The Savepoint That Grew Until It Killed Itself: Reproducing a Kafka Connector Migration OOM
How a one-line connector swap silently poisoned a savepoint, why raising heap memory only bought time, and what it took to cut the stale state out.
Technical articles on stream processing, Apache Flink, architecture, and the internals that power real-time data platforms.
How a one-line connector swap silently poisoned a savepoint, why raising heap memory only bought time, and what it took to cut the stale state out.
From Kappa to Lakehouse and now Streamhouse, explore how each help address modern data challenges and unlock unified batch and stream processing systems.
Learn about Streamhouse, a data processing pattern that combines real-time stream processing with the structured nature of data warehouses.
Step by step: how the Flink Kafka source stores partition offsets in checkpoints, how barriers keep operator state consistent, and how offsets are restored after a failure.
How Apache Flink combines checkpoints with a two-phase commit protocol to deliver end-to-end exactly-once semantics with Apache Kafka transactions.
How Apache Flink incremental checkpoints work with RocksDB: only new SST files are uploaded, a shared state registry counts references, and old files are pruned.
MediaMath rescaled its stateful Flink reporting job after the 2017 S3 outage. A deep dive into how Flink redistributes operator state and keyed state with key groups.
How Apache Flink responds to backpressure: bounded network buffer pools let a slow task throttle its upstream tasks, all the way back to a durable source, without losing data.
Streaming fault-tolerance architectures compared (record acks, micro-batching, transactional updates, distributed snapshots), with Flink vs Storm throughput and latency benchmarks.