Watermarks in Apache Flink Made Easy
Why event-time processing needs watermarks, explained with a simple stream sorter: four observations on buffering, waiting and bounded out-of-orderness in Apache Flink.
4 min read
from engineers to engineers
Developers Center
Technical articles on stream processing, Apache Flink, architecture, and the internals that power real-time data platforms.
Latest articles
How Apache Flink manages Kafka consumer offsets
Step by step: how the Flink Kafka source stores partition offsets in checkpoints, how barriers keep operator state consistent, and how offsets are restored after a failure.
4 min read
State TTL for Apache Flink: How to Limit the Lifetime of State
How to configure state time-to-live (TTL) in Apache Flink: when the TTL is reset, whether expired state is returned, and how expired state is cleaned up.
7 min read
A Practical Guide to Broadcast State in Apache Flink
A practical guide to broadcast state in Apache Flink: broadcasting a pattern stream to all tasks and evaluating it on keyed user actions with KeyedBroadcastProcessFunction.
10 min read
End-to-End Exactly-Once Processing in Apache Flink with Apache Kafka
How Apache Flink combines checkpoints with a two-phase commit protocol to deliver end-to-end exactly-once semantics with Apache Kafka transactions.
9 min read
Large State in Apache Flink®: An Intro to Incremental Checkpointing
How Apache Flink incremental checkpoints work with RocksDB: only new SST files are uploaded, a shared state registry counts references, and old files are pruned.
10 min read
How To Size Your Apache Flink Cluster Back-of-the-Envelope Calculation
A worked example for sizing an Apache Flink cluster: start from throughput, keys and state size, then add up network, state-access and checkpoint traffic per machine.
12 min read
Apache Flink® at MediaMath: Rescaling Stateful Applications in Production
MediaMath rescaled its stateful Flink reporting job after the 2017 S3 outage. A deep dive into how Flink redistributes operator state and keyed state with key groups.
20 min read
Ufuk Celebi[1]
How Apache Flink™ handles backpressure
How Apache Flink responds to backpressure: bounded network buffer pools let a slow task throttle its upstream tasks, all the way back to a durable source, without losing data.
8 min read
High-throughput, low-latency, and exactly-once stream processing
Streaming fault-tolerance architectures compared (record acks, micro-batching, transactional updates, distributed snapshots), with Flink vs Storm throughput and latency benchmarks.
27 min read