Monitoring Large-Scale Apache Flink Applications, Part 1
Learn best practices for monitoring large-scale Apache Flink applications, focusing on key metrics, troubleshooting, and effective tools for continuous monitoring.
Technical articles on stream processing, Apache Flink, architecture, and the internals that power real-time data platforms.
Learn best practices for monitoring large-scale Apache Flink applications, focusing on key metrics, troubleshooting, and effective tools for continuous monitoring.
Step by step: how the Flink Kafka source stores partition offsets in checkpoints, how barriers keep operator state consistent, and how offsets are restored after a failure.
How Apache Flink combines checkpoints with a two-phase commit protocol to deliver end-to-end exactly-once semantics with Apache Kafka transactions.
A worked example for sizing an Apache Flink cluster: start from throughput, keys and state size, then add up network, state-access and checkpoint traffic per machine.
MediaMath rescaled its stateful Flink reporting job after the 2017 S3 outage. A deep dive into how Flink redistributes operator state and keyed state with key groups.
Streaming fault-tolerance architectures compared (record acks, micro-batching, transactional updates, distributed snapshots), with Flink vs Storm throughput and latency benchmarks.