Our successful journey with Flink
At Trackunit we have based our telematic IoT processing pipeline on Flink. We started out on version 1.2 and are now on 1.5. In this session I will share the lessons learned going from one giant Flink job to many smalls and some of the problems we have seen operating Flink on AWS EMR cluster, including topics such as:
• Why external enrichment can be challenging with Flink Async operator.
• Pattern to change external enrichment into streaming join.
• Building your own source
• Why Flink restart is great but should be avoided as it will terminate your cluster.
• Why iteration can cause deadlocking when backpressure occurs.
• Kinesis rate exceeded exception
• Why throttling Flink source read during catchup is needed.
• Why we moved from EMR/Kinesis and into DC/OS and kafka.
• And much more.
Lasse NedergaardSystem Architect Trackunit A/S
Lasse Nedergaard System Architect @ Trackunit. More details at https://www.linkedin.com/in/lassenedergaard/