Technology Deep Dive Track  

Becoming a Smooth Operator: A look at low-level Flink APIs and what they enable

 

As Flink has evolved, it has continued to layer on new, higher-level APIs and improve older APIs to help succinctly solve problems in different domains. However, sometimes it is useful to understand what is happening at a low-level and possibly even use the lower level constructs to enable complex functionality that isn't otherwise possible.

 

In this talk, we will dig into the internals of the low-level StreamOperator API and understand what this API provides and how Flink uses and builds upon this functionality in other layers. Additionally, we will look at a real-life use case at Instructure where the StreamOperator API was used to build a system that reads an archived stream from files and then seamlessly transitions to a live stream in the same Flink job, something not readily possible with the more well known Flink APIs.

Authors

Addison Higham
Addison Higham
Instructure

Addison Higham

Addison Higham a software engineer at Instructure, focused on building out a suite of infrastructure and tools for large scale stream processing to enable real-time learning analytics for student, teachers and administrators.