Technology Deep Dive Track
Becoming a Smooth Operator: A look at low-level Flink APIs and what they enable
As Flink has evolved, it has continued to layer on new, higher-level APIs and improve older APIs to help succinctly solve problems in different domains. However, sometimes it is useful to understand what is happening at a low-level and possibly even use the lower level constructs to enable complex functionality that isn't otherwise possible.
In this talk, we will dig into the internals of the low-level StreamOperator API and understand what this API provides and how Flink uses and builds upon this functionality in other layers. Additionally, we will look at a real-life use case at Instructure where the StreamOperator API was used to build a system that reads an archived stream from files and then seamlessly transitions to a live stream in the same Flink job, something not readily possible with the more well known Flink APIs.
Addison Higham a software engineer at Instructure, focused on building out a suite of infrastructure and tools for large scale stream processing to enable real-time learning analytics for student, teachers and administrators.