From Stream Processor to a Unified Data Processing System
The Apache Flink community has pushed (and continues to push) the boundary for Stream Processing over the last years, following the understanding that Stream Processing is unifying paradigm to build data processing applications, beyond real-time analytics.
The latest major effort in the Flink community is nothing less then re-architecting the API and runtime stack, with the goal to naturally support the spectrum of analytics and data-driven applications, to unify the APIs for batch and streaming (Table API and DataStream API), and to build a streaming runtime that is not only state-of-the-art in stream processing, but also in batch processing performance.
In this keynote, we give an overview of the goals and technology behind the above effort, and look at the adoption of Apache Flink for Stream Processing and "beyond streaming" use cases, as well as various efforts in the community to support the growth in users, applications, and ecosystem.
Stephan Ewen is CTO and co-founder at Ververica where he leads the development of the stream processing platform based on open source Apache Flink. He is also a PMC member and one of the original creators of Apache Flink. Before working on Apache Flink, Stephan worked on in-memory databases, query optimization, and distributed systems. He holds a Ph.D. from the Berlin University of Technology.
Xiaowei Jiang is a Senior Director at Alibaba. He leads the StreamCompute Platform for AliCloud. The platform provides the real time data processing both internally and externally. Previously, he worked as Tech Lead at Facebook and Principal Engineer at Microsoft SQL Server.
Robert Metzger is a PMC member of the Apache Flink project and a co-founder and an engineering lead at Ververica (former data Artisans). He is the author of many Flink components including the Kafka and YARN connectors. Robert studied Computer Science at TU Berlin and worked at IBM Germany and at the IBM Almaden Research Center in San Jose. He is a frequent speaker at conferences such as the Hadoop Summit, ApacheCon and meetups around the world.