A couple of weeks back more than 400 members of the Apache Flink and stream processing communities gathered in Berlin for the 5th edition of Flink Forward Europe. If you haven’t had the chance to participate at the conference, you can find all session recordings on the Flink Forward YouTube channel and access all presentations on SlideShare.
In this post, we will recap some of the highlights of the conference and major trends we observed from the sessions and community discussions.
Witnessing an ever-growing Apache Flink adoption
Flink Forward Europe 2019 showcased, once again, the sheer growth of the Apache Flink community. The keynotes revealed an accelerated adoption, with two major vendors now offering products and services for Apache Flink. At the same time, the community pushes the boundaries of stream processing and the scope of supported use cases by expanding the capabilities of Flink’s processing engine and adding new APIs.
The keynote by Ververica’s CTO & Apache Flink PMC Stephan Ewen, announced Stateful Functions, a new open source framework that brings together the benefits of modern stream processing and Function-as-a-Service (FaaS) to provide a powerful abstraction for the next generation of event-driven architectures. Stateful Functions is an open source framework that reduces the complexity of building and orchestrating distributed stateful applications at scale. Andrew Psaltis and Marton Balassi presented how Cloudera’s Data-in-Motion platform is adopting Flink for collecting, processing and analyzing event streams. Finally, speakers from Google shared their learnings from building and operating a global streaming processing infrastructure while Rahul Pathak from AWS gave an overview of how customers and their internal use of streaming data have evolved on AWS — now providing support for Apache Flink in two services: as a hosted service in Amazon Kinesis Data Analytics for Java, and through Amazon EMR.
Breakout session highlights
During Flink Forward Europe 2019, more than 80 speakers presented their use cases, technical deep dives, Flink ecosystem and community talks. We particularly enjoyed the following sessions — among others:
-
Dongwon Kim and Haemee Park from SK Telecom showcased FLOW, a web service that lets users do FLink On Web. FLOW aims to minimize the effort of handwriting streaming applications by letting users drag and drop graphical icons representing streaming operators on GUI.
-
Steven Bairos-Novak presented Pinterest’s experience in building a self-service streaming platform, based on Apache Flink, to provide to internal teams of engineers, data analysts and data scientists a seamless job deployment pipeline, common monitoring, alerting and tooling.
-
Jesse Anderson and Hassene Ben Salem showed an interesting use case from Airbus Defense and Space and explained how they used Apache Flink to develop, deploy and release AirSense, a new generation Air Traffic Management (ATM) service that provides business insights based on real-time aircraft position data to Airbus’ internal and external customers.
-
Lakshmi Rao gave an overview of Lyft’s streaming platform, including the platform architecture, deployment model and user experience. She also presented some of the challenges and the lessons learned from running Flink jobs at scale.
-
The Panel Discussion: Building Self Service Platforms for Apache Flink provided some great tips, lessons learned and best practices for building self-service platforms for Apache Flink from companies such as AWS, Yelp and Pinterest. The participants shared their experience building both internal and public Flink-based streaming platforms. Our panelist Kenny Gorman from Eventador wrote a blog post presenting some of his highlights from Flink Forward Europe 2019.
Hot topics at the conference
From the talks and discussions, a few major themes in the streaming space in general, and for Flink specifically came up:
-
Flink SQL is on its way to becoming a truly unified batch and streaming API for all kinds of analytics workloads. There were plenty of talks on improving Flink’s batch SQL capabilities, interactive programming and adding a Python Table API to Flink.
-
The announcement of Stateful Functions and other talks in a similar direction created a lot of excitement about expanding the scope of use cases for stream processing at the conference. There is generally a lot happening in the space of application development on data streams and many attendees were interested in trying out Stateful Functions on Flink themselves.
-
Conventional stream processing with Flink was still the main topic at the conference with plenty of companies presenting their use-cases, committers explaining internals and conference sponsors presenting products built on Flink.
A more diverse and international Flink community than ever
This year’s Flink Forward in Berlin revealed a truly diverse and global community. We saw noticeable growth in the number of female attendees and speakers (20% of attendees and 12% of the speakers were women) while participants from more than 45 countries attended the conference. Such numbers showcase the community’s efforts towards becoming a more diverse and inclusive project and are supported by efforts like the Diversity Scholarship for Flink Forward that brought 25 people from underrepresented groups in the technical community. Our conference program this year was also diversified, including a research track with 8 talks from leading European Universities as well as a community track with lunch talks about how to contribute to Apache Flink and how non-code contributions can become a path to more diversity in open source software.
We were once again amazed by the number of committers, Flink PMC members, users, students and every attendee sharing their experience, best — and worst — practices with Apache Flink and how to take advantage of Flink’s unique capabilities. While we’re still excited about the three amazing days at Flink Forward Europe 2019, preparations for the events in Asia (Beijing, November 28 - 30, 2019) and North America (Spring 2020, San Francisco) are already underway! Follow @FlinkForward on Twitter and stay tuned for CFP announcements and exciting updates in our upcoming events!
We are thankful to everyone involved in making the conference a huge success!
