Key Highlights from Flink Forward Beijing 2019

On November 28-30, Beijing welcomed the 2019 Flink Forward conference for the second time. This year’s attendance number exceeded 2,000 people, an increase of nearly 100% year-on-year.

In addition to the rapid increase in the number of participants, diversification of the presented topics and use cases was another highlight from this year's event. About 25 companies from North America, Europe and Asia, universities and research institutions shared more than 45 talks. The biggest internet companies both from China and abroad gathered together to learn more about the latest developments around open source Apache Flink®. Specifically in China, we saw all the major Chinese tech companies — including Alibaba, Baidu, Tencent, DiDi and ByteDance — using Flink as a real-time computing solution, as illustrated in the picture below.

Flink Forward Beijing presented two major development directions for Apache Flink:

1. Flink's path towards unifying offline data (batch), real-time data (streaming) and event-driven applications at the execution level.

2. A more diverse Flink ecosystem including the establishment of an AI environment around Flink and support for more technology frameworks, such as Hive, Python, Kubernetes, etc.

Opening sessions

The event opened with a keynote from Alibaba Cloud Intelligence who invited Alibaba Group's CTO and Alibaba Cloud President Jeff Zhang to talk about Alibaba’s data intelligence-driven, "All in Cloud" commitment and determination. Mr. Jeff Zhang highlighted the key role that open source Apache Flink plays in this journey.

The session followed an award ceremony of the Apache Flink Geek Challenge, jointly organized by Alibaba Cloud, Tianchi Platform and Intel. The competition attracted more than 4,000 participants from around the world and resulted — through 4 months and 4 rounds of selection — in 10 final winning teams.

Hot topics and key highlights

Some of the hot topics at this year's Flink Forward Beijing were:

• Stateful Functions

The first highlight was Stateful Functions. Stephan Ewen presented how Stateful Functions enables Flink to become a general-purpose engine bridging between data processing and online applications or microservices. Stateful Functions was first introduced during Flink Forward Europe 2019 as an open source project with the intention of donating the project to Apache Flink. Currently, the Flink community discusses how to merge the code in Flink. You can access the code and documentation on the official Stateful Functions website.

• Flink's path Towards a Unified Data Processing Engine

The second highlight was Flink’s evolution into a unified framework for offline (batch), real-time (streaming) data processing and online application integration. Additional Flink directions include Flink’s development on AI infrastructure and becoming a truly cloud-native technology framework.

Specifically on the AI front, there were two important highlights. Firstly, Alibaba released Alink, an open source project on GitHub. Alink is a library of machine learning algorithms built on top of Apache Flink. You can find more information about the project on Github. Secondly, the community presented a key investment direction of Flink AI next year: AI Flow. AI Flow has customized a complete solution for AI workflow: including a complete set of solutions from data acquisition, preprocessing, to model training, model validation & serving, as well as model inference. This solution is customized to solve the problems of time-consuming data preprocessing and the mismatch between offline training and online prediction.

On the cloud-native front, some notable highlights from the conference involved the community’s efforts for a deeper integration with Kubernetes. Kubernetes is widely used in various online businesses and scenarios making a deeper integration between Flink and Kubernetes a great match for resource management at large scale.

• The emergence of Flink-based, large-scale, real-time data analytics platforms

One of the main themes from the conference was related to the emergence of various large-scale real-time, end-to-end data analytics platforms from multiple companies, such as the one developed at Lyft. The real-time data analytics platform presented by Lyft is an end-to-end platform that supports data delay of no more than 5 minutes, and is primarily used for interactive data queries at the ride-sharing company. In addition to Lyft, many companies also participated in the conference, sharing their experiences and lessons from using Flink in production.

Summary

Flink Forward Beijing 2019 revealed a truly diverse and global Flink community. Apart from the noticeable growth in the number of attendees and speakers (nearly 100% growth YoY), we saw a more diverse community with talks from all major internet players in China as well as many tech companies from North America and Europe showcasing their use of Flink and getting excited about Flink’s development in 2020. The conference program this year was also diversified, including 5 tracks with more than 45 talks related to Apache Flink Core Technology, Artificial Intelligence, Enterprise Practice, Data Warehouse with Real Time Ingestion and Open Source Big Data Ecosystem.

We were once again amazed by the number of attendees and speakers eager to share their learning and experience with Flink. Attendees were able to connect and network with other members of the Flink community and find out the key developments around open source Apache Flink®. While Flink Forward Beijing 2020 ended only a couple of weeks ago, the Call for Papers for Flink Forward San Francisco 2020 is already open. Make sure to submit your talk and share your Flink experience with other members of the community. Follow @FlinkForward on Twitter to stay tuned for all announcements and exciting updates about our upcoming events!