Key Highlights from Flink Forward Beijing 2019

December 05, 2019 | by Yuan Mei & Markos Sfikas

On November 28-30, Beijing welcomed the 2019 Flink Forward conference for the second time. This year’s attendance number exceeded 2,000 people, an increase of nearly 100% year-on-year.

Flink-Forward-Asia-2019-Banner

In addition to the rapid increase in the number of participants, diversification of the presented topics and use cases was another highlight from this year's event. About 25 companies from North America, Europe and Asia, universities and research institutions shared more than 45 talks. The biggest internet companies both from China and abroad gathered together to learn more about the latest developments around open source Apache Flink®. Specifically in China, we saw all the major Chinese tech companies — including Alibaba, Baidu, Tencent, DiDi and ByteDance — using Flink as a real-time computing solution, as illustrated in the picture below.

Flink-Forward-Asia-2019-China-company-logos

Flink Forward Beijing presented two major development directions for Apache Flink:

  1. Flink's path towards unifying offline data (batch), real-time data (streaming) and event-driven applications at the execution level.

  2. A more diverse Flink ecosystem including the establishment of an AI environment around Flink and support for more technology frameworks, such as Hive, Python, Kubernetes, etc.

 

Opening sessions

The event opened with a keynote from Alibaba Cloud Intelligence who invited Alibaba Group's CTO and Alibaba Cloud President Jeff Zhang to talk about Alibaba’s data intelligence-driven, "All in Cloud" commitment and determination. Mr. Jeff Zhang highlighted the key role that open source Apache Flink plays in this journey.

Flink-Forward-Asia-Alibaba-CTO

The session followed an award ceremony of the Apache Flink Geek Challenge, jointly organized by Alibaba Cloud, Tianchi Platform and Intel. The competition attracted more than 4,000 participants from around the world and resulted — through 4 months and 4 rounds of selection — in 10 final winning teams.

Flink-Forward-Asia-2019-Geek-Challenge-Winners

 

Hot topics and key highlights

Some of the hot topics at this year's Flink Forward Beijing were:

 

  • Stateful Functions

The first highlight was Stateful Functions. Stephan Ewen presented how Stateful Functions enables Flink to become a general-purpose engine bridging between data processing and online applications or microservices. Stateful Functions was first introduced during Flink Forward Europe 2019 as an open source project with the intention of donating the project to Apache Flink. Currently, the Flink community discusses how to merge the code in Flink.  You can access the code and documentation on the official Stateful Functions website.

Flink-Forward-Asia-2019-Stateful-Functions

 

  • Flink's path Towards a Unified Data Processing Engine

The second highlight was Flink’s evolution into a unified framework for offline (batch), real-time (streaming) data processing and online application integration. Additional Flink directions include Flink’s development on AI infrastructure and becoming a truly cloud-native technology framework.

Specifically on the AI front, there were two important highlights. Firstly, Alibaba released Alink, an open source project on GitHub. Alink is a library of machine learning algorithms built on top of Apache Flink. You can find more information about the project on Github. Secondly, the community presented a key investment direction of Flink AI next year: AI Flow. AI Flow has customized a complete solution for AI workflow: including a complete set of solutions from data acquisition, preprocessing, to model training, model validation & serving, as well as model inference. This solution is customized to solve the problems of time-consuming data preprocessing and the mismatch between offline training and online prediction.

On the cloud-native front, some notable highlights  from the conference involved the community’s efforts for a deeper integration with Kubernetes. Kubernetes is widely used in various online businesses and scenarios making a deeper integration between Flink and Kubernetes a great match for resource management at large scale.

Flink-Forward-Asia-2019-Flink-Integration-K8s

 


  • The emergence of Flink-based, large-scale, real-time data analytics platforms

One of the main themes from the conference was related to the emergence of various large-scale real-time, end-to-end data analytics platforms from multiple companies, such as the one developed at Lyft. The real-time data analytics platform presented by Lyft is an end-to-end platform that supports data delay of no more than 5 minutes, and is primarily used for interactive data queries at the ride-sharing company. In addition to Lyft, many companies also participated in the conference, sharing their experiences and lessons from using Flink in production.

Flink-Forward-Asia-2019-Lyft-real-time-data-platform


 

Summary

Flink Forward Beijing 2019 revealed a truly diverse and global Flink community. Apart from the noticeable growth in the number of attendees and speakers (nearly 100% growth YoY), we saw a more diverse community with talks from all major internet players in China as well as many tech companies from North America and Europe showcasing their use of Flink and getting excited about Flink’s development in 2020. The conference program this year was also diversified, including 5 tracks with more than 45 talks related to Apache Flink Core Technology, Artificial Intelligence, Enterprise Practice, Data Warehouse with Real Time Ingestion and Open Source Big Data Ecosystem. 

We were once again amazed by the number of attendees and speakers eager to share their learning and experience with Flink. Attendees were able to connect and network with other members of the Flink community and find out the key developments around open source Apache Flink®. While Flink Forward Beijing 2020 ended only a couple of weeks ago, the Call for Papers for Flink Forward San Francisco 2020 is already open. Make sure to submit your talk and share your Flink experience with other members of the community. Follow @FlinkForward on Twitter to stay tuned for all announcements and exciting updates about our upcoming events!

[2019-12]-CTA-Submit your talk-FLINKFORWARD[2019-12]-CTA-Follow-twitter-FLINKFORWARD

 

 

 

 

 

Topics: Flink Forward

Related articles

Comments

Sign up for Monthly Blog Notifications

Please send me updates about products and services of Ververica via my e-mail address. Ververica will process my personal data in accordance with the Ververica Privacy Policy.

Our Latest Blogs

by Seth Wiesman November 11, 2020

Flink Forward Global 2020 Recap

On October 19-22, 2020 we virtually welcomed more than 1,500 members of the Apache Flink® community at Flink Forward Global 2020. The second virtual Flink Forward featured several exciting keynote...

Read More
by Konstantin Knauf November 09, 2020

Ververica Platform 2.3: Getting Started with Flink SQL on Ververica Platform

Flink SQL is one of the three main programming interfaces of Apache Flink.  It is declarative and, because the community kept the syntax inline with the SQL standard, anyone who knows traditional...

Read More

How mitigating event-time skewness can reduce checkpoint failures and task manager crashes

Introduction

Support for event time and stateful event processing are some of the features that make Apache Flink stand out compared to other stream processors. However, using event time may cause...

Read More