Technology Deep Dive Track  

When Table meets AI: Build Flink AI Ecosystem on Table API

 

Flink is the best engine for streaming processing and now is evolving into a streaming/batch unified platform. Table API will be the new unified high level API for both streaming and batch processing. AI is now a MUST-HAVE for every aspect of big data processing. However, till now, AI is still the weakest point of Flink ecosystem.

 

We are trying to combine the best of both to create Flink's AI ecosystem based on Table API. In this talk, we will talk about our work on new ML Pipeline to create interoperability standard for Table API and how it can form a base for bigger Flink AI ecosystem. And we will talk about our work on TensorFlow on Flink to provide native TensorFlow experience for both training and inference and seamless data communication between Flink and TensorFlow. And we will demonstrate how to perform end-to-end AI training/inference using the combination of these two technologies.

Authors

shaoxuan_profile_pic-1
Shaoxuan Wang
Alibaba

Shaoxuan Wang

Shaoxuan is a senior staff engineer and director of engineering at Alibaba, working on Flink SQL and AI platform. Alibaba runs Blink, a fork of Flink, at very large scale and offers it as a service to internal and cloud customers. Prior to Alibaba, Shaoxuan was a senior software engineer working on social graph and core infrastructure at Facebook. Shaoxuan received his Ph.D degree from UC San Diego. He is an Apache Flink® committer. Shaoxuan and his colleagues are heavily contributing to Apache Flink®.