Technology Deep Dive Track  

High performance ML library based on Flink

We hope to take this opportunity to share some of our technical tips on building a high-performance library. We have implemented some common machine algorithms based on Flink and achieved comparable or better performance than SparkML. In the development of the algorithm library, we also accumulated intermediate function libraries and tools. Now, we are working with Flink community to contribute back our work to FlinkML to provide more efficient ML algorithms for Flink users. Flink developers can quickly develop more Flink ML algorithms based on our intermediate libraries and tools.

Authors

Xu Yang
Xu Yang
Alibaba

Xu Yang

Hao is a software engineer on Airbnb’s Data Platform team. He has been leading the development and driving the adoption of real-time stream processing at Airbnb. Before that, he helped building the data and machine learning infrastructure for IBM Watson. He received his PhD from the University of Southern California.