Use Case Track  

Real-time Processing with Flink for Machine Learning at Netflix

 

Machine learning plays a critical role in providing a great Netflix member experience. It is used to drive many parts of the site including video recommendations, search results ranking, and selection of artwork images. Providing high-fidelity, near real-time data is increasingly important for these machine learning pipelines, especially as multi-armed bandit and reinforcement learning techniques, in addition to more ""traditional"" supervised learning, become more prevalent. With access to this data, models are able to converge more quickly, features can be updated more frequently, and analysis can be done in a more timely manner.

 

In this talk, we will focus on the practical details of leveraging Flink to process trillions of events per day, work with the time dimension, and manage large and frequently-changing state. We will discuss different processing schemes and dataflows, scalability and resiliency challenges we tackled, operational considerations, and instrumentation we added for monitoring job health in production.

Authors

elliot-chow copy
Elliot Chow
Netflix

Elliot Chow

Elliot is a software engineer at Netflix on the Personalization Infrastructure team. He builds big data systems tailored for machine learning and analytics use cases with a variety of technologies including Scala, Spark, Flink, Kafka, and Cassandra.