Use Case Track  

Scaling a real-time streaming warehouse with Apache Flink, Parquet and Kubernetes

 

At Branch, we process more than 12 billions events per day, and store and aggregate terabytes of data daily. We use Apache Flink for processing, transforming and aggregating events, and parquet as the data storage format. This talk covers our challenges with scaling our warehouse, namely:

  • How did we scale our Flink-Parquet warehouse to handle 3x increase in traffic?
  • How do we ensure exactly once, event-time based, fault tolerant processing of events?

In this talk, we also provide an overview on deploying and scaling our streaming warehouse. We give an overview on:

  • How we scaled our Parquet warehouse by tuning memory
  • Running on Kubernetes cluster for resource management
  • How we migrated our streaming jobs with no disruption from Mesos to Kubernetes
  • Our challenges and learnings along the way

 

Authors

Aditi Verma
Aditi Verma
Branch Metrics

Aditi Verma

Aditi is a senior software engineer at Branch, working on developing and scaling their data platform, that processes tens of billion events per day. Prior to Branch, she worked at Yahoo to develop data systems that provide actionable insights and audience targeting from petabytes of data. She has a wide range of experience in the data domain, from stream and batch processing, to resource management, scaling and monitoring.

Ramesh
Ramesh Shanmugam
Branch Metrics

Ramesh Shanmugam

Ramesh Shanmugam is a Senior Data Platform Engineer at Branch Metrics. At Branch, currently, he is building streaming and batch pipelines at a huge scale using Apache Flink, Spark, and Airflow. He has been creating distributed applications for more than 15 years. Passionate about building data-intensive applications.