From Stream Processor to a Unified Data Processing System

From Stream Processor to a Unified Data Processing System

The Apache Flink community has pushed (and continues to push) the boundary for Stream Processing over the last years, following the understanding that Stream Processing is unifying paradigm to build data processing applications, beyond real-time analytics...

Keynote
Speakers Stephan Ewen (Ververica) Xiaowei Jiang (Alibaba) Robert Metzger (Ververica) View Video & Slides
Flink Powered Customer Experience: Scaling from 5 Billion down to One

Flink Powered Customer Experience: Scaling from 5 Billion down to One

What turns a simple interaction into a great customer experience?  How do we transform a digital interaction into a personalized conversation?  Over the past year our team scaled a Customer...

Keynote
Speakers Dave Torok
(Comcast)
View Video & Slides
Screen Shot 2019-04-11 at 15.31.50

The Trade Desk's Year in Flink

At advertising technology leader, The Trade Desk, we built a data pipeline with three distinct large-scale products using Flink. The keynote gives you a peek into our journey, the lessons learned ...

Keynote
Speakers Jonathan Miles
(The Trade Desk)
View Video & Slides
Screen Shot 2019-04-09 at 12.28.39

Managing Flink on Kubernetes - FlinkK8sOperator

The goal of Lyft is to “Improve people’s lives with the world’s best transportation”. Our product is fundamentally real-time and building a reliable platform that consumes and processes massive...

Operations
Speakers Anand Swaminathan (Lyft) Ketan Umare (Lyft) View Video & Slides
How John Deere uses Flink to process millions of sensor measurements per second

How John Deere uses Flink to process millions of sensor measurements per second

The John Deere data platform receives and processes millions of sensor measurements per second from machines around the world. In this talk, we'll discuss the importance of stream data processing...

Use Case
Speakers Greg Finch (John Deere) Adam Butler (John Deere) View Video & Slides
Building a Streaming Analytics Stack with Open Source Technologies

Building a Streaming Analytics Stack with Open Source Technologies

The maturation and development of open source technologies has made it easier than ever for companies to derive insights from vast quantities of data. In this talk, we will cover how data analytic...

Technology Deep Dive
Speakers Fangjin Yang
(Imply)
View Video & Slides
Streaming for Enterprises

Streaming for Enterprises

 

 

 

 

Use Case
Speakers Srikanth Satya
(Dell EMC)
View Video & Slides
High cardinality data stream processing with large states

High cardinality data stream processing with large states

At Klaviyo, we process more than a billion events daily with spikes as high as 75,000/s on peak days. The workload is growing exponentially year over year. We migrated our legacy event...

Use Case
Speakers Ning Shi
(Klaviyo)
View Video & Slides
Future of Apache Flink Deployments: Containers, Kubernetes and More

Future of Apache Flink Deployments: Containers, Kubernetes and More

Container technology experiences an ever increasing adoption throughout many industries. Not only does this technology make your applications portable across different machines and operating...

Technology Deep Dive
Speakers Till Rohrmann
(Ververica)
View Video & Slides
How to Join Two Data Streams?

How to Join Two Data Streams?

Joins are one of the most common operations in SQL. However it is far from trivial how to express and execute them in Streaming environment with continuously running queries.During this talk...
Technology Deep Dive
Speakers Piotr Nowojski
(Ververica)
View Video & Slides
TensorFlow Extended: An end-to-end machine learning platform for TensorFlow

TensorFlow Extended: An end-to-end machine learning platform for TensorFlow

As machine learning evolves from experimentation to serving production workloads, so does the need to effectively manage the end-to-end training and production workflow including model...
Use Case
Speakers Robert Crowe
(Google)
View Video & Slides
Build a Table-centric Apache Flink Ecosystem

Build a Table-centric Apache Flink Ecosystem

Flink Table API was initially created to address the relational query use case. It has been a good addition to DataStream and DataSet API for users to write declarative queries. Moreover, Table API...

Technology Deep Dive
Speakers Shaoxuan Wang
(Alibaba)
View Video & Slides
Building Financial Identity Platform using Apache Flink

Building Financial Identity Platform using Apache Flink

To power financial prosperity around the world, Intuit needs to create personalized product experience and new data centric products. Some of these use cases include Enabling 360 Customer...

Use Case
Speakers Vivek Thakre
(Intuit.com)
View Video & Slides
Elastic Data Processing with Apache Flink and Apache Pulsar

Elastic Data Processing with Apache Flink and Apache Pulsar

More and more applications are using Flink for low-latency data processing. Flink unifies batch and stream processing using one computation engine. However in reality, in order to really unify...

Ecosystem
Speakers Sijie Guo
(Apache Pulsar)
View Video & Slides
High performance ML library based on Flink

High performance ML library based on Flink

We hope to take this opportunity to share some of our technical tips on building a high-performance library. We have implemented some common machine algorithms based on Flink and achieved...

Use Case
Speakers Xu Yang
(Alibaba)
View Video & Slides
Building production Flink jobs with Airstream at Airbnb

Building production Flink jobs with Airstream at Airbnb

AirStream is a realtime stream computation framework that supports Flink as one of its processing engines. It allows engineers and data scientists at Airbnb to easily leverage Flink to build real...

Use Case
Speakers Pala Muthiah (Airbnb) Hao Wang (Airbnb) View Video & Slides
Analytics for the masses

Analytics for the masses

One of the big operational challenges when running streaming applications is to cope with varying workloads. Variations, e.g. daily cycles, seasonal spikes or sudden events, require that allocated resources are constantly adapted. Otherwise, service quality deteriorates or money is wasted. Apache ...

Technology Deep Dive
Speakers Aslam Tajwala
(Cogilty)
View Video & Slides
Creating millions of user sessions using Complex Event Processing

Creating millions of user sessions using Complex Event Processing

Every day, Yelp connects millions of consumers with great local businesses through the website and mobile apps. We strive to provide our users with an ever-evolving, excellent experience...

Use Case
Speakers Prem Santosh Udaya Shankar
(Yelp)
View Video & Slides
Hunting for Attack Chains in Event Streams

Hunting for Attack Chains in Event Streams

Arctic Wolf Networks processes over 9 billion events a day across its customer base. These represent HTTP and DNS transactions from customer networks, log lines from customer infrastructure devices...

Use Case
Speakers Ray Ruvinskiy (Arctic Wolf Networks) Jonathan Walsh (Arctic Wolf Networks) View Video & Slides
Deploying ONNX models on Flink

Deploying ONNX models on Flink

The Open Neural Network exchange format (ONNX) is a popular format to export models to from a variety of frameworks. It can handle the more popular frameworks like PyTorch and MXNet...

Ecosystem
Speakers Isaac Mckillen-Godfried
(AI Stream)
View Video & Slides
Integrate Flink with Hive Ecosystem

Integrate Flink with Hive Ecosystem

Along with the community's effort, at Alibaba we have explored Flink's potential as an execution engine not just for stream processing but also for batch processing. The findings are...

Ecosystem
Speakers Xuefu Zhang (Alibaba) Bowen Li (Alibaba) View Video & Slides
Developing and operating real-time applications with Oceanus

Developing and operating real-time applications with Oceanus

The Tencent Data team (data.qq.com) is responsible to build a reliable and scalable infrastructure for various products at Tencent including Tencent Games, Tencent Video and WeChat...

Operations
Speakers Xiaogang Shi
(Tencent)
View Video & Slides
Apache Beam: Portability in the times of Real Time Streaming

Apache Beam: Portability in the times of Real Time Streaming

Apache Beam was open sourced by the big data team at Google in 2016, and has become an active community with participants from all over. Beam is a framework to define data processing workflows...

Ecosystem
Speakers Pablo Estrada
(Google)
View Video & Slides
Screen Shot 2019-04-09 at 12.26.05

Adventures in Scaling from Zero to 5 Billion Data Points per Day

At Flink Forward San Francisco 2018 our team at Comcast presented the operationalized streaming ML framework which had just gone into production. This year in just a few short months we...

Use Case
Speakers Dave Torok
(Comcast)
View Video & Slides
Streaming your Lyft Ride Prices

Streaming your Lyft Ride Prices

At Lyft we dynamically price our rides with a combination of various data sources, machine learning models, and streaming infrastructure for low latency, reliability and scalability ...

Use Case
Speakers Thomas Weise (Lyft) Akshay Balwally (Lyft) View Video & Slides
Screen Shot 2019-04-11 at 15.46.13

Practical Experience running Flink in Production

The stream processing platform team at Uber has been running Flink in Production for > 2 years. Currently, the platform runs and manages > 1000 jobs across multiple data centers, and powers....

Operations
Speakers Rong Ring (Uber) Shuyi Chen (Uber) View Video & Slides
Screen Shot 2019-04-11 at 15.55.35

Scaling a real-time streaming warehouse with Apache Flink, Parquet and Kubernetes

At Branch, we process more than 12 billions events per day, and store and aggregate terabytes of data daily. We use Apache Flink for processing, transforming and aggregating events, and parquet...

 

Use Case
Speakers Aditi Verma (Branch Metrics) Ramesh Shanmugam (Branch Metrics) View Video & Slides
Screen Shot 2019-04-11 at 16.02.29

Massive Scale Data Processing at Netflix using Flink

Over 137 million members worldwide are enjoying TV series, feature films across a wide variety of genres and languages on Netflix. It leads to petabyte scale of user behavior data. At Netflix, our client...

 

Use Case
Speakers Snehal Nagmote (Netflix) Pallavi Phadnis (Netflix) View Video & Slides
Using Flink to inspect live data as it flows through a data pipeline

Using Flink to inspect live data as it flows through a data pipeline

One of the hardest challenges with authoring a data pipeline in Flink is understanding what your data looks like at each stage of the pipeline. Pipeline authors would love to answer questions like...

Use Case
Speakers Matthew Dailey
(Splunk)
View Video & Slides
Towards Flink 2.0: Rethinking the stack and APIs to unify Batch & Stream

Towards Flink 2.0: Rethinking the stack and APIs to unify Batch & Stream

Flink currently features different APIs for bounded/batch (DataSet) and streaming (DataStream) programs. And while the DataStream API can handle batch use cases, it is much less efficient in that...

Technology Deep Dive
Speakers Aljoscha Krettek (Ververica) Stephan Ewen (Ververica) View Video & Slides
Moving from Lambda and Kappa Architectures to Kappa+ at Uber

Moving from Lambda and Kappa Architectures to Kappa+ at Uber

Kappa+ is a new approach developed at Uber to overcome the limitations of the Lambda and Kappa architectures. Whether your realtime infrastructure processes data at Uber scale (well over...

Technology Deep Dive
Speakers Roshan Naik
(Uber)
View Video & Slides
Build a Table-centric Apache Flink Ecosystem

When Table meets AI: Build Flink AI Ecosystem on Table API

Flink is the best engine for streaming processing and now is evolving into a streaming/batch unified platform. Table API will be the new unified high level API for both streaming and batch processing. AI is now a MUST-HAVE for every aspect of big data processing...

Technology Deep Dive
Speakers Shaoxuan Wang
(Alibaba)
View Video & Slides
Realtime Store Visit Predictions at Scale

Realtime Store Visit Predictions at Scale

This talk aims to inspire attendees with a multidisciplinary Flink application, where different fields have come together with a graceful synergy. You will hear about geospatial clustering...

Use Case
Speakers Luca Giovagnoli
(Yelp)
View Video & Slides
Screen Shot 2019-04-12 at 12.39.09

Real-time Processing with Flink for Machine Learning at Netflix

Machine learning plays a critical role in providing a great Netflix member experience. It is used to drive many parts of the site including video recommendations, search results ranking...

Use Case
Speakers Elliot Chow
(Netflix)
View Video & Slides
Screen Shot 2019-04-18 at 10.39.50

Becoming a Smooth Operator: A look at low-level Flink APIs and what they enable

As Flink has evolved, it has continued to layer on new, higher-level APIs and improve older APIs to help succinctly solve problems in different domains. However, sometimes it is useful to understand...

Technology Deep Dive
Speakers Addison Higham
(Instructure)
View Video & Slides