Apache Flink Meetups

Streaming from Iceberg Data Lake & Multi Cluster Kafka Source

Apache Iceberg brings numerous benefits like snapshot isolation, transactional commit, fast scan planning and time travel support. Those features solved important correctness and performance challenges for batch processing use cases. While originally adopted for batch, Iceberg can be leveraged as a streaming source. Streaming reads can further reduce the processing delay from hours to minutes compared to periodically scheduled batch ETL jobs.

In this talk, we are going to discuss how the Flink Iceberg source enables streaming reads from Iceberg tables, where long-running Flink jobs continuously poll and process data as soon as committed. We will discuss the design of the source operator focusing in particular on the streaming read mode. We will compare the Kafka and Iceberg sources for streaming read, and discuss how the Iceberg streaming source can power common stream processing use cases. Finally, we will present performance evaluation results of the Iceberg streaming read.

-------------

Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure.

In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover.

Register here->

Netflix and Pinterest on Apache Flink in their business models

Building a Complete, Real-time Timeline of User Activities at Netflix with Flink

Netflix strives to bring joy to its customers with personalized and relevant content. Proving implicit and explicit functions to understanding user behavior. These functions have contributed to building high-quality recommendation systems models.

Join our Meetup on May 23rd at 2pm PDT to learn:

•    How Flink Redis and Cassandra constructs a timeline to user activity in real time
•   A deep-dive into RockDB backed by Flink state and coupled with Redis as a data-serving layer
•   A discussion of several performance optimizations done to operate the system at Netflix scale
•   Challenges we faced around supporting as-fast-as-possible updates, while minimizing duplicate writes and dropped messages
•   A view on general scaling issues

Flink-powered stream processing at Pinterest

Find out how Pinterest is utilizing Flink-powered stream processing to power its engine. Pinterest has onboarded 90+ applications that significantly transformed data processing and serving from batch to (near) real-time flavor in a range of use cases from Experiment Analytics, Metrics Aggregation, Ad Reporting, Shopping Catalog Ingestion, Rich Content Signal Generation and ML inference. Join our meetup to discover how Pinterest provides a buildable and deployable skeleton project (code / test / configs) with job developers plugging in their business logic with Ververica Flink. With CI/CD, load testing framework and job debugging tools, Pinterest is able to significantly reduce the time from prototype to production with sufficient job validation and minimal human efforts involved.

Upstream Processing with Apache Flink for Immediate Intelligence with Apache Druid

Applications built with Apache Druid databases are revolutionary: a real-time event hub and processor such as Apache Flink is an essential component that keeps the user up-to-date and the statistics meaningful. Peter Marshall (Sr. Developer Advocate in Imply’s Community Team) talks about the importance of fast upstream processing and delivery to modern “immediate intelligence” applications, drilling into some examples of work done upstream of Apache Druid.

Register!

Apache Flink 1.15 AMA

Apache Flink 1.15 is coming soon! Are you ready?

Build your knowledge on Apache Flink; Ease of operations, streaming, SQL, the unification of stream and batch + much more!

Join us on March 30th at 12pm EDT / 5pm CET as we kick off with a short overview of the recent additions and how this will pay into the overarching strategy of Apache Flink. Followed by an AMA with some of the engineers that contributed to this release (Disclaimer: they have broad knowledge, but aren’t THE experts in all regards), so prepare your questions.)

In conclusion of the AMA the Ververica Product Manager team will give an outlook of the future next steps.

Register->

January 2021 Apache Flink Meetup Hosted by Netflix

It is with great pleasure that we announce the January 2021 Apache Flink meetup. The virtual event is hosted by Netflix, with an exciting lineup of topics and speakers from Ververica, Lyft, and Netflix. The event also provides great opportunities to mingle with others who share the same passion.

Agenda:
11:00 - 11:20: Building an End-to-End Analytics Pipeline with PyFlink
(Marta Paes - Ververica)

11:20 - 11:40: Flink Empowered Near Real-time Streaming Data Processing at Lyft
(Ying Xu/Yash Kumaraswamy - Lyft)

11:40 - 11:45: Break

11:45 - 12:05: Backfilling stateful Flink pipelines at a fractional cost using Iceberg
(Sundaram Ananthanarayanan - Netflix)

12:05 - 12:30: Breakout Q&A session with your favorite speaker/topic

Ask-Me-Anything: Apache Flink 1.12

We’re excited to host an ask-me-anything (AMA) session with Flink PMCs and committers Aljoscha Krettek, Timo Walther, Stephan Ewen, and Arvid Heise moderated by Flink PMC, Robert Metzger! Come join us for the last community event of 2020 and chat about the upcoming 1.12 release.

📆 Agenda:
7 pm - 7:15 pm Welcome & preview of the release
7:15 pm - 7:45 pm AMA with Aljoscha Krettek, Timo Walther, Arvid Heise, Stephan Ewen, moderated by Robert Metzger
7:45 pm - 8:30 pm Hangout/networking on Spatial Chat

What's New in Flink 1.11?

Join this virtual meetup to learn about the latest updates in Flink 1.11, including the following features:

- Unified Source API
- Support for CDC with Debezium and a new FileSystem Connector for Table API/SQL
- Support for Pandas UDFs in PyFlink

The event is hosted by the Flink meetup in Seattle in collaboration with Ververica, the original creators of Apache Flink.

AGENDA
-----------------
- "What's new in Apache Flink 1.11?" by Marta Paes Moreira, Ververica
- "Ask-Me-Anything" with Flink PMC Aljoscha Krettek, Ververica

20:00 ~ 20:20 "Data driven matchmaking streaming at HyperConnect" by Jaehyeuk Oh, HyperConnect (in Korean)

Apache Flink Seoul Meetup

Join us for the first virtual meetup for the Apache Flink community in South Korea! Talks will be in Korean and English.

The event is hosted by the Flink meetup in Seoul in collaboration with Ververica, the original creators of Apache Flink.

[Session Program]

19:00 ~ 19:20 "Introduction to Stateful Functions" by Tzu-Li (Gordon) Tai, Ververica (in English)

19:30 ~ 19:50 "Flink on Kubernetes Operator" by Eui Heo, SK Telecom (in Korean)

20:00 ~ 20:20 "Data driven matchmaking streaming at HyperConnect" by Jaehyeuk Oh, HyperConnect (in Korean)

WeAreDevelopers Live: Introduction to Stream Processing with Apache Flink

Tune in to the online event organized as part of the WeAreDevelopers Live series to learn the basic concepts of streaming with Flink and walk through the main features and abstractions that give it a competitive edge over similar frameworks.

June 2019 Bay Area Apache Flink Meetup

Join the upcoming Flink Meetup in San Francisco hosted by Zendesk! Hear the Flink use case at Pinterest by Parag Kesar and Ben Liu and the Flink with Fasttext talk by Ken Krugler.

Webinar: Deep Dive on Apache Flink State

Apache Flink is a world class stateful stream processor presents a huge variety of optional features and configuration choices to the user. Determining out the optimal choice for any production environment and use-case be challenging. In this talk, we will explore and discuss the universe of Flink configuration with respect to state and state backends.

We will start with a closer look under the hood, at core data structures and algorithms, to build the foundation for understanding the impact of tuning parameters and the costs-benefit-tradeoffs that come with certain features and options. In particular, we will focus on state backend choices (Heap vs RocksDB), tuning checkpointing (incremental checkpoints, ...) and recovery (local recovery), serializers and Apache Flink's new state migration capabilities.

Beam Summit Europe 2019

Beam Summit Europe is a 2-day, multitrack event held on June 19-20 which brings together experts on Beam, new contributors and other participants interested in learning more about Apache Beam.

Check out the Apache Flink + Apache Beam talks and register for the event here. Registration is free.

Berlin Buzzwords 2019

Berlin Buzzwords is a conference on storing, processing, streaming and searching large amounts of digital data. It focuses on open source software projects. The 10th edition of Berlin Buzzwords will be held on June 16-18, 2019 at Kulturbrauerei.

Check out the Flink talks by Lyft, Qualys, and Ververica. 

GOTO Amsterdam 2019

GOTO Conference is a software development conference designed for developers, team leads, architects, and project managers and is organized "for developers by developers".

Drop by our sponsor booth on June 18-20 and hear the "Introduction to Stream Processing with Apache Flink" talk by Robert Metzger, Apache Flink PMC and Co-founder & Engineering Lead at Ververica, on June 19.

Streaming at King, Migrating batch-ETLs to streaming Flink

Join the upcoming Flink meetup in Stockholm to hear the talks on streaming at King & migrating batch-ETLs to streaming Flink. Pizza, beer, and other drinks & snacks will be provided by the event host.

Stream Processing Beyond Streaming Data at WeAreDevelopers 2019

Hear Stephan Ewen, CTO at Ververica & Apache Flink PMC member, explore how stream processing is moving towards a "grand unifying paradigm" for data processing and the efforts of the Flink community to build a unified stream processor data processing and data-driven applications.

Let's talk about "Stream Processing with Apache Flink"!

Meet the authors of the "Stream Processing with Apache Flink" book and take the opportunity to win a free copy, signed by Fabian Hueske and Vasia Kalavri.

[Flink + AWS Kinesis Analytics] Running Flink applications in Kinesis Analytics

Join the fun evening full of interesting talks & conversations, free food and drinks! Hear talks by speakers from AWS Kinesis and Alibaba.