Broadcast State Pattern in Apache Flink: 4 important considerations

November 09, 2018 | by Markos Sfikas

This post explores the Broadcast State pattern that was introduced in Apache Flink 1.5.0. In the following sections, we describe what is a Broadcast State Pattern, how Broadcast State differs from other types of operator state, and finally, we go over some important considerations when using the feature in Apache Flink.

What is a Broadcast State Pattern?

The Broadcast State Pattern refers to streaming applications where a low-throughput event stream (that, for example, contains a set of rules) is broadcasted to all parallel instances of an operator and is then evaluated against all elements coming from another event stream with raw data (for example financial or credit card transactions). Some motivating use cases for the Broadcast State Pattern are the following:

  • Application of rules from a low-throughput event stream to raw incoming data: for example, broadcasting the rule that an alert should be sent when a transaction value is greater than $1 million to all parallel tasks of your operator.

  • Data Enrichment: for example, enriching a stream of transactions that contain a user ID, with the broadcasted data associated with that user ID.

To allow for such applications, a critical component is the Broadcast State, which we describe below.

What is Broadcast State?

The Broadcast State is the third supported type of operator state in Apache Flink. Broadcast State enables Flink users to store in a fault-tolerant and re-scalable way the elements from the broadcasted, low-throughput event stream (see examples above). Events from the second stream can then flow through the individual instances of the same operator that processes them together with the events in the broadcast state. For more information on the other types of state and how to use them visit the Flink documentation here.
There are three main differences between Broadcast State and the rest of operator states. Contrary to the remaining types of operator state, the Broadcast State:

  1. has a map format,

  2. has as input a broadcasted event stream, and

  3. the operator tasks can have multiple Broadcast States with different names.

You can check our previous blog post to explore a step-by-step practical guide to Broadcast State in Apache Flink

Important considerations

For Flink users keen to get started with Broadcast State, the Apache Flink documentation provides details about the APIs and how to use the feature in your pipelines.

There are 4 important things to keep in mind when using Broadcast State:

  • With Broadcast State, operator tasks do not communicate with each other

This is the reason why only the broadcast side of a (Keyed)-BroadcastProcessFunction can modify the contents of the Broadcast State. The user should ensure that all operator tasks modify the contents of the Broadcast State in the same way for each incoming element. Alternatively, different tasks might have different contents, leading to inconsistent results.

  • The order of events in Broadcast State may differ across tasks

Although broadcasting the elements of a stream guarantees that all elements will (eventually) go to all downstream tasks, elements may arrive in a different order to each task. As a result, any state update for each incoming element should not depend on the ordering of the incoming events.

  • All operator tasks checkpoint their Broadcast State

Upon checkpointing, all tasks checkpoint their Broadcast State, and not just one of them, even though all tasks store the same elements in their Broadcast State. This is a design decision to avoid reading from a single file during a restore and consequently avoiding hotspots. However, there is a tradeoff of increasing the size of the checkpointed state by a factor p (= parallelism). Flink guarantees that upon restoring/rescaling there will be neither duplicates nor missing data. In case of recovery with the same or smaller parallelism, each task reads its checkpointed state. Upon scaling up, each task reads its own state, and the remaining tasks (p_new-p_old) read checkpoints of previous tasks in a round-robin approach.

  • RocksDB state backend is not available for Broadcast State

Broadcast State is kept in memory at runtime. Since currently, the RocksDB state backend is not available for operator state, Flink users should arrange their application’s memory provisioning accordingly for all operator states.

Sign up to Apache Flink Public Training in a location near you for a comprehensive introduction to stateful stream processing with Apache Flink or contact us with your questions and recommendations below.

Public Training Flink, Flink training, Ververica training, Apache Flink training

Ververica Contact




Topics: Apache Flink, Flink Features

Markos Sfikas
Article by:

Markos Sfikas

Related articles


Sign up for Monthly Blog Notifications

Please send me updates about products and services of Ververica via my e-mail address. Ververica will process my personal data in accordance with the Ververica Privacy Policy.

Our Latest Blogs

by Chen Qin September 21, 2021

The Apache Flink Story at Pinterest - Flink Forward Global 2021

On October 27, at the annual Apache Flink user conference, Flink Forward Global 2021, Pinterest Tech Lead, Chen Qin will deliver a keynote talk on “Sharing what we love: The Apache Flink Story at...

Read More
by Holger Temme August 16, 2021

Ververica named a 'Strong Performer' in Streaming Analytics by Forrester

We are excited to see Ververica Platform, developed by the original creators of Apache Flink, debut on the Forrester Wave™ 2021: Streaming Analytics report as a Strong Performer! Back in 2019,...

Read More
by Victor Xu July 13, 2021

Troubleshooting Apache Flink with Byteman


What would you do if you need to see more details of some Apache Flink application logic at runtime, but there's no logging in that code path? An option is modifying the Flink source...

Read More