Skip to content

Ververica Cloud, a fully-managed cloud service for stream processing!

Learn more

Dataflow proposed as Apache Incubator project


by

Today, we are delighted to see Dataflow publicly proposed as an Apache Incubator project with an initial project team from Google, data Artisans, Cloudera, and others. data Artisans is the second biggest contributing organization after Google with a total of 4 proposed initial committers.

Dataflow stems from the Google’s growing internal needs for a better programming model and runtime engine for data intensive applications. While Google originally developed MapReduce (which was the blueprint for Apache Hadoop), they very quickly saw the need for high-volume stream processing, as well as the need for integrating stream and batch processing. Dataflow is the successor to many internal Google projects, including MapReduce, FlumeJava, and MillWheel.

One difference between Dataflow and earlier Google systems is that it is actually available to the public, not only to Google engineers. The Dataflow SDK is open source (Apache 2.0 License). Programs written in this SDK can run locally, at Google Compute Engine using Google’s proprietary cloud service, or one of the currently available open source runners, namely Apache Flink™ and Apache Spark. The SDK together with these runners constitute the code that is proposed to become an incubating project.

apache dataflow

The streaming models of both Flink and Dataflow are based on the Dataflow model, with very similar features such as support for event time and out of order streams, a mechanism to define custom windows based on window assigners and triggers. These features distinguish Flink and Dataflow from older-generation streaming systems. The Dataflow Flink runner allows users to execute both batch and streaming Dataflow programs on an open source engine using alternative cloud providers to Google, or even on premise. The runner also supports all Flink data sources (e.g., Apache Kafka), enabling a better integration of the Dataflow model with the open source ecosystem.>

At data Artisans, we are the primary developers of the Dataflow Flink runner, and have already contributed to the Dataflow SDK. We are looking forward to continue driving this effort forward as committers of the proposed Apache project, if accepted for incubation. Most importantly, we share a common vision with the creators of Dataflow: we believe that the future of writing data applications is a streaming programming model, and Flink and Google Cloud Dataflow are the best embodiments of this model right now, one as an open source Apache project, and the other as a Google-hosted cloud service.>

By bringing Dataflow and the Flink Dataflow runner to the Apache family, this relationship will be further strengthened and accelerated. We believe that evolution within the Apache ecosystem will bring Flink and Dataflow further together, leading eventually to a unified programming model for both batch and stream data processing applications. We are looking forward to continue working with Google, the Flink community, and the future Dataflow community to help companies realize their data infrastructure based on modern streaming technology.

Ververica Academy

Kostas Tzoumas
Article by:

Kostas Tzoumas

Comments

Our Latest Blogs

Streamhouse Unveiled featured image
by Jing Ge 17 November 2023

Streamhouse Unveiled

Apache Flink: History of Reliability Every year, Apache Flink® sets new records in its development journey. Standing as a testament to its growing popularity, Flink now boosts over 1.6k contributors,...
Read More
Highlights from Flink Forward Seattle 2023 featured image
by Karin Landers 16 November 2023

Highlights from Flink Forward Seattle 2023

Now that Flink Forward Seattle 2023 is over, we’re excited to share that the event was a big (Flinking) success! Flink Forward is *the* conference dedicated entirely to Apache Flink®, and includes...
Read More
Join me at Flink Forward! featured image
by Jing Ge 12 October 2023

Join me at Flink Forward!

Real-time data streaming is a HOT topic, which comes as no surprise considering how stream processing helps companies create new business opportunities, strengthens their advantage over competitors,...
Read More