Announcing Early Access Program for Flink SQL in Ververica Platform

July 06, 2020 | by Konstantin Knauf

Been wondering what's next for Ververica Platform? Maybe you've already guessed: Flink SQL is coming to Ververica Platform later this year! Today we are excited to announce our Early Access Program for Flink SQL in Ververica Platform with selected users and partners.

Ververica Platform SQL, Flink SQL, Streaming SQL, SQL, Flink, Apache Flink

Over the years, we have helped support many large-scale Apache Flink adopters such as Netflix, Alibaba and Lyft as they developed in-house Apache Flink-based low- or no-code streaming platforms. A decisive factor in choosing a low- or no-code solution has always been to increase developer efficiency and make streaming data accessible to a larger set of developers and data scientists within their organization. In this process, Flink SQL has emerged as the de facto standard for low-code data analytics and processing across both bounded and unbounded data. It is highly performant, standard-compliant and feature-rich.

Over the last few months, we have been working to extend Ververica Platform to become an integrated environment for Flink SQL. You will be able to manage your tables, functions and catalogs, develop SQL scripts and operate and monitor the derived continuous queries directly in the Platform. We would now like to involve the wider Apache Flink community in our development process through our early access program: interested companies and partners can sign up for our exclusive program and receive access to our product offering around Flink SQL before it becomes generally-available in fall 2020. Besides access to this pre-release version of Ververica Platform, selected companies will benefit from direct access to Ververica’s product team and will have a direct impact on the roadmap of the Ververica Platform SQL offering leading up to general availability later this year. 

If you want to get a first look at Flink SQL in Ververica Platform, make sure to sign up and express your interest today. Over the next few weeks, we will go through a selection process and schedule introductory calls with all selected users. 

 

What can you expect from the program?

  • Pre-release access to Ververica Platform SQL including regular product updates

  • Direct access to Ververica’s product team and regular feedback sessions

  • Shared Slack channel for Q&A, feedback and support

SQL EAP GRAPH copyFigure 1: The SQL Early Access Program at a glance

Since the initial release of Flink SQL in 2016 (Flink 1.1.0) the Apache Flink community has put countless development hours into extending, evolving and hardening the API. And as more and more language features, connectors and formats have been introduced, the set of use cases for Flink SQL has naturally grown, too. So, let’s have a look at three of the most popular usage scenarios for Flink SQL today.

 

Materialized View Maintenance

A materialized view is a pre-computed data set derived from a SQL query. As such, materialized views can greatly speed up query processing and reduce load on source systems, such as legacy mainframe systems. On the other hand, they need to be actively maintained by the SQL engine, a process that is very closely related to stream processing. With Flink SQL you will benefit from our years of experience in the field of stateful stream processing leading us towards a materialized view maintenance engine that goes well beyond what traditional relational databases and data warehouses offer today:

  • very low latency, high resource efficiency & scalability

  • strong end-to-end consistency guarantees and a mechanism to trade off latency & completeness depending on your requirements

  • expressive language features: various streaming joins, user-defined (aggregate/table) functions, stream-table joins

  • easy interoperability (e.g. out-of-the-box support for Debezium & Canal CDC formats) and operational flexibility

Materialized View Maintenance with Flink SQL, Materialized View Maintenance, Materialized Views, Flink SQL, RDBMS, Relational DatabasesFigure 2: Materialized View Maintenance with Flink SQL

 

Streaming ETL

Processing and moving real-time data from one place to another (streaming ETL) is a common use case for Apache Flink adopters.

With Flink SQL you will get the best of both worlds: you will benefit from the operational simplicity, low latency and expressiveness of Apache Flink without learning a new language or API. In addition, you can re-use your ETL queries for both bounded data (e.g. during reprocessing or bulk imports) and unbounded data. The results will be consistent across the different processing modes.

A particular highlight for this use case is the new Flink SQL file system connector added in Flink 1.11. It allows you to stream the results of your transformations directly into Hive partitions while also consistently updating HiveMetastore. It has never been easier to integrate your streaming data sources with your existing Hive infrastructure.

 

Complex Event Processing & Pattern Matching

Complex Event Processing (CEP) is fast becoming a critical paradigm for staying ahead of the curve. The introduction of row pattern recognition in the SQL standard and the rise of MATCH_RECOGNIZE as a native SQL clause made Complex Event Processing in Flink more efficient, scalable and accessible. Such patterns and rules often tend to be highly-specific, requiring deep knowledge of the specific domain by subject matter experts in the organization. With Flink SQL & MATCH_RECOGNIZE these rules and patterns can now be defined directly by product managers, product analysts and data scientists who hold domain knowledge and expertise in an organization. 

We would love to learn more about your use cases and areas where Flink SQL with Ververica Platform could be a perfect fit. Make sure to sign up for our early access program today and tell us more about the data processing challenges that you would like to address with Flink SQL and Ververica Platform. We are very excited to bring this new addition to our product offering later this year and want to make you a part of it! 


Ververica Platform, Flink SQL, Ververica Platform SQL, Apache Flink, Streaming SQLVerverica Contact, Apache Flink Contact, Stream Processing

 

 

 

 

Topics: Ververica Platform

Konstantin Knauf
Article by:

Konstantin Knauf

Related articles

Comments

Sign up for Monthly Blog Notifications

Please send me updates about products and services of Ververica via my e-mail address. Ververica will process my personal data in accordance with the Ververica Privacy Policy.

Our Latest Blogs

by Alexander Fedulov July 09, 2020

Presenting our Streaming Concepts & Introduction to Flink Video Series

Transitioning from the batch data processing world into the world of stream processing and real time analytics can be challenging. Throughout this process, there are many new concepts you need to...

Read More
by Konstantin Knauf July 06, 2020

Announcing Early Access Program for Flink SQL in Ververica Platform

Been wondering what's next for Ververica Platform? Maybe you've already guessed: Flink SQL is coming to Ververica Platform later this year! Today we are excited to announce our Early Access Program

Read More

Data-driven Matchmaking at Azar with Apache Flink

The Hyperconnect team attended Flink Forward for the first time a couple of months back and presented how we utilize Apache Flink to perform real time matchmaking for the video-based social...

Read More