A few weeks ago, Apache Flink 1.13 was released, and today we are happy to announce the availability of Ververica Platform 2.5, a must-have for stream processing application developers working with Flink 1.13.
Apache Flink 1.13 was a huge release, including over 1,000 new features and fixes. The Ververica Platform team set an equally ambitious goal to incorporate support for v1.13 within a few weeks of its release. We’re happy to have accomplished that. The new version makes Apache Flink even easier to use so that you can innovate new ways to serve customers in real-time.
To that end, Ververica Platform 2.5 has been enhanced with several powerful new capabilities that include the following:
Apache Flink 1.13 support
Custom catalogs
Java 11 support
Savepoint deletion
SQL improvements
Read on for more detail about Ververica Platform 2.5 enhancements.
As mentioned above, Apache Flink 1.13 was a huge release, and it’s been a big step forward in making stream processing applications as natural and as simple to manage as any other application.
Ververica Platform 2.5 supports the functionality recently announced in Apache Flink 1.13. Here are some of the most notable features:
Ververica Platform 2.5 adds support for Apache Flink 1.13 improvements that help you better understand stream processing application performance and why streams aren’t flowing as fast as you’d hoped:
Load and backpressure visualizations to help you identify bottlenecks
Flamegraph visualizations to identify hot code paths in your applications
Exception history - showing what exceptions caused a job to fail
State backend latency metrics - help you see whether your state backend is responsive
SQL support makes developing and maintaining stream processing application functionality easy for anyone. Apache Flink 1.13 added more functionality to its already-broad support for SQL with the following powerful new SQL capabilities, which are all exposed by the SQL tooling in Ververica Platform 2.5:
With the addition of Apache Flink 1.13 support in Ververica Platform 2.5, we have decided to sunset our support for Apache Flink 1.10. Per our support policy, Ververica Platform 2.5 is supported for use with Apache Flink 1.11 and higher.
Read about the full list of Apache Flink 1.13 enhancements.
Savepoints are self-contained snapshots of the distributed state of your Apache Flink applications and Flink SQL queries. Savepoints are created manually and serve various purposes, such as helping you migrate, fork, backup, or upgrade applications.
Ververica Platform 2.5 makes it easier to manage Savepoints by enabling you to delete them via Ververica Platform’s Web UI or REST API.
As an additional safety measure, Ververica Platform ensures that you do not delete savepoints that might still be required to recover your applications.
Ververica Platform 2.5 adds support for DataStream and Table API applications written in Java 11. For this, we publish an additional set of Apache Flink Docker images built on top of JDK 11 for Apache Flink 1.12 or later.
Another data access enhancement is support for custom catalogs. Catalogs store metadata about database objects, such as databases, tables, table attributes, functions, and views.
In addition to Ververica Platform’s built-in catalog and the JDBC catalog, v2.5 enables you to register custom catalogs for a namespace via the Web UI or the REST API. This allows users to use external tools to manage the data structures. This might be particularly useful, as administrators could manage the data structures in other tools that are then exposed to data analysts and data scientists through Ververica Platform. For example, it could expose fixed structures like given by an IMAP mailbox. As usual, the platform scans your archives and automatically detects your catalog implementation with all of its options.
We have also added support for the SQL “USE” keyword, which lets you set the default catalog and database in SQL scripts in Ververica Platform’s SQL Editor and SQL Deployments. Custom catalogs can be marked as read-only to prevent changes to the catalog, such as dropping tables.
Apache Flink-powered Ververica Platform offers advanced streaming SQL functionality, and version 2.5 strengthens it even further with the following enhancements:
SQL views can now be stored within Ververica Platform’s built-in catalog and are shown in the Schema Explorer alongside tables and functions. Views are frequently used to encapsulate logic that is shared between multiple queries or scripts, for example, for common pre-processing steps. Views are listed in the Schema Explorer:
The SET statement can now be used to set table configuration options directly in your SQL scripts. Thereby, they are picked up not only in the resulting Deployments, but also already when you preview your queries in the editor. Users can set local settings like timezone, parameters that could be used in UDFs, and Flink-specific settings that are specific for the use case like reusing objects in the pipeline to make it more efficient.
The Ververica Platform 2.5 SQL Editor now supports EXPLAIN <QUERY>. EXPLAIN statements are used to explain the logical and optimized query plans of a SELECT or an INSERT statement. This helps SQL experts optimize queries for faster, more efficient application performance.
If you would like to try Ververica Platform 2.5, the getting started guide and the corresponding playground repository are the best places to start. We look forward to your feedback and suggestions.