Ververica has been tirelessly working on several new products and projects in the past couple of months, including the release of Ververica Cloud, which you can read more about here.
We are always dedicated to working with the community, customers, gathering feedback, and using knowledge from our own expertise and experience to deliver products with the features that can deliver the best user experience and business value.
On top of offering new possibilities to work with Apache Flink with our new products, we have continued to bring new improvements to Ververica Platform, which has seen two new releases (2.9, 2.9.1, 2.10, 2.10.1). This post will take a look at what new features have been added to those releases.
What’s new in Ververica Platform 2.9
As you may remember about the change in our release process, Ververica Platform minor versions will correspond to either a new minor version of Flink or a new major Ververica Platform feature. Thus, Ververica Platform 2.9.0, 2.9.1 continues to support Apache Flink® 1.15 and Apache Flink® 1.14 under SLA. Per our support policy, Apache Flink® 1.13 is deprecated in this version and supported on a best-effort basis.
Running PyFlink Jobs
You can now write Flink jobs in Python or call Python UDFs and run them on Ververica Platform much easier with the release of 2.9, where we provide a Python interpreter and Pyflink in our built-in images out-of-the-box.
Previously, in order to run Pyflink jobs on the platform, you had to build a new custom Flink image that included a Python interpreter and PyFlink, which you can read more about here.
Changes to the Community Edition
We are starting to change the terms of the Community Edition of Ververica Platform as we start to revamp the business direction of our products. Starting from this release, CPU cores are limited to 10 and the new Flink High Availability Service will only be available in the Stream Edition.
Deployment improvements
We regularly ask for user feedback and have now made it easier to change the state of your deployments in one location on the UI. Now all you have to do is navigate to the Deployments section or the Session Clusters section in order to manage the state (“start”, “suspend”, “cancel”) of all of your deployments. In addition, we have added a Restart button (on the Deployments overview page) which allows you to restart a running deployment with one click, instead of manually stopping a certain job and then starting it again.
In this release, we have also enabled automatic calculation of the number of TaskManagers based on the parallelism and number of task slots set. The default number of TaskManagers has also been set to 1 (taskmanager.numberofTasksSlots = 1).
What’s new in Ververica Platform 2.10
With the addition of support for Apache Flink 1.16 in Ververica Platform 2.10, we are sunsetting our support for Apache Flink 1.13. Per our support policy, Ververica Platform 2.10 is supported for use with Apache Flink 1.14 and higher.
Flink SQL version upgraded to 1.16.1
Flink SQL has been upgraded to 1.16.1 on the platform. This means that the SQL editor now includes new syntax such as “CREATE TABLE AS SELECT”. You can have a look at the list of changes to the Table API and SQL in the release notes.
PyFlink Improvements
Besides SQL and Java, Python is another popular API supported by the Flink community and so we want to make it seamless to support PyFlink jobs on the Ververica Platform.
The 2.10 release supports submitting PyFlink jobs natively and you can now manage PyFlink deployments directly in the UI in this release.
Deployment Improvements
In terms of deployment, we have also added a validation check of non-empty Service Account names in Kubernetes resources. This enforces a one-to-one mapping between [VVP Namespace + Deployment/Session Cluster Name] and the [Service Account Name] and allows administrators to safely let the users work with their own permissions. Previously, users can specify any Service Account to a Deployment/Session Cluster via Flink Pod Templates.
Moreover, we improved artifact fetching via HTTP Basic Auth when deploying in application mode, which means credentials are no longer revealed in the URL when performing basic authentication.
Integration with Azure Kubernetes Service
In this release, users are now able to use Azure Blob Storage to store information about checkpoints, savepoints, and artifacts directly in their Microsoft Azure accounts. By utilizing Azure workload identity technology, VVP users do not need any extra credentials to authenticate in Azure Blob Storage. The user just needs to properly configure the Kubernetes service account during the environment installation process.
This integration allows users to manage and deploy Apache Flink applications seamlessly on AKS while leveraging Azure services for authentication and resource access.
Kubernetes High Availability Service
In version 2.8 of the Ververica Platform, we implemented the long-awaited Flink Kubernetes HA option alongside the existing VVP Kubernetes HA option. The purpose was to allow for a smooth transition between the old and the new availability options, which both feature retaining the latest state. In the 2.10 release, Flink Kubernetes is set as the default and VVP Kubernetes is marked as deprecated. Please check Kubernetes High-Availability Service for more information.
More Info & Get Ververica Platform
In addition to building support for Apache Flink 1.16, we also resolved a number of Ververica Platform vulnerability issues, which you can read about in our release notes.
If you would like to try Ververica Platform, the getting started guide and the corresponding playground repository are the best places to start. We look forward to your feedback and suggestions. :)
