We are pleased to announce the general availability of data Artisans Platform 1.3 as of today! This release is packed with features that strengthen enterprise security and application management, as well as improvements to the web-based User Interface that makes configuring these features as straightforward as a click.
Fancy a walkthrough?
Support for SSL Encryption and Mutual Authentication:
With the new release of data Artisans Platform, encryption and mutual authentication of network communications within Flink clusters can be enabled through a simple configuration flag in Application Manager (flink.security.ssl.enabled) — starting from Flink 1.6.0. From Flink 1.6.2, this support is further extended to communications between Application Manager instances and Flink, guaranteeing that only authenticated users can exert control over jobs.
Certificates and encryption keys are managed in a shared effort between Application Manager and Kubernetes, as described in the data Artisans Platform documentation.
More robust state management:
The second major feature shipping with data Artisans Platform 1.3 introduces a new restore strategy that reinforces the measures for keeping the state of a Flink application: even in cases of failure or edge cases such as the application being stuck in a restart-loop. The supported restart strategies are:
- LATEST_STATE: use the latest successful checkpoint or savepoint known to Application Manager;
- LATEST_SAVEPOINT: use the latest successful savepoint known to Application Manager that may have been previously triggered by a user request;
- NONE: do not start from any checkpoint or savepoint.
This development extends state retention to a broader set of failure scenarios and ensures that users can restore their stateful Flink jobs with little to no manual intervention, at the same time reducing the amount of reprocessing involved in this operation. As part of these changes, we have also adapted the Flink invocation to a broader set of abstractions introduced since Flink 1.6 (FLIP-6), overall reducing the time to launch a cluster or upgrade a deployment substantially.
Deprecation: in the API, Deployment.spec.startFromSavepoint is superseded by Deployment.spec.restoreStrategy. We still support “startFromSavepoint”, but encourage users to move to “restoreStrategy”.
New metrics dashboard:
More than just providing the tools to easily build and plug monitoring into your applications, data Artisans Platform 1.3 includes a default Grafana dashboard that allows you to start gathering and visualizing important metrics out-of-the-box. The underlying time series data relies on a buffered metrics reporter that writes asynchronously to InfluxDB at fixed intervals, leveraging performance without compromising memory.
Support for Apache Flink 1.7:
In sync with the release of the latest Flink version, we are also announcing that data Artisans Platform 1.3 will support Apache Flink 1.7 from the get-go. This means you can easily upgrade your applications to the most recent version and start exploring all the exciting new features, such as exactly-once S3 writers and temporal table joins.
Improvements to the web-based User Interface:
We have expanded the User Interface to 1) be timezone-aware and 2) accommodate the handful of implementations around multi-tenancy and authentication we introduced in the previous release — all nicely packed under a renewed Settings tab. Deployment targets, secrets, API tokens and namespaces can now be managed in an effortless way through a dedicated visual abstraction layer.
The sidebar gets an additional tab that allows you to easily switch between namespaces using a drop-down menu. If you are rolling out data Artisans Platform in an environment spread across multiple teams or projects, this improvement allows you to individually access and manage all namespaces in your Kubernetes cluster using Application Manager.
Ability to pull from private Docker registries:
For deployments on top of Kubernetes, clusters use a specific secret type to pull private Docker images: the imagePullSecret. Up to this point, users of data Artisans Platform had to attach these secrets to the default service account of their Kubernetes namespace to access Flink containers from private Docker registries — which made the secrets globally available in the namespace.
This release adds the flinkImagePullSecret configuration option to Application Manager, allowing users to seamlessly pull from these private Docker registries. If the option is present, the configured imagePullSecret is attached to a specific Flink cluster deployment.
For more details, check out the changelog in the updated documentation. You can get your hands on the latest trial Virtual Machines and Docker images on our download page and play around with data Artisans Platform 1.3. In case you are already using Application Manager and working with the data Artisans repository, you simply have to update the image reference to: v1.3/appmanager:1.3.0.
We are eager to hear your feedback about this release! Reach out to firstname.lastname@example.org to let us know of any suggestions or questions you might have.