How to Apply a Custom Logging Plugin to the Ververica Platform

Written by Jun Qin | 29 October 2023

Question

How can I use a custom logging plugin jar in my deployment in the Ververica Platform?

Answer

Note: This section applies to Flink 1.12 and later, with Ververica Platform 2.4 and later.

This article is aiming for applying a custom logging plugin to the Ververica Platform. To move forward, we'll use JSON Template Layout plugin as an example to explain the detailed steps.

Step-1: Prepare the plugin jar file

Download the latest log4j-layout-template-json-2.17.1.jar.

Step-2: Load the plugin jar into the Flink framework

There're two ways to achieve this step: creating a custom docker image or mounting the plugin jar file into the pods via Kubernetes. Both options are equally as powerful but give you some flexibility depending on your own requirements.

Option-1: Create a custom docker image

Create a Dockerfile like the following to include the logging plugin jar under /flink/lib. For example:

FROM registry.ververica.com/v2.5/flink:1.13.2-stream1-scala_2.12-java8
COPY log4j-layout-template-json-2.17.1.jar /flink/lib/

Build the docker image and push it to your docker registry.

Option-2: Add the plugin jar as a Kubernetes volume mount

The first step to allow mounting a custom jar into the pod is to make it available to Kubernetes. For this, we create a ConfigMap with the log4j-layout-template-json-2.17.1.jar file in the Kubernetes namespaces where you want to run your jobs:

kubectl create configmap jsonlogging --from-file=log4j-layout-template-json-2.17.1.jar \
  --namespace vvp-ops-jobs

Now you have to add a volume mount to your Ververica Platform deployment which can either be done in the old, but simple, Ververica Platform pod templates or the new Kubernetes (fully-fledged) pod templates section. Both are configured in your deployment YAML file:

Full-fledged Pod Templates (Recommended)

kind: Deployment
spec:
  template:
    spec:
      kubernetes:
        taskManagerPodTemplate:
          spec:
            containers:
              - name: flink-taskmanager
                volumeMounts:
                  - name: jsonlogging
                    mountPath: /flink/lib/log4j-layout-template-json-2.17.1.jar
                    subPath: log4j-layout-template-json-2.17.1.jar
            volumes:
              - name: jsonlogging
                configMap:
                  name: jsonlogging
        jobManagerPodTemplate:
          spec:
            containers:
              - name: flink-jobmanager
                volumeMounts:
                  - name: jsonlogging
                    mountPath: /flink/lib/log4j-layout-template-json-2.17.1.jar
                    subPath: log4j-layout-template-json-2.17.1.jar
            volumes:
              - name: jsonlogging
                configMap:
                  name: jsonlogging

Simplified Pod Options

kind: Deployment
spec:
  template:
    spec:
      kubernetes:
        pods:
          volumeMounts:
            - name: jsonlogging
              volume:
                name: jsonlogging
                configMap:
                  name: jsonlogging
              volumeMount:
                name: jsonlogging
                mountPath: /flink/lib/log4j-layout-template-json-2.17.1.jar
                subPath: log4j-layout-template-json-2.17.1.jar

Warning: You cannot just add this jar as an "additional dependency" to the VVP Deployment, because the jar would then only be available in the user class loader. Flink needs this jar to be available in the parent class loader in order to format logs. This is what we are doing with the volume mount.

Step-3: Configure the logging template for the VVP Deployment

Customize the Logging Template via the VVP UI (for example):

<Appenders>
  <Appender name="StdOut" type="Console">
    <!--Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n" type="PatternLayout"/-->
    <Layout type="JsonTemplateLayout" eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json" />
  </Appender>
  <Appender name="RollingFile" type="RollingFile" fileName="${sys:log.file}" filePattern="${sys:log.file}.%i">
    <Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n" type="PatternLayout"/>
    <Policies>
      <SizeBasedTriggeringPolicy size="5 MB"/>
    </Policies>
    <DefaultRolloverStrategy max="1"/>
  </Appender>
</Appenders>

Tip: You can also set this as a default configuration in a Ververica Platform's namespace so that all deployments in that namespace can benefit from it.

Related Information

View full post