I seem to have performance problems with Ververica Platform or see unexpected CPU spikes in the Ververica Platform pod. How can I connect a profiler/JMX to the containers in this pod to get a CPU sampling snapshot for diagnosis?
Note: This section applies to Ververica Platform 2.3 or later.
In order to connect JMX to a container inside the Ververica Platform pod just likeconnecting JMX to the Flink cluster, you need to set a few properties in the JVM to accept JMX connections. These properties, however, cannot be set by the helm template, so we have to employ some manual work to get it done.
In order to configure the JVMs used by Ververica Platform, you can pass custom Java options via the JAVA_TOOL_OPTIONS environment variable and configure that in the Ververica Platform deployment resource. In the example below, we do so for the appmanager container, but if you suspect your problem in any of the other containers, they can be changed accordingly.
Find the deployment to configure:
kubectl -n <namespace> get deployments -l component=ververica-platform
Set the JAVA_TOOL_OPTIONS environment variable to allow unauthenticated, unencrypted JMX connections (don't worry, we won't expose these ports publicly):
kubectl -n <namespace> set env deployments <deployment> -c appmanager \
JAVA_TOOL_OPTIONS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.port=1099 \
-Dcom.sun.management.jmxremote.rmi.port=1099 \
-Djava.rmi.server.hostname=127.0.0.1"Note:If you want to introspect more than one container at a time, you should use a separate port for each of them.
If you don't know the Ververica Platform pod name yet, you can find it via
kubectl -n <namespace> get pods -l component=ververica-platform
You should actually see it restarting due to the change above. If It isn't, you can restart manually via
kubectl -n <namespace> delete pod <pod>
With the pod name from above, you can forward the port that you configured for JMX:
kubectl -n <namespace> port-forward <pod> 1099
jconsole 127.0.0.1:1099
Note: Since we have not configured SSL in this setup, you will have to allow an insecure connection.
Similarly, you can configure VisualVM, Mission Control, or the JMX tool of your choice to connect to a remote connection on 127.0.0.1:1099 and gather statistics and CPU samples for debugging.