How to integrate Istio with Ververica Platform

Question

How to integrate Istio with Ververica Platform?

Answer

In general, there are no specific VVP-related settings required to work with Istio. Nevertheless, as of writing this article, there is a little issue related to Kubernetes sidecar containers (not imposed by VVP). Istio-created sidecar containers may prevent VVP pods from normal completion. This issue will be addressed further below.

Inject Istio to VVP

The below instructions use two namespaces, which will eventually be managed by the Istio service mesh.

Kubernetes namespaces:

1) vvp - VVP control plane and MinIO deployment for VVP Artifact Storage. MinIO is optional, in case external object storage is configured with your VVP, then MinIO can be ignored below.

2) vvp-jobs - for all Flink applications which will be created by VVP deployments.

First, check the current status of pods in both namespaces.

Expecting 2 pods in vvp namespace.

kubectl get po -n vvp

NAME READY STATUS RESTARTS AGE
minio-f487cb456-jm86x 1/1 Running 0 5m7s
vvp-ververica-platform-fdb85fd68-dr5wm 3/3 Running 0 5m5s

Checking the list of containers across the existing two pods:

kubectl get po -n vvp -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]''\n' |\
sort

1 minio/minio:latest
1 registry.ververica.com/v2.9/vvp-appmanager:2.9.0
1 registry.ververica.com/v2.9/vvp-gateway:2.9.0
1 registry.ververica.com/v2.9/vvp-ui:2.9.0

There are no Istio containers yet.

Checking the second namespace.

kubectl get po -n vvp-jobs

No resources found in vvp-jobs namespace.

Namespace vvp-jobs has no pods deployed yet.

Adding Istio label

Istio requires a special namespace label to manage applications in it. Adding it:

kubectl label namespace vvp istio-injection=enabled
kubectl label namespace vvp-jobs istio-injection=enabled

Restart pods in vvp namespace to get Istio to add its own sidecar containers. Replace pod names with your pod names in the below command:

kubectl delete po minio-f487cb456-jm86x -n vvp
kubectl delete po vvp-ververica-platform-fdb85fd68-dr5wm -n vvp

Checking the current list of containers in the vvp namespace:

kubectl get pods -n vvp \
  -o jsonpath="{.items[*].spec['initContainers','containers'][*]['image']}" | \
  tr -s '[[:space:]]' '\n'

docker.io/istio/proxyv2:1.16.1
docker.io/istio/proxyv2:1.16.1
minio/minio:latest
registry.ververica.com/v2.9/vvp-appmanager:2.9.0
registry.ververica.com/v2.9/vvp-gateway:2.9.0
registry.ververica.com/v2.9/vvp-ui:2.9.0
docker.io/istio/proxyv2:1.16.1
docker.io/istio/proxyv2:1.16.1

Now, there are two Istio containers (init and proxy) per each pod injected.

Explore the list of containers in the pods in detail to see that they have different names. All Istio sidecar containers will have the same container image.

Run VVP Deployment

Istio Gateway - Optional

Deploy Istio Gateway and VirtualService resources in case VVP UI needs to be accessed via external IP and no existing ingress configuration is set up.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: vvp-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  -port:
     number: 80
     name: http
     protocol: HTTP
   hosts:
   -"*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: vvp
spec:
  hosts:
  - "*"
  gateways:
  - vvp-gateway
  http:
  - match:
    - uri:
      prefix: /app
    - uri:
      prefix: /ui
    - uri:
      prefix: /namespaces
    - uri:
      prefix:/sql
    - uri:
      prefix: /catalog
    - uri:
      prefix: /artifacts
    - uri:
      prefix: /api/v1
    - uri:
      prefix: /flink-ui/v1
    route:
    - destination:
        host: vvp-ververica-platform
        port:
          number: 80

Save the above YAML definition as a local file and apply it to the vvp Kubernetes namespace by running the command:

kubectl apply -n vvp istio/vvp-gateway.yaml

The next step is to determine the host address of the Ingress gateway by using this subsection from the Istio documentation: https://istio.io/latest/docs/setup/getting-started/#determining-the-ingress-ip-and-ports.

Example of opening the VVP UI via ingress gateway: http://<determined host>/app/

Sidecar termination

Before starting any new or restarting existing VVP deployment, we need to solve the issue mentioned earlier about sidecar containers, which may hold a pod from completion. This issue will lead to problems in the VVP control plane, as it won't be able to restart Flink jobs to apply new settings.

Below is one of the solutions which require changing the VVP Deployment Kubernetes pod template. The following YAML code needs to be added under spec.template.spec:

...
kubernetes:
  jobManagerPodTemplate:
    spec:
      containers:
        - command:
            - sh
            - '-c'
            - >
            /bin/bash <<'EOSCRIPT'
            set -e
            sleep 10
            while true; do pgrep java || break; sleep 5; done 
            pgrep envoy && kill $(pgrep envoy)
            EOSCRIPT
          image: 'ubuntu:19.04'
          name: istio-proxy-terminator
          resources:
            limits:
              cpu: 10m
              memory: 20M
            requests:
              cpu: 10m
              memory: 20M
    shareProcessNamespace: true
...

Use it to update existing VVP deployments. This is how it may look on the VVP UI:

Verify Istio Integration

Kiali tool is a very convenient way to visualize service-to-service network traffic, thus we can check all the Istio proxies used in the vvp and vvp-jobs namespaces.

Observations:

1. vvp-ververica-platform application is accessed from Istio Ingress Gateway. The Gateway host is accessed via web browsers.

2. vvp-ververica-platform application uses MinIO to list or upload artifacts for VVP deployments.

3. The pods in vvp-jobs are created as part of a Flink cluster and VVP deployment: jobmanager, taskmanager.

4. The pods in vvp-jobs also access external namespace services such as MinIO, Kafka, Kubernetes API, etc.

5. Traffic to MinIO is in red status due to some percentage of 4xx HTTP statuses.

Stop VVP Deployment

Suspend or Cancel running the deployment to verify the pods are terminated successfully. The vvp-jobs namespace should not have any running pods of the tested deployment in it after a stop completion:

kubectl get po -n vvp-jobs
No resources found in vvp-jobs namespace

Mutual TLS Authentication

In order to enable Istio service-to-service mTLS authentication for VVP services, just add Istio PeerAuthentication resource to the Istio root namespace (istio-system) or to the vvp and your VVP deployment namespace (such as vvp-jobs in our case). First option applies authentication policy to all namespaces automatically.

# peer-authentication.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT

Apply above YAML configuration to you cluster:

kubectl apply -f peer-authentication.yaml -n vvp
kubectl apply -f peer-authentication.yaml -n vvp-jobs

Then start new or restart existing VVP deployment to get mTLS added to the VVP services, Flink Job and Tasks Managers communication.

Below is a Kiali service graph for vvp and vvp-jobs namespaces. You can see small black lock icon on the arrows among services. It means that client and server sides are authenticated via mTLS. As you can see, VVP does not require any special configuration rather than standard Istio authentication policy applied to a Kubernetes namespace.

Related Information

• VVP docs: add sidecar containers
• Sidecar container exit trick
• Istio Authentication Policies