Using Azure Key Vault in Ververica Platform

Question

How can I use the secrets stored in Azure Key Vault if I deploy Ververica Platform on the Azure Kubernetes Service (AKS)?

Answer

You can use the Azure Key Vault provider for Secrets Store CSI driver to access Azure Key Vault. It allows getting secret contents stored in an Azure Key Vault instance and use the Secrets Store CSI driver interface to mount them into the Ververica Platform pods or into your Flink job pods running on Ververica Platform. At the high level, the procedure consists of four steps:

1. Identify the clientId and the principalId you want to use to access Azure Key Vault
2. In Azure, grant Key Vault access permissions to the identified clientId and principalId
3. In AKS, create a SecretProviderClass that specifies the parameters to connect to Key Vault
4. For the Kubernetes pods, configure volumes with the created storage class and mount them

We will walk through these steps in detail in the sections below.

0) Prepare a Key Vault

Before we dive into the details, let us first create a Key Vault and a secret in Azure to play with. Skip this section if you have already created a key vault and set a secret.

Azure Key Vault supports two different permission models: vault access policy and azure role-based access control. We show both scenarios here, but you can choose one of them to proceed with. Both work with the four Key Vault access modes described in the next section.

Option A: Create a Key Vault with vault access policy

# create key vault and store its id in $keyVaultId
az keyvault create --resource-group ${resourceGroup} --name ${keyVaultName}
keyVaultId=$(az keyvault show --resource-group ${resourceGroup} 
             --name ${keyVaultName} --query id -o tsv)
# set a secret
az keyvault secret set --vault-name ${keyVaultName} --name ${secretName} \
  --file ${secretFile}

Option B: Create a Key Vault with Azure role-based access control

# create a RBAC enabled key vault and store its id in $keyVaultId
az keyvault create --resource-group ${resourceGroup} \
  --name ${keyVaultName} --enable-rbac-authorization
keyVaultId=$(az keyvault show --resource-group ${resourceGroup} 
             --name ${keyVaultName} --query id -o tsv)

# add the required role assignment to yourself in order to set a secret
az role assignment create --role "Key Vault Administrator" \
  --assignee <yourlogin> --scope ${keyVaultId}
# now set a secret
az keyvault secret set --vault-name ${keyVaultName} --name ${secretName} \
  --file ${secretFile}

* <yourlogin> above is the username you used to log in to the Azure portal.

1) Identify clientId and principalId

The Azure Key Vault Provider offers four modes for accessing a Key Vault instance. The following four subsections show how to identify the clientId and principalId for each of these modes. You can choose one to proceed with. While the Service Principal mode requires a Kubernetes secret to work, the Pod Identity mode and the User/System-assigned Managed Identity modes can access a Key Vault instance without the need for a Kubernetes secret. The System-assigned Managed Identity mode needs only the principalId.

Service Principal Mode

If your AKS cluster was created in the following way:

az aks create --name $clusterName ... \
  --service-principal $clientId \
  --client-secret $clientSecret

and you know the values of clientId and clientSecret, then you can use this clientId. Otherwise, you can also create a new service principal. For example:

SP=$(az ad sp create-for-rbac --skip-assignment --name <svcPrincipalName>)
clientId=$(echo $SP | jq -r '.appId')
clientSecret=$(echo $SP | jq -r '.password')

* <svcPrincipalName>: use a meaningful service principal name of your choice.

When using service principals to access Key Vault, the clientId is also the principalId.

To use a service principal to access a Key Vault instance, you will need to store the clientId and the clientSecret into a Kubernetes secret in the same namespace as the referencing pods. As you will see later, the secret will be used by the Secrets Store CSI driver to connect to the Key Vault instance.

kubectl create secret generic <svcPrincipalSecretName> \
  --from-literal clientid=$clientId \
  --from-literal clientsecret=$clientSecret \
  --namespace <namespace>

Pod Identity Mode

Refer to the Azure Key Vault Provider doc on the specific steps to set up pod identity for Key Vault access. In this mode, you can get the clientId and the principalId from the created identity:

az identity create --resource-group $nodeResourceGroup --name <identityName>
clientId=$(az identity show --resource-group $nodeResourceGroup 
  --name <identityName> --query clientId -o tsv)
principalId=$(az identity show --resource-group $nodeResourceGroup 
  --name <identityName> --query principalId -o tsv)

* <identityName>: use a meaningful identity name of your choice.

User-assigned Managed Identity Mode

Similar to the pod identity mode, the clientId and the principalId are from the created identity.

System-assigned Managed Identity Mode

To have a system-assigned managed identity for an AKS virtual machine scale set (vmss), run:

nodePoolVMSS=$(az vmss list --resource-group $nodeResourceGroup 
  | jq -r '.[0].name')
az vmss identity assign --resource-group $nodeResourceGroup \
  --name $nodePoolVMSS
principalId=$(az vmss identity show --resource-group $nodeResourceGroup 
  --name $nodePoolVMSS --query principalId -o tsv)

In this case, we only have the principalId from the generated identity. clientId is not needed in this mode.

2) Grant Access Permissions to the identified clientId and principalId

For Key Vault with vault access policy:

az keyvault set-policy --name ${keyVaultName} --spn ${clientId} \
  --secret-permissions get

Here we grant only the secret permission get. You can add or reduce permissions if necessary. In the System-assigned Managed Identity mode, because we do not get a clientId, we use with --object-id $principalId instead:

az keyvault set-policy --name ${keyVaultName} --object-id ${principalId} \
  --secret-permissions get

For Key Vault with Azure role-based access control:

az role assignment create --role "Key Vault Secrets User" \
  --assignee $principalId --scope $keyVaultId

3) Create a SecretProviderClass

First, install the Azure Key Vault Provider for Secrets Store CSI Driver:

# add helm repo if not done yet
helm repo add csi-secrets-store-provider-azure \
  https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts

# install
helm install csi-secrets \
  csi-secrets-store-provider-azure/csi-secrets-store-provider-azure

Verify the CSI driver is installed and the pods are running

% kubectl get pod
NAME                                                 READY   STATUS    RESTARTS   AGE
csi-secrets-csi-secrets-store-provider-azure-vqpwn   1/1     Running   0          94s
csi-secrets-secrets-store-csi-driver-86kft           3/3     Running   0          94s

Depending on the access mode you use above, you need to supply different parameters when creating the SecretProviderClass:

Service Principal Mode:

cat << EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: <secretProviderName>
  namespace: <namespace>
spec:
  provider: azure
  parameters:
    keyvaultName: ${keyVaultName}
    objects:  |
      array:
        - |
          objectName: ${secretName}
          objectType: secret
    tenantId: ${tenantId}
EOF

* <secretProviderName>: Use a meaningful secret provider name of your choice.
* <namespace>: the namespace of the pod which needs access to the Key Vault

Otherwise, you need to modify the YAML file above by adding the following settings under spec.parameters:

Pod Identity Mode:

usePodIdentity: "true"

User-Assigned Managed Identity Mode:

useVMManagedIdentity: "true"
    userAssignedIdentityID: $clientId

System-Assigned Managed Identity Mode:

useVMManagedIdentity: "true"
    userAssignedIdentityID: ""

4) Configure and Mount Volumes in Pods

Now you can reference this SecretProviderClass in a volume and mount the volume in a pod. To access Key Vault via the Service Principal in the jobmanager pod, for example, add the following into your deployment spec:

spec:
  templates:
    spec:
      kubernetes:
        jobManagerPodTemplate:
          spec:
            containers:
              - name: flink-jobmanager
                volumeMounts:
                  - mountPath: /azkvsecret
                    name: azkvsecret
            volumes:
              - name: azkvsecret
                csi:
                  driver: secrets-store.csi.k8s.io
                  readOnly: true
                  volumeAttributes:
                    secretProviderClass: <secretProviderName>
                  # only needed in the Service Principal Mode
                  nodePublishSecretRef:  
                    name: <svcPrincipalSecretName>

If you want to access Key Vault in the Ververica Platform pod, add the following into the values.yaml file, then use the file to setup/upgrade Ververica Platform via helm:

volumeMounts:
  - name: azkvsecret
    mountPath: /azkvsecret
    readOnly: true  
volumes:
  - name: azkvsecret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: <secretProviderName>
      nodePublishSecretRef: # only needed in the service principal mode
        name: <svcPrincipalSecretName>

Accessing Key Vault via a Pod Identity or a User/System-assigned Managed Identity is the same as above, except that there is no need to add nodePublishSecretRef to the volume specification because the SecretProviderClass already contains the information about which identity to use.

The Pod Identity Mode also requires labeling the pods with the label aadpodidbinding (see the Pod Identity access mode for more details). You can label Flink jobmanager pods as follows:

spec:
  templates:
    spec:
      kubernetes:
        jobManagerPodTemplate:
          metadata:
            labels:
              aadpodidbinding: <the selector specified in AzureIdentityBinding>

For Ververica Platform pods, you can use the following Values file to add extra labels:

extraLabels:
  aadpodidbinding: <selector you specified in AzureIdentityBinding>

Important: this extraLabels feature is only available in Ververica Platform 2.5 or later. For older versions, you would need to add the label manually.

Now, when the Ververica Platform or Flink job pods are started, you should see the secret in the mounted volume.

Related Information

• Azure Key Vault
• Azure Key Vault Provider for Secrets Store CSI Driver
• CSI Driver Identity Access Modes
• Add extra labels to Ververica Platform pods