How to Access Amazon S3 via IRSA on Ververica Platform running on AWS EKS

Written by Jun Qin | 26 July 2023

Question

We run Ververica Platform on Amazon EKS. For security reasons, my organization does not allow to access Amazon S3 based on credentials, nor based on EKS NodeInstanceRole. How can I configure Ververica Platform to allow platform pods and job pods to access Amazon S3 via IAM Role for Service Account (IRSA)?

Answer

Note: This section applies to Ververica Platform 2.4.3 (or later) and 2.5.1 (or later), combing with the Flink image 1.12.5-stream1 (or later) or 1.13.2-stream1 (or later).

Prerequisite

To access S3 (or other AWS resources) via IRSA, your EKS cluster must have an IAM OIDC Provider enabled. Follow this AWS doc if you have not enabled it.

Note: the remainder of the article uses the following variables:

$clusterName: the name of the EKS cluster
$vvpNamespace: the Kubernetes namespace where Ververica Platform runs
$jobNamespace: the Kubernetes namespace where Ververica Platform deployments run

Create a Service Account in `$vvpNamespace` and attach an IAM Role

Run the following command to create a service account in $vvpNamespace and attach an IAM policy to it:

eksctl create iamserviceaccount \
  --namespace $vvpNamespace \
  --name vvp-sa \
  --cluster $clusterName \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \
  --role-name vvp-full-s3-access \
  --approve

If you already have an existing service account that you want to use, use the option --override-existing-serviceaccounts with the command.

Note: this command attaches the policy AmazonS3FullAccess to the service account. If desired, you can attach a policy that has the least and necessary privileges.

Configure Ververica Platform to Use the Service Account

Now you need to tell Ververica Platform to use this service account. To do so, create a Helm Values file:

rbac:
  create: false
  serviceAccountName: vvp-sa

then run:

helm upgrade --install vvp ververica/ververica-platform --version <version>
  --namespace $vvpNamespace
  --values <the Values file above>
  ... <any other options/Values files you need>

In order to allow this service account to create and manage job pods in $jobNamespace, you need to run the following commands, which are otherwise done automatically during Ververica Platform installation when rbac.create is set to true.

# create a Kubernetes role "vvp-role" in $jobNamespace
cat << EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vvp-role
  namespace: $jobNamespace
rules:
- apiGroups: [ "apps", "extensions" ]
  resources: [ "deployments" ]
  verbs: [ "create", "delete", "get", "list", "patch", "update", "watch" ]
- apiGroups: [ "" ]
  resources: [ "configmaps", "pods", "services", "secrets", "serviceaccounts" ]
  verbs: [ "create", "delete", "get", "list", "patch", "update", "watch" ]
- apiGroups: [ "batch" ]
  resources: [ "jobs" ]
  verbs: [ "create", "delete", "get", "list", "patch", "update", "watch" ]
- apiGroups: [ "rbac.authorization.k8s.io" ]
  resources: [ "roles", "rolebindings" ]
  verbs: [ "create", "delete", "get", "list", "patch", "update", "watch" ]
EOF

# and bind it to the service account in $vvpNamespace
cat << EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vvp-rolebind
  namespace: $jobNamespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: vvp-role
subjects:
- kind: ServiceAccount
  name: vvp-sa
  namespace: $vvpNamespace
EOF

Create a Service Account in `$jobNamespace` and attach an IAM Role

Since the Flink job pods also need to access S3 (e.g., to download job artifacts, create checkpoints/savepoints, or write job results), similarly, you need to have a service account in $jobNamespace and attach an IAM role to it:

eksctl create iamserviceaccount \
  --namespace $jobNamespace \
  --name flink-sa \
  --cluster $clusterName \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \
  --role-name flink-full-s3-access \
  --approve

In order for Kubernetes HA to work, the service account in $jobNamespace requires permissions to access Kubernetes ConfigMaps. To grant the permission:

# create a Kubernetes role in $jobNamespace
cat << EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: flink-role
  namespace: $jobNamespace
rules:
- apiGroups: [ "" ]
  resources: [ "configmaps"]
  verbs: [ "create", "delete", "get", "list", "patch", "update", "watch" ]
EOF

# and bind it to the service account in $jobNamespace
cat << EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: flink-rolebind
  namespace: $jobNamespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: flink-role
subjects:
- kind: ServiceAccount
  name: flink-sa
  namespace: $jobNamespace
EOF

Configure Ververica Platform Deployments to Use the Service Account

With the service account created in $jobNamespace, you can configure now Ververica Platform deployments to use it by adding the following into the deployment spec:

# set the service account in deployment
spec:
  template:
    spec:
      kubernetes:
        jobManagerPodTemplate:
          spec:
            serviceAccountName: flink-sa
        taskManagerPodTemplate:
          spec:
            serviceAccountName: flink-sa

If your Flink job writes the job results to S3 (e.g., s3a://bucket/path/object) via StreamingFileSink using the flink-s3-fs-Hadoop dependency, you additionally need the following Flink configuration in your deployment spec:

fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider

Configure AWS SDK to use your regional STS service

If you need to use the AWS regional Security Token Service endpoint (e.g., sts.eu-central-1.amazonaws.com) because the global one (sts.amazonaws.com) is blocked in your environment for security reasons, please take the additional steps below:

1) Configure the S3 region for your blob storage:

vvp:
  blobStorage:
    baseUri: s3://vvp-bucket
    s3:
      region: eu-central-1

2) Add this annotation to the Service Account:

 eks.amazonaws.com/sts-regional-endpoints=true

Debugging

If you still face issues, you can enable DEBUG logging by adding the following configuration into your helm chart:

topLevelConfig:
  logging.level.com.dataartisans: DEBUG
  logging.level.com.amazonaws: DEBUG
  logging.level.com.ververica: DEBUG
  logging.level.org.springframework: DEBUG

Warning: topLevelConfig is at the root level of the YAML file, it is not under vvp!

Note: These are VVP internals and may be changed in future versions of Ververica Platform.

Related Information

View full post