Flink Forward 2025 Barcelona: The Future of AI is Real-Time

Buy Ticket

Start for Free
Request a Trial

Multiple Jobs are Started from one Ververica Platform Deployment

Issue

When a deployment starts in Ververica Platform, it may happen that multiple Flink jobs are started. The following TimeoutException may also appear in Ververica Platform appmanager logs:

2020-04-14 15:15:23.873 DEBUG 1 --- [eduler-worker-1] c.d.a.c.c.drivers.ScheduledController : Exception while invoking the controller DeploymentControllerLogic{deploymentId=12d6b0e7-b9e8-4736-9ff4-d63688cc6882}. Will retry after backing off for 3210 milliseconds

java.lang.RuntimeException: java.util.concurrent.TimeoutException
 at com.google.common.base.Throwables.propagate(Throwables.java:241) ~[guava-28.0-jre.jar!/:na]
 at com.dataartisans.appmanager.controller.api.MoreFutures.deref(MoreFutures.java:41) ~[appmanager-controller-api-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.JobRepositoryFacade.deref(JobRepositoryFacade.java:147) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.JobRepositoryFacade.create(JobRepositoryFacade.java:58) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.domain.deployment.ToRunning.createJob(ToRunning.java:248) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.domain.deployment.ToRunning.tryTransition(ToRunning.java:145) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.domain.deployment.DeploymentControllerLogic.dispatchOnSpecState(DeploymentControllerLogic.java:84) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.domain.deployment.DeploymentControllerLogic.invoke(DeploymentControllerLogic.java:72) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.drivers.ScheduledController.invoke(ScheduledController.java:56) ~[appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.drivers.SchedulerWorker.invokeSilently(SchedulerWorker.java:50) [appmanager-controller-2.1.0.jar!/:na]
 at com.dataartisans.appmanager.controller.core.drivers.SchedulerWorker.run(SchedulerWorker.java:27) [appmanager-controller-2.1.0.jar!/:na]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_242]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_242]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
Caused by: java.util.concurrent.TimeoutException: null
 at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) ~[na:1.8.0_242]
 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[na:1.8.0_242]
 at com.dataartisans.appmanager.controller.api.MoreFutures.deref(MoreFutures.java:39) ~[appmanager-controller-api-2.1.0.jar!/:na]
 ... 12 common frames omitted

Environment

This happens in Ververica Platform 2.1.0 with persistent volumes backed up by Amazon Elastic File System (EFS).

This may happen in Ververica Platform 2.1.1 or later with EFS if a longer timeout is needed (see how to increase the timeout value as described below).

Resolution

Upgrade to Ververica Platform 2.1.1 or later and, if necessary, set a longer timeout in your values.yaml, e.g.,:

vvp:
  appmanager:
    persistence.repository.timeout.ms: 15000

The time unit here is millisecond. The default value is 10000 (10 seconds). Increase the timeout if necessary.

Cause

Accessing persistent volumes backed by Amazon EFS takes a long time.

Related Information