Spring Boot scheduled Runnable tasks continue executing even after server is shut down with an actuator

I’m currently developing a Spring-based web platform which makes use of several scheduled processes that access a central database. I wanted to introduce actuators for shutdown and restart. However, I am experiencing an issue: even though the shutdown request is accepted with a 200 response, the application context begins shutting down:

2022-10-10 17:37:25.235  INFO 9067 --- [       Thread-3] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2022-10-10 17:37:25.240  INFO 9067 --- [       Thread-3] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.

And after that, I repeatedly get the following exception:

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is java.lang.IllegalStateException: EntityManagerFactory is closed
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467) ~[spring-orm-5.3.16.jar:5.3.16]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.16.jar:5.3.16]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.16.jar:5.3.16]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.16.jar:5.3.16]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) ~[spring-aop-5.3.16.jar:5.3.16]
    at usi.si.seart.service.TaskService$TaskServiceImpl$$EnhancerBySpringCGLIB$$4bf83a1d.getNext(<generated>) ~[classes/:na]
    at usi.si.seart.scheduling.TaskRunner.run(TaskRunner.java:68) ~[classes/:na]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.16.jar:5.3.16]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.lang.IllegalStateException: EntityManagerFactory is closed
    at org.hibernate.internal.SessionFactoryImpl.validateNotClosed(SessionFactoryImpl.java:547) ~[hibernate-core-5.6.5.Final.jar:5.6.5.Final]
    at org.hibernate.internal.SessionFactoryImpl.createEntityManager(SessionFactoryImpl.java:636) ~[hibernate-core-5.6.5.Final.jar:5.6.5.Final]
    at org.hibernate.internal.SessionFactoryImpl.createEntityManager(SessionFactoryImpl.java:158) ~[hibernate-core-5.6.5.Final.jar:5.6.5.Final]
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.createNativeEntityManager(AbstractEntityManagerFactoryBean.java:585) ~[spring-orm-5.3.16.jar:5.3.16]
    at jdk.internal.reflect.GeneratedMethodAccessor112.invoke(Unknown Source) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.invokeProxyMethod(AbstractEntityManagerFactoryBean.java:487) ~[spring-orm-5.3.16.jar:5.3.16]
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean$ManagedEntityManagerFactoryInvocationHandler.invoke(AbstractEntityManagerFactoryBean.java:734) ~[spring-orm-5.3.16.jar:5.3.16]
    at com.sun.proxy.$Proxy175.createNativeEntityManager(Unknown Source) ~[na:na]
    at org.springframework.orm.jpa.JpaTransactionManager.createEntityManagerForTransaction(JpaTransactionManager.java:485) ~[spring-orm-5.3.16.jar:5.3.16]
    at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:410) ~[spring-orm-5.3.16.jar:5.3.16]
    ... 17 common frames omitted

So in spite of the fact that the server is reported as “shutting down”, the scheduled processes still continue executing. My scheduler definition within the configuration class is as follows:

@Bean
public ThreadPoolTaskScheduler taskScheduler() {
    ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
    threadPoolTaskScheduler.setClock(Clock.systemUTC());
    threadPoolTaskScheduler.setPoolSize(3);
    threadPoolTaskScheduler.setThreadNamePrefix("PlatformScheduler");
    threadPoolTaskScheduler.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
    threadPoolTaskScheduler.setRemoveOnCancelPolicy(true);
    threadPoolTaskScheduler.initialize();

    threadPoolTaskScheduler.schedule(getRepoMaintainer(), new CronTrigger("0 0 0 * * SUN"));
    threadPoolTaskScheduler.schedule(getTaskCleaner(), new CronTrigger("0 */15 * * * *"));
    threadPoolTaskScheduler.scheduleWithFixedDelay(getTaskRunner(), 500);

    return threadPoolTaskScheduler;
}

Where the getRepoMaintainer, getTaskCleaner and getTaskRunner are all methods that produce custom Runnable instances: RepoMaintainer, TaskCleaner and TaskRunner respectively.

Answer

Unfortunately, none of the fixes suggested by the other users worked. In the end, I managed to solve my problem by virtue of a custom ThreadPoolTaskScheduler implementation:

ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler() {
    private final Set<ScheduledFuture<?>> futures = new HashSet<>();

    @Override
    public ScheduledFuture<?> scheduleWithFixedDelay(Runnable task, long delay) {
        ScheduledFuture<?> future = super.scheduleWithFixedDelay(task, delay);
        futures.add(future);
        return future;
    }

    @Override
    public ScheduledFuture<?> schedule(Runnable task, Trigger trigger) {
        ScheduledFuture<?> future =  super.schedule(task, trigger);
        futures.add(future);
        return future;
    }

    @Override
    public void shutdown() {
        futures.forEach(future -> future.cancel(true));
        super.shutdown();
    }
};

This way I ensure that all scheduled tasks are cancelled pre-shutdown no matter what.

Advertisement

Answer