ENVIRONMENT:
When nexus starting post DB migration to external DB, scheduler may fail to start system tasks to recreate search index and browse nodes, with below error message
1 tasks require frequency updates: Update repository indexes - (repo1, repo2)
nexus.log may have below ERROR reported
2024-05-11 23:07:49,794+0000 ERROR [status-delayed-tasks-2-thread-1] *SYSTEM org.sonatype.nexus.quartz.internal.datastore.DatastoreQuartzSchedulerSPI - Missing job-detail for key: nexus.26689c93-6cf1-4418-8eaa-ae3dd2a1194c
2024-05-11 23:07:49,913+0000 INFO [status-delayed-tasks-2-thread-1] *SYSTEM org.sonatype.nexus.quartz.internal.datastore.DatastoreQuartzSchedulerSPI - Recovering job nexus.d31287dd-cbbf-4fd2-a3d3-92a995fa5c15
2024-05-11 23:07:49,919+0000 ERROR [status-delayed-tasks-2-thread-1] *SYSTEM org.sonatype.nexus.quartz.internal.datastore.DatastoreQuartzSchedulerSPI - Failed to recover job nexus.d31287dd-cbbf-4fd2-a3d3-92a995fa5c15
org.quartz.JobPersistenceException: Couldn't store trigger 'DEFAULT.6da64b5bd2ee-a4650afa-5df5-4967-87f8-e94e69dd48a2' for 'nexus.d31287dd-cbbf-4fd2-a3d3-92a995fa5c15' job:Couldn't retrieve job because a required class was not found: org.sonatype.nexus.quartz.internal.task.QuartzTaskJob
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1228)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$4.executeVoid(JobStoreSupport.java:1164)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3780)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$VoidTransactionCallback.execute(JobStoreSupport.java:3778)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3864)
at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:93)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1160)
at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:901)
at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:258)
at org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI.recoverJob(QuartzSchedulerSPI.java:254)
at org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI.forEachNexusJob(QuartzSchedulerSPI.java:235)
at org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI.lambda$2(QuartzSchedulerSPI.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at org.sonatype.nexus.thread.DatabaseStatusDelayedExecutor.lambda$1(DatabaseStatusDelayedExecutor.java:104)
at org.sonatype.nexus.thread.DatabaseStatusDelayedExecutor.lambda$0(DatabaseStatusDelayedExecutor.java:90)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found: org.sonatype.nexus.quartz.internal.task.QuartzTaskJob
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1393)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1210)
... 20 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.sonatype.nexus.quartz.internal.task.QuartzTaskJob
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.quartz.simpl.InitThreadContextClassLoadHelper.loadClass(InitThreadContextClassLoadHelper.java:72)
at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:114)
at org.quartz.simpl.CascadingClassLoadHelper.loadClass(CascadingClassLoadHelper.java:138)
at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.selectJobDetail(StdJDBCDelegate.java:852)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveJob(JobStoreSupport.java:1390)
... 21 common frames omitted
SOLUTION:
This error can occur due to rare race condition which would be fixed by restarting nexus. If issue
still persists after restart, we can use below groovy script to remove the stale entries.
1. Enable scripttask creation, by adding below line to nexus.properties file and restart nexus
nexus.scripts.allowCreation=true
2. Create a new "Admin - Execute script" task with below content as source, set Manual as frequency and execute the task
import org.sonatype.nexus.scheduling.TaskInfo
import org.sonatype.nexus.scheduling.TaskScheduler
TaskScheduler scheduler = container.lookup(TaskScheduler.class.name)
def TASK_TYPE_IDS = ['repository.search.update']
def filterTasks = {
scheduler.listsTasks().findAll { TaskInfo info -> info.configuration.typeId in TASK_TYPE_IDS }
}
def tasks = filterTasks()
int deleteCount = 0
int triedCount = 0
log.info("Task list : $tasks")
tasks.each { TaskInfo info ->
triedCount++
def id = info.id
def name = info.name
def wasRemoved = false
try {
wasRemoved = scheduler.getTaskById(id).remove()
}
catch (Exception e) {
log.error('Problem removing task', e)
}
if (wasRemoved) {
deleteCount++
}
log.info('Task id: {} ,name: {} , removed?: {}.', id, name, wasRemoved)
}
log.info('Deleted {} of {} {} tasks', deleteCount, triedCount, TASK_TYPE_IDS)
3. Manually create and run "Repair - Rebuild repository search" task to rebuild the search index
4. If required, also create and run "Repair - Rebuild repository browse" task to recreate the browse node entries
Both these tasks can take long time to complete, depending on the no. of components in the nexus repo.
Users can continue using nexus while these tasks are running.