Job engine troubleshooting

When jobs stop moving, check the job’s status first

  5 minute read  

Work from the status to the appliance, then to the logs. Most of what follows also produces the information a support engineer will ask for, so keep notes as you go.

Start with the job’s status

Note the job number, status, and URL of the job that made you notice the problem, then find its status below. For what each status means, see Job engine.

Status What to check
PENDING Whether the job is scheduled for a future time, which is normal. Otherwise, confirm that a worker is running and check whether anything is preventing the job engine from starting.
QUEUED Whether the worker that claimed the job is still alive, and whether it is out of free threads. A worker with no free threads records that in its log. If the worker has died, the job stays QUEUED, because nothing reclaims it. Cancel it and order the work again.
RUNNING The job’s own log, for its last step and any error. The job may still be working normally. If its worker has died, the job engine cancels the job on its next start.
PAUSED, STEP, NEXT, RETURN The job is in debug mode and waiting for someone to drive it, holding a thread until it ends. An administrator can resume or cancel it from the job’s detail page.
CANCELED The job’s log. If the worker died, it says the process running the job is no longer active. A user cancellation and a timeout both log Canceling this job, so check the job engine log to tell those two apart. Then run a small job to check the health of the job engine.
SUCCESS, WARNING, FAILURE The job engine is working, so troubleshoot the job itself: its log, the action or plugin it ran, and the resource it was working on.

After checking the status, consider the following: which other jobs are affected, and which job ran most recently? Has anything run since? If some jobs are progressing and others are not, compare their types and owners: a worker may be restricted to specific job types, which can make the excluded types appear stuck while everything else runs.

Common causes

The job engine is not running. Nothing moves off PENDING, and no recurring job fires. Each of the following writes the reason to the job engine log:

  • Maintenance mode is on. The job engine exits at startup when maintenance mode is active, which keeps jobs from running during an upgrade or restore.
  • The appliance does not have a valid license.
  • The Supervisor configuration has been edited so the workers no longer start, usually by setting autostart to false.

A worker died mid-job. Its jobs are left in QUEUED or RUNNING with nothing driving them. When the job engine next starts, it cancels the jobs that were RUNNING whose worker process no longer exists. Jobs that were only QUEUED are not covered by that cleanup and stay QUEUED until someone cancels them. In a high-availability deployment running more than one job engine, that cleanup can cancel jobs another engine is still running. It is controlled by the jobengine::cancel_orphan_jobs feature flag, which is enabled by default; disable it and the RUNNING jobs of a worker that really has died stay RUNNING until someone cancels them.

A job never finishes. An infinite loop or a deadlock in a plugin or script holds its thread open. The global job timeout cancels the job, and if that does not work, the job engine can be configured to restart itself.

The job log says the process running the job is no longer active. That message means the worker went away while the job was running, which usually means it crashed. A plugin that exhausts the appliance’s memory can bring the worker down with it. The job engine log for that worker will show the cause.

The database is unhealthy. The job queue is a database table, and every running job holds a connection, so problems there surface as jobs that will not queue or will not start.

Where to look on the appliance

Check the workers first. Supervisor reports the state of each one:

supervisorctl status jobengine:*

The logs live in the Cloud Management Platform log directory under /var/log:

File What it holds
jobengine.log The job engine as a whole: startup checks, orphaned job cleanup, timeouts, and crashes.
jobengine-worker00.log One file per worker process, numbered from worker00. This is where a single worker records what it claimed and whether it ran out of threads.
supervisord.log Whether Supervisor started, stopped, or kept restarting the workers.
application.log The web application, including whatever created the job in the first place.

Each job also keeps its own log, which you can download from the job’s detail page.

Open a support ticket

Help us help you describes what every ticket needs. For a job engine issue, add the following:

  • The job numbers you gathered, each with the status it shows and the status you expected, and the steps you have already tried.
  • The output of the Supervisor status command above.
  • The log archive from Admin > Download All CloudBolt Logs. It collects the current logs listed above, including the job engine and Supervisor logs. Individual job logs are not in the archive, so download those from each job’s detail page.
  • Job engine: How jobs move through the engine, what every status means, and worker settings.
  • Restart the job engine: Restarting workers and the optional automatic restart for jobs that outlive the timeout.
  • System status: The page that reports on the appliance’s services, including the job engine workers.