Job engine on CloudBolt Documentation

Job engine troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Work from the status to the appliance, then to the logs. Most of what follows also produces the information a support engineer will ask for, so keep notes as you go.

Start with the job’s status

Note the job number, status, and URL of the job that made you notice the problem, then find its status below. For what each status means, see Job engine.

Status	What to check
`PENDING`	Whether the job is scheduled for a future time, which is normal. Otherwise, confirm that a worker is running and check whether anything is preventing the job engine from starting.
`QUEUED`	Whether the worker that claimed the job is still alive, and whether it is out of free threads. A worker with no free threads records that in its log. If the worker has died, the job stays QUEUED, because nothing reclaims it. Cancel it and order the work again.
`RUNNING`	The job’s own log, for its last step and any error. The job may still be working normally. If its worker has died, the job engine cancels the job on its next start.
`PAUSED`, `STEP`, `NEXT`, `RETURN`	The job is in debug mode and waiting for someone to drive it, holding a thread until it ends. An administrator can resume or cancel it from the job’s detail page.
`CANCELED`	The job’s log. If the worker died, it says the process running the job is no longer active. A user cancellation and a timeout both log `Canceling this job`, so check the job engine log to tell those two apart. Then run a small job to check the health of the job engine.
`SUCCESS`, `WARNING`, `FAILURE`	The job engine is working, so troubleshoot the job itself: its log, the action or plugin it ran, and the resource it was working on.

After checking the status, consider the following: which other jobs are affected, and which job ran most recently? Has anything run since? If some jobs are progressing and others are not, compare their types and owners: a worker may be restricted to specific job types, which can make the excluded types appear stuck while everything else runs.

Restart the job engine

Mon, 01 Jan 0001 00:00:00 +0000

Restart them manually when one is stuck, and enable the automatic restart for jobs that the timeout cannot cancel.

A worker runs for at least a minute. Once it has finished the jobs it started and has nothing active, it exits, and Supervisor starts a new one. This keeps a long-running plugin from leaking memory into later jobs. A worker process ID that changes throughout the day is evidence of this cycle.

Run or exclude specific job types

Mon, 01 Jan 0001 00:00:00 +0000

This is controlled by environment variables CBJE_INCLUDE_JOB_TYPES and CBJE_EXCLUDE_JOB_TYPES. Each accepts a comma-separated list of job types, using the type names in the table below.

Set the variables

Variable	Effect
`CBJE_INCLUDE_JOB_TYPES`	The job engine runs only jobs of the listed types.
`CBJE_EXCLUDE_JOB_TYPES`	The job engine runs everything except jobs of the listed types.

Jobs of an excluded type stay PENDING until another job engine claims them. If no other job engine runs them, they never run.