Job engine

The job engine runs background Cloud Management Platform (CMP) work, such as provisioning, server actions, blueprint deployments, rules, and recurring jobs

4 minute read

Each unit of that work is a job, with its own status, progress messages, and log. The job engine retrieves pending jobs from the database, runs each in its own thread, and writes progress back as each job runs.

How jobs are processed

CMP creates the job with a PENDING status. A job created as INIT is held back and does not run until set to PENDING.
Workers check for PENDING jobs once every second and set claimed jobs to QUEUED. A job scheduled for a future time is left until that time passes.
Workers also check recurring job schedules once every minute and create a PENDING job for anything due.
The worker starts a thread for each QUEUED job and sets the job to RUNNING.
The job’s run() method finishes and returns a result. The job then ends as SUCCESS, WARNING, or FAILURE.

Potential interruptions:

The Cancel Job button in the UI sets the job to TO_CANCEL. The worker then stops the job, writes a message to the job’s log, and the job ends as CANCELED.
If a job is still running when the global job timeout expires, it is canceled in the same way, through TO_CANCEL. The timeout control is Default Job Timeout, configurable on the Resource management tab of Miscellaneous Settings, and it defaults to 8 hours.

Job statuses

Check a job’s status first when work looks stuck.

Status	Shown as	Meaning
`INIT`	Initializing	Created, but held back on purpose. Workers ignore the job until something sets it to PENDING.
`PENDING`	Pending	Waiting to be claimed. A job scheduled for a future time stays here until that time arrives.
`QUEUED`	Queued	Claimed by a worker and waiting for a free thread.
`RUNNING`	Running	Executing in a worker thread.
`PAUSED`	Paused	Stopped inside plugin code that is running in debug mode. Only an administrator can resume it.
`STEP`	Step	Resumed from a pause to run one line, stepping into any call it makes.
`NEXT`	Next	Resumed from a pause to run one line, stepping over any call it makes.
`RETURN`	Run until return	Resumed from a pause to run until the current function returns.
`TO_CANCEL`	In cancellation process	Cancellation has been requested. A job that is already running may sit here briefly before it reaches CANCELED.
`CANCELED`	Canceled by user	Stopped before it finished. The global job timeout and the cleanup of jobs whose workers died also end in this status.
`SUCCESS`	Completed successfully	Finished with no errors reported.
`WARNING`	Completed with warnings	Finished, but something in the job reported a warning. Read the job’s log before treating the work as done.
`FAILURE`	Completed with errors	Finished with an error. The job engine itself is working; troubleshoot the job.

You can use shell_plus to list jobs by state:

Job.objects.filter(status__in=Job.ACTIVE_STATUSES)
Job.objects.filter(status__in=Job.RUNNING_STATUSES)
Job.objects.filter(status__in=Job.COMPLETED_STATUSES)

Configuration

The job engine runs under Supervisor on the appliance, configured by /etc/supervisord.d/jobengine.conf.

Setting	Default	Description
`numprocs`	`1`	How many worker processes the job engine runs. Each worker runs individual jobs in threads, and each process can use a separate CPU core. Raising this can improve throughput on a busy appliance.
`stopwaitsecs`	`600`	How long Supervisor waits for a worker to exit after telling it to stop, before forcing it with a kill signal.
`autostart`	`true`	Whether the workers start when Supervisor starts. Set it to `false` on a node that should not process jobs, such as a standby node in a high availability deployment.

The file also carries other Supervisor settings, including how often a stopped worker is retried; see the Supervisor documentation. Upgrading CMP writes an example file, jobengine.conf.template. That file holds the built-in settings for the upgraded release. Use this as a reference when comparing any customized settings on your appliance.

How many jobs run at once

numprocs sets how many workers exist. A separate Django setting, MAX_JOB_CONCURRENCY, sets how many jobs each worker runs at a time, and it defaults to 200. Override it in customer_settings.py.

Each running job holds a database connection, so a worker will stop claiming jobs when PostgreSQL runs short of connections. When a worker cannot start any jobs, it logs that it has no free threads.

Change the number of worker processes

Edit /etc/supervisord.d/jobengine.conf and set numprocs to the desired number of workers.
Wait until no jobs are running, then reload Supervisor so it picks up the change: supervisorctl reload.

Reloading Supervisor restarts every service it manages, not only the job engine. Jobs running at the time are killed and left in RUNNING until the engine cleans them up.