# Job engine

> The job engine runs background Cloud Management Platform (CMP) work, such as provisioning, server actions, blueprint deployments, rules, and recurring jobs

---

Site [llms.txt](/llms.txt)

---

Each unit of that work is a job, with its own status, progress messages, and log. The job engine retrieves pending jobs from the database, runs each in its own thread, and writes progress back as each job runs.

## How jobs are processed

1. CMP creates the job with a PENDING status. A job created as INIT is held back and does not run until set to PENDING.
1. Workers check for PENDING jobs once every second and set claimed jobs to QUEUED. A job scheduled for a future time is left until that time passes.
1. Workers also check recurring job schedules once every minute and create a PENDING job for anything due.
1. The worker starts a thread for each QUEUED job and sets the job to RUNNING.
1. The job's `run()` method finishes and returns a result. The job then ends as SUCCESS, WARNING, or FAILURE.

Potential interruptions:

- The **Cancel Job** button in the UI sets the job to TO_CANCEL. The worker then stops the job, writes a message to the job's log, and the job ends as CANCELED.
- If a job is still running when the global job timeout expires, it is canceled in the same way, through TO_CANCEL. The timeout control is **Default Job Timeout**, configurable on the **Resource management** tab of [Miscellaneous Settings](/docs/cmp/configure/administration/system/miscellaneous-settings/), and it defaults to 8 hours.

## Job statuses

Check a job's status first when work looks stuck.

| Status | Shown as | Meaning |
| --- | --- | --- |
| `INIT` | Initializing | Created, but held back on purpose. Workers ignore the job until something sets it to PENDING. |
| `PENDING` | Pending | Waiting to be claimed. A job scheduled for a future time stays here until that time arrives. |
| `QUEUED` | Queued | Claimed by a worker and waiting for a free thread. |
| `RUNNING` | Running | Executing in a worker thread. |
| `PAUSED` | Paused | Stopped inside plugin code that is running in debug mode. Only an administrator can resume it. |
| `STEP` | Step | Resumed from a pause to run one line, stepping into any call it makes. |
| `NEXT` | Next | Resumed from a pause to run one line, stepping over any call it makes. |
| `RETURN` | Run until return | Resumed from a pause to run until the current function returns. |
| `TO_CANCEL` | In cancellation process | Cancellation has been requested. A job that is already running may sit here briefly before it reaches CANCELED. |
| `CANCELED` | Canceled by user | Stopped before it finished. The global job timeout and the cleanup of jobs whose workers died also end in this status. |
| `SUCCESS` | Completed successfully | Finished with no errors reported. |
| `WARNING` | Completed with warnings | Finished, but something in the job reported a warning. Read the job's log before treating the work as done. |
| `FAILURE` | Completed with errors | Finished with an error. The job engine itself is working; troubleshoot the job. |

<div class="alert alert-info" role="alert">


You can use `shell_plus` to list jobs by state:

```python
Job.objects.filter(status__in=Job.ACTIVE_STATUSES)
Job.objects.filter(status__in=Job.RUNNING_STATUSES)
Job.objects.filter(status__in=Job.COMPLETED_STATUSES)
```

</div>


## Configuration

The job engine runs under Supervisor on the appliance, configured by `/etc/supervisord.d/jobengine.conf`.

| Setting | Default | Description |
| --- | --- | --- |
| `numprocs` | `1` | How many worker processes the job engine runs. Each worker runs individual jobs in threads, and each process can use a separate CPU core. Raising this can improve throughput on a busy appliance. |
| `stopwaitsecs` | `600` | How long Supervisor waits for a worker to exit after telling it to stop, before forcing it with a kill signal. |
| `autostart` | `true` | Whether the workers start when Supervisor starts. Set it to `false` on a node that should not process jobs, such as a standby node in a high availability deployment. |

The file also carries other Supervisor settings, including how often a stopped worker is retried; see the [Supervisor documentation](http://supervisord.org/configuration.html). Upgrading CMP writes an example file, jobengine.conf.template. That file holds the built-in settings for the upgraded release. Use this as a reference when comparing any customized settings on your appliance.

### How many jobs run at once

`numprocs` sets how many workers exist. A separate Django setting, `MAX_JOB_CONCURRENCY`, sets how many jobs each worker runs at a time, and it defaults to 200. Override it in `customer_settings.py`.

Each running job holds a database connection, so a worker will stop claiming jobs when PostgreSQL runs short of connections. When a worker cannot start any jobs, it logs that it has no free threads.

### Change the number of worker processes

1. Edit `/etc/supervisord.d/jobengine.conf` and set `numprocs` to the desired number of workers.
1. Wait until no jobs are running, then reload Supervisor so it picks up the change: `supervisorctl reload`.

<div class="alert alert-warning" role="alert">


Reloading Supervisor restarts every service it manages, not only the job engine. Jobs running at the time are killed and left in RUNNING until the engine cleans them up.
</div>

---

Section pages:

- [Job engine troubleshooting](/docs/cmp/configure/administration/system/job-engine/troubleshooting/): When jobs stop moving, check the job's status first
- [Restart the job engine](/docs/cmp/configure/administration/system/job-engine/restart/): The job engine recycles its own workers as it runs
- [Run or exclude specific job types](/docs/cmp/configure/administration/system/job-engine/job-types/): A job engine can be limited to certain kinds of jobs