# Restart the job engine

> The job engine recycles its own workers as it runs

---

Site [llms.txt](/llms.txt)

---

Restart them manually when one is stuck, and enable the automatic restart for jobs that the timeout cannot cancel.

A worker runs for at least a minute. Once it has finished the jobs it started and has nothing active, it exits, and Supervisor starts a new one. This keeps a long-running plugin from leaking memory into later jobs. A worker process ID that changes throughout the day is evidence of this cycle.

## Restart the workers

Supervisor owns the worker processes, so to restart the job engine, you tell Supervisor to stop and start the processes:

```bash
supervisorctl restart jobengine:*
```

Supervisor starts a new worker in place of each one that stopped, and pending jobs begin running again.

<div class="alert alert-warning" role="alert">


Restarting kills running jobs, and those jobs remain in RUNNING status. The job engine cancels them when it next starts. Restart when the queue is quiet.
</div>


## Automatic restart after a stuck job

A job that outlives the global job timeout is canceled, and its status moves to TO_CANCEL. Cancellation cannot reach a job that is blocked outside Cloud Management Platform (CMP), though, such as one waiting on a call to a remote system that never returns. That job stays active, and the worker running it never frees up.

The job engine can break that deadlock. When a job has been active for twice the global job timeout and nothing else is running, the engine stops its own process so Supervisor can start a clean one. This occurs only when the normal cancellation path fails.

The global job timeout is **Default Job Timeout** in [Miscellaneous Settings](/docs/cmp/configure/administration/system/miscellaneous-settings/). It defaults to 8 hours, meaning the engine will consider a forced restart after 16 hours.

### Enable the automatic restart

This behavior is disabled by default and is enabled via a feature flag. Run the following on the appliance:

```bash
sudo su -
/opt/cloudbolt/manage.py features -e "jobengine::hard_reset_past_timeout"
```

To check the current state of the flag, list the features with the `-l` option, and to disable it, run the command with `-d`.

### Email alert

When the engine forces a restart and finds a job that has been active for three times the global job timeout, Cloud Management Platform (CMP) also sends an email.

- The subject is **NOTICE: Long-running Job Engine**. The body names the jobs still running and the timeout value.
- It goes to the addresses in **CB Admin Email Address(es)**, configured on the **Resource management** tab of **Miscellaneous Settings**. If the field is empty, CMP logs it instead.
- The address the alert is sent from is determined by your [email configuration](/docs/cmp/configure/administration/system/email/).

## Related

- [Job engine](/docs/cmp/configure/administration/system/job-engine/): How jobs move through the engine, status meanings, and settings controlling workers.
- [Job engine troubleshooting](/docs/cmp/configure/administration/system/job-engine/troubleshooting/): What to check for each stuck status, and where to find the logs on the appliance.