Restart the job engine

Restart them manually when one is stuck, and enable the automatic restart for jobs that the timeout cannot cancel.

A worker runs for at least a minute. Once it has finished the jobs it started and has nothing active, it exits, and Supervisor starts a new one. This keeps a long-running plugin from leaking memory into later jobs. A worker process ID that changes throughout the day is evidence of this cycle.

Restart the workers

Supervisor owns the worker processes, so to restart the job engine, you tell Supervisor to stop and start the processes:

supervisorctl restart jobengine:*

Supervisor starts a new worker in place of each one that stopped, and pending jobs begin running again.

Automatic restart after a stuck job

A job that outlives the global job timeout is canceled, and its status moves to TO_CANCEL. Cancellation cannot reach a job that is blocked outside Cloud Management Platform (CMP), though, such as one waiting on a call to a remote system that never returns. That job stays active, and the worker running it never frees up.

The job engine can break that deadlock. When a job has been active for twice the global job timeout and nothing else is running, the engine stops its own process so Supervisor can start a clean one. This occurs only when the normal cancellation path fails.

The global job timeout is Default Job Timeout in Miscellaneous Settings. It defaults to 8 hours, meaning the engine will consider a forced restart after 16 hours.

Enable the automatic restart

This behavior is disabled by default and is enabled via a feature flag. Run the following on the appliance:

sudo su -
/opt/cloudbolt/manage.py features -e "jobengine::hard_reset_past_timeout"

To check the current state of the flag, list the features with the -l option, and to disable it, run the command with -d.

Email alert

When the engine forces a restart and finds a job that has been active for three times the global job timeout, Cloud Management Platform (CMP) also sends an email.

• The subject is NOTICE: Long-running Job Engine. The body names the jobs still running and the timeout value.
• It goes to the addresses in CB Admin Email Address(es), configured on the Resource management tab of Miscellaneous Settings. If the field is empty, CMP logs it instead.
• The address the alert is sent from is determined by your email configuration.