Continuous Infrastructure Testing

Continuous Infrastructure Testing (CIT) can execute end-to-end functional tests on the elements of your virtualization environment that are most sensitive to change. You can create CIT tests that can be scheduled or run on demand and the feature provides web-based reporting and test management.

How Does It Work?

CIT can ensure that your environment is working correctly by re-running previously successful orders and making sure that they still succeed. Action tests can also be used to test anything you can write an action for. You can be immediately notified of test failures via email.

Why Is It Useful?

Modern virtualized IT environments are constantly changing. The unexpected effects of changes to storage, network, server configuration, passwords, DNS, VM images, and applications can cause failures in your VM environment.

Rather than testing infrastructure components in isolation, CIT executes end-to-end functional tests against the VM environment using the CloudBolt web interface. If the Continuous Infrastructure Tests pass, you have proven that all of the systems that participate in provisioning and decommissioning are functioning. Depending on your environment, these systems may include:

  • Network
  • Storage
  • Virtualization host systems
  • Public cloud providers
  • DNS
  • Asset tracking systems
  • Monitoring systems
  • Backup systems
  • Hostname reservation systems
  • Change management systems
  • Other infrastructure CloudBolt is managing

If a Continuous Infrastructure Test fails, the software will notify an administrator who can intervene to resolve the problem.

Functional Test Description

CIT’s provision and decommission tests, based on orders, verify that the following actions complete as expected:

  • Provision a server or resource with specified parameters
  • Install any applications included in the original order
  • Execute enabled orchestration hooks
  • Test the expected status and output of jobs
  • Decommission the server

Action tests can be used to verify any aspect of your environment that is not dependent on provisioning, whether that’s connecting to an external system for a status or just wanting to make sure your scripts are running successfully.

Getting Started with CIT

In order to make use of the CloudBolt CIT feature, start by creating your tests. A CIT test can be made from either an action or an order.

Create tests from orders

When creating CIT tests from orders, CIT operates by re-running previously successful or failed orders. So, the first step is to create these orders.

  1. Create a new order and ensure it gets submitted and approved.
  2. Follow the job progress to make sure the order completes.
  3. Once the order has completed, a new button will appear on the order details page labeled Add to CIT. This will add a CIT test for this order.

Create tests from actions

  1. Go to Admin > Continuous Infrastructure Testing
  2. Click on New Action Test, located near the top of the page.
  3. From here, you may choose an existing action or create a new one.

Running Tests

To run the Continuous Infrastructure Tests manually:

  1. In the CloudBolt web interface, click the Admin tab.
  2. In the Orchestration section, click the Continuous Infrastructure Testing link.
  3. The Continuous Infrastructure Testing list page loads.
  4. Click Execute Tests to run either all tests or a subset based on labels, or click a play button for individual testing.

Scheduling Tests

To set the schedule for your Continuous Infrastructure Tests, edit the schedule for the “Run all CIT Tests” recurring job. See Recurring Jobs for more details on scheduling.

Owner of Test Jobs & Orders

By default, when CIT tests are run by the recurring job, the jobs and orders that they create will not have an owner. If you need those jobs and orders to have an owner, you can use the Edit CIT Settings button on the Continuous Infrastructure Testing page to select a default owner for your CIT tests. When configured, CIT runs triggered by the recurring job will use that user as the owner for all their jobs and orders. We recommend that if you set a default owner for CIT tests, you may want to use a user with a name that makes it obvious it is not an actual human initiating these tests, such as cit_owner. Please also ensure that the selected user has any permissions necessary for your particular tests, although there are not any specific requirements out-of-the-box.

Note that when tests are run manually the user who started them will always be the owner, regardless of whether a default CIT owner is configured.

Retrying Tests

To make a test automatically retry on failure, edit the test details and set the “Max Retries” attribute. The test will automatically rerun until the test passes or the max number of retries is met. Tests that initially failed but succeeded on retry will be marked with a status of “Warning” instead of “Failure”.

CIT will wait for 10 minutes (default) before rerunning failed tests. To change the default, click Edit CIT Settings.

Emailing Reports

CloudBolt can email the results of Continuous Infrastructure Tests to one or more recipients. To configure email reports:

  1. In the CloudBolt web interface, click the Admin tab.
  2. In the Miscellaneous Settings section’s “CB Admin E-mail” field, type a comma-separated list of the email addresses you want CloudBolt to send Continuous Infrastructure Test reports to.
  3. Click Save changes.

CloudBolt will email the results of Continuous Infrastructure Tests on failure.

Parallel Execution of Tests

CIT is able to run groups of tests in parallel. It will look for tests with a label like “serialgroup:<group name>” and run the tests within the same group (exact same label) in serial, while letting the different groups run in parallel. Unlabelled tests will be treated as an implicit group, running in serial amongst themselves but at the same time as any defined groups. If you do not use any labels starting with “serialgroup:”, all CIT tests will run in serial.


Do not attempt to add more than one label starting with “serialgroup:” to any test. Each test can only be in one serial group.

Interpreting Test Results

In the Progress section of the Job Details page, you will see detailed output for each test. There you will see important details about failures and successes as well as links to any relevant jobs.