Continuous Infrastructure Testing

Continuous Infrastructure Testing (CIT) executes end-to-end functional tests on the elements of your virtualization environment that are most sensitive to change. CIT ships with tests that can be scheduled or run on demand and provides web-based reporting and test management.

How Does It Work?

CIT ensures that your environment is working correctly by re-running previously successful orders and making sure that they still succeed. You can be immediately notified of failures via email.

Why Is It Useful?

Modern virtualized IT environments are constantly changing. The unexpected effects of changes to storage, network, server configuration, passwords, DNS, VM images, and applications can cause failures in your VM environment.

Rather than testing infrastructure components in isolation, CIT executes end-to-end functional tests against the VM environment using the CloudBolt web interface. If the Continuous Infrastructure Tests pass, you have proven that all of the systems that participate in provisioning and decommissioning are functioning. Depending on your environment, these systems may include:

  • Network
  • Storage
  • Virtualization host systems
  • Public cloud providers
  • DNS
  • Asset tracking systems
  • Monitoring systems
  • Backup systems
  • Hostname reservation systems
  • Change management systems
  • Other infrastructure CloudBolt is managing

If a Continuous Infrastructure Test fails, the software will notify an administrator who can intervene to resolve the problem.

Configuring CIT

Use the following procedure before your first use of the CloudBolt CIT feature:

Create your tests

A CIT can be made from either an action or an order.

Create tests from orders

When creating CIT tests from orders, CIT operates by re-running previously successful or failed orders. So, the first step is to create these orders.

  1. Create a new order and ensure it gets submitted and approved.
  2. Follow the job progress to make sure the order completes.
  3. Once the order has completed, a new button will appear on the order details page labeled Add to CIT. This will add a CIT test for this order.

Create tests from actions

  1. Go to Admin > Continuous Infrastructure Testing
  2. Click on New Action Test, located near the top of the page.
  3. From here, you may choose an existing action or create a new one.

Functional Test Description

CIT’s provision and decommission test verifies that the following actions complete as expected:

  • Provision a server or service with specified parameters
  • Install any applications included in the original order
  • Execute enabled orchestration hooks
  • Test the expected status and output of jobs
  • Decommission the server

Running Tests

To run the Continuous Infrastructure Tests:

  1. In the CloudBolt web interface, click the Admin tab.
  2. In the Jobs section, click the Continuous Infrastructure Testing link.
  3. The Continuous Infrastructure Testing list page loads.
  4. Click Execute All Tests or click a play button for individual testing.

Scheduling Tests

To set the schedule for your Continuous Infrastructure Tests, edit the schedule for the “Run all CIT Tests” recurring job. See Recurring Jobs for more details on scheduling.

Retrying Tests

To make a test automatically retry on failure, edit the test details and set the “Max Retries” attribute. The test will automatically rerun until the test passes or the max number of retries is met. Tests that initially failed but succeeded on retry will be marked with a status of “Warning” instead of “Failure”.

Emailing Reports

CloudBolt can email the results of Continuous Infrastructure Tests to one or more recipients. To configure email reports:

  1. In the CloudBolt web interface, click the Admin tab.
  2. In the Miscellaneous Settings section’s “CB Admin E-mail” field, type a comma-separated list of the email addresses you want CloudBolt to send Continuous Infrastructure Test reports to.
  3. Click Save changes.

CloudBolt will email the results of Continuous Infrastructure Tests on failure.

Parallel Execution of Tests

CIT is able to run groups of tests in parallel. It will look for tests with a label like “serialgroup:<group name>” and run the tests within the same group (exact same label) in serial, while letting the different groups run in parallel. Unlabelled tests will be treated as an implicit group, running in serial amongst themselves but at the same time as any defined groups. If you do not use any labels starting with “serialgroup:”, all CIT tests will run in serial.


Do not attempt to add more than one label starting with “serialgroup:” to any test. Each test can only be in one serial group.

Interpreting Test Results

In the Progress section of the Job Details page, you will see detailed output for each test. There you will see important details about failures and successes as well as links to any relevant jobs.