Orchestration

CloudBolt provides an orchestration framework that enables customization and extension of many back-end processes such as server provisioning and lifecycle management, user authentication, and order approval flows.

This framework has been used to:

  • Create DNS records for servers before they are provisioned
  • Automatically add servers to an external asset tracking system
  • Enable monitoring of new servers
  • Communicate with a storage array to allocate a new LUN when requested by the user
  • Send messages to chat systems when new resources or servers are built
  • Synchronize user-group membership with Active Directory when users log in

Ways to Orchestrate Your Cloud

Actions are executable code in the form of CloudBolt plug-ins, remote scripts, webhooks, emails, and external orchestration flows. Actions can modify jobs, orders, servers, groups, environments, etc. They can take additional input and can be used to communicate progress to the end user or instruct CloudBolt to make calls into external systems (ex. vCenter, AWS, ServiceNow, Infoblox, etc). For details, see Actions.

Ways in which actions may be executed (described in detail below):

  • Orchestration Actions - hook your custom logic into CloudBolt processes at various trigger points, e.g. during server ordering or provisioning.
  • Blueprint Actions - customize the deployment of your blueprint.
  • Server Actions - run actions on individual servers after they are provisioned.
  • Resource Actions - run actions on deployed resources.
  • Rules - periodically tested conditions that may execute actions.
  • External Orchestrators - use your existing orchestration content (HPSA, vCO) from within CloudBolt.

Content Library

The CloudBolt Content Library hosts samples of the above use cases. Admins can browse these from within the CB user interface and import them with a few clicks. To try it out, go to the admin view of orchestration actions, server actions, and resource actions as well as rules and recurring jobs.

Orchestration Actions

Orchestration Actions can be triggered at one of the many pre-defined trigger points that exist throughout the product. Four categories of action triggers and the points where they are invoked are described below. If any of these points invoke actions that require additional inputs, an admin is expected to provide defaults for those action while running in this context. The same underlying action can take different defaults when invoked from different trigger points.

An Orchestration Action can be set to continue on failure, which means that if it fails that will not impact the rest of the job in which the Orchestration Action runs. It will not cause the overall job to fail or change the flow of the job. This can be useful for actions that occasionally fail but whose failure does not interfere with the rest of the job completing successfuly. Note that there are a few exceptions to this behavior (namely the 2 Parameter Trigger Points below, Generate Hostname Overwrite, Pre-Server Refresh, Order Form Validation, and Compute Server Rate), and other trigger points where using it may not be very logical (e.g., Validate Hostname).

Job Trigger Points

Actions can be executed at many points throughout the job flow. The list of trigger points are found at Admin -> Orchestration Actions and are grouped by job type. For each of these job types, actions are executed in the order they are listed where <job-type> represents the job being run:

  1. Pre-Job [1]
  2. Pre-<job-type>
  3. The actual job is run [2]
  4. Post-<job-type> [3]
  5. Post-Job [1] [3]
[1](1, 2) Pre-Job and Post-Job action triggers can be found under the “Other” tab on the Orchestration Actions list page
[2]Jobs can call out to additional actions at trigger points that are specific to that job type (the Provision Job for example)
[3](1, 2) By default, the post job actions will run regardless of the results of the main job, even if the job fails. However, starting in CB 5.4 it is possible to edit the action at that trigger point and set it to only run if the main job has a particular status.

Provisioning Job Trigger Points

The “Provision Server” job has trigger points that happen during the job (not just pre and post-job). The Orchestration Actions list page shows these trigger points in the order that they are executed during a provision job.

Order Trigger Points

There are three trigger points relating to orders:

  • Order Approval - executes when an order is submitted for approval. Mainly used to integrate CloudBolt with third party change management systems.
  • Pre-Order Execution - executes after an order is approved and before jobs are created. Used to alter job parameters before they are executed.
  • Post-Order Completion - executes when an order is marked as completed (either as a success or a failure).

Parameter Trigger Points

There are two trigger points relating to parameters:

  • Parameter Change - executes whenever a user changes a parameter on a server. This can be used to alter the server or contact another system when users change parameter values on servers.
  • Generated Parameter Options - used to dynamically determine what options should be presented to a user for a parameter. Can also specify an initial value amongst those options. Contain a get_options_list method that returns either a list of option tuples or a dictionary with both those and the initial value. See the documentation on providing options for action inputs in CloudBolt plug-ins, where the method has the same possible return values.

User Trigger Points

The one user-related trigger point, External Users Sync, executes selected actions each time an external user (AD/LDAP) logs in. An action can be used to update users’ permissions and group membership in CloudBolt based on data in AD.

Blueprint Actions

A blueprint can include one or more server tiers. However, there is sometimes a need to add custom logic to the blueprint deployment process and Blueprint Actions enable this.

Admins manage blueprint actions in the Build tab of a blueprint detail view. An action item can be associated with all server tiers or only specific ones. In addition, it can be given a name and description that makes more sense in the context of that blueprint.

Custom Actions on Servers and Resources

It is possible to expose arbitrary actions on deployed servers and resources, that appear as buttons to users on the server and resource details views. These can be managed at Admin -> Server Actions and Admin -> Resource Actions. When executed, users will be prompted for any inputs needed, and then CloudBolt jobs will be created to execute the actions asynchronously, and also to keep an audit trail of who ran what actions on what resources when.

When creating a server or resource action, it is possible to add a new action or choose an existing shared action with a label that will be presented to end-users. All of the Action Types are supported.

If using a CloudBolt plugin, the job parameters will include a servers attribute of all servers the action is being run on. In the case of a Resource Action, this will include all servers in the resource, and a resources attribute will also be available on the job parameters.

Troubleshooting tip: If an action is not showing up on the server detail page check that the server has an OS build. If it does not, use the ‘Change Attributes’ button to assign one.

Rules

Rules allow CloudBolt admins to define conditional logic that is periodically executed and that may trigger custom actions based on the result. They can be used, for example, to automate policy enforcement through corrective actions, adapt proactively based on the state of your environment, or perform health checks and notify stakeholders of required actions.

Enabled rules run on a regular schedule as defined by a CloudBolt recurring job which is run nightly at midnight by default (see Admin -> Recurring Jobs). You can also run rules on demand from the Admin -> Rules page in the UI, either individually or in bulk.

A rule is a logical statement comprised of a condition action (“If”) and one or more actions that are triggered (“Then”) as a result of a positive test.

Action Types for rules

Both the Condition and Action(s) of a rule are CloudBolt Actions of type CloudBolt Plug-in. For implementation details, refer to the CloudBolt Plug-ins section. However, be aware that the underlying Python method and return signatures for a Condition CloudBolt Plug-in differ from other CloudBolt Plug-ins in two ways:

  • Condition CloudBolt Plug-ins return a four-tuple (instead of a three-tuple); the additional value is a parameters dictionary which must have a value (not None) for the condition to be satisfied; otherwise, if the fourth tuple member is None, the condition is not satisfied and the Actions do not run.
  • Condition CloudBolt Plug-ins are called with a function called check(), instead of a function called run() as in Orchestration Actions or Server Actions.
check(job, logger)

Is passed a CloudBolt Job object and a standard Python Logger object (http://docs.python.org/2/library/logging.html). Should return a tuple of 3 strings and 1 dictionary: (status, output, errors, params). On success, the plug-in should return (“”, “”, “”, {params_dictionary_values}) if the condition is met or (“”, “”, “”, None) if the condition is not met.

If the params dictionary returned is None, the job will end and the Actions for this rule will not run.

If the params dictionary has values, the Actions will run as separate jobs and will each be passed the params dictionary via the job parameters associated with each sub job created.

Out-of-the-box Rules

CloudBolt ships with a set of sample rules, some of which are enabled by default. Go to Admin -> Rules to view and enable/disable them. Since rules are made of actions, you can view and modify their source code by visiting Admin -> All Actions.

Below are some examples of rules that ship with CloudBolt.

Email CB Admin when License Warning for Thresholds

This rule ships out of the box with CloudBolt 5.3 and above. It is enabled by default.

It sends an email alert to the CB admin when the CB license is going to expire in less than a threshold number of days or more than a threshold percentage of its maximum number of servers are in use. The out of the box value for the expiration days threshold is 7 and for the percent of servers used threshold it is 75%.

Notify Admins when a Newer Version of CloudBolt is Available

This rule ships out of the box with CloudBolt 5.3.1 and above, and is enabled by default.

It checks for a version of CloudBolt newer than what you are running, stores that information, and emails the CB Admin if there is one.

Email Group Admins when Quota Usage over Threshold

This rule ships with CloudBolt out of the box with CloudBolt 5.1 and above. It is disabled by default.

When a Group’s quota reaches a defined threshold, sends email alerts to all users who have the group.manage_subgroup_quotas permission as a virtue of their roles in that Group. The out of the box value for the threshold is 80%.

Email owners of AWS VMs Missing Required Tags

Checks servers in AWS environments for a set of required AWS tags and emails the server owner if any are missing. Added in CloudBolt 5.2 and disabled by default.