Discovery/sync configuration

This is especially useful for bringing brownfield environments under management. The main component of the discovery/sync logic is the Discovery plugin. You can also choose whether an existing resource associated with the blueprint is automatically marked Historical if the Discovery plugin does not find it when it runs. The Sync Resources recurring job must be enabled for this.

Writing a Discovery plugin

A Discovery plugin is optional, but useful when you want to make management resource actions available as self-service for previously deployed services. The plugin can import and synchronize the state of resources into Cloud Management Platform (CMP). Before the plugin runs, enable the Sync Resources recurring job; it is disabled by default.

Discovery plugins also help if you copy state data into CMP parameters: the plugin can refresh that information automatically so it does not go stale.

Some details of your Discovery plugin vary depending on its purpose and the external system it connects to, but some aspects of the structure are prescribed.

Planning

• For the type of resources you want to discover and sync, determine which of their attributes uniquely identify them. This identifier only needs to be unique across resources associated with this particular blueprint.
• Then determine the other attributes you want to store about them, including the type of value each has.
• Consider permissions: which group should own the discovered resources, and who their owner should be (if anyone).

Requirements

• The plugin must define a global variable RESOURCE_IDENTIFIER whose value is a string or an iterable collection of strings, matching the keys in the discovered dictionaries whose values uniquely identify resources (within the resources associated with that blueprint).
• The plugin must contain a method called discover_resources that returns a list of dictionaries, where each dictionary represents a discovered resource.
• All keys in those dictionaries should map to attributes that can be set on a resource object. This includes resource model fields and parameters (using their name as the key). If a key does not match a field or an existing parameter, a new parameter is created with that name, and its label is derived from the name. If the value is an integer, decimal/float, datetime object, or boolean, the parameter is of that type; otherwise it defaults to a string. If the created type is not what you intended, delete that parameter and create one that matches your needs. All other parameter attributes are set to their defaults.
• The plugin code must be added to the Discovery tab on the blueprint.

Special cases for setting attributes

• Some resource model fields are always set by CMP and are ignored if included in a discovered dictionary: the blueprint the resource is associated with (always the one with the Discovery plugin), the date and time the resource was created, and the jobs associated with the resource. A new resource takes its resource type from the blueprint unless the dictionary provides one.
• Some resource model fields can be provided by the discovered dictionary, but are set to default values if not. If no group is provided, the “Unassigned” group is used. If a name is not provided, the value of the first resource identifier is used. If you want all identifying attributes used in a new resource’s name, or you have other naming conventions, we strongly suggest passing a ‘name’ attribute in the discovered dictionary. If no valid value is provided for the lifecycle, it defaults to Active.
• To set the owner or group of the resource, find and return a UserProfile or Group object from the discovered dictionary, respectively.
• Whenever an invalid value is provided for an attribute, it is not set. This ranges from the wrong type of value for a parameter to something that is not actually a Group object. The values in the discovered dictionary should be the intended type.

Below is a skeleton example of a Discovery plugin. It defines the resource identifier that every Discovery plugin requires, then uses the AWS resource handler to connect to AWS and look for S3 buckets. In this example, the value specified for the resource identifier is used to name each discovered S3 bucket when the plugin code runs.

"""
Discover and create S3 Bucket records with some basic identifying attributes

As all Discovery Plug-ins must do, we define the global `RESOURCE_IDENTIFIER` variable
and return a list of dictionaries from the `discover_resources` function
"""
from botocore.client import ClientError
from common.methods import set_progress
from resourcehandlers.aws.models import AWSHandler

RESOURCE_IDENTIFIER = 's3_bucket_name'


def discover_resources(**kwargs):

    discovered_buckets = []
    for handler in AWSHandler.objects.all():
        try:
            wrapper = handler.get_api_wrapper()
        except Exception as e:
            set_progress(f"Could not get wrapper: {e}")
            continue
        set_progress('Connecting to Amazon S3 for handler: {}'.format(handler))
        conn = wrapper.get_boto3_resource(
            handler.serviceaccount,
            handler.servicepasswd,
            None,
            service_name='s3'
        )

        try:
            for bucket in conn.buckets.all():
                discovered_buckets.append({
                    "name": bucket.name,
                    "s3_bucket_name": bucket.name,
                    "aws_rh_id": handler.id,
                    "created_in_s3": str(bucket.creation_date)
                })
        except ClientError as e:
            set_progress('AWS ClientError: {}'.format(e))
            continue

    return discovered_buckets

Updatable attributes and resource identifiers

In addition to all parameters that can be defined as attributes of a resource in its resource type definition, there are direct CMP model attributes available on all resources that can be set or updated during discovery.

An important consideration when writing Discovery plugins is deciding which attributes identify existing resources to update, versus resources to create. Every Discovery plugin must define at least one identifier, using one of these forms:

• RESOURCE_IDENTIFIER = 'xaas_resource_id' — a single identifying field
• RESOURCE_IDENTIFIER = ['xaas_resource_id', 'name', 'xaas_resource_region'] — multiple identifying fields, as a list

The following model fields can be used by Discovery plugins:

Advanced

Considerations

Example code: see hands-on-lab/xaas_discovery.py.

How a sync runs

Each time the Sync Resources job runs, it looks for every active blueprint that has a Discovery plugin and runs that plugin’s logic. The dictionaries the plugin returns are matched against the resources already associated with that blueprint, using the values of the attributes named as resource identifiers. A dictionary that matches an existing resource updates it; one that matches nothing creates a resource.

Updating covers parameter values as well as model fields, which is what keeps discovered data current. If a resource has a value of 42 for an Available Storage parameter and the matching dictionary returns 24 the next time the job runs, the resource’s value becomes 24.

CMP runs each blueprint’s sync as its own job, so one blueprint failing does not stop the others from syncing. Within a single blueprint, individual dictionaries or keys can be skipped without failing the job; those are recorded in the job log.

To sync one blueprint without waiting for the job, open its Discovery tab and select Sync Resources for this Blueprint.

Automatically mark resources historical

The Discovery tab carries an Automatically Mark Resources Historical switch, off by default. Turn it on and any resource associated with the blueprint whose identifier value appears in none of the discovered dictionaries is marked historical, on the assumption that it no longer exists. That changes where the resource is visible in CMP, and it is the reason a resource identifier has to be stable: an identifier that changes on the provider side reads as a deletion.

Next steps

After you add the Discovery plugin on the blueprint’s Discovery tab, go to Admin > Recurring Jobs and enable the Sync Resources job. It is disabled by default.