CloudBolt High Availability (HA) Guide

Overview

This guide contains a general overview of the CloudBolt high availability solution, and describes the process of setting up your environment. Although this guide will use HAProxy the overall outcome is generally the same regardless of your chosen technology.

If high availability is a requirement for your organization a CloudBolt application cluster can provide horizontal scalability, high availability of servers via fail-over protection as well as load balanced requests to the CloudBolt application.

The process is as follows:

  1. Deploy and configure a load balancer to direct traffic.

  2. Deploy the most recent CloudBolt OVA four times:

    1. Two will be used for the CloudBolt web application
    2. Two will be used as replicated databases
../../_images/HA.png

Deployment Steps

The steps to deploy the solutions are as follows:

  1. Deploy or leverage your existing load balancer. For this guide, a single CentOS 6 (ha-proxy, 192.168.1.100) server running HAProxy was deployed and configured using the following commands:
yum -y update
yum -y install haproxy
sed -i “/127.0.0.1:500/d” /etc/haproxy/haproxy.cfg
sed -i “s/frontend main *:5000/frontend main *:80/” /etc/haproxy/haproxy.cfg
service haproxy start
  1. Deploy the CloudBolt appliance (.ova) into the desired environment four times.

    1. All the necessary components are included in the appliance.

    2. Configure the network so that each appliance can communicate with each other.

    3. Ensure that each VM is given a descriptive hostname. For this guide the following naming convention is used:

      > CloudBolt App 1 - cb-ha1

      > CloudBolt App 2 - cb-ha2

      > CloudBolt DB 1 - cb-db1

      > CloudBolt DB 2 - cb-db2

  2. For all for conventions (cb-ha1/2 & cb-db1/2):

    1. Login with (root:cloudbolt)
    2. Configuration wizard will launch and guide you through the network config
    3. Provide the appropriate hostname/FQDN, IP, gateway, DNS server, and save the config
  3. On the main CloudBolt appliance (cb-ha1):

    1. Navigate to the web interface https://192.168.1.101 and upload an appropriate license file
    2. Run ssh root@cb-ha1 and copy the DATABASES section:

    Copy the DATABASES section from /opt/cloudbolt/settings_local.py to /var/opt/cloudbolt/proserv/customer_settings.py. Then change ‘HOST’ to:

    'HOST': '<FQDN-of-DB1>',
    
    1. To ensure rsync jobs can run via cron, set up password-less ssh to cb-ha2

      > Run ssh-keygen -t rsa

      > Copy ssh-copy-id -i .ssh/id_rsa.pub root@cb-ha2

    2. Setup rsync via cron between cb-ha1 & cb-ha2:

    1.0,15,30,45 * * * * rsync -av /var/opt/cloudbolt/ root@cb-ha2:/var/opt/cloudbolt/ >> <log_location> 2>&1
    2.0,15,30,45 * * * * rsync -av /var/www/html/cloudbolt/static/uploads/ root@cb-ha2:/var/www/html/cloudbolt/static/uploads/ >> <log_location> 2>&1
    3.0,15,30,45 * * * * rsync -av /var/log/cloudbolt/jobs/ root@cb-ha2:/var/log/cloudbolt/jobs/ >> <log_location> 2>&1
    
    1. Create a new RabbitMQ user, as the default ‘guest’ account can only connect via localhost. Be sure to choose a strong password.
    rabbitmqctl add_user cloudbolt <PASSWORD>
    rabbitmqctl set_user_tags cloudbolt administrator
    

    Add the following to /var/opt/cloudbolt/proserv/customer_settings.py, using the password created for the previous step:

    CELERY_BROKER_URL = "amqp://cloudbolt:<PASSWORD>@localhost:5672"
    
  4. On the secondary CloudBolt appliance (ssh root@cb-ha2):

    1. Stop/disable mysqld:

      > Run service mysqld stop

      > Run chkconfig mysqld off

    2. Stop Job Engine processes:

      > Disable the scheduling cron jobs: crontab -e

      # * * * * * /usr/local/bin/python /opt/cloudbolt/manage.py run_recurring_jobs > /dev/null 2>&1
      # * * * * * /usr/local/bin/python /opt/cloudbolt/manage.py run_scheduled_jobs > /dev/null 2>&1
      

      > Stop the Job Engine workers process: supervisorctl stop celeryd:*

      > Stop the Job Engine queue process: service rabbitmq-server stop

      > Run chkconfig celeryd off

      > Run chkconfig rabbitmq-server off

    3. Point to cb-ha1’s Job Engine queue:

    Add the following to /var/opt/cloudbolt/proserv/customer_settings.py, using the password created on cb-ha1:

    CELERY_BROKER_URL = "amqp://cloudbolt:<PASSWORD>@<FQDN-of-HA1>:5672"
    
    1. Copy the DATABASES section:

    Copy the DATABASES section from /opt/cloudbolt/settings_local.py to /var/opt/cloudbolt/proserv/customer_settings.py. Then change ‘HOST’ to:

    'HOST': '<FQDN-of-DB1>',
    
  5. On the main CloudBolt Database appliance (cb-db1):

    1. Stop Apache:

      > Run service httpd stop

      > Run chkconfig httpd off

    2. Stop Job Engine processes:

      > Disable job scheduling cron jobs: crontab -e

      # * * * * * /usr/local/bin/python /opt/cloudbolt/manage.py run_recurring_jobs > /dev/null 2>&1
      # * * * * * /usr/local/bin/python /opt/cloudbolt/manage.py run_scheduled_jobs > /dev/null 2>&1
      

      > Stop the Job Engine workers process: supervisorctl stop celeryd:*

      > Stop the Job Engine queue process: service rabbitmq-server stop

      > Run chkconfig celeryd off

      > Run chkconfig rabbitmq-server off

    3. Enable one way replication:

      > Edit /etc/my.cnf and comment out the line:

      bind-address=127.0.0.1
      

      > Edit /etc/my.cnf and add the following lines:

      log-bin = mysql-bin
      
      server-id = 1
      

      > Restart mysqld: service mysqld restart

      Note: For one way replication reference see How to setup MySQL Master/Slave Replication

  6. On the secondary CloudBolt Database appliance (cb-db2):

    1. Stop httpd

      > Run service httpd stop

      > Run chkconfig httpd off

    2. Edit /etc/my.cnf and comment out the line:

    bind-address=127.0.0.1
    
    1. Set up password-less ssh to cb-ha1

      > Run ssh-keygen -t rsa

      > Copy ssh-copy-id -i .ssh/id_rsa.pub root@cb-ha1

Steps to upgrade CloudBolt in HA environment

  1. Backup all four instances (cb-ha1/2 & cb-db1/2)

  2. Set cb-ha2 to maintenance mode to ensure no changes occur to the database (cb-db1/2) during the upgrade of cb-ha1

    1. In cb-ha2 run /opt/cloudbolt/utilities/maintenance_mode.py on
  3. Run the upgrader on cb-ha1

    1. Follow the Upgrading CloudBolt steps, but skip step #8 that deals with the install_config file
  4. Set cb-ha1 to maintenance mode to ensure no changes occur to the database (cb-db1/2) during the upgrade of cb-ha2

    1. In cb-ha1 run /opt/cloudbolt/utilities/maintenance_mode.py on
  5. Run the upgrader on cb-ha2. This is necessary, even though cb-ha1 and cb-ha2 were configured to sync CloudBolt code, because the upgrade may include updates to supporting packages that cb-ha2 should have as well

    1. Follow the Upgrading CloudBolt steps, but skip step #8 that deals with the install_config file
  6. Disable maintenance mode on cb-ha1

    1. In cb-ha1 run /opt/cloudbolt/utilities/maintenance_mode.py off

NOTE No need to run the upgrader in either database instance (cb-db1/2)