CloudBolt High Availability Guide

Overview

This guide contains a general overview of the CloudBolt high availability solution, and describes the process of setting up your environment. Although this guide will use HAProxy the overall outcome is generally the same regardless of your chosen technology.

If high availability is a requirement for your organization a CloudBolt application cluster can provide horizontal scalability, high availability of servers via fail-over protection as well as load balanced requests to the CloudBolt application.

The process is as follows:

  1. Deploy and configure a load balancer to direct traffic.

  2. Deploy the most recent CloudBolt OVA four times:

    1. Two will be used for the CloudBolt web application
    2. Two will be used as replicated databases
../../_images/HA.png

Deployment Steps

The steps to deploy the solutions are as follows:

  1. Deploy or leverage your existing load balancer. For this guide, a single CentOS 6 (ha-proxy, 192.168.1.100) server running HAProxy was deployed and configured using the following commands:
yum -y update
yum -y install haproxy
sed -i “/127.0.0.1:500/d” /etc/haproxy/haproxy.cfg
sed -i “s/frontend main *:5000/frontend main *:80/” /etc/haproxy/haproxy.cfg
service haproxy start
  1. Deploy the CloudBolt appliance (.ova) into the desired environment four times.

    1. All the necessary components are included in the appliance.

    2. Configure the network so that each appliance can communicate with each other.

    3. Ensure that each VM is given a descriptive hostname. For this guide the following naming convention is used:

      > CloudBolt App 1 - cb-ha1

      > CloudBolt App 2 - cb-ha2

      > CloudBolt DB 1 - cb-db1

      > CloudBolt DB 2 - cb-db2

  2. For all for conventions (cb-ha1/2 & cb-db1/2):

    1. Login with (root:cloudbolt)
    2. Configuration wizard will launch and guide you through the network config
    3. Provide the appropriate hostname/FQDN, IP, gateway, DNS server, and save the config
  3. On the main CloudBolt appliance (cb-ha1):

    1. Navigate to the web interface https://192.168.1.101 and upload an appropriate license file
    2. Run ssh root@cb-ha1 and copy the DATABASES section:
    From  /opt/cloudbolt/settings_local.py To
     /var/opt/cloudbolt/proserv/customer_settings.py
    
     Under DATABASES section, add 'HOST': '<FQDN-of-DB1>',
    
    1. To ensure rsync jobs can run via cron, set up password-less ssh to cb-ha2

      > Run ssh-keygen -t rsa

      > Copy ssh-copy-id -i .ssh/id_rsa.pub root@cb-ha2

    2. Setup rsync via cron between cb-ha1 & cb-ha2:

    1.0,15,30,45 * * * * rsync -av /var/opt/cloudbolt/ root@cb-ha2:/var/opt/cloudbolt/ >> <log_location> 2>&1
    2.0,15,30,45 * * * * rsync -av /var/www/html/cloudbolt/static/uploads/ root@cb-ha2:/var/www/html/cloudbolt/static/uploads/ >> <log_location> 2>&1
    3.0,15,30,45 * * * * rsync -av /var/log/cloudbolt/jobs/ root@cb-ha2:/var/log/cloudbolt/jobs/ >> <log_location> 2>&1
    
  4. On the secondary CloudBolt appliance (ssh root@cb-ha2):

    1. Stop/disable mysqld:

      > Run service mysqld stop

      > Run chkconfig mysqld off

    2. Copy the DATABASES section:

    From  /opt/cloudbolt/settings_local.py To
     /var/opt/cloudbolt/proserv/customer_settings.py
    
      Under the DATABASES section, add 'HOST': '<FQDN-of-DB1>',
    
  5. On the main CloudBolt Database appliance (cb-db1):

    1. Stop httpd & CB processes:

      > Run service httpd stop

      > Run chkconfig httpd off

      > Disable jobegine: crontab -e

      # * * * * * /opt/cloudbolt/jobengine/runjobs.sh >>
      /var/log/cloudbolt/runjobs.log 2>&1
      

      > Kill relevant CB process: pkill runjobs

    2. Enable one way replication:

      > Edit /etc/my.cnf and comment out the line:

      bind-address=127.0.0.1
      

      > Edit /etc/my.cnf and add the following lines:

      log-bin = mysql-bin
      
      server-id = 1
      

      > Restart mysqld: service mysqld restart

      Note: For one way replication reference see How to setup MySQL Master/Slave Replication

  6. On the secondary CloudBolt Database appliance (cb-db2):

    1. Stop httpd

      > Run service httpd stop

      > Run chkconfig httpd off

    2. Edit /etc/my.cnf and comment out the line:

    bind-address=127.0.0.1
    
    1. Set up password-less ssh to cb-ha1

      > Run ssh-keygen -t rsa

      > Copy ssh-copy-id -i .ssh/id_rsa.pub root@cb-ha1