Provisioning with the Nutanix v4 API

How Cloud Management Platform (CMP) builds, customizes, and checks a VM on a Nutanix resource handler that uses API version v4, and how to configure your answer files and orchestration actions for it

This page applies only to Nutanix resource handlers set to API version v4. On v4, CMP provisions from Nutanix VM Templates as well as disk images. It customizes the VM before its first boot, installs Nutanix Guest Tools (NGT), and waits until the guest has finished customizing before it continues.

How a v4 provision runs

  1. Nutanix creates the VM from the image or VM Template, powered off, with every network interface from the order attached.
  2. The Pre-Network Configuration orchestration actions run before the VM’s first boot and can change its hardware. At this point, the Acropolis Attach NGT CD-ROM action attaches the NGT ISO.
  3. CMP powers the VM on. Guest customization (Sysprep or cloud-init) runs and installs NGT.
  4. Guest customization ends by powering the VM off, which tells CMP that customization has finished.
  5. While the VM is off, CMP empties its CD-ROMs. For when CMP also removes the NGT CD-ROM, see CD-ROM handling.
  6. CMP powers the VM back on and waits for the guest to report ready. See OS readiness and timeouts.
  7. The Post-Provision orchestration actions run. The Acropolis Eject CD-ROM Media action runs last and checks that no ISO is left mounted.

Built-in orchestration actions

CMP includes two orchestration actions for Nutanix. Both are enabled and run only on Nutanix resource handlers that use API version v4.

Action Trigger point What it does Skipped when
Acropolis Attach NGT CD-ROM Pre-Network Configuration Before the VM first powers on, puts the guest customization ISO on the first CD-ROM of a bus and attaches the VM’s own NGT ISO on the next one. The handler does not use v4, or the provisioning job has failed.
Acropolis Eject CD-ROM Media Post-Provision Ejects every ISO still mounted on the VM, and fails if any media is still mounted afterward. It also runs for failed provisions whose VM still exists. The handler does not use v4, the provision failed before a VM existed, the VM no longer exists in Nutanix, or the server has skip_power_on set.

Note the following:

  • Keep Acropolis Eject CD-ROM Media last in Post-Provision. If an earlier Post-Provision action fails and is not set to continue on failure, the eject action does not run and media stays mounted.
  • Disabling either action persists through upgrades. CMP enables them only on creation.
  • Do not rename them. Upgrades find the actions by name and create new copies if they are renamed.
  • With Acropolis Attach NGT CD-ROM disabled, NGT is not installed. On Nutanix IPAM subnets, CMP then can’t verify that the guest is ready. See When a signal does not arrive.

Guest customization and NGT

CMP customizes each new VM with one of these:

  • CMP’s default file: used when the server has no Acropolis Sysprep File (acropolis_sysprep_file) for Windows or Acropolis User Data (acropolis_user_data) for Linux. The default installs NGT and powers the VM off when it finishes.
  • Your own file: set in Acropolis Sysprep File or Acropolis User Data. CMP uses it unchanged. It must install NGT and end by powering the VM off. See What to add to your own files.
Default file Windows Linux
Location /opt/cloudbolt/resourcehandlers/windows_unattend.xml, shared with Hyper-V /opt/cloudbolt/resourcehandlers/acropolis/linux_cloud_config.yaml.tmpl
How it installs NGT A one-time Administrator AutoLogon runs three FirstLogonCommands: wait for the NUTANIX_TOOLS drive, install NGT, power off. runcmd mounts the NUTANIX_TOOLS drive, runs the installer, and waits for the NGT service. power_state then powers the VM off.
Logs in the guest C:\Windows\Temp\NGT-FirstLogon.log and C:\Windows\Temp\NGT-setup.log /var/log/ngt-install.log

Windows default answer file

CMP adds an AutoLogon and three FirstLogonCommands to the oobeSystem pass of the default answer file. FirstLogonCommands run only when someone signs in. A one-time AutoLogon signs in as Administrator with the server’s password to start them. The commands:

  1. Wait up to one minute for the NUTANIX_TOOLS drive.
  2. Install NGT, unless it is already installed. Before it installs, the command waits up to five minutes for Windows Installer to finish any work left from Sysprep.
  3. Power the VM off, whether or not NGT was installed.

To install NGT from your own answer file, add a block like this one to the oobeSystem pass, before <OOBE>:

<AutoLogon>
  <Enabled>true</Enabled>
  <LogonCount>1</LogonCount>
  <Username>Administrator</Username>
  <Password>
    <Value>{{ sysprep_password|salt_and_encode_windows_password:"Password" }}</Value>
    <PlainText>false</PlainText>
  </Password>
</AutoLogon>
<FirstLogonCommands>
  <SynchronousCommand wcm:action="add">
    <Order>1</Order>
    <Description>Install Nutanix Guest Tools from the NUTANIX_TOOLS CD-ROM: wait for the drive</Description>
    <CommandLine>powershell.exe -NoProfile -Command "$log='C:\Windows\Temp\NGT-FirstLogon.log'; function say($t){ Write-Host $t; Add-Content $log $t }; say '1. Waiting for the NUTANIX_TOOLS drive (up to 1 minute)...'; for($i=0;$i -lt 6;$i++){ $v=Get-Volume -FileSystemLabel 'NUTANIX_TOOLS' -EA 0 | ? DriveLetter | select -First 1; if($v){ say ('1. Found the NUTANIX_TOOLS drive at '+$v.DriveLetter+':'); sleep 5; exit 0 }; sleep 10 }; say '1. NUTANIX_TOOLS drive not found; is the Acropolis Attach NGT CD-ROM action enabled?'; sleep 5; exit 1"</CommandLine>
    <RequiresUserInput>false</RequiresUserInput>
  </SynchronousCommand>
  <SynchronousCommand wcm:action="add">
    <Order>2</Order>
    <Description>Install Nutanix Guest Tools from the NUTANIX_TOOLS CD-ROM: install</Description>
    <CommandLine>powershell.exe -NoProfile -Command "$log='C:\Windows\Temp\NGT-FirstLogon.log'; function say($t){ Write-Host $t; Add-Content $log $t }; if(Get-Service -DisplayName 'Nutanix Guest*' -EA 0){ say '2. NGT is already installed; skipping.'; exit 0 }; $v=Get-Volume -FileSystemLabel 'NUTANIX_TOOLS' -EA 0 | ? DriveLetter | select -First 1; if(-not $v){ say '2. No NUTANIX_TOOLS drive; skipping.'; sleep 5; exit 1 }; say '2. Waiting for Windows Installer to be idle...'; for($i=0;$i -lt 30;$i++){ $m=$null; if([Threading.Mutex]::TryOpenExisting('Global\_MSIExecute',[ref]$m)){ $m.Dispose(); sleep 10 } else { break } }; $s=[string]$v.DriveLetter+':\setup.exe'; say ('2. Running the NGT install: '+$s+' ...'); $p=Start-Process $s '/quiet','/norestart','ACCEPTEULA=yes','/log','C:\Windows\Temp\NGT-setup.log' -Wait -PassThru; say ('2. NGT install finished with exit code '+$p.ExitCode); sleep 5"</CommandLine>
    <RequiresUserInput>false</RequiresUserInput>
  </SynchronousCommand>
  <SynchronousCommand wcm:action="add">
    <Order>3</Order>
    <Description>Power off to end the one-time AutoLogon session</Description>
    <CommandLine>powershell.exe -NoProfile -Command "$log='C:\Windows\Temp\NGT-FirstLogon.log'; function say($t){ Write-Host $t; Add-Content $log $t }; say '3. Guest customization finished; powering off. CloudBolt powers the VM back on.'; sleep 5; shutdown.exe /s /t 0 /f"</CommandLine>
    <RequiresUserInput>false</RequiresUserInput>
  </SynchronousCommand>
</FirstLogonCommands>

Linux default cloud-config

The default cloud-config sets the hostname and fully qualified domain name, the user’s password hash, and the SSH key, then installs NGT and powers the VM off. When it installs NGT:

  • It waits up to one minute for the NGT drive.
  • The NGT installer hands the installation to a background systemd unit, ngt_guest_agent_upgrade. The cloud-config waits up to 10 minutes for that unit to finish before it unmounts the drive.
  • It powers the VM off last, because power_state runs after runcmd.

To install NGT from your own user data, add runcmd and power_state entries like these:

runcmd:
  - |
    for i in $(seq 1 6); do [ -e /dev/disk/by-label/NUTANIX_TOOLS ] && break; sleep 10; done
    mkdir -p /mnt/ngt && mount -o ro /dev/disk/by-label/NUTANIX_TOOLS /mnt/ngt && python3 /mnt/ngt/installer/linux/install_ngt.py > /var/log/ngt-install.log 2>&1
    for i in $(seq 1 120); do systemctl is-active --quiet ngt_guest_agent_upgrade || break; sleep 5; done
    umount /mnt/ngt 2>/dev/null || true

power_state:
  mode: poweroff
  message: CloudBolt guest customization finished; powering off. CloudBolt powers the VM back on.
  condition: true

What to add to your own files

If you set Acropolis Sysprep File or Acropolis User Data, add the following to your file:

  1. Install NGT from the NUTANIX_TOOLS drive, as in the examples above. Without NGT, provisions on Nutanix IPAM subnets fail because CMP can’t confirm that the guest is ready.
  2. End by powering the VM off. On Windows, add an AutoLogon so that FirstLogonCommands run, and make the last FirstLogonCommand shutdown.exe /s /t 0 /f. On Linux, add power_state with mode: poweroff. Without a final power-off, every build waits the full VM Customization Timeout before it continues.

Your own user data can also use {{ linux_username }}, {{ linux_password_hash }}, and {{ ssh_public_key }}. These are the values the default cloud-config uses to set the user’s password and SSH key.

Passwords for Windows builds

On Windows, CMP checks the answer file for blank passwords after filling in its values. If it finds one, the order fails before the VM is created. The check covers the Administrator, AutoLogon, local account, and domain join passwords, in both the default answer file and your own Acropolis Sysprep File. With the default answer file, set a password on the order or on the OS build attribute. Linux user data is not checked.

OS readiness and timeouts

During provisioning, CMP waits up to the VM Customization Timeout (vm_customization_timeout) for guest customization to power the VM off. It then powers the VM back on and waits for the guest to report ready.

Wait Default Set by
For guest customization to power the VM off 600 seconds VM Customization Timeout
For the guest to report ready after CMP powers it back on 300 seconds Fixed
Extra readiness check when the VM never powers off 60 seconds Fixed

Increase VM Customization Timeout for builds that take longer than 10 minutes to customize. See Waiting for a server to be ready. Setting prov_timeout does not change these waits. It only limits how long CMP waits for Nutanix to create the VM or deploy the VM Template (300 seconds by default).

What counts as ready

CMP checks the VM in Prism every 15 seconds. The guest is ready on the first of these signals:

  • A learned IP address on the primary network interface. If CMP assigned that interface an address, the learned address must match it. CMP ignores link-local 169.254.x.x addresses. Nutanix reports learned addresses only on subnets without Nutanix IPAM.
  • NGT installed, reachable, and reporting the server’s hostname as its DNS name.

On Linux, NGT reports a DNS name only when the guest has a fully qualified domain name. If you use your own user data, make sure it sets one. The default cloud-config uses the hostname and the server’s DNS domain, or localdomain when the server has none.

When a signal does not arrive

Situation Result
The VM does not power off within the timeout, but reports ready during the extra 60-second check The provision continues, with a warning that customization did not power the VM off.
The VM does not power off and never reports ready The provision fails.
The VM is on a Nutanix IPAM subnet and NGT is not enabled on it, because Acropolis Attach NGT CD-ROM is disabled CMP can’t verify readiness. If customization powers the VM off, the provision continues with the assigned IP address. Otherwise, CMP waits the full timeout, then continues with the assigned IP address and a warning.
No readiness signal within 300 seconds after CMP powers the VM back on The provision fails. The error says which check was missing.
Prism can’t be read during a check CMP logs the error and tries again at the next check. The timeout still applies.

Day-2 actions, such as the built-in reboot action, check readiness once. They do not wait for the provisioning timeouts and do not fail when the guest is not ready. Servers with skip_power_on set skip the readiness check.

Networking

CMP creates the VM with a network interface for each network on the order (sc_nic_0, sc_nic_1, and so on). Guest customization sees every interface at first boot.

  • Nutanix IPAM subnets. When CMP assigns an interface an address on a subnet with Nutanix IPAM enabled, CMP sets that address on the interface, and Nutanix DHCP gives it to the guest. If CMP can’t look up the subnet in Prism, the provision fails.
  • Other interfaces. CMP does not send Nutanix an address for interfaces that have no assigned address or are on subnets without Nutanix IPAM.
  • Advanced Networking. Nutanix can reject a VM with several interfaces when not all of its subnets have Advanced Networking enabled. CMP logs a warning when it creates such a VM from an image.
  • Skip Network Configuration (c2_skip_network_conf) stops CMP from sending any IP addresses to Nutanix.
  • Link-local addresses. CMP ignores 169.254.x.x addresses when it waits for the server’s IP address, and records one only if the interface has no other address.
  • VM Templates. The order’s networks replace the template’s own network interfaces. If the order has no networks, the VM keeps the template’s interfaces.

For how CMP and Nutanix assign addresses on each kind of network, see IP address management for Nutanix.

CD-ROM handling

The guest customization ISO holds the Windows answer file or the Linux user data. The answer file includes the Administrator password. CMP makes sure no ISO is left mounted on a provisioned VM.

Before first boot

The Acropolis Attach NGT CD-ROM action places the two ISOs:

VM layout Guest customization ISO NGT ISO
IDE:0 and IDE:1 CD-ROMs exist Copied to IDE:0 IDE:1
A single bus, for example a Q35 VM for UEFI or Secure Boot, with no IDE The first CD-ROM on that bus, where Nutanix put it The next CD-ROM on that bus, added if needed
Anything else, for example CD-ROMs on several buses, or several empty CD-ROMs Every CD-ROM is deleted and re-created so Nutanix attaches and labels both ISOs A CD-ROM after the customization CD-ROM on the same bus

The action fails if a CD-ROM holds other media or if it can’t confirm that both ISOs are in place.

After customization powers the VM off

  • When exactly one bus has exactly two CD-ROMs, CMP deletes the second one, which is the NGT CD-ROM. CMP identifies it by its position rather than its contents, because Nutanix may already have ejected the NGT ISO.
  • With any other layout, no CD-ROM is deleted. Examples are a single CD-ROM, a template with more CD-ROMs on that bus, or two buses with two CD-ROMs each.
  • Every other CD-ROM with media is emptied and kept, including template CD-ROMs and the one Nutanix used for customization.
  • If a CD-ROM can’t be removed or emptied, CMP logs a warning and powers the VM on. The eject action removes any leftover media later.

With the attach action disabled and a template that has one CD-ROM, Nutanix adds the customization CD-ROM after it on the same bus. Because that bus then has two CD-ROMs, CMP keeps the template’s CD-ROM and deletes the customization CD-ROM.

After provisioning

The Acropolis Eject CD-ROM Media action ejects any ISO still mounted and fails if any remains. It leaves the empty CD-ROMs on the VM, where NGT can use them later.

Ejecting the ISO does not remove the copy of the customization data saved inside the guest. On Linux, cloud-init saves it to /var/lib/cloud/instance/user-data.txt, which only root can read. On Windows, the answer file is cached under C:\Windows\Panther.

VM Templates

On v4, the handler’s Images tab imports Nutanix VM Templates alongside disk images. See Import images and VM Templates.

When CMP deploys a VM Template:

  • It uses the template’s active version. CMP looks up the active version at deploy time. A newer version activated in Prism is used without importing the template again.
  • The order’s CPU and memory replace those saved in the template. The VM gets one socket, with one core for each CPU and one thread per core.
  • The order’s networks replace the template’s network interfaces. See Networking.
  • Guest customization is applied at deploy time, using CMP’s default file or your own. See Guest customization and NGT.
  • The Storage Container parameter applies. Nutanix deploys into the template’s storage container, then CMP moves the disks to the requested one. Moving the disks slows the deploy. The deploy fails if the requested container is on external storage.

Before you create a template from a VM:

  • Do not install NGT. The NGT certificate is unique to each VM. A template with NGT installed breaks NGT on every VM created from it. Instead, the attach action mounts an NGT ISO on each new VM.
  • Remove every network interface. Nutanix restores the template’s saved interfaces before CMP replaces them. If one is on a subnet with no free addresses in its IPAM pool, the deploy fails with error VMM-30601.
  • Remove empty CD-ROMs. An extra CD-ROM moves the customization ISO off the first CD-ROM.

Leave VMs powered off

To leave new VMs powered off, create a Boolean parameter named skip_power_on and set it on the order, environment, or group. When it is set on a v4 server:

  • CMP never powers the VM on and does not wait for an IP address or OS readiness.
  • Guest customization runs when someone first powers the VM on. The default answer file and cloud-config then power the VM off when they finish. Power it on again yourself.
  • The eject action leaves the media mounted, because the guest needs it at first boot. Eject it after the guest has finished customizing.

Failed provisions

When a provision fails after Nutanix created the VM, for example while migrating its disks to another storage container, CMP keeps the VM’s UUID on the server. The automatic cleanup then deletes the VM in Nutanix.

  • No UUID, nothing deleted. When a provision fails before Nutanix created a VM, the cleanup warns that no Nutanix VM UUID was recorded and nothing was deleted in Nutanix. If Nutanix has a VM left over from that provision, delete it there.
  • Missing UUIDs are reported. If a server has no Nutanix VM UUID, Refresh Info says so.
  • Template deploy failures show the cause. Nutanix reports a generic error for a failed deploy task. CMP also logs the detailed errors from its subtasks.
  • Media is still ejected. The eject action runs for failed provisions whose VM exists.

Manage CD-ROMs from your own actions

Your own orchestration actions can manage CD-ROMs on v4 through four Nutanix API wrapper methods instead of calling the Nutanix API directly. The built-in actions and the readiness check rely on the same methods.

Method What it does Returns
add_cdrom_v4(vm_uuid, bus_type=None, attach_ngt_iso=False, ngt_capabilities=None) Adds an empty CD-ROM. bus_type is "SATA" or "IDE". Without it, the method uses the bus of the guest customization CD-ROM, otherwise the first CD-ROM’s bus, otherwise SATA. With attach_ngt_iso=True, it also loads the NGT ISO. The new CD-ROM’s extId
insert_iso_v4(vm_uuid, iso_type="NGT", capabilities=None, cdrom_uuid=None, image_uuid=None) Loads an ISO into an empty CD-ROM. With "NGT", Nutanix generates the VM’s own NGT ISO, with the SELF_SERVICE_RESTORE and VSS_SNAPSHOT capabilities by default, and picks the empty CD-ROM. With "GUEST_CUSTOMIZATION", it loads the image image_uuid into the CD-ROM cdrom_uuid. The Nutanix task UUID
eject_cdrom_v4(vm_uuid, cdrom_ext_id, etag=None) Removes the ISO and keeps the CD-ROM. Works whether the VM is running or powered off. The Nutanix task UUID
delete_cdrom_v4(vm_uuid, cdrom_ext_id) Removes the CD-ROM. Nutanix removes CD-ROMs only while the VM is powered off. The Nutanix task UUID

All four methods wait for the Nutanix task to finish and raise AcropolisError if it fails. Use them only on handlers that use API version v4.

Calls that change a VM need its current ETag. eject_cdrom_v4 reads a fresh one unless you pass etag. To find a CD-ROM’s extId and what it holds, read the cdRoms list returned by wrapper.get_vm_v4(vm_uuid). Each entry has extId, diskAddress (bus and index), backingInfo (empty when no ISO is loaded), and isoType.

This Pre-Network Configuration action adds a CD-ROM with the NGT ISO while the VM is still off:

def run(job, logger=None, **kwargs):
    server = job.server_set.last()
    wrapper = server.resource_handler.cast().get_api_wrapper()
    vm_uuid = server.resource_handler_svr_id

    cdrom_id = wrapper.add_cdrom_v4(vm_uuid, attach_ngt_iso=True)
    return "SUCCESS", f"Added CD-ROM {cdrom_id} with the NGT ISO", ""

This action ejects every loaded ISO:

def run(job, logger=None, **kwargs):
    server = job.server_set.last()
    wrapper = server.resource_handler.cast().get_api_wrapper()
    vm_uuid = server.resource_handler_svr_id

    for cdrom in wrapper.get_vm_v4(vm_uuid).get("cdRoms") or []:
        if cdrom.get("backingInfo"):
            wrapper.eject_cdrom_v4(vm_uuid, cdrom.get("extId"))
    return "SUCCESS", "Ejected all ISOs", ""

Before you upgrade

If your installation has Nutanix resource handlers set to API version v4, check these changes.

Change Who is affected What to do
The built-in orchestration actions are created and enabled Every installation Nothing, unless you want them disabled. Keep Acropolis Eject CD-ROM Media last in Post-Provision.
Provisioning waits for guest customization to power the VM off Your own Acropolis Sysprep File or Acropolis User Data without a final power-off Add the power-off. See What to add to your own files. Otherwise, each build waits the full VM Customization Timeout (600 seconds by default), then logs a warning.
Your own files do not install NGT Your own files, on Nutanix IPAM subnets Add the NGT installation. Otherwise, provisions on those subnets fail when the readiness wait ends.
Windows builds without an Acropolis Sysprep File use the CMP default answer file Windows image and template deploys Set a password on the order or the OS build attribute. Without one, the order fails before the VM is created.
Blank passwords in your own answer file fail the order An Acropolis Sysprep File that leaves a password blank Fix the file or set a password. The order fails before the VM is created.
Linux builds without Acropolis User Data use the CMP default cloud-config Linux image and template deploys Nothing. To change it, copy it to /var/opt/cloudbolt/proserv/acropolis_linux_cloud_config.yaml.tmpl and edit the copy.
The Hyper-V default answer file moved from resourcehandlers/hyperv/windows_unattend.xml to resourcehandlers/windows_unattend.xml, and Nutanix now shares it Hyper-V installations that edited the shipped file in place Move your edits to the Hyper-V override file.
Template deploys use the Storage Container parameter Template deploys with a storage container set Expect slower deploys. Deploys fail when the container is on external storage.
VM Customization Timeout applies to Nutanix All v4 provisioning Increase it for builds that take longer than 10 minutes to customize.
New Boot Disk Adapter Type, Nutanix Image - Boot Disk Adapter Type, and Nutanix Image - Boot Type parameters Installations with a Nutanix resource handler Nothing. Upgrading adds them. See Per-image boot settings.