Offline Instance Migration with Nova Compute

In managing or testing an OpenStack private cloud environment, it may be periodically necessary to offload nova compute workloads to perform planned server maintenance. Live migration is often preferred and is generally the best option, as it requires zero instance downtime. Motivations for choosing to do an offline migration over a live migration can certainly vary, depending on any number of variables such as shared storage configuration, hardware condition, hypervisor flavor, or perhaps the nova compute, qemu or KVM versions that are in play. Whatever the reason, here is a simple example of an offline migration.

This environment is OpenStack Icehouse 2014.1, Juju-deployed onto MAAS-managed Ubuntu Server 14.04 LTS Trusty Tahr nodes. Storage back-end is ceph RBD, but other shared storage options should do as well. It’s all open source, and it’s all free as in … freedom to learn, or freedom to buy support if or as needed. By the way, I just love that about open source. Ok back on track, here goes:

The short version:

# For offline migration, the instance should be
# in a STOPPED / shutoff state!  Use nova stop for
# non-graceful shutdown of the vm, if you must.

# Get the instance ID
nova list --all-tenants | grep my-instance

# Confirm the current state and hypervisor node
nova show ab90cc7a-7022-4157-b32a-11c49b7701cc

# Migrate the instance, nova scheduler decides where
nova migrate ab90cc7a-7022-4157-b32a-11c49b7701cc

# Watch & wait
watch nova migration-list

# Seal the deal
nova resize-confirm ab90cc7a-7022-4157-b32a-11c49b7701cc

# Confirm the goods
nova show ab90cc7a-7022-4157-b32a-11c49b7701cc

# Fire it up
nova start ab90cc7a-7022-4157-b32a-11c49b7701cc


The more descriptive version:

Prepare for Offline Migration

Find and copy the instance ID:
$ nova list --all-tenants | grep my-instance

+--------------------------------------+------------+---------+
| ID                                   | Name       | Status  |
+--------------------------------------+------------+---------+
| ab90cc7a-7022-4157-b32a-11c49b7701cc | my-instance| SHUTOFF |
+--------------------------------------+------------+---------+

You should gracefully shut down the instance from within the guest OS if at all possible to avoid data or file system corruption. If your instance isn’t responsive, the drastic and dangerous gamble of a forced power pull is another option. This would be akin to an abrupt power outage of a running server. Generally BAD, but if you must:
$ nova stop ab90cc7a-7022-4157-b32a-11c49b7701cc

Assess the instance: identify the current hypervisor host node and virtual machine state. Confirm that it is down, whether by grace or by force:
$ nova show ab90cc7a-7022-4157-b32a-11c49b7701cc

 OS-EXT-SRV-ATTR:hypervisor_hostname  | nc-node-2.beisner.local
 OS-EXT-STS:vm_state                  | stopped 
 status                               | SHUTOFF

Migrate the Offline Instance

Before issuing the migrate command, you may want to start this in a separate screen or window to monitor the migration:
$ watch nova migration-list

And… go! Nova scheduler will determine the new compute node.
$ nova migrate ab90cc7a-7022-4157-b32a-11c49b7701cc

Watch the migration-list as the status becomes ‘migrating,’ and eventually ‘finished.’ While the migration status says ‘finished,’ the instance still needs some attention.

Confirm Instance Migration

At this point, the instance is in a VERIFY_RESIZE state, as can be seen with another nova show:
nova show ab90cc7a-7022-4157-b32a-11c49b7701cc

| OS-EXT-STS:vm_state                  | resized
| status                               | VERIFY_RESIZE

Although the storage may not have been resized, we still need to confirm:
$ nova resize-confirm ab90cc7a-7022-4157-b32a-11c49b7701cc

Now the instance should be moved and awaiting a start command:
$ nova show ab90cc7a-7022-4157-b32a-11c49b7701cc

 OS-EXT-STS:vm_state                  | stopped 
 status                               | SHUTOFF

Fire it up!:
$ nova start ab90cc7a-7022-4157-b32a-11c49b7701cc

Verify that the instance has been migrated to another nova compute node, and that it is running:
$ nova show ab90cc7a-7022-4157-b32a-11c49b7701cc

 OS-EXT-SRV-ATTR:hypervisor_hostname | nc-node-1.beisner.local
 OS-EXT-STS:vm_state | active
 status | ACTIVE

As with any of my posts, tweets, and videos, this is not official documentation, and it may not apply to your systems or your specific environment. Take care, test in a non-production environment, and Read The Full Manual. 😉 Best to you!

Related Links & More Information:

Comment is closed.