Objectives

In this exercise, you will start up a VM with a shared storage disk image stored on a file storage server, and then migrate it to another group's host, while it is still running.

VM disk images have already been prepared, one for each of you, and are on a central class block storage server.

Assuming you are working in pairs on the same host, then

• group 1 (host1) has disk image 1 and 2
• group 2 (host2) has disk image 3 and 4
• ...
• group N (hostN) has disk image 2N-1 and 2N

Please do try to use the correct image and not connect to someone else's image; this will result in corruption if two VMs are using the same image at once. However in case of an accident it is very quick for the instructors to copy a clean disk image for you.

Create VM

Both people in a pair have their own disk image, so they can work on this part independently.

ssh into your group's host server - you should be logged in as the "nsrc" user, not as root.

The disk image already exists, but you will still have to create a VM to attach the disk to.

This time, you're going to do it using an XML file. Check you are in the nsrc user's home directory, which is /home/nsrc

$ pwd
/home/nsrc

On Debian hosts: also check that you have the libvirt configuration set up correctly by running this command:

$ cat .config/libvirt/libvirt.conf

It should show:

uri_default = "qemu:///system"

If it does not, then create this file with this line as the contents.

Now copy and paste the following XML into a file called "serverX.xml", where X is your disk image number:

<domain type='kvm'>
  <name>serverX</name>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source protocol='nbd' name='diskXX'>
        <host name='s1.ws.nsrc.org'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:FF:00:XX'/>
      <source bridge='br-lan'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Change three things:

• In <name>serverX</name>, change X to your disk image number
• In name='diskXX', change XX to your disk image number (1 and up)
• In <mac address='52:54:00:FF:00:XX'/>, change XX to your disk image number as two decimal digits. (So disk image 40 becomes 52:54:00:FF:00:40)
• Save the file

Now define your virtual machine using the file you have created:

$ virsh define serverX.xml

If this is successful, you should see that it is present and be able to start it.

$ virsh list --all
$ virsh start --console serverX    # replace X with your disk image number

After 10-15 seconds you should get a login prompt at the console The username and password will be given to you in class.

In this console session, type ifconfig to find the name of the interface. It may be eth0 or enp2s0. Once you find out (is it eth0 ? is it enp2s0 ?), you will need to check, and possibly update, the /etc/netplan/50-cloud-init.yaml so that the interface is correct (e.g.: change eth0 to enp2s0 if necessary).

If you DO update the netplan file:

• run netplan apply
• log back in using the console if necessary
• verify that your VM got an IP using DHCP:

$ ifconfig eth0

or

$ ifconfig enp2s0

Note: you can always use ifconfig -a to see ALL interfaces on your VM, even those that aren't configured.

If it did NOT get an IP address, please verify /etc/netplan/50-cloud-init.yaml, and change the interface name (for instance, from eth0 to enp2s0) if necessary, then reboot the VM.

If you DO see an IP address, write it down, then disconnect from the serial console (press ctrl and right-hand square bracket: CTRL - ] ).

Now you are back on your host server. Use these commands to show that your KVM process is running:

$ virsh list
$ ps auxwww | egrep '(kvm|qemu-system)'

Look for -name serverX in the kvm command line, so you know it's yours.

ssh into VM

Now using another ssh connection, e.g. another putty window, ssh directly to this running VM, using the IP address you just noted. Obviously this is using the same username/password that you were given before, as it's just a different way of connecting to the same VM.

All the VMs start off as being identical, so make some changes to yours so that it is distinctive. For example:

• Type sudo -s to get a root shell
• Edit /etc/hostname to change the hostname to anything of your choice
• Edit /etc/hosts with the new hostname
• Type service hostname start to refresh the hostname

If you then disconnect and reconnect your ssh client, you should see your new hostname at the prompt.

Now you want to get the VM to do some visible work while you migrate it. Type the following command line:

$ while true; do date; sleep 0.5; done

This will make it display the current date and time twice per second, so you should see this scrolling up the screen. Leave it running.

Suspend and resume

Just to prove that this is a running VM, now go back to your ssh session to the host server, while keeping the VM session window open so you can see the scrolling text.

On the host server, type the following commands:

$ virsh suspend serverX

$ virsh resume serverX

The 'suspend' should stop the messages from the VM scrolling up the screen, and 'resume' should restart them. Leave the VM running and the messages scrolling up.

Migrate

Now to migrate the virtual machine to another host. You will migrate to the next server numerically: those using host1 will migrate their servers to host2; those on host2 will migrate their servers to host3; and so on. The group on the last host will migrate their servers to host1.

This can be done either using the virsh command line, or the virt-manager GUI. Try the virsh command line first.

In the virsh command line

In the host server ssh window:

$ virsh migrate --live --verbose serverX qemu+ssh://hostN.ws.nsrc.org/system

... where X is your disk image number and N is your group number plus 1.

(Contact the instructor if you get this error; "error: internal error: Attempt to migrate guest to the same host")

You'll be prompted for the ssh password to the next machine.

That's it. Your VM should still be running, but will be on a different host!

To check, use:

$ virsh list
$ ps auxwww | egrep '(kvm|qemu-system)'

on your host, and on the next group's host. (Ask them to do it for you, or carefully ssh into their host and do it)

From there, you can get them to migrate it back again:

$ virsh migrate --live --verbose serverX qemu+ssh://hostM.ws.nsrc.org/system

where M is your group number. If you try without --live you may see a short pause in the output of the running VM.

Using virt-manager GUI

Note: You will have to explicitly install ssh-askpass for the following lab to function. So start by doing this:

# apt-get install ssh-askpass

Now, start the virt-manager GUI. You should see a connection to your own hypervisor: QEMU/KVM.

Now you are going to add a connection to another hypervisor.

• File > Add Connection (You might get a warning about a dialog failing to open - you can ignore this and click on Close)
• Hypervisor: QEMU/KVM
• [X] Connect to remote host
  • Method: SSH
  • Username: nsrc
  • Hostname: hostN.ws.nsrc.org (where N is the next group)
• Connect

The first time you connect, you may be questioned about the authenticity of the other host's SSH key. Type "yes" in full, and press OK.

You will then be prompted for the ssh password to the other server. Type the same password as you would normally use to login to that machine. (In a real environment you would streamline this using ssh keys).

You should now see the other hypervisor listed in the window, together with any VMs running there. To perform the migration:

• Double-click on the VM you want to migrate to get a console.
• Select Virtual Machine > Migrate...
• The drop down "New host" chooses the host you want to migrate to - if you see QEMU/KVM without a hostname after it, that's your own server
• Leave the Address, Port, and Advanced Options as-is
• Click Migrate

The console will disconnect because the machine is no longer running where it was before.

Close the console window, look at the list of VMs and you should see that your VM is now running on the other host, so you can double-click to see it there. If this is a remote host you will need to enter the ssh password again.

You should be able to migrate the VM back again in exactly the same way, from the same GUI instance. (In this way, a single machine running virt-manager could control a whole network of VM servers)

Optional exercises

Find two other groups who are happy to host your VM. Migrate your VM from yours to the first, then from the first to the second, then back to your host.

You can do this easily in the virt-manager GUI.

Alternatively, it's also straightforward in the virsh command line. You can issue commands to other hosts using the -c flag. For example, this is how you run virsh list on a remote hypervisor:

virsh -c qemu+ssh://<hostname>/system list

A third-party command to migrate a VM from <oldhost> to <newhost> looks like this:

virsh -c qemu+ssh://<oldhost>/system migrate --live --verbose \
      <vmname> qemu+ssh://<newhost>/system

Notes on persistence

When a virtual machine is defined in an XML file on disk, it is called "persistent". When the virtual machine is migrateed to another host, it can be done in one of two ways:

• transient, or "temporary" (running on the new host, but not permanently so: stopping the VM will make it "disappear" from the new host). Also note that the virtual machine remains defined, in a "shut down" state on the original host!

• persistent: the migration is done in a way that the virtual machine is "undefined" on the old host, and created permanently on the new host (definition is saved to disk).

But you need to be aware that, the command line tool (virsh) and the GUI (virt-manager) have different default behaviours!

Using virsh, the default is for the migration to bee transient or "temporary". This means that:

• When you migrate from host A to host B, it remains defined on host A but in a shutdown state. There is a risk that you might start it again on A while it is also running on B.

• When you then migrate from host B to host C, you will see it vanish from host B (but it remains defined on host A)

To change the behaviour of virsh, you can use the following options:

virsh migrate --undefinesource --persistent ...

While in the GUI, the migration will be persistent. This means that:

• When you migrate from host A to host B, the virtual machine is undefined on host A, and permanently defined on host B.

• When you migrate from host B to host C, the virtual machine is, once again, undefined on host B, and permanently defined on host C.

For full details see libvirt migration doc