In this lab, you’re going to add new nodes to the cluster, and then perform a live migration of a virtual machine.
There are three clusters, each with seven nodes, connected like this:
Managing node1 is a shared responsibility between all the groups using a particular cluster; but then each group is responsible for one of the other nodes. Check your group number in this table to find out which node you’ll be working on (unless told differently by your instructors).
| group | cluster/node | group | cluster/node | group | cluster/node | ||
|---|---|---|---|---|---|---|---|
| group 1 | cluster1-node2 | group 7 | cluster2-node2 | group 13 | cluster3-node2 | ||
| group 2 | cluster1-node3 | group 8 | cluster2-node3 | group 14 | cluster3-node3 | ||
| group 3 | cluster1-node4 | group 9 | cluster2-node4 | group 15 | cluster3-node4 | ||
| group 4 | cluster1-node5 | group 10 | cluster2-node5 | group 16 | cluster3-node5 | ||
| group 5 | cluster1-node6 | group 11 | cluster2-node6 | group 17 | cluster3-node6 | ||
| group 6 | cluster1-node7 | group 12 | cluster2-node7 | group 18 | cluster3-node7 |
It is VERY IMPORTANT that the groups on the same cluster work together to do this ONE STEP at a time! That is:
Following recommended practice, we are going to use a separate NIC for cluster communication (corosync), on a separate subnet, 100.64.4.0/24. This NIC has already been set up for you; in real life you’d have to configure this network interface first.
Every node must select its 100.64.4.X address when joining the cluster, otherwise they won’t be able to communicate.
For this part, ONE person is going to prepare node1 for the whole cluster. So, groups 1-6 should nominate one person and gather around their laptop; groups 7-12 choose another person, and so on.
node1 has to be initialized as a standalone cluster. There are two ways to do this: choose whichever way you prefer.
Login to clusterX-node1 GUI as you did before.
Go to Datacenter (first column), Cluster (second column)
Click “Create Cluster”, which gives you a dialog box.
Link 0, select address
100.64.4.XXX (eth1)
It should generate a few lines of output ending with “TASK OK”. Close the output window.
You should then see your single node under “Cluster Nodes” in the GUI.
If there’s a problem, ask your instructors for help.
Login to clusterX-node1 GUI, then get a command line by selecting
clusterX-node1 in the left-hand menue, then clicking
>_ Shell in the next column.
Then enter this command:
pvecm create clusterX --link0 100.64.4.1X1 --nodeid 1
^ ^
change X to ------'-------------------'
your cluster numberYou should see some results. To confirm the cluster status afterwards, enter this command:
pvecm statuswhich should end like this:
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 100.64.4.1X1 (local)If there’s a problem, ask your instructors for help.
Now the group which is managing node2 (see the table at the top) needs to add it to the cluster. One person in the group should do these steps, and the rest of the people in the cluster watch and help.
Again, you have two options for doing this. Choose either one; it doesn’t matter which one was used to set up node1.
This requires you to use both node1 and node2 GUIs. Open both on the same laptop.
First, login to clusterX-node1 GUI. Go to Datacenter (first column), Cluster (second column), then click “Join Information”.
This will give you a dialog box showing three fields: IP address (of node1), Fingerprint, and Join Information which is a long blob of encoded text.
Click “Copy Information” at the bottom left, and it will copy the Join Information to the clipboard.
Now open a window to clusterX-node2 GUI and login there.
Go to Datacenter (first column), Cluster (second column)
Click “Join Cluster”, which gives you a dialog box. Paste the text blob into the “Information” field, and more fields will open up:
Under “Cluster Network”, next to Link 0, select address 100.64.4.XXX
(interface enp6s0)
Finally, you have to enter the peer’s root password (i.e. the root password for node1, which you already know), and click “Join ‘ClusterX’”
At this point you’ll start getting some results:
Establishing API connection with host '100.64.0.101'
Login succeeded.
check cluster join API version
Request addition of this nodeAt this point the web interface may hang, or it may get a bit further:
Join request OK, finishing setup locally
stopping pve-cluster serviceand/or you may see an error like “permission denied - invalid PVE ticket (401)”. This is due to the web server restarting after joining the cluster. If so, refresh the web page and login again. If the browser shows that the site now has an invalid certificate, try quitting and restarting the browser.
When logged back in, you should be able to see the success in the “Tasks” log at the bottom of the screen:
You should also see both nodes in the GUI of both node1 and node2, when you go to “Datacenter” in column 1, “Cluster” in column 2.
Login to clusterX-node2 GUI, then get a command line by selecting
clusterX-node1 in the left-hand menue, then clicking
>_ Shell in the next column.
Then enter this command:
pvecm add clusterX-node1.ws.nsrc.org --link0 100.64.4.1XN --nodeid N
^ ^^ ^
'-------------------------------------'`----------'
change X to your change N to
cluster number your node number
(e.g. 2 for node2)The first parameter, clusterX-node1.ws.nsrc.org, is the
existing cluster node that we want to join.
You will be prompted for the root password for node1; enter it.
If you get an error saying “hostname verification failed” then you’ll need a longer version of this command:
pvecm add clusterX-node1.ws.nsrc.org --link0 100.64.4.1XN --nodeid N \
--fingerprint ZZ:ZZ:ZZ:ZZ...(all on one line), where you replace ZZ:ZZ:ZZ:ZZ... with
the fingerprint shown from the cluster “Join Information” in the GUI
from node1.
You should see some results like this:
Establishing API connection with host 'cluster0-node1.ws.nsrc.org'
Login succeeded.
check cluster join API version
Request addition of this node
Join request OK, finishing setup locally
stopping pve-cluster service
backup old database to '/var/lib/pve-cluster/backup/config-1753349590.sql.gz'
waiting for quorum...OK
(re)generate node files
generate new node certificate
merge authorized SSH keys
generated new node certificate, restart pveproxy and pvedaemon services
successfully added node 'cluster0-node2' to cluster.To confirm the cluster status afterwards, enter this command:
pvecm statuswhich should end like this:
Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 2
Quorum: 2
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 100.64.4.1X1
0x00000002 1 100.64.4.1X2 (local)Note the “Quorum”. For the cluster to work, more than half of the nodes have to be online to agree on cluster state. Since we have only two nodes, the quorum is 2; both nodes must be online.
“Quorate” means that the cluster does currently have sufficient active nodes to form a quorum.
ONE GROUP AT A TIME, repeat the above steps which were done for node2 on the remaining nodes. That is: the next group should add their node3; only when that is complete and working move to node4; and so on.
Once this is finished, you should have a 7-node cluster. (This is a good cluster size; it is recommended that clusters should have an odd number of nodes)
At this point, you can break up into your separate groups. Each group can run this exercise in parallel with the others.
You’re going to live-migrate the VM you created earlier
(groupX-web) to your group’s new node.
Choose someone in the group who hasn’t done much so far
Find your groupX-web VM in the GUI. It should be under
clusterX-node1. Start it, if it’s not already running, and then open the
console.
Login to the VM (username “ubuntu” and the password you set when creating the VM), then start the following command:
ping 8.8.8.8You should see pings scrolling up the screen, with increasing
sequence numbers (icmp_seq).
You are now going to move this VM, while it is running, to another node.
Click the “Migrate” button, which is at the top right above the console.
For the target, choose the node which your group is managing (see the table above). For example, if you’re group6, you will migrate to cluster1-node7.
Click “Migrate” and a progress window should pop up. You will see the disk contents being migrated:
...
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 6.2 MiB of 8.0 GiB (0.08%) in 0s
drive-scsi0: transferred 1.1 GiB of 8.0 GiB (13.37%) in 1s
drive-scsi0: transferred 1.3 GiB of 8.0 GiB (16.71%) in 2s
drive-scsi0: transferred 1.6 GiB of 8.0 GiB (20.20%) in 3s
drive-scsi0: transferred 1.9 GiB of 8.0 GiB (23.45%) in 4s
...This is follwed by the migration of the VM state (RAM). Hopefully it will end with TASK OK after a few seconds - with the ping still running in the console. Close the task viewer window.
But if you look in the left-hand pane, you’ll see that your VM is now running on a different node!
Live migration involves copying the entire RAM and CPU state, and because this VM has local disk, it also involves copying the entire disk image. That’s why there’s a warning that it might take a long time. Since RAM and disk can be changing, any “dirty” data may have to be copied again. There may be a short pause of a second or two while the final copy takes place.
Now that all the nodes share a replicated cluster database, it’s possible to create user accounts which can login to the GUI and manage all nodes, without having to create system users on every node.
Using these steps, everyone in the class can create their own user account, so they don’t need to use the PAM ‘root’ login any more.
In the GUI, go to Datacenter (first column), Permissions > Groups (second column)
If you find there’s already a group called “admin”, then somebody else has beaten you to this: go straight to the next section, “Create your own user account”.
Otherwise, you are the first. Click on “Create”
In the “Create: Group” box, enter name “admin”, then press Create. (If it fails with “group ‘admin’ already exists” then someone else has beaten you to it, skip to the next section).
Next, click on “Permissions” in the second column (directly above “Users”). Click Add > Group Permission.
From the drop-down menus, select:
/adminAdministratorthen click Add. This gives anyone in the “admin” group all permissions to all resources.
Now you are going to create your own user account.
Go to Permissions > Users and click “Add”. In the dialog box enter:
You should see your new user in the list of users:
Now logout, by clicking the “root@pam” drop-down menu at the top-right corner.
You should be prompted to login again. Login with your new credentials, remembering to select realm “Proxmox VE authentication server” if it’s not already selected.
A cluster set up this way means that most users don’t need to know
the root password. Furthermore, any actions taken in the GUI (such as
stopping and starting VMs) will be shown in the log with their username
instead of root@pam.
You can also assign different levels of permissions to groups or users. We will not investigate that here.
For this lab, we have installed the web server certificates for you. To do this manually, you’d upload
/etc/pve/nodes/<nodename>/pveproxy-ssl.key # private key
/etc/pve/nodes/<nodename>/pveproxy-ssl.pem # certificateand then systemctl restart pveproxy. Alternatively, the
web interface has options for uploading certificates and configuring
ACME for automated certificate issuance - we don’t have time to go into
this. We recommend you configure this after you have joined
your nodes into the cluster, since adding a node to the cluster wipes
its local copy of /etc/pve and synchronizes it to the
shared cluster database.
To see how we configured the NICs, go to clusterX-nodeY (first column), System > Network (second column). For the maximum cluster reliability it is even possible to configure two corosync NICs, on separate subnets and connected to separate switches. Again, we won’t do that here. Instead we have reserved an extra NIC for use for storage replication traffic.