Before we create and install VMs, we need to change one setting in the cluster (this should have been done before, but we're doing it to be safe).
Do this on the MASTER node:
# gnt-cluster modify -H kvm:kernel_path="",initrd_path=""We are going to install our VMs using debootstrap - we don't feel like spending time installing an OS using an ISO image: it takes too long, and that's not what we would be doing in production if we had many customers or users asking for VMs all the time.
First, we need to adjust a few things.
We need to upgrade the debootstrap instance builder for Ganeti - there is a newer one. It fixes some bugs that could give some trouble later.
First, remove the ganeti-instance-debootstrap that was installed automatically when we installed ganeti:
# apt-get remove ganeti-instance-debootstrapYou will need to do this on both nodes (a secondary node can become a primary if the primary fails, and must be running the same software as the master).
# apt-get install make
# wget https://ganeti.googlecode.com/files/ganeti-instance-debootstrap-0.14.tar.gz
# tar xvfz ganeti-instance-debootstrap-0.14.tar.gz
# cd ganeti-instance-debootstrap-0.14
# ./configure -prefix=/usr --localstatedir=/var --sysconfdir=/etc
[ some output ]
# make; make install
[ some output ]Test that the new "instance OS" type is recognized:
gnt-os listThis should show: debootstrap+default
We would like all our instances to have the same root password. Ganeti provides us with a mechanism for this, which comprises two files: a small shell script, and a data file containing the desired password.
You need to be in the directory ganeti-instance-debootstrap-0.14 you extracted above.
# mkdir -p /etc/ganeti/instance-debootstrap/hooks/confdata
# cp examples/hooks/defaultpasswords /etc/ganeti/instance-debootstrap/hooks/
# cp examples/hooks/confdata/defaultpasswords \
/etc/ganeti/instance-debootstrap/hooks/confdata/Now, edit /etc/ganeti/instance-debootstrap/hooks/confdata/defaultpasswords and change the default password from s3cr3t to the password we use in class. This file should be just one line:
root:XXXXXXwhere XXXXXX is the password.
We're almost ready to build our VM. Before, we need to set a few parameters so it builds the version and architecture we want.
Modify the file /etc/default/ganeti-instance-debootstrap, which contains the default debootstrap settings for Ganeti.
If you take a quick look, you can see various options that we covered before in the scripted installs:
ARCH, SUITE and EXTRA_PKGS in particular, control which version of the OS and architecture, as well as which package, get installed.
We need to override (change) two of these parameters: ARCH and EXTRA_PKGS, but we're not going to modify this file.
Instead, we'll edit the file /etc/ganeti/instance-debootstrap/variants/default.conf, which allows us to make the change without touching the installed version. Like this, when Ganeti is upgraded, your settings file isn't overwritten.
Therefore:
# editor /etc/ganeti/instance-debootstrap/variants/default.confAnd add the following lines:
ARCH="amd64"
EXTRA_PKGS="acpi-support-base,console-tools,udev,linux-image-amd64,grub2"Summary:
# gnt-instance add -t plain \
-o debootstrap+default \
-B memory=512M --disk 0:size=4G \
-n vmX.ws.nsrc.org \
--no-ip-check --no-name-check \
-H kvm:vnc_bind_address=0.0.0.0,kernel_path="" \
debianNote: if you are only going to have one disk, instead of writing --disk 0:size=4G, you could simply write -s 4G.
The build is going to take a while the first time - it should fetch most packages from the local cache, which should speed things up.
It needs to fetch packages just like for debootstrap in the previous labs. Be patient...
Sat Jan 18 09:52:14 2014 * disk 0, size 4.0G
Sat Jan 18 09:52:14 2014 * creating instance disks...
Sat Jan 18 09:52:15 2014 adding instance debian to cluster config
Sat Jan 18 09:52:15 2014 - INFO: Waiting for instance debian to sync disks
Sat Jan 18 09:52:15 2014 - INFO: Instance debian's disks are in sync
Sat Jan 18 09:52:15 2014 * running the instance OS create scripts...
Sat Jan 18 10:02:53 2014 * starting instance...Ok, the build is done! The good thing is, in the future, Ganeti has cached the image of your host (by default, for two weeks), so if you install more VMs with the same command, it will only take a few seconds to create and start the machine 1.
Unfortunately, there is currently a problem (bug) currently with this way of installing VMs on Ganeti.
Your VM will not come up, and KVM will hang with 99% cpu:
# top
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5981 root 20 0 765m 11m 3144 S 98.8 1.1 6:05.41 kvmLet's fix it.
Bring down the host (shutdown):
# gnt-instance shutdown --timeout=0 debian
Waiting for job 211 for debian ...We're going to boot the instance using the kernel from the Host, just temporarily, so we can fix the bug:
# gnt-instance start \
-H kernel_path=/vmlinuz,initrd_path=/initrd.img,kernel_args="ro single" \
debianWait a few seconds, then start a serial console:
# gnt-instance console debianPress ENTER
You should see:
Login incorrect.
Give root password for maintenance
(or type Control-D to continue):Here, enter the password we set for the root (class password). If all goes well, you are in:
root@debian:~#Now we can fix the GRUB bootloader on the VM:
Note: you are typing this in the console, NOT ON YOUR HOST!
Update the GRUB bootloader:
# update-grub
Generating grub.cfg ...
cat: /boot/grub/video.lst: No such file or directory
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
doneReinstall the boot loader in the boot sector of our VM disk:
# grub-install /dev/vda
Installation finished. No error reported.Now you can shutdown the machine (still inside the guest!)
# halt -p
INIT: Switching to runlevel: 0
INIT: Sending processes the TERM signal
INIT: Sending processes the KILL signal
[ 247.024934] EXT4-fs (vda1): re-mounted. Opts: (null)
[ 249.108473] ACPI: Preparing to enter system sleep state S5
[ 249.150817] Power down.
Connection to vmX.ws.nsrc.org closed.You should be back at the shell prompt of your host. On the host, let's try and restart the instance (VM) normally.
# gnt-instance start debianWait a few seconds, and if all goes well, you should now be able to get a console on your instance, either using gnt-instance console debian or using VNC 2
Our machine still doesn't have any network configuration - we will deal with it now.
One thing remains to be done - our machines still don't have any network connectivity.
Log in using VNC to the console of your VM (if you're in doubt which Node it's running on, use
# gnt-instance list -o name,pnode,snodes,oper_ram,nic.bridge/0,network_port,nic.link/0,sda_size,statusTo find out which Node + Port it's listening on.
Once logged in to the guest:
# ifconfig eth0Notice there isn't any IP.
Remember, that this VM is attached to br-svc from the point of view of the HOST. That network's address and subnet is 10.10.255.0/24.
You will give an IP address to your VM, from the following table:
| Cluster/Group | IP address |
|---|---|
| Group 1 | 10.10.255.1 |
| Group 2 | 10.10.255.17 |
| Group 3 | 10.10.255.33 |
| Group 4 | 10.10.255.49 |
| Group 5 | 10.10.255.65 |
| Group 6 | 10.10.255.81 |
| Group 7 | 10.10.255.97 |
We need to edit /etc/network/interfaces and give your VM an IP address from the table above. Do this now, and it should end up looking like this:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 10.10.255.XX
netmask 255.255.255.0
gateway 10.10.255.254Replace XX with the IP from the table above.
Finally, modify /etc/resolv.conf and set the nameserver as follows:
nameserver 10.10.0.241You can now bring up the interface:
# ifup eth0Test that connectivity works, by pinging the class webserver:
# ping www.ws.nsrc.orgDoes it work ?
Let's make sure we are using the in-class package cache:
Edit the file /etc/apt/apt.conf and add this line:
Acquire::http::Proxy "http://apt.ws.nsrc.org:3142/";Now we can install OpenSSH 3
# apt-get install openssh-serverYou should now be able to log in to your VM as root, on the IP address you gave it earlier (10.10.255.X).
Ok, if our machine is going to be running some important services, we need to make it "fully redundant".
We can tell Ganeti to convert our machine to go from 'plain' disk (a single copy of the disk in the LVM store of the primary node) to using 'drbd', where data is written at the same time on the primary and secondary nodes.
This is done with a single command, but first we need to shut down our machine temporarily.
# gnt-instance shutdown debian
Waiting for job 277 for debian ...Wait 10-15 seconds, and check that the instance is down:
# gnt-instance list debianIf it says ADMIN_down for the Status, you can now run the command to convert from plain to drbd:
# gnt-instance modify -t drbd -n vmY.ws.nsrc.org debianImmediately the process will begin, and you will see some output similar to this:
Sat Jan 18 11:33:23 2014 Converting template to drbd
Sat Jan 18 11:33:23 2014 Creating additional volumes...
Sat Jan 18 11:33:23 2014 Renaming original volumes...
Sat Jan 18 11:33:24 2014 Initializing DRBD devices...
Sat Jan 18 11:33:26 2014 - INFO: Waiting for instance debian to sync disks
Sat Jan 18 11:33:27 2014 - INFO: - device disk/0: 1.80% done, 1m 1s remaining (estimated)
Sat Jan 18 11:34:27 2014 - INFO: - device disk/0: 90.10% done, 6s remaining (estimated)
Sat Jan 18 11:34:34 2014 - INFO: - device disk/0: 99.30% done, 0s remaining (estimated)
Sat Jan 18 11:34:34 2014 - INFO: - device disk/0: 99.70% done, 0s remaining (estimated)
Sat Jan 18 11:34:34 2014 - INFO: - device disk/0: 100.00% done, 0s remaining (estimated)
Sat Jan 18 11:34:34 2014 - INFO: Instance debian's disks are in sync
Modified instance debian
- disk_template -> drbd
Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.Notice that we provide, as a parameter to -n, the name of the node that we will be replicating to. This node should already be part of the cluster (we did this at the end of the Ganeti install lab).
Time to restart the instance!
# gnt-instance start debianOk, how do we know that the replication is really taking place ?
The best way it to use the ifstat package, to see how much bandwidth is being used on our br-rep network, which DRBD is using to copy data from the primary to the secondary node.
# apt-get install ifstatOnce the ifstat tool is installed, run it to see the network traffic on your network interfaces:
# ifstat -i br-rep
br-rep
KB/s in KB/s out
0.00 0.00
2.23 78.96
0.00 0.00
0.00 0.00Now, log in to your guest, and install a package there, for example, apache:
Log in via SSH to your guest (remember the IP you gave it from the table above- it's in the 10.10.255.0/24 network), and install apache2:
# ssh root@10.10.255.NN # here NN is the IP of your VM on the service net
...
# apt-get install apache2Keep an eye on your host window, and see the bandwitdh usage on br-rep
0.00 0.00
86.08 4886.99
394.32 14893.37
359.16 16467.22
23.04 881.35You should see the bandwidth utilization peak for a few seconds while the packages are being retrieved and installed.
root@vmX:~# gnt-instance migrate debian
Instance ubuntu-plain will be migrated. Note that migration might
impact the instance if anything goes wrong (e.g. due to bugs in the
hypervisor). Continue?
y/[n]/?: y
Sat Jan 18 04:37:05 2014 Migrating instance ubuntu-plain
Sat Jan 18 04:37:05 2014 * checking disk consistency between source and target
Sat Jan 18 04:37:06 2014 * switching node vmX.ws.nsrc.org to secondary mode
Sat Jan 18 04:37:06 2014 * changing into standalone mode
Sat Jan 18 04:37:06 2014 * changing disks into dual-master mode
Sat Jan 18 04:37:07 2014 * wait until resync is done
Sat Jan 18 04:37:07 2014 * preparing vmX.ws.nsrc.org to accept the instance
Sat Jan 18 04:37:08 2014 * migrating instance to vmX.ws.nsrc.org
Sat Jan 18 04:37:10 2014 * starting memory transfer
Sat Jan 18 04:37:15 2014 * memory transfer complete
Sat Jan 18 04:37:15 2014 * switching node vmY.ws.nsrc.org to secondary mode
Sat Jan 18 04:37:16 2014 * wait until resync is done
Connection to vmY.ws.nsrc.org closed.
Sat Jan 18 04:37:16 2014 * changing into standalone mode
Sat Jan 18 04:37:16 2014 * changing disks into single-master mode
Sat Jan 18 04:37:17 2014 * wait until resync is done
Sat Jan 18 04:37:17 2014 * done
root@vmX:~#Verify that the instance is now running on the secondary node:
(remember, all gnt-* commands must be run from the MASTER node)
# gnt-instance list -o name,pnode,snodes,oper_ram,nic.bridge/0,network_port,nic.link/0,sda_size,status
Instance Primary_node ConfigMaxMem Memory NicBridge/0 Network_port NicLink/0 Disk/0 Status
debian vmY.ws.nsrc.org 512M 512M br-lan 11000 br-lan 4.0G runningNote the VNC port at the same time, and reconnect to the VNC console on the primary node, port as listed in the output above.
We need to install a support package called ganeti-instance-image, which will allow us to install an operating system, just like we did with VirtualBox and KVM, that is:
The package can be installed as follows 4:
wget https://code.osuosl.org/attachments/download/2169/ganeti-instance-image_0.5.1-1_all.deb
dpkg -i ganeti-instance-image_0.5.1-1_all.debIf using DRBD, the ISO images used for CD installs must be present on all nodes in the cluster, in the same path.
Copy from your USB key the antix Linux distribution ISO, found under the iso directory, to /home/nsrc, on the master node.
It should be called antiX-13.2_386-full.iso.
We can use the gnt-cluster copyfile we learned about earlier, to distribute the ISO image to all nodes:
$ sudo gnt-cluster copyfile /home/nsrc/antiX-13.2_386-full.isoThat should copy the file over to all the other nodes.
From here, we have to decide whether we want to create an LVM (plain) instance, or a redundant DRBD instance:
gnt-instance add -t plain -o image+default
-s 3GB -B memory=128M
-n vmX.ws.nsrc.org
--no-install --no-start --no-ip-check --no-name-check
-H kvm:vnc_bind_address=0.0.0.0,boot_order=cdrom,cdrom_image_path=/home/nsrc/antiX-13.2_386-full.iso
antix-plain
... you will see some messages about creating the instance being created.
gnt-instance add -t drbd -o image+default
-s 3GB -B memory=128M
-n vmX.ws.nsrc.org:vmY.ws.nsrc.org
--no-install --no-start --no-ip-check --no-name-check
-H kvm:vnc_bind_address=0.0.0.0,boot_order=cdrom,cdrom_image_path=/home/nsrc/antiX-13.2_386-full.iso
antix-drbd
Tue Jan 14 05:40:34 2014 * disk 0, size 1.0G
Tue Jan 14 05:40:34 2014 * creating instance disks...
Tue Jan 14 05:40:34 2014 adding instance antix to cluster config
Tue Jan 14 05:40:34 2014 - INFO: Waiting for instance antix to sync disks
Tue Jan 14 05:40:35 2014 - INFO: Instance antix's disks are in syncRegardless of whether you created a plain or drbd (or both!), check using gnt-instance list to see that the instance was created properly:
$ sudo gnt-instance list
Instance Hypervisor OS Primary_node Status Memory
antix-plain kvm image+default vm1 ADMIN_down -Start the VM
$ sudo gnt-instance start antix-plain # if it's called antix-drbd, use that
Waiting for job 125 for antix ...Check that your VM is running:
# gnt-instance list -o name,pnode,snodes,oper_ram,nic.bridge/0,network_port,nic.link/0,sda_size,statusLook for the port number under Network_port, then open a VNC viewer and point it at the port listed, on the Host mentioned as pnode. Remember that we have set a console password.
Install the OS following the instructions.
When done, and the installer says you can, shut down the instance:
# gnt-instance shutdown --timeout=0 antix-plainNote that --timeout=0 is a "rough" way of shutting down. Only do this if you have problems (the VM doesn't answer) or if you are certain the VM is finished running (sometimes the VM doesn't properly shut down the virtual hardware using ACPI).
If you want to shut down the instance properly, you can always issue:
# gnt-instance shutdown antix-plainThis will send ACPI shutdown request (similar to pressing the power button on a modern PC).
Now modify the instance to boot on the disk. We leave the CD image in the virtual cdrom drive:
# gnt-instance modify -H boot_order=disk,cdrom_image_path="",kernel_path="" \
antix-plainYou can now restart the instance with gnt-instace start antix-plain (or whatever you named it)
You don't have to run this, but if you want to you can.
Moving the network interface 0 to another network:
# gnt-instance modify --net 0:modify,link=br-lan INSTANCE_NAMETry to do this to move the network interface of one of the instances you created earlier, onto the br-lan.
You'll need to reboot the instance after you made the change:
# gnt-instance reboot INSTANCE_NAMEYou'll have to get into the VM's VNC console (since it won't be reachable over the network), and then you can run dhclient eth0 to see if you can get an IP on the br-lan network.
If you are curious: ls -l /var/cache/ganeti-instance-debootstrap/↩
Reminder: gnt-instance list -o name,pnode,network_port then connect with VNC to the host & port listed ~↩
Note: would it not have been possible to add openssh-server to the EXTRA_PKGS section of the Ganeti debootstrap earlier ? Then we wouldn't need to install openssh manually on every instance we create.↩
You could, as an alternative, fetch and build the newest version of ganeti-instance-image, from here (building instructions are included) http://ftp.osuosl.org/pub/osl/ganeti-instance-image/↩