Validator Lab

Routing Infrastructure and Security Operations Workshop

Introduction

This lab demonstrates how to install the NLnetLabs Routinator 3000 Validator Cache in preparation for ultimately running route origin validation on our lab network.

Lab Topology

The diagram below is a reminder of the lab topology:

Installing NLnetLabs Routinator

These instructions are borrowed from NLnetLabs excellent page describing how to install Routinator. Always consult NLnetLabs latest instructions before starting off in this set up.

Preparing to build Routinator

We will be installing Routinator on the SRV appliance provided for your group. This is a virtual machine running on the training VTP and is called srvX.groupX.ws.nsrc.org.

To start off, log in to the SRV appliance on its console using the isplab username. The password will be provided by the workshop instructors. You can log in via the console, or by using SSH. Note that if you use your own SSH client, we recommend connecting to oob.srvX.groupX.ws.nsrc.org which connects to the out-of-band interface of the SRV appliance (rather than the internal lab network interface).

The GNS3 based lab infrastructure using Cisco’s IOSv and CSR1000v is limited to 1Mbps, as it is intended for investigation of routing concepts, not moving packets for production. If we tried to install Routinator over a 1Mbps link, it would take around 3 hours from start to finish. This isn’t practical for our lab schedule.

The SRV appliance has a second ethernet interface which connects it directly to the lab infrastructure - its a “backdoor” to make the operation of our lab more efficient.

Updating the SRV appliance external connectivity

Let’s disable the interface connecting to the Core router first:

srvX$ sudo ip link set ens3 down

(Note that if you use SSH to access the SRV appliance, you need to connect to oob.srvX.ws.nsrc.org otherwise you’ll lose your connection to the appliance.)

Then point the default route on the SRV appliance to the “backdoor” interface:

srvX$ sudo ip route add default via 100.64.0.1

Check the routing table now. It should look like this:

srvX$ ip route
default via 100.64.0.1 dev ens4 
100.64.0.0/22 dev ens4 proto kernel scope link src 100.64.0.X0

Updating the software packages on the SRV appliance

First lets make sure the repository on the lab server is up to date:

srvX$ sudo apt update
srvX$ sudo apt -y upgrade

and that will bring all installed packages up to the current versions (around 200 at the time of writing).

Once you have done this, check that the server is still using the “backdoor” path using the ip route command introduced in the previous section. If you find that ens3 has returned because of the package update steps, then rerun the commands to shut the interface down, and have the static default route pointing to the “backdoor”.

Confirm that ip route shows the following:

srvX$ ip route
default via 100.64.0.1 dev ens4 
100.64.0.0/22 dev ens4 proto kernel scope link src 100.64.0.X0

We then make sure we have the gnupg2 package installed (it might already be there):

srvX$ sudo apt install -y gnupg2

Preparing to install Routinator

Routinator is conveniently provided by NLnetLabs as a Debian/Ubuntu package. Using their instructions, reproduced here, it is very simple to install Routinator on your lab server.

Check the version of Ubuntu running on the lab server. At time of writing, it is 20.04, otherwise known by the codename Focal. To verify, run this command:

srvX$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:    22.04
Codename:   jammy

Now we are going to update the apt source list to include Routinator:

srvX$ cd /etc/apt/sources.list.d
srvX$ sudo editor routinator.list

and add the following single line:

deb [arch=amd64] https://packages.nlnetlabs.nl/linux/ubuntu jammy main

Be careful with spelling and note the spaces too!

Save the new file and then run:

srvX$ wget -qO- https://packages.nlnetlabs.nl/aptkey.asc | sudo tee /etc/apt/trusted.gpg.d/nlnetlabs.asc

which will get the public key that is used to verify the signature of the package (to verify its authenticity). After that we can then run the update again to get information about the Routinator package.

We also need to make sure that the apt configuration will allow connections to sources which use https rather than http:

srvX$ cd /etc/apt/apt.conf.d
srvX$ sudo editor 99proxy

and make it look like this (the first line will already exist):

Acquire::http::Proxy "http://100.64.0.1:3142/";
Acquire::https::Proxy "DIRECT";

Now run the update again, and apt will get the package list from NLnetLabs as well.

srvX$ sudo apt update

You will see a query to NLnetLabs being made in the output, like this:

srvX$ sudo apt update
...
Get:6 https://packages.nlnetlabs.nl/linux/ubuntu jammy InRelease [4323 B]
Get:7 https://packages.nlnetlabs.nl/linux/ubuntu jammy/main amd64 Packages [5325 B]
Fetched 9648 B in 1s (7858 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

We are now ready to install Routinator.

Installing Routinator

To install Routinator, run:

srvX$ sudo apt install routinator

You will see the package download and new routinator user and group being created, and so on.

Running Routinator

Once the installation is complete, routinator will be set up to start running by the install script. We can check that it is running using:

srvX$ sudo systemctl status routinator

You should see output looking similar to this:

● routinator.service - Routinator 3000
     Loaded: loaded (/lib/systemd/system/routinator.service; enabled; vendor pr>
     Active: active (running) since Fri 2021-10-29 03:48:45 UTC; 6s ago
       Docs: man:routinator(1)
   Main PID: 45705 (routinator)
      Tasks: 6 (limit: 1136)
     Memory: 7.2M
     CGroup: /system.slice/routinator.service
             └─45705 /usr/bin/routinator --config=/etc/routinator/routinator.co>

Oct 29 03:48:45 srv4 systemd[1]: Starting Routinator 3000...
Oct 29 03:48:45 srv4 systemd[1]: Started Routinator 3000.

(noting that the lines are truncated at the screen width and show a >).

Also check the logs using:

srvX$ sudo journalctl -f --unit=routinator

The -f (follow) option prints output until routinator has completed the VRP download.

The first time Routinator is run it will download all the VRPs from the various Certificate Authorities (CAs) - this will take a while (unless on NSRC’s co-located VTPs which all have high bandwidth Internet connectivity).

Note: Once you have started this command running, it might be a good idea to have a refreshment break. With 8 Groups installing and configuring Routinator at the same time, it takes around 20 minutes to sync the VRPs (around 2.4Gbytes worth at time of writing) assuming a 100Mbps Internet connection.

The VRPs are synced mostly using HTTP and RRDP, but there are still some CAs only supporting RSYNC, which seems to result in a much slower rate of copying them to your lab server.

Progress of Routinator when it is running in daemon mode can really only be checked by running du -sh /var/lib/routinator until the disk usage stabilises. Then run the above journalctl command with -f option.

Once journalctl shows that the Routinator VRP download has completed (basically journalctl doesn’t report any more VRP downloads for a few minutes), you are ready to move on to the next step. You will see log output looking similar to:

...
Jul 28 07:45:55 srv1 routinator[25261]: rsync://ca.rg.net/rpki/RGnet-OU/ovsCA/W>
Jul 28 07:45:55 srv1 routinator[25261]: rsync://ca.rg.net/rpki/RGnet-OU/ovsCA/W>
Jul 28 07:45:55 srv1 routinator[25261]: CA for rsync://ca.rg.net/rpki/RGnet-OU/>
Jul 28 07:45:55 srv1 routinator[25261]:    45.132.188.0/22
Jul 28 07:45:55 srv1 routinator[25261]:    147.28.2.0-147.28.6.255
Jul 28 07:45:55 srv1 routinator[25261]:    147.28.8.0/21
Jul 28 07:45:58 srv1 routinator[25261]: RRDP https://invalid.rov.koenvanhove.nl>
Jul 28 07:46:01 srv1 routinator[25261]: rsync://invalid.rov.koenvanhove.nl/repo>
Jul 28 07:46:01 srv1 routinator[25261]: CA for rsync://invalid.rov.koenvanhove.>
Jul 28 07:46:01 srv1 routinator[25261]:    203.119.21.123/32
...

which is a log capture from SRV1 in a previous workshop.

Make a note of the last few lines of the log messages in a separate text file, for future reference.

Using Routinator

We are now ready to run the Routinator server for our routers.

First, we should restore our server’s connection to our Core router, as we will need this for the labs ongoing, so that the routers can talk to the validator. We simply run:

srvX$ sudo netplan apply

Then check the routing table again:

srvX$ ip route
default via 100.68.X.29 dev ens3 proto static 
100.64.0.0/22 dev ens4 proto kernel scope link src 100.64.0.10 
100.68.X.28/30 dev ens3 proto kernel scope link src 100.68.X.30

Note that the default route now points to our Core router again.

Next, we need to modify the Routinator configuration file to suit our lab environment. The default file is called routinator.conf and is located in the /etc/routinator directory. Let’s edit the default configuration file:

srvX$ sudo editor /etc/routinator/routinator.conf

and make sure the following settings are in the file (remembering to replace X in the addresses with your group number):

# Routinator Configuration
#
# <lots of preamble explaining the configuration file>
#
repository-dir = "/var/lib/routinator/rpki-cache"
extratal-dir = "/var/lib/routinator/tals"
rsync-args = ["-4"]
rtr-listen = ["127.0.0.1:3323", "[::1]:3323", "100.68.X.30:3323", "[2001:DB8:X:21::30]:3323"]
http-listen = ["127.0.0.1:8323"]
#

Some explanation of the options:

The repository-dir entry is the default value Routinator expects for where the VRPs are kept. Routinator won’t run without these being set. The TALs for the 5 RIRs are now include with Routinator.
Restrict Routinator to using IPv4 for rsync from our lab network - most CAs now support HTTP or RRDP for synchronisation.
The rtr-listen entry has routinator listening to all connections made to it over IPv4 and IPv6 to its loopback address (to allow for testing) and its ethernet LAN address, sufficient for our needs in our lab.
Routinator by default has a web server listening on port 8323 to allow the operator to check the its status.

Save the modified configuration file.

Next, restart Routinator by running:

srvX$ sudo systemctl restart routinator

and verify that it is indeed running, as you did earlier.

When you restart Routinator, it rechecks all the VRPs it downloaded earlier - but this process should be quick, taking less than 5 minutes. Again run the journalctl -f —unit=routinator command to check the logs - once the last logs you see match what you had earlier, Routinator is ready to listen for connections from routers (otherwise if routers try and connect to it while it is doing its checks, it will refuse the connection).

Finally, verify that Routinator is indeed listening by using telnet to connect to port 3323 on the SRV appliance. You should get a connection being offered. For example:

srvX$ telnet localhost 3323
Trying ::1...
Connected to localhost.
Escape character is '^]'.

telnet> q
Connection closed.

The Routinator is now ready to use in the upcoming labs.

You can check that it is listening for connections too:

srvX$ ss -an | grep 3323
tcp     LISTEN   0        128           100.68.X.30:3323            0.0.0.0:* 
tcp     LISTEN   0        128             127.0.0.1:3323            0.0.0.0:*
tcp     LISTEN   0        128   [2001:db8:X:21::30]:3323               [::]:*
tcp     LISTEN   0        128                 [::1]:3323               [::]:*

(note that these notes have X inserted instead of the actual address)

We are now ready to move on. Please show the lab instructors your working Routinator configuration and the output of the diagnostic commands above.