DNSSEC Service Monitoring with Nagios

Notes

Commands preceded with "$" imply that you should execute the command as a general user - not as root.
Commands preceded with "#" imply that you should be working as root.
Commands with more specific command lines (e.g. "rtrX>" or "mysql>") imply that you are executing commands on remote equipment, or within another program.

Overview

DNSSEC signatures expire if they are not replaced. This can make DNS names disappear for clients who are doing DNSSEC validation. We can avoid this kind of thing by monitoring our zones in software and sounding the alarm if signatures look like they are getting dangerously close to their expiry.

The goal is to find out that something bad might happen before it actually does. This is actually a useful goal for everything in life, but in this workshop we are mainly concentrating on DNSSEC.

Objectives

In this exercise we are going to:

install nagios4 (monitoring platform) with Apache2 (web server)
install the world-famous NSRC DNSSEC signature validity plugin
configure our lab hosts for monitoring
configure DNSSEC validity monitoring for our signed zone
play with the web page and extend as seems useful
relax and luxuriate in the comfortable certainty that EVERYTHING is FINE

Installing Nagios

Since we are more interested in using the software in this workshop than understanding in enormous detail how it is built, we will use packages to make this part very easy.

We will install everything on the lab machine soa.grpXX.dns.te-labs.training. (your SOA server)

Make sure you are on this machine - you can verify by typing hostname:

$ hostname

soa.grpXX.dns.te-labs.training

Update the local package repository so that it is current

# sysadm@soa:~$ sudo apt-get update

If this doesn't work, check that the resolver on your server is pointing somewhere that lets you resolve names on the real Internet, not just the lab. In this lab, you can use 10.0.0.254.

$ cat /etc/resolv.conf
nameserver 10.0.0.254

Install Nagios4 from packages

$ sudo apt-get install nagios4 nagios-nrpe-plugin nagios-plugins-contrib

You may be asked some questions during the package installation process, depending on what packages are already installed.

If postfix is installed as a dependency, select "Local only" for the general type of mail system this server will be.

If you are asked to specify the default "mail name" for the server because postfix is being installed, just accept the default (the hostname of the machine).

Next, load the following apache modules and then restart the webserver


$ sudo a2enmod auth_digest authz_groupfile cgi rewrite 
$ service apache2 restart

Backup and then open the nagios web configuration file


$ sudo cp -v /etc/apache2/conf-enabled/nagios4-cgi.conf  /etc/apache2/conf-enabled/nagios4-cgi.conf.backup
$ sudo nano /etc/apache2/conf-enabled/nagios4-cgi.conf

Replace EVERYTHING in the nagios4-cgi.conf file with the following lines.


   ScriptAlias /cgi-bin/nagios4 /usr/lib/cgi-bin/nagios4
   ScriptAlias /nagios4/cgi-bin /usr/lib/cgi-bin/nagios4

   # Where the stylesheets (config files) reside
   Alias /nagios4/stylesheets /etc/nagios4/stylesheets

   # Where the HTML pages live
   Alias /nagios4 /usr/share/nagios4/htdocs

   <DirectoryMatch (/usr/share/nagios4/htdocs|/usr/lib/cgi-bin/nagios4|/etc/nagios4/stylesheets)>
      Options ExecCGI
      AllowOverride None
      Order allow,deny
      Allow from all
      AuthName "Restricted Area"
      AuthType Basic
      AuthUserFile /etc/nagios4/htdigest.users
      Require valid-user
   </DirectoryMatch>

Make sure that there are no syntax errors. If you encounter any, go ahead and fix them.


$ sudo apachectl configtest

Then create a nagios administrator username and password by issuing the following command:


$ sudo htpasswd /etc/nagios4/htdigest.users nagiosadmin

You will be asked for a password for a web user "nagiosadmin". To make things easy in the workshop, use the same password as you have been using for the sysadm user.

Install DNSSEC signature validity Nagios plugin

The plugin we will use is this one:

https://github.com/ableyjoe/checksig.sh

It's a simple shell script that you should feel very free to extend and modify on your own. If you have any good ideas about how it could be made better, you know who to contact!

We will download the script itself directly to the server:

$ cd
$ wget 'https://raw.githubusercontent.com/ableyjoe/checksig.sh/master/checksig.sh'

You can now copy the script to the Nagios plugins directory and make it executable:

$ sudo mv checksig.sh /usr/lib/nagios/plugins/
$ sudo chmod 755 /usr/lib/nagios/plugins/checksig.sh

This plugin needs a particular utility to be installed, "gawk". So make sure that is installed:

$ sudo apt-get install gawk

Add workshop configuration to Nagios

We will add the various hosts in our group environment to Nagios, and install some DNSSEC signature validity checks against all the servers.

Depending on how the workshop has been arranged this time, some of your servers might not be running nameservers. Don't worry about that -- they will show up as warnings or errors in Nagios, and that's ok. Add them to the configuration anyway.

$ cd /etc/nagios4/conf.d

We are going to create a new file in this directory that contains all of our workshop-specific configuration. We shall call it nsrc-workshop.cfg.

$ sudo editor nsrc-workshop.cfg

Adapt the following configuration and type it in. Don't just cut and paste the following or it won't work: you need to set addresses and names that are appropriate to your group. Feel free to modify things, especially if you are a Secret Nagios Ninja and know all the Secret Tricks.

Remember to replace X with your group number

define host {
        host_name               soa.grpX
        alias                   GRPX Master Server
        address                 10.X.1.1
        use                     linux-server
}

define host {
        host_name               resolv.grpX
        alias                   GRPX DNSSEC Validator
        address                 10.x.1.2
        use                     linux-server
}

define hostgroup {
        hostgroup_name  grpX-servers
        alias           GRPX Servers
        members         soa.grpX,resolv.grpX
}

define service {
        hostgroup_name          grpX-servers
        service_description     grpX.mytld-sig-validity
        check_command           checksig!mytld
        use                     generic-service
        max_check_attempts      5
        check_interval          5
        retry_interval          3
}

define command {
        command_name            checksig
        command_line            /usr/lib/nagios/plugins/checksig.sh $HOSTADDRESS$ $ARG1$ 40m 20m
}

Check the new configuration, and then restart Nagios to activate it

$ sudo nagios4 -v /etc/nagios4/nagios.cfg

You can ignore warnings for "no default contacts or contactgroups defined".

Check that Apache and Nagios are running

(Re)start Nagios and Apache

$ sudo service nagios4 restart
$ sudo service apache2 restart

Point a browser on your laptop at:

http://52.47.150.68:8XX1/nagios4

Where XX is your group number, from 01 to 30 - so group 7 would use port 8071

You should be challenged for a username and password. Use the username "nagiosadmin" and the password you specified during the nagios4 package installation.

If you just clicked on that link and it didn't work, look at it again and realise that you have to replace XX with your group number. If you do this before you ask for help, you will make the world a better place, and your instructor will be a happier person.

$ sudo service nagios4 restart

Use your web browser to check your services

Same URL as before; look under "Services" in the left-hand-side navigation panel. You should see signature validity checks for the DNS servers you configured in step 6.