Agenda: exercises-smokeping.txt

Network Management & Monitoring

Smokeping

Notes:
------
* Commands preceded with "$" imply that you should execute the command as
  a general user - not as root.
* Commands preceded with "#" imply that you should be working as root.
* Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>") 
  imply that you are executing commands on remote equipment, or within 
  another program.

Exercises
----------

0. Log in to your PC or open a terminal window as the sysadmin user.

Once you are logged in you can continue with these exercises.

1. Install Smokeping

    $ sudo apt-get install smokeping

        (Probably already installed on your machines)
    

2. Initial Configuration

    $ cd /etc/smokeping/config.d
    $ ls -l

    -rwxr-xr-x 1 root root  578 2010-02-26 01:55 Alerts
    -rwxr-xr-x 1 root root  237 2010-02-26 01:55 Database
    -rwxr-xr-x 1 root root  413 2010-02-26 05:40 General
    -rwxr-xr-x 1 root root  271 2010-02-26 01:55 pathnames
    -rwxr-xr-x 1 root root  859 2010-02-26 01:55 Presentation
    -rwxr-xr-x 1 root root  116 2010-02-26 01:55 Probes
    -rwxr-xr-x 1 root root  155 2010-02-26 01:55 Slaves
    -rwxr-xr-x 1 root root 8990 2010-02-26 06:30 Targets
 
    The files you need to touch (at a minimum) are:

    * Alerts
    * General
    * Probes 
    * Targets

    Edit Alerts

    $ sudo vi Alerts

    Update the top of the file where it says:

        *** Alerts ***
        to = alertee@address.somewhere
        from = smokealert@company.xy

    to include a proper "to" and "from" field for your server.
    Something like:

        *** Alerts ***
        to = sysadmin@localhost
        from = smokeping-alert@localhost

        If you were going to create tickets from Smokeping alerts the "to"
        address would be an alias for the ticketing system, for example
        "net@localhost".

        We will do this a bit later.

    Add a new alert for later use:

        +rttbadstart
        type = rtt
        # in milliseconds
        pattern = ==S,==U
        priority = 1
        comment = offline at startup

    * "==S, ==U" means "at Startup" and "not Up"
    * "priority = 1" means if multiple alerts are defined for a host
      and multiple alerts match one the one with the highest priority 
      is executed.

        Now save the file and exit, then edit the file General:

        $ sudo vi General

         Change the following lines:

            owner
            contact
            cgiurl
            mailhost

        Something like this should work:

        owner    = NOC
        contact  = sysadmin@localhost
        cgiurl   = http://localhost/cgi-bin/smokeping.cgi
        mailhost = localhost

        Now save the file and exit, then edit the file Probes:

        $ sudo vi Probes

        The current entry in Probes is fine, but if you wish to
        use additional Smokeping checks you can add them in here
        and you can specify their default behavior. You can do
        this, as well, in the Targets file if you wish.

        Here is an example of a Probes file that would specify
        what to use to check for HTTP and DNS latency as well as
        the FPing probe that is used for ping latency:

        *** Probes ***

        + FPing

        binary = /usr/bin/fping

        + EchoPingHttp

        + DNS
        binary = /usr/bin/dig
        pings = 5
        step = 180
        lookup = www.nsrc.org

        Go ahead and update your Probes file with this information.
        Then save the file and exit. And, now let's restart the
        Smokeping service to verify that no mistakes have been made
        before going any further:

        $ sudo /etc/init.d/smokeping stop
        $ sudo /etc/init.d/smokeping start

        You could, also do:

        $ sudo /etc/init.d/smokeping restart

        or

        $ sudo /etc/init.d/smokeping reload

        to reload configuration changes.

    NB! Due to potential problems in the smokeping init script we recommend
        using:

    $ sudo /etc/init.d/smokeping stop
    $ sudo /etc/init.d/smokeping start

    ... instead of the "restart" option


3. Configure monitoring of devices

        The majority of your time and work configuring Smokeping
        will be done in the file /etc/smokeping/config.d/Targets.
        
        For this class please do the following:

        Use the FPing probe to check:

      - all the student NOC PCs
      - classroom NOC
      - switches
      - routers
      
    You can use the classroom Network Diagram on the classroom wiki (http://noc/) to 
    figure out addresses for each item, etc.

        Create some hierarchy to the Smokeping menu for your
        checks. Such as:

            PCs
            Routers
            Switches

        Add a check for HTTP latency for all the classroom PCs. 
        This will mean adding another category, such as:

            HTTP Servers

        If you have time, consider checking some machines that are
        external to our classroom and the conference (your organization's
        website, a popular web page, etc...)

        Look at additional Smokeping probes and consider implementing
        some of them:

        http://oss.oetiker.ch/smokeping/probe/index.en.html

        As trying to explain all syntactical details of how the file
        /etc/smokeping/config.d/Targets is used would require several
        pages we will go through some examples in class, and you can
        refer to the Smokeping configuration files that are in use on 
        the classroom NOC box by going to:

        http://noc/configs/etc/smokeping
        http://noc/configs/etc/smokeping/config.d

    Review these files and try to do all the suggested steps from above.

4. Add DNS Latency Checks

        You can check either or both internal or external names using
        the DNS latency probe.

        Add a menu hierarchy for DNS Latency. Check an external address
        (nsrc.org) and an internal address (noc). This will look something
        like this (in Targets):

    $ sudo vi /etc/smokeping/config.d/Targets

        ++ DNS
        probe = DNS
        menu = External DNS Check
        title = DNS Latency

        +++ nsrc
        host = nsrc.org

        +++ noc
        host = noc.mgmt

    Exit and save your changes to the file Targets.

    Restart Smokeping to see the changes:

    $ sudo /etc/init.d/smokeping stop
    $ sudo /etc/init.d/smokeping start


5. Send Smokeping alerts to our Request Tracker Net queue

    We've already set this up in RT and in /etc/aliases. You just 
    need to point Smokeping alerts to our RT instance. Edit the file
    Alerts:

    $ sudo vi /etc/smokeping/config.d/Alerts

    And change:

        to = sysadmin@localhost

    to

        to = net@localhost

    Now whenever Smokeping sends an alert email with that alert text 
    will arrive to the Net queue in Request Tracker.

    Next, be sure you have alerts defined for some of your Targets.
    You can either turn on alerts by defining alerts for a probe in
    the /etc/smokeping/config.d/Probes file, or by individual Targets
    entries. 

    In our case let's edit the Targets file and turn on alerts for our
    DNS Latency checks. In addition, if you add a DNS latency check for
    a host that does not exist, then we can see a ticket being created
    in RT.

    $ sudo vi /etc/smokeping/config.d/Targets

    Find the following section in the file:

        ++ DNS
        probe = DNS
        menu = External DNS Check
        title = DNS Latency

        +++ nsrc
        host = nsrc.org

        +++ noc
        host = noc.mgmt

    And, add the following host after "+++ noc"

        +++ noexist
        host = does.not.exist
        alerts = rttbadstart

    Save and exit from the file, then restart smokeping:

    $ sudo /etc/init.d/smokeping stop
    $ sudo /etc/init.d/smokeping start

    You will see an error message on the screen:

        WARNING: Hostname 'does.not.exist' does currently not resolve to 
        an IPv6 or IPv4 address

    This is to be expected as the host "does.not.exist" is not a valid
    host. But, Smokeping still starts, and the rttbadstart Alert will 
    now send email to the Net queue for Request Tracker. If you open a
    web browser to your RT instance:

    http://MyMachine/rt/

    and log in as "sysadmin" you will see a new ticket in the home screen
    that has a subject of:

    "[SmokeAlert] rttbadstart is active on 
    AROC.DNSProbe.RT-test"
    

6. MultiHost Graphs

    Once you have defined a group of hosts under a single probe type in your
    /etc/smokeping/config.d/Targets file, then you can create a single graph
    that will show you the results of all smokeping tests for all hosts that
    you define. This has the advantage of letting you quickly compare, for
    example, a group of hosts that you are monitoring with the FPing probe.

    The MultiHost graph functional in Smokeping is extremely picky - pay close
        attention.

    To create a MultiHost graph first edit the file Targets:

    $ sudo vi Targets

    If you had a section for the FPing probe defined that looked like this
    (this is an example only - your Targets file may look different):

        + Local
        menu = Local
        title = Local Network

        ++ LocalMachine
        menu = Local Machine
        title = This host
        host = localhost

        ++ pc1
        menu = pc1
        title = pc1
        host = pc1

        ++ pc2
        menu = pc2
        title = pc2
        host = pc2

        ++ pc3
        menu = pc3
        title = pc3
        host = pc3

    Right now smokeping displays the results of the FPing probe for each
    host defined in separate graphs. If you wish to see the results in a 
    single graph with multiple lines, then you would do this after the last
    FPing probe host definition:

        + MultiHostPCs
        menu = MultiHost Ping
        title = Consolidated Ping Response Time
        host = /Local/LocalMachine /Local/pc1 /Local/pc2 /Local/pc3

    (Note: if the lines get too long, you can have multiple lines for the
    "host" entry by using the "\" character to indicate another line - ask about
    this if you are unsure!)

    Now save and exit the file Targets and restart smokeping:

    $ sudo /etc/init.d/smokeping stop
    $ sudo /etc/init.d/smokeping start

    You should see a new graph under the "MultiHost Ping" menu in your
    smokeping web interface. This graph will have different color lines
    for each host you have defined.


7. Slave instances - only done if we have the time.

    This is a description only for informational purposes in case you wish
    to attempt this type of configuration once the workshop is over.

    The idea behind this is that you can run multiple smokeping instances 
    at multiple locations that are monitoring the same hosts and/or services
    as your master instance. The slaves will send their results to the 
    master server and you will see these results side-by-side with your
    local results. This allows you to view how users outside your network
    see your services and hosts.

    This can be a powerful tool for resolving service and host issues that
    may be difficult to troubleshoot if you only have local data.

    Graphically this looks this:

          [slave 1]     [slave 2]      [slave 3]
                |             |              |
                +-------+     |     +--------+
                        |     |     |
                        v     v     v
                        +---------------+
                        |    master     |
                        +---------------+

    You can see example of this data here:

    http://oss.oetiker.ch/smokeping-demo/

    Look at the various graph groups and notice that many of the graphs
    have multiple lines with the color code chart listing items such as
    "median RTT from mipsrv01" - These are not MultiHost graphs, but rather
    graphs with data from external smokeping servers.

    To configure a smokeping master/slave server you can see the documentation
    here:

    http://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html

    In addition, a sample set of steps for configuring this is available in
    the file sample-smokeping-master-slave.txt.