Agenda: exercises-network-metrics-and-analysis.txt

Network Performance Definitions and Measurement Exercises
=========================================================

Notes:
------
* Commands preceded with "$" imply that you should execute the command as
  a general user - not as root.
* Commands preceded with "#" imply that you should be working as root.
* Commands with more specific command lines (e.g. "GW-RTR>" or "mysql>") 
  imply that you are executing commands on remote equipment, or within 
  another program.
* If a command line ends with "\" this indicates that the command continues
  on the next line and you should treat this as a single line.

Exercises Part I
================

0. Log in to your PC/VM or open a terminal window as the sysadm user.

Network Performance Metrics
---------------------------

1. ping
-------

ping is a program that sends ICMP echo request packets to target hosts and
waits for an ICMP response from the host. Depending on the operating system
on which you are using ping you may see the minimum, maximum, and the mean 
round-trip times, and sometimes the standard deviation of the mean for the
ICMP responses from the target host. For more details see:

http://en.wikipedia.org/wiki/Ping

Blocking ping is generally a bad idea.

With all this in mind, try using ping in a few different ways:

    $ ping localhost

Press ctrl-c to stop the process. Here is typical output from the above
command:

    PING localhost (127.0.0.1) 56(84) bytes of data.
    64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.020 ms
    64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.006 ms
    64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.006 ms
    64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.006 ms
    64 bytes from localhost (127.0.0.1): icmp_seq=5 ttl=64 time=0.006 ms
    64 bytes from localhost (127.0.0.1): icmp_seq=6 ttl=64 time=0.009 ms
    64 bytes from localhost (127.0.0.1): icmp_seq=7 ttl=64 time=0.007 ms
    ^C
    --- localhost ping statistics ---
    7 packets transmitted, 7 received, 0% packet loss, time 5994ms
    rtt min/avg/max/mdev = 0.006/0.008/0.020/0.005 ms

Question: why did the first ICMP response take 20ms while the remaining
responses were much quicker? This is a type of delay. What kind is it?


2. traceroute
-------------

You may have used traceroute before, but have you really looked at what it is
doing? If not, read this:

http://en.wikipedia.org/wiki/Traceroute

Now try:

        $ traceroute nsrc.org

Here's sample output from traceroute to nsrc.org (lines wrapped due to length):

    traceroute to nsrc.org (128.223.157.19), 30 hops max, 60 byte packets
     1  192.168.5.129 (192.168.5.129)  4.291 ms  5.757 ms  6.725 ms
     2  192.168.17.2 (192.168.17.2)  1.933 ms  1.932 ms  2.150 ms
     3  192.168.0.1 (192.168.0.1)  2.140 ms  2.127 ms  2.598 ms
     4  10.0.0.129 (10.0.0.129)  2.586 ms  2.576 ms  4.548 ms
     5   (168.234.72.1)  4.792 ms  4.786 ms  4.750 ms
     6  200.0.204.69 (200.0.204.69)  7.456 ms  5.665 ms  5.890 ms
     7  panama-salvador.core.redclara.net (200.0.204.185)  64.651 ms  64.884 ms  64.870 ms
     8  panama-santiago.core.redclara.net (200.0.204.22)  124.865 ms  124.853 ms  124.841 ms
     9  saopaulo-santiago.core.redclara.net (200.0.204.38)  172.008 ms  171.793 ms  172.019 ms
    10  ge-7-1-0.0.rtr.chic.net.internet2.edu (64.57.28.114)  172.006 ms 
        xe-2-2-0.88.rtr.wash.net.internet2.edu (198.32.11.105)  244.441 ms  244.675 ms
    11  xe-0-1-0.0.rtr.atla.net.internet2.edu (64.57.28.6)  258.151 ms  258.384 ms  258.618 ms
    12  xe-0-0-0.0.rtr.salt.net.internet2.edu (64.57.28.24)  207.383 ms  
        207.602 ms xe-1-0-0.0.rtr.hous.net.internet2.edu (64.57.28.112)  282.040 ms
    13  xe-2-0-0.0.rtr.losa.net.internet2.edu (64.57.28.96)  314.004 ms 
        xe-1-0-0.0.rtr.seat.net.internet2.edu (64.57.28.105)  224.293 ms  224.527 ms
    14  vl-101.xe-0-0-0.core0-gw.pdx.oregon-gigapop.net (198.32.165.65)  328.948 ms 
        vl-102.xe-1-0-0.core0-gw.pdx.oregon-gigapop.net (198.32.163.69)  227.015 ms 
        vl-101.xe-0-0-0.core0-gw.pdx.oregon-gigapop.net (198.32.165.65)  328.184 ms
    15  vl-105.uonet9-gw.eug.oregon-gigapop.net (198.32.165.92)  330.660 ms  330.891 ms  229.940 ms
    16  vl-3.uonet2-gw.uoregon.edu (128.223.3.2)  331.359 ms  229.748 ms  229.727 ms
    17  nsrc.org (128.223.157.19)  229.458 ms  229.460 ms  330.862 ms

Do you understand what each item means? If not, see the Wikipedia page and type:

    $ man traceroute

for more information. What does it mean if you see lines like this?

    15  * * *
    16  * * *
    17  * * *

Again, read "man traceroute" for details.

As you can see traceroute can be used to determine where problems are taking place
between two endpoints on a network.


3. mtr
------

The mtr tool combines ping and traceroute in to a single, dynamically updating display.
Give it a try:

    $ mtr nsrc.org

The output of the command looks different on different Linux and UNIX flavors, but in
general you'll see a summary of packet loss to each node on the path to the remote
target host, number of ICMP echo request packets sent, last rtt (round-trip-time) to 
the host, average, best and worst rtt as well as the standard deviation of rtt's.

By showing the percent loss of packets in this format it makes it much easier to see
where you may be having network issues.


4. ping with variable packet size
---------------------------------

By default, ping sends out IP datagrams of size 84 bytes:

* 20 bytes IP header
*  8 bytes ICMP header
* 56 bytes data padding

However, you can send out larger packets using the -s option. Using
`-s 1472` will give you a 1500-byte IP datagram, which is the maximum for
most networks before fragmentation takes place (MTU = Maximum Transmission
Unit)

This simple mechanism can be used to debug all sorts of problems, and even
distinguish between transmission delay and propagation delay.

For this exercise, first determine your default gateway, which is the first
hop in a traceroute, or use `netstat -rn` for destination 0.0.0.0

Send 20 standard pings to that address:

    ping -c20 10.108.240.254

Make a note of the *minimum* round-trip time seen (t1).

Now send 20 maximum-sized pings:

    ping -c20 -s1472 10.108.240.254

Again, make a note of the *minimum* round-trip time seen (t2).

The propagation delay is the same in both cases, so the larger round-trip
time must be due to transmission delay.

You can now estimate the transmission delay and hence the bandwidth of
the link.

    increase in transmission time   =  t2 - t1
    increase in bits sent           =  (1500-84) * 8 * 2  = 22656

(x2 because the round-trip time involves sending the packet twice)

Divide the bits by time to get an estimate of bits per second. Remember to
convert milliseconds to seconds first.

By doing this for subsequent hops, it's possible to estimate the bandwidth
on each hop, even those remote from you. There is a tool available which
does this automatically - it's called "pathchar" but you have to build it
from source.


---------------------------------------------------------------------------


Exercises Part II
=================

Network Analysis
----------------

1. lsof and netstat
-------------------

See what services are running on your machine. You can use the
presentation as a reference.

Or, utilize "man lsof", "man netstat", "lsof -h" and "netstat -h" to see
the available options (there are a lot!). Remember to use
sudo when using lsof and netstat to give yourself necessary permissions
to view everything.

* Using lsof, what IPv4 services are listening on your machine?
* Using netstat, what IPv4 and IPv6 services are listening on your machine?


2. tcpdump and tshark
---------------------

To use tcpdump you need to use sudo, or be root. To use wireshark you need 
to open a terminal and use sudo as a normal user (i.e. userid "sysadm"):

Use tcpdump like this:

        $ sudo tcpdump -i lo -A -s1500 -w /tmp/tcpdump.log

Now, generate some traffic on your lo interface in another terminal.

For example:

        $ ping localhost
        $ ssh localhost
 
etc. Afterwords press CTRL-C to terminate the tcpdump session.

Note: ssh generates much more "interesting" output. Now let's read the
output from tcpdump using shark:

        $ sudo tshark -r /tmp/tcpdump.log | less

What do you see? Can you follow the SSH session you initiated earlier?

Now try something like this:

        $ sudo rm /tmp/tcpdump.log
        $ sudo tcpdump -i eth0 -A -s1500 -w /tmp/tcpdump.log

In another terminal do:

        $ ftp limestone.uoregon.edu
 
        Connected to limestone.uoregon.edu.
        220 FTP Server ready.
        Name (limestone.uoregon.edu:sysadmin): anonymous
        Password: <anything you want>
        ftp> exit

End the tcpdump session in the other terminal (CTRL-C). Now view the 
contents of the log file:

        $ sudo tshark -r /tmp/tcpdump.log | less

Can you see your password? If you have a lot of traffic on  your network, then
the tcpdump.log file may be fairly large. You can search for your FTP session
by typing:

        "/FTP"

in the output screen. Since you piped your shark command output to the "less"
command using the "/" to search for strings works. Now press the "n" key for 
"n"ext to follow the FTP session. You should see a line with the string:

        "FTP Request: PASS PasswordYouTypedIn"

Sniffing unencrypted passwords on wireless lans is very easy with a tool like 
this.

3. Using iperf
--------------

Use "man iperf" or "iperf -h" for help.

Ask your neighbor to run:

        $ iperf -s

Connect to your neighbor's machine using:

        $ iperf -c ipNeighbor

How is the throughput between  your machines?

Consider connecting both your PCs directly together (one cable, 
no switch). Use a private IP address on both machines, verify
you can ping each other, then repeat the previous steps with
your new connection. Has your throughput improved? 

If you have time continue playing with iperf options. If you have a
remote PC running UNIX or Linux you might want to try installing iperf
and testing your connection from the workshop lab to your remote 
machine.

Some more things to try...

* Test TCP using various window sizes (-2).

* Verify TCP MSS (-m). How does this affect throughput? What is
  Path MTU discovery?

* Test with two parallel threads (-P) and compare the totals. Is
  there any difference? Why?

* Test with different packet sizes and the TCP_NODELAY (-N) option.