Network Management

and

Network Operations

I have a network, now what?

Slides based on work by Abha Ahuja

<ahuja@merit.edu>

some slides based on the netmgt talk in T4-98 by Scott Bradner

Outline

What is network management?

Fault Management

Fault detection and tracking

Performance Monitoring

Basic Network Operations

What are typical network problems?

Other parts of network management

Outline (con)

Network Management Tools

what do I need?

what is available?

Pros and Cons of various tools

Network Management - What is it?

Making sure the network is up, running and performing well

Parts of Network Management

fault management

performance management

security management

trouble tracking

statistics and accounting

Fault Management

one of the most important parts of network management

detect network problems

transient/persistent

failure/overload

examples: router down, serial link down

detect server problems

isolating problems

Fault Management (con)

reporting mechanism

link to help desk

notify on-call personnel

setup & control alarm procedures

repair/recovery procedures

ticket system

Fault Management - Fault Detection

Who notices a problem with the network?

Network Operations Center w/ 24x7 operations staff

open trouble ticket to track problem

preliminary troubleshooting

escalate to engineer or call carrier

Fault Management -
Fault Detection (con)

How can you tell if there is a problem with the network?

Network Monitoring Tools

common utilities

ping

traceroute

snmp

Report state or unreachability

detect node down

routing problems

Fault Management -
Fault Detection (con)

“Alert” shows up for NOC

rover

spectrum

NOCol

HP Openview

other

Other methods

customer complaint via phone/email

another ISP notices problem

Fault Detection Example -
Using Rover

Rover = network monitoring system

http://www.merit.edu/internet.tools/rover/

Keep it Simple

add nodes and tests to hostfile

run Display to see status

NOC notices alert on board for failed node

opens ticket

investigates

The Alert Display Program

Time of Alert that failed

Name as in hostfile

IPAddress as in hostfile

Name of Test that failed

Place for status updates

Command line: ‘Help’ Problem #1

hostfile

InetRover

Pingd

Other tests

dixie-X.500()

SMTP(),FTP()

NAMED(),TROUBLE()

WWWTest

InetRover (cont’d)

Extensibility

Generic tests

InetRoverd

file existence

Any # of Displays

telnet/web display

Simple, right?

pingd

Generic test script

Generic test script

Fault Management -
Ticket System (Why all the fuss?)

Very Important!

Need mechanism to track:

failures

current status of outage

carrier ticket #s

Fault Management -
Ticket Systems (Why all the fuss?)

system provides for:

short term memory & communication

scheduling and work assignment

referrals and dispatching

oversight

statistical analysis

long term accountability

Fault Management -
Ticket Systems (Why all the fuss?)

Goal: make your NOC the communication and coordination center!

Central repository for all information

current status

troubleshooting information

Engineers can coordinate their work through the NOC

Fault Management - Ticket Usage

create a ticket on ALL calls

create a ticket on ALL problems

create a ticket for ALL scheduled events

copy of ticket mailed to reporter and mailing list(s)

all milestones in resolution of problem create a new ticket entry with reference to original

ticket stays "open" until problem resolved according to problem reporter

Fault Management - Ticket Example

sample opening ticket

TT0000033975 has been OPENED. Here is the trouble ticket contents:

Create-date : 06/09/99 12:46:42

Ticket ID : TT0000033975

Node + : rs2.mae-west.rsng.net

Equipment Type : host

NOC Customer : RA

Trouble Reported : Unreachable

Next Action : Investigate

Next Action Date : 06/09/99 12:46:42

Outage type : unscheduled

Source of Report : Noc/roverStatus : Assigned

Assigned-to : Noc

Contact Name : rsng

Group Member :

Contact pager#/email address :

Contact Phone : .

Carrier Ticket History :

Carrier :

Carrier Phone :

Ticket information log :

06/09/99 12:46:42 noc-op

toppingb@facesofdeath.ns.itd.umich.edu said ...

11 Wed12:23 rs2MW_O/C 198.32.136.2 PING

Fault Management - Ticket Example

sample progress ticket

TT0000033975 has been MODIFIED. Here are the fields that have been changed:

CopyOfTime : 5

TTC Temp : 0

Ticket information log : toppingb@facesofdeath.ns.itd.umich.edu said ...

While I was investigating this, Debbie from UUNet called (via Merit main

number) to tell us they were seeing it down. She can be reached at

xxx-xxxx. The UUNet ticket is xxxxx..

Fault Management - Ticket Example

sample closing ticket

includes previous ticket contents plus resolution

T0000033975 has been CLOSED. Here is the trouble ticket contents:

01/15/99 12:50:06 noc-op

mgf@wonka.ns.itd.umich.edu said ...

Email response from Abha suggesting contacting peers directly -- see

internal log.

01/15/99 14:25:22 noc-op

aubinc@augustus2.ns.itd.umich.edu said ...

The alerts cleared shortly before 14:00. I called MCI/Worldcom for an

update, and found out their ticket was closed. According to them the

outage was due solely to a power problem.

Closing.

Last-modified-by : noc-op

Modified-date : 01/15/99 14:25:22

Submitter : btracy

Fault Management - typical failures

Node unpingable

no ip connectivity to router

possible reasons:

serial link down

call telco

router down/hardware problem

call engineer

routing problem

troubleshoot with traceroute

routeviews machine

Performance Management

evaluate the behavior of network elements

information used in planning

interface stats

throughput

error rates

software stats

usage

queues

system load

disk space

percent availability

Security Management

tends to be host-based

protect your stats, data and NOC info

protect other services

security required to operate network and protect managed objects

security services

Kerberos

PGP key server

secure time

Security Management (con)

security tools

cops - host configuration checker (www.cert.org)

swatch - email reports of activity on machine

tcpwrappers

ssh/skey

tripwire

distribute security information

bug reports

CERT advisories

bug fixes

intruder alerts

Security Management (con)

reporting procedure for security events

e.g. break-ins

abuse email address for customers to report complaints (abuse@your-isp.net)

control internal and external gateways

control firewalls (external and internal)

security logs

privacy issues a conflict

Security Management

Network based security

Types of attacks

DOS - Denial of Service

ping floods

smurf

attacks that make your network unusable

Spoofing

packets with “spoofed” source address

What types of problems?

Blocking and tracing denial of service attacks

Tracing incoming forged packets back to their source

Blocking outgoing forged packets

Most other security problems are not specific to backbone operators

Deal with complaints

smurf

attacker sends many ping request packets:

from forged (victim) source address

to broadcast address on “amplifier” network

many ping responses from systems on amplifier network

attacker on dialup modem can saturate victim’s T1 using a T3-connected amplifier

http://users.quadrunner.com/chuegen/smurf/

Protection against smurf

configure “no directed-broadcast” on all interfaces

so you can’t be used as an amplifier

trace forged packets back, hop by hop

block outgoing forged packets from your customers

limit the bandwidth that can be used by ICMP traffic

Smurf Attack

victim

attacker

amplifier

215.23.16.0/24

src IP=132.34.65.1

dst IP= 215.23.16.255

5*100 byte packets

132.34.65.1

24.3.2.1

253*5*100

SYN flooding

attacker sends many TCP SYN packet from forged source address

victim sends SYN+ACK packets to invalid address

gets no response

connection hangs in half open state

wastes OS resources, possibly crashing system

Protection against SYN flooding

Make operating system more robust

not a backbone problem, except on routers

Trace and block forged packets

Limit bandwidth that can be used by TCP SYN traffic

Syn attack

attacker

victim

24.13.51.2

132.16.12.5

src IP=230.55.65.1

dst IP=132.16.12.5

connection request packets

( syn packets)

230.55.65.1

Replies go to spoofed IP

Notice a pattern?

Forged packets

Need a way of preventing customers from sending forged packets

Need a way of tracing where forged packets really come from

Tracing forged packets

Start on router near victim

Find how packets get to that router

Repeat on next router

Continue until edge of your AS

Ask next AS to trace further

Need cooperation

IMPORTANT - Should have a 24hour security contact!

Security Management

Protecting your network

traffic shapers

use CAR to limit ICMP traffic

anti-spoofing filters

RFC 2267 (Network Ingress Filtering)

for singly-homed customers

IF packet's source address from within your network

THEN forward as appropriate

IF packet's source address is anything else

THEN deny packet

Filter on the outbound

Preventing forged packets from customers

packet filters!

you know what IP addresses are used (at least for dialup and statically routed customers)

make a filter for each customer that denies other source addresses

very recent cisco code has “ip verify source-address”

Preventing forged packets from you to outside world

you might know all the IP addresses that are used in your AS

if your connections to the outside world and your transit arrangements are not too complicated

make a filter that denies other source addresses

apply that filter to all links from you to other Ases

Configuration and Name Management

track network vitals

ip addresses, interfaces, console phone numbers, etc

NOC needs valid contact info for nodes

network state information

network topology

operation status of network elements

including resources

network element configuration

Configuration and Name Management

inventory management

database of network elements

history of changes & problems

directory maintenance

all hosts & applications

nameserver database

host and service naming coordination

"Information is not information if you can't find it"

Config. Mgmt. - Network State Info.

e.g. SNMP driven display

nnhvd

husc6

harvard

geo

oitgw1

mghgw

sphgw1

wjhgw1

wjh12

generali

talcott

harvisr

huelings

pitirium

nngw

lmagw1

dfch

tch

tch

Network Management Tools

many use SNMP

ping

traceroute

References:

MON - http://www.kernel.org/software/mon/

NOCol - ftp://ftp.navya.com/pub/vikas/nocol.tar.gz

Sysmon - ftp://puck.nether.net/pub/jared

Rover - http://www.merit.edu/~rover

Concord - http://www.concord.com

What is SNMP? (the quick version...)

Simple Network Management Protocol

query - response system

can obtain status from a device

standard queries

enterprise specific

uses database defined in MIB

management information base

What do we use SNMP for?

query routers for:

in and out bytes per second

CPU load

uptime

BGP peer session status

query hosts for:

network status

SNMP Network Management Tools

mrtg (http//:www.ee.ethz.ch/~oetiker/webtools/mrtg

why we like it

simple to use and configure

quickly determine spikes/drops in traffic

ping floods

in/out bps

uptime

supplement to monitoring tools

MRTG

Netscarf/Scion

free

snmp collector and analyzer package

collects snmp data

display on web pages

http://www.merit.net/~netscarf

Other Network Tools

netflow

cflowd (http://www.caida.org/Tools/Cflowd)

collects flow information from cisco routers

AS to AS information

src and destination ip and port information

useful for accounting and statistics

how much of my traffic is port 80?

how much of my traffic goes to AS237?

Netflow examples

Top ten lists (or top five)

##### Top 5 AS's based on number of bytes #######

srcAS dstAS pkts bytes

6461 237 4473872 3808572766

237 237 22977795 3180337999

3549 237 6457673 2816009078

2548 237 5215912 2457515319

##### Top 5 Nets based on number of bytes ######

Net Matrix

----------

number of net entries: 931777

SRCNET/MASK DSTNET/MASK PKTS BYTES

165.123.0.0/16 35.8.0.0/13 745858 1036296098

207.126.96.0/19 198.108.98.0/24 708205 907577874

206.183.224.0/19 198.108.16.0/22 740218 861538792

35.8.0.0/13 128.32.0.0/16 671980 467274801

##### Top 10 Ports #######

input output

port packets bytes packets bytes

119 10863322 2808194019 5712783 427304556

80 36073210 862839291 17312202 1387817094

20 1079075 1100961902 614910 62754268

7648 1146864 419882753 1147081 414663212

25 1532439 97294492 2158042 722584770

More Tools!

http://www.caida.org/Tools/

OC3Mon/Coral

http://www.merit.edu/~ipma

RouteTracker

IRRj

ASExplorer

http://www.geektools.com/

http://www.merit.edu/ipma/tools/other.html

ASexplorer

Route Flap Stats

Looking Glass Tools

route-views.oregon-ix.net>show ip bgp 35.0.0.0

BGP routing table entry for 35.0.0.0/8, version 56135569

Paths: (17 available, best #12)

11537 237

198.32.8.252 from 198.32.8.252

Origin incomplete, localpref 100, valid, external

Community: 11537:900 11537:950

2914 5696 237

129.250.0.3 (inaccessible) from 129.250.0.3

Origin IGP, metric 0, localpref 100, valid, external

Community: 2914:420

2914 5696 237

129.250.0.1 (inaccessible) from 129.250.0.1

Origin IGP, metric 0, localpref 100, valid, external

Community: 2914:420

3561 237 237 237

204.70.4.89 from 204.70.4.89

Origin IGP, localpref 100, valid, external

267 1225 237

204.42.253.253 from 204.42.253.253

Origin IGP, localpref 100, valid, external

Community: 267:1225 1225:237

http://www.merit.edu/~ipma/tools/lookingglass.html

More Looking Glass Tools

Traceroute servers

http://www.merit.edu/ipma/tools/trace.html

Query: trace

Addr: www.isoc.org

Translating "www.isoc.org"...domain server (206.205.242.132) [OK]

Type escape sequence to abort.

Tracing the route to info.isoc.org (198.6.250.9)

1 iad1-core2-fa5-0-0.atlas.digex.net (165.117.129.2) 0 msec 0 msec 4 msec

2 dca5-core2-s5-0-0.atlas.digex.net (165.117.53.41) 0 msec 4 msec 0 msec

3 dca5-core1-fa5-1-0.atlas.digex.net (165.117.56.117) 4 msec 0 msec 4 msec

4 Hssi3-1-0.BR1.DCA1.ALTER.NET (209.116.159.98) 0 msec 0 msec 4 msec

5 101.ATM2-0.XR1.DCA1.ALTER.NET (146.188.160.226) [AS 701] 4 msec 0 msec 4 msec

6 195.ATM7-0.XR1.TCO1.ALTER.NET (146.188.160.102) [AS 701] 4 msec 0 msec 0 msec

7 193.ATM8-0-0.GW1.TCO1.ALTER.NET (146.188.160.33) [AS 701] 4 msec 4 msec 4 msec

8 charlie.isoc.org (198.6.250.1) [AS 701] 8 msec 8 msec 8 msec

9 info.isoc.org (198.6.250.9) [AS 701] 8 msec * 12 msec

Accounting Management

what do you account for?

if you count packets sent

it can inhibit anonymous ftp & web sites

QoS differences in the future

want to charge "user" of service

application dependent determination of "user"

if count hosts

is a PC equal to a mainframe?

cost?

usage-based billing may become common as telcos take over Internet service providers

getoctets

simple traffic stats collector

cron-driven shell procedure

get-octets router1 router2 router3

figures out interface list for each router

then gets

ifInOctets, ifOutOctets, ifInUcastPkts, ifOutUcastPkts

ifInNUcastPkts, ifOutNUcastPkts, system.sysUpTime

ftp://ndtl.harvard.edu/pub/SNMPoll/octets.tar

needs cmu snmp package

getoctets, contd.

makes separate stats file for each interface

example filename: 128.103.1.2.WJHgw1

example data

1997,06,23,160,09,1,00,02,37,EDT,1764089502,1045789221,99138769,92200835,10,628226,758006814

1997,06,23,160,09,1,00,22,37,EDT,1766362487,1047093977,99151676,92213338,10,628281,758126831

1997,06,23,160,09,1,00,42,36,EDT,1768439726,1048266407,99163118,92224546,10,628342,758246748

processing a bit hard

must deal with counter wrap & router reboots

sample period must be < 59 min for an Ethernet

link utilization calculation complex

must include link encapsulation etc

getoctets, processing

UpDate routine

bug in 32 bit versions of perl (gives bad results)

example output

week millions of bits per second millions of octets

ending peak in peak out 95% in 95% out in out total

1997.06.01 5.0976 0.9330 1.3389 0.4104 18782 13752 32534

Accounting, Cont.

could do settlements based in routing information

try to minimize size of routing tables

Telco model

everyone shares in revenue

call an 800 number from a pay phone

800 destinations pays pay phone owner

receive a long distance call to your own switch

you get free for local delivery

Importance of Network Statistics

Accounting

Troubleshooting

Long-term trend analysis

Capacity Planning

Management Tools have statistical functionality

Management for Real

Which tools should I use? What do I really need?

Keep it simple!

Need to consider engineers working remotely

Don’t want to spend too much time maintaining the tool (it should be helping you!)

Different tools for NOC and engineers

Different tools for statistics

RELIABILITY!

Monitoring

simple monitoring tools do 95% of task

e.g. ftp://ndtl.harvard.edu/pub/SNMPoll

e.g. http://www.merit.edu/internet.tools/rover/

monitor should be both poll & trap based for best reliability

but just polling will do better than just traps

and will work fine other than response latency

simple, terse, messages on problems

A Day in the Life of Merit’s NOC

running rover

prefer because easy to tell when change occurs

quickly can determine type of problem

no sifting through GUIs

quick screen display

alert appears on screen

27 Wed02:07 MCH_MSU:S6/1/7.6-->STOCKBRIDG 198.109.177.41 PING

28 Tue16:00 MCH_STOCKBRIDGE:S0.2-->JACKSO 198.109.177.46 PING

29 Tue16:00 MCH_STOCKBRIDGE:E0-GW 207.74.125.129 PING

30 Tue16:00 MCH_STOCKBRIDGE:S0.1-->MSU 198.109.177.42 PING

A Day in the Life of Merit’s NOC

open ticket

investigate

the two most important questions:

can you ping it?

can you trace to it?

get to the the node from somewhere else in the network?

dial-in to the router?

serial line problem? call telco

If necessary, escalate to engineer

Another example - Sluggishness

customer calls NOC - reports sluggishness

open ticket

investigate

check mrtg

more traffic now than normal?

use netflow to determine what type of traffic

possible denial of service attack

circuit problem?

call telco to test

always call customer back to get okay to close

Another example - DOS

Customer reports possible Denial of Service

Open ticket

Investigate

notice a large amount of packets from one destination?

log onto router

ip accounting

sho ip route cache flow

install packet filter

report to offending ISP

Tracing packets on cisco - interface access-group

cisco access list

permit everything, but log packets from 10.2.3.4 to 195.176.0.0/16

access-list 199 permit ip 10.2.3.4 0.0.0.0 195.176.0.0 0.0.255.255 log-input

access-list 199 permit ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255

apply access-list to interface

interface serial3

ip access-group 199 out

Tracing packets on cisco - debug ip packet

cisco access list

permit packets from 10.2.3.4 to 195.176.0.0/16, deny others

access-list 199 permit ip 10.2.3.4 0.0.0.0 195.176.0.0 0.0.255.255 log-input

access-list 199 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255

use access-list with “debug ip packet”

debug ip packet 199

Problems

we are early in the internet management game

there is still a lot to learn

prices still high for functionality

many new NMSs will be on the market soon, will help lower price and expand capabilities

data networks are not "plug and play" with large scale

More Problems

not so good at provoking simple, easy to understand, warning to non-gurus

should have database & logic about when to cry wolf

critical vs, noncritical device, access restrictions, who to call when

needs to be usable by "normal" people

needs to say when users will complain

Even more Problems

training your Network Operations Staff

keeping your database up-to-date

router configs

contact information

communication with the telco

More things you can do!

secure your router

tacacs

radius

restrict login and snmp access

enable syslog logging

security

debugging

More things you can do!

educate your NOC

provide adequate documentation

escalation procedures

register your routers in DNS

traceroutes easier to follow

coolbeans% traceroute www.above.net

traceroute to www.above.net (207.126.96.163), 30 hops max, 40 byte packets

1 eth0-2.michnet1.mich.net (198.108.61.1) 1.074 ms 0.888 ms 0.696 ms

2 hssi1-0-0.msu.mich.net (198.108.22.102) 77.602 ms 75.356 ms 12.437 ms

3 aads.above.net (198.32.130.71) 9.981 ms 15.098 ms 11.342 ms

4 chicago-core1.ord.above.net (209.249.0.129) 9.634 ms 9.834 ms 9.590 ms

5 sjc-chicago-oc3.above.net (209.249.0.125) 71.261 ms 71.232 ms 71.305 ms

6 main2-core1-oc3-3.sjc.above.net (209.133.31.97) 123.499 ms 71.512 ms 71.8

7 www.above.net (207.126.96.163) 72.861 ms 72.624 ms 74.529 ms

More things you can do!

Prevent excessive route-flapping

enable route-flap dampening

use CIDR

use filters

References

http://www.merit.edu/ipma/docs/isp.html

http://www.nanog.org

http://www.caida.org

http://www.nlanr.net

http://www.cisco.com

http://www.amazing.com/internet/

http://www.isp-resource.com/

http://www.merit.edu/ipma

http://www.ripe.net

Network Operations

Abha Ahuja (ahuja@merit.edu)

1996-02-07T17:49:24

ymbk