Agenda: exercises-log-management-tenshi.page

File exercises-log-management-tenshi.page, 5.1 KB (added by brian, 7 years ago)

Line
1	% Log Management Part 2: Using Tenshi
2	%
3	% Network Management & Monitoring
4
5	# Notes
6
7	* Commands preceded with "$" imply that you should execute the command as
8	a general user - not as root.
9	* Commands preceded with "#" imply that you should be working as root.
10	* Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>")
11	imply that you are executing commands on remote equipment, or within
12	another program.
13
14	# Exercises
15
16	First make sure that your routers are configured to send logs to your PC
17	(this should have been done in the previous exercise).
18
19	## Update rsyslog configuration
20
21	Configure rsyslog to save all router logs in one file for monitoring purposes.
22	Edit `/etc/rsyslog.d/30-routerlogs.conf`, find the line
23
24	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
25	local5.* -?RouterLogs
26	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
27
28	... and add the following new line immediately after this:
29
30	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31	local5.* /var/log/network/everything
32	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
33
34	(but before the line which says '& ~'). So what you should end up with is:
35
36	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
37	# editor /etc/rsyslog.d/30-routerlogs.conf
38
39	$template RouterLogs,"/var/log/network/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%-%$HOUR%.log"
40	local5.* -?RouterLogs
41	local5.* /var/log/network/everything
42	& ~
43	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
44
45	This will enable logging of ALL messages matching the local5 facility to a
46	single file, so that we can run a monitoring script on the messages.
47
48	Now restart rsyslog:
49
50	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
51	# service rsyslog restart
52	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
53
54
55	## Log rotation
56
57	Create a daily automated script to truncate the log file so it doesn't
58	grow too big:
59
60	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
61	# editor /etc/logrotate.d/everything
62
63	/var/log/network/everything {
64	daily
65	copytruncate
66	rotate 1
67	postrotate
68	/etc/init.d/tenshi restart
69	endscript
70	}
71	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
72
73	(Then save and exit)
74
75
76	##Â Install tenshi
77
78	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
79	# apt-get install tenshi
80	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
81
82
83	## Configure tenshi
84
85	Configure Tenshi to send you alarms when the routers are configured
86
87	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
88	# editor /etc/tenshi/includes-available/network
89
90	set logfile /var/log/network/everything
91	set queue network_alarms tenshi@localhost sysadm@localhost [/1 * * *] Tenshi Network Alarms
92
93	group_host rtr
94	network_alarms SYS-5-CONFIG_I
95	network_alarms PRIV_AUTH_PASS
96	network_alarms LINK
97	group_end
98	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
99
100	(Then save and exit)
101
102	Create a symlink so that Tenshi loads your new file:
103
104	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
105	# ln -s /etc/tenshi/includes-available/network /etc/tenshi/includes-active
106	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
107
108	Finally restart Tenshi:
109
110	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
111	# service tenshi restart
112	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
113
114
115	##Â Testing
116
117	Log in to your router, and run some "config" commands (example below):
118
119	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
120	$ ssh cisco@rtrX [where "X" is your router number]
121	rtrX> enable
122	Password: <password>
123	rtrX# config terminal
124	rtrX(config)# int FastEthernet0/0
125	rtrX(config-if)# description Description Change for FastEthernet0/0 for Tenshi
126	rtrX(config-if)# ctrl-z
127	rtrX# write memory
128	rtrX# exit
129	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
130
131	Just as in the previous exercise, attempt to shutdown / no shutdown
132	a loopback interface
133
134	Verify that you are receiving emails to the sysadm user from Tenshi.
135	A quick check is to look in the mail directory:
136
137	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
138	$ ls -l /var/mail
139	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
140
141	Make sure you are logged in as sysadm (not root), then do:
142
143	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
144	$ mutt
145	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
146
147	Scroll `up/down` to select a message, hit `Enter` to view it, and `q` to quit.
148
149	If mails are not arriving, then check the following:
150
151	* Are logs arriving in the file `/var/log/network/everything`?
152
153	tail /var/log/network/everything
154
155	* Do these logs show a hostname like 'rtr5'? Remember that the way we have
156	configured tenshi, it only looks at hostnames matching the pattern 'rtr'
157
158	* Check your tenshi configuration file. Restart tenshi if you change it.
159