1 Analyzing SYSLOG

1.1 Editing, Analyzing Streams of Text

It is often extremely useful as a Systems / Network admin to be able to edit, display and count fields from text streams. Luckily, Linux distributions like CentOS and Ubuntu come configured with tools for editing and searching streams of text. In this tutorial, we will learn the basics of the following tools:

• cat
• less
• grep
• awk

1.1.1 CAT

root@ubuntu-server:/var/log# man cat

NAME
       cat - concatenate files and print on the standard output

SYNOPSIS
       cat [OPTION]... [FILE]...

DESCRIPTION
       Concatenate FILE(s), or standard input, to standard output.
...

1.1.2 Example: Show authentication attempts from 'auth.log'

root@ubuntu-server:~# cd /var/log
root@ubuntu-server:/var/log# cat auth.log

In this very simple example we will take a look at our 'auth.log'. To do so we will use the 'cat' tool.

• 'cat' can take an arbitrary number of files and print them to stdout (your terminal). Files will be printed in the order they are listed. For instance, issuing 'cat auth.log kern.log' would print 'auth.log' first, and 'kern.log' second to the terminal.

root@ubuntu-server:/var/log# cat auth.log
Jan  5 15:48:51 ubuntu-server sshd[818]: Server listening on 0.0.0.0 port 22.
Jan  5 15:48:51 ubuntu-server sshd[818]: Server listening on :: port 22.
Jan  5 15:48:59 ubuntu-server login[899]: pam_unix(login:session): session opened for user bob by LOGIN(uid=0)
Jan  5 15:53:14 ubuntu-server sudo:      bob : TTY=tty1 ; PWD=/home/bob ; USER=root ; COMMAND=/usr/sbin/dpkg-reconfigure -plow unattended-upgrades
Jan  5 15:53:14 ubuntu-server sudo: pam_unix(sudo:session): session opened for user root by bob(uid=0)
Jan  5 15:53:57 ubuntu-server sudo: pam_unix(sudo:session): session closed for user root
Jan  5 16:17:01 ubuntu-server CRON[1249]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan  5 16:17:01 ubuntu-server CRON[1249]: pam_unix(cron:session): session closed for user root
Jan  5 16:18:08 ubuntu-server sudo:      bob : TTY=tty1 ; PWD=/home/bob ; USER=root ; COMMAND=/bin/su -
Jan  5 16:18:08 ubuntu-server sudo: pam_unix(sudo:session): session opened for user root by bob(uid=0)
...

1.1.3 Example: Using pipes and pagers

root@ubuntu-server:~# cd /var/log
root@ubuntu-server:/var/log# cat auth.log | less

In this example, we're doing something rather silly (silly because we don't strictly need 'cat') to illustrate a point. In the first example, we used 'cat' to look at the 'auth.log' file. You may have noticed that this hard to navigate and impossible to search. The solution to this problem is often to pipe to a pager. A pager is a tool used to view (but not change) the contents of a text file one screen at a time. ¹

• Pagers do not read an entire file into memory at once, they are useful for reading large files
• 'less' is by no means the only or best pager, there are many others including 'more'
• 'less' starts you at the end of a file
• Notice that we issue a '|' after the 'cat auth.log' above. This is referred to as a 'pipe', it will take the text that was destined for your terminal window and redirect it to another program. In this case, we redirect the text to the pager 'less'.

Jan  6 09:25:07 ubuntu-server sudo:      bob : TTY=tty1 ; PWD=/home/bob ; USER=root ; COMMAND=/usr/bin/vim /etc/network/interfaces
Jan  6 09:25:07 ubuntu-server sudo: pam_unix(sudo:session): session opened for user root by bob(uid=0)
Jan  6 09:25:19 ubuntu-server sudo: pam_unix(sudo:session): session closed for user root
Jan  6 09:25:42 ubuntu-server sudo:      bob : TTY=tty1 ; PWD=/home/bob ; USER=root ; COMMAND=/usr/sbin/service networking restart
Jan  6 09:25:42 ubuntu-server sudo: pam_unix(sudo:session): session opened for user root by bob(uid=0)
Jan  6 09:25:42 ubuntu-server sudo: pam_unix(sudo:session): session closed for user root
Jan  6 09:25:58 ubuntu-server sudo:      bob : TTY=tty1 ; PWD=/home/bob ; USER=root ; COMMAND=/etc/init.d/networking restart
Jan  6 09:25:58 ubuntu-server sudo: pam_unix(sudo:session): session opened for user root by bob(uid=0)
Jan  6 09:25:58 ubuntu-server sudo: pam_unix(sudo:session): session closed for user root
Jan  6 09:26:01 ubuntu-server sudo:      bob : TTY=tty1 ; PWD=/home/bob ; USER=root ; COMMAND=/sbin/reboot
Jan  6 09:26:01 ubuntu-server sudo: pam_unix(sudo:session): session opened for user root by bob(uid=0)
Jan  6 09:26:01 ubuntu-server sudo: pam_unix(sudo:session): session closed for user root
Jan  6 09:27:57 ubuntu-server sshd[880]: Server listening on 0.0.0.0 port 22.
Jan  6 09:27:57 ubuntu-server sshd[880]: Server listening on :: port 22.
Jan  6 09:28:05 ubuntu-server login[997]: pam_unix(login:session): session opened for user bob by LOGIN(uid=0)
Jan  6 09:33:44 ubuntu-server sshd[1123]: Invalid user rotsted from 192.168.56.1
Jan  6 09:33:44 ubuntu-server sshd[1123]: input_userauth_request: invalid user rotsted [preauth]
:/fail

• When you enter 'less', you will see the same text from the previous example, only this time you can use the directional keys to navigate around the text
• There is a ':' at the bottom of the pager. This is the place where you can issue commands to the pager
  • 'less' allows you to search the text in the pager by placing a /<search string> in the pager's command prompt
  • If you wish to exit less type :q

1.1.4 GREP

root@ubuntu-server:/var/log# man grep

NAME
       grep, egrep, fgrep, rgrep - print lines matching a pattern

SYNOPSIS
       grep [OPTIONS] PATTERN [FILE...]
       grep [OPTIONS] [-e PATTERN | -f FILE] [FILE...]

DESCRIPTION
       grep  searches the named input FILEs (or standard input if no files are named, or if a single hyphen-minus (-) is given as file name) for lines containing a match to the
       given PATTERN.  By default, grep prints the matching lines.

       In addition, three variant programs egrep, fgrep and rgrep are available.  egrep is the same as grep -E.  fgrep is the same as grep -F.  rgrep is the  same  as  grep -r.
       Direct invocation as either egrep or fgrep is deprecated, but is provided to allow historical applications that rely on them to run unmodified.
...

1.1.4.1 Example: 'grep' with a file as input, find failed logins

root@ubuntu-server:/var/log# grep -i "fail" auth.log 

In this example, we will search for failed authentication messages. We will use the following options:

-F, --fixed-strings
    Interpret PATTERN as a list of fixed strings, separated by newlines, any of which is to be matched.
-i, --ignore-case
    Ignore case distinctions in both the PATTERN and the input files.

• By default, grep's pattern matching engine is case sensitive and takes a regular expression as input.
• We use the fixed-string option because we are feeding grep a fixed string as opposed to a regular expression.
• We use the ignore-case option because we want 'grep' to be more 'greedy' or forgiving in matching patterns for us.
• Take a look at the output below, you will notice below that if we did not specify the 'ignore-case' option, we would not have matched the 'sshd' message because 'sshd' has a capital 'F' in it's failure message.

root@ubuntu-server:/var/log# grep -i "fail" auth.log
Jan  6 09:24:14 ubuntu-server login[942]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=  user=root
Jan  6 09:24:17 ubuntu-server login[942]: FAILED LOGIN (1) on '/dev/tty1' FOR 'root', Authentication failure
Jan  6 09:24:22 ubuntu-server login[942]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost= 
Jan  6 09:24:25 ubuntu-server login[942]: FAILED LOGIN (2) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
Jan  6 09:25:02 ubuntu-server sudo: pam_unix(sudo:auth): authentication failure; logname=bob uid=1000 euid=0 tty=/dev/tty1 ruser=bob rhost=  user=bob
Jan  6 17:27:35 ubuntu-server sudo: pam_unix(sudo:auth): authentication failure; logname=bob uid=1000 euid=0 tty=/dev/pts/0 ruser=bob rhost=  user=bob
Jan  7 13:36:09 ubuntu-server sudo: pam_unix(sudo:auth): authentication failure; logname=bob uid=1000 euid=0 tty=/dev/pts/0 ruser=bob rhost=  user=bob
Jan  7 13:36:13 ubuntu-server sudo: pam_unix(sudo:auth): conversation failed
Jan  7 17:31:17 ubuntu-server sudo: pam_unix(sudo:auth): authentication failure; logname=bob uid=1000 euid=0 tty=/dev/pts/0 ruser=bob rhost=  user=bob
Jan  7 17:31:22 ubuntu-server sshd[4465]: Failed password for root from 192.168.56.1 port 59379 ssh2

1.1.4.2 Example: Extract IP for failed logins using awk

root@ubuntu-server:/var/log# grep -i "Failed password" auth.log | awk '{ print $11 }'

• In the previous examples, we looked at excerpts from the man pages. The 'awk' man page is rather difficult to follow, so I will forego it in this example.
• Today we're going to stick to a very simple use for awk: extracting fields from log files

1. Find the 'sshd' logs that indicate a 'failed' login.

In the previous example, we used 'grep' to find all log messages with the term "fail" in them. Lets try that again, this time, lets only match messages that contain the string 'sshd'. One way to accomplish this is to pipe one 'grep' into another:

root@ubuntu-server:/var/log# grep -iF "sshd" auth.log | grep -iF "fail"
Jan  7 17:31:22 ubuntu-server sshd[4465]: Failed password for root from 192.168.56.1 port 59379 ssh2
Jan  7 17:31:22 ubuntu-server sshd[4465]: Failed password for root from 192.168.56.1 port 59379 ssh2
Jan  7 17:31:36 ubuntu-server sshd[4473]: Failed password for root from 192.168.56.1 port 59383 ssh2
Jan  7 17:31:37 ubuntu-server sshd[4473]: Failed password for root from 192.168.56.1 port 59383 ssh2
Jan  7 22:38:04 ubuntu-server sshd[5073]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.56.1  user=root
Jan  7 22:38:06 ubuntu-server sshd[5073]: Failed password for root from 192.168.56.1 port 53875 ssh2
Jan 21 15:33:17 ubuntu-server sshd[2595]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost  user=root
Jan 21 15:33:19 ubuntu-server sshd[2595]: Failed password for root from ::1 port 58866 ssh2
Jan 21 15:33:26 ubuntu-server sshd[2595]: message repeated 2 times: [ Failed password for root from ::1 port 58866 ssh2]
Jan 21 15:33:26 ubuntu-server sshd[2595]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost  user=root

2. Now that we know the format of these messages, find the field number for the source IP

• The 'auth.log' is formatted in a 'space delimited' fashion -- white space is used to delimit the data contained in the log.
• Pretend for a moment that each field of the log has a number associated with it, starting with the number 1
• Counting from the left to right there are 14 space delimited fields, the field we're interested in, the source IP is at field 12

1    2 3        4             5           6      7        8   9    10   11           12   13    14  
Jan  7 22:38:06 ubuntu-server sshd[5073]: Failed password for root from 192.168.56.1 port 53875 ssh2

3. Craft an 'awk' command that will extract the 11th field.

• 'awk' commands consist of a pattern and an action. A pattern defines the text to which awk applies an action.
• 'awk' actions are always contained within curly brackets: awk '<pattern> { <action> } <file>'
• We will use 'grep' to select the lines to pipe to 'awk' instead of a selector pattern.
• We will be piping to 'awk' we will not need to provide a file to the 'awk' command
• The action we want to perform is "print" field 11: awk '{print $11}'
• Note: If we wanted to, we could use the following 'awk' command to select and print lines: awk '/sshd.*Fail/ { print }' auth.log

4. Execute the command!

root@ubuntu-server:/var/log# grep -i "Failed password" auth.log | awk '{ print $11 }'
192.168.56.1
192.168.56.1
192.168.56.1
192.168.56.1
192.168.56.1
::1