These exercices demonstrate some of the tools used for tasks that every system administrator should perform when installing or hardening a system.
# apt-get install lsof psmiscFirst you can see what is running on your machine by typing something like:
$ ps auxwwwYou will see lots and lots of stuff go by. So, let's look at this a bit more closely:
$ ps auxwww | less(press [spacebar] to go one page down, and [b] to go one page up)
Now, browsing through all this we can see there are a bunch of initial system processes that start to support our hardware (items in "[ ]"). Let's filter all of this out and see what we are left with:
$ ps auxwww | grep -v "\["(Hint: You might want to copy and paste this in to a command window)
What's left?
Have a look and see if you can identify everything in the remaining list. Your list of processes should look something like:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.8 0.4 4452 2348 ? Ss 18:38 0:00 /sbin/init
root 163 0.0 0.2 4916 1340 ? S 18:38 0:00 mountall --daemon
root 264 0.0 0.1 3008 616 ? S 18:38 0:00 upstart-udev-bridge --daemon
root 269 0.0 0.2 12160 1488 ? Ss 18:38 0:00 /lib/systemd/systemd-udevd --daemon
message+ 312 0.0 0.1 4232 972 ? Ss 18:38 0:00 dbus-daemon --system --fork
root 347 0.0 0.3 4212 1620 ? Ss 18:38 0:00 /lib/systemd/systemd-logind
syslog 352 0.0 0.2 30476 1068 ? Ssl 18:38 0:00 rsyslogd
root 383 0.0 0.1 3012 640 ? S 18:38 0:00 upstart-file-bridge --daemon
root 448 0.0 0.3 5508 1852 ? Ss 18:38 0:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
root 622 0.0 0.1 4168 808 tty4 Ss+ 18:38 0:00 /sbin/getty -8 38400 tty4
root 623 0.0 0.1 4168 812 tty5 Ss+ 18:38 0:00 /sbin/getty -8 38400 tty5
root 628 0.0 0.1 4168 816 tty2 Ss+ 18:38 0:00 /sbin/getty -8 38400 tty2
root 629 0.0 0.1 4168 808 tty3 Ss+ 18:38 0:00 /sbin/getty -8 38400 tty3
root 631 0.0 0.1 4168 812 tty6 Ss+ 18:38 0:00 /sbin/getty -8 38400 tty6
root 667 0.0 0.4 7796 2436 ? Ss 18:38 0:00 /usr/sbin/sshd -D
root 679 0.0 0.1 2192 600 ? Ss 18:38 0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
root 681 0.0 0.1 3052 768 ? Ss 18:38 0:00 cron
root 721 0.0 0.1 3264 920 ? S 18:38 0:00 upstart-socket-bridge --daemon
root 729 0.0 0.4 5608 2508 ? Ss 18:38 0:00 /usr/sbin/apache2 -k start
www-data 732 0.0 0.3 228240 1972 ? Sl 18:38 0:00 /usr/sbin/apache2 -k start
www-data 733 0.0 0.3 228240 1972 ? Sl 18:38 0:00 /usr/sbin/apache2 -k start
root 812 0.0 0.1 4168 816 tty1 Ss+ 18:38 0:00 /sbin/getty -8 38400 tty1
root 813 0.0 0.1 2416 736 ttyS0 Ss+ 18:38 0:00 /sbin/getty -L 115200 ttyS0 xterm
sysadm 849 0.0 0.3 11192 1812 ? S 18:38 0:00 sshd: sysadm@pts/0
sysadm 850 0.0 0.3 5176 1860 pts/0 Ss 18:38 0:00 -bash
sysadm 861 0.0 0.2 4740 1112 pts/0 R+ 18:39 0:00 ps auxwwwYou can type "man" or search in Google to figure out what all this is. For instance:
$ man acpid
$ man getty
$ man cron
$ man sshdWhen investigating a security incident or suspicious process, one of the most important things you will want to know is how a process was started.
To visualize the relationships between processes use 'pstree'.
# pstreeinit---acpid
|-apache2---2*[apache2---26*[{apache2}]]
|-avahi-daemon---avahi-daemon
|-cron
|-cups-browsed
|-dbus-daemon
|-dhclient
|-7*[getty]
|-mountall
|-rsyslogd---3*[{rsyslogd}]
|-snmpd
|-sshd---sshd---sshd---bash---sudo---su---bash---pstree
|-systemd-logind
|-systemd-udevd
|-upstart-file-br
|-upstart-socket-
|-upstart-udev-brOnce you feel pretty comfortable with what's running on your system you might consider if you need each item. If there is something running that is unnecessary, then you should consider uninstalling or disabling the service.
To uninstall a service, use the APT package manager's "remove" function:
# apt-get remove <pkg_name>The way a service is disabled at startup depends on which mechanism the developers used to initialized it.
A service can be started using the traditional System V init script:
# update-rc.d <pkg_service> removeServices can also be via a system called "upstart". To find out if a service was started by upstart issue the following command:
# ls -alh /etc/init | grep <pkg_service>If you see a .conf file that corresponds with the service you want to disable, the service is controlled by upstart.
To disable a service initialized by upstart, issue the following command:
# echo "manual" > /etc/init/<pkg_service>.overrideThe next step is to see if any of these services are listening to the network for connections:
# lsof -iYou'll see something like:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 1005 root 3u IPv4 5150 0t0 TCP *:ssh (LISTEN)
sshd 1005 root 4u IPv6 5152 0t0 TCP *:ssh (LISTEN)
cupsd 1063 root 5u IPv6 5318 0t0 TCP localhost:ipp (LISTEN)
cupsd 1063 root 6u IPv4 5319 0t0 TCP localhost:ipp (LISTEN)
sshd 10340 root 3r IPv4 18747 0t0 TCP pc4.pacnog.bluesky.as:\
ssh->noc.pacnog.bluesky.as:34634 (ESTABLISHED)Again, Google and man to figure out what is going on:
$ man sshd
$ man cupsdWhat's cupsd? is this necessary on every server?
Notice that sshd is listening to all incoming connection requests (the "*"). This is a typical, potential security hole.
In our case, we will leave ssh up, but we are aware they are running and need to be patched for security updates as they come out.
For example, it is a good idea to lock down sshd a bit by not allowing the root user to log in with a passwords.
As you are not printing, let's turn off the cups printing service. Do you remember how to do this?
# ls /etc/init.d <-- to find the service script name
# service cups stop
# lsof -iNow we only see:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 1005 root 3u IPv4 5150 0t0 TCP *:ssh (LISTEN)
sshd 1005 root 4u IPv6 5152 0t0 TCP *:ssh (LISTEN)
sshd 10340 root 3r IPv4 18747 0t0 TCP pc4.pacnog.bluesky.as:\
ssh->noc.pacnog.bluesky.as:34634 (ESTABLISHED)To prevent this service to start when the machine is rebooted, type:
# echo "manual" > /etc/init/cups.overrideIt's usually a good idea to see how your machine looks to other users.
Log in to a PC different than yours. For example:
$ ssh sysadm@pcXMake sure that nmap is installed by doing:
# apt-get install -y nmapNow let's scan your machine using the nmap command:
# nmap -sV pcX [Where "pcX" is _your_ pc]You should see something like:
Starting Nmap 5.00 ( http://nmap.org ) at 2010-06-30 09:59 SST
Interesting ports on pc2.pacnog.bluesky.as (67.218.55.102):
Not shown: 998 closed ports
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 5.1p1 Debian 6ubuntu2 (protocol 2.0)
25/tcp open smtp Postfix smtpd
MAC Address: 00:0F:1F:E6:62:94 (WW Pcba Test)
Service Info: Host: pc2.pacnog.bluesky.as; OS: Linux
Service detection performed. Please report any incorrect results at \
http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 1.76 secondsThis looks reasonable. The machine is exposing smtp and ssh to the world as well as the type of OS that it is running.
Now let's scan a bit more aggressively:
# nmap -A -T4 pcXTake a look at the information presented. This will take some time, but it will contain more detail.
Now, remember to log out of your classmate's PC!
$ exitYou can use nmap to scan entire networks and find all the machines and services that are running. This is what network attack scripts do - but, they usually scan for a specific port and service, then they launch an attack when they find a service that they think they can break.
Be careful with nmap! If you scan aggressively or against an entire network you will likely set off detection alarms and you could get in trouble. Let people know before you scan if you are not in charge of the remote machines.
Now read about nmap to understand what -sV, -A, -T4 and -F are doing:
$ man nmapLet's install fcheck. This is an intrusion detection tool that is very simple to set up and is preconfigured to do most of what you want:
# apt-get install fcheckChoose default actions "Internet Site" and "pcX.ws.nsrc.org"
Once this is done you can look around to see how fcheck is configured. By default Ubuntu installs and configures fcheck in a reasonable manner and you probably don't need to do anything else.
$ man fcheckConfiguration of check is in /etc/fcheck/fcheck.cfg. Let's have a look:
# editor /etc/fcheck/fcheck.cfgRead through the file to see what directories fcheck is checking, which directories are excluded, etc. The check process is run once every two hours on the 1/2 hour. You can view this by looking at:
$ less /etc/cron.d/fcheckThe text that reads:
30 */2 * * *is telling our system cron process to run the long check command listed in the file once every 2 hours on the 1/2 hour.
Now force fcheck to run for the first time and create a database:
# fcheck -acLook at the baseline file that fcheck has created:
# less /var/lib/fcheck/fcheck.dbfNow let's make a change to a file in one of the directories that fcheck is checking.
# editor /etc/hostsAdd a blank line at the end of the file. Save the file.
Now do another forced run of fcheck:
# fcheck -aYou'll see lots of stuff go by on the screen.
you see something like:
PROGRESS: validating integrity of /etc/
STATUS:
WARNING: [cv-macbook] /etc/hosts
[Sizes: 257 - 258, Times: Jul 22 21:36 2010 - Mar 14 16:10 2012]This tells you that the file /etc/hosts has changed. The cron job installed by Ubuntu will e-mail these kinds of reports to you.
Inotify in the kernel can provide real-time notification of filesystem changes. Install the incron package and configure incrond to monitor important filesystems.
# apt-get install incron
# tail /var/log/syslog
# cd /etc/incron.d
# editor globals
add the following line (one line) to the globals file:
/etc IN_MODIFY,IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/logger -p news.warn "$% $@/$#"For a description of the syntax of incron table files, see:
$ man 5 incrontabThat's it. The changes you make to incron are updated automatically. Because incron can recognize changes, it even recognizes when you change the configuration for incron, and it updates.
Now add a file to the /etc directory:
# touch /etc/dogTake a look at /var/log/syslog. What does it say???
# tail /var/log/syslogFrom now on, any changes you make in the /etc directory will generate syslog messages.
There is a meta package called unattended-upgrades to do this. To install:
# apt-get install unattended-upgradesThat's it. Any time a security update is placed in the Ubuntu repositories it will be automatically installed on your system. You will probably want to look at how unattended-upgrades is configured.
# cd /etc/apt/apt.conf.dThis package is configured in the file 50unattended-upgrades. Let's have a look and we will make a change to the configuration:
# editor 50unattended-upgradesNote at the very top of the file. If you were to change this:
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}-security";
// "${distro_id}:${distro_codename}-updates";
// "${distro_id}:${distro_codename}-proposed";
// "${distro_id}:${distro_codename}-backports";
};
To look like:
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}-security";
"${distro_id}:${distro_codename}-updates";
// "${distro_id}:${distro_codename}-proposed";
// "${distro_id}:${distro_codename}-backports";
};then all software package updates would be installed as well. You may, or may not, want to do this. This is generally safer for user desktops than for servers.
Let's change this line:
//Unattended-Upgrade::Mail "root";To be:
Unattended-Upgrade::Mail "root@localhost";That way your root account will get an email when an update is installed.
Note that you can even have your machine automatically reboot if required after an update by editing the following stanza.
// Automatically reboot *WITHOUT CONFIRMATION*
// if the file /var/run/reboot-required is found after the upgrade
//Unattended-Upgrade::Automatic-Reboot "false";
Save the file and exit.
That's it. If a security update is applied you will be notified.
There is a nice tool called "chkrootkit" - This is used to see if a machine has been compromised with known software kits that install once security has been breached. You can read about this software here: http://www.chkrootkit.org/
To install, do this:
# apt-get install chkrootkitTo use it, do:
# chkrootkit"chrootkit" will find one file it thinks is infected:
Searching for Suckit rootkit... Warning: /sbin/init INFECTEDThis file probably isn't infected, but tests positive due to Ubuntu's configuration.
You should not see anything found or infected (hopefully!). However, it's possible for the tool to give you some false positives. You can go back to the http://www.chkrootkit.org/ web site for more information in the README and FAQ pages and you should use Google. If you don't see other people reporting false positivies like yours, then you probably need to format your hard drive, reinstall and restore data from backups.
Let's do something to make chkrootkit give you a warning:
Place your ethernet interfaces in to promiscuous mode (i.e. it listens for all packets on the network, not just packets coming to your machine).
# ifconfig lo promiscNow let's re-run chkrootkit:
# chkrootkitand you will see that it detects that the loopback network interface (lo) is now in promiscuous mode. To just see this vs. all the other messages do:
# chkrootkit | grep PROMISCIf your PC is running a DHCP client daemon, you may also see that eth0 is in promiscuous mode:
eth0: PROMISC PACKET SNIFFER(/sbin/dhclient3[564])Turn off promiscuous mode for lo:
# ifconfig lo -promisc"rkhunter" is another rootkit checker like "chkrootkit" - This is used to see if a machine has been compromised with known software kits that install once security has been breached. You can read about this software here: http://rkhunter.sourceforge.net/
To install, do this:
# apt-get install rkhunterTo use it, do:
# rkhunter -cOn Ubuntu rkhunter might find a suspicious file "/usr/bin/unhide.rb". It also may find suspicious or hidden files or directories in the filesystem check. This is because Ubuntu is designed differently than many other distributions of Linux
System accounting gives us logs of all the commands that have run and terminated on the system. Let's see if we have the acct package:
$ which saDid "which" find the command? If not install the package:
# apt-get install acct
$ which saLet's run a command and see if acct records it.
$ whoami
# sa -uDid "sa" show a record for the command?
Let's try the "lastcomm" command as well:
# lastcomm root