Securing Network Services and Infrastructure
Return to NSRC Help Desk home.
Network Host Security
- Staying Informed
- Critical Steps to Securing a Network Server
- Physical Security
- Secure Passwords
- Updating and Patching Software
- System Integrity Checking and Intrusion Detection Systems
- System Logging
- Encryption and PGP
Network Infrastructure Security
This document presents some tips and suggestions for system administrators and network managers who are responsible for securing their networks. The document begins with an outline and basic steps for those new to server and network security. Each section below goes into more detail and includes links to configuration examples, software, and other helpful informations sources.
This document is largely aimed at the new or intermediate Unix/Linux system administrator who is looking to secure a server, or servers, on their network. Some discussion of general network security issues is included as well in the "Network Infrastructure" section of this document.
If you have feedback on the information you see here, or additional suggestions for content, please contact us at firstname.lastname@example.org.
Because security vulnerabilities are constantly being revealed, staying informed about recent security issues is crucial. The 'Secunia Advisories' mailing list and RSS feed are an excellent resource for up-to-date security advisories.
You can sign up for the Secunia Advisories mailing list at : http://secunia.com/advisories/mailing_lists/
Critical Steps to Securing a Network Server
If you are new to security and are considering installing a Unix-based or Linux server there are a few things that you should consider before installation. If you have already installed and setup your server and it is on a publicly accessible network, then we strongly recommend that you take these steps to begin securing your box immediately. The assumption made here is that you have not taken steps to secure your server(s). If you have, then you may wish to use this list to review what you have done.
Ideally, you would install any network server box and secure it before connecting the box to the network. The reason for this is because many default installations of Linux or Unix have known security holes. If you install a network server and place it on a publicly accessible network for any time, then it may be compromised before you manage to secure the box. In addition, if the box is compromised, and then you install System Integrity software you may not notice your system is compromised, which defeats the purpose of using this type of software.
During your initial network server installation you can do this with your machine not connected to the network, or just connected to an internal, and secure network. You can connect your server as needed to update your software for security purposes, and then disconnect once updates are downloaded. Additionally, you can consider downloading any software updates you will need to another network server, and then copy this software to your new network server to avoid placing it on the network before you feel it is properly secured.
Before we get started with securing your server's operating system, and installed software, it's important that we mention physical security. You can secure your server fully and completely, but if someone gains physical access to the box, then it's likely there is little you can do to protect it. In extreme cases there are some things you could do, such as encrypting data on the drive, but for the most part it is trivial to gain root (administrator) access to a server box once it has been physically compromised.
Consider the people who have physical access to your server or servers. Can you trust all of them? If not, perhaps you should consider placing your server(s) in a locked room of some type where only trusted individuals have physical access to the box.
If physical security still presents an issue, encourage users to lock their terminal when leaving a computer unattended.
Choosing a strong password is crucial in maintaining the security of networked host. A password that is 'strong' is a password that is less prone to be guessed. The strength of a password increases with the number of characters in the password, the randomness of the password's characters and the variety of characters in the password.
It is fine to tell your users that their passwords must follow a rule set like this:
- New passwords must be 6 to 14 characters in length.
- You cannot choose any word that is in a dictionary, or any names, places, or personal information like your birth date.
- A secure password should contain at least two numbers and two letters and it's a good idea to mix upper and lower case letters.
The open source security project 'openwall' offers a PAM (Pluggable Authentication Module) called "pam_passwdqc" with the ability of enforcing a password policy on your host. This module is available at:
Services are the heart and soul of a network server machine. Services represent the permanently running programs that listen for requests and respond to them. Understanding what you want to do with your machine will directly lead to deciding what services you need to run. Once you make this decision, then you should do the following:
- Disable all unnecessary network services.
- Update or patch the program to be run to fix potential and known security holes.
- Restrict access to running services to only those who need or should have access.
- Change the service's listening port to a non-standard port number (Optional).
This philosophy is absolutely essential to running a secure network server. If you install your network server, run a bunch of services by default, don't restrict access, and don't update or patch the programs running then it is guaranteed that your machine will be compromised quickly. To see just how quickly check out the HoneyNet Project at http://www.honeynet.org/papers/stats/, and in particular their Statistics White Paper from July, 2001.
Securing Common Services
Below are links to sites explaining how to secure a few common Unix/Linux services:
- SSH: http://ubuntu-tutorials.com/2007/02/14/what-you-ought-to-know-about-securing-ssh/
- Apache2: http://httpd.apache.org/docs/trunk/misc/security_tips.html
- MySQL: http://dev.mysql.com/doc/mysql-security-excerpt/5.1/en/index.html
What Services to Run
On most Linux/Unix systems you can look in the directory '/etc/rc.d/init.d/' or '/etc/init.d' to see the startup script files that are called as your operating system starts. In Red Hat Linux derivitives like Fedora and CentOS the command 'chkconfig' allows for the easy listing, enabling and disabling of services.
A tutorial on how to use 'chkconfig' is located at http://www.linuxjournal.com/article/4445
The man page for 'chkconfig' is located at http://linuxcommand.org/man_pages/chkconfig8.html
Debian Linux derivatives such as Ubuntu offer a tool called 'update-rc.d' that offers functionality similar to 'chkconfig'. To disable a service from starting at all runlevels run the following command, where service is the name of the service you wish to disable:
- update-rc.d service remove
The man page for 'update-rc.d' is located at: http://xgen.iit.edu/cgi-bin/man/man2html?update-rc.d+8
If your operating system does not include the 'chkconfig' or 'sysvconfig' utility, changing which scripts start up at boot time is a bit more tedious. The 'Initscripts' section of the gentoo linux user manual explains how the linux boot process works, this document will help you change which scripts start at boot time. This document is located at: http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4
'Iptables' is a common service used to 'firewall' a Linux host. Actually, a more accurate description of iptables function is as a 'packet filter'; iptables essentially filters packets destined or originating from your host. This is absolutely essential to protecting your system. For instance, you may have certain services that you only want accessed by people on or from your local network—using network filters like 'iptables' you can do this rather easily.
As a general rule of thumb you will enable a service such as iptables in run-levels 2, 3, 4, and 5. The iptables services will start from a script located somewhere like /etc/rc.d/init.d/iptables, and this script may refer to a rule set that you define under /etc/sysconfig/iptables. There are other ways to do this, but this is one typical example. Or, if you use something like pfilter (mentioned above), simply add it in under /etc/rc.d/init.d/ and then use something like chkconfig to make sure the service starts each time you boot your server.
Most modern Linux distributions come bundled with iptables; some distributions like CentOS, offer a GUI front-end to iptables. To see if your host is already running iptables type "iptables –v" at a command prompt, if your host is running iptables the version of iptables you are running will be listed.
- An excellent iptables tutorial is located at: http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch14_:_Linux_Firewalls_Using_iptables
- An iptables reference guide is located at: http://iptables-tutorial.frozentux.net/iptables-tutorial.html
- An example iptables script is located at: http://nsrc.org/security/pfilter
Updating and Patching Software
All Linux and Unix distributions are "out-of-date" by the time you install them. What this means is that it is very likely that there are already updates and security patches for many of the services included in base installation of the operating system. It is extremely important to keep your operating system up to date to mitigate the security risks to your host.
Most modern Linux/Unix distributions come bundled with package management software. Package managers are a collection of software tools used to ease the installation, maintenance and of software packages. It is highly recommended that you use a package management system to manage the software on your host because it makes it far easier to keep your system up-to-date and secure.
- Debian Linux dervitives such as Ubuntu use the 'apt' package manager, instructions on how to use 'apt' are located at: http://www.debian.org/doc/manuals/apt-howto/
- Red Hat Linux derivitives such as Fedora and CentOS use the 'yum' and 'rpm' package managers, instructions on how to use 'yum' and 'rpm' are located at: http://www.fedorafaq.org/#installsoftware
System Integrity Checking and Intrusion Detection Systems
Your first question may be, "What is System Integrity Checking?" The basic concept is that once your system is configured, patched, and secured, then how do you know if something changes that you don't expect? Answering this question is actually harder than you might think, and it's at the heart of system integrity checking. In a nutshell, a system integrity checker examines how your filesystem and network services behave. If your computer operates outside the systems administrator's prescribed boundaries the system integrity checker notifies the administrator, or in drastic circumstances, halts the system.
One very simple method to do some basic checking is to generate md5 checksums on each and every file on your system. You can then run a script (using cron, for example) that compares expected md5sums with newly generated md5sums for all files on your system.
Immediately you should see some problems associated with this:
- How do you keep track of new files, and automatically generate an md5sum for these files?
- You have to decide what constitute valid files. That is, you probably don't want to generate md5sums for the /proc directory.
- What about files that are erased? You should have a way of knowing that the file no longer exists.
- If your system has many users, then you may not want to generate md5sums on their files as they are in a state of constant flux.
- How do you interpret the results of rescanning your system? You may need to look at each report to understand what's going on.
Programs like 'Samhain' and 'Osiris' address the issues of system integrity and intrusion protection.
- 'Samhain' is an open source 'host-based intrusion detection system'. It can be downloaded at: http://www.la-samhna.de/samhain/
- 'Osiris' is an open source "Host Integrity monitor". It can be downloaded at: http://osiris.shmoo.com/
- An excellent article on "Best Practices" in Host Integrity Monitoring is located at: http://www.securityfocus.com/infocus/1771
Backing up your server once everything is installed and setup just the way you want it is very important.
You might be asking why we include backup as a topic under security. If your server is compromised, then you will need a good set of backups to recover from the compromise. If you know when the compromise took place, then it may be possible to restore your server and it's data relatively quickly. If you are unsure of when your server was compromised, but you know what the compromise was, then you could decide to rebuild your server, but restore user data. Naturally you are running the risk that you have a compromised user on your system with tools to break in to your system from the inside. Restoring user data in this case is a decision you would have to take. Finally, a system with a backup, including your configuration files, can be much more quickly rebuilt if you have your old configuration files to refer to. You may wish to take a snapshot of your system once you have it running just as you want it for the first time in case you ever need to refer to your original configuration files to rebuild your server.
Once you make the decision to put a good backup system in place, then your first set of decisions concerning backups include the following:
- What do you want to backup?
- What do you need to backup?
- How often must you backup?
- User data? System configuration files? Operating system files?
- What is the backup rotation? Daily, weekly, monthly, semi-annually, yearly?
- What type of backup media are you going to use?
- Will you use the same media and software for each piece of your backup process?
- Where will you backup your data?
- Where will you keep copies of your backups?
- Have you tested your backups? I.E. have you tried a restore?
- What will you do if you lose your server? Do you have a place to restore your data in this case?
This is quite a few questions, but they are really important. If you don't believe us, wait until you lose data or a server, and see what happens if these questions have not been answered ahead of time!
The first thing to remember is to never backup your data to the same partition, or same disk. This should be obvious, but we state it just in case... Ideally you should back up your data to a different physical device than your server. At the very least, you should back up your data to media that can be removed from your server. Remember, if you back up to CD, Tape, DVD, another disk drive that can be removed, etc., and your server is completely lost (i.e. - it burns up, is destroyed, etc.), then you will need a device that can read that media in order to restore your files to a new server! This fairly obvious, but a key point that is sometimes forgotten.
Now you should decide what gets backed up and how. You could just do an entire disk image backup and always be able to restore your entire machine. This might work if you are lucky enough to have something large enough to fit your entire disk, but more likely than not, you will only want to back up what's necessary, when it's necessary. In this case you can consider backing up your system files and configuration files as necessary, maybe weekly. Now you need to backup your user's data and your constantly changing system data. User data and log files should be backed up daily. In practical terms you'll want to consider something like:
- Everything not in daily backups, and /proc, /tmp, /scratch
- /var/spool (includes user mail files on many systems)
This is still not complete because you need to think about how long you keep your backups. That is, do you keep the weekly backups (where you should do both daily and weekly backups) for a month? Then do monthly backups and keep them for a year? Then do semi-annual backups and keep them permanently?
The suggestion above is just that, a suggestion. Almost certainly you will do something different, but it's just to give you an idea of how to approach backing up your system. In addition, you may noted that some policies are implied by your backup routines. For instance, if you do not back up files in /tmp or /scratch, then users who have work files in these locations will lose them. This is generally considered to be acceptable practice as anyone using a directory such as /scratch should be able to copy the files they need to their own directory for safekeeping. But, on some systems, particularly heavy research servers, this may not be acceptable. You, as a system administrator, will have to set such policies based on how your server is used.
You may find that you can back up your system files only when they change if your server is fairly static. But, if you apply patches on a regularl basis, particularly for security reasons, then you will want to back up your system files fairly regularly as well.
- 'A.M.A.N.D.A' is an open source tool that can manage the back ups of numerous Linux and Windows hosts. A.M.A.N.D.A is available at: http://www.amanda.org/
What you decide to log, where you decide to log information, and how you decide to do this can go a long ways toward securing your system. Not only can logging help you to detect security breaches, but it can help you to verify that software is running as expected, detect hardware problems, and get a general feel for how your box and it's users work.
Most modern Linux distributions configure logging by default, if you are using Ubuntu, your syslog configuration file is located in '/etc/syslog.conf'.
In your syslog configuration file you will have an option to configure remote logging. Configuring remote logging on production systems is HIGHLY recommended. Remote logging is useful both in the event of an intrusion and in the event of a disk failure; your log files MUST be kept safe! See the man page for your syslog daemon for more details on how to configure remote logging.
- Foremost: Foremost is a console program to recover files based on their headers, footers, and internal data structures. This process is commonly referred to as data carving. Foremost can work on image files, such as those generated by dd, Safeback, Encase, etc, or directly on a drive. The headers and footers can be specified by a configuration file or you can use command line switches to specify built-in file types. These built-in types look at the data structures of a given file format allowing for a more reliable and faster recovery.
- LSOF - LiSt Open Files: Lsof is a Unix-specific diagnostic tool. Its name stands for LiSt Open Files, and it does just that. It lists information about any files that are open by processes currently running on the system. It can also list communications open by each process.
- chkrootkit: chkrootkit is a tool to locally check for signs of a rootkit.
- ModSecurity: ModSecurity is an open source web application firewall that runs as an Apache module.
- Wireshark (formerly Ethereal): Wireshark is a program that captures network traffic on a given network interface.
- Kismet: A powerful wireless packet sniffer
- tcpreplay: tcprelay is a a program that gives you the ability to use previously captured traffic in libpcap format to test a variety of network devices. It allows you to classify traffic as client or server, rewrite Layer 2, 3 and 4 headers and finally replay the traffic back onto the network and through other devices such as switches, routers, firewalls, NIDS and IPS's.
- nmap: nmap is an extremely powerful port scanner for the *nix platform.
Encryption with PGP
PGP is an implementation of public key cryptography that is often used to sign and encrypt e-mail messages and files. To use PGP, you must generate a "key-pair"; a PGP "key-pair" consists of both a "private key" and a "public key". A private key is not to be shared with ANYONE, and is to be stored in a place that only you can access it like your home directory. A "public key" is to be distributed to anyone with whom you would like to communicate with over an encrypted channel.
Generating a PGP Key
To generate a PGP key-pair using GnuPG enter the following command:
- gpg --gen-key
After you enter the command you will be prompted with a series of questions about what kind of key-pair you wish to create. The GnuPG documentation recommends a 'DSA/ ElGamal' key because it is so widely used.
Next, the tool will ask how long you wish the key to be. It is recommended that you create a key of 2048 bits. Creating a key of this length takes a bit more time, but is far less vulnerable to cracking.
The last step in creating a PGP key-pair is setting a password. When choosing a password for your PGP key-pair be sure to make it secure. Keep in mind that PGP passwords allow spaces to be used, in that way, a PGP password can be a long sentence if you would like. See the creating secure password portion of this document for more information on how to create a secure password.
Uploading Your Public key to a Keyserver
Once you have created your PGP key-pair, it is important to distribute your public key so that people will be able to send you encrypted and signed messages.
First you must locate your public key. Your public key is often located in your home directory in the folder '.gnupg'.
Next you must view your public key with your favorite text editor, and copy the contents of the file to your clipboard.
Next go to http//pgp.mit.edu scoll down to the heading "Submit a Key", paste your key in the provided textbox, and click submit.
Configuring your E-mail Client for PGP
If you use Mozilla Thunderbird for your e-mail client, an extension called 'Enigmail' is used to encrypt, decrypt and sign messages with GnuPG.
You may download Enigmail from: http://enigmail.mozdev.org/
Securing Routers and Switches
If you look at the two "Border router config" files listed below you willl notice they both block incoming udp requests on ports 137-139, 445, as well as incoming tcp requests on ports 137, 139, and 445. Ports 137-139 have to do with NetBIOS access, while port 445 is used by Windows 2000 for SMB over TCP; see http://ntsecurity.nu/papers/port445 for a discussion of this. Windows client boxes are almost certainly not secured and extremely easy to compromise if you allow outside users access to your internal network ports 137-139, and port 445. There are many other ports that you may or may not want to block. This generally tends to be an internal organizational discussion about the trade offs in security vs. the inconvenience of not allowing outside access to some of the services on your internal network servers. This discussion is different for most every organization. Many groups will decide to block the RPC (Sun Remote Procedure Calls) tcp and udp port 111, as well as the LPR tcp port 515. Note that Samba (the Linux/Unix NetBIOS service) uses port 111.
Another option on your part is to create what is called a DMZ (Demilitarized Zone). One way to do this is to simply block all incoming services at your border routers. While this may sound reasonable, it very rarely works. Generally the larger your organization, the harder it will be to enforce such rules. For instance, eventually someone (perhaps an important administrator) will insist on external access to some server. You might decide to give them access via SSH, so that at least they are using a secure protocol when accessing your network. While this is a good idea, users can now use SSH to tunnel other services in and out of your network, and, SSH has had various vulnerabilities exposed over the years. If you forget to patch SSH, or if you have many servers running SSH, then it's entirely possible that one of them may eventually run an insecure version of the protocol. And, this is just one service! Imagine what happens if several "holes" are opened in your DMZ. At this point you might as well concentrate on securing your individual servers on your internal network.
Now that you may be thinking it's time to look at securing your individual servers again, then you will want to understand and implement firewalls well. Please refer to the Firewalls and Protecting Services section of this document for a complete discussion, examples, and references.
Cisco has provided some very good tools to learn about security as it pertains to routing, particularly from the perspective of an ISP. You can find all of this at ftp://ftp-eng.cisco.com/cons/isp/security/ (mostly in PDF format).
Here a few sample items included at this location:
Cisco Security for ISPs: Sample Items
- ACL Update Procedures (PDF)
- BGP Risk Assessment (PDF)
- http://nsrc.org/security/large-net.conf: A Cisco Border router config for a large university network. ( IP addresses have been replaced with "abc.xyz")
- http://nsrc.org/security/small-net.conf: A Cisco Border router config for a small network. (IP addresses have been replaced with "abc.xyz")
- Cisco Guide to Harden Cisco IOS Devices: An in depth guide about how to secure Cisco IOS devices.
- Identifying Incidents Using Firewall and IOS Router Syslog Events: An explanation of how to use IOS and ASA syslogging to identify security incidents.
- http://www.sans.org/reading_room/whitepapers/firewalls/794.php: An article from SANS about hardening Cisco Routers.
- http://www.cs.cmu.edu/~dwendlan/routing/: A collection of BGP and routing security resources
- Cisco Security Response: Rootkits on Cisco IOS devices: A discussion of the implications of rootkits on Cisco IOS devices. This document presents a "Best Practices Guide" for securing your Cisco IOS devices.
- TechRepublic.com: Lock Down Cisco switch port security An article from TechRepublic about switch port security and how to enable it on Cisco Catalyst switches.
There are, naturally, several issues surrounding security and wireless networks. The major issue is that most the methods built in to the 802.11a/b networking protocols for security can be broken by a determined hacker. The methods generally available for securing your clients with Access Points are WEP 40/64 bit and 128 bit, required SSID's (Service Set Identifier), MAC (Media Access Control) access authentication, and passwords (if supported by your client). The issue with these methods are the following:
- WEP can be broken by a determined hacker. Thus passwords can be discovered.
- The WEP key is stored, plain text, on each machine configured to connect to the network. Access to any of these machines will give someone the WEP key in a trivial manner.
- Clients can "spoof" MAC addresses.
- SSIDs can be found out easily
In addition, you should change the default SSIDs and passwords that come with your equipment. With that said, you should still use 128 bit WEP encryption, and, if feasible, use a MAC address filtering mechanism. Most access points have MAC address filters. This means that if a machine's MAC address (really the wireless card in that machine) is not registered with the access point, then that machine is not allowed on the network - even if the WEP key, passwords, etc. presented are correct. These steps will make it considerably more difficult for someone to break in to your wireless network - at least at a first attempt. There are users who look for open wireless networks to gain free access to the Internet. Putting basic protection in place will stop many of these types of users.
Below we will present several resources discussion how to further protect your wireless network, including the possible use of VPN's to add another layer of authentication to your wireless network. With this said, however, it's important to remember that the best protection is to ensure that your users are working with secure services from end-to-end on your wireless network. That is, if you protect someone on your wireless network from having their password guessed this may be great for your part of the network, but what if they are still using insecure POP to get their email. Once their password leaves your protected network, then it will, once again, be out in the clear. The best long-term defense is to work toward migrating all your clients to secure network use with protocols such as POP/IMAP over SSL, SSH instead of Telnet, SCP instead of FTP, and HTTPS instead of HTTP when reading email over the web.
By MAC authentication we mean that most wireless access points allow you to specify a list of Ethernet hardware addresses and IP numbers. This generally does not scale well, but if your network is small enough, then you can insist that all new members of your network first see you with their wireless card so that you can register its MAC address in your wireless access point, and assign an IP address to that MAC address. This is a 1-to-1 mapping of hardware on your network, which can be very powerful in terms of security - but, once again, can be very hard to scale to a larger number of users.
- Kismet - Kismet is a wireless network detector, sniffer and intrusion detection system.
- Aircrack-ng - Aircrack-ng is a a suite of utilities used to crack 802.11 WEP and WPA-PSK keys