Here we present an overview of the pieces you need to run an ISP and how these pieces come together to provide service to your users. We document the critical pieces for you, point to collections of useful tools, free software, and other useful information.

This document presents information in sections with links to useful resources. After the links a more detailed topic discussion may be available as well.

Information Index

Outline [Return to Top]

The NSRC has been helping ISPs around the world for many years. Over time we have seen many tools, tactics, and different methodologies for putting together ISPs. Based on our experience and expertise we present what we believe to be relevant information to help you get started and tools and methods that we have found to work. We present this information based on experience. And, based on this experience we do not, generally, recommend Microsoft solutions to build a reliable and secure ISP infrastructure. We do, however, present information about Microsoft products to allow you to place them in context. In addition, we cannot stress strongly enough the importance of planning for the future in every aspect of your ISP's development. If you are successful as an ISP, then your implementation must be able to scale with your success.

For many of these reasons we recommend the use of FreeBSD as your principal server operating system. For those of you who are new to FreeBSD or would like some more information about getting started with FreeBSD take a look at our FreeBSD Planning, Installation and Security Tips guide at:

In addition, be sure to see our discussion of server operating systems later on in this document for more information.

General Links and Pages [Return to Top]

Some of these may be a bit outdated, but they'll give you a good feel for some of the issues that you may be dealing with as you go forward:

In addition you can read about the related topics of network administration and help desk creation. See the Guide to Administrative Procedures of the Internet Infrastructure (RFC 2901) in the various formats listed below:

and the Help Desk & Support Organization Resources pages at:

Connectivity [Return to Top]

If you are in a region where network connectivity is at a premium it is likely that you will be looking for a way to get a satellite connection setup for your ISP.

Satellites and Operators

Undersea Cables and Trans-Oceanic Backbones

And, if you are interested in Hybrid Routing for your ISP (Satellite link for one route and trans-oceanic cables for the other), then you should see this excellent whitepaper on this topic:

Getting Address Space [Return to Top]

To get the network addresses that you'll need to get started we recommend that you read the RFC 2901, the Guide to Administrative Procedures of the Internet Infrastructure that is referenced above.

And, as you go forward trying to get the address space each of the four organizations that deal with IP address registration and administration have pages discussing how to request IP address space for new and continuing ISPs. Note that AfriNIC, the African IP address Registry, is undergoing formal recognition and transition to become the RIR (Regional Internet Registry) for Africa. The regions adminstered and the corresponding organizations pages discussing initial IP allocation are listed below:

AfriNIC
African Network Information Center (not yet a formal RIR)

APNIC
Asia Pacific Network Information Centre (62 economies)

ARIN
North America and sub-Saharan Africa:

LACNIC
Latin America and the Caribbean:

RIPE NCC
Pan-European


Setting up Routing [Return to Top]

As this is such a large topic we strongly suggest that you go over in detail the excellent resources presented at the AfNOG Workshop web site for the Scalable Network Infrastructure track at:
In addition, we present the "Links and References" section to this track below as well as some additional links:

IP Numbers and AS Numbers

Understand how you request a range of IP addresses for your network and how data from your network will be routed to the rest of the Internet.

Operational Guidelines

Learn how to set up your network and your network hardware in a practical manner.

Cisco-Specific Documentation

If you plan on using Cisco hardware in your network, then you should read these documents closely.

Software Tools

Registries

Router Manufacturers

Core Services and Managing Them Locally [Return to Top]

As an ISP you will be responsible for maintaining several core services that your clients use to gain Internet connectivity. Some of the most critical are DNS, DHP, and probably dialup, often via a product called RADIUS. Each of these is discussed below with pointers to information about setting them up and using them.

DNS (Domain Name Server)

It's important that you set this up correctly from a functional standpoint. For instance, you run your primary DNS server locally, but ideally you should have your secondary, or backup, DNS server on a different network. This way if your other network functions are up, but your DNS box is down your clients can still make use of their Internet connections.

Below we present some first-level discussion of DNS, then we have more comprehensive list of Registries and DNS setup and maintenance links, including mulitple RFCs and their descriptions.

The Domain Internet Groper (DIG) comes as part of the Bind (http://www.isc.org/products/BIND/) distribution. DIG (the command "dig" is lowercase when you use it) allows you to find domain information as needed. This tool is used in place of nslookup (http://www.linux.com/develop/man/8/nslookup/). Here are some examples of how to use this tool:

DHCP (Dynamic Host Control Protocol)

DHCP is not all that difficult to setup, but in a large environment it is important to decide upon the rules of engagement. If you have a fixed number of IP addresses then how you allocate them, for how long, and who gets static IP addresses becomes important. As you implement your DHCP services consider the following issues:

  • What devices get static IP addresses. Generally this includes things like printers, routers, servers, and anything else that must have the same IP address everytime it is turned on.
  • How long are your DHCP leases? That is, how long until an IP address is recycled back in to your IP address pool? For dialup connections this is usually as soon as the client disconnects.
  • Build a dhcpd.conf file that is well documented and laid out by located (networks) so that it is easy to update and maintain.

    One of the best DHCP resources is the man pages for dhcp. Simply type "man dhcp" at the operating system prompt under any UNIX system to get started. The current version of the DHCP server for UNIX can be found at:

And, information about DHCP is available from these sites:

Dialup (RADIUS)

Setting up a process by which you authenticate users dialing in to your network (modem pool) is likely to be a critical piece of your operation as an ISP. In addition this is one point at which you can monitor user access to your services and keep track of usage for billing or statistical purposes. Below is a short list of some excellent Radius resources:

Client Services [Return to Top]

These are services that you are very likely to run that allow your users (your clients) to make use of the Internet. These are all TCP/IP based. While some of these services are protocols in themselves, such as SMTP, which is part of your MTA (Mail Transfer Agent), others are programs that take advantage of these protocols, such as Majordomo, which lets you use mail as a discussion list. Each and every one of these services has considerable configuration options, security considerations, and should be studied in some detail by yourself or someone on your staff to understand how to implement them well, particularly on a large scale.

For all the services listed below one of the best resources available to you for finding clients to use these services is "Tucows" at http://www.tucows.com/. After the descriptions below there is a table listing well-known or recommended clients for accessing these services from Windows, UNIX/Linux variants, and Macintosh users.

If you are interested, we have a page discussing setting up web pages, and accessing these services from the point of view of the client. This is located at the NGO/Nonprofits Web Resources discussion [english | spanish].

Web

There are many web server choices available. Depending on if you use a UNIX variant or Windows your two most likely products that you'll use include Apache or Internet Information Server (IIS) from Microsoft. Both these and others are discussed below:

Apache

At the time of this writing (July 2001), the Apache web server running under UNIX (FreeBSD, Solaris, Linux, etc.) accounted for over 62% of all web servers on the Internet. The Apache web server is well understood, has many free and commercial modules that can be added, supports ssl, java, python, has hooks in to various databases such as MySQL and PostgreSQL, and scales to intermediate and fairly large scale implementations. Apache runs under Windows as well, but is not considered of the same quality when run under Windows.

Apache runs commonly as version 1.3 or the newer version 2.x product. You should go to http://httpd.apache.org/ and determine which version of Apache you wish to run. Both are acceptable, but by Fall 2001 version 2.x should be well enough tested to be considered as reliable as the 1.3 version of the product.

If you wish to use Apache with SSL for secure connections here are some useful links:

Internet Information Server (IIS)

Version 4 of IIS runs on Windows NT 4 and Version 5 runs on Windows 2000 servers. It is currently unclear what version will run on the upcoming Windows XP operating system. IIS is the second most popular web server on the Internet. There are free or inexpensive options available for IIS users to integrate database functionality in to your web sites, but if you need something more reliable or robust you will pay considerable licensing fees to Microsoft or other companies (thousands of dollars US). In general IIS can make setting up a complex web site easier, but in the end your site will not be as stable or secure as one setup using tools under UNIX and Apache.

Both products have security vulnerabilites (as do all services on the Internet) so you should stay on top of security alerts and patch levels at all times if you rely on web services as an integral part of your operations.

You can go to http://www.microsoft.com/iis/ to be directed to the current web services pages at Microsoft. Note, it is very difficult to get straight information out of Microsoft's pages as to what actual product you need, where it is, and what it costs. The bottom line is that IIS is available for free to any licensed user of Microsoft Windows NT Server or Windows 2000 Server products. You can run IIS with the workstation versions of these products, but the web server will be artificially constrained and will not function as you need. This is all marketing on the part of Microsoft as there is no reason why their web server component of IIS should run constrained on their workstation version of these products.

Proxy and Cache Servers

World Wide Web proxy servers speed up the access to web based resources by locally storing copies of frequently accessed web sites. Instead of having to reconnect to a slow webserver, after the first view, the web page will be served from the local proxy cache. This results in much faster page views than reloading the page from the origional server. The more disk space on your proxy, the better chance you will have of improving the performance of your client's web browsing as well as reducing the total bandwitdth used for your incoming network connection(s). The most widely used, and freely available, proxy server for Linux and FreeBSD is squid. You can find this at http://www.squid-cache.org/. In addition, here are some more useful links:

Web-Based Search Engines

How you seach the web for information can greatly affect your experience and effectiveness when using the web. There are many, many search engines currently available, including some popular ones listed below:

In addition, learning how to search is probably as important, or even more so, then where you search. Here are some articles that can help:

Finally, you can create your own search engine for your site. There are many, many site search engine products available. Two of the more popular ones include the freeware solution "ht://Dig" from http://www.htdig.org/ and the Altavista localized software search solutions from http://solutions.altavista.com/.

Telnet/Ssh/Shell Access

Any version of FreeBSD/UNIX/Linux will come with a telnet daemon that you can use and configure. Some distributions of these products will not ship with a version of ssh depending on the country you are in, local laws, or version of the product. If ssh is not available to you in your current UNIX product you can always download it from http://www.openssh.org/. There are artificial telnet daemons that run under Windows that will give you a DOS-like command prompt access to the box you connect to. This can be interested if you connect as Administrator, but this is not recommended as it is very insecure and not a standard way of interacting with your Windows box.

Telnet

To be blunt, telnet is a bad idea and if you are starting a new ISP you should avoid it from the start. telnet sends username and password information across the wire in plain text format. In addition, root can login via telnet as well. We recommend that you do not use telnet to gain access to any of your boxes or allow any of your users access to a shell on your boxes via telnet. Instead you should use ssh from the beginning.

Ssh

There are currently two flavors of ssh, these are ssh1 and ssh2. We recommend that you use ssh2. In general the client connecting to your ssh server will either "speak" ssh1 or ssh2. If you use Openssh from http://www.openssh.org/ this can support both. ssh has the advantage that username and password information is sent across the line encrypted. It is non-trivial to break this encryption. There is one weakness to ssh that we will describe below.

On the client side there are numerous ssh clients available for Windows and UNIX based environments. If your users are using the Macintosh OS before Mac OS X, then their options are quite limited.

The major vulnerability for ssh users is the "man in the middle" style attack. If you connect to a machine using an ssh client for the first time and you do not have that machine's ssh key locally recorded, then you will be asked to "trust" the machine you are connecting to the first time. That machine's key will be written to your local ssh client's host file. If the key ever changes in the future you will receive a warning when you next attempt to connect. This can be an indication that someone has taken over the machine name you are attempting to use. Even if they do not have username and password information for that machine they can record attempts, then at their leisure they can attempt to decrypt username and password combinations to break in to the real machine at a later time. This requires considerable effort, but it is something to be aware of.

Ftp

The first thing to mention about ftp is that by default this is an insecure protocol. Passwords are sent as plain text between your user sessions and any ftp server you are likely to run. Even if you insist that users connect to your servers using ssh, if their ftp and shell login passwords are the same (which is the default), then someone snooping around can gain access to your system via ssh by sniffing ftp user sessions for passwords.

With that said here are some ftp servers you can run under UNIX and Windows and some comments about them:

UNIX (FreeBSD, Linux, Solaris)

Most distributions come with either (or both) WuFTP and ftp Pro. WuFTP has had several security issues over the past few years, so you should be sure to install the latest version of this product if you decide to use it. In both cases, be sure that ftp is not started by default in the distribution that you install. In addition, double-check that root is not allowed ftp access to your server (this is usually the default action). You can find WuFTP at:

and ftp Pro, which we recommend at this time, at:

In both cases, however, we recommend that you try to avoid ftp completely and use products like Secure ftp, or Secure CoPy (scp) built in to the Openssh product.

Windows (NT and 2000)

Windows NT and 2000 both have Microsoft's Internet Information Server (IIS) available for services such as ftp, web, news and limited email. ftp services under IIS work fairly well, but there are a few known issues, particularly if you wish to interact with UNIX machines. These include:

  • IIS version 3.x under Windows NT displays files in DOS listing format by default. This can cause problems for numerous clients. You should set the listing style to UNIX format in the IIS options.
  • IIS version 4 and 5 do not automatically default to binary file transfers. This can cause problems for clients and programs that assume binary file transfer mode by default and do not set this explicitly.

    Additionally under Windows there are many other ftp servers available. An excellent resource for finding a comprehensive list of such servers is the Tucows software list (http://www.tucows.com/), which we've mentioned previously.

    Two recommended and very popular ftp servers for Windows include FTP Serv-U (http://www.serv-u.com/), which is a commercial product for more than one user access, and WarFTP (http://www.warftp.org/), which is freeware, but quite difficult to configure. Both these servers offer very good and reliable ftp server performance. Note that ftp Serv-U has the same default file transfer issues as Microsoft IIS.

Scp (Secure CoPy)

If you are running an ssh daemon on your server, then clients can copy files to and from their machines and your server using the Secure Copy (scp) protocol. This is preferable to using ftp for file transfers. In general, if you have installed Openssh on a UNIX/Linux workstation or server, this will be available and easy to use. If you are using Windows there are several scp clients available from Tucows (see above). Clients using a Mac with Mac OS before OS X, willl need to purchase a commercial ssh and Secure Copy software, such as F-Secure for the Mac. You can find the F-Secure products, which are excellent, from:

F-Secure, also, makes one of the only ssh clients for Windows that can do indirect port forwarding. Note that Tera Term Pro with the ssh addition can do port forwarding, but not in an indirect method. You can find Tera Term Pro ssh version at http://www.zip.com.au/~roca/ttssh.html.

MTA (Mail Transfer Agent)

There are threes interface involved in this discussion. These include:

In general you will run one command to transfer mail from your machine to another, or to receive incoming mail. Under UNIX the most used server to do this is sendmail. The next most used product is qmail. After this comes Exim. The NSRC recommends and uses Exim whenever possible running under FreeBSD. Exim is very secure and has been used in extremely large-scale implementations (up to a million users!).

Exim

Exim has been shown to scale to massive implementations, is generally easier to implement than sendmail, and is quite secure. You can read about the latest book, written by the auther of Exim (Philip Hazel), at http://www.oreilly.com/catalog/exim/. Otherwise, be sure to see the official web site at http://www.exim.org/.

Postfix

Another alternative to sendmail. Previously known as vmailer and as the IBM Secure Mailer when released in 1998. Postfix is preferred by administers at some sites. Read all about it, download it, and find updates at http://www.postfix.org/.

Qmail

qmail is famous for its ease of use, quick response to reader input for changes, and the $1,000 reward to the first person who could crack it - the prize went unclaimed and there is currently a $500 prize that you can read about at http://cr.yp.to/qmail/guarantee.html. You can read about qmail at http://www.qmail.org/.

Sendmail

Sendmail is the grandfather of Open Source MTA programs. O'Reilly has printed the definitive sendmail guide that you can read about at http://www.oreilly.com/catalog/sendmail2/, and you can see the Desktop reference at http://www.oreilly.com/catalog/sendmailqr/. The main sendmail site is located at http://www.sendmail.org/. One thing to be aware of is if you use sendmail there are numerous security exploits with this product. Whatever you do, do not run old versions of sendmail! There are well-known, and commonly exploited security issues. You should make it a point of keeping your sendmail versions up-to-date.

SMTP under IIS

This MTA is minimal tool included for free from Microsoft. It does not scale, is not secure, and will not work for any reasonable size ISP. See http://www.microsoft.com/iis/ for more information.

Microsoft Exchange Server

This product will work for a small ISP (500 to 1,000 users), but will cost you considerable money in licensing costs and is really aimed at corporate workgroups that are tied in to the Microsoft model using such things as the calendar features in Outlook (not Outlook Express), Microsoft Office groupwise components. See http://www.microsoft.com/exchange/ for more information. If you read independent reviews of MS Exchnage you will see that most recommend this product for corporate workgroups of less than 500 people, and not large-scale ISPs.

General Issues with MTAs

sendmail has been the de facto standard of mail delivery agents for years, but with the maturing of products such as Exim and qmail this is no longer as true.

If you need to scale your mail server there are two things that can help dramatically:

  1. Use an OS like FreeBSD. FreeBSD does not use flat file system for the password file. If you have hundreds, or thousands of users constantly checking their email, then your server must constantly verify their username and password. If the password file is not in database format, then each connection requires the server OS to search the password file from top to bottom until it finds the user and verifies the password. This can cause dramatic system response slowdowns on Linux boxes, or on any OS that uses a flat file system for the password file.
  2. If you have thousands of users, then move their mailboxes to their home directories. This way your mail delivery agent will not need to search an entire massive directory of mailboxes each time it needs to deliver mail.
Email Delivery - IMAP/POP/Webmail/Console


This is a difficult topic. Which protocol or method do you wish to use to allow your users access to their email on your site? There is no perfect answer to this question. Generally the answer you come up with will be a balance of security, reliability, and convenience.

The fundamental issue will come down to whether you use IMAP, POP, or something like webmail as your primary mail access method. The advantages and disadvantages of each method are described below. If you choose to use IMAP, then there are several IMAP servers available for use. Generally IMAP servers will include POP servers as well. You can read about some of the IMAP servers available for FreeBSD/UNIX/Linux at http://www.imap.org/. A popular IMAP server for UNIX/FreeBSD/Linux come from the University of Washington and is availble at http://www.washington.edu/imap/. And for a long list of IMAP products see:

http://www.imap.org/products/longlist.htm

Summary of Email Delivery Mechanisms and Links

IMAP: The email client that understands the IMAP protocol can see the message headers and folder names on your server, but actual email is not downloaded to the client machine until a message is opened. In addition, email stays on the server once a session is over.

Advantages
You can use any IMAP enabled client and see your email in a consistent manner from any workstation with an IMAP-enbabled client.

The end-user does not need to backup their email as it remains on a central server, which is most likely to be backed up regularly.

Disadvantages
You (the system administrator) must allocate enough space to hold email for each user. If email is an important aspect of a user's life, then you may need to consider 100MB/User, or more, of space!

The user must remain (generally) connected during an entire IMAP session. In many countries this is very expensive, or impractical.

Even though, in theory, users can check their email from any IMAP-enabled client, this is non-trivial. The user is likely to leave traces of email, password and account information, etc. on any machine where they set this up. In addition, setup is non-trivial.

If the user cannot access the IMAP server, then they do not have access to their email.

POP: With this protocol the email client downloads email from the server to the client machine. The POP protocol is smart enough to know what messages are new and in what state they are. That is, if you read them on-line, then when they are downloaded they will not be marked as new. In addition. the user can decide to leave the messages on the server, delete them, delete them after a certain period of time (POP3), or delete them only when they empty trash on their machine. The UW IMAP server comes with POP as well and can be found at:

Qualcomm makes the Eudora line of products including several POP servers for Macintosh OS, Windows and UNIX flavors. You can find these at:

Advantages
As long as the user can connect to the Internet, then they can probably download their messages using POP.

The end user can have their email saved in multiple places (server, desktop at work, laptop for the road, desktop at home).

POP allows you to "pop" (i.e., "get") email in a single, quick session, then work on the email received offline and send them at the next opportunity without needing to be constantly connected.

Many, many clients speak and understand POP.

Disadvantages
Many users become very confused as to the state of their email. The often do not know where the latest download is or how to keep their mail syncronized. POP on multiple machines is not for novice users.

Running POP from multiple locations can cause problems.

Many uses forget to ever delete email from the server, and will fill their disk space, which results in rejected email by most systems.

If the user evers gets (i.e., "pops") their email to a foreign system, then their email is on that system unless the user remembers to delete it.


Webmail: This method allows a user to connect to the Internet and get their email from within a web browser. Generally email stays on the system and it is generally much easier to deliver email in a secure fashion using webmail. One list of webmail servers can be found at http://dmoz.org/Computers/Software/Internet/Clients/Mail/Web-Based/.

Advantages
 To secure webmail you just need to run a secure webserver. This is generally easier than generating certificats to run POP or IMAP servers in secure mode.

Users can get their email from any machine with a web browser and an Internet connection. This bears repeating... Users can get their email from any machine in the world with a web browser and an Internet connection. This singlehandedly has made Microsoft's Hotmail one of the most successful email clients in the world.

Their are many webmail servers to choose from and many are free.

Disadvantages
Email remains on the server. If you don't have an Internet connection, then you don't have access to your email.

The webmail interface, filter abilities, etc. are often inferior to what standalone clients may offer.

You may have to put up with advertising or large amounts of spam (see Microsoft Hotmail in this regard) to use a webmail service.


Console-Based: This is generally any email client that runs directly on the server where your email resides, but that presents your email in a text-only format usually in a telnet or ssh terminal session window. We strongly recommend you use console-based clients with ssh connections only. A few common console-based email programs include Pine (http://www.washington.edu/pine/), which is the most popular, or Mutt (http://www.mutt.org/), which is catching on quickly, and Elm (http://www.instinct.org/elm/), which is no longer under serious development. The "mail" program is almost always available in any UNIX/Linux/FreeBSD flavor. Mail is much more rudimentary, but is functional for sending and receiving email. Note that console-based clients do not really apply to Windows servers as the concept of connecting to the server wth a shell is not well supported.

Advantages
Your email is always available and backed up on a central server.

Console-based email packages such as Pine are 100% compliant when it comes to email standards (i.e., they tend to send ASCII only!).

You can manipulate your email directly and with considerable power once you learn the console-based client well.

You can access your email from any Internet connected machine that runs telnet (most all machines, but this is not secure!) or ssh.

Disadvantages
You must connect to the server to get your email. This is not strictly true. If you understand how to do this you can download your current "Inbox" from the server to a local machine and run a console-based email client, like Pine, on your local machine. This is similiar to what you can do with POP.

If you connect to the server, then you must stay connected to send and receive email. This is expensive and impractical in some situations or countries.

If you are connected over a slow connection, then typing in telnet or ssh can be painfully slow.

Conclusion

When you pick the system that you are going to support you will have several tradeoffs between ease of use, security, reliability of the connection method, and cost. Webmail has become very popular and allows you to secure the email sessions easily. Remember, SSL connections require considerably more CPU time so you must scale your hardware appropriately. Webmail, however, requires a constant connection. POP clients are good for users who want to get their email, download it, disconnect, respond, reconnect and send. But, POP is largely insecure. There are a few POP clients now available that will talk to a POP server via SSL. We strongly recommend you implement this since POP without SSL passes a user's password and ID in clear text each time they check for mail. At this time the following clients (Windows versions) support POP over SSL:

One final issue to remember if you, or your users, run multiple protocols for accessing email is that these methods can cause problems with the state of email. For instance, some versions of POP and IMAP do not mix well. If you "touch" your mailbox using IMAP, and then attempt to access it using POP you may find that POP suddenly thinks that all your messages are new again. In addition, dependent on the servers in use, a user trying to access their email from multiple locations using multiple protocols at once can cause problems. Generally the first process accessing the email will lock out others from having write access, but this is not always guarranteed under all circumstances.

  • Eudora, version 5.1 (http://www.eudora.com/)
  • All current Microsoft Outlook clients. However, we strongly recommend against advocating Outlook as it has two major flaws:
    • 1.) By default Outlook almost always sends email with HTML formatting. This causes numerous problems, including clients that cannot read HTML-formatted email.
    • 2.) Most of the recent security issues with email clients on the Internet have targeted Outlook specifically both due to its popularity and its design flaws in terms of security.
  • Mulberry, version 2.1 (http://www.cyrusoft.com/)

Mailing Lists (Majordomo, Mhonarc)

There are many tools available for managing mailing lists, but the one tool we recommend over the rest is Majordomo. You can read about Majordomo in detail at the links below:

In addition, see the Majordomo workshop pages for the 2001 AfNOG Workshop at http://www.ws.afnog.org/afnog2001/services/mailing-lists/index.html. These include discussions about using and implementing Majordomo. Majordomo has been shown to scale to massive sizes, including lists with thousands of users and many lists of lists in a single organization running off of one machine.

One of the most important aspects of running a mailing list, or a set of mailing lists, is the etiquitte behind the job and the responsibility. The AfNOG Workshop notes discuss these issues in detail and give practical tips for dealing with them.

Additional tools and mailing list software that you may want to look at include:

News

News is a popular method for facilitating discussion on thousands of topics to anyone who has a News reader. If you decide to implement News you can create local groups, you can mirror and interact with them via the web, or you can give users access to News with Newsreader clients. In general, if you wish to carry some of the Newsgroup hierarchies, then you will need to peer with another News server that can send you the groups you wish. If you are interested in doing this you can contact the NSRC for help.

You can find the InteNet News (INN) package at http://www.isc.org/products/INN/, and the INN FAQ at http://www.eyrie.org/~eagle/faqs/inn.html. The INN package page contains many useful links and much good information.

For Newsreader clients, see the selection under News clients for the operating system of your choice at Tucows (see above discussions). A very popular choice for Windows is Free Agent from forte at http://www.forteinc.com/.

News can require significant bandwidth and lots of space depending on how much peering you do and how many groups you decide to carry on your server. As an example, the University of Oregon carries around 40,000 separate Newsgroups that require many gigabytes of space. If you wish to get an idea of what Newsgroups are available see the largest on-line web News server at http://www.deja-news.com/.

Server Operating Systems [Return to Top]

One of the most important decisions you will make is what operating system to run on the servers that support your ISP infrastructure. There are pros and cons to each OS we list here. From experience, trying to scale day-to-day operations on Windows has not proven to be very successful and costs considerably more. We recommend FreeBSD, some other UNIX flavor (say Solaris), then Linux in terms of reliability and robustness. Below we give some more details about these issues for several operating system choices.

This discussion is based on years of experience implementing large scale services solutions. This discussion is not aimed at determining the best desktop system, which interface is better, etc., but just how you can run your core services in a reliable, scalable, and affordable manner.

Quick Decision Matrix

OS Costs Advantages Disadvantages Links
FreeBSD

Base OS is free.

Several thousand free and "industrial strength" software packages are available.

 Delivers rock-solid reliable core Internet services on very large scales.

Lacks some of the commercial software choices available to Linux.

Supports less hardware than Linux or Windows.

 www.freebsd.org
Linux

Base OS is free.

Several thousand free and industrial strength software packages are available as well.

Has large-scale industry and community support.

Has considerable commercial software support as well as support for newer hardware.

Lacks indexed password store, thus does not scale well as a mail server.

Kernel 2.4.x overcomes limits of memory and filesize in 2.2 kernel, but is not as stable as FreeBSD.

 www.linux.org
Solaris

Base OS is free for Intel platform.

Base OS costs thousands on Sun platform, but is included with new hardware costs.

Sun hardware is expensive.

Delivers rock-solid reliable core Internet services on a very large scale.

Run on proprietary Sun hardware, which can improve reliability (but not always).

Costs considerable money on Sun hardware.

Commercial options available are very expensive.

Support less hardware.

 www.sun.com
Windows NT

Base OS costs approx. $700 for 5 client license.

Cost per user connection if acting as a File and Print Server approx. $28.

Cost per user connection if using Microsoft's Back Office and Exchange solutions for Core Internet Services approx. $280 for 5 users.

Has a considerable number of commercial software tools available.

Supports a moderate to decent selection of hardware.

Licensing fees are exorbitant.

Is very unstable when multiple services are run.

Can corrupt and lose data due to serious memory leaks.

Has unwarranted security holes due to design.

 www.microsoft.com
Windows 2000

Base OS costs approx. $850 for 5 client license.

Cost per user connection if acting as a File and Print Server approx. $28.

Cost per user connection if using Microsoft's Back Office and Exchange solutions for Core Internet Services approx. $280 for 5 users.

Is more stable than Windows NT due to improved modular design.

Supports more hardware.

Licensing fees are exorbitant.

Hung services are applications can still bring down the OS.

Has unwarranted security holes due to design.

 www.microsoft.com

 

Detailed Decision Discussion

There are endless discussions and sites that can prove to you that using FreeBSD is better than using Linux is better than using Windows NT or 2000 is etc. Bottom line, and in a nutshell we recommend using FreeBSD first. After FreeBSD the NSRC can help to support you using some UNIX variation (HP/UX, Solaris, etc.), and then Linux. If you choose to run Windows NT/2000 or the upcoming XP OS do not expect too much help as these are not operating systems that we can support or recommend as backbone servers for an ISP. Many of us have worked extensively with Microsoft operating systems over the years and there is a reason why we have deliberately chosen to no longer support or attempt to build organizations around them. This does not mean that we do not use Microsoft software in our daily work on our desktops and laptops, but we do not endorse it in any way as a means to support your users in a reliable and scalable fashion.

A Few Reasons Why Not to Use Microsoft Windows for Core Services

  • Microsoft software is very complex and not modular. It is full of bugs that can cause your system to crash. When the system crashes your server is down and, thus, your ISP is down.
  • By not modular please study the concept of Dynamic Link Libraries and why Microsoft has spent considerable resources trying to fix this piece of their Operating Systems.
  • In addition, study and understand how the Registry works in Windows (all versions). This is a major weak link in Operating System stability and recoverability.
  • Windows does not scale - period. Even Microsoft has to run many, many, many Windows boxes in parallel to try and replicate the same type of functionality of single UNIX-based servers for core services. For examples, search and read the history of Hotmail.
  • Windows (particularly NT) does not perform, or degrade gracefully, under load. In fact, Windows will often crash under load causing data loss. A UNIX-based server, if overloaded, may stop responding, but it will generally not crash and cause data loss.
  • Microsoft consistently takes agreed upon international standards and breaks them when creating their software. For proof and examples please study how they implement kerberos, LDAP (for their Active Directory scheme), wpad (auto discovery protocol attempt that is implemented but not approved - see http://www.ws.afnog.org/afnog2001/services/squid/proxyie.html), email (countless, countless problems with Outlook versions), html-formatted email by default in their clients (see Outlook again), naming standards (see ".htm"), ftp server services (non-standard directory listings under IIS until version 4 of IIS and ASCII file transfers by default in all IIS ftp servers), DNS and Dynamic DNS designed to work with Active Directory, Active Directory will not work with standards built Dymanic DNS servers, and the list goes on, and on, and on...
  • Others don't recommend this. Gartner, one of the leaders in enterprise research and analysis recommends dropping IIS immediately (Fall 2001) due to gaping security holes in this software. See http://www.zdnet.com/zdnn/stories/news/0,4586,2814546,00.html for more details.
  • For an in depth look at some of these issues and more see http://www.dwheeler.com/oss_fs_refs.html.

FreeBSD vs. Linux

This questions comes up quite a bit. Why not run Linux instead of FreeBSD? This is perfectly reasonable and Linux is a quite stable OS and runs many core Internet services well, but, it does not scale as well as FreeBSD and it is not as stable. If your game is to provide rock-solid, reliable core services to a large user base, then very few Operating Systems can beat FreeBSD in this arena, and none can come close in cost.

FreeBSD will run on standard Intel-based hardware very efficiently. FreeBSD can scale core services to thousands of users. And, FreeBSD offers a huge number available applications. Here's a bit more "Why FreeBSD" from http://www.ws.afnog.org/afnog2001/services/intro/freebsd_intro.htm.

  • Very stable, especially under load
  • Heavily used and tested in large service providers
  • Scalability features as standard: e.g., pwd.db (indexed password database), which give you much better performance and scales well for very large sites. Linux uses a flat structure password file that is inefficient for large sites.
  • Optional "softupdates" filesystem combines crash-safety of BSD filesystem with speed of Linux filesystem - see /usr/src/sys/contrib/softupdates/README
  • Similarities to BSDI and other "industrial strength" Unixes

Linux users will find some annoyances: for example, that 'bash' is not installed as standard, and new package management tools to learn. However, we feel that the robustness of FreeBSD is the most overriding factor for ISPs, and it's worth you getting to know and use FreeBSD. The FreeBSD web site is located at http://www.freebsd.org/.

Security [Return to Top]

Securing your system from attack is a basic responsibility of any good system administrator. Doing this is not an easy task and staying on top of how to secure you system is of critical importance. At this stage of the Internet (Summer 2003) if you do not secure your site, then you are guaranteed to be compromised. Your site will be broken in to and you are likely to lose data, have data corrupted, and lose service. Securing your system and knowing how to respond to security threats and breaches is of the utmost importance. If you have any doubts about this you should see statistics from the HoneyNet Project at http://project.honeynet.org/, which includes reports of one unsecured Linux box that was compromised within 45 minutes of being placed on the net.

Below we give you a few basic guidelines about securing network servers, but before you install, configure and place a server on-line we strongly suggest that you read our entire security document located at (opens in a separate window):

http://nsrc.org/security/
This document contains detailed advice, pointers, and specific information about the steps you need to secure your network servers. You should do this before you allow any of your servers to be connected to the Internet.

Security is an ongoing process, but there are some basic guidelines you can follow to secure most servers. Securing an entire network involves additional planning and work. Larger organizations will usually have an entire staff person dedicated to security. Generally, security is much easier to implement if you do it from the start and you provide just the services you need vs. providing everything out of the box and then trying figure out what you can turn off.

General Guidelines

Very quickly, here are some general guidelines for securing a server on your network. If you are new to security be sure that you go to some of the links listed below and read them in depth. A few pointers include:

  • Turn off unnecessary services: If you don't need to run that printer daemon on your FreeBSD box, then don't. It has security holes!
  • Use Filters: That is filter the packets that come in to your system and do not let all of them through to the services that are running. This includes using tools like inetd or xinetd on your Linux server.
  • Segregate services on different boxes: If you plan on running telnet or sendmail or other insecure servers, and you need to run core services to keep your business up, then consider splitting these to separate boxes. That way if one is compromised your whole network set of services is not compromised.
  • Put some services behind firewalls: Many organizations see firewalls as a guaranteed way to protect their network. But, the problem with a firewall is that you often must open ports on the firewall to allow your users access to the services they want to use. And, you probably guessed this, the services your clients want to use are the most likely ones to come under attack. All is not lost, however, as a firewall can certainly reduce security problems from port scans, less used services, etc.
  • Keep up with patches! Pay attention to security patches for the operating system and services you are using. Apply them, especially when they address security holes. If a security hole is found in a service you run it is almost a guarrantee that your server will be attacked and compromised quickly (hours or days) once this hole becomes well know.
  • Consider System Integrity Checkers: Including things such as the Tripwire project (http://www.tripwire.org). These products have secured known databases of your filesystem. If anything changes that you do not expect to change, then you are likely to be notified and you can deal with the problem. Tripwire, and such systems are hard to install and are really only effective if you install them right when you first build your server. Once you rserver has been built and running for a while, then you do not know if it has been compromised or not.
  • See a detailed list on how to conifgure a secure system at http://ns.uoregon.edu/security/linux.html.

Some Security Links

Software

Software and Information Lists

Backup [Return to Top]

This cannot be stressed enough. At some point you will lose data. Your system will experience a hardware failure. You may have a security breach that results in data loss or corruption. The power will go out or or spike. You will experience a natural or man-made disaster. One of these, or more, is guarranteed to happen. Your only method for recovering from such situations is to have a safe, usable, tested (many people forget this step), and reliable backup of your data.

How you decide to implement this will depend on your hardware setup, amount of data that needs backing up, and your budget. The general rule of thumb is the easier and more data backed up, then the more expensive the solution. Often administrator's attempt to backup entire systems, when the real goal is to be able to get your user's data back and your current configuration up and running.

When designing your network and server setup try to design so that unique configuration files and user data is segmented in to distinct areas. If, for instance, you know that you need to backup /home, /data/ and a few system directories like /etc, /usr/local, etc., then your backup setup and configuration will be much easier.

Your next step is to decide how quickly you need to be able to recover your system and user's data in case of failure. Remember, even if you decide this must be fast, you will need the additional hardware and ability to restore to that hardware to make this happen.

Here are some good sites that discuss backup solutions (commercial and freeware) for servers:

Physical Infrastructure [Return to Top]

When planning to set up an ISP be sure that you pay close and careful attention to your physical space and infrastructure. It is easy to think about hardware and software, but forget about the supporting pieces. Below are items that you should consider and links to some good pages that discuss these are in more detail:

Space and Power

Do you have a large enough space set aside to hold your staff and your equipment? In addition, can you get the necessary power and phone lines to your chosen location. Are there any issues with the location and bringing the network to that location? What about power? If you include air conditioning, servers, desktop machines, modems, routers, switches, etc are there enough? You should probably do some planning to figure out how much power your equipment will need, then how far you can scale before you run in to problems.

Racks and Wiring

As your organization grows you will quickly realize that having a well thought out and clean system for your patch cables in place can save you countless hours of crawling around, tracking ends, and accidentally disconnecting machines. In addition, this can be faciliated by placing your switches, routers, hubs, and CPUs in racks.

UPS and/or Generators

If your area experiences power outtages, brown-outs, or power spikes, then you will need more robust UPS equipment. But, in all cases you need to protect your equipment with a Uninterrupted Power Supply (UPS) system and/or a generator. Power that is not clean can cause serious damage to equipment. If you lose your modems due to a power outtage or spike, then how long will it take before you are back up and running? If the answer is days or weeks, then your business is not likely to survive such outtages. A reliable and robust system of UPSs and/or generators becomes critical. A great place to start planning for what you will need is the American Power Conversion web site at http://www.apcc.com/. They have excellent UPS products, but they also have very useful on-line planning tools as well.

Phones/Phone Lines


Often this is a difficult area -- no matter what country you are in. Getting enough phone lines to support the number of dialup modems you wish to provide may take months, political wrangling, or may require that you physically upgrade your phone infrastructure to support what you wish to do.

Modems

What modems to buy for dialup is important. If you can afford it, newer digital modems use very little room and power, or easily configurable, and generally reliable. Modems that you use for your dialup pool are likely to be very different than the modems your clients will use.

Routers

The router or routers you buy will be largely dependent on the size of your organization. This is another area where planning for scale is important. If your router cannot grow with your traffic and network size, then replacing the router with a new one may be considerably more expensive than buying the correct router from the start.

Switches (vs. Hubs)

The simple and short answer to this is use switches not hubs. At this point in time switches are so inexpensive and give you so much gain in network efficiency and security that there is no reason to buy hubs. Your largest decision will be to decide whether you want a managed or unmanaged switch. It may be critical, depending on the hardware you have, to be able to manage the individual ports on a switch. For instance, if some equipment is only 10Mb/sec capable, then you may need to set the corresponding port on a 10/100Mb/sec switch to run at 10Mb/sec. This is not as common a problem as a few years ago, but it does still exist.

Scalability [Return to Top]

You may have noticed that we have discussed how items scale throughout this document. We include this section separately to emphasize this point. As you design your new ISP it is important to think to the future. What does your ISP look like if it is successful? What if you build for a few hundred users and you end up with a few thousand? Can you support this? What about future services such as e-commerce, web portals, large scale email, newsfeeds, database deployments, etc.?

These are critical questions. There are some technical issues to consider when considering how you might scale your operation and what pieces may need to scale.

Email

Issues of scaling email were discussed above in the section titled, "General Issues with MTAs." Please refer to this for information.

Databases

This is the one area where most of the OSes mentioned will not suffice for very large scale databases. The one OS that can deal with this is Solaris. By very large scale we mean 100s of gigabytes with 100s or 1,000s of transactions per second. In reality you are not likely to experience this level of database need in your organization. If you do, then you will likely use a product such as Oracle or DB2. If you are interested in a large list of databases that run on Linux (and UNIX/FreeBSD for many) see http://linas.org/linux/db.html for more information.

Otherwise, an acceptable database for departmental level, or medium size use would be something like PostgreSQL. You can download and read about PostgreSQL at http://www.greatbridge.org/. In addition, O'Reilly has some excellent books on databases and UNIX/Linux at: http://www.oreilly.com/.

The other popular Open Source database product that is in use is MySQL. You can read about MySQL in depth at http://www.mysql.orgl and at O'Reilly (http://www.oreilly.com/) as well. MySQL lacks some of the fundamental database features that PostgreSQL has, but it is a very well known tool and has many, many add-ons and utilities available.

For those running Windows NT and Microsoft's SQL Server product the two pieces of advice we can give are:
  • Backup your database files daily!
  • Run SQL Server on a separate machine with no other major services running

This comes from real world experience with many, many small to medium size shops. MS SQL Server will crash under Windows NT from time-to-time due to memory leaks and it will corrupt your database files beyond recovery when this happens. Under Windows 2000 and XP we can only assume this situation has improved. MS SQL wants lots of RAM and it runs much more stable on NT Server machines that are dedicated to the product.

If you need access to large amounts of data in large database files see the table below for file size and filesystem size comparisons among operating systems. Note that we recommend the use of a journaling file system when possible as this increases your data reliability and will greatly improve startup time on very large file systems. Be aware, however, that a journaling file system will be somewhat slower than standard file systems.

Large Memory and File Access, SMP, and Price
OS/Feature(1) FreeBSD 4.2 FreeBSD 5 Linux 2.2.x Linux 2.4.x Solaris 8 Windows NT Windows 2000
Max. RAM 4GB 4GB 2GB 64GB [3GB] 4GB(5) 2GB 8/64GB
Max. File System 16TB [1TB] 16TB [1TB] Terabytes Petabytes(2) Terabytes 4/2048/16EB(3) 4/32/16EB(4)
Max. File Size 16TB [128GB] 16TB [128GB] Terabytes Petabytes(2) Terabytes 4/4/16EB(2) 4/4/16EB(4)
SMP No Nov. 2001 4 [2] 8 [2] 128 8 4/32
Price Free Free Free Free $45/75/1000s Approx $700 Approx $850

(1) Items in "[ ]" are "everday" maximumns. Items split (8/64GB for example) imply sizes for product versions.
(2) Petabytes using IBM JFS filesystem. Terabytes is a more likely number for now as petabyte file size has not been proven.
(3) FAT/HPFS/NTFS (after SP3). EB = Exabytes, or 2^64 bytes.
(4) FAT/FAT32/NTFS
(5)On 32-bit architectures.

Futures

A few areas that you may want to watch as an ISP include:

Client Support/Help Desk [Return to Top]

Implementing a proper Help Desk from the beginning will be critical to your long term success. If you can get clients up and running quickly with your service and you can support them well, then it will be all that much easier to grow. For a detailed look at how to set this up and to see what some other organizations have done take a look at our Help Desk section at:

In addition you can find additional Help Desk information at:

Trying to implement a help desk after you have attracted a large user base can be quite painful. If you read through the pages above, particularly the ones at the NSRC, you will get a good feeling for what you are likely to be implementing as your ISP becomes successful and grows. A Help Desk and Support organization does not stop at just creating a number, or email address where your customers can reach you, but becomes a philosophy behind how you provide service. Remember, this document stresses scalability in all phases of your operation. This includes how what you do will affect your ability to be able to support your clients. The NSRC Help Desk pages, particularly the Setting up a Functional Support Organization section (http://www.nsrc.org/helpdesk/creating.html) goes over these ideas in detail with practical examples and solutions.

Conclusion [Return to Top]

This document is meant to give you a first level look at the pieces involved with setting up an ISP and then point you to much more detailed documents and tools that you can use to get up and running. We welcome your feedback. As you can see, there are a very large number of links in this document. If you wish to report broken links, suggest new, different or better links to resources, changes to content, etc. please fee free to email us at nsrc@nsrc.org.

If you have other questions or coments about content in this document please feel free to send email to nsrc@nsrc.org as well.