Scalable Web Services for Unix

Joel Jaeggli

Lucy Lynch

Hervey Allen

Academic User Services

University of Oregon

Agenda

Agenda Bashing

Overview

Squid

overview of server requirements

Installation

Configuration/Performance Tweaking

Two Example Servers

Configuration for clients

Peering

Agenda (Cont)

Apache

Installation

Configuration

Performance Tweaking

Virtual Hosts

Modules

suexec

ssl

Agenda (Cont)

Other Topics

Webmail Services

other apache modules

Squid Overview

Why are Cache boxes important?

What are the minimal requirements for a cache

What should a cache do?

Reduce latency

Conserve bandwidth

Squid Installation

Installation

Squid can be installed two ways:

As part of the FreeBSD ports collection

Has the advantage of being well integrated with freebsd

From the source distribution package

Is self contained.

To build from source:

tar -zxvf squid-xxx-xxx-src.tar.gz

cd squid-xxx-xxx

./configure --prefix=/usr/local/squid

Squid Installation

make all

make install

Squid should now be installed in the directory specified in the prefix command

to launch Squid on boot in FreeBSD:

create a shell script in /usr/local/etc/rc.d that does

/usr/local/sbin/RunCache >/dev/null 2>&1 &

Squid Configuration

Before Starting the cache for the first time configure it.

Using adduser create a user squid to own the cache (the cache should not run as root)

change the ownership of the /usr/local/squid/cache and /usr/local/squid/logs to be owned by the squid user

edit the file

/usr/local/squid/etc/squid.conf

Squid Configuration

Squid.conf lists all of the available options for squid

In most cases the defaults are commented out.

To change a value uncomment and edit to taste.

Important things to set:

#cache_mem 8

affects the amount of ram memory squid will use to cache data, the program itself will use additional memory, as will the database of files stored on disk.

Squid Configuration

#maximum_object_size 4096 KB

If your cache is small this number might result in relativly large files cluttering up the cache.

#ipcache_size 1024

defines the number of IP addresses the DNS cache will hold. More will improve performance at the expense of memory

#fqdncache_size 1024

maximum number of fqdn cache entries in dns cache

Squid Configuration

#cache_effective_user nobody

Since we've created a squid user already we should set the effective user to squid. That way when squid is invoked it will become the effective user.

Squid Configuration

#cache_dir ufs /usr/local/squid-2.4/cache 100 16 256

Each cache_dir line speciefes the location of one of squid's file caches, there can be more than one.

The first nuumber specifies the size in MB the second is the number of directories on the first level and the third is the number of directorieson the second level.

The more directories there are the fewer files there are in each directory.

For larger filesystems you want to increase the number of first level directories to 32 or 64

Squid Configuration

#ftp_user Squid@

change tosomething more informative such as squid@yourfqdn

#dns_children 5

you generally want this sent to the number of clients you want be able simultanously the maximum is 32

ACL's

The ACL's section of the squid.conf defines who can and can't connect to the proxy.

Best and fastest way to allows hosts is by netmask

Squid Configuration

That completes The most basic configuration.

You should be able to invoke

/usr/local/squid/bin/squid -z

This will create the cache directory structure

Then

/usr/local/squid/bin/squid

Should launch a working squid

Squid Configuration - Kernel

Optimizing FreeBSD for use with squid

One major problem with using squid on FreeBSD is the UFS filesystem.

Current solution is soft-updates for ufs

Rebuilding the kernel to support soft updates

cd /usr/src/sys/ufs/ffs

ln -s ../../contrib/softupdates/*.[ch] .

cd /usr/src/sys/i386/conf

Copy the generic config file to a file of your own

cp GENERIC SOFTUPDATE

Squid Configuration Kernel

Need to edit the SOFTUPDATE file to enable the softupdatge option

emacs -nw SOFTUPDATE

page down to bottom of file

add the line:

options SOFTUPDATES

Now it's time to build the kernel

/usr/sbin/config SOFTUPDATE

cd ../../compile/SOFTUPDATE

make depend

Squid Configuration - Kernel

make

make install

Now it's time to reboot the machine in single user mode.

boot -s

Make sure the filesystem you want to enable softupdate on is unmounted and run the following command on it's mountpoint

tunefs -n enable /mountpoint

then reboot

Squid Configuration - Kernel

Why softupdates is important

One of the most serious bottlenecks in squid is the creation, reading, and replacement of files off of the disk.

A high-end proxy server must be able to serve several hundred connections per second some of which will replace objects currently in the cache

How many create, write, destroy operations can you do per second on a filesystem

Squid Configuration - Kernel

Additional optimization for DiskD

What is DiskD

DiskD is a feature new to Squid 2.4 it creates a child process for each cache filesystem in order to keep the Squid cache processes from blocking on writes.

In the 2nd NLANR cache bakeoff this resulted in a 4-fold improvement in the performance of the squid boxes on FreeBSD

Squid Configuration - Kernel

What does DiskD require?

Sys V message que support

Shared memory support

FreeBSD has both on by default, however paramaters need to be tweaked.

For SYSVMSG

options MSGMNB=16384

options MSGMNI=41

options MSGSEG=2049

Squid Configuration - Kernel

options MSGSSZ=64

options MSGTQL=512

For Shared Memory

options SHMSEG=16

options SHMMNI=32

options SHMMAX=2097152

options SHMALL=4096

Then configure, and recompile your kernel

Squid Examples

Two server hardware configurations

Small

Pentium 180mhz

96mb ram

2.5GB ide disk

4 x 2gb fw seagate hawk 2x disks

symbios 53c875 UW controller

Intel Fast Ethernet

Squid Examples

Filesystem layout

/ 1GB ide

/usr/local/squid/logs 1.5GB ide

/usr/local/squid/cache0 2GB scsi

/usr/local/squid/cache1 2GB scsi

/usr/local/squid/cache2 2GB scsi

/usr/local/squid/cache3 2GB scsi

Squid Examples

squid.conf

cache_mem 40 MB

cache_swap_low 80

cache_swap_high 90

maximum_object_size 2048 KB

ipcache_size 8000

Squid Examples

cache_dir /usr/local/squid/cache0 1800 32 256

cache_dir /usr/local/squid/cache1 1800 32 256

cache_dir /usr/local/squid/cache2 1800 32 256

cache_dir /usr/local/squid/cache3 1800 32 256

dns_children 10

Squid Examples

Large Server

Pentium-III 500

384MB of RAM

8.4GB ide disk

3 x 9GB U2W Seagate Barracuda 7200rpm disks

Symbios 53c895 u2w controller

SMC Etherpower II 10/100

Squid Examples

Filesystem Layout

/ 4GB ide

/usr/local/squid/logs 4.4GB ide

/usr/local/squid/cache0 9GB scsi

/usr/local/squid/cache1 9GB scsi

/usr/local/squid/cache2 9GB scsi

Squid Examples

squid.conf

cache_mem 64 MB

ipcache_size 8192

ipcache_low 90

ipcache_high 95

fqdncache_size 4096

cache_dir /usr/local/squid/cache0 7000 128 256

cache_dir /usr/local/squid/cache1 7000 128 256

cache_dir /usr/local/squid/cache2 7000 128 256

Squid Examples

ftp_user Squid@proxy.uoregon.edu

ftp_list_width 60

dns_children 32

acl ourallowedhosts src 128.223.0.0/255.255.0.0

acl mesd src 198.236.63.0/255.255.255.0

acl owencache src 198.237.157.0/255.255.255.0

acl owen srcdomain or.us

acl lane srcdomain lane.edu

acl orst src 128.193.0.0/255.255.0.0

Squid Examples

store_avg_object_size 20 KB

Proxy Configuration for Clients

Three different ways cliensts can be configured ot use the proxy server

manual configuration

proxy autoconf file

wpad (ie5)

Squid Peering

What is cache peering?

Why peer?

In order to connect to the nlanr cache hierarchy in the United States.

To provide better service to downstream customers with their own cache boxes.

To peer with other service providers cache boxes across a public exchange point

Squid Peering

Two kinds of cache peers.

Parent

A cache which accepts requests from peers and, if the object is not found in the cache retries the object directly

Sibling

A cache which accepts requests from peers, and if the object is not found returns a miss, after which the requesting cache retrieves the object itself.

Squid Peering

Example configuration on a machine with two Parent caches and one sibling

cache_peer proxy2.uoregon.edu sibling 3128 3130 no-query

cache_peer pa.us.ircache.net parent 3128 3130 round-robin

cache_peer_domain pa.us.ircache.net !com !edu !org !net !ca !gov !us !128.223

cache_peer sd.us.ircache.net parent 3128 3130 round-robin

cache_peer_domain sd.us.ircache.net !com !edu !org !net !ca !gov !us !128.223

References

Current versions of Squid at:

http://www.squid-cache.org/Versions/v2/

Squid FAQ

http://www.squid-cache.org/Doc/FAQ/FAQ.html

FreeBSD Handbook building a custom kernel

http://www.freebsd.org/handbook/kernelconfig-building.html

Apache Agenda Revisted

Installation

Configuration

Performance Tweaking

Virtual Hosts

Modules

suexec

ssl

Apache Installation

As with squid, you can use the FreeBSD ports copy of apache, or build your own.

Much of how you install and configure apche will depend on how the server will be used.

Will the server host lots of user websites, (the UO 20,0000) , or just a few web-sites?

Is the machine to be a dedicated webserver

Is the webserver an interface to other applications

Apache Installation

Building apache today

were going to build apache with two optional componets

ssl support

suexec

What is suexec

suexec is an Apache module which allows cgi programs to run as the user who put them in place rather than as the uid of the webserver (in this case nobody)

This fixes some security problems an creates others

Apache Installation

For this build therefore, we need three components

openssl-0_9_4_tar.gz

apache_1_3_12_tar.gz

apache_1_3_12+ssl_1_40_tar.gz

Apache Installation

Lets start by building openssl

tar -zxvf openssl-0_9_4_tar.gz

cd openssl-0.9.4/

./config

make

make test

make install

Apache Install

Now lets uncompress and patch apache

tar -zxvf apache_1_3_12_tar.gz

cd apache_1.3.12/

tar -zxvf ../apache_1_3_12+ssl_1_40_tar.gz

take a look at the readme.ssl

./FixPatch

Apache Installation

Now lets configure and build Apache

./configure --prefix=/usr/local/apache+ssl+suexec --enable-suexec \suexec-caller=nobody

make

make install

cd src

make certificate

Apache Configuration

This is the hairy bit

Couple of example server configs

using ssl means you're running two virtual servers

one on port 80 (the regular server)

one on port 443 (the ssl server)

more directories to keep track of because your keys

key generated is valid for only one hostname

so a key per virtual host is a good idea if you're doing virtual hosts with ssl servers as well

Apache Configuration

Unsigned keys are fine for things like running your webmail services through ssl, for ecommerce type applications having a key signed by a reliable CA (certificate authority) is considered normal.

CA's include Verisign (USA), Thawte (South Africa) and others

Apache configuration

http.conf examples

a generic config for ssl

a webserver config with a few virtual hosts (1894 user hosted websites, 69 virtual hosts, ~740,000 requests a day)

2000-04-19T23:12:42