Moving a Mailman List Across Servers: Case Study

archived information

Return to NSRC Help Desk home.

Introduction

African Network Operators Group (AfNOG) had several mailing lists hosted on servers in Uganda. The group received a new server box and hosting location in South Africa. It was decided to migrate the lists from the Ugandan servers to the one in South Africa. Box boxes were running Mailman under Unix/Linux. The time frame was August 2006.

We will describe the details for bringing across one public list, afnog@afnog.org. The server, afnog.org, was running Fedora Core 5 and Mailman 2.1.8. Mailman was installed via RPM. The specific directories, therefore, may be different if you are using a different Linux flavor, or have installed differently. You can replace "afnog" and "afnog.org" with your own list name(s) and/or domain(s).

In addition, while it was possible to turn off incoming mail for the Ugandan servers, we decided not to. This meant it was possible to miss a few posts to the list in the archive files and possibly subscription requests. These could be dealt with once the move is made as list traffic was not that high. If you have a very busy list, or lists, then you will likely want to stop delivery of incoming mail to your lists while the MX record for your domain is swung. One simple trick is to point the mailbox for each list you are moving to a directory where the list user has no rights to create files. This will cause a "4xx temporary failure" for incoming mail. This works well as remote servers will eventually see your MX record change and re-attempt delivery to your new box where mail can be successfully delivered.

Overview: Steps Taken

  • Install Mailman via RPM.
  • Decide upon the MTA you will be using. In our case we used Sendmail.
  • Update Sendmail as needed for your site.
  • Copy between servers the Mailman archives and configuration files.
  • Place the files in the appropriate locations.
  • Run the appropriate scripts on the files.
  • Update each lists configuration as is appropriate.
  • Edit the URL in the Mailman archives directory to point to your new server.
  • Consider changing MX record timeouts (TTL) a day, or two, before your move to 300 seconds (5 minutes).
  • Update MX records as needed to point incoming traffic to the new server. If the MX Time To Live (TTL) is short enough, then mail should start arriving quickly.
  • Assuming email is being delivered as expected you are done, except for standard mailing list administrative tasks.

Note, in the instructions below if "#" is used for a command, then this means you must be root to run the command.

Specifics: Steps Taken

Install RPM

The RPM mailman-2.1.8-0.FC5.1.i386.rpm was installed:

# rpm -ivh mailman-2.1.8-0.FC5.1.i386.rpm

You should really read the Mailman Installation Manual:

Text version

And, you should become familiar with the Mailman administrative scripts available to you. For our installation they reside in:

/usr/lib/mailman/bin

You can read descriptions for each script here:

http://www.gnu.org/software/mailman/site.html

Decide upon Your MTA

As noted we are using Sendmail. The Mailman Installation Manual noted above includes step-by-step instructions for getting Mailman to work with other MTAs. To get Mailman and Sendmail to work together requires several steps. Details on these steps are included later, but in general you must:

  • Make sure you have a logical link point to the Mailman executable in the Sendmail "smrsh" directory (SendMail Restricted SHell utility) so that Mailman can execute Sendmail commands as needed. Under Fedora Core 5 the Mailman RPM install did this for us.
  • Create an aliases file for Mailman-specific aliases that are required for each mailing list created. These aliases are not automatically updated for you when creating new mailing lists and must be properly updated by hand. The Mailman Installation Manual includes a potential work-around for this.
  • Edit /etc/mail/sendmail.cf to point to the new aliases file.
  • Create an empty new db file for your new aliases, and then run the "newaliases" command to populate this file.
  • Edit /etc/mail/sendmail.cf to allow for external connection to your site from other mail servers (i.e. smtp port 25 access).
  • Verify that you can telnet to port 25 to an external mail server and to your mail server both internally and externally.

Under Fedora Core 5 the "smrsh" directory is located in:

/etc/smrsh/

The logical link required looks like:

lrwxrwxrwx   1 root mailman   29 Jun 11 03:54 mailman -> /usr/lib/mailman/mail/mailman

This should already exist if you install by RPM. Other locations for the smrsh directory include /var/smrsh and /usr/local/smrsh.

To tell Sendmail to respond to remote smtp connections edit the file /etc/mail/sendmail.cf and update the line that reads:

O DaemonPortOptions=Port=smtp,Addr=127.0.0.1, Name=MTA

to read: O DaemonPortOptions=Port=smtp, Name=MTA

You will need to be root to edit /etc/mail/sendmail.cf.

The final step will be to create an additional aliases file as noted above, but you cannot do this until you've run through the actual moving of your Mailman lists. This final step will be described a bit later in this document.

You can choose to restart Sendmail now, or you can wait until the end of all these steps. There are several ways to restart Sendmail, but in Fedora Core 5 the following should work:

# /etc/rc.d/init.d/sendmail restart

Copying Mailman Files Across Servers

In our example the administrator on the server in Uganda created three sets of archive files for each mailing list. We'll run through the steps for a single list. You can simply re-iterate these steps for each list you need to move. On the original server the actual directories where the Mailman archives and list configuration files resided were different than on our target box, but the archives were named to indicate where the should be expanded. The three archives copied across for the AfNOG mailing list were named:

var-lib-mailman-archives-private-afnog.mbox.tgz
var-lib-mailman-archives-private-afnog.tgz
var-lib-mailman-lists-afnog.tgz

We'll assume these are first copied to /tmp. Now lets copy and expand these where they belong. We keep a copy of each archive in /tmp in case mistakes are made.

# cp /tmp/var-lib-mailman-archives-private-afnog.mbox.tgz /var/lib/mailman/archives/private/
# cp /tmp/var-lib-mailman-archives-private-afnog.tgz /var/lib/mailman/archives/private/
# cp /tmp/var-lib-mailman-lists-afnog.tgz /var/lib/mailman/lists/

Now let's expand the files:

# cd /var/lib/mailman/archives/private
# tar xvzf var-lib-mailman-archives-private-afnog.mbox.tgz
# tar xvzf var-lib-mailman-archives-private-afnog.tgz
# cd /var/lib/mailman/lists
# tar xvzf var-lib-mailman-lists-afnog.tgz

Now what you'll end up with are the following directories:

  1. /var/lib/mailman/archives/private/afnog.mbox/
  2. /var/lib/mailman/archives/private/afnog/
  3. /var/lib/mailman/lists/afnog/

Directory "1" contains the mbox of all posts to the list from which the archives directory is derived. Directory "2" contains the actual archives files and database directories for viewing archives by date, thread, subject, and author. Directory "3" contains the actual list configuration, membership, and list status files. This information is kept inside database files with the extension ".pck". You cannot edit these files directly, but rather must use Mailman provided scripts to extract their contents to a text file. You can edit this file, then import it back in. Once you are done you will likely use the Mailman web interface to make further changes.

Now that you have all the files in place for the list to work you need to make a few updates to make sure that things work as expected.

Use the Appropriate Mailman Scripts

First you need to verify that permissions are correct on all your lists. Use the Mailman check_perms scripts like this:

# /usr/lib/mailman/bin/check_perms -f

You should run this script twice to verify that permissions have been fixed properly. The "-f" option is to fix found problems. You are likely to see multiple issues with the archive files.

With almost all the Mailman provided scripts you can type scriptname --help to get details on how to use the script, or read the Mailman web page here:

http://www.gnu.org/software/mailman/site.html

Now that permission are fixed you need to set the default web page URL for your lists. As the list has been moved from another domain this setting is currently incorrect and will cause problems in your Mailman web interface. You can set this via a configuration variable for each list, but the recommended method is to do the following:

# /usr/lib/mailman/bin/withlist -l -r fix_url afnog -v -u afnog.org

Naturally you would replace "afnog" with the list name you are currently moving. The last item is the domain where your lists reside, nothing else.

Next you need to set the password for the list. If you don't wish to change this (and you know it), then this step is not necessary. To do this:

# /usr/lib/mailman/bin/change_pw -l afnog -p password

The "-l" is the list and the "-p" is the password you wish to use.

The last step is the most complex. You need to export out the list configuration to a text file like this:

# /usr/lib/mailman/bin/config_list -o /tmp/afnog.conf afnog

Again, replace "afnog" with the name of your list. You can now edit the file "/tmp/afnog.conf" and update entries as needed. If you list is configured exactly as you need it, then this step is not necessary. The one item that may change is the "owner" specification.

If you do make a change to your configuration using this method, then you should re-import the file back in to the Mailman list database by doing:

# /usr/lib/mailman/bin/config_list -i /tmp/afnog.conf afnog

Optionally you could have simply done this via the web interface if you new who the list owner was, or if you have set your Mailman site password and used this to enter in as list administrator. You can set your Mailman site password like this:

# /usr/lib/mailman/bin/mmsitepass password

At this point you are close to finished. You are, however, lacking the Sendmail aliases required for each list on your site. First, decide where you want these aliases to reside. In our case they are in /etc/aliases.mailman, so we'll use this example.

Mailman has a utility to extract the aliases you need for all your lists. First do this:

# /usr/lib/mailman/bin/genaliases > /etc/aliases.mailman

Mailman outputs a note that you must edit out of the file you've just created. You can do this each time you create a new list. Don't worry about overwriting the /etc/aliases.mailman file as all Mailman list aliases are outputted by this command. So, to clean up the file you must edit it:

# vi /etc/aliases.mailman

And, remove the text at the top that reads:

To finish creating your mailing list, you must edit your /etc/aliases (or equivalent) file by adding the following lines, and possibly running the `newaliases' program:

Now you must edit your /etc/mail/sendmail.cf file and edit/update the following line:

O AliasFile=/etc/aliases, /etc/aliases.mailman

Once you have done this you need to create an empty /etc/aliases.mailman.db file by doing:

# touch /etc/aliases.mailman.db

Then, you must restart Sendmail so that it knows about the location of the additional aliases file and you must run the newaliases command to populate the new aliases database file, like this:

# /etc/rc.d/init.d/sendmail restart
# newaliases

Remember, for each new mailing list created you need to run the genaliases script, edit the /etc/aliases.mailman file to clean it up, and then run newaliases.

Edit the URL in the Archives Directory

This is not critical to getting your lists to work, but it will save your users from going to the wrong location if they view your Mailman archives. At a minimum you need to edit the file:

/var/lib/mailman/archives/private/afnog/index.html

You will notice the line near the top of the file that reads, "You can get more information about this list", with the "more information" hot-linked to the old server where the list used to reside. Update this with the proper domain name for your new server. In addition, you have a decision to make. Your archives are likely filled with links to the old mail server. It is your decision whether you want to change these links, or leave them for posterity. Note, if you are saving attachments in your archives, and you want your users to be able to access them, then you'll need to update your archives entries to point to the proper domain. One method to do this is to use a regular expression search and replace trick using Perl. For instance, you could go to the /var/lib/mailman/archives/private/afnog/ directory and run the command:

# perl -pi -e 's/old.domain/new.domain/gi' *

This would update all files in this directory, but not files in subdirectories. The "*" is your file string. You could be more specific and use "*.html", etc. if you wish.

MX Record Changes

As noted earlier, to minimize the amount of time it takes for mail to start being delivered to your new domain from the old one you may wish to change the TTL on your domain's MX records 1 or 2 days in advance. A setting of 300 seconds (5 minutes) is a good choice to allow for the switch to take place quickly when you are ready. Remember, whatever your TTL is currently set to is how long (worst case) it will take for servers to notice the new TTL that you set.

Once you are confident that your lists and your server are ready, then you can have someone update the MX record to point to your new mail server. To watch traffic starting to flow you can do:

# tail -f /var/log/maillog

On a server using Sendmail. Remember to eventually set the TTL back to something reasonable for your MX record(s).

Now you are ready for standard Mailman list administration tasks. If you use the Mailman web interface a few URLs, in general, to know are:

http://domain.name/mailman
http://domain.name/mailman/listinfo

The first URL just redirects to the second. This will display all public lists. To view private lists you need to know the list name. Then you can go to:

http://domain.name/mailman/listinfo/listname

This works for public lists as well. Once you are on the initial admin page you will see links to the lists' administrative interface (http://domain.name/mailman/admin/listname), and to the lists' archives if there are any (http://domain.name/mailman/private/listname/ for private archives).

Finally, if you have files left in /tmp you may want to delete or move them. In this document we had the *.tgz files and the afnog list configuration file, /tmp/afnog.conf left over to be removed or moved.

The Mailman home page is here.