Currently our mail servers are mx1, mx2, and mx3.

We use qmail as our MTA, and have it configured to deliver messages to users' mail volumes in AFS (mounted in $HOME/Maildir).

Contents

  1. Mail (qmail)
    1. Packages
    2. Custom programs and scripts for /usr/local
    3. Installation
    4. Configuration
    5. Startup
  2. Mail for Orgs
  3. Mailing Lists (ezmlm)
    1. Quick tips
  4. TODOS (when upgrade to wheezy is complete)
  5. Bring-up for Debian 12
  6. TODOs for qmail package

1. Mail (qmail)

1.1. Packages

1.1.1. Unmodified debian packages

1.1.2. Packages that required minor modifications

1.1.3. Packages that required major modifications or were not in Debian

1.2. Custom programs and scripts for /usr/local

1.3. Installation

All club users must be present on mail servers. To do this (as root):

Toggle line numbers 
touch /etc/passwd.user
/afs/club/system/scripts/sh/passwd-update.sh

Install the Debian packages listed above.

Be sure to save off the users the packages create into /etc/passwd.system!

Toggle line numbers 
grep '^\(dspam\|alias\|qmail.\):' /etc/passwd >> /etc/passwd.system

Build and install the wrappers:

Toggle line numbers 
cd /tmp
svn co https://svn.club.cc.cmu.edu/cclub/wrapit/trunk ./wrapit
cd wrapit
make

# Remaining steps require root

mkdir -p /usr/local/stow/wrappers-001
make DESTDIR=/usr/local/stow/wrappers-001 install

cd /usr/local/stow/wrappers-001
mv ./usr/local/* .
rmdir ./usr/local
rmdir ./usr
mv ./var/log/wrapit /var/log
rmdir ./var/log
rmdir ./var

ln -s /var/log/wrapit/svc /etc/service/wrapit-logs

cd ./bin
svn export https://svn.club.cc.cmu.edu/cclub/scripts/trunk/perl/ezmlm-issubn.pl ./ezmlm-issubn.pl
# Assumes you've downloaded the wrapper-symlinks.tar.gz, which is attached to this page, somewhere locally
tar -zxvf /path/to/wrapper-symlinks.tar.gz

cd /usr/local/stow
stow wrappers-001

1.4. Configuration

The easiest way to configure qmail for cclub, is to start with configuration from an existing mail server.

1.4.1. /var/qmail/control

Most of qmail's configuration exists as several files in the /var/qmail/control directory. For a list of such files, see qmail-control (5), though please note that our qmail supports additional control files (see the actual man page on one of the mailservers for a complete list).

1.4.1.1. Things that need to be changed if you start with an existing mail server's files

Edit me --> local FQDN.

Edit locals --> add local FQDN to the list. Copy the new version of locals to all the other mail servers.

1.4.1.2. Descriptions of the control files, some with cclub's values

Here are some important control files, and how they are configured for the cclub environment:

concurrencylocal
maximum number of local deliveries to perform simultaneously 
5
concurrencyremote
maximum number of remote deliveries to perform simultaneously 
30
defaultdomain
domain to use when a recipient address is given with an unqualified hostname 
club.cc.cmu.edu
defaulthost
domain to use when a recipient address is given without any hostname 
club.cc.cmu.edu
locals
domains that are handled by local delivery (list below is valid as of 10 Feb 2014): 
localhost
club.cc.cmu.edu
«hostname».club.cc.cmu.edu
thorin.dementia.org
thorin.club.cc.cmu.edu
aberrant.org
cmucc.org
me
the fully-qualified hostname of the machine qmail is running on 
«hostname».club.cc.cmu.edu
plusdomain
what should be appended to recipient addresses given with a trailing '+' character 
cmu.edu
queuelifetime
how long to keep a message in the queue before treating temporary delivery failures as permanent 
345600
rcpthosts

domains for which mail will be accepted, with wildcards indicated by a leading '.' character

This list is quite lengthy (at some point we may want to go through this and identify domains in it that we are no longer hosting mail for, and remove them). I'm not including it here, due to its length. It should be fine to copy the file off of some existing mail server.

servercert.pem
concatenated, PEM encoded SSL private key, CA key chain, and SSL certificate
spfbehavior
controls SPF validation: whether it's done, and how to treat failures 
1
timeoutremote
how long to wait for responses from remote SMTP servers 
300
virtualdomains

maps "virtual" users and domains to local user accounts

This list is quite lengthy (at some point we may want to go through this and identify domains in it that we are no longer hosting mail for, and remove them). I'm not including it here, due to its length. It should be fine to copy the file off of some existing mail server.

1.4.2. /var/keys

Create the directory.

Extract the "mailtabs" keytab to /var/keys/mailtabs.

Toggle line numbers 
mkdir -p /var/keys
kinit -S kadmin/admin «username»/admin
kadmin ext -k /var/keys/mailtabs mailtabs
kdestroy

The mailtabs will be added to the directory by update-mailtabs.sh.

1.4.3. Cron jobs

These all run as root. Probably a good idea to run all of them first manually. 

# Mail stuff
00 *            * * *   /afs/club.cc.cmu.edu/system/scripts/perl/mailassign.pl /afs/club/user > /var/qmail/users/assign && /usr/sbin/qmail-newu
02 *            * * *   /afs/club.cc.cmu.edu/system/scripts/sh/update-alias.sh
30 *            * * *   /afs/club.cc.cmu.edu/system/scripts/sh/update-mailtabs.sh
00 0            * * *   /usr/sbin/update_tmprsadh > /dev/null 2>&1

1.4.4. SMTP Access Control

See tcprules(1) for information on the rule files' syntax.

For the standard SMTP service (port 25), allow relaying for club machines. For non-club machines, optionally support authentication and relay for authenticated users. Otherwise, non-club machines are only allowed to send mail addressed to hosts/domains in rcpthosts.

/var/qmail/tcp.smtp: 

# B6 machines
128.237.157.:allow,RELAYCLIENT=""
# GERMANIUM
128.2.204.94:allow,RELAYCLIENT=""
# TECHNETIUM + DomUs
128.2.204.148-153:allow,RELAYCLIENT=""
# TRANSIT-1
128.2.207.106:allow,RELAYCLIENT=""
# loopback IPs
127.:allow,RELAYCLIENT=""
# catch-all
:allow,SMTPAUTH=""

For the submission service (port 587), always require authentication.

/var/qmail/tcp.submission: 

:allow,SMTPAUTH="!"

To use the rules with tcpserver, they need to be compiled into a .cdb:

Toggle line numbers 
tcprules /var/qmail/tcp.smtp.{cdb,tmp} < /var/qmail/tcp.smtp
tcprules /var/qmail/tcp.submission.{cdb,tmp} < /var/qmail/tcp.submission

1.4.5. SMTP Authentication

We use checkpassword-pam to authenticate club users connecting to SMTP. This requires a modified PAM configuration.

/etc/pam.d/smtpd: 

#
# /etc/pam.d/smtpd - PAM settings for SMTP-AUTH
#

# Unfortunately, for this case, we can't use any of the common-* files.
#   - pam_krb5.so, by default, wants to create FILE ccaches and chown them to
#     the authenticated user; that would fail, since we don't run as root
#   - we want to support mail passwords; setting that up requires additional
#     arguments for pam_krb5.so
#   - there's no need to create a PAG or get AFS tokens

auth     required       pam_krb5.so minimum_uid=110 alt_auth_map=%s/email \
                                keytab=/var/qmail/smtp.keytab ccache=MEMORY:

account  required       pam_krb5.so minimum_uid=110 alt_auth_map=%s/email \
                                keytab=/var/qmail/smtp.keytab ccache=MEMORY:

password required       pam_deny.so

session  required       pam_permit.so
session  optional       pam_krb5.so minimum_uid=110 alt_auth_map=%s/email \
                                keytab=/var/qmail/smtp.keytab ccache=MEMORY:

What's this all about?

alt_auth_map=%s/email
  • This allows us to provide alternate passwords that users can use for email. E.g., this would allow users to use a different password than there regular cclub password for email access from mobile devices.
keytab=/var/qmail/smtp.keytab
  • A Kerberos-enabled server uses a shared keytab with the KDC in order to authenticate the KDC (i.e., to prevent somebody to succeed in authenticating by spoofing the KDC). Usually /etc/krb5.keytab is used, but since checkpassword-pam is not run as root, it is not readable. To work around, we create a dedicated keytab for the smtpd to use.
ccache=MEMORY:
  • For a login user's convenience, Kerberos authentication will create a ticket file for the authentication user in /tmp, and then chown it to the authenticated user. This does not work when checkpassword-pam is not run from. For mail, the tickets are needed, so we can solve the problem by using an in-memory only credentials cache.

The next step is actually creating the above-mentioned keytab:

Toggle line numbers 
kinit -S kadmin/admin user/admin
kadmin ank -r SMTP/hostname.club.cc.cmu.edu
kadmin ext -k /var/qmail/smtp.keytab SMTP/hostname.club.cc.cmu.edu
chown qmaild:root /var/qmail/smtp.keytab
chmod 400 /var/qmail/smtp.keytab

1.4.6. DSPAM Configuration

A few things need to change for the Cclub environment:

/etc/dspam/dspam.conf

StorageDriver ---> /usr/lib/x86_64-linux-gnu/dspam/libmysql_drv.so
TrustedDeliveryAgent ---> "/bin/cat"
Add: UntrustedDeliveryAgent "/bin/cat"
Remove: all Trust lines except for Trust root

/etc/dspam/dspam.d/mysql.conf
Fill in with the details for the club MySQL server.

1.5. Startup

To have qmail start on boot, create supervise service directories, link to them from /etc/services, and then create wrappers in /etc/init.d.

1.5.1. Create supervise service directories

Toggle line numbers 
cd /var/qmail

########################################################################
# qmail system
########################################################################

mkdir -p svc/qmail/log/main
cat > svc/qmail/run << EOF
#!/bin/sh
exec /usr/sbin/qmail-start ./Maildir/
EOF
chmod 0755 svc/qmail/run
touch svc/qmail/down
cat > svc/qmail/log/run << EOF
#!/bin/sh
umask 0077
exec multilog t n50 ./main
EOF
chmod 0755 svc/qmail/log/run

########################################################################
# normal smtpd
########################################################################

mkdir -p svc/smtp/log/main
cat > svc/smtp/run << EOF
#!/bin/sh
exec /usr/bin/tcpserver -v -R -x /var/qmail/tcp.smtp.cdb \\
                        -u 1001 -g 1002 0 smtp \\
                        /usr/bin/rblsmtpd -b -r "zen.spamhaus.org" \\
                        /usr/sbin/qmail-smtpd \\
                        /usr/bin/checkpassword-pam -e -s smtpd /bin/true 2>&1
EOF
chmod 0755 svc/smtp/run
touch svc/smtp/down
cat > svc/smtp/log/run << EOF
#!/bin/sh
umask 0077
exec multilog t n50 ./main
EOF
chmod 0755 svc/smtp/log/run

########################################################################
# submission smtpd
########################################################################

mkdir -p svc/submission/log/main
cat > svc/submission/run << EOF
#!/bin/sh
exec /usr/bin/tcpserver -v -R -x /var/qmail/tcp.submission.cdb \\
                        -u 1001 -g 1002 0 submission \\
                        /usr/sbin/qmail-smtpd \\
                        /usr/bin/checkpassword-pam -e -s smtpd /bin/true 2>&1
EOF
chmod 0755 svc/submission/run
touch svc/submission/down
cat > svc/submission/log/run << EOF
#!/bin/sh
umask 0077
exec multilog t n50 ./main
EOF
chmod 0755 svc/submission/log/run

Toggle line numbers 
ln -s /var/qmail/svc/qmail /etc/service/qmail
ln -s /var/qmail/svc/smtp /etc/service/smtp
ln -s /var/qmail/svc/submission /etc/service/submission

1.5.3. Create wrappers in /etc/init.d

Toggle line numbers 
svinitd-create qmail > /etc/init.d/qmail
sed -i -e '/^svinitd/{' -e 'x' -e 's/./&/' -e 'x' -e 't' -e 'h' -e "i\\" -e 'unset INIT_VERSION' -e '}' /etc/init.d/qmail
chmod 0755 /etc/init.d/qmail

svinitd-create smtp submission > /etc/init.d/smtpd
sed -i -e '/^svinitd/{' -e 'x' -e 's/./&/' -e 'x' -e 't' -e 'h' -e "i\\" -e 'unset INIT_VERSION' -e '}' /etc/init.d/smtpd
chmod 0755 /etc/init.d/smtpd

1.5.4. Enable the services at boot

Toggle line numbers 
update-rc.d qmail defaults
update-rc.d smtpd defaults

2. Mail for Orgs

I set up mail this way for the KGB.

Unless otherwise noted, each step requires admin tokens.

  1. Create PTS users for the organization's mail service 
    pts createuser ${org}mail
# Note the pts ID assigned to the user.  E.g.,
# ----
# kbare@conch:~$ pts createuser ${org}mail
# User ${org}mail has id 1891
#                        ^^^^
pts createuser ${org}mail.mail
  2. Add the organization's user to passwd.user 
    echo "${org}mail:x:${uid}:20:${long_org_name} Mail Forwarding:${org_subdir_in_andrew_afs}:/bin/false" \
     >> /afs/club.cc.cmu.edu/service/etc/passwd.user
  3. Create a symlink in /afs/club.cc.cmu.edu/usr pointing to the org's directory 
    ln -s ${org_subdir_in_andrew_afs} /afs/.club.cc.cmu.edu/usr/${org}mail
vos release club.usr
  4. Create a Kerberos mail principal, extract it to the mailtabs directory 
    kadmin ank -r --use-defaults ${org}mail/mail
kadmin ext -k /afs/club.cc.cmu.edu/service/mail/mailtabs/${org}mail ${org}mail/mail
chown ${uid}:dialout /afs/club.cc.cmu.edu/service/mail/mailtabs/${org}mail
  5. Add the hosted domain to the mailserver configurations 
    # XXX: if we end up doing a lot of org mail hosting, consider some automated
#      way to handle this.
# Do the following as root on *EACH* of the mail servers.
# Currently the mail servers are:  MX1, MX2, MX3, and WHEEZY-TEST-MX
echo -e "${orgdomain}\n.${orgdomain}" >> /var/qmail/control/rcpthosts
echo "${orgdomain}:${org}mail" >> /var/qmail/control/virtualdomains
  6. Things won't work until various things sync (via cron jobs)
    • passwd-update.sh on all the mail servers (happens on the hour every hour)
    • update-mailtabs.sh on all the mail servers (happens on the half hour every hour)

3. Mailing Lists (ezmlm)

ezmlm is our mailing list manager and is remarkably understandable considering it's from djb. There are extensive manpages, which should be the first place to look. Some questions are only answered by looking at the source, which isn't that ugly.

Currently, we keep most (all?) of our mailing lists under the "ezmlm" user's Maildir.

3.1. Quick tips

Mailing lists are identified by the directory in which the data is stored. In our case, we use directories under /afs/club.cc.cmu.edu/usr/ezmlm/Maildir/, such as /afs/club.cc.cmu.edu/usr/ezmlm/Maildir/announcef07.

3.1.1. Creating a mailing list

To create an announcement-style mailing list, the following will usually work (swapping the name of the list for announcef07): 

# ezmlm-make -5 gripe@club.cc.cmu.edu -m /afs/club.cc.cmu.edu/usr/ezmlm/Maildir/announcef07 /afs/club.cc.cmu.edu/usr/ezmlm/.qmail-announcef07 announcef07 club.cc.cmu.edu

After this, one needs to add this list to /afs/club/service/mail/subusers to add this as subuser of ezmlm. An (currently) hourly cronjob then uses this to update /var/qmail/users/assign and regenerate the corresponding cdb with qmail-newu.

3.1.2. Adding someone to a mailing list

To add someone to a mailing list, run: 

# ezmlm-sub <mailing list directory> <email address>

4. TODOS (when upgrade to wheezy is complete)

Email Aaron before and after putting the last wheezy MX into place.

Update the DSPAM database to the new schema (changes some strings into integers).

Ensure the DSPAM-purge job is running from cron somewhere (once MAGNESIUM is retired).

Start updating old absolute paths into /usr/local.

5. Bring-up for Debian 12

# apt-get install cclub-pre-configuration
# apt-get update
# apt-get install qmail-uids-gids
# apt-get install cclub-base-configuration cclub-afs-client-configuration cclub-passwd-update-configuration cclub-xen-pvh-domu-configuration qmail qmail-mail-transport-agent
# : copy the qmail users from /etc/passwd.OLD -> /etc/passwd.system !
# /usr/share/cclub-scripts/passwd_update_v2.sh

6. TODOs for qmail package

CategoryServices CategoryMemberServices