Currently our mail servers are mx1, mx2, and mx3.

We use qmail as our MTA, and have it configured to deliver messages to users' mail volumes in AFS (mounted in $HOME/Maildir).

Contents

  1. Mail (qmail)
    1. Packages
    2. Custom programs and scripts for /usr/local
    3. Installation
    4. Configuration
    5. Startup
  2. Mail for Orgs
  3. Mailing Lists (ezmlm)
    1. Quick tips
  4. Old TODOS (when upgrade to wheezy is complete)
  5. TODOs
    1. for qmail package
    2. for wrapper-symlinks
    3. for wrapit
    4. for dspam
    5. for qmail-qfilter
  6. Scratch work

1. Mail (qmail)

1.1. Packages

1.1.1. Unmodified debian packages

1.1.2. Packages that required minor modifications

1.1.3. Packages that required major modifications or were not in Debian

1.2. Custom programs and scripts for /usr/local

1.3. Installation

Note that you need to be a little careful during clubification, where cclub-base-configuration would by default install exim4-- the basic MTA we run on general purpose (i.e., non-mail) servers.

If you clubify normally, you can switch from using exim4 as the mail-transport-agent provider to using qmail, with: 

apt-get install -exim4 -exim4-base -exim4-daemon-light +qmail +qmail-mail-transport-agent

Alternatively, you can install qmail and qmail-mail-transport-agent together with cclub-base-configuration. E.g., a full command performing clubification might look like: 

apt-get install cclub-base-configuration cclub-afs-client-configuration cclub-passwd-update-configuration cclub-xen-pvh-domu-configuration qmail qmail-mail-transport-agent

When you install qmail, it will pull in the qmail-uids-gids package as a dependency. It creates users with uids that are not in the standard range for system users. This means you will need to manually save off the users to /etc/passwd.system! 

grep -h '^\(alias\|qmail.\):' /etc/passwd /etc/passwd.OLD | sort | uniq >> /etc/passwd.system

All club users must be present on mail servers. To do this (as root): 

touch /etc/passwd.user
rm /etc/passwd.core
/usr/share/cclub-scripts/passwd_update_v2.sh

Install the rest of the Debian packages listed above.

Build and install the wrappers: 

cd /tmp
svn co https://svn.club.cc.cmu.edu/cclub/wrapit/trunk ./wrapit
cd wrapit
make

# Remaining steps require root

mkdir -p /usr/local/stow/wrappers-001
make DESTDIR=/usr/local/stow/wrappers-001 install

cd /usr/local/stow/wrappers-001
mv ./usr/local/* .
rmdir ./usr/local
rmdir ./usr
mv ./var/log/wrapit /var/log
rmdir ./var/log
rmdir ./var

ln -s /var/log/wrapit/svc /etc/service/wrapit-logs

cd ./bin
svn export https://svn.club.cc.cmu.edu/cclub/scripts/trunk/perl/ezmlm-issubn.pl ./ezmlm-issubn.pl
# Assumes you've downloaded the wrapper-symlinks.tar.gz, which is attached to this page, somewhere locally
tar -zxvf /path/to/wrapper-symlinks.tar.gz

cd /usr/local/stow
stow wrappers-001

1.4. Configuration

The easiest way to configure qmail for cclub, is to start with configuration from an existing mail server.

1.4.1. /var/qmail/control

Most of qmail's configuration exists as several files in the /var/qmail/control directory. For a list of such files, see qmail-control (5), though please note that our qmail supports additional control files (see the actual man page on one of the mailservers for a complete list).

1.4.1.1. Things that need to be changed if you start with an existing mail server's files

Edit me --> local FQDN.

Edit locals --> add local FQDN to the list. Copy the new version of locals to all the other mail servers.

1.4.1.2. Descriptions of the control files, some with cclub's values

Here are some important control files, and how they are configured for the cclub environment:

bouncehost
host part of sender address for bounce messages 
club.cc.cmu.edu
concurrencylocal
maximum number of local deliveries to perform simultaneously 
5
concurrencyremote
maximum number of remote deliveries to perform simultaneously 
30
defaultdomain
domain to use when a recipient address is given with an unqualified hostname 
club.cc.cmu.edu
defaulthost
domain to use when a recipient address is given without any hostname 
club.cc.cmu.edu
locals
domains that are handled by local delivery (list below is valid as of 10 Feb 2014): 
localhost
club.cc.cmu.edu
«hostname».club.cc.cmu.edu
thorin.dementia.org
thorin.club.cc.cmu.edu
aberrant.org
cmucc.org
me
the fully-qualified hostname of the machine qmail is running on 
«hostname».club.cc.cmu.edu
plusdomain
what should be appended to recipient addresses given with a trailing '+' character 
cmu.edu
queuelifetime
how long to keep a message in the queue before treating temporary delivery failures as permanent 
345600
rcptchecks
programs to run that determine whether a recipient of a non-relayed message is valid, see qmail-rcptcheck(8) 
/usr/sbin/qmail-rcptcheck-realrcptto
rcpthosts

domains for which mail will be accepted, with wildcards indicated by a leading '.' character

This list is quite lengthy (at some point we may want to go through this and identify domains in it that we are no longer hosting mail for, and remove them). I'm not including it here, due to its length. It should be fine to copy the file off of some existing mail server.

servercert.pem
concatenated, PEM encoded SSL private key, CA key chain, and SSL certificate
smtpfilters
list of filters to run on messages injected into the system via SMTP, see qmail-qfilter-queue(8)
smtpplugins
configuration file specifying the plugins to run when SMTP commands are executed 
[rcpt]
:if [ "${RELAYCLIENT-unset}" = unset ]; then exec /usr/sbin/qmail-rcptcheck; fi
smtproutes
can be used override the server outgoing mail is directed to
spfbehavior
controls SPF validation: whether it's done, and how to treat failures 
1
timeoutremote
how long to wait for responses from remote SMTP servers 
300
tlsclientciphers
overrides the TLS ciphers used by qmail-remote; to deal with sending things to our wheezy mailservers, I set: 
HIGH:MEDIUM:!MD5:!RC4:!3DES:@SECLEVEL=0
virtualdomains

maps "virtual" users and domains to local user accounts

This list is quite lengthy (at some point we may want to go through this and identify domains in it that we are no longer hosting mail for, and remove them). I'm not including it here, due to its length. It should be fine to copy the file off of some existing mail server.

1.4.2. /var/keys

Create the directory.

Extract the "mailtabs" keytab to /var/keys/mailtabs. 

mkdir -p /var/keys
kinit -S kadmin/admin «username»/admin
kadmin ext -k /var/keys/mailtabs mailtabs
kdestroy

The mailtabs will be added to the directory by update-mailtabs.sh.

1.4.3. Cron jobs

These all run as root. Probably a good idea to run all of them first manually. 

# Mail stuff
00 *            * * *   /afs/club.cc.cmu.edu/system/scripts/perl/mailassign.pl /afs/club/user > /var/qmail/users/assign && /usr/sbin/qmail-newu
02 *            * * *   /afs/club.cc.cmu.edu/system/scripts/sh/update-alias.sh
30 *            * * *   /afs/club.cc.cmu.edu/system/scripts/sh/update-mailtabs.sh
00 0            * * *   /usr/sbin/update_tmprsadh > /dev/null 2>&1

1.4.4. SMTP Access Control

See tcprules(1) for information on the rule files' syntax. See qmail-smtpd(8) for the behaviors the environment variables control.

For the standard SMTP service (port 25), allow relaying for club machines. For non-club machines, optionally support authentication and relay for authenticated users. Otherwise, non-club machines are only allowed to send mail addressed to hosts/domains in rcpthosts.

/var/qmail/tcp.smtp: 

# B6 machines
128.237.157.:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
172.29.24.:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
# Gates machines
128.2.220.21:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
128.2.220.33:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
128.2.220.55:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
128.2.220.97:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
128.2.220.100:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
128.2.220.175:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
# loopback IPs
127.:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",RELAYCLIENT="",SMTPAUTH=""
# catch-all
:allow,QMAILQUEUE="/usr/sbin/qmail-qfilter-queue",QMAILQUEUEFILTERS="control/smtpfilters",SMTPAUTH=""

For the submission service (port 587), always require authentication.

/var/qmail/tcp.submission: 

:allow,SMTPAUTH="!"

To use the rules with tcpserver, they need to be compiled into a .cdb: 

tcprules /var/qmail/tcp.smtp.{cdb,tmp} < /var/qmail/tcp.smtp
tcprules /var/qmail/tcp.submission.{cdb,tmp} < /var/qmail/tcp.submission

1.4.5. SMTP Authentication

We use checkpassword-pam to authenticate club users connecting to SMTP. This requires a modified PAM configuration.

/etc/pam.d/smtpd: 

#
# /etc/pam.d/smtpd - PAM settings for SMTP-AUTH
#

# Unfortunately, for this case, we can't use any of the common-* files.
#   - pam_krb5.so, by default, wants to create FILE ccaches and chown them to
#     the authenticated user; that would fail, since we don't run as root
#   - we want to support mail passwords; setting that up requires additional
#     arguments for pam_krb5.so
#   - there's no need to create a PAG or get AFS tokens

auth     required       pam_krb5.so minimum_uid=110 alt_auth_map=%s/email \
                                keytab=/var/qmail/smtp.keytab ccache=MEMORY:

account  required       pam_krb5.so minimum_uid=110 alt_auth_map=%s/email \
                                keytab=/var/qmail/smtp.keytab ccache=MEMORY:

password required       pam_deny.so

session  required       pam_permit.so
session  optional       pam_krb5.so minimum_uid=110 alt_auth_map=%s/email \
                                keytab=/var/qmail/smtp.keytab ccache=MEMORY:

What's this all about?

alt_auth_map=%s/email
  • This allows us to provide alternate passwords that users can use for email. E.g., this would allow users to use a different password than there regular cclub password for email access from mobile devices.
keytab=/var/qmail/smtp.keytab
  • A Kerberos-enabled server uses a shared keytab with the KDC in order to authenticate the KDC (i.e., to prevent somebody to succeed in authenticating by spoofing the KDC). Usually /etc/krb5.keytab is used, but since checkpassword-pam is not run as root, it is not readable. To work around, we create a dedicated keytab for the smtpd to use.
ccache=MEMORY:
  • For a login user's convenience, Kerberos authentication will create a ticket file for the authentication user in /tmp, and then chown it to the authenticated user. This does not work when checkpassword-pam is not run from. For mail, the tickets are needed, so we can solve the problem by using an in-memory only credentials cache.

The next step is actually creating the above-mentioned keytab: 

kinit -S kadmin/admin user/admin
ktutil -k /var/qmail/smtp.keytab get SMTP/freedom.club.cc.cmu.edu
chmod 400 /var/qmail/smtp.keytab
chown qmaild:root /var/qmail/smtp.keytab

1.4.6. DSPAM Configuration

A few things need to change for the Cclub environment:

/etc/dspam/dspam.conf

StorageDriver ---> /usr/lib/x86_64-linux-gnu/dspam/libmysql_drv.so
TrustedDeliveryAgent ---> "/bin/cat"
Add: UntrustedDeliveryAgent "/bin/cat"
Remove: all Trust lines except for Trust root

/etc/dspam/dspam.d/mysql.conf
Fill in with the details for the club MySQL server.

1.5. Startup

To have qmail start on boot, create supervise service directories, link to them from /etc/services, and then create wrappers in /etc/init.d.

1.5.1. Create supervise service directories

cd /var/qmail

########################################################################
# qmail system
########################################################################

mkdir -p svc/qmail/log/main
cat > svc/qmail/run << EOF
#!/bin/sh
exec /usr/sbin/qmail-start ./Maildir/
EOF
chmod 0755 svc/qmail/run
touch svc/qmail/down
cat > svc/qmail/log/run << EOF
#!/bin/sh
umask 0077
exec multilog t s10485760 n15 ./main
EOF
chmod 0755 svc/qmail/log/run
ln -s /var/qmail/svc/qmail/log/main /var/log/qmail

########################################################################
# normal smtpd
########################################################################

mkdir -p svc/smtp/log/main
cat > svc/smtp/run << EOF
#!/bin/sh
exec /usr/bin/tcpserver -v -R -x /var/qmail/tcp.smtp.cdb \\
                        -u 1001 -g 1002 0 smtp \\
                        /usr/bin/rblsmtpd -b -r "zen.spamhaus.org" \\
                        /usr/sbin/qmail-smtpd \\
                        /usr/bin/checkpassword-pam -e -s smtpd /bin/true 2>&1
EOF
chmod 0755 svc/smtp/run
touch svc/smtp/down
cat > svc/smtp/log/run << EOF
#!/bin/sh
umask 0077
exec multilog t s10485760 n15 ./main
EOF
chmod 0755 svc/smtp/log/run
ln -s /var/qmail/svc/smtp/log/main /var/log/qmail-smtp

########################################################################
# submission smtpd
########################################################################

mkdir -p svc/submission/log/main
cat > svc/submission/run << EOF
#!/bin/sh
exec /usr/bin/tcpserver -v -R -x /var/qmail/tcp.submission.cdb \\
                        -u 1001 -g 1002 0 submission \\
                        /usr/sbin/qmail-smtpd \\
                        /usr/bin/checkpassword-pam -e -s smtpd /bin/true 2>&1
EOF
chmod 0755 svc/submission/run
touch svc/submission/down
cat > svc/submission/log/run << EOF
#!/bin/sh
umask 0077
exec multilog t s10485760 n15 ./main
EOF
chmod 0755 svc/submission/log/run
ln -s /var/qmail/svc/submission/log/main /var/log/qmail-submission



ln -s /var/qmail/svc/qmail /etc/service/qmail
ln -s /var/qmail/svc/smtp /etc/service/smtp
ln -s /var/qmail/svc/submission /etc/service/submission

1.5.3. Create wrappers in /etc/init.d

svinitd-create qmail | sed -e '/^#!/a\' \
  -e '\' \
  -e '### BEGIN INIT INFO\' \
  -e '# Provides:          qmail\' \
  -e '# Default-Start:     2 3 4 5' \
  -e '# Default-Stop:      0 1 6' \
  -e '# Required-Start:    $local_fs $network daemontools-run\' \
  -e '# Required-Stop:     $network\' \
  -e '# Short-Description: qmail\' \
  -e '# Description:       The qmail mail transport agent queue management and\' \
  -e '#                    delivery processes.\' \
  -e '### END INIT INFO' \
  > /etc/init.d/qmail
chmod 0755 /etc/init.d/qmail

svinitd-create smtp submission | sed -e '/^#!/a\' \
  -e '\' \
  -e '### BEGIN INIT INFO\' \
  -e '# Provides:          smtpd\' \
  -e '# Default-Start:     2 3 4 5' \
  -e '# Default-Stop:      0 1 6' \
  -e '# Required-Start:    $local_fs $network daemontools-run\' \
  -e '# Required-Stop:     $network\' \
  -e '# Short-Description: smtpd\' \
  -e '# Description:       The qmail mail transport agent smtp daemon, receiving\' \
  -e '#                    mail via (E)SMTP(S)(A) on ports 25 and 587.\' \
  -e '### END INIT INFO' \
  > /etc/init.d/smtpd
chmod 0755 /etc/init.d/smtpd

1.5.4. Enable the services at boot

update-rc.d qmail defaults
update-rc.d smtpd defaults

2. Mail for Orgs

I set up mail this way for the KGB.

Unless otherwise noted, each step requires admin tokens.

  1. Create PTS users for the organization's mail service 
    pts createuser ${org}mail
# Note the pts ID assigned to the user.  E.g.,
# ----
# kbare@conch:~$ pts createuser ${org}mail
# User ${org}mail has id 1891
#                        ^^^^
pts createuser ${org}mail.mail
  2. Add the organization's user to passwd.user 
    echo "${org}mail:x:${uid}:20:${long_org_name} Mail Forwarding:${org_subdir_in_andrew_afs}:/bin/false" \
     >> /afs/club.cc.cmu.edu/service/etc/passwd.user
  3. Create a symlink in /afs/club.cc.cmu.edu/usr pointing to the org's directory 
    ln -s ${org_subdir_in_andrew_afs} /afs/.club.cc.cmu.edu/usr/${org}mail
vos release club.usr
  4. Create a Kerberos mail principal, extract it to the mailtabs directory 
    kadmin ank -r --use-defaults ${org}mail/mail
kadmin ext -k /afs/club.cc.cmu.edu/service/mail/mailtabs/${org}mail ${org}mail/mail
chown ${uid}:dialout /afs/club.cc.cmu.edu/service/mail/mailtabs/${org}mail
  5. Add the hosted domain to the mailserver configurations 
    # XXX: if we end up doing a lot of org mail hosting, consider some automated
#      way to handle this.
# Do the following as root on *EACH* of the mail servers.
# Currently the mail servers are:  MX1, MX2, MX3, and WHEEZY-TEST-MX
echo -e "${orgdomain}\n.${orgdomain}" >> /var/qmail/control/rcpthosts
echo "${orgdomain}:${org}mail" >> /var/qmail/control/virtualdomains
  6. Things won't work until various things sync (via cron jobs)
    • passwd-update.sh on all the mail servers (happens on the hour every hour)
    • update-mailtabs.sh on all the mail servers (happens on the half hour every hour)

3. Mailing Lists (ezmlm)

ezmlm is our mailing list manager and is remarkably understandable considering it's from djb. There are extensive manpages, which should be the first place to look. Some questions are only answered by looking at the source, which isn't that ugly.

Currently, we keep most (all?) of our mailing lists under the "ezmlm" user's Maildir.

3.1. Quick tips

Mailing lists are identified by the directory in which the data is stored. In our case, we use directories under /afs/club.cc.cmu.edu/usr/ezmlm/Maildir/, such as /afs/club.cc.cmu.edu/usr/ezmlm/Maildir/announcef07.

3.1.1. Creating a mailing list

To create an announcement-style mailing list, the following will usually work (swapping the name of the list for announcef07): 

# ezmlm-make -5 gripe@club.cc.cmu.edu -m /afs/club.cc.cmu.edu/usr/ezmlm/Maildir/announcef07 /afs/club.cc.cmu.edu/usr/ezmlm/.qmail-announcef07 announcef07 club.cc.cmu.edu

After this, one needs to add this list to /afs/club/service/mail/subusers to add this as subuser of ezmlm. An (currently) hourly cronjob then uses this to update /var/qmail/users/assign and regenerate the corresponding cdb with qmail-newu.

3.1.2. Adding someone to a mailing list

To add someone to a mailing list, run: 

# ezmlm-sub <mailing list directory> <email address>

4. Old TODOS (when upgrade to wheezy is complete)

Email Aaron before and after putting the last wheezy MX into place.

Update the DSPAM database to the new schema (changes some strings into integers).

Ensure the DSPAM-purge job is running from cron somewhere (once MAGNESIUM is retired).

Start updating old absolute paths into /usr/local.

5. TODOs

5.1. for qmail package

5.3. for wrapit

5.4. for dspam

5.5. for qmail-qfilter

6. Scratch work

apt-get install python3-srs python3-dkim python3-authres



root@freedom:~# dpkg-query -W python3-srs python3-dkim python3-authres  
python3-authres 1.2.0-3
python3-dkim    1.1.4-1
python3-srs     1.0.4-2

adduser --system --ingroup nofiles --home /var/qmail/srs srs
echo "+SRS0=:srs:`getent passwd srs | cut -d : -f 3-4`:/var/qmail/srs:-::" > /var/qmail/srs/assign
echo "+SRS1=:srs:`getent passwd srs | cut -d : -f 3-4`:/var/qmail/srs:-::" >> /var/qmail/srs/assign

Change the crontab entry: 

00 *            * * *   { cat /var/qmail/srs/assign; /afs/club.cc.cmu.edu/system/scripts/perl/mailassign.pl /afs/club/user; } > /var/qmail/users/assign && /usr/sbin/qmail-newu

Extract keytab for srs/mail into /var/qmail/srs/keytab. Chown it to srs:nofiles.

Need config.json. Owned by srs:nofiles, 0600, but with ACL allowing qmailr and qmaild read permissions. 

chown srs:nofiles ./config.json
setfacl -m u:qmaild:r-- -m u:qmailr:r-- ./config.json

.qmail-default (also needs to be owned by srs:nofiles): 

|/usr/local/bin/srs-unwrap-and-forward

Logging for srs-wrap-and-send: 

root@independence:~# mkdir -p /var/log/srs-wrap/svc
root@independence:~# mkfifo /var/log/srs-wrap/fifo
root@independence:~# chmod 622 /var/log/srs-wrap/fifo

cat > /var/log/srs-wrap/svc/run << EOF
#!/bin/sh
exec 3<> /var/log/srs-wrap/fifo
exec < /var/log/srs-wrap/fifo
exec multilog t s1048576 n25 /var/log/srs-wrap
EOF
chmod 0755 /var/log/srs-wrap/svc/run

ln -s /var/log/srs-wrap/svc /etc/service/srs-wrap-logs

CategoryServices CategoryMemberServices

Services/Club Mail (last edited 2024-03-06 02:28:17 by kbare@CLUB.CC.CMU.EDU)