Differences between revisions 24 and 25

Contrib-CGI

The CGI back-end server gets passed requests for CGI scripts that come into the front-ends.

Jessie

Caution: This is currently a work-in-progress.

Packages to install packages.jessie.contrib-cgi.

Install packages
Set apache to use the prefork MPM
- a2dismod mpm_event && a2enmod mpm_prefork
Setup multilog for apache
- mkdir /var/log/apache2/{access_log,error_log}
- mkfifo /var/log/apache2/{access_log,error_log}/fifo
- Add apache_access_log and apache_error_log multilog setups in /etc/service
PHP
- dpkg-divert --local --divert /usr/bin/php5-cli --rename /usr/bin/php5
- ln -s php5-cgi /usr/bin/php5
- Disable cgi.force_redirect in /etc/php5/cgi/php.ini
Setup apache users and groups
- egrep '^(mycontrib|contribkey|cgi-bin|boguscgi):' /afs/club/service/etc/passwd.service >> /etc/passwd.system
- egrep '^(mycontrib|cgi-bin):' /afs/club/service/etc/group.service >> /etc/group
- touch /etc/passwd.user
- /afs/club/system/scripts/sh/passwd-update.sh # Be careful; don't run if near the hour...
- adduser mycontrib www-data
Apache support stuff
- Setup /var/apache/andrew-contrib (contains org and usr symlink trees)
  - mkdir -p /var/apache/andrew-contrib/{org,usr}
  - touch /var/apache/andrew-contrib/passwd.old
- cwscript (scripts that make contrib go round)
  - Copy from collections
    - As of this writing, cwscript-002 was the latest version - if a newer exists in AFS, do some diffs to double check for breaking changes
    - mkdir -p /usr/local/stow/cwscript-002/cwscript
    - rsync -rltp --exclude .svn /afs/club/system/src/local/cwscript/002/ /usr/local/stow/cwscript-002/cwscript
  - Setup database passwords
    - Files in cwscript-002/cwscript/etc
    - Fix perms first!
      - chmod 0600 /usr/local/stow/cwscript-002/cwscript/etc/*
    - Get from old contrib-cgi or backup
    - Old orgtracker-related stuff is safe to keep running, but will simply fail -- maybe someday we can reconnect with them...
  - A couple things needed to be fixed
    - sed -i -e 's%/etc/apache/contrib-org.conf%/etc/apache2/contrib-org.conf%' /usr/local/stow/cwscript-002/cwscript/contrib_orgs/makeorgconf.pl
    - sed -i -e 's%^/usr/local/bin/apachectl%/usr/sbin/apache2ctl%' /usr/local/stow/cwscript-002/cwscript/cron/contrib_cgi_user_update.sh
  - Install
    - (cd /usr/local/stow && stow cwscript-002)
    - touch /etc/passwd.user
    - Run each of the scripts in the crontab below once, in the listed order
      - The second one complaining about contrib-org.conf not being found is ok; complaint is it trying to create a backup, but there is no pre-existing file to back up
    - /afs/club/system/scripts/sh/passwd-update.sh # Be careful; don't run if near the hour...
    - Setup cron jobs:
```
00 5            * * *   /usr/local/cwscript/cron/contrib_user_update.sh
45 0,12         * * *   /usr/local/cwscript/cron/contrib_cgi_user_update.sh 1> /dev/null
```
- Scary stuff in andrew-contrib-internal
  - Check over and copy from the old machine
  - ??? Ownership and perms of setdbpass need to be checked, was root:Debian-exim on the old host which seems wrong; whole thing is probably obsolete though ???
  - ??? enablecontribkey.sh is referring to gallium, a host that no longer exists; seems wrong ???
- Cgikeys directory
  - mkdir /var/apache/cgikeys && chmod 0771 /var/apache/cgikeys && chown root:mycontrib /var/apache/cgikeys
  - The keyfiles themselves can either be copied or regenerated
    - To regenerate, start with the current list of keys from the old machine
    - Identify which of the users still have Andrew home directories and users, E.g.,
```
for user in `cat old-cgikeys-list`; do
    if grep -q "^${user}:" /etc/passwd &&
       [ -d "/var/apache/andrew-contrib/usr/$user" -o \
         -d "/var/apache/andrew-contrib/org/$user" ]; then
           echo "$user"
    fi
done > confirmed-cgikeys-list
```
    - Then get new keytabs (with Kerberos admin privileges for kadmin)
```
for user in `cat confirmed-cgikeys-list`; do
    ktutil -k "/var/apache/cgikeys/$user" get "contrib/${user}@CLUB.CC.CMU.EDU"
    sleep 0.2
done
```
  - In either case, the permissions need to be fixed/verified
```
(cd /var/apache/cgikeys &&
for key in *; do
    chmod 0640 "$key"
    chown mycontrib:mycontrib "$key"
    setfacl -m "u:${key}:r--" "$key"
done)
```
- Mycontrib directory
  - Copy from old machine, or regenerate parts as desired
  - Need to be careful about ownership and perms

Squeeze

Nonbasic Debian Packages Installed

Note: dependencies are not necessarily included here. Installing the enumerated packages should pull all those in too.

Apache
- apache2-mpm-prefork
- libapache2-mod-auth-kerb (not really used on contrib-cgi, but why not)
PHP
- php5-cgi
- php5-cli
- php5-curl
- php5-gd
- php5-imagick
- php5-ldap
- php5-mysql
- php5-pgsql
- php5-xsl
- php5-sqlite
- php5-remctl
python/python2
- python-xapian
- python-remctl
- python-yaml
- python-sqlite
- python-opencv
- python-gd
- python-mysqldb
- python-pgsql
- python-recaptcha
- python-gdbm
- python-sqlite
- python-webpy
python3
- python3-yaml
- python3-gdbm
- python3-yaml
ruby
- ruby1.8
- ruby1.9
Packages for perl modules installed (not including their dependencies):
- libcrypt-passwdmd5-perl (Crypt::PasswdMD5)
- libdbd-mysql-perl (DBD::mysql)
- libdbd-pg-perl (DBD::pg)
- libdbi-perl (DBI)
- libwww-perl (LWP)
- libcrypt-ssleay-perl (Crypt::SSLeay)
- libgd-gd2-perl (GD)
- libnet-ldap-perl (Net::LDAP)
- libarchive-zip-perl (Archive::Zip)
- libhtml-template-perl (HTML::Template)
- libnet-finger-perl (Net::Finger)
- libgraphics-magick-perl (GraphicsMagick)
- libdbd-sqlite3-perl (DBD::SQLite)
- libnet-remctl-perl (Net::Remctl)
Other nonstandard packages installed:
- acl (extended POSIX ACLs)
- daemontools daemontools-run svtools
Other useful things not likely necessary for operation
- mysql-client
- postgresql-client
- gs

Setup Procedure

Install packages
multilog for apache
- move svscanboot above init.d stuff in /etc/inittab (daemontools-run package bug)
- add apache_access_log and apache_error_log multilog setups in /etc/service
- mkdir /var/log/apache2/{access_log,error_log}
php
- dpkg-divert --divert /usr/bin/php5-cli --rename /usr/bin/php5
- cd /usr/bin
- ln -s php5-cgi php5
- Disable cgi.force_redirect in /etc/php5/cgi/php.ini
apache
- Add users (currently mycontrib, contribkey, cgi-bin, and boguscgi) to /etc/passwd.system
apache support stuff
- setup /var/apache/andrew-contrib (contains org and usr symlink trees)
  - mkdir /var/apache/andrew-contrib/{org,usr}
  - touch /var/apache/andrew-contrib/passwd.old
- cwscript (scripts that make contrib go round)
  - copy cwscript-001 from collections into /usr/local/stow
  - make sure to get sql_*.pm with passwords from old contrib-cgi or backup
  - cd /usr/local/stow; stow cwscript-001
  - add /usr/local/cwscript/cron/contrib_user_update.sh to cron for daily
  - add /usr/local/cwscript/cron/contrib_cgi_user_update.sh to cron for twice-daily
  - run /usr/local/cwscript/cron/contrib_user_update.sh once for setup
  - (etch -> squeeze required update of paths in scripts from /etc/apache -> /etc/apache2)
- keep going on /var/apache
  - copy scary stuff in andrew-contrib-internal
  - copy cgikeys
    - for i in *; do setfacl -m u:$i:r-- $i; done
  - fix owner/group ownership of mycontrib stuff
apache - debian package
- configs in /etc/apache2 - update paths and port to new configfile format as needed
pubcookie debs (cclub)
- install libapache2-mod-pubcookie pubcookie-config pubcookie-key-client
- Copy andrew defaults into /etc/pubcookie/config
- Drop granting, www.contrib, my.contrib keys into /var/lib/pubcookie/keys from previous machine
- Tweak mods-available/pubcookie.conf to set PubcookieAuthTypeNames and comment out their defaults
suexec deb (cclub)
- Forward-port any necessary updates to deb src and rebuild
  - Should be pretty stable across apache-2.2.*
- requires cgi_limits.db in /etc/apache2
  - recompile update_cgi_limits & dump_cgi_limits in andrew-contrib-internal
  - rebuild from cgi_limits.conf using update_cgi_limits if dump_cgi_limits fails to read the db (eg architecture/version change)
binfmt
- copy /var/lib/binfmts from source machine or backup
- Formats so far: python (magic), php (extension) -- do .php, .php5, and .php4 for backwards-compat
Setup /var/log/apache2/cgi for userlogging
- Make sure it's on its own partition
- Copy /var/apache/andrew-contrib-internal/rotate_userlogs.sh and /etc/logrotate.d/cgilogs from prior machine or backup
contribkey (on a remote machine)
- Fetch the source from svn, https://svn.club.cc.cmu.edu/cclub/contributed-webserver/contribkey/trunk, and build the binary (requires heimdal-multidev)
- On the remote machine add a contribkey user with home /var/contribkey
- Install keytab and binary in /var/contribkey (as keytab and contribkey, respectively)
- Setup .ssh/authorized_keys so login using contrib-cgi's private key only allows execution of /var/contribkey/contribkey
- Configure the mycontrib scripts to contact the correct remote machine

Etch

Nonbasic Debian Packages Installed

Note: dependencies are not necessarily included here. Installing the enumerated packages should pull all those in too.

Etch

Apache
- apache2-mpm-prefork
- apache2-prefork-dev
- libapache2-mod-pubcookie (club package; none exists in debian [yet?])
- libapache2-mod-auth-kerb (club package; not really used on contrib-cgi, but why not)
PHP4
- php4
- php4-cgi
- php4-cli
- php4-common
- php4-curl
- php4-dev
- php4-gd
- php4-imagick
- php4-ldap
- php4-mysql
- php4-pgsql
PHP5
- php5
- php5-cgi
- php5-cli
- php5-common
- php5-curl
- php5-dev
- php5-gd
- php5-imagick
- php5-ldap
- php5-mysql
- php5-pgsql
Other scripting langs
- python
- ruby
Packages for perl modules installed (not including their dependencies):
- libcrypt-passwdmd5-perl (Crypt::PasswdMD5)
- libdbd-mysql-perl (DBD::mysql)
- libdbd-pg-perl (DBD::pg)
- libdbi-perl (DBI)
- libwww-perl (LWP)
- libcrypt-ssleay-perl (Crypt::SSLeay)
- libgd-gd2-perl (GD)
- libnet-ldap-perl (Net::LDAP)
- libarchive-zip-perl (Archive::Zip)
- libhtml-template-perl (HTML::Template)
- libnet-finger-perl (Net::Finger)
Other nonstandard packages installed:
- acl (extended POSIX ACLs)
Other useful things not likely necessary for operation
- mysql-client
- postgresql-client
- gs

Setup Procedure

Install packages
apache - debian package
- configs in /etc/apache2
- various files in /var/apache
mod_auth_kerb (cclub) - installed but not really used on contrib-cgi
pubcookie debs (cclub)
- /var/pubcookie
suexec
- right now, divert debian suexec & copy in our own patched version from apache collections
  - dpkg-divert --divert /usr/lib/apache2/suexec.apache --rename /usr/lib/apache2/suexec
- could make this into a deb someday
- requires cgi_limits.db in apache config directory
  - recompile update_cgi_limits & dump_cgi_limits in andrew-contrib-internal
  - rebuild from cgi_limits.conf using update_cgi_limits if dump_cgi_limits fails to read the db (eg architecture/version change)
multilog for apache
- symlink /var/apache/logs/apache* to /var/service
- mess with djbdaemon foo as necessary
cwscript collection
- put cron/contrib_{user,cgi_user}_update.sh in cron
add mycontrib to /etc/passwd.system
binfmt
- /var/lib/binfmts
php
- dpkg-divert --divert /usr/bin/php4-cli --rename /usr/bin/php4
- dpkg-divert --divert /usr/bin/php5-cli --rename /usr/bin/php5
- cd /usr/bin
- ln -s php4-cgi php4
- ln -s php5-cgi php5

CategoryServices CategoryContributedServices

Deletions are marked like this.	Additions are marked like this.
Line 47:	Line 47:
* '''??? Is the !OrgTracker stuff still relevant; host of interest doesn't resolve ???'''	* Old orgtracker-related stuff is safe to keep running, but will simply fail -- maybe someday we can reconnect with them...