• Services
• Contrib CGI

Contrib-CGI

The CGI back-end server gets passed requests for CGI scripts that come into the front-ends.

Jessie

Caution: This is currently a work-in-progress.

Packages to install packages.jessie.contrib-cgi.

• Install packages
• Set apache to use the prefork MPM

  • a2dismod mpm_event && a2enmod mpm_prefork

• Setup multilog for apache
  • mkdir /var/log/apache2/{access_log,error_log}
  • mkfifo /var/log/apache2/{access_log,error_log}/fifo
  • Add apache_access_log and apache_error_log multilog setups in /etc/service
• PHP
  • Make "php5" run php5-cgi instead of php5-cli by default
    • dpkg-divert --local --divert /usr/bin/php5-cli --rename /usr/bin/php5
    • ln -s php5-cgi /usr/bin/php5
    • (Last time this broke /usr/lib/php5/sessionclean run by cron -- it has a place where it invokes "php5" expecting php5-cgi -- so just change that)
  • Make a couple modifications to /etc/php5/cgi/php.ini
    • Disable cgi.force_redirect
    • Enable short_open_tag
• Setup apache users and groups

  • egrep '^(mycontrib|contribkey|cgi-bin|boguscgi):' /afs/club/service/etc/passwd.service >> /etc/passwd.system

  • egrep '^(mycontrib|cgi-bin):' /afs/club/service/etc/group.service >> /etc/group

  • touch /etc/passwd.user
  • /afs/club/system/scripts/sh/passwd-update.sh # Be careful; don't run if near the hour...
  • adduser mycontrib www-data
• Apache support stuff
  • Setup /var/apache/andrew-contrib (contains org and usr symlink trees)
    • mkdir -p /var/apache/andrew-contrib/{org,usr}
    • touch /var/apache/andrew-contrib/passwd.old
  • cwscript (scripts that make contrib go round)
    • Copy from collections
      • As of this writing, cwscript-002 was the latest version - if a newer exists in AFS, do some diffs to double check for breaking changes
      • mkdir -p /usr/local/stow/cwscript-002/cwscript
      • rsync -rltp --exclude .svn /afs/club/system/src/local/cwscript/002/ /usr/local/stow/cwscript-002/cwscript
    • Setup database passwords
      • Files in cwscript-002/cwscript/etc
      • Fix perms first!
        • chmod 0600 /usr/local/stow/cwscript-002/cwscript/etc/*
      • Get from old contrib-cgi or backup
      • Old orgtracker-related stuff is safe to keep running, but will simply fail -- maybe someday we can reconnect with them...
    • A couple things needed to be fixed
      • sed -i -e 's%/etc/apache/contrib-org.conf%/etc/apache2/contrib-org.conf%' /usr/local/stow/cwscript-002/cwscript/contrib_orgs/makeorgconf.pl
      • sed -i -e 's%^/usr/local/bin/apachectl%/usr/sbin/apache2ctl%' /usr/local/stow/cwscript-002/cwscript/cron/contrib_cgi_user_update.sh
    • Install

      • (cd /usr/local/stow && stow cwscript-002)

      • touch /etc/passwd.user
      • Run each of the scripts in the crontab below once, in the listed order
        • The second one complaining about contrib-org.conf not being found is ok; complaint is it trying to create a backup, but there is no pre-existing file to back up
      • /afs/club/system/scripts/sh/passwd-update.sh # Be careful; don't run if near the hour...

      • Setup cron jobs:

        00 5            * * *   /usr/local/cwscript/cron/contrib_user_update.sh
        45 0,12         * * *   /usr/local/cwscript/cron/contrib_cgi_user_update.sh 1> /dev/null

  • Scary stuff in andrew-contrib-internal
    • Check over and copy from the old machine
      • There was some cruft in here the most recent time this process was performed, and blatantly old stuff put in old/
      • A little more harmless cruft related to org wikis etc. is still here
    • Mycontrib directory
      • Check/fix perms - ensure dir is 750 root.mycontrib, files are 640 root.mycontrib
      • Check contents if any values have changed
  • Cgikeys directory

    • mkdir /var/apache/cgikeys && chmod 0771 /var/apache/cgikeys && chown root:mycontrib /var/apache/cgikeys

    • The keyfiles themselves can either be copied or regenerated
      • To regenerate, start with the current list of keys from the old machine

      • Identify which of the users still have Andrew home directories and users, E.g.,

        for user in `cat old-cgikeys-list`; do
            if grep -q "^${user}:" /etc/passwd &&
               [ -d "/var/apache/andrew-contrib/usr/$user" -o \
                 -d "/var/apache/andrew-contrib/org/$user" ]; then
                   echo "$user"
            fi
        done > confirmed-cgikeys-list

      • Then get new keytabs (with Kerberos admin privileges for kadmin)

        for user in `cat confirmed-cgikeys-list`; do
            ktutil -k "/var/apache/cgikeys/$user" get "contrib/${user}@CLUB.CC.CMU.EDU"
            sleep 0.2
        done

    • In either case, the permissions need to be fixed/verified

      (cd /var/apache/cgikeys &&
      for key in *; do
          chmod 0640 "$key"
          chown mycontrib:mycontrib "$key"
          setfacl -m "u:${key}:r--" "$key"
      done)

• apache - use debian package
  • configs in /etc/apache2 - update paths and port to new configfile format as needed

  • copy configs from prior machine or backup, check over for validity with current apache version & config format

  • Key changes for global configuration:
    • log file paths (use the fifos for multilog), request log file format
  • Disable some things Debian turns on by default that we do not want
    • a2disconf serve-cgi-bin.conf # we don't want to serve CGI scripts the way it does

    • Add modules-available/shib2.conf, with "ShibDisable On"; we re-enable it on a vserver-by-vserver basis

  • Enable some other things
    • Modules (with a2enmod): authn_dbm authz_dbm authz_groupfile cgi imagemap include rewrite shib2 ssl suexec userdir

  • FIXME: still a work in-progress

• suexec deb (cclub)
  • Generate apache2-suexec-contrib deb package if not already existing
    • Forward-port any necessary updates to deb src and rebuild (should be pretty stable ish, hah...)
  • requires cgi_limits.db in /etc/apache2

    • recompile update_cgi_limits & dump_cgi_limits in andrew-contrib-internal

    • copy /etc/apache2/cgi_limits.{db,conf} from prior machine or backup
    • rebuild from cgi_limits.conf using update_cgi_limits if dump_cgi_limits fails to read the db (eg architecture/version change)
      • just run update_cgi_limits from /etc/apache2 with no arguments
      • verify with dump_cgi_limits *default
  • sanity check paths
    • /usr/lib/apache2/suexec -V
• pubcookie debs (cclub)
  • Uh oh, this looks really dead - luckily don't need on contrib-cgi we think
• shibboleth
  • First install shib package in package list
    • Note we're using special cclub build that enables cfg directives in a few new places, fixes ssl redirect behavior, ...

  • Copy contents of /etc/shibboleth (configs & keys) from prior machine or backup

    • Format changes slightly frequently - may want to diff and merge to use new format
  • Setup shibd access to centralized shibd over vpn to central shibd (currently on "shibboleth" machine)
    • Disable shibd daemon (systemctl disable shibd I think?)

    • Copy contents of /etc/openvpn (configs & keys) from prior machine or backup

    • Edit /etc/default/openvpn and set AUTOSTART="shibd"

    • Start openvpn & verify can ping server over vpn

    • Make sure local vpn IP is in the acl list in /etc/shibboleth/shibboleth2.xml on the server (must be enumerated, doesn't do CIDR or globbing, sigh)
• binfmt
  • The python packages automatically install configurations for executing compiled python code
  • We also add support executing PHP scripts based on extension (.php, .php5, and .php4)
    • update-binfmts --install PHP /usr/bin/php5 --extension php
    • update-binfmts --install PHP5 /usr/bin/php5 --extension php5
    • update-binfmts --install PHP4 /usr/bin/php5 --extension php4
• Setup /var/log/apache2/cgi for user CGI stderr output logging
  • Make sure it's on its own partition
  • Copy /var/apache/andrew-contrib-internal/rotate_userlogs.sh and /etc/logrotate.d/cgilogs from prior machine or backup
  • suexec will just know to put logs here
• Copy over /etc/rc.local from prior or backup
  • Sanity check it
• Test stuff
  • Access everything in ~cl0x from a browser, check it prints what it should (cat the sources)

Squeeze

Nonbasic Debian Packages Installed

Note: dependencies are not necessarily included here. Installing the enumerated packages should pull all those in too.

• Apache
  • apache2-mpm-prefork
  • libapache2-mod-auth-kerb (not really used on contrib-cgi, but why not)
• PHP
  • php5-cgi
  • php5-cli
  • php5-curl
  • php5-gd
  • php5-imagick
  • php5-ldap
  • php5-mysql
  • php5-pgsql
  • php5-xsl
  • php5-sqlite
  • php5-remctl
• python/python2
  • python-xapian
  • python-remctl
  • python-yaml
  • python-sqlite
  • python-opencv
  • python-gd
  • python-mysqldb
  • python-pgsql
  • python-recaptcha
  • python-gdbm
  • python-sqlite
  • python-webpy
• python3
  • python3-yaml
  • python3-gdbm
  • python3-yaml
• ruby
  • ruby1.8
  • ruby1.9
• Packages for perl modules installed (not including their dependencies):
  • libcrypt-passwdmd5-perl (Crypt::PasswdMD5)
  • libdbd-mysql-perl (DBD::mysql)
  • libdbd-pg-perl (DBD::pg)
  • libdbi-perl (DBI)
  • libwww-perl (LWP)
  • libcrypt-ssleay-perl (Crypt::SSLeay)
  • libgd-gd2-perl (GD)
  • libnet-ldap-perl (Net::LDAP)
  • libarchive-zip-perl (Archive::Zip)
  • libhtml-template-perl (HTML::Template)
  • libnet-finger-perl (Net::Finger)

  • libgraphics-magick-perl (GraphicsMagick)

  • libdbd-sqlite3-perl (DBD::SQLite)
  • libnet-remctl-perl (Net::Remctl)
• Other nonstandard packages installed:
  • acl (extended POSIX ACLs)
  • daemontools daemontools-run svtools
• Other useful things not likely necessary for operation
  • mysql-client
  • postgresql-client
  • gs

Setup Procedure

• Install packages
• multilog for apache
  • move svscanboot above init.d stuff in /etc/inittab (daemontools-run package bug)
  • add apache_access_log and apache_error_log multilog setups in /etc/service
  • mkdir /var/log/apache2/{access_log,error_log}
• php
  • dpkg-divert --divert /usr/bin/php5-cli --rename /usr/bin/php5
  • cd /usr/bin
  • ln -s php5-cgi php5
  • Disable cgi.force_redirect in /etc/php5/cgi/php.ini
• apache
  • Add users (currently mycontrib, contribkey, cgi-bin, and boguscgi) to /etc/passwd.system
• apache support stuff
  • setup /var/apache/andrew-contrib (contains org and usr symlink trees)
    • mkdir /var/apache/andrew-contrib/{org,usr}
    • touch /var/apache/andrew-contrib/passwd.old
  • cwscript (scripts that make contrib go round)
    • copy cwscript-001 from collections into /usr/local/stow
    • make sure to get sql_*.pm with passwords from old contrib-cgi or backup
    • cd /usr/local/stow; stow cwscript-001
    • add /usr/local/cwscript/cron/contrib_user_update.sh to cron for daily
    • add /usr/local/cwscript/cron/contrib_cgi_user_update.sh to cron for twice-daily
    • run /usr/local/cwscript/cron/contrib_user_update.sh once for setup

    • (etch -> squeeze required update of paths in scripts from /etc/apache -> /etc/apache2)

  • keep going on /var/apache
    • copy scary stuff in andrew-contrib-internal
    • copy cgikeys
      • for i in *; do setfacl -m u:$i:r-- $i; done
    • fix owner/group ownership of mycontrib stuff
• apache - debian package
  • configs in /etc/apache2 - update paths and port to new configfile format as needed
• pubcookie debs (cclub)
  • install libapache2-mod-pubcookie pubcookie-config pubcookie-key-client
  • Copy andrew defaults into /etc/pubcookie/config
  • Drop granting, www.contrib, my.contrib keys into /var/lib/pubcookie/keys from previous machine

  • Tweak mods-available/pubcookie.conf to set PubcookieAuthTypeNames and comment out their defaults

• suexec deb (cclub)
  • Forward-port any necessary updates to deb src and rebuild
    • Should be pretty stable across apache-2.2.*
  • requires cgi_limits.db in /etc/apache2

    • recompile update_cgi_limits & dump_cgi_limits in andrew-contrib-internal

    • rebuild from cgi_limits.conf using update_cgi_limits if dump_cgi_limits fails to read the db (eg architecture/version change)
• binfmt
  • copy /var/lib/binfmts from source machine or backup
  • Formats so far: python (magic), php (extension) -- do .php, .php5, and .php4 for backwards-compat
• Setup /var/log/apache2/cgi for userlogging
  • Make sure it's on its own partition
  • Copy /var/apache/andrew-contrib-internal/rotate_userlogs.sh and /etc/logrotate.d/cgilogs from prior machine or backup
• contribkey (on a remote machine)

  • Fetch the source from svn, https://svn.club.cc.cmu.edu/cclub/contributed-webserver/contribkey/trunk, and build the binary (requires heimdal-multidev)

  • On the remote machine add a contribkey user with home /var/contribkey
  • Install keytab and binary in /var/contribkey (as keytab and contribkey, respectively)
  • Setup .ssh/authorized_keys so login using contrib-cgi's private key only allows execution of /var/contribkey/contribkey
  • Configure the mycontrib scripts to contact the correct remote machine

Etch

Nonbasic Debian Packages Installed

Note: dependencies are not necessarily included here. Installing the enumerated packages should pull all those in too.

Etch

• Apache
  • apache2-mpm-prefork
  • apache2-prefork-dev
  • libapache2-mod-pubcookie (club package; none exists in debian [yet?])
  • libapache2-mod-auth-kerb (club package; not really used on contrib-cgi, but why not)
• PHP4
  • php4
  • php4-cgi
  • php4-cli
  • php4-common
  • php4-curl
  • php4-dev
  • php4-gd
  • php4-imagick
  • php4-ldap
  • php4-mysql
  • php4-pgsql
• PHP5
  • php5
  • php5-cgi
  • php5-cli
  • php5-common
  • php5-curl
  • php5-dev
  • php5-gd
  • php5-imagick
  • php5-ldap
  • php5-mysql
  • php5-pgsql
• Other scripting langs
  • python
  • ruby
• Packages for perl modules installed (not including their dependencies):
  • libcrypt-passwdmd5-perl (Crypt::PasswdMD5)
  • libdbd-mysql-perl (DBD::mysql)
  • libdbd-pg-perl (DBD::pg)
  • libdbi-perl (DBI)
  • libwww-perl (LWP)
  • libcrypt-ssleay-perl (Crypt::SSLeay)
  • libgd-gd2-perl (GD)
  • libnet-ldap-perl (Net::LDAP)
  • libarchive-zip-perl (Archive::Zip)
  • libhtml-template-perl (HTML::Template)
  • libnet-finger-perl (Net::Finger)
• Other nonstandard packages installed:
  • acl (extended POSIX ACLs)
• Other useful things not likely necessary for operation
  • mysql-client
  • postgresql-client
  • gs

Setup Procedure

• Install packages
• apache - debian package
  • configs in /etc/apache2
  • various files in /var/apache
• mod_auth_kerb (cclub) - installed but not really used on contrib-cgi
• pubcookie debs (cclub)
  • /var/pubcookie
• suexec

  • right now, divert debian suexec & copy in our own patched version from apache collections

    • dpkg-divert --divert /usr/lib/apache2/suexec.apache --rename /usr/lib/apache2/suexec
  • could make this into a deb someday
  • requires cgi_limits.db in apache config directory

    • recompile update_cgi_limits & dump_cgi_limits in andrew-contrib-internal

    • rebuild from cgi_limits.conf using update_cgi_limits if dump_cgi_limits fails to read the db (eg architecture/version change)
• multilog for apache
  • symlink /var/apache/logs/apache* to /var/service
  • mess with djbdaemon foo as necessary
• cwscript collection
  • put cron/contrib_{user,cgi_user}_update.sh in cron
• add mycontrib to /etc/passwd.system
• binfmt
  • /var/lib/binfmts
• php
  • dpkg-divert --divert /usr/bin/php4-cli --rename /usr/bin/php4
  • dpkg-divert --divert /usr/bin/php5-cli --rename /usr/bin/php5
  • cd /usr/bin
  • ln -s php4-cgi php4
  • ln -s php5-cgi php5

CategoryServices CategoryContributedServices