Contents

  1. Jessie
    1. DomU
    2. Dom0
  2. Wheezy
    1. DomU
    2. DomU (the hard way)
    3. Dom0
  3. Squeeze
    1. DomU
    2. Dom0
  4. Building an Etch DomU
  5. Building a Dom0

Jessie

(Under construction; follow with care.)

DomU

/!\ This will create PV DomU. You should convert it to HVM when you finish these steps. /!\

Pre-steps (Copied from Wheezy DomU Instructions)

Step 1: Register domU on netreg.net.cmu.edu using the cl0x user

Step 2: Update the zonefile information so DNS behaves properly

Edit the file /afs/club/service/dns/DB.club.cc.cmu.edu with this new set of lines at an appropriate place. Include contact information if the machine is being hosted for an outside group. 

# the_machine_name
=XXXXXX.club.cc.cmu.edu:127.237.157.XXX:7200

Note: These changes propagate through the machines at Sync Time. Until this has happened, you will not be able to ssh from other machines using DNS, and ksu will not work.

Step 3: Add some info about the machine to the CategoryInfrastructure page.

Step 4: Find a dom0 to host the domU on. http://www.club.cc.cmu.edu/~kbare/list-xen.cgi is helpful.

Initial VM Construction

# change XXX as appropriate
# for instance: magic-byte-burper, 188, BC (respectively)
# feel free to change memory and disk sizes as well
xen-create-image --hostname XXXXXX.club.cc.cmu.edu --memory=1Gb --size=6Gb --fs=ext4 \
                 --swap=2Gb --ip=128.237.157.XXX --mac=00:00:80:ed:9d:XX \
                 --broadcast=128.237.157.255 --gateway=128.237.157.1 --netmask=255.255.255.0 \
                 --lvm=dom0.root --dist=jessie --pygrub --nohosts
# IMPORTANT: remember root password

#fixup long hostname -> short hostname in a few places
mv /etc/xen/XXXXXX.club.cc.cmu.edu.cfg /etc/xen/XXXXXX

vim /etc/xen/XXXXXX
#  Or use sed -i.bak s/XXXXXX.club.cc.cmu.edu/XXXXXX/g /etc/xen/XXXXXX
#  change name = 'XXXXXX.club.cc.cmu.edu' to 'XXXXXX'
#  change LVs from XXXXXX.club.cc.cmu.edu-{disk,swap} to XXXXXX-{disk,swap}
:%s/XXXXXX.club.cc.cmu.edu/XXXXXX/g

lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-disk XXXXXX-disk
lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-swap XXXXXX-swap

Booting and Initial Setup

xl create -c /etc/xen/XXXXXX

TODO:kbare - xen-create-image doesn't seem to handle the fact Debian 12 (bookworm) DomUs use enXN as their network interface names.

Login as root, using the password displayed by xen-create-image.

Change the root password. 

passwd

Fix the initramfs so that fsck runs properly. 

update-initramfs -k all -u

Clubification

echo "deb http://debian.club.cc.cmu.edu/debian/ jessie-cclub contrib" >> /etc/apt/sources.list
apt-get update
apt-get dist-upgrade
apt-get install cclub-pre-configuration
apt-get update

Then pick one of the two variants below.

Variant 1: Rsync slave + passwd.update.sh

Recommended for high-availability services and hosts that otherwise do not need AFS clients. 

apt-get install cclub-rsync-slave cclub-passwd-update-configuration

Variant 2: OpenAFS + passwd-update.sh

Do not use for critical club services! We don't want circular dependencies involving AFS.

Note, that as before, it is advisable to have a separate file system mounted on /var/cache/openafs. 

apt-get install cclub-afs-client-configuration cclub-passwd-update-configuration

Finish Clubification

apt-get clean
shutdown -r now

Dom0

Install Jessie on the machine. (TODO: make this easy with preseeding.)

Fix the clock, if necessary. (When you run the Debian installer on a machine with a private IP, it doesn't sync the clock.) 

service stop ntp
ntpdate ntp1.club.cc.cmu.edu ntp2.club.cc.cmu.edu ntp3.club.cc.cmu.edu
service start ntp

Clubification

echo "deb http://debian.club.cc.cmu.edu/debian/ jessie-cclub contrib" >> /etc/apt/sources.list
apt-get update
apt-get dist-upgrade
apt-get install cclub-pre-configuration
apt-get update

Substitute cclub-proliant-gen7-configuration with some other configuration package, depending on the machine type. There's also a generic cclub-hardware-configuration package. 

apt-get install cclub-proliant-gen7-configuration cclub-xen-dom0-configuration

Finish Clubification

apt-get clean
shutdown -r now

Wheezy

These instructions come to you courtesy of kbare and rharwood. Please only install Wheezy DomUs on Wheezy Dom0s, and do not install Squeeze, Lenny, or Etch anything.

DomU

Step 1: Register domU on netreg.net.cmu.edu using the cl0x user

Step 2: Update the zonefile information so DNS behaves properly

Edit the file /afs/club/service/dns/DB.club.cc.cmu.edu with this new set of lines at an appropriate place. Include contact information if the machine is being hosted for an outside group. 

# the_machine_name
=XXXXXX.club.cc.cmu.edu:127.237.157.XXX:7200

Note: These changes propagate through the machines at Sync Time. Until this has happened, you will not be able to ssh from other machines using DNS, and ksu will not work.

Step 3: Add some info about the machine to the CategoryInfrastructure page.

Step 4: Find a dom0 to host the domU on. http://www.club.cc.cmu.edu/~kbare/list-xen.cgi is helpful.

Step 5.1: Create the domU's volumes and configuration file. 

# change XXX as appropriate
# for instance: squeeze, 188, BC (respectively)
# feel free to change memory and disk sizes as well
xen-create-image --hostname XXXXXX.club.cc.cmu.edu --memory=512Mb --size=4Gb --fs=ext4 \
                 --swap=1Gb --ip=128.237.157.XXX --mac=00:00:80:ed:9d:XX \
                 --broadcast=128.237.157.255 --gateway=128.237.157.1 --netmask=255.255.255.0 \
                 --lvm=dom0.root --pygrub --nohosts
# IMPORTANT: remember root password

#fixup long hostname -> short hostname in a few places
mv /etc/xen/XXXXXX.club.cc.cmu.edu.cfg /etc/xen/XXXXXX

vim /etc/xen/XXXXXX
#  Or use sed -i.bak s/XXXXXX.club.cc.cmu.edu/XXXXXX/g /etc/xen/XXXXXX
#  change name = 'XXXXXX.club.cc.cmu.edu' to 'XXXXXX'
#  change LVs from XXXXXX.club.cc.cmu.edu-{disk,swap} to XXXXXX-{disk,swap}
:%s/XXXXXX.club.cc.cmu.edu/XXXXXX/g

lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-disk XXXXXX-disk
lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-swap XXXXXX-swap

# To have it start on bootup
mkdir -p /etc/xen/auto
ln -s /etc/xen/XXXXXX /etc/xen/auto

Step 5.2: Start the domU, then do some basic setup. 

xm create -c XXXXXX

#login, and then change root passwd !!!
passwd

echo "deb http://debian.club.cc.cmu.edu/debian/ wheezy-cclub contrib" >> /etc/apt/sources.list

aptitude update
aptitude install cclub-keyring <<< "Yes"
aptitude update
aptitude install cclub-debconf-settings
aptitude dist-upgrade

# For 32-bit only machines, substitute -amd64 with -686
# also replace with the correct kernel version, if different
aptitude install '~pstandard' linux-image-3.2.0-4-amd64

Step 6 Clubification

scp «user»@«host»:/afs/club/service/etc/skel/wheezy/packages.wheezy.domU . aptitude install $(cat packages.wheezy.domU) # take defaults at pam prompt 

# log in as root

cd /root

# DO NOT DO THE NEXT COMMAND!
# I don't have time to fix this right now, but this is horribly insecure. If you
# can't tell why, ask someone before proceeding.
# I have messed up the url so that you can't run this command because you shouldn't.
aptitude install $(wget -O - htp://random-unsecure-website.mit.edu/afs/club.cc.cmu.edu/service/etc/skel/wheezy/packages.wheezy.domU) # take defaults at pam prompt

Step 7a Real AFS

*if you want fake AFS, please skip to Step 7b:*

It is highly advisable to create a dedicated afscache volume. 

# on the Dom0
lvcreate -L 1G -n $SHORTNAME-afscache dom0.root
mkfs.ext2 /dev/dom0.root/$SHORTNAME-afscache

# Add an entry to the "disk" list in /etc/xen/$SHORTNAME mapping the afscache
# volume to a virtual block device in the DomU.
#
# E.g., the whole thing should end up looking something like:
# disk        = [
#                   'phy:/dev/dom0.root/$SHORTNAME-disk,xvda2,w',
#                   'phy:/dev/dom0.root/$SHORTNAME-swap,xvda1,w',
#                   'phy:/dev/dom0.root/$SHORTNAME-afscache,xvda3,w',
#               ]

Then, restart the DomU to pick up the new blockdevice. 

# in the DomU

# Add the afscache device to fstab.
# E.g., fstab line should like:
# /dev/xvda3 /var/cache/openafs ext2 defaults 0 2

# Mount the cache filesystem.
mkdir -p /var/cache/openafs
mount /var/cache/openafs

# Install the AFS client
aptitude install openafs-modules-3.2.0-4-amd64
aptitude install openafs-client libpam-afs-session
kinit -S kadmin/admin «user»/admin

Step 7b Fake AFS 

# only if you want fake AFS, do the next four things:
# Edit /etc/ssh/ssh_config
# Change
#   #   GSSAPIDelegateCredentials no
# To
#       GSSAPIDelegateCredentials yes
# scp «user»@«host»:/afs/club/system/scripts/sh/newrsync-wheezy.sh .
sed -i.bak "s/.*\(GSSAPIDelegateCredentials\).*/\\1 yes/" /etc/ssh/ssh_config
wget http://lost-contact.mit.edu/afs/club.cc.cmu.edu/system/scripts/sh/newrsync-wheezy.sh
kinit -S kadmin/admin «user»/admin
chmod +x ./newrsync-wheezy.sh
./newrsync-wheezy.sh
# sometimes this fails chicken/egg problem, fix by editing ./newrsync-wheezy.sh: all unix.club.cc.cmu.edu to conch.club.cc.cmu.edu
# chicken/egg problem resolved in Step 8

Step 8 Final clubification 

# create a new kerberos host key
kadmin ank -r host/$SHORTNAME.club.cc.cmu.edu # accept all defaults
kadmin ext_keytab host/$SHORTNAME.club.cc.cmu.edu
/afs/club/system/scripts/sh/newmachine-wheezy.sh
/afs/club/system/scripts/sh/snmp-configure-monitoring.sh

Step 9 Making a shell 

# if you are not setting up a shell machine, you are DONE
# otherwise, please continue

aptitude update
aptitude install $(cat /afs/club/service/etc/skel/wheezy/packages.wheezy.shell-extra) # take defaults

# rsync crap for shell machines
# this is a dirty

# rsync -av the /var/rsync directory from an existing shell machine
rsync -av root@oyster.club.cc.cmu.edu:/var/rsync /var/rsync

# rsync should be in /etc/passwd.system (should be in /etc/shadow automatically)
# this is because they need to run cronjobs (TODO: please fix it so it doesn't; ask rharwood or rzhou if you want to know why)

# root's crontab should include a motd update script
EDITOR=vim crontab -e
append "00 *            * * *   /afs/club.cc.cmu.edu/system/scripts/sh/motd-update.sh"

# rsync crontab should have the rsync-master.sh script at 50 pass the hour
# /var/mkasick is part of a DNS system; ignore it

# ssh host keys need to match each other (/etc/ssh/ssh_host*)
# rsync them from existing shell machines

# kerberoskerberoskerberos (gssapi)
# this is from an email by mkasick:
# This one step in particular, is probably the most nuanced of things we do.  I do it entirely too cautiously, becuase I'd rather not trash the heimdal database.  Steps are:
0.  Delete "/var/heimdal/delme_the_next_time_you_see_this" on sodium.
1.  Backup existing Heimdal database on sodium (/var/heimdal/heimdal.db).
2.  Run "/etc/init.d/kdc stop; /etc/init.d/ipropd-master stop" on sodium.
3.  Run "kadmin -l", "dump fooasdfbar"
4.  Open fooasdfbar, replace "host/foo.club.cc.cmu.edu" line with copy of
    "host/.club.cc.cmu.edu" line. 's/unix/foo/' on that line.  In
    effect, you're replacing the contents of the host/foo key with the
    host/unix key.  See host/cobalt for example.  Save as barasdfbar.
5.  "diff -U 0 fooasdfbar barasdfbar" and make sure only the appropriate
     line has been changed.
6.  "rm /var/heimdal/heimdal.db"
7.  "kadmin -l", "load barasdfbar", "dump bazasdfbar"
8.  "cmp barasdfbar bazasdfbar" to ensure they're the same.
9.  Run "/etc/init.d/kdc stop; /etc/init.d/ipropd-slave stop" on potassium,
    barium.
10. "rm /var/heimdal/heimdal.db" on potassium, barium.
11. "/etc/init.d/kdc start; /etc/init.d/ipropd-master start" on sodium.
12. "/etc/init.d/ipropd-slave start" on potassium, barium.
13. Verify iprop worked correctly by "ls -l /var/heimdal/heimdal.db" on
    potassium, barium, and looking at the output of /var/heimdal/slave-stats
    on sodium.
14. Run "/etc/init.d/kdc start" on potassium, barium.
15. Delete fooasdfbar, barasdfbar, bazasdfbar, or alternatively place them
    in a folder called "delme_the_next_time_you_see_this" if you're like me
    and liable to have screwed this up.
# if keys become desynchronized to this
# sourced from https://wiki.club.cc.cmu.edu/org-auth/ccwiki/Infrastructure/cobalt.club.cc.cmu.edu

# now any any machine on which the database has been changed needs to run
rm /etc/krb5.keytab
kadmin ext host/XXXXXX.club.cc.cmu.edu
# note that XXXXXX is strictly the name of the machine (it doesn't seem to have to be unix)

# reboot the shell

DomU (the hard way)

Step 1–Step 4: Same as above.

Step 5: Machine set up 

# note: please only do this once at a time on any given machine

# on the host machine:

lvcreate -L 4G -n $SHORTNAME-root dom0.root # replace dom0.root as well
lvcreate -L 1G -n $SHORTNAME-swap dom0.root # see above

mkfs.ext3 /dev/mapper/dom0.root-$SHORTNAME--root
mkswap /dev/mapper/dom0.root-$SHORTNAME--swap

mount /dev/mapper/dom0.root-$SHORTNAME--root /mnt

debootstrap wheezy /mnt/ http://mirrors.mit.edu/debian/ # and wait

mount --bind /dev/ /mnt/dev
mount -t proc proc /mnt/proc
mount -t sysfs sysfs /mnt/sys

cat > /mnt/etc/apt/sources.list << EOF
deb http://mirrors.mit.edu/debian/ stable main
deb http://security.debian.org/ stable/updates main
deb http://mirrors.mit.edu/debian/ stable-updates main
deb http://debian.club.cc.cmu.edu/debian/ wheezy-cclub contrib
EOF

chroot /mnt /bin/bash

in the chroot: 

echo $SHORTNAME > /etc/hostname

# fix /etc/hosts
# prepend to file:
# 128.237.157.«###» $SHORTNAME.club.cc.cmu.edu $SHORTNAME

# set up /etc/network/interfaces
# prepend to the file:
# auto eth0
# iface eth0 inet static
#       address 128.237.157.«###»
#       netmask 255.255.255.0
#       network 128.237.157.0
#       broadcast 128.237.157.255
#       gateway 128.237.157.1
#       mtu 16110

# set up /etc/fstab
cat > /etc/fstab << EOF
/dev/xvda1 / ext3 rw,noatime,errors=remount-ro 0 1
/dev/xvda2 swap swap sw 0 0
EOF

# divert start-stop-daemon
dpkg-divert --local --rename --divert /sbin/start-stop-daemon.real --add /sbin/start-stop-daemon
cat > /sbin/start-stop-daemon << EOF
#!/bin/sh
echo invoked fake start-stop-daemon... > /dev/stderr
exit 0
EOF
chmod a+x /sbin/start-stop-daemon

aptitude update
aptitude install cclub-keyring
aptitude update
aptitude install cclub-debconf-settings
aptitude dist-upgrade

# For 32-bit only machines, substitute -amd64 with -686
# also replace with the correct kernel version, if different
aptitude install '~pstandard' linux-image-3.2.0-4-amd64

# Pick US/Eastern as the local time zone
dpkg-reconfigure tzdata

# Edit /etc/inittab
# After,
#   6:23:respawn:/sbin/getty 38400 tty6
# Add
#   H0:2345:respawn:/sbin/getty 38400 hvc0

rm /sbin/start-stop-daemon
dpkg-divert --rename --remove /sbin/start-stop-daemon

passwd root

# exit the chroot
exit

outside the chroot: 

umount /mnt/{dev,proc,sys,}

# create xen configuration
cat > /etc/xen/$SHORTNAME << EOF
kernel = '/boot/vmlinuz-3.2.0-4-amd64'
ramdisk = '/boot/initrd.img-3.2.0-4-amd64'

vcpus = '1'
memory = '1024'

root = '/dev/xvda1 ro'
disk = [ 'phy:/dev/dom0.root/$SHORTNAME-root,xvda1,w',
         'phy:/dev/dom0.root/$SHORTNAME-swap,xvda2,w'
       ]

name = '$SHORTNAME'

vif = [ 'ip=128.237.157.«###» ,mac=00:00:08:ed:9d:«hex(«###»)»' ]

on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
EOF

# enable automatic start on boot
mkdir -p /etc/xen/auto
ln -sv /etc/xen/$SHORTNAME /etc/xen/auto/

xm create -c $SHORTNAME

If you are not clubifying, you are DONE!

Step 6–Step 9: Same as above.

Dom0

TODO: test, document, and fill from https://wiki.club.cc.cmu.edu/org-auth/ccwiki/kbare%20at%20CLUB.CC.CMU.EDU/Wheezy

Pre-req: Note the MAC address for the machine's first GbE port. Usually this can be determined by looking under the "Integrated Devices" menu in a machine's BIOS.

Step 1: Register the machine in NetReg.

Login in as cl0x. Be sure to register the machine with the correct MAC address so that DHCP works properly.

Once the machine is registered, NetReg will take you to the machine's main page.

Click the "View Advanced Options" link.

Scroll down to the "DHCP Options" section and click the "Add DHCP Option" link.

Use this to add two options:

filename

"/netinstall-wheezy/debian-installer/amd64/pxelinux.0"

next-server

storage-2.club.cc.cmu.edu

If you need to install a non-64-bit-capable machine, replace "amd64" with "i386".

Wait for the change to propagate to the Andrew DHCP servers. NetReg will tell you when the next DHCP server update happens on the signout page.

Step 2: Netboot

Once the change has propagated to the Andrew DHCP servers, netboot the machine.

You can usually force a machine to netboot by pressing F12 as it is booting (or ESC-@ if on the serial console). If that doesn't work, you can try moving network booting earlier in the BIOS boot sequence.

Step 3: Netinstall Boot Menu

This will bring you to a boot menu.

We have modified the Debian netinstall boot menu in some useful ways:

Hit enter at the "Cclub Options" menu item.

This gives you a grand total of six options.

You can choose either a serial or VGA console depending on which way you're accessing the machine.

You also choose how Debian is installed to the machine's disks:

Select an appropriate option and hit enter.

Note on setting up LVM: This guide generally assumes that you name the volume group 'dom0.root'

Step 4: Set up networking.

The Debian installer will boot and take you to its main menu.

First "Detect Network Hardware."

Once the network hardware has been detected "Configure the Network." 

Choose the correct network device (probably eth0).
Manually define the IP settings.
# Add DNS info: 128.237.157.12, 128.237.157.14 128.2.204.150
Manually define the hostname and domainname (these should be all lowercase).

Step 5: Mostly-automated install.

Select "Download debconf preconfiguration file."

This will start the mostly-automated install process. However, there still will be one or more prompts that you will need to answer.

You will always be prompted to set a root password. Use the club root password if you know it.

If you selected manual partitioning at the installer boot menu, you will need to define partitions and file systems.

You may be prompted to confirm deleting or modifying existing on-disk state. If you are sure it is ok to overwrite the existing data on the system, select "Ok."

When the installation completes, the machine will reboot into the newly installed Debian system.

Step 6: Install packages and clubify

Debian install note: do not install "standard packages." Do it later so we don't install any packages (most notably exim4) that we'd like to preseed with the cclub-debconf-settings package. 

# Add wheezy-cclub to /etc/apt/sources.list.  Either manually add it, or scp
# conch:/afs/club/service/etc/skel/wheezy/apt/sources.list.
# To manually add it, the appropriate line is:
#     deb http://debian.club.cc.cmu.edu/debian/ wheezy-cclub contrib

aptitude update
aptitude install cclub-keyring
aptitude update
aptitude install cclub-debconf-settings

aptitude dist-upgrade

aptitude install '~pstandard'

# For machines that should run Xen.
# For 32-bit only machines, substitute -amd64 with -686
aptitude install xen-hypervisor-4.1-amd64 xen-utils-4.1 xen-tools

scp «user»@conch:/afs/club/service/etc/skel/wheezy/packages.wheezy.dom? .
aptitude install `cat packages.wheezy.domU`
aptitude install `cat packages.wheezy.dom0`

# If you need/want to create a new Kerberos host key
# Run
kinit -S kadmin/admin «user»/admin
kadmin ank --use-defaults -r host/«my-hostname».club.cc.cmu.edu
kadmin ext_keytab host/«my-hostname».club.cc.cmu.edu

Fake AFS: 

# Edit /etc/ssh/ssh_config
# Change
#   #   GSSAPIDelegateCredentials no
# To
#       GSSAPIKeyExchange yes
#       GSSAPIDelegateCredentials yes

scp «user»@conch:/afs/club/system/scripts/sh/newrsync-wheezy.sh .
kinit -S kadmin/admin «user»/admin
./newrsync-wheezy.sh

Note: If you get "gss_init_context failed", try using a specific host rather than the unix.club redirect 

/afs/club/system/scripts/sh/newmachine-wheezy.sh

# Edit /etc/network/interfaces.
# Replace all instances of "eth0" with "br0".
# Change "allow-hotplug br0" to "auto br0".
# Add "bridge_ports eth0" to the stanza for br0.
# Add "post-up ifconfig eth0 mtu 9000"

shutdown -r now

Following shouldn't be necessary, since the newmachine script should do an appropriate diversion for the same effect... --Keith

Please consider installing a RAID monitoring package if appropriate. The specific packages and how to set them up will depend on the Dom0's hardware. Please see the relevant page under Hardware Documentation. Configuring smartd to monitor the drives is also a good idea.

Squeeze

You should only install Squeeze DomUs on Squeeze Dom0s. However, Squeeze Dom0s should support Etch and Lenny DomUs.

DomU

Notes: Make sure you have /sbin and /usr/sbin in your path. A quick fix is to ssh into the DomO as root from the DomO.

Step 1: Register domU on netreg.net.cmu.edu using the cl0x user

Step 2: Update the zonefile information so DNS behaves properly

Edit the file /afs/club/service/dns/DB.club.cc.cmu.edu with this new set of lines at an appropriate place. Include contact information if the machine is being hosted for an outside group. 

# the_machine_name
+XXXXXX.club.cc.cmu.edu:127.237.157.XXX:7200

IMPORTANT: Before closing the file, update the timestamp at the first line. It should look something like this: 

Zclub.cc.cmu.edu:sodium.club.cc.cmu.edu.:gripe.club.cc.cmu.edu.:2012042000:3600:600:9600:3600:86400

Edit the number of the form 20XXXXXXXX to be a new datestamp. It is very important that this number be strictly greater than the old number, otherwise terrible things will happen.

Note: These changes propagate through the machines at Sync Time. Until this has happened, you will not be able to ssh from other machines using DNS, and ksu will not work.

Step 3: Add some info about the machine to the CategoryInfrastructure page.

Step 4: Find a dom0 to host the domU on. http://www.club.cc.cmu.edu/~kbare/list-xen.cgi is helpful. 

# change XXX as appropriate
# for instance: squeeze, 188, BC (respectively)
# feel free to change memory and disk sizes as well
xen-create-image --hostname XXXXXX.club.cc.cmu.edu --memory=256Mb --size=4Gb \
                 --swap=1Gb --ip=128.237.157.XXX --mac=00:00:80:ed:9d:XX \
                 --broadcast=128.237.157.255 --gateway=128.237.157.1 --netmask=255.255.255.0 \
                 --lvm=dom0.root --nohosts
# IMPORTANT: remember root password

#fixup long hostname -> short hostname in a few places
mv /etc/xen/XXXXXX.club.cc.cmu.edu.cfg /etc/xen/XXXXXX
vim /etc/xen/XXXXXX
#  change name = 'XXXXXX.club.cc.cmu.edu' to 'XXXXXX'
#  change LVs from XXXXXX.club.cc.cmu.edu-{disk,swap} to XXXXXX-{disk,swap}
:%s/XXXXXX.club.cc.cmu.edu/XXXXXX/g

lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-disk XXXXXX-disk
lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-swap XXXXXX-swap

# To have it start on bootup
mkdir -p /etc/xen/auto
ln -s /etc/xen/XXXXXX /etc/xen/auto

xm create -c XXXXXX
#login, and then change root passwd !!!
passwd

dpkg-reconfigure debconf
Change priority to medium.

scp www-node-2:/afs/club/service/etc/skel/squeeze/packages* . # you can't use unix for this anymore
scp www-node-2:/afs/club/system/scripts/sh/newrsync-squeeze.sh . # or this
apt-get update
aptitude install `cat packages.squeeze.domU`
 # Mail server configuration
 # - mail sent by smarthost; no local mail
 # - defaults except:
 #    +  "system mail name" => "<host>.club.cc.cmu.edu"
 #    +  "visible domain name" => "club.cc.cmu.edu" (no machine name)
 #    +  "outgoing smarthost" => "smtp.club.cc.cmu.edu" (add smtp)
 # ca-certificate configuration
 # - yes
 # man-db
 # - no
 # Kerberos and PAM
 # - defaults for everything

vi /etc/krb5.conf
 # add `allow_weak_crypto = true` under `default_realm = ...` line
vi /etc/ssh/ssh_config
# Change `GSSAPIDelegateCredentials yes` and uncomment that line if not already uncommented
kinit -S kadmin/admin [YOU]/admin
./newrsync-squeeze.sh
/afs/club/system/scripts/sh/newmachine-squeeze.sh
# might have to kadmin add -r host/XXXXXX.club.cc.cmu.edu
kadmin ext host/XXXXXX.club.cc.cmu.edu

# to set the default locale to something reasonable
# we generally install all en_US locales, and set en_US.UTF8 as the system default
dpkg-reconfigure locales

Step 5: If you are setting up AFS access perform the following steps. Otherwise you are DONE. 

su rsync
crontab -e
# delete entries
# close rsync shell
aptitude install linux-headers-<version>-xen-amd64 linux-image-<version>-xen-amd64

# on host
lvcreate -L 1G -n XXXXXXX-afscache dom0.root
emacs /etc/xen/XXXXXX
# add mapping for new disc
xm shutdown XXXXXX
xm create XXXXXX

# on guest
aptitude install libpam-afs-session openafs-client openafs-fileserver openafs-krb5 openafs-modules-dkms openafs-modules-source
# try to maximize space of afs cache used but do not come too close as its estimation algorithm is weird
# do not encrypt queries
# everything else defaults
reboot
# you should now see entries in /afs

cp /afs/club.cc.cmu.edu/service/etc/skel/squeeze/pam.d/common-session.afs /etc/pam.d/common-session

Step 6: Perform the following steps if you are setting up a shell machine. Otherwise you are DONE. 

aptitude install $(cat /afs/club/service/etc/skel/squeeze/packages.squeeze.shell-extra)
# for zephyr-clients config, set servers to: zephyr1.club.cc.cmu.edu zephyr2.club.cc.cmu.edu
emacs /etc/pam.d/common-session
# add the line
# session required        pam_afs_session.so minimum_uid=110
# as the second line

# rsync crap for shell machines
# this is a dirty
# /bin/rsync -av the /var/rsync directory from an existing shell machine
# rsync should be in /etc/user (should be in /etc/shadow automatically)
# this is because they need to run cronjobs; is this still the case?
# root's crontab should include a motd update script
# rsync crontab should have the rsync-master.sh script at 50 pass the hour
# /var/mkasick is part of a DNS system; ignore it

# ssh host keys need to match each other (/etc/ssh/ssh_hosts*)
# rsync them from existing shell machines

# kerberoskerberoskerberos (gssapi)
# this is from an email by mkasick:
# This one step in particular, is probably the most nuanced of things we do.  I do it entirely too cautiously, becuase I'd rather not trash the heimdal database.  Steps are:
0.  Delete "/var/heimdal/delme_the_next_time_you_see_this" on sodium.
1.  Backup existing Heimdal database on sodium (/var/heimdal/heimdal.db).
2.  Run "/etc/init.d/kdc stop; /etc/init.d/ipropd-master stop" on sodium.
3.  Run "kadmin -l", "dump fooasdfbar"
4.  Open fooasdfbar, replace "host/foo.club.cc.cmu.edu" line with copy of
    "host/unix.club.cc.cmu.edu" line. 's/unix/foo/' on that line.  In
    effect, you're replacing the contents of the host/foo key with the
    host/unix key.  See host/cobalt for example.  Save as barasdfbar.
5.  "diff -U 0 fooasdfbar barasdfbar" and make sure only the appropriate
     line has been changed.
6.  "rm /var/heimdal/heimdal.db"
7.  "kadmin -l", "load barasdfbar", "dump bazasdfbar"
8.  "cmp barasdfbar bazasdfbar" to ensure they're the same.
9.  Run "/etc/init.d/kdc stop; /etc/init.d/ipropd-slave stop" on potassium,
    barium.
10. "rm /var/heimdal/heimdal.db" on potassium, barium.
11. "/etc/init.d/kdc start; /etc/init.d/ipropd-master start" on sodium.
12. "/etc/init.d/ipropd-slave start" on potassium, barium.
13. Verify iprop worked correctly by "ls -l /var/heimdal/heimdal.db" on
    potassium, barium, and looking at the output of /var/heimdal/slave-stats
    on sodium.
14. Run "/etc/init.d/kdc start" on potassium, barium.
15. Delete fooasdfbar, barasdfbar, bazasdfbar, or alternatively place them
    in a folder called "delme_the_next_time_you_see_this" if you're like me
    and liable to have screwed this up.
# if keys become desynchronized to this
# sourced from https://wiki.club.cc.cmu.edu/org-auth/ccwiki/Infrastructure/cobalt.club.cc.cmu.edu

# now any any machine on which the database has been changed needs to run
rm /etc/krb5.keytab
kadmin ext host/XXXXXX.club.cc.cmu.edu
# note that XXXXXX is strictly the name of the machine (it doesn't seem to have to be unix)

Dom0

Install Debian:

Choose your own adventure. You can either netboot, or use a netinst CD.

Netboot

You will need to know the MAC address for the machine you need to install.

On a PE2850, you can:

Hit F2 to enter setup.

Go under the "Integrated Devices" menu. Make sure the first gigabit ethernet adapter has PXE booting enabled. Also, note the MAC address.

Now, go to NetReg. If you want to use a new hostname, you will need to register a new machine. Be sure to specify the correct MAC address. Otherwise, if you are re-using an old hostname, got to its entry, change the MAC address, and click update.

Now click the "View Advanced Options" link.

Now scroll down to the "DHCP Options" section and click the "Add DHCP Option" link.

Use this to add two options:

filename

"/netinstall-squeeze/debian-installer/amd64/pxelinux.0"

next-server

storage-2.club.cc.cmu.edu

If you need to install a non-64-bit machine, replace "amd64" with "i386".

Wait for the change to propagate to the Andrew DHCP servers.

Once the change has propagated, you can usually netboot the machine by pressing F12 as its booting. (If that doesn't work, you can try moving network booting earlier in the BIOS boot sequence.)

This will bring you to a boot menu. There's a "Cclub Options" sub-menu with various useful setup configurations. Choose serial or VGA console depending on which way you're accessing the machine. Generally you'll want to use one of the automated install options, rather than manual partitioning. Which of /dev/sda or /dev/sdb is correct depends on whether the machine has a DRAC virtual drive enabled.

Netinst CD

Boot the netinst CD.  Use an AMD64 CD whenever possible (e.g., for 2850s).

Open the "Advanced options" menu.
Highlight the "Expert install" item and press tab.  Edit the command line, adding 'auto=true url=www.club.cc.cmu.edu' before the '--'.
Press enter.

Select "Detect and mount CD-ROM."  Select "Continue" at all prompts.
Select "Load installer components from CD."  Select "Continue" at all prompts.
Select "Detect network hardware."

Debian Installer

Select "Configure the network."
Choose the correct network device (probably eth0).
Manually define the IP settings.
# Add DNS info: 128.237.157.12, 128.237.157.14
Manually define the hostname and domainname (these should be all lowercase).

Select "Download debconf preconfiguration file" to start a mostly-automated install.  You will be prompted to:
1. Set a root password.  Use the club root password if you know it.
2. Confirm writing the partition table to disk
3. Confirm formatting filesystems.

When the installation completes, the machine will reboot into the newly installed Debian system.

Note: If you need to edit the preconfiguration file, the file is in /afs/club.cc.cmu.edu/www/d-i/.  In particular, you may need to replace /dev/sda with /dev/sdb on 2850 machines.

Install Xen (can be skipped if the machine will be dedicated to a single service):

aptitude install xen-hypervisor-4.0-amd64 linux-image-xen-amd64 xen-tools
# (OR, for non-amd64 machines): aptitude install xen-hypervisor-4.0-i386 linux-image-xen-686 xen-tools

vi /etc/network/interfaces
# %s/eth0/br0/g, %s/allow-hotplug/auto/g, add `bridge_ports eth0` to the br0 stanza

mkdir /etc/xen/auto

Clubification:

dpkg-reconfigure debconf
# Select "Dialog" interface
# Select "medium" priority

scp unix:/afs/club/service/etc/skel/squeeze/packages* .
scp unix:/afs/club/system/scripts/sh/newrsync-squeeze.sh .
aptitude install `cat packages.squeeze.domU`
# For all debconf pop-ups, select the defaults, except:
# exim4-config - Select "mail sent by smarthost; no local mail"
# exim4-config - Use "visible domain name" => "club.cc.cmu.edu" (no machine name)
# exim4-config - Use "outgoing smarthost" => "smtp.club.cc.cmu.edu" (add smtp)
aptitude install `cat packages.squeeze.dom0`
# For all debconf pop-ups, select the defaults

vi /etc/krb5.conf
# Add `allow_weak_crypto = true` under `default_realm = ...` line
vi /etc/ssh/ssh_config
# Change `GSSAPIDelegateCredentials yes`

kinit -S kadmin/admin <USER>/admin
./newrsync-squeeze.sh
/afs/club/system/scripts/sh/newmachine-squeeze.sh
# might have to kadmin add -r host/<MACHINE>.club.cc.cmu.edu
kadmin ext host/<MACHINE>.club.cc.cmu.edu

reboot

Building an Etch DomU

To build a Lenny DomU look at the install instructions in mkasick's public/lenny/ folder.

Netreg machine name with cl0x and an empty MAC to get an IP. Then use the IP to update that record with the correct MAC.

Implicit step:

Create LVM volumes for the DomU

Mkasick's magic image, uncompress it onto the root device.

Make the swap partition.

Make the afs cache filesystem.

Mount the root filesystem (probably want to chroot).

Update config files on the new domU.

Do magic on the domU to fix the change from Xen emulating device tty1 to hvc0 (could put in magic image eventually)

Remove udev persistent net rules the domU.

Xen configuration file on the dom0.

If installing without AFS, don't have the AFS Cache in /etc/fstab. If afscache is ext3 instead of ext2, make that change in fstab as well.

Start the domain.

Upgrade packages.

Create kerb instances for the machine.

If not using afs, copy the rsync script from a machine that does have afs and run it as kerb admin.

Run the new machine script as kerb admin.

By default only passwd.admin is allowed to log in.

Add to DNS

Building a Dom0

Install lenny:

Boot from the netinst CD.
Tab on the Expert Install item and add 'auto=true url=www.club.cc.cmu.edu' before the '--'.
Press enter.

Select "Detect and mount CD-ROM"
Select "Load installer components from CD"
Select "Detect network hardware"
Select "Configure the network"

Choose the correct network device (probably eth0).
Manually define the IP settings.
Manually define the hostname and domainname (these should be all lowercase).

Select "Support for automatic installs"
Select "Download debconf preconfiguration file"

That's it!

Install Xen (optional):

Clubification:

Install etch:

Before "Partition disks":

#the actual cylinders don't really matter
#just make sure that hda1 starts at 1 and is 128M
fdisk /dev/hda
    /dev/hda1:   1-  260 83  (128 MB) # /boot
    /dev/hda2: 261-      8e           # lvm

modprobe dm-mod
pvcreate /dev/hda2
vgcreate xx.root /dev/hda2
lvcreate -L 1G -n xx.root xx.root      # /
lvcreate -L 512M -n xx.swap xx.root    # swap
#Size of xx.xensave should be the same as the amount of physical ram
lvcreate -L 512M -n xx.xensave xx.root # /var/lib/xen/save

During "Partition disks":
/boot should be jfs
xx.root should be jfs
xx.swap should be swap
xx.xensave should be jfs

Install linux-image-2.6-686.
During package selection, do not install base system.
Install grub to MBR.

After reboot:

dpkg-reconfigure debconf
Change priority to medium.

apt-get update
apt-get dist-upgrade
apt-get install vim

vi /boot/grub/menu.lst
#Replace \n with a new line
Add "serial --unit=0 --speed=9600\nterminal serial" before
"BEGIN AUTOMATIC KERNELS LIST".
Change "# kopt=root=/dev/mapper/xx.root-xx.root ro console=ttyS0".
Change "# xenhopt=com1=9600,8n1".
Change "# xenkopt=".

vi /etc/inittab
Uncomment "#TO:23:respawn:/sbin/getty -L ttyS0 9600 vt100".

#If on an IA32 system:
apt-get install bridge-utils libc6-xen linux-image-2.6-xen-686 xen-hypervisor-3.0.3-1-i386-pae

#If on an AMD64 system:
#apt-get install bridge-utils linux-image-2.6-xen-amd64 xen-hypervisor-3.0.3-1-amd64

vi /etc/default/xendomains
Change 'XENDOMAINS_SAVE=""'. #do not forget quotes

vi /etc/network/interfaces
Change "allow-hotplug eth0" to "auto br0".
Change "iface eth0 inet static" to "iface br0 inet static".
Add "bridge_ports eth0" under "gateway 128.237.157.1".

vi /etc/apt/apt.conf
Replace with 'APT::Default-Release "etch-cclub";'

vi /etc/apt/sources.list
Add "deb http://debian.club.cc.cmu.edu/debian/ etch-cclub contrib".

apt-get update
apt-get dist-upgrade

apt-get install linux-image-2.6.18-4-xen-686 linux-image-2.6.18-5-xen-686 linux-image-2.6.18-6-xen-686 linux-image-2.6.24-1-xen-686

apt-get install openssh-client
scp unix.club.cc.cmu.edu:/afs/club/service/etc/skel/packages.etch.dom* .
apt-get install `cat packages.etch.domU`
apt-get install `cat packages.etch.dom0`

#adjust hostname as necessary, username as necessary
kinit mkasick/admin
kadmin add -r host/osmium.club.cc.cmu.edu
kadmin ext host/osmium.club.cc.cmu.edu

#if this is a non-AFS domain
scp unix.club.cc.cmu.edu:/afs/club/system/scripts/sh/newrsync-etch.sh .
./newrsync-etch.sh
#endif

/afs/club/system/scripts/sh/newmachine-etch.sh

shutdown -r now

Common Maintenance Tasks/Building Xen Domains (last edited 2023-12-29 17:30:00 by kbare@CLUB.CC.CMU.EDU)