Squeeze

You should only install Squeeze DomUs on Squeeze Dom0s. However, Squeeze Dom0s should support Etch and Lenny DomUs.

DomU

Notes: Make sure you have /sbin and /usr/sbin in your path. A quick fix is to ssh into the DomO as root from the DomO.

Step 1: Register domU on netreg.net.cmu.edu using the cl0x user

Step 2: Update the zonefile information so DNS behaves properly

Edit the file /afs/club/service/dns/DB.club.cc.cmu.edu with this new set of lines at an appropriate place. Include contact information if the machine is being hosted for an outside group. 

# the_machine_name
+XXXXXX.club.cc.cmu.edu:127.237.157.XXX:7200

IMPORTANT: Before closing the file, update the timestamp at the first line. It should look something like this: 

Zclub.cc.cmu.edu:sodium.club.cc.cmu.edu.:gripe.club.cc.cmu.edu.:2012042000:3600:600:9600:3600:86400

Edit the number of the form 20XXXXXXXX to be a new datestamp. It is very important that this number be strictly greater than the old number, otherwise terrible things will happen.

Note: These changes propagate through the machines at Sync Time. Until this has happened, you will not be able to ssh from other machines using DNS, and ksu will not work.

Step 3: Add some info about the machine to the CategoryInfrastructure page.

Step 4: Find a dom0 to host the domU on. http://www.club.cc.cmu.edu/~kbare/list-xen.cgi is helpful. 

# change XXX as appropriate
# for instance: squeeze, 188, BC (respectively)
# feel free to change memory and disk sizes as well
xen-create-image --hostname XXXXXX.club.cc.cmu.edu --memory=256Mb --size=4Gb \
                 --swap=1Gb --ip=128.237.157.XXX --mac=00:00:80:ed:9d:XX \
                 --broadcast=128.237.157.255 --gateway=128.237.157.1 --netmask=255.255.255.0 \
                 --lvm=dom0.root
# IMPORTANT: remember root password

#fixup long hostname -> short hostname in a few places
mv /etc/xen/XXXXXX.club.cc.cmu.edu.cfg /etc/xen/XXXXXX
vim /etc/xen/XXXXXX
#  change name = 'XXXXXX.club.cc.cmu.edu' to 'XXXXXX'
#  change LVs from XXXXXX.club.cc.cmu.edu-{disk,swap} to XXXXXX-{disk,swap}
:%s/XXXXXX.club.cc.cmu.edu/XXXXXX/g

lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-disk XXXXXX-disk
lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-swap XXXXXX-swap

# To have it start on bootup
mkdir -p /etc/xen/auto
ln -s /etc/xen/XXXXXX /etc/xen/auto

xm create -c XXXXXX
#login, and then change root passwd !!!
passwd

dpkg-reconfigure debconf
Change priority to medium.

scp www-node-1:/afs/club/service/etc/skel/squeeze/packages* . # you can't use unix for this anymore
scp www-node-1:/afs/club/system/scripts/sh/newrsync-squeeze.sh . # or this
apt-get update
aptitude install `cat packages.squeeze.domU`
 # Mail server configuration
 # - mail sent by smarthost; no local mail
 # - defaults except:
 #    +  "system mail name" => "<host>.club.cc.cmu.edu"
 #    +  "visible domain name" => "club.cc.cmu.edu" (no machine name)
 #    +  "outgoing smarthost" => "smtp.club.cc.cmu.edu" (add smtp)
 # ca-certificate configuration
 # - yes
 # man-db
 # - no
 # Kerberos and PAM
 # - defaults for everything

vi /etc/krb5.conf
 # add `allow_weak_crypto = true` under `default_realm = ...` line
vi /etc/ssh/ssh_config
# Change `GSSAPIDelegateCredentials yes` and uncomment that line if not already uncommented
kinit -S kadmin/admin [YOU]/admin
./newrsync-squeeze.sh
/afs/club/system/scripts/sh/newmachine-squeeze.sh
# might have to kadmin add -r host/XXXXXX.club.cc.cmu.edu
kadmin ext host/XXXXXX.club.cc.cmu.edu

# to set the default locale to something reasonable
# we generally install all en_US locales, and set en_US.UTF8 as the system default
dpkg-reconfigure locales

Step 5: If you are setting up AFS access perform the following steps. Otherwise you are DONE. 

su rsync
crontab -e
# delete entries
# close rsync shell
aptitude install linux-headers-<version>-xen-amd64 linux-image-<version>-xen-amd64

# on host
lvcreate -L 1G -n XXXXXXX-afscache dom0.root
emacs /etc/xen/XXXXXX
# add mapping for new disc
/etc/init.d/xendomains restart
# there is probably a more elegant way to do the above but I don't know it

# on guest
aptitude install libpam-afs-session openafs-client openafs-fileserver openafs-krb5 openafs-modules-dkms openafs-modules-source
# try to maximize space of afs cache used but do not come too close as its estimation algorithm is weird
# do not encrypt queries
# everything else defaults
reboot
# you should now see entries in /afs

Step 6: Perform the following steps if you are setting up a shell machine. Otherwise you are DONE. 

aptitude install $(cat /afs/club/service/etc/skel/squeeze/packages.squeeze.shell-extra)
# for zephyr-clients config, set servers to: zephyr1.club.cc.cmu.edu zephyr2.club.cc.cmu.edu
emacs /etc/pam.d/common-session
# add the line
# session required        pam_afs_session.so minimum_uid=110
# as the second line

# rsync crap for shell machines
# this is a dirty
# /bin/rsync -av the /var/rsync directory from an existing shell machine
# rsync should be in /etc/user (should be in /etc/shadow automatically)
# this is because they need to run cronjobs; is this still the case?
# root's crontab should include a motd update script
# rsync crontab should have the rsync-master.sh script at 50 pass the hour
# /var/mkasick is part of a DNS system; ignore it

# ssh host keys need to match each other (/etc/ssh/ssh_hosts*)
# rsync them from existing shell machines

# kerberoskerberoskerberos (gssapi)
# this is from an email by mkasick:
# This one step in particular, is probably the most nuanced of things we do.  I do it entirely too cautiously, becuase I'd rather not trash the heimdal database.  Steps are:
0.  Delete "/var/heimdal/delme_the_next_time_you_see_this" on sodium.
1.  Backup existing Heimdal database on sodium (/var/heimdal/heimdal.db).
2.  Run "/etc/init.d/kdc stop; /etc/init.d/ipropd-master stop" on sodium.
3.  Run "kadmin -l", "dump fooasdfbar"
4.  Open fooasdfbar, replace "host/foo.club.cc.cmu.edu" line with copy of
    "host/unix.club.cc.cmu.edu" line. 's/unix/foo/' on that line.  In
    effect, you're replacing the contents of the host/foo key with the
    host/unix key.  See host/cobalt for example.  Save as barasdfbar.
5.  "diff -U 0 fooasdfbar barasdfbar" and make sure only the appropriate
     line has been changed.
6.  "rm /var/heimdal/heimdal.db"
7.  "kadmin -l", "load barasdfbar", "dump bazasdfbar"
8.  "cmp barasdfbar bazasdfbar" to ensure they're the same.
9.  Run "/etc/init.d/kdc stop; /etc/init.d/ipropd-slave stop" on potassium,
    barium.
10. "rm /var/heimdal/heimdal.db" on potassium, barium.
11. "/etc/init.d/kdc start; /etc/init.d/ipropd-master start" on sodium.
12. "/etc/init.d/ipropd-slave start" on potassium, barium.
13. Verify iprop worked correctly by "ls -l /var/heimdal/heimdal.db" on
    potassium, barium, and looking at the output of /var/heimdal/slave-stats
    on sodium.
14. Run "/etc/init.d/kdc start" on potassium, barium.
15. Delete fooasdfbar, barasdfbar, bazasdfbar, or alternatively place them
    in a folder called "delme_the_next_time_you_see_this" if you're like me
    and liable to have screwed this up.
# if keys become desynchronized to this
# sourced from https://wiki.club.cc.cmu.edu/org-auth/ccwiki/Infrastructure/cobalt.club.cc.cmu.edu

# now any any machine on which the database has been changed needs to run
rm /etc/krb5.keytab
kadmin ext host/XXXXXX.club.cc.cmu.edu
# note that XXXXXX is strictly the name of the machine (it doesn't seem to have to be unix)

Dom0

Install Debian:

Choose your own adventure. You can either netboot, or use a netinst CD.

Netboot

You will need to know the MAC address for the machine you need to install.

On a PE2850, you can:

Hit F2 to enter setup.

Go under the "Integrated Devices" menu. Make sure the first gigabit ethernet adapter has PXE booting enabled. Also, note the MAC address.

Now, go to NetReg. If you want to use a new hostname, you will need to register a new machine. Be sure to specify the correct MAC address. Otherwise, if you are re-using an old hostname, got to its entry, change the MAC address, and click update.

Now click the "View Advanced Options" link.

Now scroll down to the "DHCP Options" section and click the "Add DHCP Option" link.

Use this to add two options:

filename

"/netinstall-squeeze/debian-installer/amd64/pxelinux.0"

next-server

storage-2.club.cc.cmu.edu

If you need to install a non-64-bit machine, replace "amd64" with "i386".

Wait for the change to propagate to the Andrew DHCP servers.

Once the change has propagated, you can usually netboot the machine by pressing F12 as its booting. (If that doesn't work, you can try moving network booting earlier in the BIOS boot sequence.)

This will bring you to a boot menu. There's a "Cclub Options" sub-menu with various useful setup configurations. Choose serial or VGA console depending on which way you're accessing the machine. Generally you'll want to use one of the automated install options, rather than manual partitioning. Which of /dev/sda or /dev/sdb is correct depends on whether the machine has a DRAC virtual drive enabled.

Netinst CD

Boot the netinst CD.  Use an AMD64 CD whenever possible (e.g., for 2850s).

Open the "Advanced options" menu.
Highlight the "Expert install" item and press tab.  Edit the command line, adding 'auto=true url=www.club.cc.cmu.edu' before the '--'.
Press enter.

Select "Detect and mount CD-ROM."  Select "Continue" at all prompts.
Select "Load installer components from CD."  Select "Continue" at all prompts.
Select "Detect network hardware."

Debian Installer

Select "Configure the network."
Choose the correct network device (probably eth0).
Manually define the IP settings.
# Add DNS info: 128.237.157.12, 128.237.157.14
Manually define the hostname and domainname (these should be all lowercase).

Select "Download debconf preconfiguration file" to start a mostly-automated install.  You will be prompted to:
1. Set a root password.  Use the club root password if you know it.
2. Confirm writing the partition table to disk
3. Confirm formatting filesystems.

When the installation completes, the machine will reboot into the newly installed Debian system.

Note: If you need to edit the preconfiguration file, the file is in /afs/club.cc.cmu.edu/www/d-i/.  In particular, you may need to replace /dev/sda with /dev/sdb on 2850 machines.

Install Xen (can be skipped if the machine will be dedicated to a single service):

aptitude install xen-hypervisor-4.0-amd64 linux-image-xen-amd64 xen-tools
# (OR, for non-amd64 machines): aptitude install xen-hypervisor-4.0-i386 linux-image-xen-686 xen-tools

vi /etc/network/interfaces
# %s/eth0/br0/g, %s/allow-hotplug/auto/g, add `bridge_ports eth0` to the br0 stanza

mkdir /etc/xen/auto

Clubification:

dpkg-reconfigure debconf
# Select "Dialog" interface
# Select "medium" priority

scp unix:/afs/club/service/etc/skel/squeeze/packages* .
scp unix:/afs/club/system/scripts/sh/newrsync-squeeze.sh .
aptitude install `cat packages.squeeze.domU`
# For all debconf pop-ups, select the defaults, except:
# exim4-config - Select "mail sent by smarthost; no local mail"
# exim4-config - Use "visible domain name" => "club.cc.cmu.edu" (no machine name)
# exim4-config - Use "outgoing smarthost" => "smtp.club.cc.cmu.edu" (add smtp)
aptitude install `cat packages.squeeze.dom0`
# For all debconf pop-ups, select the defaults

vi /etc/krb5.conf
# Add `allow_weak_crypto = true` under `default_realm = ...` line
vi /etc/ssh/ssh_config
# Change `GSSAPIDelegateCredentials yes`

kinit -S kadmin/admin <USER>/admin
./newrsync-squeeze.sh
/afs/club/system/scripts/sh/newmachine-squeeze.sh
# might have to kadmin add -r host/<MACHINE>.club.cc.cmu.edu
kadmin ext host/<MACHINE>.club.cc.cmu.edu

reboot

Building an Etch DomU

To build a Lenny DomU look at the install instructions in mkasick's public/lenny/ folder.

Netreg machine name with cl0x and an empty MAC to get an IP. Then use the IP to update that record with the correct MAC.

Implicit step:

Create LVM volumes for the DomU

Mkasick's magic image, uncompress it onto the root device.

Make the swap partition.

Make the afs cache filesystem.

Mount the root filesystem (probably want to chroot).

Update config files on the new domU.

Do magic on the domU to fix the change from Xen emulating device tty1 to hvc0 (could put in magic image eventually)

Remove udev persistent net rules the domU.

Xen configuration file on the dom0.

If installing without AFS, don't have the AFS Cache in /etc/fstab. If afscache is ext3 instead of ext2, make that change in fstab as well.

Start the domain.

Upgrade packages.

Create kerb instances for the machine.

If not using afs, copy the rsync script from a machine that does have afs and run it as kerb admin.

Run the new machine script as kerb admin.

By default only passwd.admin is allowed to log in.

Add to DNS

Building a Dom0

Install lenny:

Boot from the netinst CD.
Tab on the Expert Install item and add 'auto=true url=www.club.cc.cmu.edu' before the '--'.
Press enter.

Select "Detect and mount CD-ROM"
Select "Load installer components from CD"
Select "Detect network hardware"
Select "Configure the network"

Choose the correct network device (probably eth0).
Manually define the IP settings.
Manually define the hostname and domainname (these should be all lowercase).

Select "Support for automatic installs"
Select "Download debconf preconfiguration file"

That's it!

Install Xen (optional):

Clubification:

Install etch:

Before "Partition disks":

#the actual cylinders don't really matter
#just make sure that hda1 starts at 1 and is 128M
fdisk /dev/hda
    /dev/hda1:   1-  260 83  (128 MB) # /boot
    /dev/hda2: 261-      8e           # lvm

modprobe dm-mod
pvcreate /dev/hda2
vgcreate xx.root /dev/hda2
lvcreate -L 1G -n xx.root xx.root      # /
lvcreate -L 512M -n xx.swap xx.root    # swap
#Size of xx.xensave should be the same as the amount of physical ram
lvcreate -L 512M -n xx.xensave xx.root # /var/lib/xen/save

During "Partition disks":
/boot should be jfs
xx.root should be jfs
xx.swap should be swap
xx.xensave should be jfs

Install linux-image-2.6-686.
During package selection, do not install base system.
Install grub to MBR.

After reboot:

dpkg-reconfigure debconf
Change priority to medium.

apt-get update
apt-get dist-upgrade
apt-get install vim

vi /boot/grub/menu.lst
#Replace \n with a new line
Add "serial --unit=0 --speed=9600\nterminal serial" before
"BEGIN AUTOMATIC KERNELS LIST".
Change "# kopt=root=/dev/mapper/xx.root-xx.root ro console=ttyS0".
Change "# xenhopt=com1=9600,8n1".
Change "# xenkopt=".

vi /etc/inittab
Uncomment "#TO:23:respawn:/sbin/getty -L ttyS0 9600 vt100".

#If on an IA32 system:
apt-get install bridge-utils libc6-xen linux-image-2.6-xen-686 xen-hypervisor-3.0.3-1-i386-pae

#If on an AMD64 system:
#apt-get install bridge-utils linux-image-2.6-xen-amd64 xen-hypervisor-3.0.3-1-amd64

vi /etc/default/xendomains
Change 'XENDOMAINS_SAVE=""'. #do not forget quotes

vi /etc/network/interfaces
Change "allow-hotplug eth0" to "auto br0".
Change "iface eth0 inet static" to "iface br0 inet static".
Add "bridge_ports eth0" under "gateway 128.237.157.1".

vi /etc/apt/apt.conf
Replace with 'APT::Default-Release "etch-cclub";'

vi /etc/apt/sources.list
Add "deb http://debian.club.cc.cmu.edu/debian/ etch-cclub contrib".

apt-get update
apt-get dist-upgrade

apt-get install linux-image-2.6.18-4-xen-686 linux-image-2.6.18-5-xen-686 linux-image-2.6.18-6-xen-686 linux-image-2.6.24-1-xen-686

apt-get install openssh-client
scp unix.club.cc.cmu.edu:/afs/club/service/etc/skel/packages.etch.dom* .
apt-get install `cat packages.etch.domU`
apt-get install `cat packages.etch.dom0`

#adjust hostname as necessary, username as necessary
kinit mkasick/admin
kadmin add -r host/osmium.club.cc.cmu.edu
kadmin ext host/osmium.club.cc.cmu.edu

#if this is a non-AFS domain
scp unix.club.cc.cmu.edu:/afs/club/system/scripts/sh/newrsync-etch.sh .
./newrsync-etch.sh
#endif

/afs/club/system/scripts/sh/newmachine-etch.sh

shutdown -r now