Squeeze
You should only install Squeeze DomUs on Squeeze Dom0s. However, Squeeze Dom0s should support Etch and Lenny DomUs.
DomU
Notes: Make sure you have /sbin and /usr/sbin in your path. A quick fix is to ssh into the DomO as root from the DomO.
Step 1: Register domU on netreg.net.cmu.edu using the cl0x user
Step 2: Update the zonefile information so DNS behaves properly
Edit the file /afs/club/service/dns/DB.club.cc.cmu.edu with this new set of lines at an appropriate place. Include contact information if the machine is being hosted for an outside group.
# the_machine_name +XXXXXX.club.cc.cmu.edu:127.237.157.XXX:7200
IMPORTANT: Before closing the file, update the timestamp at the first line. It should look something like this:
Zclub.cc.cmu.edu:sodium.club.cc.cmu.edu.:gripe.club.cc.cmu.edu.:2012042000:3600:600:9600:3600:86400
Edit the number of the form 20XXXXXXXX to be a new datestamp. It is very important that this number be strictly greater than the old number, otherwise terrible things will happen.
Note: These changes propagate through the machines at Sync Time. Until this has happened, you will not be able to ssh from other machines using DNS, and ksu will not work.
Step 3: Add some info about the machine to the CategoryInfrastructure page.
Step 4: Find a dom0 to host the domU on. http://www.club.cc.cmu.edu/~kbare/list-xen.cgi is helpful.
# change XXX as appropriate # for instance: squeeze, 188, BC (respectively) # feel free to change memory and disk sizes as well xen-create-image --hostname XXXXXX.club.cc.cmu.edu --memory=256Mb --size=4Gb \ --swap=1Gb --ip=128.237.157.XXX --mac=00:00:80:ed:9d:XX \ --broadcast=128.237.157.255 --gateway=128.237.157.1 --netmask=255.255.255.0 \ --lvm=dom0.root # IMPORTANT: remember root password #fixup long hostname -> short hostname in a few places mv /etc/xen/XXXXXX.club.cc.cmu.edu.cfg /etc/xen/XXXXXX vim /etc/xen/XXXXXX # change name = 'XXXXXX.club.cc.cmu.edu' to 'XXXXXX' # change LVs from XXXXXX.club.cc.cmu.edu-{disk,swap} to XXXXXX-{disk,swap} :%s/XXXXXX.club.cc.cmu.edu/XXXXXX/g lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-disk XXXXXX-disk lvrename /dev/dom0.root/XXXXXX.club.cc.cmu.edu-swap XXXXXX-swap # To have it start on bootup mkdir -p /etc/xen/auto ln -s /etc/xen/XXXXXX /etc/xen/auto xm create -c XXXXXX #login, and then change root passwd !!! passwd dpkg-reconfigure debconf Change priority to medium. scp www-node-1:/afs/club/service/etc/skel/squeeze/packages* . # you can't use unix for this anymore scp www-node-1:/afs/club/system/scripts/sh/newrsync-squeeze.sh . # or this apt-get update aptitude install `cat packages.squeeze.domU` # Mail server configuration # - mail sent by smarthost; no local mail # - defaults except: # + "system mail name" => "<host>.club.cc.cmu.edu" # + "visible domain name" => "club.cc.cmu.edu" (no machine name) # + "outgoing smarthost" => "smtp.club.cc.cmu.edu" (add smtp) # ca-certificate configuration # - yes # man-db # - no # Kerberos and PAM # - defaults for everything vi /etc/krb5.conf # add `allow_weak_crypto = true` under `default_realm = ...` line vi /etc/ssh/ssh_config # Change `GSSAPIDelegateCredentials yes` and uncomment that line if not already uncommented kinit -S kadmin/admin [YOU]/admin ./newrsync-squeeze.sh /afs/club/system/scripts/sh/newmachine-squeeze.sh # might have to kadmin add -r host/XXXXXX.club.cc.cmu.edu kadmin ext host/XXXXXX.club.cc.cmu.edu # to set the default locale to something reasonable # we generally install all en_US locales, and set en_US.UTF8 as the system default dpkg-reconfigure locales
Step 5: If you are setting up AFS access perform the following steps. Otherwise you are DONE.
su rsync crontab -e # delete entries # close rsync shell aptitude install linux-headers-<version>-xen-amd64 linux-image-<version>-xen-amd64 # on host lvcreate -L 1G -n XXXXXXX-afscache dom0.root emacs /etc/xen/XXXXXX # add mapping for new disc /etc/init.d/xendomains restart # there is probably a more elegant way to do the above but I don't know it # on guest aptitude install libpam-afs-session openafs-client openafs-fileserver openafs-krb5 openafs-modules-dkms openafs-modules-source # try to maximize space of afs cache used but do not come too close as its estimation algorithm is weird # do not encrypt queries # everything else defaults reboot # you should now see entries in /afs
Step 6: Perform the following steps if you are setting up a shell machine. Otherwise you are DONE.
aptitude install $(cat /afs/club/service/etc/skel/squeeze/packages.squeeze.shell-extra) # for zephyr-clients config, set servers to: zephyr1.club.cc.cmu.edu zephyr2.club.cc.cmu.edu emacs /etc/pam.d/common-session # add the line # session required pam_afs_session.so minimum_uid=110 # as the second line # rsync crap for shell machines # this is a dirty # /bin/rsync -av the /var/rsync directory from an existing shell machine # rsync should be in /etc/user (should be in /etc/shadow automatically) # this is because they need to run cronjobs; is this still the case? # root's crontab should include a motd update script # rsync crontab should have the rsync-master.sh script at 50 pass the hour # /var/mkasick is part of a DNS system; ignore it # ssh host keys need to match each other (/etc/ssh/ssh_hosts*) # rsync them from existing shell machines # kerberoskerberoskerberos (gssapi) # this is from an email by mkasick: # This one step in particular, is probably the most nuanced of things we do. I do it entirely too cautiously, becuase I'd rather not trash the heimdal database. Steps are: 0. Delete "/var/heimdal/delme_the_next_time_you_see_this" on sodium. 1. Backup existing Heimdal database on sodium (/var/heimdal/heimdal.db). 2. Run "/etc/init.d/kdc stop; /etc/init.d/ipropd-master stop" on sodium. 3. Run "kadmin -l", "dump fooasdfbar" 4. Open fooasdfbar, replace "host/foo.club.cc.cmu.edu" line with copy of "host/unix.club.cc.cmu.edu" line. 's/unix/foo/' on that line. In effect, you're replacing the contents of the host/foo key with the host/unix key. See host/cobalt for example. Save as barasdfbar. 5. "diff -U 0 fooasdfbar barasdfbar" and make sure only the appropriate line has been changed. 6. "rm /var/heimdal/heimdal.db" 7. "kadmin -l", "load barasdfbar", "dump bazasdfbar" 8. "cmp barasdfbar bazasdfbar" to ensure they're the same. 9. Run "/etc/init.d/kdc stop; /etc/init.d/ipropd-slave stop" on potassium, barium. 10. "rm /var/heimdal/heimdal.db" on potassium, barium. 11. "/etc/init.d/kdc start; /etc/init.d/ipropd-master start" on sodium. 12. "/etc/init.d/ipropd-slave start" on potassium, barium. 13. Verify iprop worked correctly by "ls -l /var/heimdal/heimdal.db" on potassium, barium, and looking at the output of /var/heimdal/slave-stats on sodium. 14. Run "/etc/init.d/kdc start" on potassium, barium. 15. Delete fooasdfbar, barasdfbar, bazasdfbar, or alternatively place them in a folder called "delme_the_next_time_you_see_this" if you're like me and liable to have screwed this up. # if keys become desynchronized to this # sourced from https://wiki.club.cc.cmu.edu/org-auth/ccwiki/Infrastructure/cobalt.club.cc.cmu.edu # now any any machine on which the database has been changed needs to run rm /etc/krb5.keytab kadmin ext host/XXXXXX.club.cc.cmu.edu # note that XXXXXX is strictly the name of the machine (it doesn't seem to have to be unix)
Dom0
Install Debian:
Choose your own adventure. You can either netboot, or use a netinst CD.
Netboot
You will need to know the MAC address for the machine you need to install.
On a PE2850, you can:
Hit F2 to enter setup.
Go under the "Integrated Devices" menu. Make sure the first gigabit ethernet adapter has PXE booting enabled. Also, note the MAC address.
Now, go to NetReg. If you want to use a new hostname, you will need to register a new machine. Be sure to specify the correct MAC address. Otherwise, if you are re-using an old hostname, got to its entry, change the MAC address, and click update.
Now click the "View Advanced Options" link.
Now scroll down to the "DHCP Options" section and click the "Add DHCP Option" link.
Use this to add two options:
filename |
"/netinstall-squeeze/debian-installer/amd64/pxelinux.0" |
next-server |
storage-2.club.cc.cmu.edu |
If you need to install a non-64-bit machine, replace "amd64" with "i386".
Wait for the change to propagate to the Andrew DHCP servers.
Once the change has propagated, you can usually netboot the machine by pressing F12 as its booting. (If that doesn't work, you can try moving network booting earlier in the BIOS boot sequence.)
This will bring you to a boot menu. There's a "Cclub Options" sub-menu with various useful setup configurations. Choose serial or VGA console depending on which way you're accessing the machine. Generally you'll want to use one of the automated install options, rather than manual partitioning. Which of /dev/sda or /dev/sdb is correct depends on whether the machine has a DRAC virtual drive enabled.
Netinst CD
Boot the netinst CD. Use an AMD64 CD whenever possible (e.g., for 2850s). Open the "Advanced options" menu. Highlight the "Expert install" item and press tab. Edit the command line, adding 'auto=true url=www.club.cc.cmu.edu' before the '--'. Press enter. Select "Detect and mount CD-ROM." Select "Continue" at all prompts. Select "Load installer components from CD." Select "Continue" at all prompts. Select "Detect network hardware."
Debian Installer
Select "Configure the network." Choose the correct network device (probably eth0). Manually define the IP settings. # Add DNS info: 128.237.157.12, 128.237.157.14 Manually define the hostname and domainname (these should be all lowercase). Select "Download debconf preconfiguration file" to start a mostly-automated install. You will be prompted to: 1. Set a root password. Use the club root password if you know it. 2. Confirm writing the partition table to disk 3. Confirm formatting filesystems. When the installation completes, the machine will reboot into the newly installed Debian system. Note: If you need to edit the preconfiguration file, the file is in /afs/club.cc.cmu.edu/www/d-i/. In particular, you may need to replace /dev/sda with /dev/sdb on 2850 machines.
Install Xen (can be skipped if the machine will be dedicated to a single service):
aptitude install xen-hypervisor-4.0-amd64 linux-image-xen-amd64 xen-tools # (OR, for non-amd64 machines): aptitude install xen-hypervisor-4.0-i386 linux-image-xen-686 xen-tools vi /etc/network/interfaces # %s/eth0/br0/g, %s/allow-hotplug/auto/g, add `bridge_ports eth0` to the br0 stanza mkdir /etc/xen/auto
Clubification:
dpkg-reconfigure debconf # Select "Dialog" interface # Select "medium" priority scp unix:/afs/club/service/etc/skel/squeeze/packages* . scp unix:/afs/club/system/scripts/sh/newrsync-squeeze.sh . aptitude install `cat packages.squeeze.domU` # For all debconf pop-ups, select the defaults, except: # exim4-config - Select "mail sent by smarthost; no local mail" # exim4-config - Use "visible domain name" => "club.cc.cmu.edu" (no machine name) # exim4-config - Use "outgoing smarthost" => "smtp.club.cc.cmu.edu" (add smtp) aptitude install `cat packages.squeeze.dom0` # For all debconf pop-ups, select the defaults vi /etc/krb5.conf # Add `allow_weak_crypto = true` under `default_realm = ...` line vi /etc/ssh/ssh_config # Change `GSSAPIDelegateCredentials yes` kinit -S kadmin/admin <USER>/admin ./newrsync-squeeze.sh /afs/club/system/scripts/sh/newmachine-squeeze.sh # might have to kadmin add -r host/<MACHINE>.club.cc.cmu.edu kadmin ext host/<MACHINE>.club.cc.cmu.edu reboot
Building an Etch DomU
To build a Lenny DomU look at the install instructions in mkasick's public/lenny/ folder.
Netreg machine name with cl0x and an empty MAC to get an IP. Then use the IP to update that record with the correct MAC.
note IP address => MAC function
- just encode the IP in hex as the MAC address
- ok, there will never be Cray ethernet hardware in B6
Implicit step:
- apt-get update
- apt-get dist-upgrade
- If krb5-config gets upgraded, rsync /etc/krb5.conf from bromine
- If kernel upgrade is required, then it takes a while
Create LVM volumes for the DomU
- pick a short name for the domU (eg atomic symbol)
- figure out short name of dom0 (hint: vgdisplay)
- lvcreate -L 1G -n [domU-short].root [dom0-short].root
- lvcreate -L 1G -n [domU-short].swap [dom0-short].root
- lvcreate -L 1G -n [domU-short].afscache [dom0-short].root
Mkasick's magic image, uncompress it onto the root device.
- on osmium (and most dom0s too)
- etch-i386-2007041000.img.bz2
bzcat etch-etc.img.bz2 > /dev/[dom0-short].root/[domU-short].root
- note: jfs filesystem
Make the swap partition.
- mkswap /dev/[dom0-short].root/[domU-short].swap
Make the afs cache filesystem.
- mkfs.ext3 /dev/[dom0-short].root/]domU-short].afscache
Mount the root filesystem (probably want to chroot).
- mkdir /mnt/[domU-short].root
- mount /dev/[dom0-short].root/[domU-short].root /mnt/[domU-short].root
- mount -o remount,resize /mnt/domU-short].root
if you made a > 1GB root LV
- chroot /mnt/[domU-short].root
Update config files on the new domU.
- Need to make sure they have the right IP, MAC, hostname, etc.
- /etc/hostname
- /etc/hosts
- /etc/network/interfaces
- /etc/fstab
Do magic on the domU to fix the change from Xen emulating device tty1 to hvc0 (could put in magic image eventually)
- /etc/inittab
- change the relevant line (eg 1:2345:respawn:/sbin/getty 38400 tty1) to 1:2345:respawn:/sbin/getty 38400 hvc0
- /etc/securetty
- add hvc0
Remove udev persistent net rules the domU.
- rm /etc/udev/rules.d/z25_persistent-net.rules
Xen configuration file on the dom0.
- /etc/xen/...
- symlink from auto, so starts on boot
If installing without AFS, don't have the AFS Cache in /etc/fstab. If afscache is ext3 instead of ext2, make that change in fstab as well.
Start the domain.
Upgrade packages.
- apt-get update
- apt-get dist-upgrade
- apt-get install linux-modules-2.6.24-1-xen-686
- apt-get install openafs-modules-2.6.24-1-xen-686
- apt-get install openafs-client
- apt-get install libpam-afs-session
- max cache size for 1G afscache is 980400 kb
- do not dynamically generate /afs
- get /afs/club/service/etc/skel/packages.etch.domU into /root
- replaces old version in magic image
- afs may or may not be working until reboot, may have to scp off another machine
- cat /root/packages.etch.domU | xargs apt-get install -y --force-yes
- exim4 configuration
- do not split config files
- smarthost, no local mail
- accept defaults until...
"visible domain name" => "club.cc.cmu.edu" (no machine name)
"outgoing smarthost" => "smtp.club.cc.cmu.edu" (add smtp)
- accept the defaults for the rest
- exim4 configuration
- you will probably want to reboot at this point
Create kerb instances for the machine.
- kinit you/admin
- kadmin add -r host/[domU].club.cc.cmu.edu
- accept defaults
- kadmin ext host/[domU].club.cc.cmu.edu
If not using afs, copy the rsync script from a machine that does have afs and run it as kerb admin.
- kinit you/admin
- /afs/club.cc.cmu.edu/system/scripts/sh/newrsync-etch.sh
Run the new machine script as kerb admin.
- kinit you/admin
- /afs/club/system/scripts/sh/newmachine-etch.sh
By default only passwd.admin is allowed to log in.
- touch /etc/passwd.user
- will allow all users on the next /etc/passwd sync
Add to DNS
- /afs/club/service/dns/DB.club.cc.cmu.edu
Building a Dom0
Install lenny: Boot from the netinst CD. Tab on the Expert Install item and add 'auto=true url=www.club.cc.cmu.edu' before the '--'. Press enter. Select "Detect and mount CD-ROM" Select "Load installer components from CD" Select "Detect network hardware" Select "Configure the network" Choose the correct network device (probably eth0). Manually define the IP settings. Manually define the hostname and domainname (these should be all lowercase). Select "Support for automatic installs" Select "Download debconf preconfiguration file" That's it! Install Xen (optional): Clubification:
Install etch: Before "Partition disks": #the actual cylinders don't really matter #just make sure that hda1 starts at 1 and is 128M fdisk /dev/hda /dev/hda1: 1- 260 83 (128 MB) # /boot /dev/hda2: 261- 8e # lvm modprobe dm-mod pvcreate /dev/hda2 vgcreate xx.root /dev/hda2 lvcreate -L 1G -n xx.root xx.root # / lvcreate -L 512M -n xx.swap xx.root # swap #Size of xx.xensave should be the same as the amount of physical ram lvcreate -L 512M -n xx.xensave xx.root # /var/lib/xen/save During "Partition disks": /boot should be jfs xx.root should be jfs xx.swap should be swap xx.xensave should be jfs Install linux-image-2.6-686. During package selection, do not install base system. Install grub to MBR. After reboot: dpkg-reconfigure debconf Change priority to medium. apt-get update apt-get dist-upgrade apt-get install vim vi /boot/grub/menu.lst #Replace \n with a new line Add "serial --unit=0 --speed=9600\nterminal serial" before "BEGIN AUTOMATIC KERNELS LIST". Change "# kopt=root=/dev/mapper/xx.root-xx.root ro console=ttyS0". Change "# xenhopt=com1=9600,8n1". Change "# xenkopt=". vi /etc/inittab Uncomment "#TO:23:respawn:/sbin/getty -L ttyS0 9600 vt100". #If on an IA32 system: apt-get install bridge-utils libc6-xen linux-image-2.6-xen-686 xen-hypervisor-3.0.3-1-i386-pae #If on an AMD64 system: #apt-get install bridge-utils linux-image-2.6-xen-amd64 xen-hypervisor-3.0.3-1-amd64 vi /etc/default/xendomains Change 'XENDOMAINS_SAVE=""'. #do not forget quotes vi /etc/network/interfaces Change "allow-hotplug eth0" to "auto br0". Change "iface eth0 inet static" to "iface br0 inet static". Add "bridge_ports eth0" under "gateway 128.237.157.1". vi /etc/apt/apt.conf Replace with 'APT::Default-Release "etch-cclub";' vi /etc/apt/sources.list Add "deb http://debian.club.cc.cmu.edu/debian/ etch-cclub contrib". apt-get update apt-get dist-upgrade apt-get install linux-image-2.6.18-4-xen-686 linux-image-2.6.18-5-xen-686 linux-image-2.6.18-6-xen-686 linux-image-2.6.24-1-xen-686 apt-get install openssh-client scp unix.club.cc.cmu.edu:/afs/club/service/etc/skel/packages.etch.dom* . apt-get install `cat packages.etch.domU` apt-get install `cat packages.etch.dom0` #adjust hostname as necessary, username as necessary kinit mkasick/admin kadmin add -r host/osmium.club.cc.cmu.edu kadmin ext host/osmium.club.cc.cmu.edu #if this is a non-AFS domain scp unix.club.cc.cmu.edu:/afs/club/system/scripts/sh/newrsync-etch.sh . ./newrsync-etch.sh #endif /afs/club/system/scripts/sh/newmachine-etch.sh shutdown -r now