First, locally install the appropriate packages.

apt-get install virtinst virt-manager virt-viewer dconf-tools

Getting a Libvirt Connection

I assume that either your local username is the same as your club username, or you've configured SSH to use your club username whenever you try to connect to a club host (see Useful clientside CClub configurations).

If you're going to use the CLI remotely, I recommend turning on shared, persistent SSH connections as detailed in Useful clientside CClub configurations, to prevent each action opening and closing a new SSH connection.

Note that SSH configurations work in libvirt URIs; so, for example, if you have

# ~/.ssh/config
Host kvm01
    User sbaugh
    Hostname kvm01.club.cc.cmu.edu

then you can connect to qemu+ssh://kvm01/system.

CLI

# ~/.config/libvirt/libvirt.conf
uri_aliases = [
  "kvm01=qemu+ssh://kvm01.club.cc.cmu.edu/system",
  "kvm02=qemu+ssh://kvm02.club.cc.cmu.edu/system",
  "kvm03=qemu+ssh://kvm03.club.cc.cmu.edu/system",
  "kvm04=qemu+ssh://kvm04.club.cc.cmu.edu/system",
]

#uri_default = "qemu:///system"
#uri_default = "kvm01"

When working with a specific host on the command line, you can either:

# on your own machine
virsh -c kvm01 list
virsh -c kvm01 start myvm
virsh -c kvm01 stop myvm
virt-install --connect kvm01 --name myvm

or,

# on your own machine
export LIBVIRT_DEFAULT_URI=kvm01
virsh list
virsh start myvm
virsh stop myvm
virt-install --name myvm

Alternatively, you could directly ssh to the host to do all these things, like so:

ssh kvm01.club.cc.cmu.edu
# on kvm01.club.cc.cmu.edu
virsh -c qemu:///system list
virsh -c qemu:///system start myvm
virsh -c qemu:///system stop myvm
virt-install --connect qemu:///system --name myvm

GUI

To tell virt-manager about all the CClub libvirt hosts, run one of the following commands on the machine you are running virt-manager on. Note that this will wipe out any existing connections you've configured in virt-manager.

dconf write /org/virt-manager/virt-manager/connections/uris "
['qemu+ssh://kvm01.club.cc.cmu.edu/system',
'qemu+ssh://kvm02.club.cc.cmu.edu/system',
'qemu+ssh://kvm03.club.cc.cmu.edu/system',
'qemu+ssh://kvm04.club.cc.cmu.edu/system']"

# ALTERNATIVELY: to connect to Xen hosts too
dconf write /org/virt-manager/virt-manager/connections/uris "
['qemu+ssh://kvm01.club.cc.cmu.edu/system',
'qemu+ssh://kvm02.club.cc.cmu.edu/system',
'qemu+ssh://kvm03.club.cc.cmu.edu/system',
'qemu+ssh://kvm04.club.cc.cmu.edu/system',
'xen+ssh://root@neon.club.cc.cmu.edu/',
'xen+ssh://root@polonium.club.cc.cmu.edu/',
'xen+ssh://root@chromium.club.cc.cmu.edu/',
'xen+ssh://root@cesium.club.cc.cmu.edu/',
'xen+ssh://root@cerium.club.cc.cmu.edu/',
'xen+ssh://root@bohrium.club.cc.cmu.edu/',
'xen+ssh://root@bismuth.club.cc.cmu.edu/',
'xen+ssh://root@erbium.club.cc.cmu.edu/',
'xen+ssh://root@gadolinium.club.cc.cmu.edu/',
'xen+ssh://root@germanium.club.cc.cmu.edu/',
'xen+ssh://root@krypton.club.cc.cmu.edu/',
'xen+ssh://root@mercury.club.cc.cmu.edu/',
'xen+ssh://root@thulium.club.cc.cmu.edu/',
'xen+ssh://root@tungsten.club.cc.cmu.edu/',
'xen+ssh://root@lutetium.club.cc.cmu.edu/']"

Double click on the name of the host in virt-manager to connect to it.

Making a VM

1. Pick a host. Make sure you have set up your connection to it; that is a one-time requirement that is described above.

2. Make a NetReg entry. (NetReg gets you an external IP. If you don't want an external IP, then you don't really need to follow anything in this guide, you can just fumble around and use the host-local NAT.)

3. Make a DNS entry.

4. Pick GUI or CLI and branch.

GUI (virt-manager)

Our usage of virt-manager is not at all custom; so you may find it useful to read random tutorials on the web. Here is one.

1. Click the new VM and select the appropriate host.
2. Select "PXE boot".
3. Select the appropriate OS type and version.
4. Choose your amount of memory and CPU.
5. Choose the size of your disk. (More disk can always be added later, so keep it small, certainly under 20GB)
6. Fill in the hostname of the VM as its name.
7. Under "Advanced Options", select the option starting with "Bridge" (e.g. "Bridge br0 (Host device eth0)"), check "Set a fixed MAC address",

   and fill in the MAC address you set in NetReg.

8. Check "Customize configuration before install".
9. Fill in a description for the VM in the "Basic Details" section of the "Overview" page. Include distro, purpose, and maintainer (you).
10. Click "Begin installation" and you will soon be in the Debian installer, which I'm sure you can navigate yourself. TODO document using the PXE boot installer.
11. After installing with the Debian installer, make sure to

    clubify that machine. For maximum ease of use, choose Network Install on the first step, and paste in

    http://debian.club.cc.cmu.edu/pub/debian/dists/stable/main/installer-amd64/ (You can get other urls from the virt-install manpage)

CLI (virsh/virt-install)

# running this actually gives a useful error message
virt-install

# a full invocation would be:
virt-install --name $VM_HOSTNAME --memory 512 --disk size=10 \
             --location http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/

To find out what, say, --location does, perform the following operation:

1. open the man page
2. search for it

Nevertheless, here are some useful arguments:

• Installation sources
  • --location url will figure out what distro tree is at that URL,
    • download the kernel and initrd and directly run them in the VM. Check the man-page for useful URLs to pass it.
  • --cdrom, --pxe, --import, and others are also available
• Network
  • to not be NAT'd, --network bridge=br0,mac=what:you:put:in:net:reg
• Graphics
  • to force creation of a graphical display, --graphics vnc or --graphics spice
  • to force no creation of graphical display, --graphics none
  • if --graphics something isn't present, it'll choose the appropriate one based on whether DISPLAY is set
• Console
  • to have a text console available, --extra-args console=ttyS0
  • you can attach to the text console with virsh -c url console namium
• Description
  • --metadata description="some brief description"

# FOR THE LAZY: typical cclub invocation
virt-install --name $VM_HOSTNAME --memory 512 --disk size=10 \
             --metadata description="some brief description" \
             --network bridge=br0,mac=whatyouputinnetreg \
             --extra-args console=ttyS0 \
             --location http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/

OK, you're done! You may want to clubify that machine.

Making a Host

# Install libvirt, which pulls in QEMU/KVM by default
apt-get install libvirt-bin virtinst qemu-kvm

Now you can make VMs.

# Configure the networking bridge so we can get non-NAT'd IPs
sensible-editor /etc/network/interfaces
# Add something like the following (assumes you're using DHCP for the
# host rather than static networking)

# auto lo br0
# iface lo inet loopback
#
# iface eth0 inet manual
#
# iface br0 inet dhcp
#       bridge_ports eth0
#       bridge_stp off

Now you can make VMs that can DHCP to get NetReg'd IPs.

Clubify the machine.

Now you can ssh in.

# change something in /etc/libvirt/libvirtd.conf so the unix socket controlling libvirt is accessible by all users, or users in the "wheel" group, or something
# Under "UNIX socket access controls" replace or add  
# unix_sock_group = "wheel"
# default usually "libvirt"
sensible-editor /etc/libvirt/libvirtd.conf

Now you can make VMs as non-root, so you can remotely make VMs over ssh.

libvirtifying an existing Xen host

sensible-editor /etc/xen/xend-config.sxp
# make sure this is present and uncommented:
# (xend-unix-server yes)
service xen restart
apt-get install libvirt-bin netcat-openbsd pm-utils
# libvirt remoting wants a certain version of netcat, what can I say?
# pm-utils is also necessary to prevent a (harmless but annoying) error message

Importing an existing Xen domU to be managed by libvirt

xm destroy whatvm
rm /etc/xen/auto/whatvm

virsh domxml-from-native /etc/xen/whatvm > whatvm.cfg
# add the appropriate variation on <cmdline> root=/dev/xvda2 </cmdline>
# to the /domain/os section
# (sibling to <kernel> and <initrd>)

virsh define whatvm.cfg
virsh start whatvm
virsh autostart whatvm

Converting a VM from Xen to KVM

Ask sbaugh to do it. Alternatively follow this script:

set -o errexit
set -o nounset
# Manually take an image of the root partition of the VM
ROOT_IMAGE=root.img
NEWDISK=disk.img
MOUNTPOINT=mnt

#### make a disk image with a root and boot partition

## This is a bit of a hack - I'd really prefer a tool that does these calculations for me.

# in kilobytes (64 megabytes)
BOOT_PART_SIZE=65536
# in kilobytes
ROOT_IMAGE_SIZE=$(du $ROOT_IMAGE | cut -f 1)
# in kilobytes, with 64 kilobytes of leeway
NEWDISK_SIZE=$(expr $BOOT_PART_SIZE + $ROOT_IMAGE_SIZE + 64)

# make the disk and upload the image
guestfish -N $NEWDISK=bootroot:ext2:ext2:${NEWDISK_SIZE}k:${BOOT_PART_SIZE}k \
           upload $ROOT_IMAGE /dev/sda2

# make the first partition (the boot partition) bootable
parted $NEWDISK set 1 boot on

#### update config files
## mount the disk (automagically guessing which is /)
guestmount -o uid=$UID -a $NEWDISK -i $MOUNTPOINT

## update /etc/fstab
# delete /dev/xvda mounts from /etc/fstab 
# commenting isn't enough - some things erroneously parse lines of /etc/fstab starting with #
sed -i "s:^/dev/xvda.*::" ${MOUNTPOINT}/etc/fstab

# mount root partition by UUID
UUID_ROOT=$(virt-filesystems -a $ROOT_IMAGE -l --uuid --csv | grep sda | cut -f 7 -d ,)
echo "UUID=$UUID_ROOT / ext4 errors=remount-ro 0 1" >> ${MOUNTPOINT}/etc/fstab

# mount boot partition by UUID
UUID_BOOT=$(virt-filesystems -a $NEWDISK -l --uuid --csv | grep sda1 | cut -f 7 -d ,)
echo "UUID=$UUID_BOOT /boot ext2 errors=remount-ro 0 2" >> ${MOUNTPOINT}/etc/fstab

## also update /etc/inittab to not use xen's hvc0
sed -i "s/hvc0/tty1/" ${MOUNTPOINT}/etc/inittab

## disable AFS?

## wipe out networking?
echo "Need to manually intervene at this point to fix networking in /etc/network/interfaces"
## Replace /etc/network/interfaces with:
# auto lo
# iface lo inet loopback
#
# auto eth0
# allow-hotplug eth0
# iface eth0 inet dhcp

## backup old /boot
mv ${MOUNTPOINT}/boot ${MOUNTPOINT}/oldboot
mkdir ${MOUNTPOINT}/boot

## remount disk; now will recognize UUIDs in /etc/fstab and mount /boot
guestunmount $MOUNTPOINT
echo "Just in case, sleeping to let the FUSE unmount finish" 
sleep 15
guestmount -o uid=$UID -a $NEWDISK -i $MOUNTPOINT

## move /boot files into new boot partition
mv ${MOUNTPOINT}/oldboot/* -t ${MOUNTPOINT}/boot/
rmdir ${MOUNTPOINT}/oldboot

## final unmount
guestunmount $MOUNTPOINT
echo "Just in case, sleeping to let the FUSE unmount finish"
sleep 15

#### install syslinux
# write mbr
SYSLINUX_MBR=/usr/lib/syslinux/bios/mbr.bin 
dd if=$SYSLINUX_MBR of=$NEWDISK bs=440 count=1 conv=notrunc

# install in the vm
guestfish -a $NEWDISK -i --network sh "apt-get install extlinux syslinux-common && extlinux --install /boot/extlinux"