This really isn't rocket science.
When the DomU and both Dom0s are all running the same Debian version:
- Identify the DomU's block devices. Examine its configuration file in /etc/xen/«name». Usually it will simply be LVM LVs in the general pattern /dev/dom0.root/«name»-«whatever». But sometimes it can be more complicated (e.g., www-nodes have an extra disk dedicated for caching; hence to move a www-node, you will also need to physically move that disk, or setup a new dedicated disk at the destination Dom0).
- Shutdown the DomU.
- Create new block devices at the destination Dom0 with appropriate sizes.
I usually use lvscan to help figure this out, E.g.,:
root@radium:~# lvscan ACTIVE '/dev/dom0.root/dom0.swap' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/dom0.root' [2.00 GiB] inherit ACTIVE '/dev/dom0.root/bt2-swap' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/bt2-disk' [4.00 GiB] inherit ACTIVE '/dev/dom0.root/bt2-afscache' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/bt2-tmp' [2.00 GiB] inherit ACTIVE '/dev/dom0.root/svn-swap' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/svn-disk' [4.00 GiB] inherit ACTIVE '/dev/dom0.root/svn-afscache' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/www-node-2-swap' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/www-node-2-root' [4.00 GiB] inherit ACTIVE '/dev/dom0.root/www-node-2-afscache' [2.00 GiB] inherit ACTIVE '/dev/dom0.root/www-node-2-userlogs' [2.00 GiB] inherit
Then use lvcreate -n «lvname» -L «lvsize» «vgname»; the VG will almost always be dom0.root, unless the Dom0 was set up in a non-standard fashion. E.g.,
root@cesium:~# lvcreate -n svn-swap -L 1G dom0.root Logical volume "svn-swap" created root@cesium:~# lvcreate -n svn-disk -L 4G dom0.root Logical volume "svn-disk" created root@cesium:~# lvcreate -n svn-afscache -L 1G dom0.root Logical volume "svn-afscache" created
To verify you've created the desired LVs, you can run lvscan on the destination Dom0. E.g.,
root@cesium:~# lvscan ACTIVE '/dev/dom0.root/dom0.root' [3.81 GiB] inherit ACTIVE '/dev/dom0.root/dom0.swap' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/snail-root' [10.00 GiB] inherit ACTIVE '/dev/dom0.root/snail-swap' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/svn-swap' [1.00 GiB] inherit ACTIVE '/dev/dom0.root/svn-disk' [4.00 GiB] inherit ACTIVE '/dev/dom0.root/svn-afscache' [1.00 GiB] inherit
If you are paranoid, you can double-check that the destination devices are large enough by comparing blockdev --getsz «dev» at the source and destination Dom0s.
- Copy the data from each source block device to the corresponding destination block device as needed. (I say as needed, because some do not contain any data that's truly persistent. The most common case is for -swap and -afscache devices.)
You can use the dd and ssh commands to do the copying. E.g.,
root@radium:~# dd if=/dev/dom0.root/svn-disk bs=4M | ssh -e none cesium dd bs=4M of=/dev/dom0.root/svn-disk 1024+0 records in 1024+0 records out 4294967296 bytes (4.3 GB) copied, 153.976 s, 27.9 MB/s 0+261858 records in 0+261858 records out 4294967296 bytes (4.3 GB) copied, 154.738 s, 27.8 MB/s
Using "-e none" is important! Otherwise, if the ssh escape sequence (<CR>~) exists in the block device, it will screw up the copy.
- Properly initialize any devices that did not need to be copied. E.g.,
root@cesium:~# mkswap /dev/dom0.root/svn-swap mkswap: /dev/dom0.root/svn-swap: warning: don't erase bootbits sectors on whole disk. Use -f to force. Setting up swapspace version 1, size = 1048572 KiB no label, UUID=83760789-8312-4c4f-bbc3-660a5babe810 root@cesium:~# mkfs.ext2 /dev/dom0.root/svn-afscache mke2fs 1.42.5 (29-Jul-2012) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 65536 inodes, 262144 blocks 13107 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Writing superblocks and filesystem accounting information: done
- Copy /etc/xen/«name» from the source Dom0 to the destination Dom0.
- Remove /etc/xen/auto/«name» from the source Dom0.
- Usually this is not necessary, but sometimes you may need to edit the block device paths in the configuration (at the destination Dom0). This would needed if, for example, either the source or destination used a non-standard VG name (i.e., NOT dom0.root). Just make sure the block device paths are correct for the destination Dom0.
Try to start the DomU on the destination Dom0. I.e., xm create -c «name». Verify it boots successfully. If everything looks good, you can exit the DomU console with ^].
- On the source Dom0, rename /etc/xen/«name» to something like /etc/xen/«name».moved-to-«destdom0».«date». This is to make it less likely for the DomU to get started in two places.
- Make it so the DomU will automatically start on the destination Dom0. E.g.,
snail login: root@cesium:~# cd /etc/xen/auto root@cesium:/etc/xen/auto# ln -s ../svn root@cesium:/etc/xen/auto# ls -hl total 0 lrwxrwxrwx 1 root root 14 Oct 12 20:05 snail -> /etc/xen/snail lrwxrwxrwx 1 root root 6 Oct 25 18:36 svn -> ../svn
- Remove old volumes on the source DomU (lvscan is helpful here). e.g.,
root@thallium:~# lvremove dom0.root/rgh-swap Do you really want to remove active logical volume rgh-swap? [y/n]: y Logical volume "rgh-swap" successfully removed
When the DomU is running squeeze and the destination Dom0 is running wheezy
This is slightly complicated by the fact you need the squeeze kernel installed on the wheezy Dom0. (Though you will still be running the wheezy kernel!) That just requires a little bit of apt ninjary.
Put the following in /etc/apt/preferences.d/squeeze-kernel:
Package: * Pin: release n=squeeze Pin-Priority: -1 Package: linux-image-2.6.32-5-xen-amd64 Pin: release n=squeeze Pin-Priority: 500 Package: linux-image-2.6.32-5-xen-686 Pin: release n=squeeze Pin-Priority: 500
It's basically saying that squeeze packages should be uninstallable, except for the Xen kernel needed to boot squeeze DomUs.
Then add the following to /etc/apt/sources.list:
# pull from the squeeze repos as well (for older domU kernels) deb http://mirrors.mit.edu/debian/ squeeze main deb-src http://mirrors.mit.edu/debian/ squeeze main deb http://security.debian.org/ squeeze/updates main deb-src http://security.debian.org/ squeeze/updates main
So it will look in the squeeze repo for packages (and find the kernel). (Tick... tick... tick... eventually the regular Debian mirrors are going to stop hosting squeeze. When that happens, the URLs will need to be adjusted to go to some mirror of archive.debian.org.)
Then:
aptitude update aptitude install linux-image-2.6.32-5-xen-amd64 # or, for i386: aptitude install linux-image-2.6.32-5-xen-686
After that, moving the DomU is no different than the like-versions case detailed above.
When the DomU is running something really old... (lenny, etch, ... sarge‽)
At this point, the apt_preferences trick above doesn't work so well, since Debian's made changes in the way initrds work.
What I've been doing is installing the kernel within the DomU, setting up a /boot/grub directory, and using pygrub.
The tricky part is getting a happy grub configuration in the DomU. How to do this varied depending on the Debian version.
Another annoyance is that pygrub doesn't understand jfs. You'll need to recreate the DomU's filesystem as ext3.
If the DomU is already booting via pygrub
Good news: in the case that I (or somebody else) already set up pygrub for an old DomU, you should be able to move it as in the like-versions case without much trouble. The one thing you may need to do is change the bootloader= option in the Xen configuration to look in the correct directory for the Xen hypervisor version the destination Dom0 is running (e.g., if moving from squeeze to wheezy, change xen-4.0 to xen-4.1).
If the DomU is running lenny
This isn't actually that bad. Installing grub *almost* works.
- Install a kernel in the DomU if needed. (While the DomU is still running under the original Dom0.) E.g.,
# (You may first need to edit /etc/apt/source.list to point to an archive.debian.org mirror. aptitude install linux-image-2.6.26-2-xen-amd64 # or -686 for an i386 domU
- Install grub in the DomU. (While the DomU is still running under the original Dom0.):
aptitude install grub # Grub is confused by the fact that our virtual block devices are partitions but # there is no device for the entire disk. We can make one though. # # Edit /etc/rc.local, adding the following lines before the "exit 0" at the end # of the file. (Not including the # comment indicators!) # ---- # if [ -b /dev/xvda1 -a ! -e /dev/xvda ]; then # major=0x`stat -c %t /dev/xvda1` # mknod -m 600 /dev/xvda b $major 0 # fi # ---- vim /etc/rc.local # Then run the script to create the device node. /etc/rc.local # Now, create a /boot/grub directory and device.map file so that # update-grub will cooperate. mkdir -p /boot/grub cat > /boot/grub/device.map << EOF (hd0) /dev/xvda EOF # You can now run update-grub to create an initial /boot/grub/menu.lst. update-grub # But it doesn't put the console on the Xen virtual console by default. # Fix this by finding the "# kopt=" line in /boot/grub/menu.lst and add # "console=hvc0" to the end of the line. vim /boot/grub/menu.lst # And you then need to re-run update-grub so it applies the kopt= change. update-grub
Pygrub does not understand JFS. So if the DomU is using JFS for its root filesystem, you will need to recreate its root volume using something pygrub can understand (like ext3).
Easiest way to do this is to *not* copy the root filesystem to the destination !Dom0 with dd and ssh. Instead, create a new empty device, format it as ext3, and mount it.# On the destination Dom0: lvcreate -L «size» -n «volumename» dom0.root mkfs.ext3 /dev/dom0.root/«volumename» mount /dev/dom0.root/«volumename» /mnt
Stop the DomU on the source Dom0 if you have not already. Then mount its current root volume.# On the source Dom0: mount /dev/dom0.root/«volumename» /mnt
Carefully rsync from the original volume to the new ext3 volume.# On the source Dom0: # XXX: TODO: DANGER: pentuple check arguments rsync -aSH --numeric-ids /mnt/ «destdom0»:/mnt
Lastly, unmount the volumes from both Dom0s.- For other block devices, follow the steps for the like-versions case above.
- Copy/rename the Xen configuration file for the DomU as detailed in the like-versions case above.
- Before starting the DomU at the destination Dom0, you need to make some minor edits to the Xen configuration:
Comment out the kernel= and ramdisk= lines.
- For a jessie Dom0, add (preferably near the commented-out lines):
bootloader = "/usr/lib/xen-4.4/bin/pygrub"
- For a wheezy Dom0, add (preferably near the commented-out lines):
bootloader = "/usr/lib/xen-4.1/bin/pygrub"
- For a squeeze Dom0, add (preferably near the commented-out lines):
bootloader = "/usr/lib/xen-4.0/bin/pygrub"
- Try to start the DomU.
- If it started successfully and appears to be working, continue with the cleanup steps as described in the like-versions case above.
If the DomU is running etch
This isn't actually that bad. Installing grub actually works pretty well, but works less well if the DomU is running a 2.6.24 kernel.
- Install a kernel in the DomU if needed. (While the DomU is still running under the original Dom0.) E.g.,
# (You may first need to edit /etc/apt/source.list to point to an archive.debian.org mirror. aptitude install linux-image-2.6.18-6-xen-686 # or linux-image-2.6.24-1-686, depending on what the DomU is currently running
- Install grub in the DomU. (While the DomU is still running under the original Dom0.):
aptitude install grub # Now, create a /boot/grub directory so that update-grub will cooperate. mkdir -p /boot/grub # You can now run update-grub to create an initial /boot/grub/menu.lst. update-grub # But it doesn't put the console on the Xen virtual console by default. # Fix this by finding the "# kopt=" line in /boot/grub/menu.lst and add # "console=hvc0" to the end of the line. vim /boot/grub/menu.lst # And you then need to re-run update-grub so it applies the kopt= change. update-grub
If the DomU is running a 2.6.24 kernel, grub probably didn't add any kernels to menu.lst. To solve this
# Save Debian's version of update-grub; make it so our changes don't get # clobbered by updates. dpkg-divert --rename --local --divert /usr/sbin/update-grub.debian --add /usr/sbin/update-grub # Fix up the script. sed 's/^in_domU=$/in_domU=1/' /usr/sbin/update-grub.debian > /usr/sbin/update-grub chmod a+x /usr/sbin/update-grub # And re-run update-grub. update-grub
DomUs seem to dislike booting under more recent versions of xen if they have libc6-xen installed. (They'll get some sort of memory fault in the initrd. Be sure to uninstall libc6-xen if it is installed.
Pygrub does not understand JFS. So if the DomU is using JFS for its root filesystem, you will need to recreate its root volume using something pygrub can understand (like ext3).
Easiest way to do this is to *not* copy the root filesystem to the destination !Dom0 with dd and ssh. Instead, create a new empty device, format it as ext3, and mount it.# On the destination Dom0: lvcreate -L «size» -n «volumename» dom0.root mkfs.ext3 /dev/dom0.root/«volumename» mount /dev/dom0.root/«volumename» /mnt
Stop the DomU on the source Dom0 if you have not already. Then mount its current root volume.# On the source Dom0: mount /dev/dom0.root/«volumename» /mnt
Carefully rsync from the original volume to the new ext3 volume.# On the source Dom0: # XXX: TODO: DANGER: pentuple check arguments rsync -aSH --numeric-ids /mnt/ «destdom0»:/mnt
Lastly, unmount the volumes from both Dom0s.- For other block devices, follow the steps for the like-versions case above.
- Copy/rename the Xen configuration file for the DomU as detailed in the like-versions case above.
- Before starting the DomU at the destination Dom0, you need to make some minor edits to the Xen configuration:
Comment out the kernel= and ramdisk= lines.
- For a jessie Dom0, add (preferably near the commented-out lines):
bootloader = "/usr/lib/xen-4.4/bin/pygrub"
- For a wheezy Dom0, add (preferably near the commented-out lines):
bootloader = "/usr/lib/xen-4.1/bin/pygrub"
- For a squeeze Dom0, add (preferably near the commented-out lines):
bootloader = "/usr/lib/xen-4.0/bin/pygrub"
- Try to start the DomU.
- If it started successfully and appears to be working, continue with the cleanup steps as described in the like-versions case above.