Moving disks from Xen to KVM

Overview #

• Moving virtual machines from Xen to KVM
• The storage for both is under LVM
• Under Xen the storage for each VM isn't partitioned as a disk, it's just a filesystem. GRUB under KVM will need to see a boot sector and a partition table.
• Under Xen the storage for each VM doesn't contain a kernel. GRUB under KVM will need one to boot the VM.

Sequence #

• Measure disks on Xen server, create on KVM server. Create the new root disk 10MB larger than its counterpart. (( Later we align the original root filesystem at cylinder 1 on this new disk. A LVM cylinder is roughly 8MB ))

  kyero@xen03 ~ $ sudo lvscan
  

  ACTIVE '/dev/vg4/mail-root' [20.00 GB] inherit ACTIVE '/dev/vg4/mail-swap' [1.00 GB] inherit

  [kyero@vm02 ~]$ sudo lvcreate -L20490M -n mail-root vg4 
  

  [kyero@vm02 ~]$ sudo lvcreate -L2G -n mail-swap vg4

• Create a boot disk on the KVM server. We make it 4 cylinders long, roughly 32MB. (( File backed disk is used rather than LVM owing to grief with GRUB and LVM (Error 22: No such partition) ))

  [kyero@vm02 ~]$ sudo dd if=/dev/zero of=/var/boot-images/mail-boot bs=32901120 count=1
  

• Partition the boot disk with one partition starting at cylinder zero

  [kyero@vm02 ~]$ sudo sfdisk /var/boot-images/mail-boot
  

  last_lba(): I don't know how to handle files with mode 81a4 Warning: /var/boot-images/mail-boot is not a block device Disk /var/boot-images/mail-boot: cannot get geometry

  Disk /var/boot-images/mail-boot: 4 cylinders, 255 heads, 63 sectors/track

  sfdisk: ERROR: sector 0 does not have an msdos signature /var/boot-images/mail-boot: unrecognized partition table type Old situation: No partitions found Input in the following format; absent fields get a default value. <start> <size> <type [E,S,L,X,hex]> <bootable [-,*]> <c,h,s> <c,h,s>\nUsually you only need to specify <start> and <size> (and perhaps <type>).

  /var/boot-images/mail-boot1 :0 /var/boot-images/mail-boot1 0+ 3 4- 32129+ 83 Linux /var/boot-images/mail-boot2 : /var/boot-images/mail-boot2 0 - 0 0 0 Empty /var/boot-images/mail-boot3 : /var/boot-images/mail-boot3 0 - 0 0 0 Empty /var/boot-images/mail-boot4 : /var/boot-images/mail-boot4 0 - 0 0 0 Empty New situation: Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start End #cyls #blocks Id System /var/boot-images/mail-boot1 0+ 3 4- 32129+ 83 Linux /var/boot-images/mail-boot2 0 - 0 0 0 Empty /var/boot-images/mail-boot3 0 - 0 0 0 Empty /var/boot-images/mail-boot4 0 - 0 0 0 Empty Warning: no primary partition is marked bootable (active) This does not matter for LILO, but the DOS MBR will not boot this disk. Do you want to write this to disk? [ynq] y Successfully wrote the new partition table

  Re-reading the partition table ... BLKRRPART: Inappropriate ioctl for device

  If you created or changed a DOS partition, /dev/foo7, say, then use dd(1) to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1 (See fdisk(8).)

• Format and mount the partition

  [kyero@vm02 ~]$ sudo kpartx -av /var/boot-images/mail-boot 
  

  add map loop1p1 : 0 64259 linear /dev/loop1 1

  [kyero@vm02 ~]$ sudo mke2fs /dev/mapper/loop1p1 mke2fs 1.39 (29-May-2006) Filesystem label= OS type: Linux Block size=1024 (log=0) Fragment size=1024 (log=0) 8032 inodes, 32128 blocks 1606 blocks (5.00%) reserved for the super user First data block=1 Maximum filesystem blocks=33030144 4 block groups 8192 blocks per group, 8192 fragments per group 2008 inodes per group Superblock backups stored on blocks: 8193, 24577

  Writing inode tables: done
  Writing superblocks and filesystem accounting information: done

  This filesystem will be automatically checked every 34 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.

  [kyero@vm02 ~]$ sudo mount /dev/mapper/loop1p1 /mnt/boot/

  [kyero@vm02 ~]$ df -h /mnt/boot Filesystem Size Used Avail Use% Mounted on /dev/mapper/loop1p1 31M 389K 29M 2% /mnt/boot

• Copy a kernel to it, configure GRUB's notion of the root device and install the boot sector.

  [kyero@vm02 ~]$ sudo rsync -a /boot/ /mnt/boot/
  

  [kyero@vm02 ~]$ sudo vi /mnt/boot/grub/menu.lst [kyero@vm02 ~]$ sudo touch /mnt/boot/kvm-boot-image [kyero@vm02 ~]$ sudo grub --device-map=/dev/null

  GNU GRUB  version 0.97  (640K lower / 3072K upper memory)
  

  [ Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists the possible completions of a device/filename.] grub> device (hd0) /var/boot-images/mail-boot device (hd0) /var/boot-images/mail-boot grub> find /kvm-boot-image find /kvm-boot-image (hd0,0) grub> root (hd0,0) root (hd0,0) Filesystem type is ext2fs, partition type 0x83 grub> setup (hd0) setup (hd0) Checking if "/boot/grub/stage1" exists... no Checking if "/grub/stage1" exists... yes Checking if "/grub/stage2" exists... yes Checking if "/grub/e2fs_stage1_5" exists... yes Running "embed /grub/e2fs_stage1_5 (hd0)"... failed (this is not fatal) Running "embed /grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal) Done. grub> quit

  [kyero@vm02 ~]$ sudo umount /mnt/boot [kyero@vm02 ~]$ sudo kpartx -d /var/boot-images/mail-boot loop deleted : /dev/loop1

• Populate the new root disk with one cylinder of zeros and a copy of the old root disk. The first cylinder will contain the MBR and not much else. The padding means it's easy to align the old root disk at cylinder one. Use pv so we can see the copy progress. The dd-lvm-out script is mostly just a sudo-blessed wrapper.

  [kyero@vm02 ~]$ ( dd if=/dev/zero bs=8225280 count=1 ; ssh xen03.local -l kyero sudo /data/system/bin/dd-lvm-out.sh mail-root ) | pv -e -r -s 21474836480 | sudo dd bs=1048577 of=/dev/vg4/mail-root
  

  1+0 records in 1+0 records out 8225280 bytes (8.2 MB) copied, 0.03395 seconds, 242 MB/s 20480+0 records in 20480+0 records out 21474836480 bytes (21 GB) copied, 611.215 s, 35.1 MB/s 0+662630 records in 0+662630 records out 21483061795 bytes (21 GB) copied, 615.249 seconds, 34.9 MB/s

• Partition the new root disk with a single partition starting at cylinder one.

  [kyero@vm02 ~]$ sudo sfdisk /dev/vg4/mail-root
  

  Checking that no-one is using this disk right now ... BLKRRPART: Invalid argument OK

  Disk /dev/vg4/mail-root: 2614 cylinders, 255 heads, 63 sectors/track

  sfdisk: ERROR: sector 0 does not have an msdos signature /dev/vg4/mail-root: unrecognized partition table type Old situation: No partitions found Input in the following format; absent fields get a default value. <start> <size> <type [E,S,L,X,hex]> <bootable [-,*]> <c,h,s> <c,h,s>\nUsually you only need to specify <start> and <size> (and perhaps <type>).

  /dev/vg4/mail-root1 :1 /dev/vg4/mail-root1 1 2613 2613 20988922+ 83 Linux /dev/vg4/mail-root2 : /dev/vg4/mail-root2 0+ 0 1- 8032 83 Linux /dev/vg4/mail-root3 : /dev/vg4/mail-root3 0 - 0 0 0 Empty /dev/vg4/mail-root4 : /dev/vg4/mail-root4 0 - 0 0 0 Empty New situation: Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start End #cyls #blocks Id System /dev/vg4/mail-root1 1 2613 2613 20988922+ 83 Linux /dev/vg4/mail-root2 0+ 0 1- 8032 83 Linux /dev/vg4/mail-root3 0 - 0 0 0 Empty /dev/vg4/mail-root4 0 - 0 0 0 Empty Warning: no primary partition is marked bootable (active) This does not matter for LILO, but the DOS MBR will not boot this disk. Do you want to write this to disk? [ynq] y Successfully wrote the new partition table

  Re-reading the partition table ... BLKRRPART: Invalid argument

  If you created or changed a DOS partition, /dev/foo7, say, then use dd(1) to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1 (See fdisk(8).)

• Mount the new root filesystem

  [kyero@vm02 ~]$ sudo kpartx -av /dev/vg4/mail-root
  

  add map mail-root1 : 0 41977845 linear /dev/vg4/mail-root 16065 add map mail-root2 : 0 16064 linear /dev/vg4/mail-root 1 [kyero@vm02 ~]$ sudo e2fsck /dev/mapper/mail-root1 e2fsck 1.39 (29-May-2006) /dev/mapper/mail-root1: clean, 435213/1310720 files, 4086961/5242880 blocks

  [kyero@vm02 ~]$ sudo mount /dev/mapper/mail-root1 /mnt/tmp

• Add the new kernel's modules, tweak filesystem mounts, disable swap, check console.

  [kyero@vm02 ~]$ sudo rsync -a /lib/modules/ /mnt/tmp/lib/modules/
  

  [kyero@vm02 ~]$ sudo vi /mnt/tmp/etc/fstab

  [kyero@vm02 ~]$ sudo vi /mnt/tmp/etc/inittab

• Create the KVM guest, assigning the boot, root and swap storage

  [kyero@vm02 ~]$ sudo virt-install -n mail -r 1024 --vcpus 1 --os-type linux --os-variant generic26 --accelerate --disk path=/var/boot-images/mail-boot --disk path=/dev/mapper/vg4-mail--root,bus=ide,format=raw --disk path=/dev/mapper/vg4-mail--swap,bus=ide,format=raw --network bridge:br0 --network bridge:br1 --import --noreboot
  

  Starting install... Guest installation complete... you can restart your domain by running 'virsh start mail'

  [kyero@vm02 ~]$ sudo virsh dumpxml mail ... <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/var/boot-images/mail-boot'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/mapper/vg4-mail--root'/> <target dev='hdb' bus='ide'/> <address type='drive' controller='0' bus='0' unit='1'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/mapper/vg4-mail--swap'/> <target dev='hdd' bus='ide'/> <address type='drive' controller='0' bus='1' unit='1'/> </disk> ...

• Boot the KVM guest

  [kyero@vm02 ~]$ sudo virsh start mail
  

  [kyero@vm02 ~]$ sudo virsh autostart mail [kyero@vm02 ~]$ sudo virsh console mail

• Log into KVM guest, partition /dev/hdd and configure it as swap, update /etc/fstab.