HP LeftHand VSA (Virtual SAN Applicance), there after called VSA.

VSA is distributed by HP, as a blackbox that could be run under VmWare ESX(i) or Hyper-V. Before Lefthand was bought by HP, they were planning to get it to work under Xen, but since HP bought Lefthand, it doesn't seem much work has been done toward it.

Having some Lefthand, and wanting to reuse our old systems with disks (4-10TB), we tried to use ESXi. Having bad performances, and noway to monitor underlying hardware, we tried to get it to work with Xen.

This documentation present how to get a VSA system “working” a xen VM using paravirt. Doing so still leaves some issues unresoleved (like upgrading), and as it is not officially supported, don't expect to get any support if you have anything that goes wrong on your Xen VSA.

To retrieve a VSA image, and start working with it, I installed VSA on an ESXi server, provided the disk to an other linux, and duplicated the image to a remote server (dd, ssh and gzip were very friendly to do such).

In my environment, the target was directly a disk (in fact, it was on an iscsi device), which made it easy to access within partitions. Here after, the VSA image will be called /dev/sda (and partitions in /dev/sda1 to /dev/sda9) for access from the dom0.

/dev/sda1 is the partition used as /boot. It contains a kernel and a initrd. As those aren't xen-aware enough.

To boot the system, I will be using a debian kernel and initrd. (That's just because my dom0 are debians, and have nice enough kernels).

The system relies heavily on a few things that make it very xen-unfriendly.

• dmidecode to detect motherboard UUID. dmidecode doesn't return anything under xen. That part is only called in a single script.
• /proc/bus/pci/devices to detect what kind of hardware it is running on. Used in hwid, and a few other libraries.
• disk path (like /dev/scsi/host0/bus0/target0/lun0) for system and data disks.
• a few scripts are architecture specifics
• have the required modules
• make sure we can log in to check what's wrong

We will be using hacks for each of those points.

File: /etc/init.d/lhn_functions
Function: didMotherboardChange

Patch the function so it detect a new mother board only if required. I have no idea what it does with it, but I guess we can safely guess it matches some hardware changes, and from our Xen point of view, hardware only changes if we clone the VM and give it an other network card.

Add:

moboUUID=`dmidecode ...`
[ "$moboUUID" ] || {
  mac=`/sbin/ifconfig eth0 | sed -n '/HW/{s/ *$//;s/.* //;s/://g;p;}'`
  [ "$mac" ] || mac="deadb33fc0de"
  # need to return the same value over reboot
  # first part was generated with uuidgen
  moboUUID="160d007e-0f9e-11e1-b59f-$mac"
}

Fortunately, the configuration files that match a pci device ID with an architecture ID already have a Xen entry. So we only need to provide a fake devices list.

Content of that directory should be (\t for tabs, of course) :

0000\t58530001\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0

There are two ways to hack that part. Either binary hack every binary refering to /proc/bus/pci/devices to access something else like /hack/bus/pci/devices. That was my first PoC, and it worked well.

sed -i 's,/proc/bus/pci/devices,/hack/bus/pci/devices,'

Files that match include :

sda5:/etc/lefthand/brand/common/fst
sda5:/etc/lefthand/system/lib/hal/libhal.so
sda5:/etc/lefthand/system/liblhnsys.so
sda5:/sbin/hwid

The second way is to mount-bind an other directory on top of /proc/bus/pci.

The first way need to find every files that check pci/devices. The second need to patch the initrd to mount it before letting the system boot.

initrd:/scripts/init-bottom/pcihack

#! /bin/sh
[ -d "/root/hack/bus/pci" ] &&
mount -o bind /root/hack/bus/pci /proc/bus/pci

VmWare image wants to have SCSI host0 disk for boot, and SCSI host1 disks for data. As we will be using /dev/xvd?, we won't have the same host differences. I decided to use xvd[ab] for host0, and xvd[c-g] as host1. /dev/scsi is managed by udev. Lets add some udev rules to present xvd* in /dev/scsi too !

sda5:/etc/udev/rules.d/53-devfs-hack.rules

KERNEL=="xvd*", PROGRAM="/etc/udev/scripts/xvd-hack.sh %k", SYMLINK+="scsi/%c", MODE="0600"

sda5:/etc/udev/scripts/xvd-hack.sh

#! /bin/sh

DEV="$1"
case "$DEV" in
        xvd[abcdefg]|xvd[abcdefg][0-9]) : ;;
        *) exit 1 ;;
esac

case "$DEV" in
        xvd[ab]*) host=0 ;;
        *) host=1 ;;
esac
case "$DEV" in
        xvd[ac]*) target=0 ;;
        xvdd*) target=1 ;;
        xvd[be]*) target=2 ;;
        xvdf*) target=3 ;;
        xvdg*) target=4 ;;
esac
ext="${DEV#xvd?}"
if [ "$ext" ] ; then
        ext=part"$ext"
        ext="disc"
fi
echo host$host/bus0/target$target/lun0/$ext

Lefthand already has some mechanism to detect hardware brand, and do something about it. Xen is along with VmWare and Hyper-V in VMNSM “brand”.

Devices. As we have the /dev/scsi hack, we just get along the else case (have the same devices as vmware).

In function base_init, create link for platform.xml{.vmw,} as we use the same disk path.

check platform, and load required drivers :

xen_blkfront, xen_netfront, ext3, rd rd_size=32768, md, raid0, raid1, ext3

Duplicate /lib/modules/`uname -r` from dom0 to sda5.

As we boot on pv, we don't have tty1-tty6. Comment out ^[1-6]: et ^s0:

Include :

7:2345:respawn:/sbin/agetty -l /bin/su hvc0 38400 linux

Yeah, I use su just to make sure we don't bother with auth.

Stuffs to check in the vm config :

• root = '/dev/xvda5'
• disks: xvda for boot disk, xvdc (up to xvdg) for data disks
• do specify a mac address (used as serialid for vsa)

Once booted, log in as root, and launch

 /sbin/lhn_login -- start

That should let you manage the hostname, password, and IP.

WARNING: does not keep configuration after a reboot. No further research done on it, as LeftHand is way too bugy anyway.

From what I read, the system seems to install itself on an other partition, and tries to boot on it. The patch preventing the system to modify the used initrd, can't change anything like that, and the new system would probably need some patch applied first anyway.

The network card doesn't show properly in the Console.

You should be able to find a script vsa-xen-hack.sh. This script automate all of the “Hack the system” parts.

Requirement:

• dom0 with xen
• already available image of a VSA/vmware

Usage:
        ./vsa-xen-hack.sh --initrd <debian-initrd> <destination-hacked-initrd>
        ./vsa-xen-hack.sh --root /path/to/vsa/root/mountpoint

Exemple:
        ./vsa-xen-hack.sh --initrd /boot/initrd.img-2.6.32-5-xen-amd64 \
                /boot/initrd.img-2.6.32-5-xen-amd64-vsa

        mount /dev/sda5 /mnt &&
        ./vsa-xen-hack.sh --root /mnt

vsa-xen-hack.txt