ze's sandcastle

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

WARNING: does not keep configuration after a reboot. No further research done on it, as LeftHand is way too bugy anyway.

====== HP Lefthand VSA under Xen ======

[[http://h18006.www1.hp.com/products/storage/software/vsa/index.html|HP LeftHand VSA]] (Virtual SAN Applicance), there after called VSA.

VSA is distributed by HP, as a blackbox that could be run under VmWare
ESX(i) or Hyper-V. Before Lefthand was bought by HP, they were planning
to get it to work under Xen, but since HP bought Lefthand, it doesn't
seem much work has been done toward it.

Having some Lefthand, and wanting to reuse our old systems with disks
(4-10TB), we tried to use ESXi. Having bad performances, and noway to
monitor underlying hardware, we tried to get it to work with Xen.

This documentation present how to get a VSA system "working" a xen VM
using paravirt. Doing so still leaves some issues unresoleved (like
upgrading), and as it is not officially supported, don't expect to get
any support if you have anything that goes wrong on your Xen VSA.

====== Retrievinig a VSA image ======

To retrieve a VSA image, and start working with it, I installed VSA on
an ESXi server, provided the disk to an other linux, and duplicated the
image to a remote server (''dd'', ''ssh'' and ''gzip'' were very friendly to do
such).

In my environment, the target was directly a disk (in fact, it was on an
iscsi device), which made it easy to access within partitions. Here
after, the VSA image will be called ''/dev/sda'' (and partitions in
''/dev/sda1'' to ''/dev/sda9'') for access from the dom0.

''/dev/sda1'' is the partition used as ''/boot''. It contains a kernel and a
initrd. As those aren't xen-aware enough.

To boot the system, I will be using a debian kernel and initrd. (That's
just because my dom0 are debians, and have nice enough kernels).

====== Hack the system ======

The system relies heavily on a few things that make it very
xen-unfriendly.

  * dmidecode to detect motherboard UUID. dmidecode doesn't return anything under xen. That part is only called in a single script.
  * ''/proc/bus/pci/devices'' to detect what kind of hardware it is running on. Used in hwid, and a few other libraries.
  * disk path (like ''/dev/scsi/host0/bus0/target0/lun0'') for system and data disks.
  * a few scripts are architecture specifics
  * have the required modules
  * make sure we can log in to check what's wrong

We will be using hacks for each of those points.

===== mother board UUID =====

File: /etc/init.d/lhn_functions\\
Function: didMotherboardChange

Patch the function so it detect a new mother board only if required. I
have no idea what it does with it, but I guess we can safely guess it
matches some hardware changes, and from our Xen point of view, hardware
only changes if we clone the VM and give it an other network card.

Add:
<code>
moboUUID=`dmidecode ...`
[ "$moboUUID" ] || {
  mac=`/sbin/ifconfig eth0 | sed -n '/HW/{s/ *$//;s/.* //;s/://g;p;}'`
  [ "$mac" ] || mac="deadb33fc0de"
  # need to return the same value over reboot
  # first part was generated with uuidgen
  moboUUID="160d007e-0f9e-11e1-b59f-$mac"
}
</code>

===== /proc/bus/pci/devices =====

Fortunately, the configuration files that match a pci device ID with an
architecture ID already have a Xen entry. So we only need to provide a
fake devices list.

Content of that directory should be (\t for tabs, of course) :
  0000\t58530001\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0\t0

There are two ways to hack that part. Either binary hack every binary
refering to ''/proc/bus/pci/devices'' to access something else like
''/hack/bus/pci/devices''. That was my first PoC, and it worked well.

  sed -i 's,/proc/bus/pci/devices,/hack/bus/pci/devices,'

Files that match include :
  sda5:/etc/lefthand/brand/common/fst
  sda5:/etc/lefthand/system/lib/hal/libhal.so
  sda5:/etc/lefthand/system/liblhnsys.so
  sda5:/sbin/hwid

The second way is to mount-bind an other directory on top of
''/proc/bus/pci''.

The first way need to find every files that check pci/devices. The
second need to patch the initrd to mount it before letting the system
boot.

''initrd:/scripts/init-bottom/pcihack''
<code>
#! /bin/sh
[ -d "/root/hack/bus/pci" ] &&
mount -o bind /root/hack/bus/pci /proc/bus/pci
</code>

===== /dev/scsi =====

VmWare image wants to have SCSI host0 disk for boot, and SCSI host1
disks for data. As we will be using ''/dev/xvd?'', we won't have the same
host differences. I decided to use ''xvd[ab]'' for host0, and ''xvd[c-g]'' as
host1. /dev/scsi is managed by udev. Lets add some udev rules to present
''xvd*'' in ''/dev/scsi'' too !

''sda5:/etc/udev/rules.d/53-devfs-hack.rules''
  KERNEL=="xvd*", PROGRAM="/etc/udev/scripts/xvd-hack.sh %k", SYMLINK+="scsi/%c", MODE="0600"

''sda5:/etc/udev/scripts/xvd-hack.sh''
<code>
#! /bin/sh

DEV="$1"
case "$DEV" in
        xvd[abcdefg]|xvd[abcdefg][0-9]) : ;;
        *) exit 1 ;;
esac

case "$DEV" in
        xvd[ab]*) host=0 ;;
        *) host=1 ;;
esac
case "$DEV" in
        xvd[ac]*) target=0 ;;
        xvdd*) target=1 ;;
        xvd[be]*) target=2 ;;
        xvdf*) target=3 ;;
        xvdg*) target=4 ;;
esac
ext="${DEV#xvd?}"
if [ "$ext" ] ; then
        ext=part"$ext"
        ext="disc"
fi
echo host$host/bus0/target$target/lun0/$ext
</code>

===== scripts missing xen architecture specific stuffs =====

Lefthand already has some mechanism to detect hardware brand, and do
something about it. Xen is along with VmWare and Hyper-V in VMNSM
"brand".

==== sda5:/etc/lefthand/brand/brand.init ====

Devices. As we have the /dev/scsi hack, we just get along the
else case (have the same devices as vmware).

In function base_init, create link for platform.xml{.vmw,} as we
use the same disk path.

==== sda5:/etc/lefthand/brand/rc.drivers ====

check platform, and load required drivers :
  xen_blkfront, xen_netfront, ext3, rd rd_size=32768, md, raid0, raid1, ext3

===== modules =====

Duplicate ''/lib/modules/`uname -r`'' from ''dom0'' to ''sda5''.

===== inittab =====

As we boot on pv, we don't have ''tty1-tty6''. Comment out ''^[1-6]:'' et ''^s0:''

Include :
  7:2345:respawn:/sbin/agetty -l /bin/su hvc0 38400 linux

Yeah, I use su just to make sure we don't bother with auth.

====== Configure and enjoy ======

Stuffs to check in the vm config :
  * root = '/dev/xvda5'
  * disks: xvda for boot disk, xvdc (up to xvdg) for data disks
  * do specify a mac address (used as serialid for vsa)

Once booted, log in as root, and launch 
   /sbin/lhn_login -- start

That should let you manage the hostname, password, and IP.


====== Known Issues ======

===== Do not use it =====

WARNING: does not keep configuration after a reboot. No further research done on it, as LeftHand is way too bugy anyway. 

===== Upgrading =====

From what I read, the system seems to install itself on an
other partition, and tries to boot on it. The patch preventing the
system to modify the used initrd, can't change anything like that, and
the new system would probably need some patch applied first anyway.

===== Network card =====

The network card doesn't show properly in the Console.

====== Misc ======

===== Script =====

You should be able to find a script {{vsa-xen-hack.txt|vsa-xen-hack.sh}}. This script automate all of the "Hack the system" parts.

Requirement:
  * dom0 with xen
  * already available image of a VSA/vmware

<code>
Usage:
        ./vsa-xen-hack.sh --initrd <debian-initrd> <destination-hacked-initrd>
        ./vsa-xen-hack.sh --root /path/to/vsa/root/mountpoint

Exemple:
        ./vsa-xen-hack.sh --initrd /boot/initrd.img-2.6.32-5-xen-amd64 \
                /boot/initrd.img-2.6.32-5-xen-amd64-vsa

        mount /dev/sda5 /mnt &&
        ./vsa-xen-hack.sh --root /mnt
</code>


{{:system:lefthand:vsa-xen-hack.txt|}}
ze's sandcastle

User Tools

Site Tools

Page Tools
