[Xen-users] Fwd: Xen - large virtual machines

["Alex Davies" <alex@xxxxxxxx>]

Hi Everyone,

I'm looking for a solution to run some Xen VM's with fairly significant(up to ~30TB) local storage volumes. We plan to use two Sun "Thumper" x4540's with 48*1TB drives (8 controllers), and Red Hat Enterprise Linux 5 (x86_64).

I have a few questions, and would appreciate feedback before I try to achieve something dumb or impossible :)

• Has Xen been tested / does Xen support very large volumes? We will probably end up using ext3 (as it seems its the only supported option), so no single volume in a VM will exceed ~16TB, but we will be at this end of the spectrum on several volumes.
• Can anyone suggest a better option than creating 7 RAID5 arrays of 6 disks each (1 per controller); using the remaining 6 drives for RAID-10 for DomU and hot spares and then adding the 7 RAID5 volumes to a LVM group, creating a LV per Xen VM block device? Has anybody used mdadm and/or LVM to handle this number of drives?
• We are required to be able to bring up VMs on one machine on another within a reasonable period of time (this is defined as days, not hours) and to keep the backup copy reasonably in sync (again, days not hours). Can anyone comment on the suitability of rsync to copy such massive block devices? We would use version 3, which I know has improved performance for this sort of move, but is it going to literally take forever to checksum 48TB? If anyone is able to give example of rsync performance (same switch or crossover - whichever we conclude is fastest) over gigabit Ethernet, that would be a bonus. We expect there to be significant spare resources available to domU at night for checksumming if required.
• I know XFS is not supported by RHEL, but does it work - and/or well? I've seen several reports that suggest it performs far better (best summary http://hepix.caspur.it/storage/hep_pdf/2007/Spring/Petkus_HEPiX_Spring06.storageeval.pdf)  but what is it like for reliability?
• Does anyone has any suggestions / improvements to this plan? Has anybody done this before - with similar hardware?
• [An alternative scenario, if we can get it, would be a third Thumper - in which case we would run half the VMs on each of two Thumpers running RAID-10 (8 RAID-1 volumes, each a PV) for ~24TB per machine, and use the third Thumper, partitioned as above with RAID-5 for more like RAID-5 levels of storage efficiency, for backups of VMs and disaster recovery. Comments on this would be fantastic too.]

The VMs are expected to be quite IO-heavy, hence the requirement for local storage (not iSCSI) and para virtualization.

Any comments much appreciated.

Many thanks,

Alex

[this email previously posted to fedora-xen@xxxxxxxxxx; posted here after a suggestion there :(]

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users