Fwd: [Xen-users] domU's overloaded

["Brian Stempin" <brian.stempin@xxxxxxxxx>]

whoops...forgot to copy the list.

---------- Forwarded message ----------
From: Brian Stempin <brian.stempin@xxxxxxxxx>
Date: Fri, Mar 28, 2008 at 12:56 AM
Subject: Re: [Xen-users] domU's overloaded
To: n8&abby <thoughtobject@xxxxxxxxx>

I would suggest something like having a (or a few) 10Gb cards in your server.

Our current plan is to shift from about 10 domU's per 7200rmp sata hard disk to 4 domU's...  Are you saying that buying lots more commodity disks per physical server (12 instead of 4 serving 50 domU's) may not be enough?  I do like the idea of separating out the storage over a local area network.  Should I be looking into GFS for this?

Since you're downloading torrents, your doing a lot of random writes/reads, which puts more of a strain on the disks than  a lot of synchronous reads/writes.  Having having more physical disks should help, but I have no way of knowing whether or not this will make the impact that you need it to.  It will also depend on things like your RAID controller, FSB speed, etc.

My suggestion was going in the direction of having one or many boxes that have one or many disk arrays attached.  You could then, over iSCSI, connect your DomUs to their disk space.  GFS would be useful if you have all of your DomUs writing to the same file system.  Otherwise, it's not needed.

On Thu, Mar 27, 2008 at 5:36 PM, n8&abby <thoughtobject@xxxxxxxxx> wrote:

Each domU has 50Gb space.  SCSI / SAS drives don't work for us because of the cost.  These are torrent servers, each domU constantly uploading and downloading many random parts of large files.  Each domU averages about about 1Mbps transfer speed.  Currently hovering around 30 - 50Mbps transfer speed on 40 domU's.

Each domU does not have to be super high performance, just usable. 

I am interested in what other organizations are doing to host large numbers of virtual private servers.  We are interested in using lots of inexpensive commodity hardware, rather than a high end hardware solution.

I would suggest something like having a (or a few) 10Gb cards in your server.

Our current plan is to shift from about 10 domU's per 7200rmp sata hard disk to 4 domU's...  Are you saying that buying lots more commodity disks per physical server (12 instead of 4 serving 50 domU's) may not be enough?  I do like the idea of separating out the storage over a local area network.  Should I be looking into GFS for this?

On Thu, Mar 27, 2008 at 10:46 AM, Brian Stempin <brian.stempin@xxxxxxxxx> wrote:

I would imagine lots and lots of SAN or iSCSI cards.

It sounds like you're in need of more storage bandwidth than a local RAID or disk controller can provide.  I would suggest something like having a (or a few) 10Gb cards in your server.  You can then have the Dom0 mount a series of iSCSI (AoE, whatever storage tech you want) targets and, and then pass them to the DomUs as block devices.

It may also be helpful to the list to have a general idea of what the DomUs are doing in order to help you identify other bottlenecks, etc.

On Tue, Mar 25, 2008 at 10:01 PM, n8&abby <thoughtobject@xxxxxxxxx> wrote:

>  If you want to put 100 general purpose servers on 1 piece of
>  hardware I suggest you look into a lot more than 4 disks, and
>  probably look at 10kRPM 2.5" SAS as well, as opposed to what I am
>  guessing are commodity 7200RPM 3.5" SATA disks.
>
>  Check the iowait % in your domains and dom0 - if it is more than a
>  few percent then it's IO you are needing i.e. more disks.  I always
>  run out of IO before CPU or RAM too, it's pretty common.

Thanks for all the great replies on this, very helpful.  Turns out the
iowait on some domUs are higher than 50%.  This makes sense since most
of what these domU's are doing is serving very large files.

For the next system I am building this week, I'm looking into multiple
disks on the physical server.  I will probably start with 16x 7200rpm
SATA.

However, we will be building many more of these and I am wondering
what other systems are in common use.  Is there a straightforward way
that to set this up in bulk?  Should I be looking into some type NFS
or NAS/SAN?  I don't have experience setting up anything but drives on
the local physical server, but I can learn whatever I need to.  What
is the Xen way for large storage needs with lots and lots of domU's?

Thanks,
-=nathan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users