Re: [Xen-users] Poor disk io performance in domUs

[Tom Brown <xensource.com@xxxxxxxxxxxxxxxxxxx>]

> On Jun 20, 2007, at 10:49 PM, Andrej Radonic wrote:
>
> > Hi,
> >
> > in a previously well behaving xen setup we are suddenly experiencing 
> > massive loss of disk performance.
> > Setup: two identical machines (dell, 2 quadcore cpus each, 8 GB RAM each), 
> > attached to *one* SAS storagebox (Raid5 SAS disks), CentOS 5, Xen 3.04 
> > running three domUs each.
> > Storage partitions are exposed as block-devices to the guests. Filesystem 
> > is ext3.
raid5 ... so you've got the mailserver running on multiple hard 
drives?
> > Since a domU running a rather heavily loaded mailserver (many small files, 
> > many reads/writes) has been added, io performance has deteriorated 
> > significantly.
> > In dom0 about 100 MB/s can be achieved, in domU only about 50 MB/s. This 
> > looks even worse when there are massive concurrent / parallel io jobs, then 
> > throughput goes down to 20 MB/s.
That is to be expected. If the drives are busy seeking all over the place, 
you are _not_ going to get high speed transfers off of them. I don't see 
anything xen specific about this. Although it is more likely to show up 
when you put multiple tasks on a single set of hardware (and Xen 
encourages that).
> > During all this the cpus are mostly idle...
> >
> > Any ideas? I know some performance degradation is to be expected but 
> > shurely not on this scale?
Try it on a single drive/cpu machine... fire up something like bonnie to 
get it to do multiple parallel I/Os while timing a big dd run or something 
in the background. Maxing out your disk drives is the easiest way to drive 
performance into the ground :(.
You can only get 50-100 MB/s transfers off a drive if the data is being 
read sequentially off the drive with an absolute minimum of head seeks. As 
soon as something else starts moving the drive head(s) around, your 
sustained transfer rate is going to fall dramatically... it comes down to 
how many seeks per second your drives can do.
Building up a big RAID array improves the system's ability to provide high 
throughput but is still limited by the abilities of the drives it is built 
from....
Mail servers, like news servers, "eat" drives.... If space allows it, you 
may be better off buying more smaller drives than fewer larger ones. With 
more drives you have more spindles and the ability to do more seeks per 
second (and higher combined throughput too).
I try to isolate the drives for my domUs... often dedicating one spindle 
to one domU, and using drbd for raid1. It would be nicer if this wasn't 
needed (since it is the opposite of virtualization), but AFAIK the one 
resource that is hard to schedule/limit/allocate/prioritize is disk 
access/bandwidth.
-Tom

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users