This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-users] Massive iowait with Xen 3.2

To: "Fajar A. Nugraha" <fajar@xxxxxxxxx>
Subject: Re: [Xen-users] Massive iowait with Xen 3.2
From: Antoine Benkemoun <antoine.benkemoun@xxxxxxxxx>
Date: Tue, 28 Apr 2009 14:59:31 +0200
Cc: Xen List <xen-users@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Tue, 28 Apr 2009 06:00:53 -0700
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=sDCV7ecF0uOIstpqBqxP88pi1VqBakpuXoPFCm/HwbU=; b=czuZdhZT31TxHM1g5H0IZQ8R1dBdntd1oCzLapIi3frnLjTMv/Fwf6PMXfocaU1Yih ihsaZRQTqnxahpHUATAMwrd/yzNhfA3NM4GHqkmTijWctLJEfcdAHVX+u8ExGu6JnwKf fuldNyWfu4BMWsGEdw2khHcBqE49ABHtBs6+E=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Fix7GNFdI/x8akgRkRcFWxYGhVpUw0cAc1UaBH0ZebxOoK/4iWxEXttHWPTR67pjnl gP0Ec4p94WZlkAvF14yqVsiY6LRax4CM9w5gEQbY/GHJaqbrGRoLLSEIrkRP0EtyqI4Y y3kDsCG/fojyDfQMyhjWaFQWCaHQYuTp0W3pk=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <7207d96f0904280543s6424562dq65174c12c8b7f06e@xxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <fc1cbedb0904280039me501d77vf3e6d5318e14d927@xxxxxxxxxxxxxx> <7207d96f0904280154v4cad4bcal9cf9614a6b516472@xxxxxxxxxxxxxx> <fc1cbedb0904280451v6ba09dfeh5344235df962c0e4@xxxxxxxxxxxxxx> <7207d96f0904280543s6424562dq65174c12c8b7f06e@xxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Thank you for your answer. Sorry for the headers !

4MB/s read speed is just plain ridiculous for a hard drive or is it just me ? Smartctl gives me the following output :

Device Model:     Hitachi HDP725025GLA380
Serial Number:    GEK234RS1DT3RA
Firmware Version: GM2OA5CA
User Capacity:    250 059 350 016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Apr 28 14:57:48 2009 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

It gives me the following google result : http://www.newegg.com/Product/Product.aspx?Item=N82E16822145211 . Smells like a decent hard drive to me ... I'll just go and whip my server provider...

These domains are actually stored on LVM so that's already in place but iostat doesn't seem to differenciate...

Thank you for your help,


On Tue, Apr 28, 2009 at 2:43 PM, Fajar A. Nugraha <fajar@xxxxxxxxx> wrote:
On Tue, Apr 28, 2009 at 6:51 PM, Antoine Benkemoun
> The dom0 is a pretty minimal install of Debian with nothing else then Xen,
> SSH and fail2ban running. It can easily fit on 256MB of RAM. In fact, it
> averagely uses 100MB of RAM.

Hmm, interesting. I should test Debian or Ubuntu for firewall domUs sometimes.

> I have reproduced the problem by generating IOs with 3 torrents and 1 file
> copy.
> XM top shows :
>     commun ------       2959    3.5     262144   12.7     262144
> 12.7     1    1  1828352  1567395    3    16267  3411443  1848929   32
>   Domain-0 -----r       2617    2.7     262240   12.7   no limit
> n/a     1    0        0        0    0        0        0        0   32

Since you ommit the headers, the output is quite confusing :P
What are vbd stats for the domU, does it have lots of reads/writes?

> Load average for "commun", the domain being loaded, is above 5. Memory is
> only used at about 50% and only 440k of swap is used. Along goes the usual
> symptoms, really slow web page loads and really slow reactivity of about
> everything in the domain. Reactivity in the other domains is rather normal.

So it's not swap problem. However looking at your previous iostat
output %util is over 99 (which means your disk is busy), r/s over 180,
and rsec/s almost 8000 (meaning about 4 MB/s). Based on this numbers
I'm GUESSING you only have one lowend disk (not raid), and it has
reached its IOPS caps. In other words, get more disks :P

A suggestion, it would probably be easier to manage if you use LVM and
put dom0 and domU each on its own LV. That way iostat can show per-LV
staticstic so you can easily determine who's using most disk I/O.



Xen-users mailing list
<Prev in Thread] Current Thread [Next in Thread>