WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] dom0 hangs when doing heavy I/O on domU

To: Florian Heigl <florian.heigl@xxxxxxxxx>
Subject: Re: [Xen-users] dom0 hangs when doing heavy I/O on domU
From: Kiefer Chang <zapchang@xxxxxxxxx>
Date: Thu, 21 Apr 2011 22:56:11 +0800
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Delivery-date: Thu, 21 Apr 2011 07:57:53 -0700
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=zrtjTyrLTfVLLpXXDvmCi1+GxtLBC4Z0Blz/CC4/Yyc=; b=Lrlj2TKe6w/C/bckQik6xralT1yxsH+YMiq2vOLkEWXX0iGUmrv8uOvQUluJZgOGDE y0HFGTiYtm5D7b1cYkN39fPwjamWdSj++Ae4VMF2GmbIIG7icsL3DsNGZYpkyFdu/34I I/O18fr2wZA2Dy/kgpKgprJu3wINod79AFXv8=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=tbDZSl0WyboMo5qlFFN3ql50xYyZ22+4/RhbK4+7I/C1RMXM2q9emkmqACrwRY5BMI c6k5dnU1rWYV4Nvo2+enrCIyt+FOz67acEnsRB7r5iyEgHjXMNInCp5+4UlL0MFMgqUa SneViMS9nqdqTroh1iCIwI9k0BaULQUnB5zCA=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <BANLkTinrKXZFHtGi2uEgY0bD1JRyRxNPnQ@xxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <BANLkTin3aOg2CYcvDgtwwHdrhwExr80zaw@xxxxxxxxxxxxxx> <BANLkTinrKXZFHtGi2uEgY0bD1JRyRxNPnQ@xxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Hi Florian,

- are they sharing a disk?
  Yes, 3 VMs' images are stored in the same disk sdb. sda is used for dom0 root filesystem.
- can you check what I/O scheduler you are using?
  Default to CFQ in for sdb. I tried to ionice all blkback processes to class 2 and still have no luck.
  I also tried deadline scheduler before.

Right now I found a cure is to make sdb as physical volume and setup volume groups/logical volumes on it.
Attach logical volumes to VMs by "phy" method.
The symptom is gone when 3 VMs perform the same I/Os.
I know XEN manual suggest using blktap and phy method for VM storages.
But we think it's much easier to manage VM's image files then LVM volumes since we provision VMs by downloading their images from servers.

Thanks!

--
Kiefer Chang




2011/4/21 Florian Heigl <florian.heigl@xxxxxxxxx>
Hi Chang,

2011/4/20 Kiefer Chang <zapchang@xxxxxxxxx>:
> Hi all,
> We are using XEN as hypervisor to setup our private cloud.
> The framework is Eucalyptus and using CentOS 5.4 as dom0 OS.
> Sometimes we find some machines' dom0 become unresponsive, the symptoms are:
> (1) We can't log into dom0 via ssh. After typing password, it just stops
> there.
> (2) We can ping dom0 successfully.
> (3) We can log into domU without problem.
> The unresponsive dom0 eventually "alive" after a period of time. Maybe half
> hour or even several hours.

So one of your domUs is trashing the disks and dom0 can't get enough
performance, right?
- are they sharing a disk?
- can you check what I/O scheduler you are using?
 (with cfq you can then use ionice to lower prio on all blkback
threads a little. that way dom0 will "win the race")

In general, your dom0 is privileged in terms of IO access rights, but
not in IO peformance. So if one domU goes crazy, it will affect
anything.
... until you take measures :)
I'd suggest you switch to deadline scheduler and re-test.
dom0 on a different disk media is also very advisable imho.


Flo

--
the purpose of libvirt is to provide an abstraction layer hiding all
xen features added since 2006 until they were finally understood and
copied by the kvm devs.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
<Prev in Thread] Current Thread [Next in Thread>