This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


RE: [Xen-devel] VM start time slows with large number of VMs

To: "Daniel P. Berrange" <berrange@xxxxxxxxxx>, "Keir Fraser" <Keir.Fraser@xxxxxxxxxxxx>
Subject: RE: [Xen-devel] VM start time slows with large number of VMs
From: "Carb, Brian A" <Brian.Carb@xxxxxxxxxx>
Date: Tue, 11 Sep 2007 11:27:13 -0500
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, "Carb, Brian A" <Brian.Carb@xxxxxxxxxx>
Delivery-date: Tue, 11 Sep 2007 09:28:05 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <20070905173732.GH5503@xxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <089B0D75973E1241B941D0A9854F23FC08E17469@xxxxxxxxxxxxxxxxxxxxxxxxxxxx> <C304A1BA.D3AD%Keir.Fraser@xxxxxxxxxxxx> <20070905173732.GH5503@xxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: Acfv431v4eUPM9wSTuCoXN4p9SskEgEqUdCg
Thread-topic: [Xen-devel] VM start time slows with large number of VMs
 I tried timing xenstore-ls and then tried it again with the shared memory idea 
(adding the 2 lines to the /etc/init.d/rc3.0/S10xend script before it starts 
xend). Here are the average times for xenstore-ls:

 # vms     std    shm
001-020    .67    .59
021-040   1.70   1.54
041-060   2.41   2.36
061-080   3.26   3.14
081-100   4.06   3.95
101-120   4.96   4.90
121-140   6.56   6,02
141-160   7.33   7.21
161-180   8.00   8.62
181-200   9.90   9.61

Without using shared memory, the total elapsed time to start 200 VMs is about 
95 minutes, vs 81 minutes with the shm statements. Time to start the 200th vm, 
from create to unpaused, is about 55 seconds, and with xenstore in shared 
memory that time is about 43 seconds. I'm not seeing the 200+ seconds I was 
seeing before, but that time was for vm263 and I've only been able to test with 
200 VMs. This week I'll try to test with more.

Since the shared and non-shared memory results are so close, how can I verify 
that the xenstore data is actually going in and out of shared memory? It seems 
as if the start times became reduced once I cleared out the old info from 
xenstore, regardless of shared memory.

Is it worth trying bootloader/bootentry instead of kernel/ramdisk as well?

brian carb
unisys corporation - malvern, pa

-----Original Message-----
From: Daniel P. Berrange
Sent: Wednesday, September 05, 2007 1:38 PM
Subject: Re: [Xen-devel] VM start time slows with large number of VMs

On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser wrote:
> xenstore slowness seems a reasonable hypothesis. How does performance 
> of Œtime xenstore-ls >/dev/null¹ change as you increase the number of guests?

There's another issue with the way hotplug scripts do locking which exhbits 
fairly pathelogical behaviour when a large number of guests are started 
concurrently. We've got a patch we're evaluating which changes  the use of lock 
files + looping + sleeps, to instead use the 'flock' shell util.

I suspect the performance issues will be a combination of both the locking 
stuff and the xenstore I/O. You might also want to try putting 
/var/lib/xenstored onto tmpfs eg in  initscripts before xenstored starts ...

  mkdir /dev/shm/xenstored
  mount --bind /dev/shm/xenstored /var/lib/xenstored


Xen-devel mailing list