|
xen-devel
RE: [Xen-devel] VM start time slows with large number of VMs
At 17:27 11/09/2007, Carb, Brian A wrote:
I tried timing xenstore-ls and then tried it again with the shared
memory idea (adding the 2 lines to the /etc/init.d/rc3.0/S10xend
script before it starts xend). Here are the average times for xenstore-ls:
# vms std shm
001-020 .67 .59
021-040 1.70 1.54
041-060 2.41 2.36
061-080 3.26 3.14
081-100 4.06 3.95
101-120 4.96 4.90
121-140 6.56 6,02
141-160 7.33 7.21
161-180 8.00 8.62
181-200 9.90 9.61
Without using shared memory, the total elapsed time to start 200 VMs
is about 95 minutes, vs 81 minutes with the shm statements. Time to
start the 200th vm, from create to unpaused, is about 55 seconds,
and with xenstore in shared memory that time is about 43 seconds.
I'm not seeing the 200+ seconds I was seeing before, but that time
was for vm263 and I've only been able to test with 200 VMs. This
week I'll try to test with more.
Since the shared and non-shared memory results are so close, how can
I verify that the xenstore data is actually going in and out of
shared memory? It seems as if the start times became reduced once I
cleared out the old info from xenstore, regardless of shared memory.
That would make sense.
I'm not sure how well the xenstore structure is designed, with
respect to holding hundred and thousands of different entries - I had
a quick look at it, but not really looking at it deeply, just very
briefly - the code I was looking at seemed to do a lot of small
function calls to dig data out of the database a byte or some few
bytes at a time, which didn't make me say "Wow, that's clever".
--
Mats
Is it worth trying bootloader/bootentry instead of kernel/ramdisk as well?
brian carb
unisys corporation - malvern, pa
-----Original Message-----
From: Daniel P. Berrange
Sent: Wednesday, September 05, 2007 1:38 PM
Subject: Re: [Xen-devel] VM start time slows with large number of VMs
On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser wrote:
> xenstore slowness seems a reasonable hypothesis. How does performance
> of ?time xenstore-ls >/dev/null¹ change as you increase the
number of guests?
There's another issue with the way hotplug scripts do locking which
exhbits fairly pathelogical behaviour when a large number of guests
are started concurrently. We've got a patch we're evaluating which
changes the use of lock files + looping + sleeps, to instead use
the 'flock' shell util.
I suspect the performance issues will be a combination of both the
locking stuff and the xenstore I/O. You might also want to try
putting /var/lib/xenstored onto tmpfs eg in initscripts before
xenstored starts ...
mkdir /dev/shm/xenstored
mount --bind /dev/shm/xenstored /var/lib/xenstored
Dan.
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
|
|
Home