WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] slow live magration / xc_restore on xen4 pvops

To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] slow live magration / xc_restore on xen4 pvops
From: Andreas Olsowski <andreas.olsowski@xxxxxxxxxxxxxxx>
Date: Thu, 03 Jun 2010 00:59:58 +0200
Delivery-date: Wed, 02 Jun 2010 16:01:16 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4C0578EB.2040800@xxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <2FD61F37AFF16D4DB46149330E4273C702FF9687@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <4C0578EB.2040800@xxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
I did some further research now and shut down all virtual machines on xenturio1, after that i got (3 runs): (xm save takes ~5 seconds , user and sys are always negligible so i removed those to reduce text)

xenturio1:~# time xm restore /var/saverestore-x1.mem
real    0m25.349s 0m27.456s 0m27.208s

So the fact that there were running machines did impact performance of xc_restore.

I proceeded to start create 20 "dummy" vms with 1gig ram and 4vpcus each (dom0 has 4096M fixed, 24gig total available): xenturio1:~# for i in {1..20} ; do echo creating dummy$i ; xt vm create dummy$i -vlan 27 -mem 1024 -cpus 4 ; done
creating dummy1
vm/create> successfully created vm 'dummy1'
....
creating dummy20
vm/create> successfully created vm 'dummy20'

and started them
for i in {1..20} ; do echo starting dummy$i ; xm start dummy$i ; done

So my memory allocation should now be 100% (4gig dom0 20gig domUs), but why did i have 512megs to spare for "saverestore-x1"? Oh well, onwards.

Once again i ran a save/restore, 3 times to be sure (edited the additional results in output).

With 20 running vms:
xenturio1:~# time xm restore /var/saverestore-x1.mem
real    1m16.375s 0m31.306s 1m10.214s

With 16 running vms:
xenturio1:~# time xm restore /var/saverestore-x1.mem
real    1m49.741s 1m38.696s 0m55.615s

With 12 running vms:
xenturio1:~# time xm restore /var/saverestore-x1.mem
real    1m3.101s 2m4.254s 1m27.193s

With 8 running vms:
xenturio1:~# time xm restore /var/saverestore-x1.mem
real    0m36.867s 0m43.513s 0m33.199s

With 4 running vms:
xenturio1:~# time xm restore /var/saverestore-x1.mem
real    0m40.454s 0m44.929s 1m7.215s

Keep in mind, those dumUs dont do anything at all, they just idle.
What is going on there the results seem completely random, running more domUs can be faster than running less? How is that even possible?

So i deleted the dummyXs and started the productive domUs again, in 3 steps to take further measurements:


after first batch:
xenturio1:~# time xm restore /var/saverestore-x1.mem
real    0m23.968s 1m22.133s 1m24.420s

after second batch:
xenturio1:~# time xm restore /var/saverestore-x1.mem
real    1m54.310s 1m11.340s 1m47.643s

after third batch:
xenturio1:~# time xm restore /var/saverestore-x1.mem
real 1m52.065s 1m34.517s 2m8.644s 1m25.473s 1m35.943s 1m45.074s 1m48.407s 1m18.277s 1m18.931s 1m27.458s

So my current guess is, xc_restore speed depends on the amount of used memory or rather how much is beeing grabbed by running processes. Does that make any sense?

But if that is so, explain:
I started 3 vms running "stress" that do:
load average: 30.94, 30.04, 21.00
Mem:   5909844k total,  4020480k used,  1889364k free,      288k buffers

But still:
tarballerina:~# time xm restore /var/saverestore-t.mem
real    0m38.654s

Why doesnt xc_restore slow down on tarballerina, no matter what i do?
Again: all 3 machines have 24gigs ram, 2x quad xeons and dom0 is fixed to 4096M ram.
all use the same xen4 sources, the same kernels with the same configs.

Is the Xeon E5520 with DDR3 really this much faster than the L5335 and L5410 with DDR2?

If someone were to tell me, that this is expected behaviour i wouldnt like it, but at least i could accept it. Are machines doing plenty of cpu and memory utilizaton not a good measurement in this or any case?

I think tomorrow night i will migrate all machines from xenturio1 to tarballerina, but first i have to verify that all vlans are available, that i cannot do right now.

---

Andreas

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel