Hello,
I would be grateful for the comments on possible methods to improve domain
restore performance. Focusing on the PV case, if it matters.
1) xen-4.0.0
I see a similar problem to the one reported at the thread at
http://lists.xensource.com/archives/html/xen-devel/2010-05/msg00677.html
Dom0 is 2.6.32.9-7.pvops0 x86_64, xen-4.0.0 x86_64.
[user@qubes ~]$ xm create /dev/null
kernel=/boot/vmlinuz-2.6.32.9-7.pvops0.qubes.x86_64
root=/dev/mapper/dmroot extra="rootdelay=1000" memory=400
...wait a second...
[user@qubes ~]$ xm save null nullsave
[user@qubes ~]$ time cat nullsave >/dev/null
...
[user@qubes ~]$ time cat nullsave >/dev/null
...
[user@qubes ~]$ time cat nullsave >/dev/null
real 0m0.173s
user 0m0.010s
sys 0m0.164s
/* sits nicely in the cache, let's restore... */
[user@qubes ~]$ time xm restore nullsave
real 0m9.189s
user 0m0.151s
sys 0m0.039s
According to systemtap, xc_restore uses 3812s of CPU time; besides it being
a lot, what uses the remaining 6s ? Just as reported previously, there are
some errors in xend.log
[2010-05-25 10:49:02 2392] DEBUG (XendCheckpoint:286) restore:shadow=0x0,
_static_max=0x19000000, _static_min=0x0,
[2010-05-25 10:49:02 2392] DEBUG (XendCheckpoint:305) [xc_restore]:
/usr/lib64/xen/bin/xc_restore 39 3 1 2 0 0 0 0
[2010-05-25 10:49:02 2392] INFO (XendCheckpoint:423) xc_domain_restore
start: p2m_size = 19000
[2010-05-25 10:49:02 2392] INFO (XendCheckpoint:423) Reloading memory pages:
0%
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) ERROR Internal error:
Error when reading batch size
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) ERROR Internal error:
error when buffering batch, finishing
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423)
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:4100%
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) Memory reloaded (0
pages)
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) read VCPU 0
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) Completed checkpoint
load
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) Domain ready to be
built.
[2010-05-25 10:49:11 2392] INFO (XendCheckpoint:423) Restore exit with rc=0
Note, xc_restore on xen-3.4.3 works much faster (and with no warnings in the
log), with the same dom0 pvops kernel.
Ok, so there is some issue here. Some more generic thoughts below.
2) xen-3.4.3
Firstly, /etc/xen/scripts/block in xen-3.4.3 tries to do something like
for i in /dev/loop* ; do
losetup $i
so, spawn one losetup process per each existing /dev/loopX; it hogs CPU,
especially if your system comes with maxloops=255 :). So,
let's replace it with the xen-4.0.0 version, where this problem is fixed (it
uses losetup -a, hurray).
Then, restore time for a 400MB domain, with the restore file in the cache,
with 4 vbds backed by /dev/loopX, with one vif, is ca 2.7s real time.
According to systemtap, the CPU time requirements are
xend threads- 0.363s
udevd(in dom0) - 0.007s
/etc/xen/scripts/block and its children - 1.075s
xc_restore - 1.368s
/etc/xen/scripts/vif-bridge (in netvm) - 0.130s
The obvious idea to improve /etc/xen/scripts/block shell script execution time
is to recode it, in some other language that will not spawn hundreds of
processes to do its job.
Now, xc_restore.
a) Is it correct that when xc_restore runs, the target domain memory is already
zeroed (because hypervisor scrubs free memory, before it is assigned to a
new domain) ? So, xc_save could check whether a given page contains only
zeroes and if so, omit it in the savefile. This could result in quite
significant savings when
- we save a freshly booted domain, or if we can zero out free memory in the
domain before saving
- we plan to restore multiple times from the same savefile (yes, vbd must be
restored in this case too).
b) xen-3.4.3/xc_restore reads data from savefile in 4k portions - so, one
read syscall per page. Make it read in larger chunks. It looks it is fixed in
xen-4.0.0, is this correct ?
Also, it looks really excessive that basically copying 400MB of memory takes
over 1.3s cpu time. Is IOCTL_PRIVCMD_MMAPBATCH the culprit (its
dom0 kernel code ? Xen mm code ? hypercall overhead ? ), anything
else ?
I am aware that in the usual cases, xc_restore is not the bottleneck
(savefile reads from the disk or the network is), but in case we can fetch
savefile quickly, it matters.
Is 3.4.3 branch still being developed, or pure maintenance mode only, so new
code should be prepared for 4.0.0 ?
Regards,
Rafal Wojtczuk
Principal Researcher
Invisible Things Lab, Qubes-os project
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home