[Xen-devel] measuring time of a domain creation with several device mode

[Hirofumi Tsujimura <tsujimura.hirof@xxxxxxxxxxxxxx>]

Hi all.

About five month ago, there was a topic that a domain with many disks
could not create because of a time-out of timer. I measured time of
domain creation with VBD(file) and VBD(LVM partition) and block tap.

A example of the measurement result is the following. A domain creation
spends O(n^2) time.

-----------------------------+------+-------+-------+---------
 disk                        |   1  |   10  |   50  |   100
-------+---------------------+------+-------+-------+---------
 time  |VBD(file)            | 1.35 | 10.37 | 66.23 | 205.19
[sec.] |VBD(LVM partition)   | 1.18 |  7.52 | 53.65 | 175.52
       |block tap            | 1.17 |  4.43 | 19.30 |  35.43
-------+---------------------+------+-------+-------+---------

I think that a lock contention wastes the time because a jumbo lock is
used on tools/examples/xen-backend.agent script.

Is it possible to avoid using the lock?

The measurement environment is as follows.
  - Architecture  : x86_32 with VTx
  - Xen change set: xen-unstable changeset 14535
  - CPU           : intel core duo processor 1.8GHz
  - Memory        : 1G Byte



Details of the investigation are described as follows.
On the day, I measured creation time up to 87 of disks with VBD(file). A
example of the measurement result is the following.

       table1: use VBD(file)
-----------------+------------------
 number of disk  |    time [sec.]
-----------------+------------------
        1        |       1.27
       10        |      11.43
       50        |     129.19
       87        |     491.41
-----------------+------------------

On the next day, I measured the rest from 87 of disks. A example of the
measurement result is the following.

       table2: use VBD(file)
-----------------+------------------
 number of disk  |     time [sec.]
-----------------+------------------
       87        |     158.01
       90        |     481.14
      100        |     607.35
-----------------+------------------

I wondered why the time at 87 of disks was different by the table1 and
the table2. And so, I watched any process with "top" command when
creating a domain. Then many "xen-backend.agent" processes were
executed. The number of processes was same as the number of devices
defined by a config file.

Firstly, I investigated a tools/examples/xen-backend.agent.

---------------- tools/examples/xen-backend.agent --------------------
#! /bin/bash

PATH=/etc/xen/scripts:$PATH

. /etc/xen/scripts/locking.sh

claim_lock xenbus_hotplug_global

case "$XENBUS_TYPE" in
  tap)
    /etc/xen/scripts/blktap "$ACTION"
    ;;
  vbd)
    /etc/xen/scripts/block "$ACTION"

...

esac

release_lock xenbus_hotplug_global
----------------------------------------------------------------------

The xen-backend.agent process tries to get a lock named
"xenbus_hotplug_global". For example, if a domain A with 100 of disks is
created and a domain B with 100 of disks is destroyed, 200 of the
processes tries to get the lock. I thought that the creation time is
related to the lock contention.

By way of experiment, I created a domain after confirming that
xen-backend.agent process do not exist, then I measured it. A example of
the measurement result is the following.

       table3: use VBD(file)
-----------------+------------------
 number of disk  |    time [sec.]
-----------------+------------------
        1        |       1.35
       10        |      10.37
       50        |      66.23
      100        |     205.19
-----------------+------------------

The time is not so fast.

Secondly, I investigated a tools/examples/block that is executed by the
xen-backend.agent process. The block process searches a usable loop
device from all loop devices to mount a file. At the same time, it
confirms that the file is not used on other loop devices. I thought that
the creation time is related to the searching and the checking.

By way of experiment, I measured the time with VBD(LVM partition) and
block tap.The LVM parition was created on a USB HDD. A example of the
measurement result is the following.

   table4: use VBD(LVM partition)
-----------------+------------------
 number of disk  |     time [sec.]
-----------------+------------------
        1        |       1.18
       10        |       7.52
       50        |      53.65
      100        |     175.52
-----------------+------------------

        table5: use block tap
-----------------+------------------
 number of disk  |     time [sec.]
-----------------+------------------
        1        |       1.17
       10        |       4.43
       50        |      19.30
      100        |      35.43
-----------------+------------------


Thanks.
--
Hirofumi Tsujimura


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel