This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


RE: [Xen-devel] million cycle interrupt

To: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>, "Xen-Devel (E-mail)" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Subject: RE: [Xen-devel] million cycle interrupt
From: "Tian, Kevin" <kevin.tian@xxxxxxxxx>
Date: Wed, 15 Apr 2009 07:19:11 +0800
Accept-language: en-US
Acceptlanguage: en-US
Delivery-date: Tue, 14 Apr 2009 16:19:57 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <49a04d05-4593-4fc6-9941-1aa54515d1dc@default>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <a4e61dfc-f523-4f3b-b24c-2fcb8ba0b3ac@default> <49a04d05-4593-4fc6-9941-1aa54515d1dc@default>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: Acm9JBpVLbY861byQIaANxRm6BotygAMoNQw
Thread-topic: [Xen-devel] million cycle interrupt
>From: Dan Magenheimer [mailto:dan.magenheimer@xxxxxxxxxx] 
>Sent: 2009年4月15日 1:11
>> I'll take a look at that next.
>It appears that the call to smp_timer_broadcast_ipi()
>in timer_interrupt() is the cycle hog.  And it
>definitely appears to be a scaleability problem!

So I guess processors on your platform at least support some
deep C-states (>=C3), or else that broadcast path should be
a nop, is it right? If you disable deep C-states in BIOS, I guess
it then scales well.

>maxcpus=4: avg=1600, max=15000 (cycles, rounded)
>maxcpus=5: avg=2000, max=24000
>maxcpus=6: avg=83000, max=244000
>maxcpus=7: avg=198000, max=780000
>maxcpus=8: avg=310000, max=1027000

This looks sth related to IPI overhead. When more processors
are available, more IPIs are required to contend the bus. But
I'm not sure whether the attitude of change fully comes from
that part. Which type of platform are you using? We'll also
take a try here to first reproduce and then find exact reason.

>The load is a 4vcpu PV EL5u2 32-bit domain continually
>compiling linux-2.6.28 with -j80.  I killed the load
>after only a few minutes, so the max might get worse.
>On the other hand, just booting dom0 seems to put
>max in about the same range.

And average is then very low w/o dom0 only?

Xen-devel mailing list