WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] [RFC] Physical hot-add cpus and TSC

To: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>, "Xen-Devel (xen-devel@xxxxxxxxxxxxxxxxxxx)" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Ian Pratt <Ian.Pratt@xxxxxxxxxxxxx>
Subject: RE: [Xen-devel] [RFC] Physical hot-add cpus and TSC
From: "Jiang, Yunhong" <yunhong.jiang@xxxxxxxxx>
Date: Fri, 28 May 2010 13:39:58 +0800
Accept-language: en-US
Acceptlanguage: en-US
Cc:
Delivery-date: Thu, 27 May 2010 22:42:38 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <3c99c55d-68ce-4150-b895-72fda1ff3b89@default>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <789F9655DD1B8F43B48D77C5D30659731E78D370@xxxxxxxxxxxxxxxxxxxxxxxxxxxx> <C823EF64.1603B%keir.fraser@xxxxxxxxxxxxx 789F9655DD1B8F43B48D77C5D30659731E78D500@xxxxxxxxxxxxxxxxxxxxxxxxxxxx> <3c99c55d-68ce-4150-b895-72fda1ff3b89@default>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: Acr9rpJYA+BrOykCR6yHgEJ6dUO4GgAZaAKg
Thread-topic: [Xen-devel] [RFC] Physical hot-add cpus and TSC

>-----Original Message-----
>From: Dan Magenheimer [mailto:dan.magenheimer@xxxxxxxxxx]
>Sent: Thursday, May 27, 2010 11:08 PM
>To: Jiang, Yunhong; Keir Fraser; Xen-Devel (xen-devel@xxxxxxxxxxxxxxxxxxx); Ian
>Pratt
>Subject: RE: [Xen-devel] [RFC] Physical hot-add cpus and TSC
>
>> >From: Keir Fraser [mailto:keir.fraser@xxxxxxxxxxxxx]
>> >
>> >I implemented this as xen-unstable:21468. This represents a strict
>> >improvement on what was in xen-unstable before that (no tsc sync at
>> all,
>> >ever, because I deleted it about a week ago). Open to further
>> improvements,
>> >if we can get consensus.
>>
>> Thanks for the patch. I will get the system that support CPU hotplug
>> tomorrow morning, I can try it then.
>>
>> --jyh
>

Below is test result:
a) With the patch
Before hotadd:
(XEN) TSC marked as reliable, warp = 0 (count=3)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=4)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=5)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=6)
(XEN) No domains have emulated TSC

After add
(XEN) TSC marked as reliable, warp = 1669912421214 (count=15)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 1669912421214 (count=16)
(XEN) No domains have emulated TSC

b) With the patch:
Before adding:
(XEN) TSC marked as reliable, warp = 0 (count=2)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=3)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=4)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=5)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=6)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=7)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 0 (count=8)
(XEN) No domains have emulated TSC

After add:
(XEN) TSC marked as reliable, warp = 407 (count=12)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 407 (count=13)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 407 (count=14)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 407 (count=15)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 407 (count=16)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 444 (count=17)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 444 (count=18)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 525 (count=19)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 525 (count=20)
(XEN) No domains have emulated TSC
(XEN) TSC marked as reliable, warp = 525 (count=21)
(XEN) No domains have emulated TSC

>Hmmm... I'd be very surprised if this works ever, let alone
>always across all hardware.  By "work", I mean that it will
>result in an "undetectably small difference" (less than a
>cache bounce), as defined and measured by the tsc warp test.
>Why?  Because rdtsc is a long instruction (30-100 cycles)
>and I'll bet writing to the TSC is even longer.  Plus,
>there's a cache bounce overhead added in due to the
>synchronization via the in-memory tsc_count variable.

I think that depends how we define " undetectably". If the time that guest 
migration among physical CPU is much higher than this difference, do we really 
need care about it (of course SMI/NMI is another story)?
Or I missed anything?

--jyh

>
>Our firmware guys say that TSC synchronization can't be
>implemented algorithmically in firmware... it requires
>a simultaneous "reset" signal to all sockets/cores, which
>is obviously not an option here.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>