This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] RE: Live migration fails due to c/s 20627

To: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>
Subject: Re: [Xen-devel] RE: Live migration fails due to c/s 20627
From: Andre Przywara <andre.przywara@xxxxxxx>
Date: Wed, 16 Dec 2009 16:41:23 +0100
Cc: Jeremy Fitzhardinge <jeremy@xxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxx, kurt.hackel@xxxxxxxxxx, "Dugger, Donald D" <donald.d.dugger@xxxxxxxxx>, "Xu, Dongxiao" <dongxiao.xu@xxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>, "Nakajima, Jun" <jun.nakajima@xxxxxxxxx>, "Zhang, Xiantao" <xiantao.zhang@xxxxxxxxx>
Delivery-date: Wed, 16 Dec 2009 07:42:09 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <57c748fc-49ef-4f2f-9c98-92a1dc12116f@default>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <57c748fc-49ef-4f2f-9c98-92a1dc12116f@default>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird (X11/20090329)
Dan Magenheimer wrote:
-----Original Message-----
From: Jeremy Fitzhardinge [mailto:jeremy@xxxxxxxx]
Sent: Tuesday, December 15, 2009 11:26 AM
To: Xu, Dongxiao
Cc: Dan Magenheimer; Keir Fraser; xen-devel@xxxxxxxxxxxxxxxxxxx; Kurt
Hackel; Dugger, Donald D; Nakajima, Jun; Zhang, Xiantao
Subject: Re: Live migration fails due to c/s 20627

On 12/15/2009 09:24 AM, Xu, Dongxiao wrote:
If CPU has rdtsc but no rdtscp, then the VM exec control bit in VMCS
won't be turned on. Therefore if rdtscp instruction runs,
it will encounter
invalid op code directly but no VMEXIT.
Ah, right. You'd need to make that particular illegal instruction vmexit.

Or make ALL illegal instructions vmexit, decode, if rdtscp
emulate it, else vmenter again.
But is it useful to emulate RDTSCP? I see two use cases for this instruction:
1) NUMA aware malloc:
You need to know the current node number _quickly_ to use the right bucket to take the memory from. You do not even want using a syscall for this, that's why getcpu in Linux is implemented as a vsyscall either using RDTSCP or LSL. If you emulate this, this will need a few thousand cycles.
2) Making sure TSC values are consistent:
By looking at the core ID you learn whether two consecutive RDTSCPs are from the same core and are thus reliable. If you loose a few thousand cycles with emulation, than the whole purpose of doing the RDTSCPs is in question, as your results would be spoiled due to the overhead.

These two issues are the main reason I refrained from implementing RDTSCP virtualization some months ago, as even virtualizing them introduces a slight overhead (MSR save/restore). As software seems to cope with not having this instruction (and using the perfectly virtualized lsl instruction, for instance), I thought the benefit would not justify the effort.

Dan, can you summarize the usage of RDTSCP emulation in PV? Honestly I got lost in all these threads..


Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448 3567 12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Xen-devel mailing list