This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR

To: "Zhang, Xiantao" <xiantao.zhang@xxxxxxxxx>, "Nakajima, Jun" <jun.nakajima@xxxxxxxxx>, Jeremy Fitzhardinge <jeremy@xxxxxxxx>
Subject: RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR
From: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>
Date: Sun, 13 Dec 2009 10:06:07 -0800 (PST)
Cc: "Xu, Dongxiao" <dongxiao.xu@xxxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxx, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>, "Dugger, Donald D" <donald.d.dugger@xxxxxxxxx>
Delivery-date: Sun, 13 Dec 2009 10:07:08 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <EB8593BCECAB3D40A8248BE0B6400A382E873CF2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
> > If this is true the only safe use of TSC_AUX is for
> > its originally designed intent: To determine if two
> > successive rdtscp instructions were or were not
> > executed on the same processor.  Since this cannot
> > be guaranteed in a VM, that's a reasonable argument
> > that TSC_AUX shouldn't be exposed at all (meaning the
> > rdtscp bit in cpuid should be turned off by Xen).
> Why do you think this is the design intent of this instruction ?  

The instruction was designed by AMD for this purpose
a few years ago in order to allow applications to
detect (and correct) possible TSC skew between processors.

> For guest NUMA support,  it should be a must to pin each vcpu 
> of one VM to some logical proceossors which belong to one 
> specific node(disable vcpu migration between nodes), I think, 
> otherwise, virutal numa may suffer from performance loss.

I agree that, for guest NUMA support, restricting all
vcpus to the same physical node is important.  However,
PINNING each vcpu to a fixed pcpu (and never allowing
migration) greatly reduces the value of virtualization.

> For example, in a numa system which has two nodes and each 
> node has 4G memory and 8 logical processors. And in this 
> Xen-configured system,  if we carete a VM with 2 G memory 
> with4  vcpu support,  Xen system may allocate 1 G memory from 
> physical node 0 and another 1 G memory from physical node 1.  
> And in this case, if we virtualize numa for this VM, vcpu0 
> and vcpu1 can be assinged to virtual node0 , vcpu2 and vcpu3 
> can be configured for virtual node1, certainly, we also can 
> safely pin vcpu0 and vpcu1 to the physical node0's 8 locial 
> processors and accordingly pin vcpu2 and vcpu3 to the 
> physical node1's 8 physical processors.  Since virtual 
> TSC_AUX is virtualized for each vcpu, and the value is 
> saved/restored for the vcpu when its migration occurs, so if 
> one application always runs on a virtual processors, it 
> should get a fixed value when it calls vgetcpu, envn if this 
> vcpu often migrates among logical processors of one node.   

I agree there are some cases where the TSC_AUX value
set by a guest OS may be useful.  But ensuring that its
is always useful (NEVER incorrect) requires too many restrictions,
such as pinning.

> Back to this topic, in all,  we can't mix the virtual  
> TSC_AUX of guest with the host's TSC_AUX.  If switch to HVM's 
> vcpu context,  load this vcpu's virtual  TSC_AUX_MSR to 
> physical TSC_AUX_MSR, and when it is sheduled out,  host's 
> TSC_AUX_MSR(which maybe used for pv guests) is loaded.  

I agree they can't be mixed.  My position is that a guest
does not have sufficient information to always correctly set
TSC_AUX, so the best way to avoid the issue is to tell
the guest OS that TSC_AUX doesn't exist (i.e. cpuid-rdtscp
bit is off).  Xen can still set TSC_AUX (and even emulate
it on processors that don't support it) and this information
can still be used (correctly) by virtualization-and-NUMA-aware
OS's and applications.

Xen-devel mailing list