RE: [Xen-devel] RE: Saving/Restoring IA32_TSC_AUX MSR

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

> > If this is true the only safe use of TSC_AUX is for
> > its originally designed intent: To determine if two
> > successive rdtscp instructions were or were not
> > executed on the same processor.  Since this cannot
> > be guaranteed in a VM, that's a reasonable argument
> > that TSC_AUX shouldn't be exposed at all (meaning the
> > rdtscp bit in cpuid should be turned off by Xen).
> 
> Why do you think this is the design intent of this instruction ?  

The instruction was designed by AMD for this purpose
a few years ago in order to allow applications to
detect (and correct) possible TSC skew between processors.

> For guest NUMA support,  it should be a must to pin each vcpu 
> of one VM to some logical proceossors which belong to one 
> specific node(disable vcpu migration between nodes), I think, 
> otherwise, virutal numa may suffer from performance loss.

I agree that, for guest NUMA support, restricting all
vcpus to the same physical node is important.  However,
PINNING each vcpu to a fixed pcpu (and never allowing
migration) greatly reduces the value of virtualization.

> For example, in a numa system which has two nodes and each 
> node has 4G memory and 8 logical processors. And in this 
> Xen-configured system,  if we carete a VM with 2 G memory 
> with4  vcpu support,  Xen system may allocate 1 G memory from 
> physical node 0 and another 1 G memory from physical node 1.  
> And in this case, if we virtualize numa for this VM, vcpu0 
> and vcpu1 can be assinged to virtual node0 , vcpu2 and vcpu3 
> can be configured for virtual node1, certainly, we also can 
> safely pin vcpu0 and vpcu1 to the physical node0's 8 locial 
> processors and accordingly pin vcpu2 and vcpu3 to the 
> physical node1's 8 physical processors.  Since virtual 
> TSC_AUX is virtualized for each vcpu, and the value is 
> saved/restored for the vcpu when its migration occurs, so if 
> one application always runs on a virtual processors, it 
> should get a fixed value when it calls vgetcpu, envn if this 
> vcpu often migrates among logical processors of one node.   

I agree there are some cases where the TSC_AUX value
set by a guest OS may be useful.  But ensuring that its
is always useful (NEVER incorrect) requires too many restrictions,
such as pinning.

> Back to this topic, in all,  we can't mix the virtual  
> TSC_AUX of guest with the host's TSC_AUX.  If switch to HVM's 
> vcpu context,  load this vcpu's virtual  TSC_AUX_MSR to 
> physical TSC_AUX_MSR, and when it is sheduled out,  host's 
> TSC_AUX_MSR(which maybe used for pv guests) is loaded.  

I agree they can't be mixed.  My position is that a guest
does not have sufficient information to always correctly set
TSC_AUX, so the best way to avoid the issue is to tell
the guest OS that TSC_AUX doesn't exist (i.e. cpuid-rdtscp
bit is off).  Xen can still set TSC_AUX (and even emulate
it on processors that don't support it) and this information
can still be used (correctly) by virtualization-and-NUMA-aware
OS's and applications.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel