WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] [PATCH] TSC scaling for live migration between platforms

To: John Levon <levon@xxxxxxxxxxxxxxxxx>
Subject: RE: [Xen-devel] [PATCH] TSC scaling for live migration between platforms with different TSC frequecies
From: "Zhang, Xiantao" <xiantao.zhang@xxxxxxxxx>
Date: Fri, 19 Jun 2009 23:07:28 +0800
Accept-language: en-US
Acceptlanguage: en-US
Cc: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Ian Pratt <Ian.Pratt@xxxxxxxxxxxxx>, Keir Fraser <Keir.Fraser@xxxxxxxxxxxxx>
Delivery-date: Fri, 19 Jun 2009 08:46:20 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20090619135337.GB32704@xxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4FA716B1526C7C4DB0375C6DADBC4EA3417372D32E@xxxxxxxxxxxxxxxxxxxxxxxxx> <8ef68a49-a145-4de2-898c-ab5e27937028@default> <20090618164535.GA30082@xxxxxxxxxxxxxxxxx> <706158FABBBA044BAD4FE898A02E4BC201BD7DBC6E@xxxxxxxxxxxxxxxxxxxxxxxxxxxx> <20090619135337.GB32704@xxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: Acnw5VyaW4HBhhh8Q+uftrinc5/tSAACCw3w
Thread-topic: [Xen-devel] [PATCH] TSC scaling for live migration between platforms with different TSC frequecies
John Levon wrote:
> On Fri, Jun 19, 2009 at 10:25:36AM +0800, Zhang, Xiantao wrote:
> 
>> Actually, we had done some optimizations and make most of rdtsc not
>> trap to hypervisor and always keep tsc monotonically increasing in
>> each virtual processor, but it is hard to solve the tsc drift issue
>> between all vcpus.  In our solution, tsc drift may exceeds 100000
>> cycles. You know, TSC drift maybe also exist in real platforms, but
>> maybe not quite large like 1000000 cycles. Do you know what's the
> 
> I'm not sure what you mean by "drift of 1000000 cycles". In what time
> period? Or are you talking about skew (a constant difference between
> TSC values read between CPUs) ? Solaris handles arbitrary skew (I
> think) but that's only accounted for at boot time. A fudge factor
> (tsc_max_delta) accounts for any minor post-calibration deltas.

I mean inter-cpu TSC drift at any time. In our solution, we always keep guest's 
TSC across all vcpus synced to its expected TSC in 1 ms or 0.5 ms.  That is to 
say, all vcpus' TSC monotonically increases, but may generate little drift(10^5 
~10^6 cycles) due to unsync vm exits of vcpus. 


> On HVM or VMWare we don't even try, since we can't possibly know the
> real CPUs skew: the assumption is the VM platform has already done
> this for us. And at least Xen attempts to sync up the physical CPUs.
> Significant drift (where different CPUs are ticking at different
> rates) is bad news, and can easily lead to non-monotonicity. I don't
> know what "significant" means though, unfortunately.

We can guanrantee each vcpu's TSC is increasing monotonically, but there maybe 
some diff between vcpus.  I am not sure 10^5 cycles is significant, but it 
should exceed a stable hardware's drift in general. 


> Finally, a change across all CPUs in the tsc tick rate (so no drift,
> but a sudden change after, say, a migration) is also bad news.
> Solaris used to recalibrate the scale rate once a second, but that
> was removed.

Bad news. 

> All this is HVM/metal code of course.
> 
> regards
> john


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>