This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


[Xen-devel] [RFC] Physical hot-add cpus and TSC

To: "Xen-Devel (xen-devel@xxxxxxxxxxxxxxxxxxx)" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Subject: [Xen-devel] [RFC] Physical hot-add cpus and TSC
From: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>
Date: Wed, 26 May 2010 08:19:02 -0700 (PDT)
Delivery-date: Wed, 26 May 2010 08:22:01 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
I thought I should bring this up now rather than
wait until the problem causes problems for some
future customer...

Much of the TSC-based time infrastructure in Xen,
especially as exposed to guests, is rather sensitive
to sudden dramatic differences in TSC values between
physical processors.  Hot-add of physical CPUs will
introduce a huge difference.

Current code attempts to "fix up" TSC after discontinuity
events (such as C3-state) but this works poorly (far
too imprecise), so all guest TSC uses are always emulated
on any physical machine that might have such discontinuities.

Even though a very small percentage of real-world machines
will be capable of hot-cpu-add -- and an even smaller
percentage of real-world machines will ever actually
hot-add a cpu -- Xen 4.0 currently detects and
allows this operation.  As a result, it is currently
impossible to predict if a hot-add might happen and result
in a TSC discontinuity.

Some possible solutions:
1) Always emulate all TSC uses for all guests because
   there is a (microscopic?) probability that a physical
   hot-cpu-add event might occur
2) Disable hot-cpu-add by default in Xen and require a
   Xen boot parameter to be specified if a machine might
   possibly do a hot-cpu-add.  If this boot parameter is
   specified, see (1) above.
3) Do a very precise TSC fixup on a hot-add cpu before it
   is recognized by Xen as present.  (Not sure if this is
4) Dynamically switch to TSC emulation on all guests
   when a hot-cpu-add event occurs.  (Problem:  Under
   certain circumstances, the Invariant TSC bit is
   enabled for some guests which maximizes performance
   on newer Linux kernels.  This choice would require
   Invariant TSC to always be zero.)

Thoughts?  (My favorite is (2))

Xen-devel mailing list