[Xen-devel] [RFC] Physical hot-add cpus and TSC

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

I thought I should bring this up now rather than
wait until the problem causes problems for some
future customer...

Much of the TSC-based time infrastructure in Xen,
especially as exposed to guests, is rather sensitive
to sudden dramatic differences in TSC values between
physical processors.  Hot-add of physical CPUs will
introduce a huge difference.

Current code attempts to "fix up" TSC after discontinuity
events (such as C3-state) but this works poorly (far
too imprecise), so all guest TSC uses are always emulated
on any physical machine that might have such discontinuities.

Even though a very small percentage of real-world machines
will be capable of hot-cpu-add -- and an even smaller
percentage of real-world machines will ever actually
hot-add a cpu -- Xen 4.0 currently detects and
allows this operation.  As a result, it is currently
impossible to predict if a hot-add might happen and result
in a TSC discontinuity.

Some possible solutions:
1) Always emulate all TSC uses for all guests because
   there is a (microscopic?) probability that a physical
   hot-cpu-add event might occur
2) Disable hot-cpu-add by default in Xen and require a
   Xen boot parameter to be specified if a machine might
   possibly do a hot-cpu-add.  If this boot parameter is
   specified, see (1) above.
3) Do a very precise TSC fixup on a hot-add cpu before it
   is recognized by Xen as present.  (Not sure if this is
   possible.)
4) Dynamically switch to TSC emulation on all guests
   when a hot-cpu-add event occurs.  (Problem:  Under
   certain circumstances, the Invariant TSC bit is
   enabled for some guests which maximizes performance
   on newer Linux kernels.  This choice would require
   Invariant TSC to always be zero.)

Thoughts?  (My favorite is (2))


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel