This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


RE: [Xen-devel] [RFC] Physical hot-add cpus and TSC

> >> b) With the patch:
> >> After add:
> >> (XEN) TSC marked as reliable, warp = 407 (count=12)
> >> (XEN) TSC marked as reliable, warp = 444 (count=17)
> >> (XEN) TSC marked as reliable, warp = 525 (count=19)
> >
> >Hi Yunhong --
> >
> >Does this continue to grow?  I'm concerned that
> >the hot-added CPU might be skewing as well?
> >I didn't think this was possible with an Invariant
> >TSC machine but maybe something in the hot-add
> >isolation electronics changes the characteristics
> >of the clock signal?
> >Please try:
> >
> >for i in {0..1000}; do xm debug-key t; sleep 3; done; \
> >xm dmesg | tail
> >
> >then wait an hour, and see how large the warp is.
> >Hopefully the trend (407,444,525) is a coincidence.
> I just looped 10 times, I will try with 1000 loops next Monday.
> I suspect there are any isolation electronics for hot-add. Basically
> each CPU can be hot-added except socket 0. But yes, I can have a look
> on it.

Hmmm... I'm not a system hardware expert, but the more I think
about this, the more likely it seems that any hot-plug
board must have separate QPI buses that are driven by separate
crystals.  And there would be some kind of bus bridge/repeater
to connect the two with a forwarding protocol.  That would
certainly explain a growing TSC skew.

If so, even two single socket boards connected like that at
initial boot (no hot-add) are really a "big NUMA" system due
to higher cross-node latencies and might deserve a separate
boot option anyway... this is really NOT a single system...
it is multiple systems glued together with a fast interconnect.
Xen (and users) should be warned that there is no free
lunch here and the performance degradation from TSC emulation
may be only a small part of the problem.

Some boot option like "multiboard_interconnect" (but shorter)
might be appropriate?  Or is there some way at boot-time
to determine that this box does, or might (via hot-add), or
definitely does not, go beyond point-to-point interconnect?
A boot-time decision on TSC emulation could be driven off
of that if it existed.

Xen-devel mailing list