RE: [Xen-devel] [RFC] Physical hot-add cpus and TSC

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

> 2) If disable hot-add-cpu by default, user has to reboot the system to
> enable this feature, it means hot-add CPU is meaningless at all. if
> user need reboot the system, they don't need hot-plug at all, they just
> power-off the system and add it :)

This may be true of a silly one-system administrator, but
large data centers that have systems with hot-add capability
also have documented policies and procedures that their
server administrators must obey.  Once a large data center
makes this mistake once, they will include it in their
policies so that it doesn't happen again.
 
> c) a uevent will be sent to user space of the new added device
> d) uevent script need to "echo 1 >
> /sys/device/system/xen_pcpu/xen_pcpuXXX/online", this store operation
> will trigger a hypercall ,and the CPU will be brought up in the end.
> 
> So my suggestion is, between step c/d, user space script can do more
> work before really bringup the CPU. For example, it can check if any
> special guest/application eixsting requiring strict TSC sequence, if
> xen has tsc_skew optoin passed when booting. Or worstly, it can simply
> does not notify Xen for CPU brought-up at all. I think this is more
> flexible, and is also reasonable.  And this can be done by OSV release
> (like OVM ) easily.

This is an interesting approach.  But I don't think dom0 has
the knowledge about what assumptions guests might make.
Some of the information might be possible to obtain from
Xen if we add new dom0<->Xen interfaces.  But other decisions
are made entirely inside of the guest OS or app and are
not exposed to dom0.  For example, guest A boots, checks for
Invariant TSC, finds that it is set, and selects tsc as a
clocksource; while guest B never checks Invariant TSC (even
though it is set) and never even uses TSC.  I don't think
dom0 or Xen can differentiate the two.

And failing to notify Xen because the udev script (or system
admininstrator) isn't sure about the answer is the same as
requiring a reboot to specify a boot option.

> >Are there any other questionable conditions that might
> >arise from hot-adding physical CPUs?  For example (my
> >favorite), are any order>0 allocations required?  Or
> 
> I don't remember >0 allocation,, will check it when back to office.
> 
> >what if the hot-added cpu results in mixed generations
> >(e.g. a Nehalem is added to an all-Westmere system,
> >where the apps are using AES instructions)?  Anything
> >else?
> 
> What will happen if system is booting with mixed generation? For
> example, when AES is not supported found at AP, will BSP disable the
> AES?

These were just possible examples.  I think there are probably
other examples which may cause problems.

> Agree that correctness is most important, what I suggested is, let
> dom0/adminstrator tools to guard the correctness, not hypervisor, to
> keop the flexibility. Any idea.

And I agree with you that anything that can be done in tools
should be done in tools instead of the hypervisor.  But requiring
a physical system administrator to know everything about every
feature in the underlying physical system, every feature in
any guest OS, and every app that may or may not run on any VM
now or in the future -- and then requiring that admin to make
decisions -- does not IMHO do much to guard correctness.

Requiring a boot option for hot-add guarantees correctness
(at the cost of performance only when the boot option is
specified) and is very simple to implement; that's why I
am in favor of it.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel