[Xen-devel] Issues using xenpm in 3.3.1 on Intel Nehalem cpu

[Andrew J Younge <ajy4490@xxxxxxx>]

Hello,

I am having some issues with using the new Xen Power management features in Xen 3.3.1 and I'm need of some help. If this question should be directed elsewhere, please let me know.  

My goal is to be able to dynamically turn the CPU frequency up or down for each core in dom0,
depending on the VM load.  I am hoping to do this either with
an automatic governor (ondemand) or write my own custom scripts to control this
dynamically.  To my knowledge, the new Core i7 Nehalem processors are able to dynamically control each core's
frequency and voltage independent of another, which is why I chose the
CPU.  In a normal non-xen environment (vanilla Ubuntu 8.10) I can see this
working with various governors using the cpufreq-set and cpufreq-info
tools that in turn rely on the acpi-cpufreq module.  When Xen announced
it had the same support in 3.3.1, I was very excited to be able to
recreate this effect with virtual machine loads in Xen. 

My current setup is a Intel Core i7 2.66GHz (with hyperthreading) in a X58 motherboard with 6Gb DDR3 ram and a 300GB raptor SATA HD.  I have installed Ubuntu 8.10 server x86_64 as the default OS and then compiled and installed Xen 3.3.1 from source [1],[2].  Furthermore, I am using the 2.6.27.5 xen kernel [3] for Dom0 as I had issues booting with the 2.6.18-8-xen kernel that came with the source.  I have enabled my dom0 to boot with cpufreq=xen cpuidle parameters in my grub menu.conf [4] to enable use of the xenpm tool. 

While xenpm seems to be working on my system by reading all C states and P states (with hyperthreading enabled too), the controls do nothing.  For instance, if I try to set any governor (ex: "xenpm set-scaling-governor performance"), it seems to have no effect.  No matter what command I give, the default behavior is the same.  If the system is idle, then it will show all cores as running at the lowest P-state of 1600MHz, which is correct.  However when I create load that only would use 1 core (burnP5 &), the all cores report to run at 2668Mhz (highest setting).  This is not what I want, obviously.  Like I said before, the acpi-cpufreq module only scales up 1 core in such a situation.  If I "xm create" a DomU (ex: Debian Lenny) and put load on that machine (burnP5), the result is no different, all P-states report at 2668Mhz.  I don't know if its actually turning all the CPUs up to 2.6Ghz, or xenpm just reporting at 2.6Ghz but doing the right thing under the hood.  

After a bit of google searching, I found a previous mailing list thread [5] that describes a similar issue and provides a patch fix.  However it looks like this patch was issued before the release of 3.3.1 so I don't think it applies to me. I did take a look at the code, and the patch doesn't match up with the current code structure that I have in 3.3.1, so the patch wouldn't install even if I tried.  It also looks that there were still problems being reported after the patch that are similar to mine (all cores reporting the same freq when maybe they aren't at the same freq).  To be honest some of the details discussed in the later part of this article are beyond my current level of understanding, so if I am missing something please let me know.  I have attached the output of "xm dmesg" after booting up in Dom0 and running "xenpm" once as it may be of some use.  

Any help with this issue would be greatly appreciated.  I am really looking forward to have dynamic control of each core operating frequency in Xen as I believe some valuable research can come from this.  I am not sure if there is an issue with Xen 3.3.1 and the new Core i7 system I have, or just my level of understanding.  I would be happy to test any patches or experimental versions if necessary. My lab has also ordered a dual-socket quad-core Xeon 5520 system (also Nehalem based) that I plan to use in a similar way, so if there is testing that can be done on this machine too I would also be volunteer its use.

Thank you in advance for any help you can provide me.

Andrew

References:
[1] Xen 3.3.1 source code http://bits.xensource.com/oss-xen/release/3.3.1/xen-3.3.1.tar.gz
[2] Instructions on installing Xen 3.3.1 on Debian http://www.howtoforge.com/virtualization-with-xen-3.3.1-on-debian-etch
[3] Instructions to compile 2.6.27 dom0 kernel http://www.howtoforge.com/installing-xen-3.3-with-kernel-2.6.27-on-ubuntu-8.10-x86_64
[4] Xenpm manual http://wiki.xensource.com/xenwiki/xenpm
[5] C-state and P-state conrol in Xen http://markmail.org/message/2ivrwafrue2krdox#query:depends%20on%20!PROCESSOR_EXTERNAL_CONTROL+page:1+mid:digdksyggm7ofbdp+state:results


Andrew J Younge
Rochester Institute of Technology
102 Lomb Memorial Drive
Rochester, NY 14623
ajy4490@xxxxxxx
1.585.259.5170

dmesg_04.06.2009.out
Description: Binary data

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel