[Xen-bugs] [Bug 882] New: Cpu Usage Discrepancies on Xen

[bugzilla-daemon@xxxxxxxxxxxxxxxxxxx]

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=882

           Summary: Cpu Usage Discrepancies on Xen
           Product: Xen
           Version: 3.0.2
          Platform: x86
        OS/Version: Linux-2.6
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Hypervisor
        AssignedTo: xen-bugs@xxxxxxxxxxxxxxxxxxx
        ReportedBy: pradeep.vincent@xxxxxxxxx


I see serious discrepancies between Cpu usage as reported by /proc/stat on Xen3
virts and Cpu usage as reported by the hypervisor via "xm" tool (cpu_time). I
ran some tests and observed how CPU usage is reported. The results also brought
to light a host of other discrepancies.


Physical Host Details
Processor 2x AMD Opteron 2400/200
RAM - 8192MB in 8 DIMMs

Kernel/Hypervisor - Based on Redhat's 2.6.18-1.2747.el5
Hypervisor Scheduler - Credit Scheduler

Virts Configured
Slot A 
Dom weight = 500
RAM ~ 4G
Vcpus = 2

Slot B
Dom weight = 500
RAM ~ 4G
Vcpus = 2

Dom-0

Dom weight = 1000


During both the tests Slot A ran a constant workload that shows CPU utilization
(as reported by vmstat and such) around 10 to 15% when Slot B was idle. 

Test 1 

Slot A serves the above described load and Slot B is idle

Slot A's CPU Usage from /proc/stat,

Jiffies Per second per vcpu: Usage: 12.75 Idle: 86.256 Stolen:0.983

Slot A's CPU Usage from XM,

Jiffies used Per second per vcpu from xm (cpu_time) : 21.11

While /proc/stat reports 12.75 jiffies used per second per vcpu, xm reports
21.11 jiffies. Shouldn't these two match.


Test 2

Slot A serves the above described load and Slot B serves a load that consumes
50% of overall CPU 

Slot B's CPU usage from its /proc/stat,

Jiffies per sec per vcpu Usage = 47.925 Idle = 46.37 Stolen = 5.805

Slot B's CPU usage from XM,

Jiffies used per sec per vcpu From XM (cpu_time) = 49.9705


Slot A's CPU Usage from /proc/stat,

Jiffies per sec per vcpu Usage = 25.46 Idle = 34.51 Stolen = 34.435 

Slot A's CPU Usage from XM,

Jiffies used per sec per vcpu From XM (cpu_time) = 38.002 

Note than cpu_time has been converted from seconds to jiffies. 

Number of observations can be made here

a. Slot B CPU usage is as expected.
b. Total no. of jiffies (including stolen) doesn't add upto 100 per sec per cpu
in Slot A during Test 2 (~95 jiffies /sec)
C. As with Test 1, CPU usage for Slot A via "XM" doesn't match CPU usage via
/proc/stat (25.46 via /proc/stat and 38 via "XM")
D. Jiffy usage for the same workload on Slot A INCREASES when Slot B's workload
increases (12.75 to 25.46)

I did see similar discrepancies on Intel hosts as well as Virts with 1 Vcpu.
The skew is pronounced with workloads that "sleep-wake-sleep-wake" at a high
frequency while workloads that hog the CPU don't exhibit this problem as much. 

I was able to reproduce the problem (to some extent) by executing "ls -R" from
root folder.


-- 
Configure bugmail: 
http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

_______________________________________________
Xen-bugs mailing list
Xen-bugs@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-bugs