[Xen-users] Re: xenoprof: operation 9 failed for dom0 (status: -1)

[Dante Cinco <dantecinco@xxxxxxxxx>]

As you can see from the output of "opreport -l" below, most of the cycles are coming from domain1-modules so I do need to focus on --active-domains=1 since --passive-domains=1 does not provide the profiling details for the domU modules.

The steps described in the xenoprof tutorial for active domains is pretty straightforward but I cannot get past the "write error" when I run "opcontrol --start" in dom0.

After running "opcontrol --start" in domU, I see the response below and I use lsmod to verify that oprofile module is loaded. Given the results from --passive-domains=1, the dom0 and Xenoprof interface is working. It's the domU and dom0 interface that has some problem. Is there some other way I can tell from dom0 that domU is "ready" before running "opcontrol --start" in dom0?

FYI: I'm using the same oprofile-0.9.3 with xen-r2.patch in dom0 and domU. If I boot the Debian 2.6.30.1 kernel (the same one I'm using in domU) in bare-metal (no Xen VMM), I'm able to successfully run oprofile-0.9.3.

opcontrol --start (from domU before "opcontrol --start" in dom0):
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.

opreport -l (from dom0 using --passive-domains=1 --passive-images=/boot/vmlinux-2.6.30.1):
CPU: Core 2, speed 2533.51 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        image name               app name                 symbol name
617835   38.6156  domain1-modules          domain1-modules          (no symbols)
448911   28.0576  domain1-xen-unknown      domain1-xen-unknown      (no symbols)
72460     4.5289  domain1-kernel           domain1-kernel           __down
43294     2.7059  domain1-kernel           domain1-kernel           __down_killable
34145     2.1341  domain1-kernel           domain1-kernel           validate_slab_slab

Dante

On Tue, Nov 10, 2009 at 10:22 AM, Santos, Jose Renato G <joserenato.santos@xxxxxx> wrote:

With passive domains you cannot have detailed profiling information on modules, only in kernel builtin functions and on Xen. All the samples associated with modules will be grouped under the same symbol "domain1-modules".

If you are interested in one particular module you should try to recompile the kernel with the associated code builtin (or you can use active domains, but follow the steps on the xenoprof tutorial to coordinate opcontrol in dom0 and in the guest) 

 

Renato

---

From: Dante Cinco [mailto:dantecinco@xxxxxxxxx]

Sent: Tuesday, November 10, 2009 10:13 AM

To: Santos, Jose Renato G
Cc: Xen-users
Subject: Re: xenoprof: operation 9 failed for dom0 (status: -1)

Renato,

I think I'm making progress. I followed your suggestion of using --passive-domains and --passive-images. When I run opreport, it is unable to find /domain1-modules and /domain1-xen-unknown. Where or how do I specify the kernel modules I have running in domU/domain1? I tried copying the *.ko files in /boot in dom0 and used --image-path=/boot in opreport and it is still not finding them.

Thanks.

Dante

On Mon, Nov 9, 2009 at 6:44 PM, Santos, Jose Renato G <joserenato.santos@xxxxxx> wrote:

Try replacing "--active-domains=1" with "--passive-domains=1 passive-images=<domU-kernel-image>" (use the uncompressed version of your kernel image for the guest,  vmlinux-*)

 

To use "active-domains" you need to run opcontrol in the guest in addition to running it in dom0 and you need to coordinate the execution of both instances.. This require the guest opcontrol to be ready before running "opcontrol --start" in dom0. That is why it is failing. I suspect you have not executed opcontrol in the guest

Using active-domains is very tricky. I suggest that you use --passive-domains, unless you really need active domains (it is only useful in case you need detailed profiles for user processes running in the guest)

 

Renato 

---

From: Dante Cinco [mailto:dantecinco@xxxxxxxxx]

Sent: Monday, November 09, 2009 6:13 PM

To: Santos, Jose Renato G

Cc: Xen-users
Subject: Re: xenoprof: operation 9 failed for dom0 (status: -1)

Renato,

I've narrowed down the opcontrol command sequence that causes the "write error" I'm having. If I just run "opcontrol --start" in dom0, it runs with no error and after "opcontrol --shutdown" I can run "opreport" and get a real report.

If I run "opcontrol --start-daemon --active-domains=1" in dom0, run "opcontrol --start" in domU (ID#1) and go back to dom0 and run "opcontrol --start" I get the "write error" message. It's as if "--start-daemon" is grabbing the file handle for /dev/oprofile/enable so when "--start" tries to write "1" to /dev/oprofile/enable, it is unable to do so because it is already locked.

So I can run OProfile in normal (non-Xen) mode but as soon I start using "--start-daemon" I have problems. To me it seems like a Xenoprofile problem.

I'm using OProfile 0.9.3 and oprofile-0.9.3-xen-r2.patch applied successfully.

Dante   

On Thu, Nov 5, 2009 at 6:05 PM, Santos, Jose Renato G <joserenato.santos@xxxxxx> wrote:

What version of OProfile are you using?

Did you apply the Xen patch available in http://xenoprof.sourceforge.net ?

 

Renato

---

From: Dante Cinco [mailto:dantecinco@xxxxxxxxx]
Sent: Thursday, November 05, 2009 5:16 PM
To: Santos, Jose Renato G
Cc: Xen-devel
Subject: xenoprof: operation 9 failed for dom0 (status: -1)

Renato,

When I tried running "opcontrol --start" (after previously running "opcontrol --start-daemon") in dom0, I get this error message:

/usr/local/bin/opcontrol: line 1639: echo: write error: Operation not permitted

and this message in the Xen console:

(XEN) xenoprof: operation 9 failed for dom 0 (status : -1)

It looks like opcontrol is trying to do this: echo 1 > /dev/oprofile/enable

and it is failing. "operation 9" maps to XENOPROF_start which is consistent with running "opcontrol --start." At first, I ignored the error because it gave the indication "Profiler running" but after I ran "opcontrol --shutdown" followed by "opreport" in dom0, I got this error message:

error: no sample files found: profile specification too strict ?

Do you know why the write error is occurring? I followed the steps in xenoprof_tutorial.ppt.

Dante

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users