* INAKOSHI Hiroya (inakoshi.hiroya@xxxxxxxxxxxxxx) wrote:
> Hi Mathieu,
>
> thanks for your reply. I can understand your opinion very well but a
> concern is that cpu ids on a guest OS are different from those on Xen
> because they are virtualized. The number of vcpus in a guest OS is also
> different from that of pcpus as you mentioned. I wondered if the two
> traces could be merged directly. If you translate vcpu ids to pcpu ids
> in writing records in the trace buffer in Xen, this concern is solved in
> a natural way.
>
When you are executing code in dom0 or domUs, how do you plan to get the
physical CPU number on which the tracing is done ?
> Mathieu Desnoyers wrote:
> > * INAKOSHI Hiroya (inakoshi.hiroya@xxxxxxxxxxxxxx) wrote:
> >> Hi Mathieu,
> >>
> >> I am interested in LTTng-xen because I thought that it would be nice if
> >> I can get traces on both xen and guest linux at the same time. I
> >> reviewed LTTng-xen and found that
> >>
> >> * LTTng and LTTng-xen have a quite similar structure,
> >> * a trace buffer resides in a hypervisor for LTTng-xen,
> >> * it is currently impossible to get traces from guest linux because
> >> there is no LTTng for 2.6.18-xen kernel, as you mentioned.
> >>
> >> I had coarsely ported LTTng to 2.6.18-xen, though it is only for
> >> i386. Now I can get traces on xen and guest linux simultaneously, even
> >> though they put records in different trace buffers.
> >
> > Hi Ikanoski,
> >
> > We did the same kind of coarse 2.6.18 port at our lab internally to get
> > traces from both Linux and Xen. The fact that the traces are recorded in
> > different buffers does not change anything to the fact that those trace
> > files can be copied in the same trace directory so they can be parsed
> > together by LTTV (traces coming from dom0, domUs and hypervisor). They
> > are synchronized by using the TSCs (hopefully, you will configure your
> > system to get a reliable TSC on AMD and older intels, see the
> > ltt-test-tsc kernel module in recent LTTng versions and ltt.polymtl.ca
> > website for info on that matter).
> >
> >
> >> Then I thought that
> >> it would be more useful if they put records in xen's trace buffer and I
> >> can analyze events
> >
> > LTTV merges the information from all the valid trace files that appears
> > within the trace directory, so the analysis can be done on data coming
> > from userspace, kernels and hypervisor.
> >
> >> from xen and linux guests with a single lttd and
> >> lttctl running on Domain-0. Do you have an opinion about that?
> >>
> >
> > lttctl-xen and lttd-xen, although being quite similar to lttd and
> > lttctl, use hypercalls to get the data. The standard lttctl/lttd uses
> > debugfs files as a hook to the trace buffers.
> >
> > As a distribution matter, I prefer to leave both separate for now,
> > because lttctl-xen and lttd-xen is highly tied to the Xen tree.
> >
> > Also, merging the information within the buffers between Xen and Dom0 is
> > not such a great idea: The Hypervisor and dom0 can have a different
> > number of CPUs (Xen : real CPUs, dom0: vcpus). Since I use per-cpu
> > buffers, it does not fit.
> >
> > Also, I don't want dom0 to overwrite data from the Xen buffers easily:
> > it's better if we keep some protection between dom0 and the Hypervisor.
> >
> > Thanks for looking into this, don't hesitate to ask further questions,
> >
> > Mathieu
> >
> >> Regards,
> >> Hiroya
> >>
> >>
> >> Mathieu Desnoyers wrote:
> >>> Hello,
> >>>
> >>> I made a working version of the LTTng tracer for xen-unstable for x86.
> >>> Here is the pointer to my repository so you can try it out :
> >>>
> >>> hg clone http://ltt.polymtl.ca/cgi-bin/hgweb.cgi xen-unstable-lttng.hg
> >>>
> >>> Basic usage :
> >>>
> >>> (see lttctl-xen -h)
> >>>
> >>> lttctl-xen -c
> >>>
> >>> (in a different console)
> >>> lttd-xen -t /tmp/xentrace1
> >>>
> >>> (in the 1st console)
> >>> lttctl-xen -s
> >>>
> >>> (tracing is active)
> >>>
> >>> lttctl-xen -q
> >>> lttctl-xen -r
> >>>
> >>> lttd-xen should automatically quit after writing the last buffers as
> >>> soon as lttctl-xen -r is issued.
> >>>
> >>> Then, you must copy the XML facilities :
> >>>
> >>> (see the http://ltt.polymtl.ca > QUICKSTART to see how to install the
> >>> ltt-control package which contains the XML facilities in your system)
> >>>
> >>> lttctl-xen -e -t /tmp/xentrace1
> >>>
> >>> View in the visualiser : (see the QUICKSTART to see how to install it)
> >>>
> >>> lttv -m textDump -t /tmp/xentrace1
> >>>
> >>> (not tested yet) : to visualize a dom0 trace with the xen hypervisor
> >>> information, one would have to collect the dom0 kernel trace and the Xen
> >>> trace and open them together with :
> >>> lttv -m textDump -t /tmp/xentrace1 -t /tmp/dom0trace
> >>>
> >>> The current Linux kernel instrumentation is for 2.6.20. A backport might
> >>> be needed to 2.6.18 if there is no proper Xen support in 2.6.20 (I have
> >>> not followed the recent developments).
> >>>
> >>>
> >>> Currently broken/missing :
> >>>
> >>> - Ressources are not freed when the trace channels are destroyed. So you
> >>> basically have to reboot between taking different traces.
> >>> - My code in the hypervisor complains to the console that subbuffers
> >>> have not been fully read when the trace channels are destroyed. The
> >>> error printing is just done too fast : lttd-xen is still there and
> >>> reading the buffers at that point. It will get fixed with proper
> >>> ressource usage tracking of both Xen and lttd-xen (same as the first
> >>> point above).
> >>> - x86_64 not tested, powerpc local.h and ltt.h missing (should be ripped
> >>> from my Linux kernel LTTng).
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Mathieu
> >>>
> >>>
> >>>
> >>> * Mathieu Desnoyers (compudj@xxxxxxxxxxxxxxxxxx) wrote:
> >>>> Hi,
> >>>>
> >>>> My name is Mathieu Desnoyers, I am the current maintainer of the Linux
> >>>> Trace
> >>>> Toolkit project, known as LTTng. This is a tracer for the 2.6 Linux
> >>>> kernels
> >>>> oriented towards high performance and real-time applications.
> >>>>
> >>>> I have read your tracing thread and I am surprised to see how much things
> >>>> you would like in a tracer are already implemented and tested in LTTng.
> >>>> I am
> >>>> currently porting my tracer to Xen, so I think it might be useful for
> >>>> you to
> >>>> know what it provides. My goal is to do not duplicate the effort and save
> >>>> everyone some time.
> >>>>
> >>>> Here follows some key features of LTTng :
> >>>>
> >>>> Architecture independant data types
> >>>> Extensible event records
> >>>> Self-describing traces
> >>>> Variable size records
> >>>> Fast (200 ns per event record)
> >>>> Highly reentrant
> >>>> Does not disable interrupts
> >>>> Does not take lock on the critical path
> >>>> Supports NMI tracing
> >>>> Analysis/visualization tool (LTTV)
> >>>>
> >>>> Looking at the integration of the existing LTTng implementation into
> >>>> Xen, I
> >>>> came up with those two points for my Christmas whichlist :
> >>>>
> >>>> Additionnal functionnalities that would be nice to have in Xen :
> >>>>
> >>>> - RCU-style updates : would allow freeing the buffers without impact on
> >>>> tracing.
> >>>> * I guess I could currently use :
> >>>> for_each_domain( d )
> >>>> for_each_vcpu( d, v )
> >>>> vcpu_sleep_sync(v);
> >>>> I think it will have a huge impact on the system, but it would
> >>>> only be
> >>>> performed before trace buffers free.
> >>>>
> >>>> - Polling for data in Xen from a dom0 process.
> >>>> Xentrace currently polls the hypervisor each 100ms to see if there is
> >>>> data
> >>>> that needs to be consumed. Instead of an active polling, it would be
> >>>> nice to
> >>>> use the dom0 OS capability to put a process to sleep while waiting for
> >>>> a
> >>>> resource. It would imply creating a module, loaded in dom0, that would
> >>>> wait
> >>>> for a specific virq coming from the Hypervisor to wake up such
> >>>> processes.
> >>>> We could think of exporting a complete poll() interface through sysfs
> >>>> or
> >>>> procfs that would be a directory filled with the resources exported
> >>>> from the
> >>>> Hypervisor to dom0 (which could include wait for resource freed,
> >>>> useful when
> >>>> shutting down a domU instead of busy looping). It would help dom0 to
> >>>> schedule
> >>>> other processes while a process is waiting for the Hypervisor.
> >>>>
> >>>>
> >>>> You might also be interested in looking at :
> >>>> - the website (http://ltt.polymtl.ca)
> >>>> - LTTng Xen port design document (this one is different from the one
> >>>> posted by
> >>>> Jimi)
> >>>>
> >>>> (http://ltt.polymtl.ca/svn/ltt/branches/poly/doc/developer/lttng-xen.txt)
> >>>> - OLS 2006 paper "The LTTng tracer : A Low Impact Performance and
> >>>> Behavior
> >>>> Monitor for GNU/Linux"
> >>>> (http://ltt.polymtl.ca/papers/desnoyers-ols2006.pdf)
> >>>>
> >>>>
> >>>> Questions and constructive comments are welcome.
> >>>>
> >>>> Mathieu
> >>>>
> >>>>
> >>>> OpenPGP public key:
> >>>> http://krystal.dyndns.org:8080/key/compudj.gpg
> >>>> Key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
> >>>>
> >>>> _______________________________________________
> >>>> Xen-devel mailing list
> >>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
> >>>> http://lists.xensource.com/xen-devel
> >>>>
> >>
> >>
> >
>
>
--
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home