I'm experimenting with clock_gettime(), gettimeofday(),
and rdtsc with a 2.6.30 64-bit pvguest. I have tried both
with kernel.vsyscall64 equal to 0 and 1 (but haven't seen
any significant difference between the two). I have
confirmed from sysfs that clocksource=xen
I have yet to get a measurement of either syscall that
is better than 2.5x WORSE than emulating rdtsc. On
my dual-core Conroe (Intel E6850) with 64-bit Xen and
32-bit dom0, I get approximately:
rdtsc native: 22ns
softtsc (rdtsc emulated): 360ns
gettime syscall w/softtsc: 1400ns
gettime syscall native tsc: 980ns
gettimeofday w/softtsc: 1750ns
gettimeofday native tsc: 900ns
I'm hoping this is either a bug in the 2.6.30 xen
pvclock implementation or in my measurement methodology,
so would welcome others measuring this.
A couple other minor observations:
1) The syscalls seem to be somewhat slower when usermode
rdtscs are being emulated, by approximately the cost
of emulating an rdtsc. I suppose this makes
sense since vsyscalls are executed in userland
and since vgettimeofday does a rdtsc. However it
complicates strategy if emulating rdtsc is the default.
2) The syscall clock_getres() does not seem to reflect
the fact that
> -----Original Message-----
> From: Dan Magenheimer
> Sent: Saturday, August 29, 2009 11:52 AM
> To: Jeremy Fitzhardinge
> Cc: Alan Cox; Xen-Devel (E-mail); Keir Fraser
> Subject: RE: [Xen-devel] write_tsc in a PV domain?
>
>
> (Reordered with most important points first...)
>
> > You are talking about three different cases:
>
> I agree with your analysis for case 1 and case 3.
>
> > So, there's case 2: pv usermode. There are four
> > classes of apps worth considering here:
>
> I agree with your classification. But a key point
> is that VMware provides correctness for all
> of these classes. AND provides it at much better
> performance than trap-and-emulate. AND provides
> correctness+performance regardless of the underlying
> OS (e.g. even "old" OS's such as RHEL4 and RHEL5).
> AND provides it regardless whether the guest OS is
> 32-bit or 64-bit. AND, by the way, provides it for
> your case 1 (PV OS) and case 3 (HVM) as well.
>
> > So if you want to address these problems, it seems to me
> > you'll get most
> > bang for the buck by fixing (v)gettimeofday to use pvclock, and
> > convincing app writers to trust in gettimeofday.
>
> (Partially irrelevant point, but gettimeofday returns
> microseconds which is not enough resolution for many
> cases where rdtsc has been used in apps. Clock_gettime
> is the relevant API I think.)
>
> If we can come up with a way for a kernel-loadable module
> to handle some equivalent of clock_gettime so that
> the most widely used shipping PV OS's can provide a
> pvclock interface to apps, this might be workable.
> If we tell app providers and customers: "You
> can choose either performance OR correctness but
> not both, unless you upgrade to a new OS (that is
> not even available yet)", I don't think that will
> be acceptable.
>
> Any ideas on how pvclock might be provided through
> a module that could be added to, eg. RHEL4 or RHEL5?
>
> > > There ARE guaranteed properties specified by
> > > the Intel SDM for any _single_ processor...
> >
> > Yes, but those are fairly weak guarantees. It does not
> guarantee that
> > the tsc won't change rate arbitrarily, or stop outright
> between reads.
>
> They are weak guarantees only if one uses rdtsc
> to accurately track wallclock time. They are
> perfectly useful guarantees if one simply wants to
> either timestamp data to record ordering (e.g.
> for journaling or transaction replay), or
> approximate the passing of time to provide
> approximate execution metrics (e.g. for
> performance tools).
>
> > > What is NOT guaranteed, but is widely and
> > > incorrectly assumed to be implied and has
> > > gotten us into this mess, is that
> > > the same properties applies across multiple
> > > processors.
> >
> > Yes, Linux offers even weaker guarantees than Intel. Aside from the
> > processor migration issue, the tsc can jump arbitrarily as
> a result of
> > suspend/resume (ie, it can be non-monotonic).
>
> Please explain. Suspend/resume is an S state isn't
> it? Is it possible to suspend/resume one processor
> in an SMP system and not another processor? I think
> not. Your point is valid for C-states and P-states
> but those are what Intel/AMD has fixed in the most
> recent families of multi-core processors.
>
> So I don't see how (in the most recent familes of
> processors) tsc can be non-monotonic.
>
> > Even very recent processors with "constant" tscs (ie, they
> > don't change
> > rate with the core frequency) stop in certain power states.
>
> For the most recent families of processors, the TSC
> continues to run at a fixed rate even for all the
> P-states and C-states. We should confirm this with
> Intel and AMD.
>
> > Any motherboard design which runs packages in different
> > clock-domains will lose tsc-sync between those packages,
> > regardless of what's in the packages.
>
> I'm told this is not true for recent multi-socket systems
> where the sockets are on the same motherboard. And at
> least one large vendor that ships a new one-socket-per-
> motherboard NUMA-ish system claims that it is not even
> true when the sockets are on different motherboards.
>
> Dan
>
> (no further replies below, remaining original text retained
> for context)
>
> > You are talking about three different cases:
> >
> > 1. the reliability of the tsc in a PV guest in kernel mode
> > 2. the reliability of the tsc in a PV guest in user mode
> > 3. the reliability of the tsc in an HVM guest
> >
> > I don't think 1. needs any attention. The current scheme
> works fine.
> >
> > The only option for 3 is to try make a best-effort of tsc
> > quality, which
> > ranges from trapping every rdtsc to make them all give globally
> > monotonic results, or use the other VT/SVM features to
> apply an offset
> > from the raw tsc to a guest tsc, etc. Either way the
> situation isn't
> > much different from running native (ie, apps will see
> > basically the same
> > tsc behaviour as in the native case, to some degree of
> approximation).
> >
> > So, there's case 2: pv usermode. There are four classes of
> apps worth
> > considering here:
> >
> > 1. Old apps which make unwarranted assumptions about the
> > behavour of
> > the tsc. They assume they're basically running on some
> > equivalent
> > of a P54, and so will get junk on any modernish
> system with SMP
> > and/or power management. If people are still using
> > such apps, it
> > probably means their performance isn't critically
> > dependent on the
> > tsc.
> > 2. More sophisticated apps which know the tsc has some
> limitations
> > and try to mitigate them by filtering discontinuities, using
> > rdtscp, etc. They're best-effort, but they inherently
> > lack enough
> > information to do a complete job (they have to guess at where
> > power transitions occured, etc).
> > 3. New apps which know about modern processor capabilities, and
> > attempt to rely on constant_tsc forgoing all the best-effort
> > filtering, etc
> > 4. Apps which use gettimeofday() and/or clock_gettime()
> > for all time
> > measurement. They're guaranteed to get consistent
> time results,
> > perhaps at the cost of a syscall. On systems which
> support it,
> > they'll get vsyscall implementations which avoid the
> > syscall while
> > still using the best-possible clocksource. Even if
> they don't a
> > syscall will outperform an emulated rdtsc.
> >
> > Class 1 apps are just broken. We can try to emulate a UP, no-PM
> > processor for them, and that's probably best done in an HVM domain.
> > There's no need to go to extraordinary efforts for them because the
> > native hardware certainly won't.
> >
> > Class 2 apps will work as well as ever in a Xen PV domain as-is. If
> > they use rdtscp then they will be able to correlate the tsc to the
> > underlying pcpu and manage consistency that way. If they pin
> > threads to
> > VCPUs, then they may also requre VCPUs to be pinned to PCPUs. But
> > there's no need to make deep changes to Xen's tsc handling to
> > accommodate them.
> >
> > Class 3 apps will get a bit of a rude surprise in a PV Xen
> > domain. But
> > they're also new enough to use another mechanism to get
> time. They're
> > new enough to "know" that gettimeofday can be very efficient,
> > and should
> > not be going down the rathole of using rdtsc directly. And unless
> > they're going to be restricted to a very narrow class of
> machines (for
> > example, not my relatively new Core2 laptop which stops the
> "constant"
> > tsc in deep sleep modes), they need to fall back to being a
> > class 2 or 4
> > app anyway.
> >
> > Class 4 apps are not well-served under Xen. I think the vsyscall
> > mechanism will be disabled and they'll always end up doing a real
> > syscall. However, I think it would be relatively easy to add a new
> > vgettimeofday implementation which directly uses the
> pvclock mechanism
> > from usermode (the same code would work equally well for Xen
> > and KVM).
> > There's no need to add a new usermode ABI to get quick, high-quality
> > time in usermode. Performance-wise it would be more or less
> > indistinguishable from using a raw rdtsc, but it has the benefit of
> > getting full cooperation from the kernel and Xen, and can take into
> > account all tsc variations (if any).
> >
> >
> > So if you want to address these problems, it seems to me
> > you'll get most
> > bang for the buck by fixing (v)gettimeofday to use pvclock, and
> > convincing app writers to trust in gettimeofday.
> >
> > J
> >
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home