This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] [PATCH] nestedhvm: ASID emulation

To: Keir Fraser <keir.xen@xxxxxxxxx>
Subject: Re: [Xen-devel] [PATCH] nestedhvm: ASID emulation
From: Christoph Egger <Christoph.Egger@xxxxxxx>
Date: Thu, 14 Apr 2011 11:26:05 +0200
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Thu, 14 Apr 2011 02:37:28 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <C9CB8C56.164B6%keir.xen@xxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <C9CB8C56.164B6%keir.xen@xxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; NetBSD amd64; en-US; rv: Gecko/20110202 Lightning/1.0b3pre Thunderbird/3.1.7
On 04/13/11 18:22, Keir Fraser wrote:
On 13/04/2011 16:19, "Christoph Egger"<Christoph.Egger@xxxxxxx>  wrote:

On 04/13/11 17:05, Keir Fraser wrote:
On 13/04/2011 15:26, "Christoph Egger"<Christoph.Egger@xxxxxxx>   wrote:

Is this measurable on a macro benchmark?

I measured this with xentrace while l2 guest is booting.

The speedup is noticable on the end-user side just by the feeling on how
fast the l2 guest reacts on user input.

Yikes. Does nestedhvm suck really badly then? I can't believe this patch
alone gets you from sucky to good performance. Is it an improvement from
sucky to not-quite-as-sucky?

The very first patch series I sent to xen-devel, the cost of a
VMRUN emulation was about 25000 cycles with Xen-on-Xen while hvmloader
was coming up as l2 guest. With every new patch series there were also
performance optimizations done. Now the cost of a VMRUN
emulation is about 11000 cycles and with this ASID emulation patch
the cost of a VMRUN emulation goes down to about 10000 cycles.

There is still room for more performance, I am working on.

A cost of 15000 cycles for VMRUN emulation is fast enough that Solitair
in XP mode of Windows 7 guest recognizes a double click ;-)

Solitair in XP mode runs faster than Solitair in Windows 7 ;-)

I mean this looks like a micro-optimisation on a feature that noone is going
to use for serious performance work anyway.

4. nestedhvm is enabled and we are going to run l2 guest

We run the l1 guest in the last call. The asid generation may have
changed by then. In this case the current nv_n2asid number is stale
and the value of data->next_asid is<= of nv->nv_n2asid.

How do you know for sure that next_asid will be<= nv_n2asid in this case?
What if other VCPUs have run meanwhile, and next_asid has been incremented
multiple times until it is greather than nv_n2asid?

In that case  curr->arch.hvm_vcpu.asid_generation is not equal to
data->core_asid_generation  and a new hw ASID number is assigned
regardless of the values of data->next_asid and nv_n2asid.

What if nv_n2asid wast last assigned on a previous generation, but nv_n1asid
was assigned from the current generation? Then you'd have next_asid>
nv_n2asid, but nv_n2asid is stale.

When the generation changes, next_asid is set back to 1 and the hw TLB flushed.

If next_asid is larger than nv_n2asid then it is reused and this is ok
since the hw TLB got flushed. It is just important that nv_n1asid never
gets the same hw ASID assigned and that is what the do { } while loop

The hardware doesn't enforce an incremental use of the hw ASID numbers.

The point of ASID emulation is to reduce the number of
TLB invalidations by switching forth and back between two
hw ASID numbers when switching between l1 and l2 guest.
This reduces pagetable walks of the hw MMU so much that the
performance win is noticable by the end-user.

Without this patch the ASID is invalidated on every VMRUN and VMEXIT emulation.

The Flush-by-ASID feature even let me reuse the same hw ASID number
when the l2 guest wants to flush its TLB.


  -- Keir


   -- Keir

The the value of nv->nv_n2asid is valid if l1 guest doesn't change
the virtual asid (= asid number in the virtual vmcb) and
data->next_asid is larger than nv->nv_n2asid. In this case
just reuse the same hw ASID that has been used from the last
VMRUN emulation.

5. nestedhvm is enabled and we are going to run l2 guest again

The same hw ASID should be reused unless the generation changed because
the nestedp2m got flushed or the vcpu moved to a different physical cpu,
for example.
But the hw ASID number may never match the hw ASID used to run the l1 guest.

In all cases we have to verify that the

I wouldn't bother fixing #2 unless there's a convincing answer for #1.

---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85689 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Xen-devel mailing list