WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] Poor HVM performance with 8 vcpus

To: Gianluca Guida <gianluca.guida@xxxxxxxxxxxxx>
Subject: Re: [Xen-devel] Poor HVM performance with 8 vcpus
From: Juergen Gross <juergen.gross@xxxxxxxxxxxxxx>
Date: Wed, 14 Oct 2009 12:44:35 +0200
Cc: Tim Deegan <Tim.Deegan@xxxxxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Delivery-date: Wed, 14 Oct 2009 03:45:00 -0700
Dkim-signature: v=1; a=rsa-sha256; c=simple/simple; d=ts.fujitsu.com; i=juergen.gross@xxxxxxxxxxxxxx; q=dns/txt; s=s1536b; t=1255517225; x=1287053225; h=from:sender:reply-to:subject:date:message-id:to:cc: mime-version:content-transfer-encoding:content-id: content-description:resent-date:resent-from:resent-sender: resent-to:resent-cc:resent-message-id:in-reply-to: references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:list-owner:list-archive; z=From:=20Juergen=20Gross=20<juergen.gross@xxxxxxxxxxxxxx> |Subject:=20Re:=20[Xen-devel]=20Poor=20HVM=20performance =20with=208=20vcpus|Date:=20Wed,=2014=20Oct=202009=2012:4 4:35=20+0200|Message-ID:=20<4AD5AB93.7050909@xxxxxxxxxxxx om>|To:=20Gianluca=20Guida=20<gianluca.guida@xxxxxxxxxxxx m>|CC:=20Keir=20Fraser=20<keir.fraser@xxxxxxxxxxxxx>,=20 =0D=0A=20Tim=20Deegan=20<Tim.Deegan@xxxxxxxxxxxxx>,=0D=0A =20"xen-devel@xxxxxxxxxxxxxxxxxxx"=20<xen-devel@xxxxxxxxx source.com>|MIME-Version:=201.0 |Content-Transfer-Encoding:=207bit|In-Reply-To:=20<f8877f 640910140316s7af2358bt2e34a90cfe3b48fe@xxxxxxxxxxxxxx> |References:=20<4AD588D9.4040104@xxxxxxxxxxxxxx>=09<C6FB4 BF5.1764F%keir.fraser@xxxxxxxxxxxxx>=20<f8877f64091014031 6s7af2358bt2e34a90cfe3b48fe@xxxxxxxxxxxxxx>; bh=WlQi/ycKx6/Wc7vuXEzBGN+t2wlleND2MeIrS+nqwtY=; b=Y8BDqW4ReRW9jqJUwXoLLCfkEtV2oH7VtDw4uF8msBN+OKfeuGjRnQbG NkBmJ3d2mwsbGayWyCVxbLoHHYbWJjGXAzYUMv+BdRNhV82/Cjl/szT58 CUFMbQGKUgjSnrP4FdcehQHIHROdxVblK0QJ9+aOzEmuF0N/E3ipvTuxA xFQUKoM6QBGqhi4exiH9w5N4gZqut99m256cJz8uX16L0teDTc4m6/Qey DLCQZJ6kiKkvM/XOfz6MC+AnDIbgi;
Domainkey-signature: s=s1536a; d=ts.fujitsu.com; c=nofws; q=dns; h=X-SBRSScore:X-IronPort-AV:Received:X-IronPort-AV: Received:Received:Message-ID:Date:From:Organization: User-Agent:MIME-Version:To:CC:Subject:References: In-Reply-To:X-Enigmail-Version:Content-Type: Content-Transfer-Encoding; b=wCDyI+yD0CTx14n/2mbGWO2KMXqUuVGoQfvn6vPZIuc3/8RTRsTQ7aH9 0+gUGipcVb1YdIrGAqv2IjF1bDOfvRaXJnUklmRgiIL2lQ6RrWViSbmsC hOaZKMIF3X5pIk/qvwOPjaEkFjv/KGlC8N+vTNLWblxCE44mfF6q37vPL kS8tM8pWcSuRY3Kwyaj7wow1Wkj2vKbc8ZLcm76+prOXfukX2OTxI7oWC 5tPSIj6IiLUXo9MWProkMPY8ipy7X;
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <f8877f640910140316s7af2358bt2e34a90cfe3b48fe@xxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Organization: Fujitsu Technology Solutions
References: <4AD588D9.4040104@xxxxxxxxxxxxxx> <C6FB4BF5.1764F%keir.fraser@xxxxxxxxxxxxx> <f8877f640910140316s7af2358bt2e34a90cfe3b48fe@xxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090707)
Gianluca Guida wrote:
> Ah, those good old OOS talks. I fear I am going to fail on my attempt
> to be laconic.

:-)

> 
> On Wed, Oct 14, 2009 at 10:35 AM, Keir Fraser <keir.fraser@xxxxxxxxxxxxx> 
> wrote:
>> On 14/10/2009 09:16, "Juergen Gross" <juergen.gross@xxxxxxxxxxxxxx> wrote:
>>
>>> as the performance of BS2000 seems to be hit by OOS optimization, I'm
>>> thinking of making a patch to disable this feature by a domain parameter.
>>>
>>> Is there a way to do this without having to change all places where the
>>> #if statements are placed?
>>> I think there should be some central routines where adding an "if" could
>>> be enough (setting oos_active to 0 seems not to be enough, I fear).
>>>
>>> Do you have any hint?
>> How about disabling it for domains with more than four VCPUs? Have you
>> measured performance with OOS for 1-4 VCPU guests? This is perhaps not
>> something that needs to be baked into guest configs.
> 
> In general, shadow code loses performances as the vcpus increase (>=4)
> because of the single shadow lock (and getting rid of the shadow lock,
> i.e. having per-vcpu shadows wouldn't help, since it would make much
> slower the most common operation, that is removing writable access of
> guest pages).
> But the two algorithms (always in-sync vs. OOS) will show their
> performance penalties in two different areas: in a scenario where
> guests do lot of PTE writes (read Windows in most of its operations)
> the in-sync approach will be more penalizing, because emulation is
> slow and needs the shadow lock, while scenarios were guests tend to
> have many dirty CR3 switches (that is CR3 switches with freshly
> written PTEs, as in the case with Juergen benchmark and the famous
> Windows parallel ddk build) will be penalized more by the OOS
> algorithm.
> 
> Disabling OOS for domains more than 4 vcpus might be a good idea, but
> not necessarily optimal. Furthermore, I always understood that a good
> practice for VM performance is to have many small VMs instead of a VM
> eating all of the host's CPUs, at least when shadow code is on. With
> big VMs, EPT/NPT has always been the best approach, since even with
> lot of TLB misses, the system was definitely lock-free in most of the
> VM's life.
> 
> Creating a per-domain switch should be a good idea, but a more generic
> (and correct) approach would be to have a dynamic policy for OOSing
> pages, in which we would stop putting OOS pages when we realize that
> we are resynch'ing too many pages in CR3 switches. This was taken in
> consideration during the development of the OOS, but it was finally
> discarded because performance were decent and big VMs were not in the
> interest range.
> 
> Yes, definitely away from spartan wit. But I hope this clarifies the issue.

I really does.

I think I'll start with a per-domain switch and leave the generic approach
to the specialists. ;-)

If, however, Keir rejects such a switch, I could try the generic solution,
but I think this solution would need very much work to find the correct
parameters.


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 636 47950
Fujitsu Technolgy Solutions               e-mail: juergen.gross@xxxxxxxxxxxxxx
Otto-Hahn-Ring 6                        Internet: ts.fujitsu.com
D-81739 Muenchen                 Company details: ts.fujitsu.com/imprint.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel