Re: [Xen-devel] Poor HVM performance with 8 vcpus

To:	Gianluca Guida <gianluca.guida@xxxxxxxxxxxxx>
Subject:	Re: [Xen-devel] Poor HVM performance with 8 vcpus
From:	Juergen Gross <juergen.gross@xxxxxxxxxxxxxx>
Date:	Thu, 08 Oct 2009 09:10:55 +0200
Cc:	"xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Delivery-date:	Thu, 08 Oct 2009 00:11:21 -0700
Dkim-signature:	v=1; a=rsa-sha256; c=simple/simple; d=ts.fujitsu.com; i=juergen.gross@xxxxxxxxxxxxxx; q=dns/txt; s=s1536b; t=1254985918; x=1286521918; h=from:sender:reply-to:subject:date:message-id:to:cc: mime-version:content-transfer-encoding:content-id: content-description:resent-date:resent-from:resent-sender: resent-to:resent-cc:resent-message-id:in-reply-to: references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:list-owner:list-archive; z=From:=20Juergen=20Gross=20<juergen.gross@xxxxxxxxxxxxxx> \|Subject:=20Re:=20[Xen-devel]=20Poor=20HVM=20performance =20with=208=20vcpus\|Date:=20Thu,=2008=20Oct=202009=2009:1 0:55=20+0200\|Message-ID:=20<4ACD907F.7030505@xxxxxxxxxxxx om>\|To:=20Gianluca=20Guida=20<gianluca.guida@xxxxxxxxxxxx m>\|CC:=20"xen-devel@xxxxxxxxxxxxxxxxxxx"=20<xen-devel@lis ts.xensource.com>\|MIME-Version:=201.0 \|Content-Transfer-Encoding:=207bit\|In-Reply-To:=20<f8877f 640910070937w1f055672s1b2ec84f5e218b28@xxxxxxxxxxxxxx> \|References:=20<4ACC3B49.4060500@xxxxxxxxxxxxxx>=20<f8877 f640910070937w1f055672s1b2ec84f5e218b28@xxxxxxxxxxxxxx>; bh=y2UQcLrt12u74pDyIpmQbX/tpa95G3vj6O1xrQFDP6Q=; b=Gr8c8n16HOPYIFTtBqw7blYpXzm6hNLFVeQ1VxVAtkezILUt6dNBUtA4 jfUlTEJTMULDCBwBG0QYNsWYB0R/4K4lLB21I/YcNBaBB5ogxyqZ9IQf1 YneesQrsvrt79ZcfE/yq9UDrzcQK56CSgyj8/pDSDjsCXp/JDabiK6sFO KN5MNkoIv5966qF9Fv5RN/zqxD3PH3k+N1QhZt/K3JampopQ8M+fT7A9s +H/Ew2gH25Xc9l4sTkLye9rRdofP5;
Domainkey-signature:	s=s1536a; d=ts.fujitsu.com; c=nofws; q=dns; h=X-SBRSScore:X-IronPort-AV:Received:X-IronPort-AV: Received:Received:Message-ID:Date:From:Organization: User-Agent:MIME-Version:To:CC:Subject:References: In-Reply-To:X-Enigmail-Version:Content-Type: Content-Transfer-Encoding; b=V534WG72L4Iu3c1OBuEoYbNqijTARm95cEm/fPcPzjjGTWg4b8xz0QCb hgcFivmWXJNtbaxpm9aQmweNPr424HfX/Zp9YKI52iIizPtBPzICp3azS bDNPlG0oKYmdFHCDGMCViS0VE4dOc0AoLs7shQHTQkSHHS6NTo9RegSyQ R90OwKAzwbfEqIQkcWc4hHcnT+xAyZPGDTRvctwl26UoFOC5AE8P21zLi a8x1dIc+61IndY0jGzXOSmZFW7k5D;
Envelope-to:	www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to:	<f8877f640910070937w1f055672s1b2ec84f5e218b28@xxxxxxxxxxxxxx>
List-help:	<mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id:	Xen developer discussion <xen-devel.lists.xensource.com>
List-post:	<mailto:xen-devel@lists.xensource.com>
List-subscribe:	<http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe:	<http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Organization:	Fujitsu Technology Solutions
References:	<4ACC3B49.4060500@xxxxxxxxxxxxxx> <f8877f640910070937w1f055672s1b2ec84f5e218b28@xxxxxxxxxxxxxx>
Sender:	xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent:	Mozilla-Thunderbird 2.0.0.22 (X11/20090707)

Hi,

Gianluca Guida wrote:
> Hi,
> 
> On Wed, Oct 7, 2009 at 8:55 AM, Juergen Gross
> <juergen.gross@xxxxxxxxxxxxxx> wrote:
>> we've got massive performance problems running a 8 vcpu HVM-guest (BS2000)
>> under XEN (xen 3.3.1).
>>
>> With a specific benchmark producing a rather high load on memory management
>> operations (lots of process creation/deletion and memory allocation) the 8
>> vcpu performance was worse than the 4 vcpu performance. On other platforms
>> (/390, MIPS, SPARC) this benchmark scaled rather well with the number of 
>> cpus.
>>
>> The result of the usage of the software performance counters of XEN seemed
>> to point to the shadow lock being the reason. I modified the Hypervisor to
>> gather some lock statistics (patch will be sent soon) and found that the
>> shadow lock is really the bottleneck. On average 4 vcpus are waiting to get
>> the lock!
>>
>> Is this a known issue?
> 
> Acutally, I think so. The OOS optimization is widely known not to be
> too scalable at 8vcpus in the current state, since its weak point is
> the CR3 switching time increasing linearly with the number of cpus. If
> you have lot of processes switches together with lot of PTE writings
> (as it seems to be the case for your benchmark) then that's probably
> the cause.
> 
> Could you try disabling the OOS optimization from the
> SHADOW_OPTIMIZATIONS definition?

Great!
First performance data looks okay!
We will have to run different benchmarks in different configurations, but I
think you gave an excellent hint. :-)


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 636 47950
Fujitsu Technolgy Solutions               e-mail: juergen.gross@xxxxxxxxxxxxxx
Otto-Hahn-Ring 6                        Internet: ts.fujitsu.com
D-81739 Muenchen                 Company details: ts.fujitsu.com/imprint.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

WARNING - OLD ARCHIVES

xen-devel

Re: [Xen-devel] Poor HVM performance with 8 vcpus