This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


[Xen-devel] [PATCH 0/6] add NUMA support to Xen

To: Ian Pratt <m+Ian.Pratt@xxxxxxxxxxxx>, Keir Fraser <Keir.Fraser@xxxxxxxxxxxx>
Subject: [Xen-devel] [PATCH 0/6] add NUMA support to Xen
From: Ryan Harper <ryanh@xxxxxxxxxx>
Date: Fri, 29 Sep 2006 13:58:27 -0500
Cc: "Subrahmanian, Raj" <raj.subrahmanian@xxxxxxxxxx>, xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Fri, 29 Sep 2006 11:59:01 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/1.5.6+20040907i
The following patchset adds NUMA support to the hypervisor.  This

 - A full SRAT table parser for 32-bit and 64-bit, based on the ACPI
   NUMA parser from linux with data structures to represent 
   NUMA cpu and memory topology, and NUMA emulation (fake=).
 - Changes to the Xen page allocator adding a per-node bucket for each
   zone.  Xen will continue to prioritize using the requested zone at
   the cost of using non-local memory.
 - Modified xen memory_ops to use NUMA-aware page allocator functions to
   help allocate node-local memory for guests
 - export NUMA topology through physinfo hypercall, integreated into
 - fix-up nr_cpus calculation now that nr_nodes is not static
 - Add a tool to explore NUMA allocations and Xen heap status.

Nothing has functionally changed since the last posting of the patches;
I've just been maintaining the patchset against unstable.

Additionally, I've spent the past few weeks collecting some performance
benchmarks comparing with and without the NUMA patches on a system
configured as a 4-socket server(no NUMA characteristics) and as a
multi-node 16-socket NUMA machine.

Machine: x460 4-node, 3Ghz Xeon PIV, 1MB L2, 8MB L3
         16 Socket (4 sockets per node) HT, logical 32-way,
         64GB RAM, 16G per node
Xen Changeset used: 11536:041be3f6b38e

We ran sdet and dbench, recording aggregate throughput for 4, 8, 12, and
16domUs.  This loads the system with 4domUs on each node, one node, two
nodes, etc up to 4 nodes pinning the guest VCPUs to the proper physical
cpus in their respective nodes.  

To look for possible regressions on non-NUMA systems, we used a single
node (4 sockets and 16G) for the first 4domU test as a single node does
not have any NUMA characteristics.  

In summary, there are no regressions for the single node, non-numa case.
As we scale the number of domains out the NUMA patches show significant

dbench(tmpfs)  4domU  8domU  12domU  16domU
with           590    1175   1750    2175
without        590    1090   1405    1450
%gain         +0.0%   +7.8%  +24.5%  +50.0%

sdet       4domU  8domU  12domU  16domU
with       14000  25900  39500   49000
without    13900  24950  34900   38500
%gain      +0.8%  +3.8%  +13.2%  +27.0%

I've also been working with Raj at Unisys to test out the NUMA patches
on their ES7000.  Just today Raj confirmed that the previous "dom0
slowdown" that had been reported has been reproduced without the NUMA
patches.  I believe there are no outstanding machines that have tested
the NUMA patches that have any issues.

I would like to see the NUMA patches included in xen-unstable once it
is open for wider testing prior to the next release after 3.0.3.

Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253

Xen-devel mailing list

<Prev in Thread] Current Thread [Next in Thread>
  • [Xen-devel] [PATCH 0/6] add NUMA support to Xen, Ryan Harper <=