This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] Xen 3.4.1 NUMA support

To: Andre Przywara <andre.przywara@xxxxxxx>
Subject: Re: [Xen-devel] Xen 3.4.1 NUMA support
From: Dulloor <dulloor@xxxxxxxxx>
Date: Tue, 10 Nov 2009 01:56:42 -0500
Cc: George Dunlap <george.dunlap@xxxxxxxxxxxxx>, Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Keir Fraser <Keir.Fraser@xxxxxxxxxxxxx>, Papagiannis Anastasios <apapag@xxxxxxxxxxxx>
Delivery-date: Mon, 09 Nov 2009 22:57:05 -0800
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=pj2EQvG3pHLLUQdn9gJZRhKGdPQvx4Ea3VqLzxZ2RdQ=; b=Ah/83DQgcnYbHpjXUsTHXYCYKV2Giz+cI/tqFEPNwxz68GA9ZkfdRSQh1o3dzWJ0+4 UbesGyqTgwDoWOGaSTha/wndEGNHEjSiU8ykZGVY2btbLsNPtQoxoGbnRR6Ebvm8Wv4j 3/XAvNIN/0PFDP5DzQ7KoESUm5tybfgp3BmAU=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=k4Io4R+54AQWX84tGtSKMUC5tcLPPjRCrgc1Bc/2nJj4o7Dc83OI/+9ukEqbKNpOBM RHGeQqR7FIITo0Ym6l6J9b11rZa1OaNkXuGxdxpjWKNACCfTTy3t6gXLxud9Hlw+lrk+ UhfYW1L9dCyGYg93wh03CxGevD1MqTSoWG8CQ=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4AF89D06.9010204@xxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <bd4f4a54-5269-42d8-b16d-cbdfaeeba361@default> <4AF82F12.6040400@xxxxxxx> <4AF82FD8.6020409@xxxxxxxxxxxxx> <4AF89D06.9010204@xxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
I am not finding this. Can you please point to the code ?

numa=on/off is only for setting up numa in xen (similar to the linux
knob, but turned off by default). The allocation of memory from a
single node (that you observe) could be because of the way
alloc_heap_pages is implemented (trying to allocate from all the heaps
from a node, before trying the next one) - try looking at dump_numa
output. And, affinities are not set anywhere based on the node from
which allocation happens.


On Mon, Nov 9, 2009 at 5:51 PM, Andre Przywara <andre.przywara@xxxxxxx> wrote:
> George Dunlap wrote:
>> Andre Przywara wrote:
>>> BTW: Shouldn't we set finally numa=on as the default value?
>> Is there any data to support the idea that this helps significantly on
>> common systems?
> I don't have any numbers handy, but I will try if I can generate some.
> Looking from a high level perspective it is a shame that it's not the
> default: With numa=off the Xen domain loader will allocate physical memory
> from some node (maybe even from several nodes) and will schedule the guest
> on some other (even rapidly changing) nodes. According to Murphy's law you
> will end up with _all_ the memory access of a guest to be remote. But in
> fact a NUMA architecture is really beneficial for virtualization: As there
> are close to zero cross domain memory accesses (except for Dom0), each node
> is more or less self contained and each guest can use the node's memory
> controller almost exclusively.
> But this is all spoiled as most people don't know about Xen's NUMA
> capabilities and don't set numa=on. Using this as a default would solve
> this.
> Regards,
> Andre.
> --
> Andre Przywara
> AMD-Operating System Research Center (OSRC), Dresden, Germany
> Tel: +49 351 488-3567-12
> ----to satisfy European Law for business letters:
> Advanced Micro Devices GmbH
> Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
> Geschaeftsfuehrer: Jochen Polster; Thomas M. McCoy; Giuliano Meroni
> Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
> Registergericht Muenchen, HRB Nr. 43632
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

Xen-devel mailing list