This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] [RFC][PATCH] domheap optimization for NUMA

To: Andre Przywara <andre.przywara@xxxxxxx>
Subject: Re: [Xen-devel] [RFC][PATCH] domheap optimization for NUMA
From: Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Date: Thu, 03 Apr 2008 00:21:26 +0100
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, edwin.zhai@xxxxxxxxx
Delivery-date: Wed, 02 Apr 2008 16:22:23 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <47F40D8D.7060800@xxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AciVGEWhhBJ0iwELEd2TwgAWy6hiGQ==
Thread-topic: [Xen-devel] [RFC][PATCH] domheap optimization for NUMA
User-agent: Microsoft-Entourage/
On 2/4/08 23:49, "Andre Przywara" <andre.przywara@xxxxxxx> wrote:

>> Yes, but it's a bad interface, particularlty when the function is called
>> alloc_domheap_pages_on_node(). Pass in a nodeid. Write a helper function to
>> work out the nodeid from the domain*.
> I was just looking at this code, too, so I fixed this. Eventually
> alloc_heap_pages is called, which deals with nodes only, so I replaced
> cpu with node everywhere else, too. Now __alloc_domheap_pages and
> alloc_domheap_pages_on_node are almost the same (except parameter
> ordering), so I removed the first one, since the naming of the latter is
> better. Passing node numbers instead of cpu numbers needs cpu_to_node
> and asm/numa.h, if you think there is a better way, I am all ears.

That's fine. If you reference numa stuff then you need numa.h.

But vcpu_to_node and domain_to_node as well as cpu_to_node, please. There's
no need to be open-coding v->processor everywhere. Also in future we might
care to pick node based on v's affinity map rather than just current
processor value. And usage of d->vcpu[0] without checking for != NULL is
asking to introduce edge-case bugs. We can easily do that NULL check in one
place if we implement domain_to_node().

And, while I'm thinking about the interfaces, let's just stick to
alloc_domheap_page() and alloc_domheap_pages(). Let's add a flags parameter
to the former (so it matches the latter in that respect) and let's add a
MEMF_node() flag subtype (similar to MEMF_bits). Semantics will be that if
MEMF_node(node) is provided then we try to allocate memory from node; else
we try to allocate memory from a node local to specified domain; else if
domain is NULL then we ignore locality.

Since zero is probably a valid numa nodeid we can define MEMF_node() as
something like ((((node)+1)&0xff)<<8). Then since NUMA_NO_NODE==0xff
everything works nicely: MEMF_node(NUMA_NO_NODE) is equivalent to not
specifying MEMF_node() at all, which is what we would logically expect.
NUMA_NO_NODE probably needs to be pulled out of asm-x86/numa.h and made the
official arch-neutral way to specify 'don't care' for numa nodes.

 -- Keir

Xen-devel mailing list