WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] NUMA problem with AMD G34 system

To: "marcotte@xxxxxxxxx" <marcotte@xxxxxxxxx>
Subject: Re: [Xen-devel] NUMA problem with AMD G34 system
From: Andre Przywara <andre.przywara@xxxxxxx>
Date: Mon, 21 Feb 2011 09:48:48 +0100
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Mon, 21 Feb 2011 00:52:50 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20110219085932.E08B624258@xxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20110219085932.E08B624258@xxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 2.0.0.23 (X11/20090820)
Brian Marcotte wrote:
I'm having a problem with NUMA on an AMD G34 system. Xen seems to think
that out of the 16 total cores, there are 12 on node0 and 4 on node3.

The box is:

    SuperMicro H8DGU (dual socket G34)
    Pair of AMD 6128 CPUs (8 core each)
    32 GB of RAM.
    BIOS has been upgraded three times - no help.

    xen-unstable from Feb 12 (same problem on 3.4.x).

I'm including the output from the "u" command below to show Xen's idea
of the CPU->node mapping. Complete boot log is here:

    http://www.panix.com/~marcotte/g34/boot.txt

Any help would be appreciated.

Have you checked that you populated the DIMM slots properly?
The SRAT CPU table seems sane, but the memory table seems to say that there is no memory on node 1 and 3, thus Xen will simply put those CPUs to node 0 as it cannot cope with memory-less nodes. If I understand the manual right, you first have to populate all the blue slots, then the black ones.

Hope that helps,
Andre.



Thanks.

- Brian

(XEN) Xen version 4.1.0-rc5-pre (root@xxxxxxxxxxxxxx) (gcc version 4.3.2 
(Debian 4.3.2-1.1) ) Sat Feb 12 18:53:49 EST 2011
(XEN) Command line: dom0_mem=600M com1=57600,8n1 console=com1 no-mce 
loglvl=debug

(XEN) 'u' pressed -> dumping numa info (now-0x2:5A61B881)
(XEN) idx0 -> NODE0 start->0 size->4325376
(XEN) phys_to_nid(0000000000001000) -> 0 should be 0
(XEN) idx3 -> NODE3 start->4325376 size->4194304
(XEN) phys_to_nid(0000000420001000) -> 3 should be 3
(XEN) CPU0 -> NODE0
(XEN) CPU1 -> NODE0
(XEN) CPU2 -> NODE0
(XEN) CPU3 -> NODE0
(XEN) CPU4 -> NODE0
(XEN) CPU5 -> NODE0
(XEN) CPU6 -> NODE0
(XEN) CPU7 -> NODE0
(XEN) CPU8 -> NODE3
(XEN) CPU9 -> NODE3
(XEN) CPU10 -> NODE3
(XEN) CPU11 -> NODE3
(XEN) CPU12 -> NODE0
(XEN) CPU13 -> NODE0
(XEN) CPU14 -> NODE0
(XEN) CPU15 -> NODE0
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 153600):
(XEN)     Node 0: 132232
(XEN)     Node 3: 21368

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


--
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448-3567-12


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel