On 09/09/2010 10:50 AM, Dante Cinco wrote:
> I'm using Xen 4.0.1 in a dual-socket Nehalem platform (HP Proliant
> DL380G6 with Xeon E5540). I'm also using the same pvops kernel
> (2.6.32..21-1) for both dom0 and domU.
>
> Both Xen and dom0 sees and reads the SRAT table and correctly maps the
> 16 CPUs/APICs evenly between Node0 and Node1 but they differ in the
> NUMA hash shift value (17 for Xen, 29 for dom0).
>
> domU on the other hand, reports "No NUMA configuration found" as well
> as "ACPI in unprivileged domain disabled" and "APIC: disable apic
> facility."
>
> Is it always the case that no NUMA setting can be passed to domU?
Dulloor has been working on this, but I'm not sure what the current
state is. In general PV domains don't have access to NUMA topology
info, including dom0. While dom0 can parse the SRAT tables, I don't
think its doing anything useful with the information, as its VCPUs don't
have any fixed mapping to the underlying physical CPUs.
J
> Thanks in advance.
>
> Dante
>
> ---------------------------------------------------------------------------------------------------------
>
> dom0:~# xm list
> Name ID Mem VCPUs State
> Time(s)
> Domain-0 0 7735 16 r----- 475.3
> domU 4 3072 16 -b----
> 45.3
>
> dom0:~# xm vcpu-list
> Name ID VCPU CPU State Time(s) CPU
> Affinity
> Domain-0 0 0 0 -b- 64.4 0
> Domain-0 0 1 1 -b- 30.7 1
> Domain-0 0 2 2 -b- 28.0 2
> Domain-0 0 3 3 -b- 23.9 3
> Domain-0 0 4 4 -b- 17.6 4
> Domain-0 0 5 5 -b- 62.9 5
> Domain-0 0 6 6 -b- 14.4 6
> Domain-0 0 7 7 -b- 22.0 7
> Domain-0 0 8 8 -b- 28.0 8
> Domain-0 0 9 9 -b- 18.2 9
> Domain-0 0 10 10 r-- 18.0 10
> Domain-0 0 11 11 -b- 19.7 11
> Domain-0 0 12 12 -b- 49.2 12
> Domain-0 0 13 13 -b- 25.7 13
> Domain-0 0 14 14 -b- 18.5 14
> Domain-0 0 15 15 -b- 26.7 15
> domU 4 0 0 -b- 14.5 0
> domU 4 1 1 -b- 3.8 1
> domU 4 2 2 -b- 1.9 2
> domU 4 3 3 -b- 1.5 3
> domU 4 4 4 -b- 1.2 4
> domU 4 5 5 -b- 1.3 5
> domU 4 6 6 -b- 1.0 6
> domU 4 7 7 -b- 1.5 7
> domU 4 8 8 -b- 1.9 8
> domU 4 9 9 -b- 1.1 9
> domU 4 10 10 -b- 2.9 10
> domU 4 11 11 -b- 1.1 11
> domU 4 12 12 -b- 1.5 12
> domU 4 13 13 -b- 1.7 13
> domU 4 14 14 -b- 1.7 14
> domU 4 15 15 -b- 1.9 15
> :~# xm debug-keys u
> (XEN) 'u' pressed -> dumping numa info (now-0x9B0:55D091EA)
> (XEN) idx0 -> NODE0 start->0 size->1703936
> (XEN) phys_to_nid(0000000000001000) -> 0 should be 0
> (XEN) idx1 -> NODE1 start->1703936 size->1572863
> (XEN) phys_to_nid(00000001a0001000) -> 1 should be 1
> (XEN) CPU0 -> NODE0
> (XEN) CPU1 -> NODE1
> (XEN) CPU2 -> NODE0
> (XEN) CPU3 -> NODE1
> (XEN) CPU4 -> NODE0
> (XEN) CPU5 -> NODE1
> (XEN) CPU6 -> NODE0
> (XEN) CPU7 -> NODE1
> (XEN) CPU8 -> NODE0
> (XEN) CPU9 -> NODE1
> (XEN) CPU10 -> NODE0
> (XEN) CPU11 -> NODE1
> (XEN) CPU12 -> NODE0
> (XEN) CPU13 -> NODE1
> (XEN) CPU14 -> NODE0
> (XEN) CPU15 -> NODE1
> (XEN) Memory location of each domain:
> (XEN) Domain 0 (total: 1868128):
> (XEN) Node 0: 1065165
> (XEN) Node 1: 802963
> (XEN) Domain 4 (total: 781632):
> (XEN) Node 0: 241482
> (XEN) Node 1: 540150
>
>
> domU /var/log/kern.log:
>
>
> [ 0.000000] ACPI in unprivileged domain disabled
> [ 0.000000] released 0 pages of unused memory
> [ 0.000000] BIOS-provided physical RAM map:
> [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
> [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
> [ 0.000000] Xen: 0000000000100000 - 0000000080000000 (usable)
> [ 0.000000] DMI not present or invalid.
> [ 0.000000] last_pfn = 0x80000 max_arch_pfn = 0x400000000
> [ 0.000000] init_memory_mapping: 0000000000000000-0000000080000000
> [ 0.000000] RAMDISK: 02864000 - 0d886000
> [ 0.000000] No NUMA configuration found
> [ 0.000000] Faking a node at 0000000000000000-0000000080000000
> [ 0.000000] Bootmem setup node 0 0000000000000000-0000000080000000
> [ 0.000000] NODE_DATA [0000000000001000 - 0000000000004fff]
> [ 0.000000] bootmap [0000000000008000 - 0000000000017fff] pages 10
> [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 0080000000]
> [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==>
> [0000000000 - 0000001000]
> [ 0.000000] #1 [000dc89000 - 000dcfc000] XEN PAGETABLES ==>
> [000dc89000 - 000dcfc000]
> [ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==>
> [0000006000 - 0000008000]
> [ 0.000000] #3 [0001000000 - 000273a318] TEXT DATA BSS ==>
> [0001000000 - 000273a318]
> [ 0.000000] #4 [0002864000 - 000d886000] RAMDISK ==>
> [0002864000 - 000d886000]
> [ 0.000000] #5 [000d886000 - 000dc89000] XEN START INFO ==>
> [000d886000 - 000dc89000]
> [ 0.000000] #6 [000273b000 - 0002746000] BRK ==>
> [000273b000 - 0002746000]
> [ 0.000000] #7 [0000100000 - 000048d000] PGTABLE ==>
> [0000100000 - 000048d000]
> [ 0.000000] Zone PFN ranges:
> [ 0.000000] DMA 0x00000000 -> 0x00001000
> [ 0.000000] DMA32 0x00001000 -> 0x00100000
> [ 0.000000] Normal 0x00100000 -> 0x00100000
> [ 0.000000] Movable zone start PFN for each node
> [ 0.000000] early_node_map[2] active PFN ranges
> [ 0.000000] 0: 0x00000000 -> 0x000000a0
> [ 0.000000] 0: 0x00000100 -> 0x00080000
> [ 0.000000] SMP: Allowing 16 CPUs, 0 hotplug CPUs
> [ 0.000000] No local APIC present
> [ 0.000000] APIC: disable apic facility
> [ 0.000000] PM: Registered nosave memory: 00000000000a0000 -
> 0000000000100000
> [ 0.000000] Allocating PCI resources starting at 80000000 (gap:
> 80000000:80000000)
> [ 0.000000] Booting paravirtualized kernel on Xen
> [ 0.000000] Xen version: 4.0.1 (preserve-AD)
> [ 0.000000] NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:16 nr_node_ids:1
> [ 0.000000] PERCPU: Embedded 478 pages/cpu @ffff8800113fb000
> s1927064 r8192 d22632 u1957888
> [ 0.000000] pcpu-alloc: s1927064 r8192 d22632 u1957888 alloc=478*4096
> [ 0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05
> [0] 06 [0] 07
> [ 0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13
> [0] 14 [0] 15
> [ 0.000000] Xen: using vcpu_info placement
> [ 0.000000] Built 1 zonelists in Node order, mobility grouping on.
> Total pages: 509968
> [ 0.000000] Policy zone: DMA32
> [ 0.000000] Kernel command line:
> root=UUID=0ebd3fc5-623e-4991-83bd-374ef797c126 ro iommu=soft
> swiotlb=force noirqbalance console=hvc0,115200n8
> ip=:127.0.255.255::::eth0:dhcp
> [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
> [ 0.000000] Initializing CPU#0
> [ 0.000000] DMA: Placing 64MB software IO TLB between
> ffff8800131dc000 - ffff8800171dc000
> [ 0.000000] DMA: software IO TLB at phys 0x131dc000 - 0x171dc000
> [ 0.000000] xen_swiotlb_fixup: buf=ffff8800131dc000 size=67108864
> [ 0.000000] xen_swiotlb_fixup: buf=ffff88001723c000 size=32768
> [ 0.000000] Memory: 1733712k/2097152k available (6128k kernel code,
> 384k absent, 363056k reserved, 3751k data, 2620k init)
> [ 0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0,
> CPUs=16, Nodes=1
> [ 0.000000] Hierarchical RCU implementation.
> [ 0.000000] NR_IRQS:4352 nr_irqs:656
> [ 0.000000] Console: colour dummy device 80x25
> [ 0.000000] console [tty0] enabled
> [ 0.000000] console [hvc0] enabled
> [ 0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat,
> Inc., Ingo Molnar
> [ 0.000000] ... MAX_LOCKDEP_SUBCLASSES: 8
> [ 0.000000] ... MAX_LOCK_DEPTH: 48
> [ 0.000000] ... MAX_LOCKDEP_KEYS: 8191
> [ 0.000000] ... CLASSHASH_SIZE: 4096
> [ 0.000000] ... MAX_LOCKDEP_ENTRIES: 16384
> [ 0.000000] ... MAX_LOCKDEP_CHAINS: 32768
> [ 0.000000] ... CHAINHASH_SIZE: 16384
> [ 0.000000] memory used by lock dependency info: 6367 kB
> [ 0.000000] per task-struct memory footprint: 2688 bytes
> [ 0.000000] allocated 20971520 bytes of page_cgroup
> [ 0.000000] please try 'cgroup_disable=memory' option if you don't
> want memory cgroups
> [ 0.000000] installing Xen timer for CPU 0
> [ 0.000000] Detected 2533.492 MHz processor.
> [ 0.010000] Calibrating delay loop (skipped), value calculated
> using timer frequency.. 5066.98 BogoMIPS (lpj=25334920)
> [ 0.010000] Security Framework initialized
> [ 0.010000] SELinux: Disabled at boot.
> [ 0.010000] Dentry cache hash table entries: 262144 (order: 9, 2097152
> bytes)
> [ 0.010000] Inode-cache hash table entries: 131072 (order: 8, 1048576
> bytes)
> [ 0.010000] Mount-cache hash table entries: 256
> [ 0.010000] Initializing cgroup subsys ns
> [ 0.010000] Initializing cgroup subsys cpuacct
> [ 0.010000] Initializing cgroup subsys memory
> [ 0.010000] Initializing cgroup subsys devices
> [ 0.010000] Initializing cgroup subsys freezer
> [ 0.010000] Initializing cgroup subsys net_cls
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 0/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.010000] Performance Events: unsupported p6 CPU model 26 no PMU
> driver, software events only.
> [ 0.010000] lockdep: fixing up alternatives.
> [ 0.010000] SMP alternatives: switching to UP code
> [ 0.026210] ftrace: converting mcount calls to 0f 1f 44 00 00
> [ 0.026221] ftrace: allocating 24205 entries in 95 pages
> [ 0.031813] installing Xen timer for CPU 1
> [ 0.031917] lockdep: fixing up alternatives.
> [ 0.031930] SMP alternatives: switching to SMP code
> [ 0.010000] Initializing CPU#1
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 1/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.050503] installing Xen timer for CPU 2
> [ 0.010000] Initializing CPU#2
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 2/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.051023] installing Xen timer for CPU 3
> [ 0.010000] Initializing CPU#3
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 3/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.060556] installing Xen timer for CPU 4
> [ 0.010000] Initializing CPU#4
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 4/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.062137] installing Xen timer for CPU 5
> [ 0.010000] Initializing CPU#5
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 5/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.062703] installing Xen timer for CPU 6
> [ 0.010000] Initializing CPU#6
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 6/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.063209] installing Xen timer for CPU 7
> [ 0.010000] Initializing CPU#7
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 7/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.063772] installing Xen timer for CPU 8
> [ 0.010000] Initializing CPU#8
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 8/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.064329] installing Xen timer for CPU 9
> [ 0.010000] Initializing CPU#9
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 9/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.064882] installing Xen timer for CPU 10
> [ 0.010000] Initializing CPU#10
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 10/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.070506] installing Xen timer for CPU 11
> [ 0.010000] Initializing CPU#11
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 11/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.072826] installing Xen timer for CPU 12
> [ 0.010000] Initializing CPU#12
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 12/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.073344] installing Xen timer for CPU 13
> [ 0.010000] Initializing CPU#13
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 13/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.073893] installing Xen timer for CPU 14
> [ 0.010000] Initializing CPU#14
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 14/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.074429] installing Xen timer for CPU 15
> [ 0.010000] Initializing CPU#15
> [ 0.010000] CPU: L1 I cache: 32K, L1 D cache: 32K
> [ 0.010000] CPU: L2 cache: 256K
> [ 0.010000] CPU: L3 cache: 8192K
> [ 0.010000] CPU 15/0x4 -> Node 0
> [ 0.010000] CPU: Physical Processor ID: 0
> [ 0.010000] CPU: Processor Core ID: 2
> [ 0.074712] Brought up 16 CPUs
> [ 0.076826] devtmpfs: initialized
> [ 0.083171] Grant table initialized
> [ 0.083171] regulator: core version 0.5
> [ 0.103866] Time: 165:165:165 Date: 165/165/65
> [ 0.104178] NET: Registered protocol family 16
> [ 0.104741] PCI: setting up Xen PCI frontend stub
> [ 0.120177] bio: create slab <bio-0> at 0
> [ 0.270258] ACPI: Interpreter disabled.
> [ 0.271048] xen_balloon: Initialising balloon driver with page order 0.
> [ 0.272362] vgaarb: loaded
> [ 0.273229] SCSI subsystem initialized
> [ 0.273603] usbcore: registered new interface driver usbfs
> [ 0.273603] usbcore: registered new interface driver hub
> [ 0.273603] usbcore: registered new device driver usb
> [ 0.273603] PCI: System does not support PCI
> [ 0.274978] PCI: System does not support PCI
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home