[Xen-devel] Maximum number of domains and NR_IRQS

["Carb, Brian A" <Brian.Carb@xxxxxxxxxx>]

Running xen-unstable changeset 12734 on an ES7000 4 dual-core sockets (8cpu) 16GB memory, with kernel parameters dom0_mem=512M and xenheap_megabytes=64. DOMUs are paravirtualized sles10, 96MB, 4vcpus, each on a separate physical lun.

 

We successfully start 118 DOMUs, but when we try to start the 119th, the system panics with the following messages:

  Kernel panic - not syncing: No available IRQ to bind to: increase NR_IRQS!

  (XEN) Domain 0 crashed: rebooting machine in 5 seconds.

 

The documentation in include/asm-x86_64/irq.h suggests that the value of NR_IRQS under x86_64 is limited to 256. In fact, when we rebuilt xen-unstable with NR_IRQS set to 768, the kernel panics on boot (see below).

 

On the hunch that networking in a VM uses up an IRQ, we eliminated the 'vif' statement in each DOMU config file, and we were able to start 164 VMs in the 16GB before we exhausted memory.

 

Has anyone run into this IRQ issue? Is there any work-around?

brian carb
unisys corporation - malvern, pa
brian.carb@xxxxxxxxxx

--- trace output when trying to boot xen where NR_IRQs was set to 768 ---

(XEN) Initializing CPU#0
(XEN) Detected 3400.113 MHz processor.
(XEN) extable.c:77: Pre-exception: ffff8300001713fc -> 0000000000000000
(XEN) ----[ Xen-3.0-unstable  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e010:[<ffff8300001713fc>] get_cpu_vendor+0x2c/0x90
(XEN) RFLAGS: 0000000000010006   CONTEXT: hypervisor
(XEN) rax: 00008e00e0100000   rbx: ffff830000206840   rcx: 000000006c65746e
(XEN) rdx: 0000000049656e69   rsi: 0000000000000000   rdi: ffff830000187980
(XEN) rbp: 0000000000000000   rsp: ffff8300001bfe18   r8:  0000000000410000
(XEN) r9:  000000000000003a   r10: 00000000000000ff   r11: 0000000000000000
(XEN) r12: ffff830000206840   r13: ffff8300001879a8   r14: ffff830000187980
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000000b0
(XEN) cr3: 0000000000102000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e010
(XEN) Xen stack trace from rsp=ffff8300001bfe18:
(XEN)    0000000000001180 00000000756e6547 0000000000000000 ffff830000187980
(XEN)    ffff83000019b000 00000000003f7f0d ffff830000020980 ffff83000017168a
(XEN)    0000000000000000 ffff830000165a4c ffff830003ffc080 ffff830000187980
(XEN)    0000000000003ce5 ffff830000171948 ffff83000019b000 ffff830003ffc080
(XEN)    000000000000000a ffff83000016de41 0000000000000000 ffff83000002bbc0
(XEN)    0000000800000000 000000010000006e 0000000000000003 00000000000002f8
(XEN)    0000000000000000 000000000000000a 0000000000000000 0000000000000000
(XEN)    0000000000067f0c 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffff8300001001c1 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    ffff83000028c080
(XEN) Xen call trace:
(XEN)    [<ffff8300001713fc>] get_cpu_vendor+0x2c/0x90
(XEN)    [<ffff83000017168a>] generic_identify+0x4a/0x160
(XEN)    [<ffff830000165a4c>] subarch_init_memory+0xac/0xe0
(XEN)    [<ffff830000171948>] identify_cpu+0x78/0x1f0
(XEN)    [<ffff83000016de41>] __start_xen+0x861/0xc90
(XEN)    [<ffff8300001001c1>] __high_start+0x94/0x96
(XEN)   
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL TRAP: vector = 13 (general protection fault)
(XEN) [error_code=0000] , IN INTERRUPT CONTEXT
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) machine_crash_shutdown: 0
(XEN) extable.c:77: Pre-exception: ffff830000139e9c -> 0000000000000000
(XEN) ----[ Xen-3.0-unstable  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e010:[<ffff830000139e9c>] machine_crash_shutdown+0x7c/0xf0
(XEN) RFLAGS: 0000000000010047   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 00000000000003e8   rcx: 0000ffff0000ffff
(XEN) rdx: 0000000000000001   rsi: 0000000000000400   rdi: 0000000000000000
(XEN) rbp: 0000000000000046   rsp: ffff8300001bfbc8   r8:  ffff8300000b8000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: ffff83000011c7a0
(XEN) r12: ffff830000206840   r13: ffff8300001879a8   r14: ffff830000187980
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000000b0
(XEN) cr3: 0000000000102000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e010
(XEN) Xen stack trace from rsp=ffff8300001bfbc8:
(XEN)    0000000000000000 0000000000000046 ffff830000206840 ffff83000010e18f
(XEN)    ffff830000206840 ffff83000011c24c 5254204c41544146 74636576203a5041
(XEN)    203331203d20726f 6c6172656e656728 746365746f727020 6c756166206e6f69
(XEN)    6f7272655b0a2974 303d65646f635f72 49202c205d303030 525245544e49204e
(XEN)    544e4f4320545055 000000000a545845 0000003000000008 ffff8300001bfd48
(XEN)    ffff8300001bfc78 ffff8300001d7321 0000003000000028 ffff8300001bfd68
(XEN)    ffff8300001bfc98 ffff8300000b8000 0000000000000002 000000000000000d
(XEN)    ffff83000017c2aa 0000000000000000 ffff83000017c162 0000000000000000
(XEN)    0000000000000000 ffff8300001d7356 000000008005003b ffff83000011cf7b
(XEN)    0000000000000000 0000000000102000 0000000000000096 0000000000000094
(XEN)    ffff8300001001c1 ffff8300001001c1 ffff8300001bff20 ffff8300001330a4
(XEN)    ffff830000206840 000000000000000d ffff8300001bfd68 ffff8300001334d7
(XEN)    ffff830000206840 ffff830000206840 0000000000000000 ffff8300001656c2
(XEN)    0000000000000000 ffff830000187980 ffff8300001879a8 ffff830000206840
(XEN)    0000000000000000 ffff830000206840 0000000000000000 00000000000000ff
(XEN)    000000000000003a 0000000000410000 00008e00e0100000 000000006c65746e
(XEN)    0000000049656e69 0000000000000000 ffff830000187980 0000000d00000000
(XEN)    ffff8300001713fc 000000000000e010 0000000000010006 ffff8300001bfe18
(XEN)    0000000000000000 0000000000000001 0000000000001180 00000000756e6547
(XEN)    0000000000000000 ffff830000187980 ffff83000019b000 00000000003f7f0d
(XEN) Xen call trace:
(XEN)    [<ffff830000139e9c>] machine_crash_shutdown+0x7c/0xf0
(XEN)    [<ffff83000010e18f>] machine_crash_kexec+0x2f/0x90
(XEN)    [<ffff83000011c24c>] panic+0x15c/0x1b0
(XEN)    [<ffff83000011cf7b>] __serial_putc+0xdb/0x100
(XEN)    [<ffff8300001001c1>] __high_start+0x94/0x96
(XEN)    [<ffff8300001001c1>] __high_start+0x94/0x96
(XEN)    [<ffff8300001330a4>] show_trace+0x54/0xa0
(XEN)    [<ffff8300001334d7>] fatal_trap+0x77/0xb0
(XEN)    [<ffff8300001656c2>] FATAL_exception_with_ints_disabled+0xc/0x1a
(XEN)    [<ffff8300001713fc>] get_cpu_vendor+0x2c/0x90
(XEN)    [<ffff83000017168a>] generic_identify+0x4a/0x160
(XEN)    [<ffff830000165a4c>] subarch_init_memory+0xac/0xe0
(XEN)    [<ffff830000171948>] identify_cpu+0x78/0x1f0
(XEN)    [<ffff83000016de41>] __start_xen+0x861/0xc90
(XEN)    [<ffff8300001001c1>] __high_start+0x94/0x96
(XEN)   
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL TRAP: vector = 13 (general protection fault)
(XEN) [error_code=0000] , IN INTERRUPT CONTEXT
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel