This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] XEN 4.0 + pvops kernel : system crashes on sta

To: John McCullough <jmccullo@xxxxxxxxxxx>
Subject: Re: [Xen-devel] XEN 4.0 + pvops kernel : system crashes on starting 155th domU
From: Jeremy Fitzhardinge <jeremy@xxxxxxxx>
Date: Wed, 28 Apr 2010 11:13:32 -0700
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>, Yuvraj Agarwal <yuvraj@xxxxxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Delivery-date: Wed, 28 Apr 2010 11:14:22 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4BD7DA02.3030107@xxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <C7FD6DD5.10D5B%keir.fraser@xxxxxxxxxxxxx> <4BD7DA02.3030107@xxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20100330 Fedora/3.0.4-1.fc12 Lightning/1.0b2pre Thunderbird/3.0.4
On 04/27/2010 11:47 PM, John McCullough wrote:
> I did a little testing.
> With no kernel option:
> # dmesg | grep -i nr_irqs
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] NR_IRQS:4352 nr_irqs:256
> w/nr_irqs=65536:
> # dmesg | grep -i nr_irqs
> [    0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0
> nr_irqs=65536
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] Kernel command line: root=/dev/sda1 ro quiet
> console=hvc0 nr_irqs=65536
> [    0.000000] NR_IRQS:4352 nr_irqs:256
> tweaking the NR_IRQS macro in the kernel will change the NR_IRQS
> output, but unfortunately that doesn't change nr_irqs and I run into
> the same limit (36 domus on a less-beefy dual core machine).

Yes, NR_IRQS is the hard limit (for any statically defined irq arrays,
which are deprecated now), but nr_irqs is the amount it decides to
actually allocate for dynamic irq arrays, and so represents the actual
runtime limit.

nr_irqs is computed in arch_probe_nr_irqs(), and its a function of the
number of cpus, with a bump to deal with dynamically allocated MSI
interrupts.  I should probably add something to specifically add more if
we're running under Xen, at least as a workaround (ultimately the plan
is to make all irqs completely dynamically allocated so there is no hard

> I did find this:
> http://blogs.sun.com/fvdl/entry/a_million_vms
> which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops
> kernel.

I'm pretty sure that's referring to Solaris dom0, so the fact that
there's a similarly named symbol is coincidence.  (But the root problem
is the same.)


Xen-devel mailing list