WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on sta

To: Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU
From: John McCullough <jmccullo@xxxxxxxxxxx>
Date: Tue, 27 Apr 2010 23:47:30 -0700
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Yuvraj Agarwal <yuvraj@xxxxxxxxxxx>
Delivery-date: Tue, 27 Apr 2010 23:47:32 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <C7FD6DD5.10D5B%keir.fraser@xxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <C7FD6DD5.10D5B%keir.fraser@xxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.9) Gecko/20100411 Icedove/3.0.4
I did a little testing.

With no kernel option:
# dmesg | grep -i nr_irqs
[    0.000000] nr_irqs_gsi: 88
[    0.000000] NR_IRQS:4352 nr_irqs:256

w/nr_irqs=65536:
# dmesg | grep -i nr_irqs
[ 0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0 nr_irqs=65536
[    0.000000] nr_irqs_gsi: 88
[ 0.000000] Kernel command line: root=/dev/sda1 ro quiet console=hvc0 nr_irqs=65536
[    0.000000] NR_IRQS:4352 nr_irqs:256

tweaking the NR_IRQS macro in the kernel will change the NR_IRQS output, but unfortunately that doesn't change nr_irqs and I run into the same limit (36 domus on a less-beefy dual core machine).

I did find this:
http://blogs.sun.com/fvdl/entry/a_million_vms
which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops kernel.

Watching /proc/interrupts, the domain irqs seem to be getting allocated from 248 downward until they hit some other limit:
...
 64:      59104  xen-pirq-ioapic-level  ioc0
 89:          1   xen-dyn-event     evtchn:xenconsoled
 90:          1   xen-dyn-event     evtchn:xenstored
 91:          6   xen-dyn-event     vif36.0
 92:        140   xen-dyn-event     blkif-backend
 93:         97   xen-dyn-event     evtchn:xenconsoled
 94:        139   xen-dyn-event     evtchn:xenstored
 95:          7   xen-dyn-event     vif35.0
 96:        301   xen-dyn-event     blkif-backend
 97:        261   xen-dyn-event     evtchn:xenconsoled
 98:        145   xen-dyn-event     evtchn:xenstored
 99:          7   xen-dyn-event     vif34.0
...
Perhaps the xen irqs are getting allocated out of the nr_irqs pool, while they could be allocated from the NR_IRQS pool?

-John




On 04/27/2010 08:45 PM, Keir Fraser wrote:
I think nr_irqs is specifiable on the command line on newer kernels. You may
be able to do nr_irqs=65536 as a kernel boot parameter, or something like
that, without needing to rebuild the kernel.

  -- Keir

On 28/04/2010 02:02, "Yuvraj Agarwal"<yuvraj@xxxxxxxxxxx>  wrote:

Actually, I did identify the problem (don’t know the fix) at least from
the console logs. Its related to running out of nr_irq's  (attached JPG
for the console log).


-----Original Message-----
From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
[mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Keir Fraser
Sent: Tuesday, April 27, 2010 5:44 PM
To: Yuvraj Agarwal; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
on starting 155th domU

On 27/04/2010 08:41, "Yuvraj Agarwal"<yuvraj@xxxxxxxxxxx>  wrote:

Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
but
as far as we can see we don¹t quite know what might be going causing the
system to crash (no console access anymore and system becomes
unresponsive and
needs to be power-cycled).  I have pasted only the relevant bits of
information (the last domU that did successfully start and the next one
that
failed). It may be the case that all the log messages weren¹t flushed
before
the system crashedŠ

Does anyone know where this limit of 155 domU is coming from and how we
can
fix/increase it?
Get a serial line on a test box, and capture Xen logging output on it. You
can both see if any crash messages come from Xen when the 155th domain is
created, and also try the serial debug keys (e.g., try 'h' to get help to
start with) to see whether Xen itself is still alive.

  -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>