WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on sta

To: "'Konrad Rzeszutek Wilk'" <konrad.wilk@xxxxxxxxxx>, "'Jeremy Fitzhardinge'" <jeremy@xxxxxxxx>
Subject: RE: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU
From: "Yuvraj Agarwal" <yuvraj@xxxxxxxxxxx>
Date: Thu, 29 Apr 2010 20:12:46 -0700 (PDT)
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, 'Keir Fraser' <keir.fraser@xxxxxxxxxxxxx>, 'John McCullough' <jmccullo@xxxxxxxxxxx>
Delivery-date: Thu, 29 Apr 2010 20:14:08 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20100429145603.GB28499@xxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <C7FD6DD5.10D5B%keir.fraser@xxxxxxxxxxxxx> <4BD7DA02.3030107@xxxxxxxxxxx> <20100428140437.GA29653@xxxxxxxxxxxxxxxxxxx> <006e01cae725$54b95a00$fe2c0e00$@ucsd.edu> <20100429145603.GB28499@xxxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AcrnrC9zRjMM1CCXRqyIHIu1F1LsRQAZQXxA
> [    0.000000] nr_irqs_gsi: 48
> [    0.500076] NR_IRQS:5120 nr_irqs:944

That looks to be different from the previous bootup:
[    0.000000] NR_IRQS:4352 nr_irqs:256

>> YA: I think you may have mixed up the post by John. He has a different 
>> dual core machine
>> and he is trying the same edits as me. He was getting nr_irqs as 256 
>> while I was getting 944.
>> [    0.000000] nr_irqs_gsi: 48
>> [    0.500270] NR_IRQS:5120 nr_irqs:944
>> Also it did not matter if I changed the VECTORS to 1024 instead of the 
>> original 256.

>
> Also, as earlier it crashes on the same number of domU (154). I didn’t
> mention earlier, this a dual core Nehalem machine  -- 2 (sockets) * 4 
> cores
> per CPU * 2 (hyperthreading)

Lots of logical CPUs, weird that your nr_irqs initially was that much lower.

>> No, as mentioned above, my machine has the larger number of CPUs while 
>> John McCullough
>> machine has lesser number of CPUs.

Anyhow, you mentioned that you narrowed it down to not being enough IRQs
- how did you find that out? Was there an kernel message when you
  started the 155th guest?

>>> yes, as I mentioned in an email, right after I started up the 154th 
>>> guest there was
>>> a log message printed out as below:
>>> Kernel Panic - not syncing: No available IRQs to bind to: increase 
>>> nr_irqs!

Oh, also you say that the /proc/interrupts showed the number descending
from 255 down to 89. With this it should have started at 944 and gone
down to 49? Which roughly means 175 guests?

>> Yes, the interrupts started counting down from 944 as I increased the 
>> number of domUs;
>> I am attaching the /proc/interrupts log file for three cases (no domU, 
>> after 1 domU,
>> and then after 150 domU).  The machine crashed after the 154th domU was 
>> started.

Thank you all for helping out to debug this. As I mentioned at the XEN 
summit, once I do fix these issues I’d really like to create  wiki stub for 
any one else trying to do this. It looks like in the original 2.6.18 kernel 
people did indeed try this, but I guess some of the edits/changes didn’t 
make it to the 2.6.31 pvops branch....

Attachment: interrupts-after1domU
Description: Binary data

Attachment: interrupts-after150domUs
Description: Binary data

Attachment: interrupts-beforestarting-domUs
Description: Binary data

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel