WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] Xenlinux oops and crashes with nbd, Xen no longer boots

To: Wm <bill@xxxxxxxxxxxxxxx>
Subject: Re: [Xen-devel] Xenlinux oops and crashes with nbd, Xen no longer boots
From: Keir Fraser <Keir.Fraser@xxxxxxxxxxxx>
Date: Tue, 13 Jul 2004 08:12:26 +0100
Cc: xen-devel@xxxxxxxxxxxxxxxxxxxxx
Delivery-date: Tue, 13 Jul 2004 08:14:01 +0100
Envelope-to: steven.hand@xxxxxxxxxxxx
In-reply-to: Your message of "Mon, 12 Jul 2004 23:57:12 BST." <20040712225712.GA30324@xxxxxxxxxxx>
List-archive: <http://sourceforge.net/mailarchive/forum.php?forum=xen-devel>
List-help: <mailto:xen-devel-request@lists.sourceforge.net?subject=help>
List-id: List for Xen developers <xen-devel.lists.sourceforge.net>
List-post: <mailto:xen-devel@lists.sourceforge.net>
List-subscribe: <https://lists.sourceforge.net/lists/listinfo/xen-devel>, <mailto:xen-devel-request@lists.sourceforge.net?subject=subscribe>
List-unsubscribe: <https://lists.sourceforge.net/lists/listinfo/xen-devel>, <mailto:xen-devel-request@lists.sourceforge.net?subject=unsubscribe>
Sender: xen-devel-admin@xxxxxxxxxxxxxxxxxxxxx
Okay, detangling the backtrace....

 Entry EIP    Function                  Caller's return EIP
 ---------    --------                  -------------------
 0xc00c1220   alloc_skb(1656,GFP_NOIO)  0xc00ddd58
 0xc00dd4b0   tcp_sendmsg               0xc00fa9a1
 0xc00fa960   inet_sendmsg              0xc00bde23
 0xc00bddb0   sock_sendmsg              0xc3cad25f
 unknown      nbd_xmit                  0xc3cad511
 unknown      nbd_send_req              0xc3cad91c
 unknown      do_nbd_request            0xc008fc46
 0xc008fbe0   generic_unplug_device     0xc000c342
 0xc000c2e0   __run_task_queue          n/a (tail call)
 0xc00b8f20   io_schedule               0xc000c0a8
 0xc000c050   tasklet_action            0xc000be9e
 0xc000bdc0   do_softirq                0xc006f2c0
 0xc006f220   do_IRQ(133,-)             0xc007395e
 0xc00738c0   evtchn_do_upcall          0xc006dcd7
 0xc006dca4   hypervisor_callback       n/a (async callback)

I see the problem. Well hidden because the key function (io_schedule)
ends in a tail call. The problem is that we are scheduling I/O
requests in interrupt context rather than deferring to a process
context --- a known issue that hasn't bitten until now. I'll have to
think about a fix.

 -- Keir

> Down not across
> ksymoops 2.4.9 on i686 2.4.26-xen0.  Options used
>      -v ./vmlinux (specified)
>      -k /proc/ksyms (default)
>      -l /proc/modules (default)
>      -o /lib/modules/2.4.26-xen0/ (default)
>      -M (specified)
> 
> invalid operand: 0000
> CPU:    0
> EIP:    0819:[<c00c133b>]    Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00213286
> eax: 0000003a   ebx: c048f040   ecx: c0178000   edx: 00000000
> esi: 00000030   edi: c048f160   ebp: c0179cd0   esp: c0179cc0
> ds: 0821   es: 0821   ss: 0821
> Process swapper (pid: 0, stackpage=c0179000)<1>
> Stack: c012c920 c00ddd58 c048f040 c048f040 c0179d48 c00ddd58 00000678 
> 00000030 
>        00000004 00000008 0000000e c27f9b24 c0179d18 c00cc1cc c05dac00 
> c05dac00 
>        c05d000e c0179d28 c002097c c048f098 c0179ea8 0000001c 00000000 
> 000005a8 
>        00004000 00000000 c048f160 c0179ddc c37fe4e0 c05dac00 7fffffff 
> c048f040 
>        c0179e5c c0179d78 c0179d60 c00fa9a1 c048f040 c0179e5c 0000001c 
> 00000000 
>        c0179da4 c00bde23 c0d7bf0c c0179e5c 0000001c c0179d78 00000000 
> 00000000 
>        00000000 00000000 00000000 c37fe3d8 c218e000 c218ede8 c0179ea8 
> c0d7bf0c 
>        0000001c c0179e88 c3cad25f c0d7bf0c c0179e5c 0000001c c1eaa538 
> c35581dc 
>        c1d13960 00000000 00000000 00000000 ffffffff c0179ea8 0000001c c1eaa538
>        00000000 00000030 c35581dc c0179e10 c00bbe62 c01c5444 c35581dc c250e000
>        00000000 c1d13800 c1eaa538 10000000 c0179e28 c00c550f c1eaa538 c1d13800
>        00000040 c1eaa538 c0179e6c c3c0fa28 c1eaa538 00000010 00000000 00000006
>        c0179e90 0000003a c22997ac 00000002 c0179e6c 00000000 c0179e6c 00000000
>        00000000 c0179dd4 00000001 00000000 00000000 00004000 00000002 c3caf440
>        c0d7bf0c c3caf420 c0179ed4 c3cad511 00000001 c0d7bf0c c0179ea8 0000001c
>        00000000 c0d7bf0c 13956025 00000000 c2478c64 c0109f58 00000000 00040000
>        00040000 c010b1d4 c2478c64 c3caf420 00000000 c0179ef0 c3cad91c c3caf420
>        c2478c64 00000000 c0179f0c fffffff7 c0179f00 c008fc46 c01ab6a4 c0179f0c
>        c0179f1c c000c342 c01ab6a4 c01ab704 c01ab704 00000000 c01881d8 c0179f2c
>        c000c0a8 c0147510 00000001 c0179f48 c000be9e c01881d8 00000001 c018a400
>        c1d4bbec 00002140 c0179f68 c006f2c0 00000085 c0179f9c c1d4bbec 00000000
>        00000000 fbff9000 c0179f90 c007395e 00000085 c0179f9c 00000001 00000000
>        c0179f9c c0178000 c0178000 00000006 c0179fe4 c006dcd7 c0179f9c 00000001
>        39854980 38ecb300 c0178000 00000006 c0179fe4 00000006 00000821 00000821
>        00000006 c006bcc2 00000819 00203246 c0179fe4 c0178000 00000000 c01e2200
>        c01983a0 c0179ffc c017a57c c0116f40 c01986a0 00000000 c01985a0 00000000
> Call Trace: [<c00ddd58>] [<c00ddd58>] [<c00cc1cc>] [<c002097c>] [<c00fa9a1>]
>    [<c00bde23>] [<c3cad25f>] [<c00bbe62>] [<c00c550f>] [<c3c0fa28>] 
> [<c3caf440>]
>    [<c3caf420>] [<c3cad511>] [<c0109f58>] [<c010b1d4>] [<c3caf420>] 
> [<c3cad91c>]
>    [<c3caf420>] [<c008fc46>] [<c000c342>] [<c000c0a8>] [<c000be9e>] 
> [<c006f2c0>]
>    [<c007395e>] [<c006dcd7>] [<c006bcc2>]
>  <0>Kernel panic: Aiee, killing interrupt handler!
> Warning (Oops_read): Code line not seen, dumping what data is available
> 
> 
> >>EIP; c00c133b <alloc_skb+11b/130>   <=====
> 
> >>ebx; c048f040 <_end+2bd130/3a2e170>
> >>ecx; c0178000 <init_task_union+0/2000>
> >>edi; c048f160 <_end+2bd250/3a2e170>
> >>ebp; c0179cd0 <init_task_union+1cd0/2000>
> >>esp; c0179cc0 <init_task_union+1cc0/2000>
> 
> Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
> Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
> Trace; c00cc1cc <rtnetlink_put_metrics+9c/150>
> Trace; c002097c <kmem_cache_free_one+ec/200>
> Trace; c00fa9a1 <inet_sendmsg+41/50>
> Trace; c00bde23 <sock_sendmsg+73/b0>
> Trace; c3cad25f <[nbd]nbd_xmit+11f/2e0>
> Trace; c00bbe62 <netif_be_start_xmit+122/180>
> Trace; c00c550f <dev_queue_xmit+9f/270>
> Trace; c3c0fa28 <[ipv6]ndisc_send_rs+1a8/230>
> Trace; c3caf440 <[nbd].data.end+a59/3699>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c3cad511 <[nbd]nbd_send_req+f1/180>
> Trace; c0109f58 <br_config_bpdu_generation+38/40>
> Trace; c010b1d4 <br_hello_timer_expired+14/30>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c3cad91c <[nbd]do_nbd_request+fc/1d0>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c008fc46 <generic_unplug_device+66/70>
> Trace; c000c342 <__run_task_queue+62/80>
> Trace; c000c0a8 <tasklet_action+58/b0>
> Trace; c000be9e <do_softirq+de/100>
> Trace; c006f2c0 <do_IRQ+a0/b0>
> Trace; c007395e <evtchn_do_upcall+9e/100>
> Trace; c006dcd7 <hypervisor_callback+33/49>
> Trace; c006bcc2 <cpu_idle+62/c0>
> 
> 
> 1 warning issued.  Results may not be reliable.
 -=- MIME -=- 

--u3/rZRmxL6MmkK24
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Mon, Jul 12, 2004 at 10:06:55PM +0100, Keir Fraser wrote:
> Even better, recompile your DOM0 kernel with arch/xen/kernel/traps.c
> modified with 'kstack_depth_to_print' changed to a larger value like
> 240. This will give a fuller stack backtrace (it's currently less
> useful because it's truncated). Then provide the backtrace and the
> vmlinux file.

OK I've done that the vmlinux file is at
http://www.yuri.org.uk/~murble/vmlinux-xen

New Oops file should be attached.


cheers

bill
-- 
Bill Boughton           <murble(at)@yuri.org.uk>
                        <bill@(at)xencat.murble.org.uk>
Down not across

--u3/rZRmxL6MmkK24
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=xen0-oops

ksymoops 2.4.9 on i686 2.4.26-xen0.  Options used
     -v ./vmlinux (specified)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.26-xen0/ (default)
     -M (specified)

invalid operand: 0000
CPU:    0
EIP:    0819:[<c00c133b>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00213286
eax: 0000003a   ebx: c048f040   ecx: c0178000   edx: 00000000
esi: 00000030   edi: c048f160   ebp: c0179cd0   esp: c0179cc0
ds: 0821   es: 0821   ss: 0821
Process swapper (pid: 0, stackpage=c0179000)<1>
Stack: c012c920 c00ddd58 c048f040 c048f040 c0179d48 c00ddd58 00000678 00000030 
       00000004 00000008 0000000e c27f9b24 c0179d18 c00cc1cc c05dac00 c05dac00 
       c05d000e c0179d28 c002097c c048f098 c0179ea8 0000001c 00000000 000005a8 
       00004000 00000000 c048f160 c0179ddc c37fe4e0 c05dac00 7fffffff c048f040 
       c0179e5c c0179d78 c0179d60 c00fa9a1 c048f040 c0179e5c 0000001c 00000000 
       c0179da4 c00bde23 c0d7bf0c c0179e5c 0000001c c0179d78 00000000 00000000 
       00000000 00000000 00000000 c37fe3d8 c218e000 c218ede8 c0179ea8 c0d7bf0c 
       0000001c c0179e88 c3cad25f c0d7bf0c c0179e5c 0000001c c1eaa538 c35581dc 
       c1d13960 00000000 00000000 00000000 ffffffff c0179ea8 0000001c c1eaa538
       00000000 00000030 c35581dc c0179e10 c00bbe62 c01c5444 c35581dc c250e000
       00000000 c1d13800 c1eaa538 10000000 c0179e28 c00c550f c1eaa538 c1d13800
       00000040 c1eaa538 c0179e6c c3c0fa28 c1eaa538 00000010 00000000 00000006
       c0179e90 0000003a c22997ac 00000002 c0179e6c 00000000 c0179e6c 00000000
       00000000 c0179dd4 00000001 00000000 00000000 00004000 00000002 c3caf440
       c0d7bf0c c3caf420 c0179ed4 c3cad511 00000001 c0d7bf0c c0179ea8 0000001c
       00000000 c0d7bf0c 13956025 00000000 c2478c64 c0109f58 00000000 00040000
       00040000 c010b1d4 c2478c64 c3caf420 00000000 c0179ef0 c3cad91c c3caf420
       c2478c64 00000000 c0179f0c fffffff7 c0179f00 c008fc46 c01ab6a4 c0179f0c
       c0179f1c c000c342 c01ab6a4 c01ab704 c01ab704 00000000 c01881d8 c0179f2c
       c000c0a8 c0147510 00000001 c0179f48 c000be9e c01881d8 00000001 c018a400
       c1d4bbec 00002140 c0179f68 c006f2c0 00000085 c0179f9c c1d4bbec 00000000
       00000000 fbff9000 c0179f90 c007395e 00000085 c0179f9c 00000001 00000000
       c0179f9c c0178000 c0178000 00000006 c0179fe4 c006dcd7 c0179f9c 00000001
       39854980 38ecb300 c0178000 00000006 c0179fe4 00000006 00000821 00000821
       00000006 c006bcc2 00000819 00203246 c0179fe4 c0178000 00000000 c01e2200
       c01983a0 c0179ffc c017a57c c0116f40 c01986a0 00000000 c01985a0 00000000
Call Trace: [<c00ddd58>] [<c00ddd58>] [<c00cc1cc>] [<c002097c>] [<c00fa9a1>]
   [<c00bde23>] [<c3cad25f>] [<c00bbe62>] [<c00c550f>] [<c3c0fa28>] [<c3caf440>]
   [<c3caf420>] [<c3cad511>] [<c0109f58>] [<c010b1d4>] [<c3caf420>] [<c3cad91c>]
   [<c3caf420>] [<c008fc46>] [<c000c342>] [<c000c0a8>] [<c000be9e>] [<c006f2c0>]
   [<c007395e>] [<c006dcd7>] [<c006bcc2>]
 <0>Kernel panic: Aiee, killing interrupt handler!
Warning (Oops_read): Code line not seen, dumping what data is available


>>EIP; c00c133b <alloc_skb+11b/130>   <=====

>>ebx; c048f040 <_end+2bd130/3a2e170>
>>ecx; c0178000 <init_task_union+0/2000>
>>edi; c048f160 <_end+2bd250/3a2e170>
>>ebp; c0179cd0 <init_task_union+1cd0/2000>
>>esp; c0179cc0 <init_task_union+1cc0/2000>

Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
Trace; c00cc1cc <rtnetlink_put_metrics+9c/150>
Trace; c002097c <kmem_cache_free_one+ec/200>
Trace; c00fa9a1 <inet_sendmsg+41/50>
Trace; c00bde23 <sock_sendmsg+73/b0>
Trace; c3cad25f <[nbd]nbd_xmit+11f/2e0>
Trace; c00bbe62 <netif_be_start_xmit+122/180>
Trace; c00c550f <dev_queue_xmit+9f/270>
Trace; c3c0fa28 <[ipv6]ndisc_send_rs+1a8/230>
Trace; c3caf440 <[nbd].data.end+a59/3699>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c3cad511 <[nbd]nbd_send_req+f1/180>
Trace; c0109f58 <br_config_bpdu_generation+38/40>
Trace; c010b1d4 <br_hello_timer_expired+14/30>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c3cad91c <[nbd]do_nbd_request+fc/1d0>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c008fc46 <generic_unplug_device+66/70>
Trace; c000c342 <__run_task_queue+62/80>
Trace; c000c0a8 <tasklet_action+58/b0>
Trace; c000be9e <do_softirq+de/100>
Trace; c006f2c0 <do_IRQ+a0/b0>
Trace; c007395e <evtchn_do_upcall+9e/100>
Trace; c006dcd7 <hypervisor_callback+33/49>
Trace; c006bcc2 <cpu_idle+62/c0>


1 warning issued.  Results may not be reliable.

--u3/rZRmxL6MmkK24--



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel