| |
|
 |
|
 |
|
|
|
| |
|
|
xen-devel
Re: [Xen-devel] Xenlinux oops and crashes with nbd, Xen no longer boots
Okay, detangling the backtrace....
Entry EIP Function Caller's return EIP
--------- -------- -------------------
0xc00c1220 alloc_skb(1656,GFP_NOIO) 0xc00ddd58
0xc00dd4b0 tcp_sendmsg 0xc00fa9a1
0xc00fa960 inet_sendmsg 0xc00bde23
0xc00bddb0 sock_sendmsg 0xc3cad25f
unknown nbd_xmit 0xc3cad511
unknown nbd_send_req 0xc3cad91c
unknown do_nbd_request 0xc008fc46
0xc008fbe0 generic_unplug_device 0xc000c342
0xc000c2e0 __run_task_queue n/a (tail call)
0xc00b8f20 io_schedule 0xc000c0a8
0xc000c050 tasklet_action 0xc000be9e
0xc000bdc0 do_softirq 0xc006f2c0
0xc006f220 do_IRQ(133,-) 0xc007395e
0xc00738c0 evtchn_do_upcall 0xc006dcd7
0xc006dca4 hypervisor_callback n/a (async callback)
I see the problem. Well hidden because the key function (io_schedule)
ends in a tail call. The problem is that we are scheduling I/O
requests in interrupt context rather than deferring to a process
context --- a known issue that hasn't bitten until now. I'll have to
think about a fix.
-- Keir
> Down not across
> ksymoops 2.4.9 on i686 2.4.26-xen0. Options used
> -v ./vmlinux (specified)
> -k /proc/ksyms (default)
> -l /proc/modules (default)
> -o /lib/modules/2.4.26-xen0/ (default)
> -M (specified)
>
> invalid operand: 0000
> CPU: 0
> EIP: 0819:[<c00c133b>] Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00213286
> eax: 0000003a ebx: c048f040 ecx: c0178000 edx: 00000000
> esi: 00000030 edi: c048f160 ebp: c0179cd0 esp: c0179cc0
> ds: 0821 es: 0821 ss: 0821
> Process swapper (pid: 0, stackpage=c0179000)<1>
> Stack: c012c920 c00ddd58 c048f040 c048f040 c0179d48 c00ddd58 00000678
> 00000030
> 00000004 00000008 0000000e c27f9b24 c0179d18 c00cc1cc c05dac00
> c05dac00
> c05d000e c0179d28 c002097c c048f098 c0179ea8 0000001c 00000000
> 000005a8
> 00004000 00000000 c048f160 c0179ddc c37fe4e0 c05dac00 7fffffff
> c048f040
> c0179e5c c0179d78 c0179d60 c00fa9a1 c048f040 c0179e5c 0000001c
> 00000000
> c0179da4 c00bde23 c0d7bf0c c0179e5c 0000001c c0179d78 00000000
> 00000000
> 00000000 00000000 00000000 c37fe3d8 c218e000 c218ede8 c0179ea8
> c0d7bf0c
> 0000001c c0179e88 c3cad25f c0d7bf0c c0179e5c 0000001c c1eaa538
> c35581dc
> c1d13960 00000000 00000000 00000000 ffffffff c0179ea8 0000001c c1eaa538
> 00000000 00000030 c35581dc c0179e10 c00bbe62 c01c5444 c35581dc c250e000
> 00000000 c1d13800 c1eaa538 10000000 c0179e28 c00c550f c1eaa538 c1d13800
> 00000040 c1eaa538 c0179e6c c3c0fa28 c1eaa538 00000010 00000000 00000006
> c0179e90 0000003a c22997ac 00000002 c0179e6c 00000000 c0179e6c 00000000
> 00000000 c0179dd4 00000001 00000000 00000000 00004000 00000002 c3caf440
> c0d7bf0c c3caf420 c0179ed4 c3cad511 00000001 c0d7bf0c c0179ea8 0000001c
> 00000000 c0d7bf0c 13956025 00000000 c2478c64 c0109f58 00000000 00040000
> 00040000 c010b1d4 c2478c64 c3caf420 00000000 c0179ef0 c3cad91c c3caf420
> c2478c64 00000000 c0179f0c fffffff7 c0179f00 c008fc46 c01ab6a4 c0179f0c
> c0179f1c c000c342 c01ab6a4 c01ab704 c01ab704 00000000 c01881d8 c0179f2c
> c000c0a8 c0147510 00000001 c0179f48 c000be9e c01881d8 00000001 c018a400
> c1d4bbec 00002140 c0179f68 c006f2c0 00000085 c0179f9c c1d4bbec 00000000
> 00000000 fbff9000 c0179f90 c007395e 00000085 c0179f9c 00000001 00000000
> c0179f9c c0178000 c0178000 00000006 c0179fe4 c006dcd7 c0179f9c 00000001
> 39854980 38ecb300 c0178000 00000006 c0179fe4 00000006 00000821 00000821
> 00000006 c006bcc2 00000819 00203246 c0179fe4 c0178000 00000000 c01e2200
> c01983a0 c0179ffc c017a57c c0116f40 c01986a0 00000000 c01985a0 00000000
> Call Trace: [<c00ddd58>] [<c00ddd58>] [<c00cc1cc>] [<c002097c>] [<c00fa9a1>]
> [<c00bde23>] [<c3cad25f>] [<c00bbe62>] [<c00c550f>] [<c3c0fa28>]
> [<c3caf440>]
> [<c3caf420>] [<c3cad511>] [<c0109f58>] [<c010b1d4>] [<c3caf420>]
> [<c3cad91c>]
> [<c3caf420>] [<c008fc46>] [<c000c342>] [<c000c0a8>] [<c000be9e>]
> [<c006f2c0>]
> [<c007395e>] [<c006dcd7>] [<c006bcc2>]
> <0>Kernel panic: Aiee, killing interrupt handler!
> Warning (Oops_read): Code line not seen, dumping what data is available
>
>
> >>EIP; c00c133b <alloc_skb+11b/130> <=====
>
> >>ebx; c048f040 <_end+2bd130/3a2e170>
> >>ecx; c0178000 <init_task_union+0/2000>
> >>edi; c048f160 <_end+2bd250/3a2e170>
> >>ebp; c0179cd0 <init_task_union+1cd0/2000>
> >>esp; c0179cc0 <init_task_union+1cc0/2000>
>
> Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
> Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
> Trace; c00cc1cc <rtnetlink_put_metrics+9c/150>
> Trace; c002097c <kmem_cache_free_one+ec/200>
> Trace; c00fa9a1 <inet_sendmsg+41/50>
> Trace; c00bde23 <sock_sendmsg+73/b0>
> Trace; c3cad25f <[nbd]nbd_xmit+11f/2e0>
> Trace; c00bbe62 <netif_be_start_xmit+122/180>
> Trace; c00c550f <dev_queue_xmit+9f/270>
> Trace; c3c0fa28 <[ipv6]ndisc_send_rs+1a8/230>
> Trace; c3caf440 <[nbd].data.end+a59/3699>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c3cad511 <[nbd]nbd_send_req+f1/180>
> Trace; c0109f58 <br_config_bpdu_generation+38/40>
> Trace; c010b1d4 <br_hello_timer_expired+14/30>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c3cad91c <[nbd]do_nbd_request+fc/1d0>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c008fc46 <generic_unplug_device+66/70>
> Trace; c000c342 <__run_task_queue+62/80>
> Trace; c000c0a8 <tasklet_action+58/b0>
> Trace; c000be9e <do_softirq+de/100>
> Trace; c006f2c0 <do_IRQ+a0/b0>
> Trace; c007395e <evtchn_do_upcall+9e/100>
> Trace; c006dcd7 <hypervisor_callback+33/49>
> Trace; c006bcc2 <cpu_idle+62/c0>
>
>
> 1 warning issued. Results may not be reliable.
-=- MIME -=-
--u3/rZRmxL6MmkK24
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Mon, Jul 12, 2004 at 10:06:55PM +0100, Keir Fraser wrote:
> Even better, recompile your DOM0 kernel with arch/xen/kernel/traps.c
> modified with 'kstack_depth_to_print' changed to a larger value like
> 240. This will give a fuller stack backtrace (it's currently less
> useful because it's truncated). Then provide the backtrace and the
> vmlinux file.
OK I've done that the vmlinux file is at
http://www.yuri.org.uk/~murble/vmlinux-xen
New Oops file should be attached.
cheers
bill
--
Bill Boughton <murble(at)@yuri.org.uk>
<bill@(at)xencat.murble.org.uk>
Down not across
--u3/rZRmxL6MmkK24
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=xen0-oops
ksymoops 2.4.9 on i686 2.4.26-xen0. Options used
-v ./vmlinux (specified)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.26-xen0/ (default)
-M (specified)
invalid operand: 0000
CPU: 0
EIP: 0819:[<c00c133b>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00213286
eax: 0000003a ebx: c048f040 ecx: c0178000 edx: 00000000
esi: 00000030 edi: c048f160 ebp: c0179cd0 esp: c0179cc0
ds: 0821 es: 0821 ss: 0821
Process swapper (pid: 0, stackpage=c0179000)<1>
Stack: c012c920 c00ddd58 c048f040 c048f040 c0179d48 c00ddd58 00000678 00000030
00000004 00000008 0000000e c27f9b24 c0179d18 c00cc1cc c05dac00 c05dac00
c05d000e c0179d28 c002097c c048f098 c0179ea8 0000001c 00000000 000005a8
00004000 00000000 c048f160 c0179ddc c37fe4e0 c05dac00 7fffffff c048f040
c0179e5c c0179d78 c0179d60 c00fa9a1 c048f040 c0179e5c 0000001c 00000000
c0179da4 c00bde23 c0d7bf0c c0179e5c 0000001c c0179d78 00000000 00000000
00000000 00000000 00000000 c37fe3d8 c218e000 c218ede8 c0179ea8 c0d7bf0c
0000001c c0179e88 c3cad25f c0d7bf0c c0179e5c 0000001c c1eaa538 c35581dc
c1d13960 00000000 00000000 00000000 ffffffff c0179ea8 0000001c c1eaa538
00000000 00000030 c35581dc c0179e10 c00bbe62 c01c5444 c35581dc c250e000
00000000 c1d13800 c1eaa538 10000000 c0179e28 c00c550f c1eaa538 c1d13800
00000040 c1eaa538 c0179e6c c3c0fa28 c1eaa538 00000010 00000000 00000006
c0179e90 0000003a c22997ac 00000002 c0179e6c 00000000 c0179e6c 00000000
00000000 c0179dd4 00000001 00000000 00000000 00004000 00000002 c3caf440
c0d7bf0c c3caf420 c0179ed4 c3cad511 00000001 c0d7bf0c c0179ea8 0000001c
00000000 c0d7bf0c 13956025 00000000 c2478c64 c0109f58 00000000 00040000
00040000 c010b1d4 c2478c64 c3caf420 00000000 c0179ef0 c3cad91c c3caf420
c2478c64 00000000 c0179f0c fffffff7 c0179f00 c008fc46 c01ab6a4 c0179f0c
c0179f1c c000c342 c01ab6a4 c01ab704 c01ab704 00000000 c01881d8 c0179f2c
c000c0a8 c0147510 00000001 c0179f48 c000be9e c01881d8 00000001 c018a400
c1d4bbec 00002140 c0179f68 c006f2c0 00000085 c0179f9c c1d4bbec 00000000
00000000 fbff9000 c0179f90 c007395e 00000085 c0179f9c 00000001 00000000
c0179f9c c0178000 c0178000 00000006 c0179fe4 c006dcd7 c0179f9c 00000001
39854980 38ecb300 c0178000 00000006 c0179fe4 00000006 00000821 00000821
00000006 c006bcc2 00000819 00203246 c0179fe4 c0178000 00000000 c01e2200
c01983a0 c0179ffc c017a57c c0116f40 c01986a0 00000000 c01985a0 00000000
Call Trace: [<c00ddd58>] [<c00ddd58>] [<c00cc1cc>] [<c002097c>] [<c00fa9a1>]
[<c00bde23>] [<c3cad25f>] [<c00bbe62>] [<c00c550f>] [<c3c0fa28>] [<c3caf440>]
[<c3caf420>] [<c3cad511>] [<c0109f58>] [<c010b1d4>] [<c3caf420>] [<c3cad91c>]
[<c3caf420>] [<c008fc46>] [<c000c342>] [<c000c0a8>] [<c000be9e>] [<c006f2c0>]
[<c007395e>] [<c006dcd7>] [<c006bcc2>]
<0>Kernel panic: Aiee, killing interrupt handler!
Warning (Oops_read): Code line not seen, dumping what data is available
>>EIP; c00c133b <alloc_skb+11b/130> <=====
>>ebx; c048f040 <_end+2bd130/3a2e170>
>>ecx; c0178000 <init_task_union+0/2000>
>>edi; c048f160 <_end+2bd250/3a2e170>
>>ebp; c0179cd0 <init_task_union+1cd0/2000>
>>esp; c0179cc0 <init_task_union+1cc0/2000>
Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
Trace; c00cc1cc <rtnetlink_put_metrics+9c/150>
Trace; c002097c <kmem_cache_free_one+ec/200>
Trace; c00fa9a1 <inet_sendmsg+41/50>
Trace; c00bde23 <sock_sendmsg+73/b0>
Trace; c3cad25f <[nbd]nbd_xmit+11f/2e0>
Trace; c00bbe62 <netif_be_start_xmit+122/180>
Trace; c00c550f <dev_queue_xmit+9f/270>
Trace; c3c0fa28 <[ipv6]ndisc_send_rs+1a8/230>
Trace; c3caf440 <[nbd].data.end+a59/3699>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c3cad511 <[nbd]nbd_send_req+f1/180>
Trace; c0109f58 <br_config_bpdu_generation+38/40>
Trace; c010b1d4 <br_hello_timer_expired+14/30>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c3cad91c <[nbd]do_nbd_request+fc/1d0>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c008fc46 <generic_unplug_device+66/70>
Trace; c000c342 <__run_task_queue+62/80>
Trace; c000c0a8 <tasklet_action+58/b0>
Trace; c000be9e <do_softirq+de/100>
Trace; c006f2c0 <do_IRQ+a0/b0>
Trace; c007395e <evtchn_do_upcall+9e/100>
Trace; c006dcd7 <hypervisor_callback+33/49>
Trace; c006bcc2 <cpu_idle+62/c0>
1 warning issued. Results may not be reliable.
--u3/rZRmxL6MmkK24--
-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 -
digital self defense, top technical experts, no vendor pitches,
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel
|
|
|
| |
|
Home