[Xen-users] Crashing PV DomU, Can't find the reason

["Emre Erenoglu" <erenoglu@xxxxxxxxx>]

Hi,

I have a Paravirtual DomU crashing on me at bootup on a newly setup Ubuntu 7.10 system. The system has 2.6.22-14-xen kernel and Xen 3.1. HVM guests are OK.

I was running perfectly OK with this system at Ubuntu 
7.10's beta stages, when there was 2.6.22-12-xen kernel, but when Gutsy is officially launched, I decided to wipe out my Dom0 and reinstall it. It came with kernel 2.6.22-14-xen and now crashing on me at bootup.

In order to make sure that it's not Ubuntu specific, I installed Fedora 8 with 2.6.21 Dom0 kernel. I ran this DomU with  2.6.23 standard and 2.6.21 Dom0 kernels. No change, always same crash. So it seems that there's some issue here.

Here's some log from the boot about the crash. It seems that something is related with atomic-ops, maybe it's possible to fix it within the DomU.

[  186.548094] ------------[ cut here ]------------
[  186.548100] kernel BUG at include2/asm/mach-xen/asm/maddr.h:34!
[  186.548103] invalid opcode: 0000 [#1]
[  186.548104] SMP
[  186.548107] Modules linked in: nls_cp437 cifs ext2 usbcore evdev joydev ext3
jbd mbcache raid10 raid456 xor raid1 raid0 multipath linear md_mod dm_mirror dm_
snapshot dm_mod fuse apparmor commoncap
[  186.548129] CPU:    0
[  186.548129] EIP:    0061:[<c0118dd7>]    Not tainted VLI
[  186.548130] EFLAGS: 00010206   (2.6.22-14-xen #1)
[  186.548138] EIP is at xen_l1_entry_update+0x97/0xb0
[  186.548140] eax: 00030800   ebx: 01aeb025   ecx: 00000000   edx: 000357f6
[  186.548142] esi: 357f6240   edi: 00000001   ebp: 00000000   esp: c07fbe24
[  186.548144] ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0069
[  186.548147] Process dmidecode (pid: 3512, ti=c07fa000 task=c22894c0 task.ti=c
07fa000)
[  186.548149] Stack: 00000000 08048000 ed64a318 00000000 00000000 01aeb025 0178
b067 00000001
[  186.548156]        f57f6240 c0160c23 01aeb025 00000000 80000000 ecbc3c40 0000
0000 08053fff
[  186.548164]        034c2067 c0423d80 0002f70b 00000000 2f70b000 00000000 0002
f70b 00000000
[  186.548171] Call Trace:
[  186.548174]  [<c0160c23>] unmap_vmas+0x963/0xbf0
[  186.548182
]  [<c0167757>] exit_mmap+0x77/0xf0
[  186.548186]  [<c011f8e8>] mmput+0x38/0xa0
[  186.548190]  [<c0125081>] do_exit+0x121/0x850
[  186.548193]  [<c01676d1>] remove_vma+0x41/0x50
[  186.548197
]  [<c0168146>] do_munmap+0x186/0x1e0
[  186.548201]  [<c01257d6>] do_group_exit+0x26/0x90
[  186.548204]  [<c01055a2>] syscall_call+0x7/0xb
[  186.548208]  =======================
[  186.548210
] Code: 24 04 be f0 7f 00 00 89 4c 24 0c b9 01 00 00 00 89 54 24 10
 31 d2 e8 59 82 fe ff 85 c0 78 15 83 c4 14 5b 5e 5f 5d c3 39 c2 72 93 <0f> 0b eb
 fe 90 8d 74 26 00 0f 0b eb fe 8d b6 00 00 00 00 8d bf
[  
186.548250] EIP: [<c0118dd7>] xen_l1_entry_update+0x97/0xb0 SS:ESP 0069:c07fb
e24
[  186.548257] Fixing recursive fault but reboot is needed!
[  186.548261] BUG: scheduling while atomic: dmidecode/0x00000001/3512
[  186.548264]  [<c02ff3f0>] schedule+0x550/0x900
[  186.548271]  [<c0259946>] evtchn_do_upcall+0xb6/0xf0
[  186.548277]  [<c01255bb>] do_exit+0x65b/0x850
[  186.548283]  [<c01061ff>] die+0x25f/0x260
[  186.548288]  [<c0106520>] do_invalid_op+0x0/0x90
[  186.548292]  [<c01065a1>] do_invalid_op+0x81/0x90
[  186.548297]  [<c0118dd7>] xen_l1_entry_update+0x97/0xb0
[  186.548302]  [<c01183c9>] kmap_atomic_prot+0xd9/0x180
[  186.548307]  [<c011a261>] __activate_task+0x21/0x40
[  186.548312]  [<c011c34f>] try_to_wake_up+0x3f/0x400
[  186.548317]  [<c015715a>] get_page_from_freelist+0x34a/0x430
[  186.548323]  [<c01561bc>] __rmqueue+0x9c/0xf0
[  186.548326]  [<c01040dd>] __switch_to+0x23d/0x490
[  186.548330]  [<c0301685>] error_code+0x35/0x40
[  186.548336]  [<c0118dd7>] xen_l1_entry_update+0x97/0xb0
[  186.548341]  [<c0160c23>] unmap_vmas+0x963/0xbf0
[  186.548350]  [<c0167757>] exit_mmap+0x77/0xf0
[  186.548354]  [<c011f8e8>] mmput+0x38/0xa0
[  186.548359]  [<c0125081>] do_exit+0x121/0x850
[  186.548364]  [<c01676d1>] remove_vma+0x41/0x50
[  186.548368]  [<c0168146>] do_munmap+0x186/0x1e0
[  186.548374]  [<c01257d6>] do_group_exit+0x26/0x90
[  186.548379]  [<c01055a2>] syscall_call+0x7/0xb
[  186.548385]  =======================
[  186.548525] ------------[ cut here ]------------
[  186.548528] kernel BUG at /build/buildd/linux-source-2.6.22-2.6.22/debian/bui
ld/custom-source-xen/arch/i386/mm/highmem-xen.c:38!
[  186.548530] invalid opcode: 0000 [#2]
[  186.548532] SMP
[  186.548534] Modules linked in: nls_cp437 cifs ext2 usbcore evdev joydev ext3
jbd mbcache raid10 raid456 xor raid1 raid0 multipath linear md_mod dm_mirror dm_
snapshot dm_mod fuse apparmor commoncap
[  186.548553] CPU:    0
[  186.548553] EIP:    0061:[<c0118462>]    Not tainted VLI
[  186.548554] EFLAGS: 00010282   (2.6.22-14-xen #1)
[  186.548558] EIP is at kmap_atomic_prot+0x172/0x180
[  186.548560] eax: 8178b061   ebx: c200efc0   ecx: 00000000   edx: c1a44fb0
[  186.548562] esi: 00000061   edi: 80000000   ebp: 00000007   esp: c21bbe18
[  186.548564] ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
[  186.548567] Process klogd (pid: 3494, ti=c21ba000 task=c070d4c0 task.ti=c21ba
000)
[  186.548568] Stack: c21bbe58 00008200 00000000 c21bbe7c c1a44fb0 00000000 c041
f000 00000000
[  186.548576]        c03faf00 00000000 03a18067 00000000 c21bbefc c01644e9 0000
0061 80000000
[  186.548583]        00000000 c031ebe0 c21bbed8 c21bbf60 c0423d80 c0284b5d b7e8
f270 c1898cb8
[  186.548591] Call Trace:
[  186.548594]  [<c03faf00>] request_standard_resources+0x10/0x3e0
[  186.548598]  [<c01644e9>] __handle_mm_fault+0x179/0x1a80
[  186.548602]  [<c0284b5d>] sock_aio_write+0x11d/0x130
[  186.548608]  [<c017a6d5>] do_sync_write+0xd5/0x120
[  186.548612]  [<c03028c0>] do_page_fault+0x0/0xdf0
[  186.548616]  [<c0302bbe>] do_page_fault+0x2fe/0xdf0
[  186.548620]  [<c017b027>] vfs_write+0x127/0x170
[  186.548623]  [<c03028c0>] do_page_fault+0x0/0xdf0
[  186.548627]  [<c0301685>] error_code+0x35/0x40
[  186.548631]  =======================
[  186.548632] Code: 0c 89 f9 89 f3 25 ff 0f 00 00 30 d2 09 c3 09 d1 89 5c 24 18
 89 4c 24 1c e9 4d ff ff ff 83 c4 24 89 d8 5b 5e 5f 5d e9 5e 5c 04 00 <0f> 0b eb
 fe 0f 0b eb fe 8d b6 00 00 00 00 53 83 ec 08 8b 1d 5c
[  186.548670] EIP: [<c0118462>] kmap_atomic_prot+0x172/0x180 SS:ESP 0069:c21bbe
18
[  186.548675] note: klogd[3494] exited with preempt_count 1
[  
186.548732] ------------[ cut here ]------------
[  186.548734] kernel BUG at /build/buildd/linux-source-2.6.22-2.6.22/debian/bui
ld/custom-source-xen/arch/i386/mm/highmem-xen.c:38!
[  186.548736] invalid opcode: 0000 [#3]
[  186.548737] SMP
[  186.548739] Modules linked in: nls_cp437 cifs ext2 usbcore evdev joydev ext3
jbd mbcache raid10 raid456 xor raid1 raid0 multipath linear md_mod dm_mirror dm_
snapshot dm_mod fuse apparmor commoncap
[  186.548757] CPU:    0
[  186.548757] EIP:    0061:[<c0118462>]    Not tainted VLI
[  186.548758] EFLAGS: 00010282   (2.6.22-14-xen #1)
[  186.548761] EIP is at kmap_atomic_prot+0x172/0x180
[  186.548763] eax: 8178b061   ebx: c2021ba0   ecx: 00000000   edx: c1a44fb0
[  186.548765] esi: 00000063   edi: 80000000   ebp: 00000007   esp: c21bbb8c
[  186.548767] ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0069
[  186.548769] Process klogd (pid: 3494, ti=c21ba000 task=c070d4c0 task.ti=c21ba
000)
[  186.548771] Stack: 00000000 c0121f30 00000001 fffff793 c1a44fb0 001bbc34 c031
4720 00000006
[  186.548778]        37352331 00000000 030b9067 00000000 c0437158 c016078d 0000
0063 80000000
[  186.548785]        ffffffff 00000000 00000030 0804cfff 00000001 00000002 0002
eddd 00000000
[  186.548792] Call Trace:
[  186.548794]  [<c0121f30>] release_console_sem+0x1b0/0x230
[  186.548798]  [<c016078d>] unmap_vmas+0x4cd/0xbf0
[  186.548805]  [<c0167757>] exit_mmap+0x77/0xf0
[  186.548808]  [<c011f8e8>] mmput+0x38/0xa0
[  186.548811]  [<c0125081>] do_exit+0x121/0x850
[  186.548815]  [<c01061ff>] die+0x25f/0x260
[  186.548818]  [<c0106520>] do_invalid_op+0x0/0x90
[  186.548821]  [<c01065a1>] do_invalid_op+0x81/0x90
[  186.548824]  [<c0118462>] kmap_atomic_prot+0x172/0x180
[  186.548827]  [<c017676c>] __slab_alloc+0x2dc/0x520
[  186.548831]  [<c03faf00>] request_standard_resources+0x10/0x3e0
[  186.548835]  [<c011a059>] __wake_up_common+0x39/0x60
[  186.548839
]  [<c0301685>] error_code+0x35/0x40
[  186.548842]  [<c0118462>] kmap_atomic_prot+0x172/0x180
[  186.548846]  [<c03faf00>] request_standard_resources+0x10/0x3e0
[  186.548849]  [<c01644e9>] __handle_mm_fault+0x179/0x1a80
[  186.548853]  [<c0284b5d>] sock_aio_write+0x11d/0x130
[  186.548857]  [<c017a6d5>] do_sync_write+0xd5/0x120
[  186.548861]  [<c03028c0>] do_page_fault+0x0/0xdf0
[  186.548865]  [<c0302bbe>] do_page_fault+0x2fe/0xdf0
[  186.548869]  [<c017b027>] vfs_write+0x127/0x170
[  186.548872]  [<c03028c0>] do_page_fault+0x0/0xdf0
[  186.548875]  [<c0301685>] error_code+0x35/0x40
[  186.548879]  =======================
[  186.548880] Code: 0c 89 f9 89 f3 25 ff 0f 00 00 30 d2 09 c3 09 d1 89 5c 24 18
 89 4c 24 1c e9 4d ff ff ff 83 c4 24 89 d8 5b 5e 5f 5d e9 5e 5c 04 00 <0f> 0b eb
 fe 0f 0b eb fe 8d b6 00 00 00 00 53 83 ec 08 8b 1d 5c
[  186.548918] EIP: [<c0118462>] kmap_atomic_prot+0x172/0x180 SS:ESP 0069:c21bbb
8c
[  186.548922] Fixing recursive fault but reboot is needed!
[  186.548924] BUG: scheduling while atomic: klogd/0x00000002/3494
[  186.548926]  [<c02ff3f0>] schedule+0x550/0x900
[  186.548930]  [<c0122331>] vprintk+0x1e1/0x3c0
[  186.548934]  [<c01255bb>] do_exit+0x65b/0x850
[  186.548938]  [<c01061ff>] die+0x25f/0x260
[  186.548941]  [<c0106520>] do_invalid_op+0x0/0x90
[  186.548943]  [<c01065a1>] do_invalid_op+0x81/0x90
[  186.548946]  [<c0118462>] kmap_atomic_prot+0x172/0x180
[  186.548950]  [<c025d310>] kcons_write+0x0/0xd0
[  186.548953]  [<c0121b03>] __call_console_drivers+0x53/0x60
[  186.548956]  [<c0121f87>] release_console_sem+0x207/0x230
[  186.548959]  [<c025daf5>] xencons_ring_send+0xc5/0x140
[  186.548962
]  [<c01fa5cb>] vsnprintf+0x54b/0x630
[  186.548967]  [<c025d11f>] __xencons_tx_flush+0xff/0x140
[  186.548970]  [<c0301685>] error_code+0x35/0x40
[  186.548973]  [<c025007b>] device_add_groups+0x7b/0x80
[  186.548977]  [<c0118462>] kmap_atomic_prot+0x172/0x180
[  186.548980]  [<c0121f30>] release_console_sem+0x1b0/0x230
[  186.548984]  [<c016078d>] unmap_vmas+0x4cd/0xbf0
[  186.549136]  [<c0167757>] exit_mmap+0x77/0xf0
[  186.549140]  [<c011f8e8>] mmput+0x38/0xa0
[  186.549144]  [<c0125081>] do_exit+0x121/0x850
[  186.549151]  [<c01061ff>] die+0x25f/0x260
[  186.549155]  [<c0106520>] do_invalid_op+0x0/0x90
[  186.549159]  [<c01065a1>] do_invalid_op+0x81/0x90
[  186.549163]  [<c0118462>] kmap_atomic_prot+0x172/0x180
[  186.549170]  [<c017676c>] __slab_alloc+0x2dc/0x520
[  186.549175]  [<c03faf00>] request_standard_resources+0x10/0x3e0
[  186.549180]  [<c011a059>] __wake_up_common+0x39/0x60
[  186.549185]  [<c0301685>] error_code+0x35/0x40
[  186.549190]  [<c0118462>] kmap_atomic_prot+0x172/0x180
[  186.549196]  [<c03faf00>] request_standard_resources+0x10/0x3e0
[  186.549201]  [<c01644e9>] __handle_mm_fault+0x179/0x1a80
[  186.549206]  [<c0284b5d>] sock_aio_write+0x11d/0x130
[  186.549214]  [<c017a6d5>] do_sync_write+0xd5/0x120
[  186.549219]  [<c03028c0>] do_page_fault+0x0/0xdf0
[  186.549224]  [<c0302bbe>] do_page_fault+0x2fe/0xdf0
[  186.549231]  [<c017b027>] vfs_write+0x127/0x170
[  186.549236]  [<c03028c0>] do_page_fault+0x0/0xdf0
[  186.549241]  [<c0301685>] error_code+0x35/0x40
[  186.549246]  =======================
[  186.549344] ------------[ cut here ]------------
[  186.549347] kernel BUG at /build/buildd/linux-source-2.6.22-2.6.22/debian/bui
ld/custom-source-xen/arch/i386/mm/highmem-
xen.c:38!
[  186.549350] invalid opcode: 0000 [#4]
[  186.549351] SMP
[  186.549353] Modules linked in: nls_cp437 cifs ext2 usbcore evdev joydev ext3
jbd mbcache raid10 raid456 xor raid1 raid0 multipath linear md_mod dm_mirror dm_
snapshot dm_mod fuse apparmor commoncap
[  186.549372] CPU:    0
[  186.549372] EIP:    0061:[<c0118462>]    Not tainted VLI
[  186.549373] EFLAGS: 00010282   (2.6.22-14-xen #1)
[  186.549377
] EIP is at kmap_atomic_prot+0x172/0x180
[  186.549379] eax: 8178b061   ebx: c2019a80   ecx: 00000000   edx: c1a44fb0
[  186.549381] esi: 00000063   edi: 80000000   ebp: 00000007   esp: ebec9dc8
[  186.549383
] ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
[  186.549386] Process syslogd (pid: 3491, ti=ebec8000 task=c224d9f0 task.ti=ebe
c8000)
[  186.549388] Stack: c2019a80 c2019a80 00000000 00000200 c1a44fb0 ebe7c200 ecbc
3e00 c1f9c2c0
[  186.549395]        c1fd4100 00000000 034c2067 00000000 c0437158 c0162c17 0000
0063 80000000
[  186.549402]        00000002 00000002 00000000 00000000 00000001 000240d0 c03c
1f98 c1f9c2c0
[  186.549409] Call Trace:
[  186.549412]  [<c0162c17>] copy_page_range+0x497/0xc60
[  186.549416]  [<c0184c45>] link_path_walk+0x65/0xc0
[  186.549422]  [<c011622c>] pgd_alloc+0x21c/0x2f0
[  
186.549425]  [<c0109b04>] init_new_context+0x34/0x120
[  186.549429]  [<c01201bc>] copy_process+0x84c/0x1200
[  186.549433]  [<c0179125>] nameidata_to_filp+0x35/0x40
[  186.549437]  [<c0120df6>] do_fork+0x76/0x230
[  186.549440]  [<c01fb4a0>] copy_to_user+0x30/0x60
[  186.549443]  [<c03fa240>] setup_arch+0x5d0/0x7a0
[  186.549447]  [<c0103206>] sys_clone+0x36/0x40
[  186.549450]  [<c01055a2>] syscall_call+0x7/0xb
[  186.549453]  =======================
[  186.549455] Code: 0c 89 f9 89 f3 25 ff 0f 00 00 30 d2 09 c3 09 d1 89 5c 24 18
 89 4c 24 1c e9 4d ff ff ff 83 c4 24 89 d8 5b 5e 5f 5d e9 5e 5c 04 00 <0f> 0b eb
 fe 0f 0b eb fe 8d b6 00 00 00 00 53 83 ec 08 8b 1d 5c
[  186.549493] EIP: [<c0118462>] kmap_atomic_prot+0x172/0x180 SS:ESP 0069:ebec9d
c8
[  186.549497] note: syslogd[3491] exited with preempt_count 1
[  186.549500] BUG: scheduling while atomic: syslogd/0x00000001/3491
[  186.549502]  [<c02ff3f0>] schedule+0x550/0x900
[  186.549505]  [<c025d310>] kcons_write+0x0/0xd0
[  186.549508]  [<c0121b03>] __call_console_drivers+0x53/0x60
[  186.549511]  [<c0121f87>] release_console_sem+0x207/0x230
[  186.549515]  [<c0122331>] vprintk+0x1e1/0x3c0
[  186.549518]  [<c01f8e95>] rwsem_down_failed_common+0x85/0x1b0
[  186.549521]  [<c0300e4a>] rwsem_down_read_failed+0x1a/0x30
[  186.549525]  [<c0300ed7>] call_rwsem_down_read_failed+0x7/0x10
[  186.549528]  [<c0139c84>] down_read+0x14/0x20
[  186.549531]  [<c0145c49>] acct_collect+0x39/0x180
[  186.549534]  [<c012506c>] do_exit+0x10c/0x850
[  186.549538]  [<c01061ff>] die+0x25f/0x260
[  186.549541]  [<c0106520>] do_invalid_op+0x0/0x90
[  186.549544]  [<c01065a1>] do_invalid_op+0x81/0x90
[  186.549547]  [<c0118462>] kmap_atomic_prot+0x172/0x180
[  186.549550]  [<c011848c>] kmap_atomic+0x1c/0x30
[  186.549553]  [<c01181ee>] kunmap_atomic+0x3e/0x60
[  186.549556]  [<c015715a>] get_page_from_freelist+0x34a/0x430
[  186.549560]  [<c0301685>] error_code+0x35/0x40
[  186.549563]  [<c0118462>] kmap_atomic_prot+0x172/0x180
[  186.549567]  [<c0162c17>] copy_page_range+0x497/0xc60
[  186.549571]  [<c0184c45>] link_path_walk+0x65/0xc0
[  186.549576]  [<c011622c>] pgd_alloc+0x21c/0x2f0
[  186.549579]  [<c0109b04>] init_new_context+0x34/0x120
[  186.549582]  [<c01201bc>] copy_process+0x84c/0x1200
[  186.549585]  [<c0179125>] nameidata_to_filp+0x35/0x40
[  186.549589]  [<c0120df6>] do_fork+0x76/0x230
[  186.549592]  [<c01fb4a0>] copy_to_user+0x30/0x60
[  186.549595]  [<c03fa240>] setup_arch+0x5d0/0x7a0
[  186.549598]  [<c0103206>] sys_clone+0x36/0x40
[  186.549601]  [<c01055a2>] syscall_call+0x7/0xb
[  186.549604]  =======================
[  188.395268] ------------[ cut here ]------------


Thanks for any insight,

Best regards,

Emre Erenoglu
erenoglu@xxxxxxxxx
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users