[Xen-users] Virtual Machine suddenly stops working

[Marco Weber <marco.jamendo@xxxxxxxxx>]

Hi,

This problem just occurs on a small ubuntu 8.04 LTS host.
On this host, there is just 1 VM running.
The problem always occurs, when the VM is up for about 2-3days.
Suddenly, the machine hangs and you cannot connect to it using ssh.
( Just a xm destroy  and  xm create   helps to get the VM running again. )

when i open the xen console of the vm, i start seeing such strange error messages:

[148030.316941]  [<c011cdb3>] dequeue_entity+0x13/0x40

[148030.316952]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148030.316963]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148030.316974]  [<c0167ecf>] shrink_inactive_list+0x11f/

0x3b0
[148030.316991]  [<c01681fc>] shrink_zone+0x9c/0x100
[148030.317001]  [<c016889c>] kswapd+0x44c/0x490
[148030.317014]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[148030.317027]  [<c011e270>] complete+0x40/0x60
[148030.317037]  [<c0168450>] kswapd+0x0/0x490
[148030.317042]  [<c013b812>] kthread+0x42/0x70
[148030.317046]  [<c013b7d0>] kthread+0x0/0x70
[148030.317051]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[148030.317057]  =======================
[148041.948368] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[148041.948377]
[148041.948383] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[148041.948390] EIP: 0061:[<c03285f7>] EFLAGS: 00000286 CPU: 0
[148041.948399] EIP is at _spin_lock+0x7/0x10
[148041.948405] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[148041.948410] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[148041.948416]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[148041.948423] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[148041.948430] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[148041.948435] DR6: ffff0ff0 DR7: 00000400
[148041.948441]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[148041.948456]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[148041.948471]  [<c011cdb3>] dequeue_entity+0x13/0x40
[148041.948481]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148041.948492]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148041.948504]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[148041.948520]  [<c01681fc>] shrink_zone+0x9c/0x100
[148041.948531]  [<c016889c>] kswapd+0x44c/0x490
[148041.948544]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[148041.948557]  [<c011e270>] complete+0x40/0x60
[148041.948566]  [<c0168450>] kswapd+0x0/0x490
[148041.948571]  [<c013b812>] kthread+0x42/0x70
[148041.948576]  [<c013b7d0>] kthread+0x0/0x70
[148041.948581]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[148041.948587]  =======================
[148053.543628] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[148053.543638]
[148053.543644] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[148053.543650] EIP: 0061:[<c03285f7>] EFLAGS: 00000286 CPU: 0
[148053.543660] EIP is at _spin_lock+0x7/0x10
[148053.543665] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[148053.543671] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[148053.543676]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[148053.543684] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[148053.543690] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[148053.543696] DR6: ffff0ff0 DR7: 00000400
[148053.543701]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[148053.543719]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[148053.543734]  [<c011cdb3>] dequeue_entity+0x13/0x40
[148053.543745]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148053.543755]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148053.543767]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[148053.543784]  [<c01681fc>] shrink_zone+0x9c/0x100
[148053.543795]  [<c016889c>] kswapd+0x44c/0x490
[148053.543808]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[148053.543820]  [<c011e270>] complete+0x40/0x60
[148053.543828]  [<c0168450>] kswapd+0x0/0x490
[148053.543833]  [<c013b812>] kthread+0x42/0x70
[148053.543837]  [<c013b7d0>] kthread+0x0/0x70
[148053.543842]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[148053.543849]  =======================
[148065.165553] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[148065.165563]
[148065.165569] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[148065.165575] EIP: 0061:[<c03285f5>] EFLAGS: 00000286 CPU: 0
[148065.165585] EIP is at _spin_lock+0x5/0x10
[148065.165590] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[148065.165596] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[148065.165601]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[148065.165609] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[148065.165615] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[148065.165621] DR6: ffff0ff0 DR7: 00000400
[148065.165626]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[148065.165642]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[148065.165656]  [<c011cdb3>] dequeue_entity+0x13/0x40
[148065.165667]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148065.165678]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148065.165690]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[148065.165706]  [<c01681fc>] shrink_zone+0x9c/0x100
[148065.165717]  [<c016889c>] kswapd+0x44c/0x490
[148065.165730]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[148065.165742]  [<c011e270>] complete+0x40/0x60
[148065.165752]  [<c0168450>] kswapd+0x0/0x490
[148065.165757]  [<c013b812>] kthread+0x42/0x70
[148065.165762]  [<c013b7d0>] kthread+0x0/0x70
[148065.165767]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[148065.165773]  =======================
[148076.738934] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[148076.738943]
[148076.738950] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[148076.738956] EIP: 0061:[<c03285f5>] EFLAGS: 00000286 CPU: 0
[148076.738965] EIP is at _spin_lock+0x5/0x10
[148076.738971] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[148076.738976] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[148076.738982]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[148076.738989] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[148076.738996] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[148076.739001] DR6: ffff0ff0 DR7: 00000400
[148076.739007]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[148076.739022]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[148076.739037]  [<c011cdb3>] dequeue_entity+0x13/0x40
[148076.739048]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148076.739059]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[148076.739070]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[148076.739087]  [<c01681fc>] shrink_zone+0x9c/0x100
[148076.739097]  [<c016889c>] kswapd+0x44c/0x490
[148076.739110]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[148076.739123]  [<c011e270>] complete+0x40/0x60
[148076.739133]  [<c0168450>] kswapd+0x0/0x490
[148076.739138]  [<c013b812>] kthread+0x42/0x70
[148076.739142]  [<c013b7d0>] kthread+0x0/0x70
[148076.739147]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[148076.739154]  =======================
[148088.362709] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[148088.362719]
[148088.362725] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[148088.362731] EIP: 0061:[<c03285f7>] EFLAGS: 00000286 CPU: 0
[148088.362741] EIP is at _spin_lock+0x7/0x10
[148088.362746] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[148088.362754] ESI: c17a33c0 EDI__rmqueue_smallest+0xb7/0x130
[178204.806034]  [<c02de99b>] __inet_lookup_listener+0x1b/0x140
[178204.806041]  [<c02f589d>] tcp_v4_rcv+0x7bd/0x9a0
[178204.806049]  [<c02d72b9>] ip_local_deliver_finish+0xf9/0x210
[178204.806055]  [<c02d6f4f>] ip_rcv_finish+0xff/0x370
[178204.806061]  [<c01082e3>] sched_clock+0x23/0x70
[178204.806067]  [<c02d74c0>] ip_rcv+0x0/0x290
[178204.806072]  [<c02b4419>] netif_receive_skb+0x399/0x480
[178204.806079]  [<c01623d9>] free_hot_cold_page+0x139/0x220
[178204.806085]  [<c026bd97>] netif_poll+0xa67/0[187595.019177] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[187595.019185]
[187595.019189] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[187595.019193] EIP: 0061:[<c03285f7>] EFLAGS: 00000286 CPU: 0
[187595.019200] EIP is at _spin_lock+0x7/0x10
[187595.019203] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[187595.019206] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[187595.019210]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[187595.019215] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[187595.019219] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[187595.019223] DR6: ffff0ff0 DR7: 00000400
[187595.019226]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[187595.019236]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[187595.019245]  [<c011cdb3>] dequeue_entity+0x13/0x40
[187595.019250]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187595.019256]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187595.019262]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[187595.019274]  [<c01681fc>] shrink_zone+0x9c/0x100
[187595.019279]  [<c016889c>] kswapd+0x44c/0x490
[187595.019287]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[187595.019294]  [<c011e270>] complete+0x40/0x60
[187699.479450]  [<c0168450>] kswapd+0x0/0x490
[187699.479454]  [<c013b812>] kthread+0x42/0x70
[187699.479459]  [<c013b7d0>] kthread+0x0/0x70
[187699.479464]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[187699.479470]  =======================
[187711.037292] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[187711.037301]
[187711.037308] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[187711.037315] EIP: 0061:[<c03285f5>] EFLAGS: 00000286 CPU: 0
[187711.037325] EIP is at _spin_lock+0x5/0x10
[187711.037332] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[187711.037338] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[187711.037345]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[187711.037353] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[187711.037360] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[187711.037367] DR6: ffff0ff0 DR7: 00000400
[187711.037373]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[187711.037390]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[187711.037407]  [<c011cdb3>] dequeue_entity+0x13/0x40
[187711.037420]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187711.037433]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187711.037448]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[187711.037467]  [<c01681fc>] shrink_zone+0x9c/0x100
[187711.037480]  [<c016889c>] kswapd+0x44c/0x490
[187711.037496]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[187711.037511]  [<c011e270>] complete+0x40/0x60
[187711.037524]  [<c0168450>] kswapd+0x0/0x490
[187711.037529]  [<c013b812>] kthread+0x42/0x70
[187711.037534]  [<c013b7d0>] kthread+0x0/0x70
[187711.037539]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[187711.037545]  =======================
[187722.652134] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[187722.652143]
[187722.652150] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[187722.652157] EIP: 0061:[<c03285f5>] EFLAGS: 00000286 CPU: 0
[187722.652168] EIP is at _spin_lock+0x5/0x10
[187722.652174] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[187722.652180] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[187722.652187]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[187722.652195] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[187722.652202] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[187722.652209] DR6: ffff0ff0 DR7: 00000400
[187722.652215]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[187722.652233]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[187722.652252]  [<c011cdb3>] dequeue_entity+0x13/0x40
[187722.652265]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187722.652284]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187722.652306]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[187722.652329]  [<c01681fc>] shrink_zone+0x9c/0x100
[187722.652347]  [<c016889c>] kswapd+0x44c/0x490
[187722.652372]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[187722.652387]  [<c011e270>] complete+0x40/0x60
[187722.652400]  [<c0168450>] kswapd+0x0/0x490
[187722.652406]  [<c013b812>] kthread+0x42/0x70
[187722.652410]  [<c013b7d0>] kthread+0x0/0x70
[187722.652415]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[187722.652426]  =======================
[187734.268601] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[187734.268610]
[187734.268617] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[187734.268624] EIP: 0061:[<c03285f5>] EFLAGS: 00000286 CPU: 0
[187734.268635] EIP is at _spin_lock+0x5/0x10
[187734.268641] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[187734.268647] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[187734.268654]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[187734.268662] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[187734.268669] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[187734.268676] DR6: ffff0ff0 DR7: 00000400
[187734.268682]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[187734.268700]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[187734.268717]  [<c011cdb3>] dequeue_entity+0x13/0x40
[187734.268730]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187734.268744]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187734.268758]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[187734.268777]  [<c01681fc>] shrink_zone+0x9c/0x100
[187734.268795]  [<c016889c>] kswapd+0x44c/0x490
[187734.268811]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[187734.268826]  [<c011e270>] complete+0x40/0x60
[187734.268839]  [<c0168450>] kswapd+0x0/0x490
[187734.268845]  [<c013b812>] kthread+0x42/0x70
[187734.268850]  [<c013b7d0>] kthread+0x0/0x70
[187734.268854]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[187734.268861]  =======================
[187745.853420] BUG: soft lockup - CPU#0 stuck for 11s! [kswapd0:95]
[187745.853429]
[187745.853436] Pid: 95, comm: kswapd0 Tainted: G      D (2.6.24-24-xen #1)
[187745.853443] EIP: 0061:[<c03285f7>] EFLAGS: 00000286 CPU: 0
[187745.853454] EIP is at _spin_lock+0x7/0x10
[187745.853460] EAX: cf802dd8 EBX: cf802d94 ECX: c17a33c0 EDX: 00000000
[187745.853466] ESI: c17a33c0 EDI: 00000000 EBP: cfdeff7c ESP: cfdefdcc
[187745.853473]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
[187745.853482] CR0: 8005003b CR2: 00100100 CR3: 0f5b7000 CR4: 00000660
[187745.853489] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[187745.853496] DR6: ffff0ff0 DR7: 00000400
[187745.853502]  [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[187745.853520]  [<c0167c75>] shrink_page_list+0x4c5/0x600
[187745.853537]  [<c011cdb3>] dequeue_entity+0x13/0x40
[187745.853555]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187745.853569]  [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[187745.853583]  [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[187745.853605]  [<c01681fc>] shrink_zone+0x9c/0x100
[187745.853619]  [<c016889c>] kswapd+0x44c/0x490
[187745.853636]  [<c013bad0>] autoremove_wake_function+0x0/0x40
[187745.853654]  [<c011e270>] complete+0x40/0x60
[187745.853667]  [<c0168450>] kswapd+0x0/0x490
[187745.853672]  [<c013b812>] kthread+0x42/0x70
[187745.853677]  [<c013b7d0>] kthread+0x0/0x70
[187745.853682]  [<c0105bb7>] kernel_thread_helper+0x7/0x10
[187745.853689]  =======================

Does anyone know what to do there?

Thanks a lot in advance for any reply.

Marco Weber
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users