WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] pvops domu soft lockup under load (more logs)

To: Pim van Riezen <pi+lists@xxxxxxxxxxxx>
Subject: Re: [Xen-devel] pvops domu soft lockup under load (more logs)
From: Jeremy Fitzhardinge <jeremy@xxxxxxxx>
Date: Fri, 09 Apr 2010 11:43:28 -0700
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Fri, 09 Apr 2010 11:45:12 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <2F17645D-999B-435C-97EE-508D39B71035@xxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <C6F9EF58-539C-4B05-ACE1-8C96056944E8@xxxxxxxxxxxx> <2F17645D-999B-435C-97EE-508D39B71035@xxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.9) Gecko/20100330 Fedora/3.0.4-1.fc12 Lightning/1.0b2pre Thunderbird/3.0.4
On 04/09/2010 09:50 AM, Pim van Riezen wrote:
> Some more logs of this bug in action. Stack seems to be all over the place.
>   

What kernel is this?  Did you build it, or does it come from a distro?

What devices are you using within the domain?  What storage backend are
you using?

Could you include the full kernel output?

When you're getting lockups, could you capture the output of sysrq-t (xm
sysrq <domain> t, or "Ctrl-O t" on the domain's console)?

Thanks,
    J

>   
>> Apr  9 08:21:01 host kernel: BUG: soft lockup - CPU#1 stuck for 628s! 
>> [gzip:27658]
>> Apr  9 08:21:01 host kernel: Modules linked in:
>> Apr  9 08:21:01 host kernel: CPU 1:
>> Apr  9 08:21:01 host kernel: Modules linked in:
>> Apr  9 08:21:01 host kernel: Pid: 27658, comm: gzip Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 08:21:01 host kernel: RIP: e033:[<0000000000401958>]  
>> [<0000000000401958>] 0x401958
>> Apr  9 08:21:01 host kernel: RSP: e02b:00007fff026d7e38  EFLAGS: 00000297
>> Apr  9 08:21:01 host kernel: RAX: 0000000000000020 RBX: 0000000000000002 
>> RCX: 0000000000657ce8
>> Apr  9 08:21:01 host kernel: RDX: 0000000000000048 RSI: 0000000000000002 
>> RDI: 000000000000da48
>> Apr  9 08:21:01 host kernel: RBP: 0000000000006a6c R08: 0000000000000034 
>> R09: 000000000000da48
>> Apr  9 08:21:01 host kernel: R10: 0000000000000048 R11: 0000000000000080 
>> R12: 0000000000658c06
>> Apr  9 08:21:01 host kernel: R13: 0000000000658d08 R14: 0000000000658c06 
>> R15: 0000000000000080
>> Apr  9 08:21:01 host kernel: FS:  00007fa6088946e0(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 08:21:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 08:21:01 host kernel: CR2: 00007f7addede020 CR3: 00000000267db000 
>> CR4: 0000000000002660
>> Apr  9 08:21:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 08:21:02 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 08:21:02 host kernel: Call Trace:
>> Apr  9 08:29:39 host kernel: BUG: soft lockup - CPU#1 stuck for 483s! 
>> [swapper:0]
>> Apr  9 08:29:39 host kernel: Modules linked in:
>> Apr  9 08:29:39 host kernel: CPU 1:
>> Apr  9 08:29:39 host kernel: Modules linked in:
>> Apr  9 08:29:39 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 08:29:39 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 08:29:39 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 08:29:39 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 08:29:39 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 08:29:39 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 08:29:39 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 08:29:39 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 08:29:39 host kernel: FS:  00007fa913f42720(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 08:29:39 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 08:29:39 host kernel: CR2: 00007ff39862b1f0 CR3: 0000000052c5e000 
>> CR4: 0000000000002660
>> Apr  9 08:29:39 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 08:29:39 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 08:29:39 host kernel: Call Trace:
>> Apr  9 08:29:39 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 08:29:39 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 08:29:39 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 08:29:39 host kernel: BUG: soft lockup - CPU#0 stuck for 483s! 
>> [swapper:0]
>> Apr  9 08:29:39 host kernel: Modules linked in:
>> Apr  9 08:29:39 host kernel: CPU 0:
>> Apr  9 08:29:40 host kernel: Modules linked in:
>> Apr  9 08:29:40 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 08:29:40 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 08:29:40 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 08:29:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 08:29:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 08:29:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 08:29:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 08:29:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 08:29:40 host kernel: FS:  00007f07704fe6e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 08:29:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 08:29:40 host kernel: CR2: 00007ff39e4a8000 CR3: 000000007ea99000 
>> CR4: 0000000000002660
>> Apr  9 08:29:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 08:29:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 08:29:40 host kernel: Call Trace:
>> Apr  9 08:29:40 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 08:29:40 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 08:29:40 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 08:29:40 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 08:34:34 host kernel: BUG: soft lockup - CPU#1 stuck for 275s! 
>> [ssh:2324]
>> Apr  9 08:34:34 host kernel: Modules linked in:
>> Apr  9 08:34:34 host kernel: CPU 1:
>> Apr  9 08:34:34 host kernel: Modules linked in:
>> Apr  9 08:34:34 host kernel: Pid: 2324, comm: ssh Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 08:34:34 host kernel: RIP: e033:[<00007f1b0fcac76a>]  
>> [<00007f1b0fcac76a>] 0x7f1b0fcac76a
>> Apr  9 08:34:34 host kernel: RSP: e02b:00007fffa45a7480  EFLAGS: 00010202
>> Apr  9 08:34:34 host kernel: RAX: 0000000000000000 RBX: 0000000000000000 
>> RCX: 0000000000000181
>> Apr  9 08:34:34 host kernel: RDX: 00007f1b0fcdca28 RSI: 0000000000000013 
>> RDI: 000000000000000a
>> Apr  9 08:34:34 host kernel: RBP: 0000000000000010 R08: 0000000000000004 
>> R09: 0101010101010101
>> Apr  9 08:34:34 host kernel: R10: 737361702e707466 R11: 00007f1b0fcac690 
>> R12: 0000000000000010
>> Apr  9 08:34:34 host kernel: R13: 00007f1b11f825e0 R14: 0000000000000001 
>> R15: 0000000000000004
>> Apr  9 08:34:34 host kernel: FS:  00007f1b10528710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 08:34:34 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006ad44000 
>> CR4: 0000000000002660
>> Apr  9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 08:34:34 host kernel: Call Trace:
>> Apr  9 08:34:34 host kernel: BUG: soft lockup - CPU#0 stuck for 275s! 
>> [swapper:0]
>> Apr  9 08:34:34 host kernel: Modules linked in:
>> Apr  9 08:34:34 host kernel: CPU 0:
>> Apr  9 08:34:34 host kernel: Modules linked in:
>> Apr  9 08:34:34 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 08:34:34 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 08:34:34 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 08:34:34 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 08:34:34 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 08:34:34 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 08:34:34 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 08:34:34 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 08:34:34 host kernel: FS:  00007fa913f42720(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 08:34:34 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006acb1000 
>> CR4: 0000000000002660
>> Apr  9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 08:34:34 host kernel: Call Trace:
>> Apr  9 08:34:34 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 08:34:34 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 08:34:34 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 08:34:34 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 08:39:49 host kernel: BUG: soft lockup - CPU#1 stuck for 293s! 
>> [gzip:30581]
>> Apr  9 08:39:49 host kernel: Modules linked in:
>> Apr  9 08:39:49 host kernel: CPU 1:
>> Apr  9 08:39:49 host kernel: Modules linked in:
>> Apr  9 08:39:49 host kernel: Pid: 30581, comm: gzip Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 08:39:49 host kernel: RIP: e030:[<ffffffff811ede38>]  
>> [<ffffffff811ede38>] fuse_request_send+0x60/0x22b
>> Apr  9 08:39:49 host kernel: RSP: e02b:ffff88007952bc28  EFLAGS: 00000202
>> Apr  9 08:39:49 host kernel: RAX: ffff880079652df0 RBX: ffff88007e07d2c0 
>> RCX: fffffffffffdfb5e
>> Apr  9 08:39:49 host kernel: RDX: 0000000000020001 RSI: 000000000000001d 
>> RDI: ffff88007eb1f710
>> Apr  9 08:39:49 host kernel: RBP: ffff88007e07d2c0 R08: 000000037ede1c40 
>> R09: ffff8800593170d8
>> Apr  9 08:39:49 host kernel: R10: 0000000000000000 R11: 000000000000000f 
>> R12: ffff88007eb1f400
>> Apr  9 08:39:49 host kernel: R13: ffff88007e07d2e0 R14: 0000000000001000 
>> R15: 0000000000001000
>> Apr  9 08:39:49 host kernel: FS:  00007f7b12f606e0(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 08:39:49 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 08:39:49 host kernel: CR2: 00000000006d9138 CR3: 00000000264f7000 
>> CR4: 0000000000002660
>> Apr  9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 08:39:49 host kernel: Call Trace:
>> Apr  9 08:39:49 host kernel:  [<ffffffff811ede34>] ? 
>> fuse_request_send+0x5c/0x22b
>> Apr  9 08:39:49 host kernel:  [<ffffffff811f1655>] ? 
>> fuse_wait_on_page_writeback+0x31/0x168
>> Apr  9 08:39:49 host kernel:  [<ffffffff810904b4>] ? 
>> add_to_page_cache_lru+0x41/0x58
>> Apr  9 08:39:49 host kernel:  [<ffffffff811f186a>] ? 
>> fuse_send_write+0xb8/0xc2
>> Apr  9 08:39:49 host kernel:  [<ffffffff811f29d7>] ? 
>> fuse_file_aio_write+0x497/0x62f
>> Apr  9 08:39:49 host kernel:  [<ffffffff810bab7e>] ? do_sync_write+0xce/0x113
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100f6af>] ? __switch_to+0x1ad/0x297
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ? 
>> autoremove_wake_function+0x0/0x2e
>> Apr  9 08:39:49 host kernel:  [<ffffffff8103b070>] ? 
>> finish_task_switch+0x27/0x6e
>> Apr  9 08:39:49 host kernel:  [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
>> Apr  9 08:39:49 host kernel:  [<ffffffff810bb4b2>] ? vfs_write+0xad/0x149
>> Apr  9 08:39:49 host kernel:  [<ffffffff810bb60a>] ? sys_write+0x45/0x6e
>> Apr  9 08:39:49 host kernel:  [<ffffffff81010ac2>] ? 
>> system_call_fastpath+0x16/0x1b
>> Apr  9 08:39:49 host kernel: BUG: soft lockup - CPU#0 stuck for 293s! 
>> [sshfs:2332]
>> Apr  9 08:39:49 host kernel: Modules linked in:
>> Apr  9 08:39:49 host kernel: CPU 0:
>> Apr  9 08:39:49 host kernel: Modules linked in:
>> Apr  9 08:39:49 host kernel: Pid: 2332, comm: sshfs Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 08:39:49 host kernel: RIP: e030:[<ffffffff813b93bd>]  
>> [<ffffffff813b93bd>] sk_reset_timer+0x1/0x1d
>> Apr  9 08:39:49 host kernel: RSP: e02b:ffff8800022afbc0  EFLAGS: 00000206
>> Apr  9 08:39:49 host kernel: RAX: 0000000101b312c6 RBX: 0000000000000080 
>> RCX: 0000000101b312ce
>> Apr  9 08:39:49 host kernel: RDX: 0000000101b312ce RSI: ffff88007d1dc4d0 
>> RDI: ffff88007d1dc1c0
>> Apr  9 08:39:49 host kernel: RBP: ffff88007d1dc1c0 R08: 000000007d1dc10f 
>> R09: 0000000101b312ce
>> Apr  9 08:39:49 host kernel: R10: ffff880068b8f1c0 R11: ffffffff8126167a 
>> R12: ffff880068b8f1c0
>> Apr  9 08:39:49 host kernel: R13: 0000000000000000 R14: 0000000000000020 
>> R15: ffff880068b8f1f8
>> Apr  9 08:39:49 host kernel: FS:  00007f1b10528710(0063) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 08:39:49 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 08:39:49 host kernel: CR2: 00007f66a4ede000 CR3: 000000006acb1000 
>> CR4: 0000000000002660
>> Apr  9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 08:39:49 host kernel: Call Trace:
>> Apr  9 08:39:49 host kernel:  <IRQ>  [<ffffffff81419140>] ? 
>> tcp_rcv_established+0x587/0x87b
>> Apr  9 08:39:49 host kernel:  [<ffffffff8141f7f1>] ? tcp_v4_do_rcv+0x2c/0x1c9
>> Apr  9 08:39:49 host kernel:  [<ffffffff81046156>] ? 
>> local_bh_enable+0x22/0x8f
>> Apr  9 08:39:49 host kernel:  [<ffffffff8141fe05>] ? tcp_v4_rcv+0x477/0x6f8
>> Apr  9 08:39:49 host kernel:  [<ffffffff814064f7>] ? 
>> ip_local_deliver_finish+0xf2/0x1c7
>> Apr  9 08:39:49 host kernel:  [<ffffffff81406126>] ? 
>> ip_rcv_finish+0x326/0x340
>> Apr  9 08:39:49 host kernel:  [<ffffffff814063c0>] ? ip_rcv+0x280/0x2c5
>> Apr  9 08:39:49 host kernel:  [<ffffffff8135d39d>] ? xennet_poll+0xb06/0xc68
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100e39f>] ? 
>> xen_restore_fl_direct_end+0x0/0x1
>> Apr  9 08:39:49 host kernel:  [<ffffffff81334afc>] ? 
>> do_blkif_request+0x341/0x350
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 08:39:49 host kernel:  [<ffffffff813c469b>] ? net_rx_action+0x77/0x179
>> Apr  9 08:39:49 host kernel:  [<ffffffff810462b4>] ? __do_softirq+0xdd/0x1a1
>> Apr  9 08:39:49 host kernel:  [<ffffffff81011bec>] ? call_softirq+0x1c/0x30
>> Apr  9 08:39:49 host kernel:  [<ffffffff81012edf>] ? do_softirq+0x3f/0x7c
>> Apr  9 08:39:49 host kernel:  [<ffffffff812ff975>] ? 
>> xen_evtchn_do_upcall+0x173/0x190
>> Apr  9 08:39:49 host kernel:  [<ffffffff81011c3e>] ? 
>> xen_do_hypervisor_callback+0x1e/0x30
>> Apr  9 08:39:49 host kernel:  <EOI>  [<ffffffff8100e3b2>] ? 
>> check_events+0x12/0x20
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ? 
>> hypercall_page+0x22a/0x1001
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ? 
>> hypercall_page+0x22a/0x1001
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100dd39>] ? 
>> xen_force_evtchn_callback+0x9/0xa
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100e359>] ? 
>> xen_irq_enable_direct_end+0x0/0x7
>> Apr  9 08:39:49 host kernel:  [<ffffffff8103b070>] ? 
>> finish_task_switch+0x27/0x6e
>> Apr  9 08:39:49 host kernel:  [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
>> Apr  9 08:39:49 host kernel:  [<ffffffff8103d336>] ? 
>> default_wake_function+0x0/0x9
>> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ? 
>> cap_socket_recvmsg+0x0/0x3
>> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ? 
>> cap_socket_recvmsg+0x0/0x3
>> Apr  9 08:39:49 host kernel:  [<ffffffff814d6e59>] ? 
>> schedule_timeout+0x23/0x1d9
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100dd39>] ? 
>> xen_force_evtchn_callback+0x9/0xa
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ? 
>> cap_socket_recvmsg+0x0/0x3
>> Apr  9 08:39:49 host kernel:  [<ffffffff814d824c>] ? 
>> _spin_unlock_irqrestore+0xc/0xd
>> Apr  9 08:39:49 host kernel:  [<ffffffff81456ebe>] ? 
>> unix_stream_recvmsg+0x2c4/0x644
>> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ? 
>> autoremove_wake_function+0x0/0x2e
>> Apr  9 08:39:49 host kernel:  [<ffffffff81457231>] ? 
>> unix_stream_recvmsg+0x637/0x644
>> Apr  9 08:39:49 host kernel:  [<ffffffff813b6303>] ? 
>> sock_aio_read+0x103/0x117
>> Apr  9 08:39:49 host kernel:  [<ffffffff810bac91>] ? do_sync_read+0xce/0x113
>> Apr  9 08:39:49 host kernel:  [<ffffffff81011c1b>] ? 
>> xen_hypervisor_callback+0x1b/0x20
>> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ? 
>> autoremove_wake_function+0x0/0x2e
>> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ? 
>> cap_socket_recvmsg+0x0/0x3
>> Apr  9 08:39:49 host kernel:  [<ffffffff814d824c>] ? 
>> _spin_unlock_irqrestore+0xc/0xd
>> Apr  9 08:39:49 host kernel:  [<ffffffff81011c1b>] ? 
>> xen_hypervisor_callback+0x1b/0x20
>> Apr  9 08:39:49 host kernel:  [<ffffffff810bb6f0>] ? vfs_read+0xbd/0x146
>> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ? 
>> hypercall_page+0x22a/0x1001
>> Apr  9 08:39:49 host kernel:  [<ffffffff810bb835>] ? sys_read+0x45/0x6e
>> Apr  9 08:39:49 host kernel:  [<ffffffff81010ac2>] ? 
>> system_call_fastpath+0x16/0x1b
>> Apr  9 09:11:36 host kernel: BUG: soft lockup - CPU#1 stuck for 1776s! 
>> [swapper:0]
>> Apr  9 09:11:36 host kernel: Modules linked in:
>> Apr  9 09:11:36 host kernel: CPU 1:
>> Apr  9 09:11:36 host kernel: Modules linked in:
>> Apr  9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 09:11:36 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 09:11:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 09:11:36 host kernel: FS:  00007f44bc51f710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 09:11:36 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 09:11:36 host kernel: CR2: 00007f44bc52c000 CR3: 0000000071c86000 
>> CR4: 0000000000002660
>> Apr  9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 09:11:36 host kernel: Call Trace:
>> Apr  9 09:11:36 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 09:11:36 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 09:11:36 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 09:11:36 host kernel: BUG: soft lockup - CPU#0 stuck for 1776s! 
>> [swapper:0]
>> Apr  9 09:11:36 host kernel: Modules linked in:
>> Apr  9 09:11:36 host kernel: CPU 0:
>> Apr  9 09:11:36 host kernel: Modules linked in:
>> Apr  9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 09:11:36 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 09:11:36 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 09:11:36 host kernel: FS:  00007f07704fe6e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 09:11:36 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 09:11:36 host kernel: CR2: 00000000004884b2 CR3: 0000000071c86000 
>> CR4: 0000000000002660
>> Apr  9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 09:11:36 host kernel: Call Trace:
>> Apr  9 09:11:36 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 09:11:36 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 09:11:36 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 09:11:36 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 09:20:24 host kernel: BUG: soft lockup - CPU#1 stuck for 492s! 
>> [ssh:2324]
>> Apr  9 09:20:24 host kernel: Modules linked in:
>> Apr  9 09:20:24 host kernel: CPU 1:
>> Apr  9 09:20:24 host kernel: Modules linked in:
>> Apr  9 09:20:24 host kernel: Pid: 2324, comm: ssh Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 09:20:24 host kernel: RIP: e033:[<00007f1b0e259f20>]  
>> [<00007f1b0e259f20>] 0x7f1b0e259f20
>> Apr  9 09:20:24 host kernel: RSP: e02b:00007fffa45a7338  EFLAGS: 00000206
>> Apr  9 09:20:24 host kernel: RAX: 00007f1b11f32b20 RBX: 00007f1b0e5399c0 
>> RCX: 00000000000000a0
>> Apr  9 09:20:24 host kernel: RDX: 00007f1b11f32b80 RSI: 0000000000000004 
>> RDI: 00007f1b0e539c30
>> Apr  9 09:20:24 host kernel: RBP: 0000000000000064 R08: 0000000000000001 
>> R09: 0000000000000001
>> Apr  9 09:20:24 host kernel: R10: 00007f1b0e539a30 R11: 00007f1b0e25bfb0 
>> R12: 0000000000000000
>> Apr  9 09:20:24 host kernel: R13: 00007f1b0fcd2aec R14: 0000000000000111 
>> R15: 0000000000000071
>> Apr  9 09:20:24 host kernel: FS:  00007f1b10528710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 09:20:24 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 09:20:24 host kernel: CR2: 00007fa913f53000 CR3: 000000006ad44000 
>> CR4: 0000000000002660
>> Apr  9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 09:20:24 host kernel: Call Trace:
>> Apr  9 09:20:24 host kernel: BUG: soft lockup - CPU#0 stuck for 492s! 
>> [swapper:0]
>> Apr  9 09:20:24 host kernel: Modules linked in:
>> Apr  9 09:20:24 host kernel: CPU 0:
>> Apr  9 09:20:24 host kernel: Modules linked in:
>> Apr  9 09:20:24 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 09:20:24 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 09:20:24 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 09:20:24 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 09:20:24 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 09:20:24 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 09:20:24 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 09:20:24 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 09:20:24 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 09:20:24 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 09:20:24 host kernel: CR2: 00007fff892a1e30 CR3: 000000006acb1000 
>> CR4: 0000000000002660
>> Apr  9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 09:20:24 host kernel: Call Trace:
>> Apr  9 09:20:24 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 09:20:24 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 09:20:24 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 09:20:24 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 09:24:36 host kernel: BUG: soft lockup - CPU#1 stuck for 234s! 
>> [gzip:4496]
>> Apr  9 09:24:36 host kernel: Modules linked in:
>> Apr  9 09:24:36 host kernel: CPU 1:
>> Apr  9 09:24:36 host kernel: Modules linked in:
>> Apr  9 09:24:36 host kernel: Pid: 4496, comm: gzip Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 09:24:36 host kernel: RIP: e033:[<0000000000409804>]  
>> [<0000000000409804>] 0x409804
>> Apr  9 09:24:36 host kernel: RSP: e02b:00007fff87e009f8  EFLAGS: 00000202
>> Apr  9 09:24:36 host kernel: RAX: 00000000f6019e6d RBX: 0000000000008000 
>> RCX: 00007f529598c610
>> Apr  9 09:24:36 host kernel: RDX: 0000000000f6019e RSI: 00000000000045a6 
>> RDI: 00000000006534fb
>> Apr  9 09:24:36 host kernel: RBP: 0000000000005800 R08: 0000000000004000 
>> R09: 7464697720223333
>> Apr  9 09:24:36 host kernel: R10: 6c61202236223d68 R11: 0000000000000246 
>> R12: 00000000006522a0
>> Apr  9 09:24:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 09:24:36 host kernel: FS:  00007f5295e2a6e0(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 09:24:36 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 09:24:36 host kernel: CR2: 0000000000d550c8 CR3: 0000000026474000 
>> CR4: 0000000000002660
>> Apr  9 09:24:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 09:24:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 09:24:37 host kernel: Call Trace:
>> Apr  9 09:24:37 host kernel: BUG: soft lockup - CPU#0 stuck for 234s! 
>> [swapper:0]
>> Apr  9 09:24:37 host kernel: Modules linked in:
>> Apr  9 09:24:37 host kernel: CPU 0:
>> Apr  9 09:24:37 host kernel: Modules linked in:
>> Apr  9 09:24:37 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 09:24:37 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 09:24:38 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 09:24:38 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 09:24:38 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 09:24:38 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 09:24:38 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 09:24:38 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 09:24:38 host kernel: FS:  00007f2f6a31a710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 09:24:38 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 09:24:38 host kernel: CR2: 00007f2f6a327000 CR3: 000000003f62e000 
>> CR4: 0000000000002660
>> Apr  9 09:24:38 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 09:24:38 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 09:24:38 host kernel: Call Trace:
>> Apr  9 09:24:38 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 09:24:38 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 09:24:38 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 09:24:38 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 10:05:51 host kernel: BUG: soft lockup - CPU#1 stuck for 2305s! 
>> [swapper:0]
>> Apr  9 10:05:51 host kernel: Modules linked in:
>> Apr  9 10:05:51 host kernel: CPU 1:
>> Apr  9 10:05:51 host kernel: Modules linked in:
>> Apr  9 10:05:51 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 10:05:51 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 10:05:51 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 10:05:51 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 10:05:51 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 10:05:51 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 10:05:51 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 10:05:51 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 10:05:51 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 10:05:51 host kernel: CR2: 0000000004786158 CR3: 000000007e2a6000 
>> CR4: 0000000000002660
>> Apr  9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 10:05:51 host kernel: Call Trace:
>> Apr  9 10:05:51 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 10:05:51 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 10:05:51 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 10:05:51 host kernel: BUG: soft lockup - CPU#0 stuck for 2305s! 
>> [httpd:9922]
>> Apr  9 10:05:51 host kernel: Modules linked in:
>> Apr  9 10:05:51 host kernel: CPU 0:
>> Apr  9 10:05:51 host kernel: Modules linked in:
>> Apr  9 10:05:51 host kernel: Pid: 9922, comm: httpd Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 10:05:51 host kernel: RIP: e033:[<00007ff39a44a36d>]  
>> [<00007ff39a44a36d>] 0x7ff39a44a36d
>> Apr  9 10:05:51 host kernel: RSP: e02b:00007fff4b9ad118  EFLAGS: 00000a86
>> Apr  9 10:05:51 host kernel: RAX: c0c176e403831574 RBX: 0000000003643b60 
>> RCX: c0c176e403831574
>> Apr  9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000004 
>> RDI: 000000000420e43b
>> Apr  9 10:05:51 host kernel: RBP: 0000000003643b60 R08: 0000000000000004 
>> R09: 00007fff4b9ad270
>> Apr  9 10:05:51 host kernel: R10: 00007ff39aaa9fa0 R11: 00007fff4b9ad188 
>> R12: 000000000320e6d0
>> Apr  9 10:05:51 host kernel: R13: 0000000000000000 R14: 00007fff4b9ad368 
>> R15: 0000000000000000
>> Apr  9 10:05:51 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 10:05:51 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 10:05:51 host kernel: CR2: 00000000047a4018 CR3: 000000007972c000 
>> CR4: 0000000000002660
>> Apr  9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 10:05:51 host kernel: Call Trace:
>> Apr  9 10:50:04 host kernel: BUG: soft lockup - CPU#1 stuck for 2471s! 
>> [swapper:0]
>> Apr  9 10:50:04 host kernel: Modules linked in:
>> Apr  9 10:50:04 host kernel: CPU 1:
>> Apr  9 10:50:04 host kernel: Modules linked in:
>> Apr  9 10:50:04 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 10:50:04 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 10:50:04 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 10:50:04 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 10:50:04 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 10:50:04 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 10:50:04 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 10:50:04 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 10:50:04 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 10:50:04 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 10:50:04 host kernel: CR2: 00007ff398a0ce08 CR3: 000000004ddb4000 
>> CR4: 0000000000002660
>> Apr  9 10:50:04 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 10:50:04 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 10:50:04 host kernel: Call Trace:
>> Apr  9 10:50:04 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 10:50:04 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 10:50:04 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 10:50:04 host kernel: BUG: soft lockup - CPU#0 stuck for 2471s! 
>> [httpd:16797]
>> Apr  9 10:50:04 host kernel: Modules linked in:
>> Apr  9 10:50:04 host kernel: CPU 0:
>> Apr  9 10:50:04 host kernel: Modules linked in:
>> Apr  9 10:50:04 host kernel: Pid: 16797, comm: httpd Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 10:50:04 host kernel: RIP: e033:[<00007ff39a49ae55>]  
>> [<00007ff39a49ae55>] 0x7ff39a49ae55
>> Apr  9 10:50:04 host kernel: RSP: e02b:00007fff4b9b10b8  EFLAGS: 00000246
>> Apr  9 10:50:04 host kernel: RAX: 000000000368dd40 RBX: 00000000036868d0 
>> RCX: 222879726575713e
>> Apr  9 10:50:05 host kernel: RDX: 000000000000000c RSI: 0000000003686a6c 
>> RDI: 00007fff4b9b1d00
>> Apr  9 10:50:05 host kernel: RBP: 00007fff4b9b1da0 R08: 0000000000000001 
>> R09: 0101010101010101
>> Apr  9 10:50:05 host kernel: R10: 0000000000000001 R11: 00007ff39c4ac0be 
>> R12: 00007fff4b9b1d00
>> Apr  9 10:50:05 host kernel: R13: 00007ff39aaa9fa0 R14: 00007ff39aaa9fa0 
>> R15: 0000000003695df0
>> Apr  9 10:50:05 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 10:50:05 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 10:50:05 host kernel: CR2: 00000000032068c0 CR3: 000000004ddb4000 
>> CR4: 0000000000002660
>> Apr  9 10:50:05 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 10:50:05 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 10:50:05 host kernel: Call Trace:
>> Apr  9 10:57:01 host kernel: BUG: soft lockup - CPU#1 stuck for 389s! 
>> [exim:17977]
>> Apr  9 10:57:01 host kernel: Modules linked in:
>> Apr  9 10:57:01 host kernel: CPU 1:
>> Apr  9 10:57:01 host kernel: Modules linked in:
>> Apr  9 10:57:01 host kernel: Pid: 17977, comm: exim Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 10:57:01 host kernel: RIP: e030:[<ffffffff8100922a>]  
>> [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
>> Apr  9 10:57:01 host kernel: RSP: e02b:ffff880056611ea0  EFLAGS: 00000246
>> Apr  9 10:57:01 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 
>> RCX: ffffffff8100922a
>> Apr  9 10:57:01 host kernel: RDX: 00007fd75a6d614e RSI: 0000000000000000 
>> RDI: 0000000000000000
>> Apr  9 10:57:01 host kernel: RBP: 00007fd75a6d614e R08: 00007fd75c048710 
>> R09: 0000000000000003
>> Apr  9 10:57:01 host kernel: R10: 00007fd75a909a30 R11: 0000000000000246 
>> R12: 0000000000000004
>> Apr  9 10:57:01 host kernel: R13: ffff880056611f58 R14: ffff88005aa996d0 
>> R15: ffff88005a51b3c0
>> Apr  9 10:57:01 host kernel: FS:  00007fd75c048710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 10:57:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 10:57:01 host kernel: CR2: 00007fd75a6d614e CR3: 000000007eb9a000 
>> CR4: 0000000000002660
>> Apr  9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 10:57:01 host kernel: Call Trace:
>> Apr  9 10:57:01 host kernel:  [<ffffffff8100dd39>] ? 
>> xen_force_evtchn_callback+0x9/0xa
>> Apr  9 10:57:01 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 10:57:01 host kernel:  [<ffffffff8100e359>] ? 
>> xen_irq_enable_direct_end+0x0/0x7
>> Apr  9 10:57:01 host kernel:  [<ffffffff814da477>] ? do_page_fault+0xfb/0x28d
>> Apr  9 10:57:01 host kernel:  [<ffffffff814d85b5>] ? page_fault+0x25/0x30
>> Apr  9 10:57:01 host kernel: BUG: soft lockup - CPU#0 stuck for 389s! 
>> [netstat:17966]
>> Apr  9 10:57:01 host kernel: Modules linked in:
>> Apr  9 10:57:01 host kernel: CPU 0:
>> Apr  9 10:57:01 host kernel: Modules linked in:
>> Apr  9 10:57:01 host kernel: Pid: 17966, comm: netstat Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 10:57:01 host kernel: RIP: e030:[<ffffffff814d8226>]  
>> [<ffffffff814d8226>] _spin_lock+0xa/0x15
>> Apr  9 10:57:01 host kernel: RSP: e02b:ffff88006ad4dce0  EFLAGS: 00000202
>> Apr  9 10:57:01 host kernel: RAX: 0000000000001c1c RBX: ffff88007ff77180 
>> RCX: 0000000000000012
>> Apr  9 10:57:01 host kernel: RDX: 018721dfffc35c9f RSI: ffff88006ad4ddc8 
>> RDI: ffff88007ff77188
>> Apr  9 10:57:01 host kernel: RBP: ffff88007ff77198 R08: 0000000000000000 
>> R09: 0000000000300000
>> Apr  9 10:57:01 host kernel: R10: 00000000fffffff2 R11: ffffffff8126158e 
>> R12: ffff88000ff859c0
>> Apr  9 10:57:01 host kernel: R13: ffff88006ad4ddc8 R14: 0000000000034b38 
>> R15: 0000000000000002
>> Apr  9 10:57:01 host kernel: FS:  00007fb435545710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 10:57:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 10:57:01 host kernel: CR2: 00007f0e429d6000 CR3: 000000007ee3d000 
>> CR4: 0000000000002660
>> Apr  9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 10:57:01 host kernel: Call Trace:
>> Apr  9 10:57:01 host kernel:  [<ffffffff810cb01b>] ? __d_lookup+0x7a/0x105
>> Apr  9 10:57:01 host kernel:  [<ffffffff810c251e>] ? do_lookup+0x2a/0x1b3
>> Apr  9 10:57:01 host kernel:  [<ffffffff810c4498>] ? 
>> __link_path_walk+0x348/0xdcd
>> Apr  9 10:57:01 host kernel:  [<ffffffff8100dd39>] ? 
>> xen_force_evtchn_callback+0x9/0xa
>> Apr  9 10:57:01 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 10:57:01 host kernel:  [<ffffffff810c514b>] ? path_walk+0x66/0xca
>> Apr  9 10:57:01 host kernel:  [<ffffffff810c51cf>] ? do_path_lookup+0x20/0x77
>> Apr  9 10:57:01 host kernel:  [<ffffffff810c5ae9>] ? user_path_at+0x48/0x79
>> Apr  9 10:57:01 host kernel:  [<ffffffff8100e39f>] ? 
>> xen_restore_fl_direct_end+0x0/0x1
>> Apr  9 10:57:01 host kernel:  [<ffffffff810cfd6b>] ? 
>> mntput_no_expire+0x23/0xed
>> Apr  9 10:57:01 host kernel:  [<ffffffff8129b703>] ? 
>> _atomic_dec_and_lock+0x33/0x50
>> Apr  9 10:57:01 host kernel:  [<ffffffff810cfd6b>] ? 
>> mntput_no_expire+0x23/0xed
>> Apr  9 10:57:01 host kernel:  [<ffffffff810bdf4d>] ? sys_readlinkat+0x25/0x8d
>> Apr  9 10:57:01 host kernel:  [<ffffffff81010ac2>] ? 
>> system_call_fastpath+0x16/0x1b
>> Apr  9 11:00:36 host kernel: BUG: soft lockup - CPU#1 stuck for 200s! 
>> [swapper:0]
>> Apr  9 11:00:36 host kernel: Modules linked in:
>> Apr  9 11:00:36 host kernel: CPU 1:
>> Apr  9 11:00:36 host kernel: Modules linked in:
>> Apr  9 11:00:36 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 11:00:36 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 11:00:36 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 11:00:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 11:00:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 11:00:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 11:00:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 11:00:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 11:00:36 host kernel: FS:  00007f719f5fa6e0(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 11:00:36 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 11:00:36 host kernel: CR2: 00007ff39c43cfa8 CR3: 000000007d0db000 
>> CR4: 0000000000002660
>> Apr  9 11:00:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 11:00:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 11:00:37 host kernel: Call Trace:
>> Apr  9 11:00:37 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 11:00:37 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 11:00:37 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 11:00:37 host kernel: BUG: soft lockup - CPU#0 stuck for 200s! 
>> [swapper:0]
>> Apr  9 11:00:37 host kernel: Modules linked in:
>> Apr  9 11:00:37 host kernel: CPU 0:
>> Apr  9 11:00:37 host kernel: Modules linked in:
>> Apr  9 11:00:37 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 11:00:37 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 11:00:37 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 11:00:37 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 11:00:37 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 11:00:37 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 11:00:37 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 11:00:37 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 11:00:37 host kernel: FS:  00007f07704fe6e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 11:00:37 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 11:00:37 host kernel: CR2: 00000000035750c8 CR3: 000000007ea99000 
>> CR4: 0000000000002660
>> Apr  9 11:00:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 11:00:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 11:00:37 host kernel: Call Trace:
>> Apr  9 11:00:37 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 11:00:37 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 11:00:37 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 11:00:37 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 11:08:50 host kernel: BUG: soft lockup - CPU#0 stuck for 460s! 
>> [httpd:19360]
>> Apr  9 11:08:50 host kernel: Modules linked in:
>> Apr  9 11:08:50 host kernel: CPU 0:
>> Apr  9 11:08:50 host kernel: Modules linked in:
>> Apr  9 11:08:50 host kernel: Pid: 19360, comm: httpd Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 11:08:50 host kernel: RIP: e030:[<ffffffff813b995e>]  
>> [<ffffffff813b995e>] sock_alloc_send_skb+0x3/0xc
>> Apr  9 11:08:50 host kernel: RSP: e02b:ffff88005675dc80  EFLAGS: 00000246
>> Apr  9 11:08:50 host kernel: RAX: 0000000000003e00 RBX: ffff88005675dcb8 
>> RCX: ffff88005675dce4
>> Apr  9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000057 
>> RDI: ffff88002678c900
>> Apr  9 11:08:50 host kernel: RBP: ffff88007f740b80 R08: ffff88005675dce4 
>> R09: 75646f72705f6d76
>> Apr  9 11:08:50 host kernel: R10: 5245485720607463 R11: ffffffff81261683 
>> R12: 0000000000000057
>> Apr  9 11:08:50 host kernel: R13: 0000000000000000 R14: ffff88006add3c80 
>> R15: ffff88005675dd60
>> Apr  9 11:08:50 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 11:08:50 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 11:08:50 host kernel: CR2: 00000000045cf000 CR3: 000000007f97d000 
>> CR4: 0000000000002660
>> Apr  9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 11:08:50 host kernel: Call Trace:
>> Apr  9 11:08:50 host kernel:  [<ffffffff814573c8>] ? 
>> unix_stream_sendmsg+0x18a/0x31f
>> Apr  9 11:08:50 host kernel:  [<ffffffff813b61ec>] ? 
>> sock_aio_write+0xf6/0x10a
>> Apr  9 11:08:50 host kernel:  [<ffffffff8100dd39>] ? 
>> xen_force_evtchn_callback+0x9/0xa
>> Apr  9 11:08:50 host kernel:  [<ffffffff8100e39f>] ? 
>> xen_restore_fl_direct_end+0x0/0x1
>> Apr  9 11:08:50 host kernel:  [<ffffffff810bab7e>] ? do_sync_write+0xce/0x113
>> Apr  9 11:08:50 host kernel:  [<ffffffff810c5af3>] ? user_path_at+0x52/0x79
>> Apr  9 11:08:50 host kernel:  [<ffffffff810be0b0>] ? cp_new_stat+0xe9/0xfc
>> Apr  9 11:08:50 host kernel:  [<ffffffff810561b8>] ? 
>> autoremove_wake_function+0x0/0x2e
>> Apr  9 11:08:50 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 11:08:50 host kernel:  [<ffffffff8100dd39>] ? 
>> xen_force_evtchn_callback+0x9/0xa
>> Apr  9 11:08:50 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 11:08:50 host kernel:  [<ffffffff810bb4c5>] ? vfs_write+0xc0/0x149
>> Apr  9 11:08:50 host kernel:  [<ffffffff810bb60a>] ? sys_write+0x45/0x6e
>> Apr  9 11:08:50 host kernel:  [<ffffffff81010ac2>] ? 
>> system_call_fastpath+0x16/0x1b
>> Apr  9 11:08:50 host kernel: BUG: soft lockup - CPU#1 stuck for 460s! 
>> [swapper:0]
>> Apr  9 11:08:50 host kernel: Modules linked in:
>> Apr  9 11:08:50 host kernel: CPU 1:
>> Apr  9 11:08:50 host kernel: Modules linked in:
>> Apr  9 11:08:50 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 11:08:50 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 11:08:50 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 11:08:50 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 11:08:50 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 11:08:50 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 11:08:50 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 11:08:50 host kernel: FS:  00007f0e429cb6e0(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 11:08:50 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 11:08:50 host kernel: CR2: 0000000004598000 CR3: 000000007e2a6000 
>> CR4: 0000000000002660
>> Apr  9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 11:08:50 host kernel: Call Trace:
>> Apr  9 11:08:50 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 11:08:50 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 11:08:50 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 12:00:40 host kernel: BUG: soft lockup - CPU#1 stuck for 2897s! 
>> [httpd:26903]
>> Apr  9 12:00:40 host kernel: Modules linked in:
>> Apr  9 12:00:40 host kernel: CPU 1:
>> Apr  9 12:00:40 host kernel: Modules linked in:
>> Apr  9 12:00:40 host kernel: Pid: 26903, comm: httpd Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 12:00:40 host kernel: RIP: e033:[<00007ff39a427cd2>]  
>> [<00007ff39a427cd2>] 0x7ff39a427cd2
>> Apr  9 12:00:40 host kernel: RSP: e02b:00007fff4b9afa20  EFLAGS: 00000202
>> Apr  9 12:00:40 host kernel: RAX: 0000000000000020 RBX: 00007fff4b9aff30 
>> RCX: 0000000000000013
>> Apr  9 12:00:40 host kernel: RDX: 00007ff39a77be20 RSI: 00007fff4b9aff50 
>> RDI: 0000000003bd6418
>> Apr  9 12:00:40 host kernel: RBP: 0000000000000000 R08: 00007ff39a782e60 
>> R09: 00007ff39a783d80
>> Apr  9 12:00:40 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 
>> R12: 00007ff39aaaa3e0
>> Apr  9 12:00:40 host kernel: R13: 00007fff4b9aff50 R14: 00007fff4b9aff50 
>> R15: 000000000000015e
>> Apr  9 12:00:40 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 12:00:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 12:00:40 host kernel: CR2: 0000000003bdc000 CR3: 0000000056566000 
>> CR4: 0000000000002660
>> Apr  9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 12:00:40 host kernel: Call Trace:
>> Apr  9 12:00:40 host kernel: BUG: soft lockup - CPU#0 stuck for 2897s! 
>> [swapper:0]
>> Apr  9 12:00:40 host kernel: Modules linked in:
>> Apr  9 12:00:40 host kernel: CPU 0:
>> Apr  9 12:00:40 host kernel: Modules linked in:
>> Apr  9 12:00:40 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 12:00:40 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 12:00:40 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 12:00:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 12:00:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 12:00:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 12:00:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 12:00:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 12:00:40 host kernel: FS:  00007f74fde746e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 12:00:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 12:00:40 host kernel: CR2: 00007ff39aec11a7 CR3: 000000007e2a6000 
>> CR4: 0000000000002660
>> Apr  9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 12:00:40 host kernel: Call Trace:
>> Apr  9 12:00:40 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 12:00:40 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 12:00:40 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 12:00:40 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 12:19:45 host kernel: BUG: soft lockup - CPU#1 stuck for 1065s! 
>> [swapper:0]
>> Apr  9 12:19:45 host kernel: Modules linked in:
>> Apr  9 12:19:45 host kernel: CPU 1:
>> Apr  9 12:19:45 host kernel: Modules linked in:
>> Apr  9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 12:19:45 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 12:19:45 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 12:19:45 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 12:19:45 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 12:19:45 host kernel: CR2: 00007ff39c5002b0 CR3: 0000000079588000 
>> CR4: 0000000000002660
>> Apr  9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 12:19:45 host kernel: Call Trace:
>> Apr  9 12:19:45 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 12:19:45 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 12:19:45 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 12:19:45 host kernel: BUG: soft lockup - CPU#0 stuck for 1065s! 
>> [swapper:0]
>> Apr  9 12:19:45 host kernel: Modules linked in:
>> Apr  9 12:19:45 host kernel: CPU 0:
>> Apr  9 12:19:45 host kernel: Modules linked in:
>> Apr  9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 12:19:45 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 12:19:45 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 12:19:45 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 12:19:45 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 12:19:45 host kernel: CR2: 00007ff39e404000 CR3: 000000007e2a6000 
>> CR4: 0000000000002660
>> Apr  9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 12:19:45 host kernel: Call Trace:
>> Apr  9 12:19:45 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 12:19:45 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 12:19:45 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 12:19:45 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 12:37:26 host kernel: BUG: soft lockup - CPU#1 stuck for 989s! 
>> [swapper:0]
>> Apr  9 12:37:26 host kernel: Modules linked in:
>> Apr  9 12:37:26 host kernel: CPU 1:
>> Apr  9 12:37:26 host kernel: Modules linked in:
>> Apr  9 12:37:26 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 12:37:26 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 12:37:26 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 12:37:26 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 12:37:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 12:37:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 12:37:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 12:37:26 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 12:37:26 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 12:37:26 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 12:37:26 host kernel: CR2: 00000000039f7330 CR3: 000000007e2a6000 
>> CR4: 0000000000002660
>> Apr  9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 12:37:26 host kernel: Call Trace:
>> Apr  9 12:37:26 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 12:37:26 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 12:37:26 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 12:37:26 host kernel: BUG: soft lockup - CPU#0 stuck for 989s! 
>> [httpd:31684]
>> Apr  9 12:37:26 host kernel: Modules linked in:
>> Apr  9 12:37:26 host kernel: CPU 0:
>> Apr  9 12:37:26 host kernel: Modules linked in:
>> Apr  9 12:37:26 host kernel: Pid: 31684, comm: httpd Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 12:37:26 host kernel: RIP: e033:[<00007ff39a41f7d3>]  
>> [<00007ff39a41f7d3>] 0x7ff39a41f7d3
>> Apr  9 12:37:26 host kernel: RSP: e02b:00007fff4b9ad330  EFLAGS: 00000246
>> Apr  9 12:37:26 host kernel: RAX: 00007ff39a41f7ac RBX: 00007ff39aaaa3e0 
>> RCX: 0000000000000001
>> Apr  9 12:37:26 host kernel: RDX: 00007ff39a781bdc RSI: 000000000000003e 
>> RDI: 0000000000000000
>> Apr  9 12:37:26 host kernel: RBP: 00007ff39aaa9e60 R08: 00007ff39a782e60 
>> R09: 0000000000000003
>> Apr  9 12:37:26 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 
>> R12: 0000000003a2b297
>> Apr  9 12:37:26 host kernel: R13: 00007ff39aaaa3e0 R14: 00007ff39a784da0 
>> R15: 00007ff39a783e80
>> Apr  9 12:37:26 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 12:37:26 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 12:37:26 host kernel: CR2: 0000000003aa1000 CR3: 000000004debb000 
>> CR4: 0000000000002660
>> Apr  9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 12:37:26 host kernel: Call Trace:
>> Apr  9 13:44:52 host kernel: BUG: soft lockup - CPU#1 stuck for 3768s! 
>> [swapper:0]
>> Apr  9 13:44:52 host kernel: Modules linked in:
>> Apr  9 13:44:52 host kernel: CPU 1:
>> Apr  9 13:44:52 host kernel: Modules linked in:
>> Apr  9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 13:44:52 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 13:44:52 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 13:44:52 host kernel: FS:  00007fd75c048710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 13:44:52 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 13:44:52 host kernel: CR2: 00007fff892a1e30 CR3: 0000000071c86000 
>> CR4: 0000000000002660
>> Apr  9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 13:44:52 host kernel: Call Trace:
>> Apr  9 13:44:52 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 13:44:52 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 13:44:52 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 13:44:52 host kernel: BUG: soft lockup - CPU#0 stuck for 3768s! 
>> [swapper:0]
>> Apr  9 13:44:52 host kernel: Modules linked in:
>> Apr  9 13:44:52 host kernel: CPU 0:
>> Apr  9 13:44:52 host kernel: Modules linked in:
>> Apr  9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 13:44:52 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 13:44:52 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 13:44:52 host kernel: FS:  00007f0e429cb6e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 13:44:52 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 13:44:52 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d0de000 
>> CR4: 0000000000002660
>> Apr  9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 13:44:52 host kernel: Call Trace:
>> Apr  9 13:44:52 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 13:44:52 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 13:44:52 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 13:44:52 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 13:55:43 host kernel: BUG: soft lockup - CPU#1 stuck for 607s! 
>> [swapper:0]
>> Apr  9 13:55:43 host kernel: Modules linked in:
>> Apr  9 13:55:43 host kernel: CPU 1:
>> Apr  9 13:55:43 host kernel: Modules linked in:
>> Apr  9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 13:55:43 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 13:55:43 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 13:55:43 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 13:55:43 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 13:55:43 host kernel: CR2: 0000000003c7b27f CR3: 00000000266e8000 
>> CR4: 0000000000002660
>> Apr  9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 13:55:43 host kernel: Call Trace:
>> Apr  9 13:55:43 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 13:55:43 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 13:55:43 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 13:55:43 host kernel: BUG: soft lockup - CPU#0 stuck for 607s! 
>> [swapper:0]
>> Apr  9 13:55:43 host kernel: Modules linked in:
>> Apr  9 13:55:43 host kernel: CPU 0:
>> Apr  9 13:55:43 host kernel: Modules linked in:
>> Apr  9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 13:55:43 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 13:55:43 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 13:55:43 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 13:55:43 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 13:55:43 host kernel: CR2: 00007f2f28bf62e8 CR3: 000000007e2a6000 
>> CR4: 0000000000002660
>> Apr  9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 13:55:43 host kernel: Call Trace:
>> Apr  9 13:55:43 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 13:55:43 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 13:55:43 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 13:55:43 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 14:16:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1153s! 
>> [swapper:0]
>> Apr  9 14:16:21 host kernel: Modules linked in:
>> Apr  9 14:16:21 host kernel: CPU 1:
>> Apr  9 14:16:21 host kernel: Modules linked in:
>> Apr  9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 14:16:21 host kernel: RIP: e030:[<ffffffff8100922a>]  
>> [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
>> Apr  9 14:16:21 host kernel: RSP: e02b:ffff88007f8a1dd8  EFLAGS: 00000246
>> Apr  9 14:16:21 host kernel: RAX: 0000000000030004 RBX: ffff88007db55810 
>> RCX: ffffffff8100922a
>> Apr  9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000000
>> Apr  9 14:16:21 host kernel: RBP: 0000000000000001 R08: ffff88007f8a0000 
>> R09: ffffffff8100e3b2
>> Apr  9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 14:16:21 host kernel: R13: ffff88007e0040c0 R14: 0000000000000000 
>> R15: 0000000000001000
>> Apr  9 14:16:21 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 14:16:21 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 
>> CR4: 0000000000002660
>> Apr  9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 14:16:21 host kernel: Call Trace:
>> Apr  9 14:16:21 host kernel:  [<ffffffff8100dd39>] ? 
>> xen_force_evtchn_callback+0x9/0xa
>> Apr  9 14:16:21 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 14:16:21 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 14:16:21 host kernel:  [<ffffffff8100e359>] ? 
>> xen_irq_enable_direct_end+0x0/0x7
>> Apr  9 14:16:21 host kernel:  [<ffffffff8103b070>] ? 
>> finish_task_switch+0x27/0x6e
>> Apr  9 14:16:21 host kernel:  [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
>> Apr  9 14:16:21 host kernel:  [<ffffffff814d87ba>] ? error_exit+0x2a/0x60
>> Apr  9 14:16:21 host kernel:  [<ffffffff8101149d>] ? 
>> retint_restore_args+0x5/0x6
>> Apr  9 14:16:21 host kernel:  [<ffffffff8100fe5f>] ? cpu_idle+0x80/0x82
>> Apr  9 14:16:21 host kernel: BUG: soft lockup - CPU#0 stuck for 1153s! 
>> [swapper:0]
>> Apr  9 14:16:21 host kernel: Modules linked in:
>> Apr  9 14:16:21 host kernel: CPU 0:
>> Apr  9 14:16:21 host kernel: Modules linked in:
>> Apr  9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 14:16:21 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 14:16:21 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 14:16:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 14:16:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 14:16:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 14:16:21 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 14:16:21 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 
>> CR4: 0000000000002660
>> Apr  9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 14:16:21 host kernel: Call Trace:
>> Apr  9 14:16:21 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 14:16:21 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 14:16:21 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 14:16:21 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 15:40:21 host kernel: BUG: soft lockup - CPU#1 stuck for 4104s! 
>> [swapper:0]
>> Apr  9 15:40:21 host kernel: Modules linked in:
>> Apr  9 15:40:21 host kernel: CPU 1:
>> Apr  9 15:40:21 host kernel: Modules linked in:
>> Apr  9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 15:40:21 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 15:40:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 15:40:21 host kernel: FS:  00007fd75c048710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 15:40:21 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 15:40:21 host kernel: CR2: 000000000040d0e2 CR3: 0000000071c86000 
>> CR4: 0000000000002660
>> Apr  9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 15:40:21 host kernel: Call Trace:
>> Apr  9 15:40:21 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 15:40:21 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 15:40:21 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 15:40:21 host kernel: BUG: soft lockup - CPU#0 stuck for 4104s! 
>> [swapper:0]
>> Apr  9 15:40:21 host kernel: Modules linked in:
>> Apr  9 15:40:21 host kernel: CPU 0:
>> Apr  9 15:40:21 host kernel: Modules linked in:
>> Apr  9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 15:40:21 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 15:40:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 15:40:21 host kernel: FS:  00007f07704fe6e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 15:40:21 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 15:40:21 host kernel: CR2: 00000000004c04a8 CR3: 0000000071c86000 
>> CR4: 0000000000002660
>> Apr  9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 15:40:21 host kernel: Call Trace:
>> Apr  9 15:40:21 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 15:40:21 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 15:40:21 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 15:40:21 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 16:06:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1452s! 
>> [swapper:0]
>> Apr  9 16:06:21 host kernel: Modules linked in:
>> Apr  9 16:06:21 host kernel: CPU 1:
>> Apr  9 16:06:21 host kernel: Modules linked in:
>> Apr  9 16:06:21 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 16:06:21 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 16:06:21 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 16:06:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 16:06:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 16:06:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 16:06:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 16:06:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 16:06:21 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 16:06:25 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 16:06:25 host kernel: CR2: 00007fbd3c012909 CR3: 0000000071c86000 
>> CR4: 0000000000002660
>> Apr  9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 16:06:26 host kernel: Call Trace:
>> Apr  9 16:06:26 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 16:06:26 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 16:06:26 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 16:06:26 host kernel: BUG: soft lockup - CPU#0 stuck for 1452s! 
>> [swapper:0]
>> Apr  9 16:06:26 host kernel: Modules linked in:
>> Apr  9 16:06:26 host kernel: CPU 0:
>> Apr  9 16:06:26 host kernel: Modules linked in:
>> Apr  9 16:06:26 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 16:06:26 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 16:06:26 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 16:06:26 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 16:06:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 16:06:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 16:06:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 16:06:26 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 16:06:26 host kernel: FS:  00007f07704fe6e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 16:06:26 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 16:06:26 host kernel: CR2: 00007f09a4413000 CR3: 000000007ea99000 
>> CR4: 0000000000002660
>> Apr  9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 16:06:26 host kernel: Call Trace:
>> Apr  9 16:06:26 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 16:06:26 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 16:06:26 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 16:06:26 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 16:17:43 host kernel: BUG: soft lockup - CPU#1 stuck for 637s! 
>> [httpd:32342]
>> Apr  9 16:17:43 host kernel: Modules linked in:
>> Apr  9 16:17:43 host kernel: CPU 1:
>> Apr  9 16:17:43 host kernel: Modules linked in:
>> Apr  9 16:17:43 host kernel: Pid: 32342, comm: httpd Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 16:17:43 host kernel: RIP: e030:[<ffffffff8100922a>]  
>> [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
>> Apr  9 16:17:43 host kernel: RSP: e02b:ffff88007955fea0  EFLAGS: 00000246
>> Apr  9 16:17:43 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 
>> RCX: ffffffff8100922a
>> Apr  9 16:17:43 host kernel: RDX: 000000000002ba30 RSI: 0000000000000000 
>> RDI: 0000000000000000
>> Apr  9 16:17:43 host kernel: RBP: 000000000394ca98 R08: 0000000000033240 
>> R09: 000000310f6de059
>> Apr  9 16:17:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000006
>> Apr  9 16:17:43 host kernel: R13: ffff88007955ff58 R14: ffff88007f9a8ab0 
>> R15: ffff88007d093b40
>> Apr  9 16:17:43 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 16:17:43 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 16:17:43 host kernel: CR2: 000000000394ca98 CR3: 000000006adde000 
>> CR4: 0000000000002660
>> Apr  9 16:17:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 16:17:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 16:17:43 host kernel: Call Trace:
>> Apr  9 16:17:43 host kernel:  [<ffffffff8100dd39>] ? 
>> xen_force_evtchn_callback+0x9/0xa
>> Apr  9 16:17:43 host kernel:  [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr  9 16:17:43 host kernel:  [<ffffffff8100e359>] ? 
>> xen_irq_enable_direct_end+0x0/0x7
>> Apr  9 16:17:43 host kernel:  [<ffffffff814da477>] ? do_page_fault+0xfb/0x28d
>> Apr  9 16:17:43 host kernel:  [<ffffffff814d85b5>] ? page_fault+0x25/0x30
>> Apr  9 16:17:43 host kernel: BUG: soft lockup - CPU#0 stuck for 637s! 
>> [swapper:0]
>> Apr  9 16:17:43 host kernel: Modules linked in:
>> Apr  9 16:17:43 host kernel: CPU 0:
>> Apr  9 16:17:43 host kernel: Modules linked in:
>> Apr  9 16:17:43 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 16:17:43 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 16:17:44 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 16:17:44 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 16:17:44 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 16:17:44 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 16:17:44 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 16:17:44 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 16:17:44 host kernel: FS:  00007f07704fe6e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 16:17:44 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 16:17:44 host kernel: CR2: 00000000004505b0 CR3: 000000007ea99000 
>> CR4: 0000000000002660
>> Apr  9 16:17:44 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 16:17:44 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 16:17:44 host kernel: Call Trace:
>> Apr  9 16:17:44 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 16:17:44 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 16:17:44 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 16:17:44 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 16:20:22 host kernel: BUG: soft lockup - CPU#1 stuck for 147s! 
>> [swapper:0]
>> Apr  9 16:20:22 host kernel: Modules linked in:
>> Apr  9 16:20:22 host kernel: CPU 1:
>> Apr  9 16:20:22 host kernel: Modules linked in:
>> Apr  9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 16:20:22 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 16:20:22 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 16:20:22 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 16:20:22 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 16:20:22 host kernel: CR2: 00007ff39aec11a7 CR3: 0000000071c86000 
>> CR4: 0000000000002660
>> Apr  9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 16:20:22 host kernel: Call Trace:
>> Apr  9 16:20:22 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 16:20:22 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 16:20:22 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 16:20:22 host kernel: BUG: soft lockup - CPU#0 stuck for 147s! 
>> [swapper:0]
>> Apr  9 16:20:22 host kernel: Modules linked in:
>> Apr  9 16:20:22 host kernel: CPU 0:
>> Apr  9 16:20:22 host kernel: Modules linked in:
>> Apr  9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 16:20:22 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> Apr  9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 
>> RCX: ffffffff810093aa
>> Apr  9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: ffffffffffffffff
>> Apr  9 16:20:22 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 
>> R15: 0000000000000000
>> Apr  9 16:20:22 host kernel: FS:  00007f07704fe6e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 16:20:22 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 16:20:22 host kernel: CR2: 00000000006ddc20 CR3: 0000000071c86000 
>> CR4: 0000000000002660
>> Apr  9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 16:20:22 host kernel: Call Trace:
>> Apr  9 16:20:22 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 16:20:22 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 16:20:22 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 16:20:22 host kernel:  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr  9 16:22:29 host kernel: BUG: soft lockup - CPU#1 stuck for 119s! 
>> [swapper:0]
>> Apr  9 16:22:29 host kernel: Modules linked in:
>> Apr  9 16:22:29 host kernel: CPU 1:
>> Apr  9 16:22:29 host kernel: Modules linked in:
>> Apr  9 16:22:29 host kernel: Pid: 0, comm: swapper Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 16:22:29 host kernel: RIP: e030:[<ffffffff810093aa>]  
>> [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr  9 16:22:29 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
>> Apr  9 16:22:29 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 
>> RCX: ffffffff810093aa
>> Apr  9 16:22:29 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 
>> RDI: 0000000000000001
>> Apr  9 16:22:29 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 
>> R09: ffffffff8100e3b2
>> Apr  9 16:22:29 host kernel: R10: 0000000000000001 R11: 0000000000000246 
>> R12: 0000000000000000
>> Apr  9 16:22:29 host kernel: R13: 0000000000000000 R14: 0000000000000000 
>> R15: 0000000000000000
>> Apr  9 16:22:29 host kernel: FS:  00007ff39e495710(0000) 
>> GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr  9 16:22:29 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 
>> 000000008005003b
>> Apr  9 16:22:29 host kernel: CR2: 00007ff39c506460 CR3: 0000000026469000 
>> CR4: 0000000000002660
>> Apr  9 16:22:29 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 16:22:29 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 16:22:29 host kernel: Call Trace:
>> Apr  9 16:22:29 host kernel:  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr  9 16:22:29 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr  9 16:22:29 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr  9 16:22:29 host kernel: BUG: soft lockup - CPU#0 stuck for 119s! 
>> [syslogd:1848]
>> Apr  9 16:22:29 host kernel: Modules linked in:
>> Apr  9 16:22:29 host kernel: CPU 0:
>> Apr  9 16:22:29 host kernel: Modules linked in:
>> Apr  9 16:22:29 host kernel: Pid: 1848, comm: syslogd Not tainted 
>> 2.6.32.9xls-domU #2 
>> Apr  9 16:22:29 host kernel: RIP: 0033:[<00007f74fd9e4ad5>]  
>> [<00007f74fd9e4ad5>] 0x7f74fd9e4ad5
>> Apr  9 16:22:29 host kernel: RSP: 002b:00007fff22fa00d0  EFLAGS: 00000246
>> Apr  9 16:22:29 host kernel: RAX: 000000000000002e RBX: 0000000000000000 
>> RCX: ffffffff810092eb
>> Apr  9 16:22:29 host kernel: RDX: 00000000000003fe RSI: 00007fff22fa01b0 
>> RDI: 0000000000000000
>> Apr  9 16:22:30 host kernel: RBP: 00000000000003fe R08: 0000000000000000 
>> R09: 0000000000000000
>> Apr  9 16:22:30 host kernel: R10: 0000000000000000 R11: 0000000000000246 
>> R12: 00007fff22fa01b0
>> Apr  9 16:22:30 host kernel: R13: 0000000000000000 R14: 00007fff22fa0120 
>> R15: 00007f74fde74690
>> Apr  9 16:22:30 host kernel: FS:  00007f74fde746e0(0000) 
>> GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr  9 16:22:30 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Apr  9 16:22:30 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d03a000 
>> CR4: 0000000000002660
>> Apr  9 16:22:30 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 
>> DR2: 0000000000000000
>> Apr  9 16:22:30 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 
>> DR7: 0000000000000400
>> Apr  9 16:22:30 host kernel: Call Trace:
>>     
>
> On Apr 9, 2010, at 15:22 , Pim van Riezen wrote:
>
>   
>> Good day,
>>
>> I'm getting lots of these from a couple of pvops DomU guests that are pretty 
>> heavy in their i/o and cpu-load. The issue reproduces on pvops 2.6.31 and 
>> 2.6.32 trees under Xen 3.3 as well as 3.4.1 (in the process of trying out 
>> 3.4.2 but I'm not holding my breath). Is there anything we can do be more 
>> helpful towards solving this issue?
>>
>> BUG: soft lockup - CPU#0 stuck for 561s! [swapper:0]
>> Modules linked in:
>> CPU 0:
>> Modules linked in:
>> Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2 
>> RIP: e030:[<ffffffff810093aa>]  [<ffffffff810093aa>] 
>> hypercall_page+0x3aa/0x1001
>> RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>> RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> FS:  00007f8ac7a9c6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> CR2: 00007f291b4c9000 CR3: 000000007d8c1000 CR4: 0000000000002660
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Call Trace:
>>  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>>  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>>  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>>  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>>
>> Kind regards,
>> Pim van Riezen
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-devel
>>     
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>
>   


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel