| |
|
 |
|
 |
|
|
|
| |
|
|
xen-devel
Re: [Xen-devel] Xen hypervisor external denial of service vulnerability?
On Tue, Feb 08, 2011 at 06:21:25PM +0100, Pim van Riezen wrote:
>
> On Feb 8, 2011, at 18:08 , Pim van Riezen wrote:
>
> > On Feb 8, 2011, at 17:51 , Pasi Kärkkäinen wrote:
> >>
> >> Did you also make sure VMs don't use those 2 pcpus dedicated for dom0?
> >> You have to explicitly configure each VM not to use those pcpus.
> >
> > That seems to have done the trick.
>
> Alas, I was too soon in drawing a conclusion. After a new 10 minute run:
Did you try to run the 2.6.32 pvops type kernel? Asking b/c it looks like
the issue is due to the fact that mutex lock is held for a very very long time.
The spinlock implementation in 2.6.32 changed so it might provide a better
solution.
>
> Feb 8 18:12:30 telemann kernel: INFO: task bash:12225 blocked for more than
> 120 seconds.
> Feb 8 18:12:30 telemann kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:12:30 telemann kernel: bash D ffff88006ac7bd08 0 12225
> 1 8260 (L-TLB)
> Feb 8 18:12:30 telemann kernel: ffff88006ac7bb88 0000000000000246
> 0000000300000000 ffff88007ec3a6d8
> Feb 8 18:12:30 telemann kernel: 0000000000000009 ffff88006c16e820
> ffff88007a5a9080 000000000008f03e
> Feb 8 18:12:30 telemann kernel: ffff88006c16ea08 ffffffff8022f10c
> Feb 8 18:12:30 telemann kernel: Call Trace:
> Feb 8 18:12:30 telemann kernel: [<ffffffff8022f10c>] __wake_up+0x38/0x4f
> Feb 8 18:12:30 telemann kernel: [<ffffffff880317ae>]
> :jbd:journal_stop+0x1f3/0x1ff
> Feb 8 18:12:30 telemann kernel: [<ffffffff802994d1>]
> flush_cpu_workqueue+0x83/0xb5
> Feb 8 18:12:30 telemann kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:12:30 telemann kernel: [<ffffffff80263914>] mutex_lock+0xd/0x1d
> Feb 8 18:12:30 telemann kernel: [<ffffffff80299563>]
> flush_workqueue+0x60/0x87
> Feb 8 18:12:41 telemann kernel: [<ffffffff80394af5>] release_dev+0x503/0x67b
> Feb 8 18:12:55 telemann kernel: [<ffffffff8020b860>]
> release_pages+0x158/0x165
> Feb 8 18:13:09 telemann kernel: [<ffffffff80255821>] tty_release+0x11/0x1a
> Feb 8 18:13:23 telemann kernel: [<ffffffff80213492>] __fput+0xd3/0x1bd
> Feb 8 18:13:38 telemann kernel: [<ffffffff802243cb>] filp_close+0x5c/0x64
> Feb 8 18:13:51 telemann kernel: [<ffffffff8023a392>]
> put_files_struct+0x63/0xae
> Feb 8 18:14:06 telemann kernel: [<ffffffff802160cd>] do_exit+0x31d/0x902
> Feb 8 18:14:19 telemann kernel: [<ffffffff8024ae4d>] cpuset_exit+0x0/0x88
> Feb 8 18:14:33 telemann kernel: [<ffffffff8022b920>]
> get_signal_to_deliver+0x477/0x4aa
> Feb 8 18:14:49 telemann kernel: [<ffffffff8025d19e>]
> do_notify_resume+0x9c/0x7ba
> Feb 8 18:15:01 telemann kernel: [<ffffffff80294ea1>]
> __group_send_sig_info+0xb9/0xc8
> Feb 8 18:15:08 telemann kernel: [<ffffffff8025cb0b>]
> group_send_sig_info+0x62/0x6f
> Feb 8 18:15:22 telemann kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:15:37 telemann kernel: [<ffffffff802afd73>]
> audit_syscall_entry+0x180/0x1b3
> Feb 8 18:15:49 telemann kernel: [<ffffffff80245a48>]
> sys_rt_sigreturn+0x327/0x35a
> Feb 8 18:16:03 telemann kernel: [<ffffffff802b0175>]
> audit_syscall_exit+0x336/0x362
> Feb 8 18:16:17 telemann kernel: [<ffffffff8026042c>] int_signal+0x12/0x17
> Feb 8 18:16:31 telemann kernel:
> Feb 8 18:16:44 telemann kernel: INFO: task bash:12225 blocked for more than
> 120 seconds.
> Feb 8 18:16:58 telemann kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:17:12 telemann kernel: bash D ffff88006ac7bd08 0 12225
> 1 8260 (L-TLB)
> Feb 8 18:17:26 telemann kernel: ffff88006ac7bb88 0000000000000246
> 0000000300000000 ffff88007ec3a6d8
> Feb 8 18:17:39 telemann kernel: 0000000000000009 ffff88006c16e820
> ffff88007a5a9080 000000000008f03e
> Feb 8 18:17:54 telemann kernel: ffff88006c16ea08 ffffffff8022f10c
> Feb 8 18:18:08 telemann kernel: Call Trace:
> Feb 8 18:18:21 telemann kernel: [<ffffffff8022f10c>] __wake_up+0x38/0x4f
> Feb 8 18:18:34 telemann kernel: [<ffffffff880317ae>]
> :jbd:journal_stop+0x1f3/0x1ff
> Feb 8 18:18:47 telemann kernel: [<ffffffff802994d1>]
> flush_cpu_workqueue+0x83/0xb5
> Feb 8 18:18:58 telemann kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:58 telemann kernel: [<ffffffff80263914>] mutex_lock+0xd/0x1d
> Feb 8 18:18:58 telemann kernel: [<ffffffff80299563>]
> flush_workqueue+0x60/0x87
> Feb 8 18:18:58 telemann kernel: [<ffffffff80394af5>] release_dev+0x503/0x67b
> Feb 8 18:18:58 telemann kernel: [<ffffffff8020b860>]
> release_pages+0x158/0x165
> Feb 8 18:18:58 telemann kernel: [<ffffffff80255821>] tty_release+0x11/0x1a
> Feb 8 18:18:58 telemann kernel: [<ffffffff80213492>] __fput+0xd3/0x1bd
> Feb 8 18:18:58 telemann kernel: [<ffffffff802243cb>] filp_close+0x5c/0x64
> Feb 8 18:18:58 telemann kernel: [<ffffffff8023a392>]
> put_files_struct+0x63/0xae
> Feb 8 18:18:58 telemann kernel: [<ffffffff802160cd>] do_exit+0x31d/0x902
> Feb 8 18:18:58 telemann kernel: [<ffffffff8024ae4d>] cpuset_exit+0x0/0x88
> Feb 8 18:18:58 telemann kernel: [<ffffffff8022b920>]
> get_signal_to_deliver+0x477/0x4aa
> Feb 8 18:18:58 telemann kernel: [<ffffffff8025d19e>]
> do_notify_resume+0x9c/0x7ba
> Feb 8 18:18:58 telemann kernel: [<ffffffff80294ea1>]
> __group_send_sig_info+0xb9/0xc8
> Feb 8 18:18:58 telemann kernel: [<ffffffff8025cb0b>]
> group_send_sig_info+0x62/0x6f
> Feb 8 18:18:58 telemann kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:58 telemann kernel: [<ffffffff802afd73>]
> audit_syscall_entry+0x180/0x1b3
> Feb 8 18:18:58 telemann kernel: [<ffffffff80245a48>]
> sys_rt_sigreturn+0x327/0x35a
> Feb 8 18:18:58 telemann kernel: [<ffffffff802b0175>]
> audit_syscall_exit+0x336/0x362
> Feb 8 18:18:59 telemann kernel: [<ffffffff8026042c>] int_signal+0x12/0x17
> Feb 8 18:18:59 telemann kernel:
> Feb 8 18:18:59 telemann kernel: INFO: task bash:12225 blocked for more than
> 120 seconds.
> Feb 8 18:18:59 telemann kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:18:59 telemann kernel: bash D ffff88006ac7bd08 0 12225
> 1 8260 (L-TLB)
> Feb 8 18:18:59 telemann kernel: ffff88006ac7bb88 0000000000000246
> 0000000300000000 ffff88007ec3a6d8
> Feb 8 18:18:59 telemann kernel: 0000000000000009 ffff88006c16e820
> ffff88007a5a9080 000000000008f03e
> Feb 8 18:18:59 telemann kernel: ffff88006c16ea08 ffffffff8022f10c
> Feb 8 18:18:59 telemann kernel: Call Trace:
> Feb 8 18:18:59 telemann kernel: [<ffffffff8022f10c>] __wake_up+0x38/0x4f
> Feb 8 18:18:59 telemann kernel: [<ffffffff880317ae>]
> :jbd:journal_stop+0x1f3/0x1ff
> Feb 8 18:18:59 telemann kernel: [<ffffffff802994d1>]
> flush_cpu_workqueue+0x83/0xb5
> Feb 8 18:18:59 telemann kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:59 telemann kernel: [<ffffffff80263914>] mutex_lock+0xd/0x1d
> Feb 8 18:18:59 telemann kernel: [<ffffffff80299563>]
> flush_workqueue+0x60/0x87
> Feb 8 18:18:59 telemann kernel: [<ffffffff80394af5>] release_dev+0x503/0x67b
> Feb 8 18:18:59 telemann kernel: [<ffffffff8020b860>]
> release_pages+0x158/0x165
> Feb 8 18:18:59 telemann kernel: [<ffffffff80255821>] tty_release+0x11/0x1a
> Feb 8 18:18:59 telemann kernel: [<ffffffff80213492>] __fput+0xd3/0x1bd
> Feb 8 18:18:59 telemann kernel: [<ffffffff802243cb>] filp_close+0x5c/0x64
> Feb 8 18:18:59 telemann kernel: [<ffffffff8023a392>]
> put_files_struct+0x63/0xae
> Feb 8 18:18:59 telemann kernel: [<ffffffff802160cd>] do_exit+0x31d/0x902
> Feb 8 18:18:59 telemann kernel: [<ffffffff8024ae4d>] cpuset_exit+0x0/0x88
> Feb 8 18:18:59 telemann kernel: [<ffffffff8022b920>]
> get_signal_to_deliver+0x477/0x4aa
> Feb 8 18:18:59 telemann kernel: [<ffffffff8025d19e>]
> do_notify_resume+0x9c/0x7ba
> Feb 8 18:18:59 telemann kernel: [<ffffffff80294ea1>]
> __group_send_sig_info+0xb9/0xc8
> Feb 8 18:18:59 telemann kernel: [<ffffffff8025cb0b>]
> group_send_sig_info+0x62/0x6f
> Feb 8 18:18:59 telemann kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:59 telemann kernel: [<ffffffff802afd73>]
> audit_syscall_entry+0x180/0x1b3
> Feb 8 18:18:59 telemann kernel: [<ffffffff80245a48>]
> sys_rt_sigreturn+0x327/0x35a
> Feb 8 18:18:59 telemann kernel: [<ffffffff802b0175>]
> audit_syscall_exit+0x336/0x362
> Feb 8 18:18:59 telemann kernel: [<ffffffff8026042c>] int_signal+0x12/0x17
> Feb 8 18:18:59 telemann kernel:
> Feb 8 18:18:59 telemann kernel: INFO: task bash:12225 blocked for more than
> 120 seconds.
> Feb 8 18:18:59 telemann kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:18:59 telemann kernel: bash D ffff88006ac7bd08 0 12225
> 1 8260 (L-TLB)
> Feb 8 18:18:59 telemann kernel: ffff88006ac7bb88 0000000000000246
> 0000000300000000 ffff88007ec3a6d8
> Feb 8 18:18:59 telemann kernel: 0000000000000009 ffff88006c16e820
> ffff88007a5a9080 000000000008f03e
> Feb 8 18:18:59 telemann kernel: ffff88006c16ea08 ffffffff8022f10c
> Feb 8 18:18:59 telemann kernel: Call Trace:
> Feb 8 18:18:59 telemann kernel: [<ffffffff8022f10c>] __wake_up+0x38/0x4f
> Feb 8 18:18:59 telemann kernel: [<ffffffff880317ae>]
> :jbd:journal_stop+0x1f3/0x1ff
> Feb 8 18:18:59 telemann kernel: [<ffffffff802994d1>]
> flush_cpu_workqueue+0x83/0xb5
> Feb 8 18:18:59 telemann kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:18:59 telemann kernel: [<ffffffff80263914>] mutex_lock+0xd/0x1d
> Feb 8 18:18:59 telemann kernel: [<ffffffff80299563>]
> flush_workqueue+0x60/0x87
> Feb 8 18:18:59 telemann kernel: [<ffffffff80394af5>] release_dev+0x503/0x67b
> Feb 8 18:18:59 telemann kernel: [<ffffffff8020b860>]
> release_pages+0x158/0x165
> Feb 8 18:18:59 telemann kernel: [<ffffffff80255821>] tty_release+0x11/0x1a
> Feb 8 18:18:59 telemann kernel: [<ffffffff80213492>] __fput+0xd3/0x1bd
> Feb 8 18:18:59 telemann kernel: [<ffffffff802243cb>] filp_close+0x5c/0x64
> Feb 8 18:18:59 telemann kernel: [<ffffffff8023a392>]
> put_files_struct+0x63/0xae
> Feb 8 18:18:59 telemann kernel: [<ffffffff802160cd>] do_exit+0x31d/0x902
> Feb 8 18:18:59 telemann kernel: [<ffffffff8024ae4d>] cpuset_exit+0x0/0x88
> Feb 8 18:18:59 telemann kernel: [<ffffffff8022b920>]
> get_signal_to_deliver+0x477/0x4aa
> Feb 8 18:18:59 telemann kernel: [<ffffffff8025d19e>]
> do_notify_resume+0x9c/0x7ba
> Feb 8 18:19:00 telemann kernel: [<ffffffff80294ea1>]
> __group_send_sig_info+0xb9/0xc8
> Feb 8 18:19:00 telemann kernel: [<ffffffff8025cb0b>]
> group_send_sig_info+0x62/0x6f
> Feb 8 18:19:00 telemann kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:19:00 telemann kernel: [<ffffffff802afd73>]
> audit_syscall_entry+0x180/0x1b3
> Feb 8 18:19:00 telemann kernel: [<ffffffff80245a48>]
> sys_rt_sigreturn+0x327/0x35a
> Feb 8 18:19:00 telemann kernel: [<ffffffff802b0175>]
> audit_syscall_exit+0x336/0x362
> Feb 8 18:19:00 telemann kernel: [<ffffffff8026042c>] int_signal+0x12/0x17
> Feb 8 18:19:00 telemann kernel:
>
> Feb 8 18:11:23 handel kernel: xenbr0: received tcn bpdu on port 1(eth0)
> Feb 8 18:11:23 handel kernel: xenbr0: topology change detected, propagating
> Feb 8 18:14:54 handel kernel: INFO: task syslogd:11299 blocked for more than
> 120 seconds.
> Feb 8 18:14:54 handel kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:14:54 handel kernel: syslogd D 0000031e848fed46 0 11299
> 1 11302 11268 (NOTLB)
> Feb 8 18:14:54 handel kernel: ffff880079603d88 0000000000000282
> 0000000000000000 0000000000000001
> Feb 8 18:14:54 handel kernel: 000000000000000a ffff88007e5b9100
> ffff88000002b040 0000000000026ea9
> Feb 8 18:14:54 handel kernel: ffff88007e5b92e8 0000000000000000
> Feb 8 18:14:54 handel kernel: Call Trace:
> Feb 8 18:14:54 handel kernel: [<ffffffff88036d5a>]
> :jbd:log_wait_commit+0xa3/0xf5
> Feb 8 18:14:54 handel kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:14:54 handel kernel: [<ffffffff8803178a>]
> :jbd:journal_stop+0x1cf/0x1ff
> Feb 8 18:14:54 handel kernel: [<ffffffff8023119d>]
> __writeback_single_inode+0x1e9/0x328
> Feb 8 18:19:15 handel kernel: [<ffffffff802d330d>]
> do_readv_writev+0x26e/0x291
> Feb 8 18:19:15 handel kernel: [<ffffffff802e5b8b>] sync_inode+0x24/0x33
> Feb 8 18:19:15 handel kernel: [<ffffffff8804c36d>]
> :ext3:ext3_sync_file+0xc9/0xdc
> Feb 8 18:19:15 handel kernel: [<ffffffff80251e07>] do_fsync+0x52/0xa4
> Feb 8 18:19:15 handel kernel: [<ffffffff802d3b11>] __do_fsync+0x23/0x36
> Feb 8 18:19:15 handel kernel: [<ffffffff802602f9>] tracesys+0xab/0xb6
> Feb 8 18:19:15 handel kernel:
> Feb 8 18:19:15 handel kernel: INFO: task syslogd:11299 blocked for more than
> 120 seconds.
> Feb 8 18:19:15 handel kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:19:15 handel kernel: syslogd D 0000031e848fed46 0 11299
> 1 11302 11268 (NOTLB)
> Feb 8 18:19:15 handel kernel: ffff880079603d88 0000000000000282
> 0000000000000000 0000000000000001
> Feb 8 18:19:15 handel kernel: 000000000000000a ffff88007e5b9100
> ffff88000002b040 0000000000026ea9
> Feb 8 18:19:15 handel kernel: ffff88007e5b92e8 0000000000000000
> Feb 8 18:19:15 handel kernel: Call Trace:
> Feb 8 18:19:15 handel kernel: [<ffffffff88036d5a>]
> :jbd:log_wait_commit+0xa3/0xf5
> Feb 8 18:19:15 handel kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:19:15 handel kernel: [<ffffffff8803178a>]
> :jbd:journal_stop+0x1cf/0x1ff
> Feb 8 18:19:15 handel kernel: [<ffffffff8023119d>]
> __writeback_single_inode+0x1e9/0x328
> Feb 8 18:19:15 handel kernel: [<ffffffff802d330d>]
> do_readv_writev+0x26e/0x291
> Feb 8 18:19:15 handel kernel: [<ffffffff802e5b8b>] sync_inode+0x24/0x33
> Feb 8 18:19:15 handel kernel: [<ffffffff8804c36d>]
> :ext3:ext3_sync_file+0xc9/0xdc
> Feb 8 18:19:15 handel kernel: [<ffffffff80251e07>] do_fsync+0x52/0xa4
> Feb 8 18:19:15 handel kernel: [<ffffffff802d3b11>] __do_fsync+0x23/0x36
> Feb 8 18:19:15 handel kernel: [<ffffffff802602f9>] tracesys+0xab/0xb6
> Feb 8 18:19:15 handel kernel:
> Feb 8 18:19:15 handel kernel: INFO: task syslogd:11299 blocked for more than
> 120 seconds.
> Feb 8 18:19:15 handel kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 8 18:19:15 handel kernel: syslogd D 0000031e848fed46 0 11299
> 1 11302 11268 (NOTLB)
> Feb 8 18:19:15 handel kernel: ffff880079603d88 0000000000000282
> 0000000000000000 0000000000000001
> Feb 8 18:19:15 handel kernel: 000000000000000a ffff88007e5b9100
> ffff88000002b040 0000000000026ea9
> Feb 8 18:19:15 handel kernel: ffff88007e5b92e8 0000000000000000
> Feb 8 18:19:15 handel kernel: Call Trace:
> Feb 8 18:19:15 handel kernel: [<ffffffff88036d5a>]
> :jbd:log_wait_commit+0xa3/0xf5
> Feb 8 18:19:15 handel kernel: [<ffffffff8029c48f>]
> autoremove_wake_function+0x0/0x2e
> Feb 8 18:19:16 handel kernel: [<ffffffff8803178a>]
> :jbd:journal_stop+0x1cf/0x1ff
> Feb 8 18:19:16 handel kernel: [<ffffffff8023119d>]
> __writeback_single_inode+0x1e9/0x328
> Feb 8 18:19:16 handel kernel: [<ffffffff802d330d>]
> do_readv_writev+0x26e/0x291
> Feb 8 18:19:16 handel kernel: [<ffffffff802e5b8b>] sync_inode+0x24/0x33
> Feb 8 18:19:16 handel kernel: [<ffffffff8804c36d>]
> :ext3:ext3_sync_file+0xc9/0xdc
> Feb 8 18:19:16 handel kernel: [<ffffffff80251e07>] do_fsync+0x52/0xa4
> Feb 8 18:19:16 handel kernel: [<ffffffff802d3b11>] __do_fsync+0x23/0x36
> Feb 8 18:19:16 handel kernel: [<ffffffff802602f9>] tracesys+0xab/0xb6
> Feb 8 18:19:16 handel kernel:
>
> Cheers,
> Pim
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
|
|
| |
|
Home