=========================crash log for machine one in 2.6.32.10 ================================================ INIT: Id "s0" respawning too fast: disabled for 5 minutes blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800b7a62600 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800abb3d200 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800ab3a3000 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800a7ef8e00 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800bd224e00 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800bf09f400 blktap_sysfs_destroy blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800a96b3c00 blktap_sysfs_create: adding attributes for dev ffff8800bf09ec00 INIT: Id "s0" respawning too fast: disabled for 5 minutes blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800b8daae00 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800b0ea5400 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800b8dab200 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800b8daa600 INIT: Id "s0" respawning too fast: disabled for 5 minutes blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800ab933200 blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800b6514000 BUG: scheduling while atomic: swapper/0/0x10000100 Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure serio_raw bnx2 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd pata_acpi soundcore ata_generic snd_page_alloc pcspkr iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table] CPU 0: Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure serio_raw bnx2 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd pata_acpi soundcore ata_generic snd_page_alloc pcspkr iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table] Pid: 0, comm: swapper Not tainted 2.6.32.10xen #1 Tecal RH2285 RIP: e030:[] [] hypercall_page+0x3aa/0x1000 RSP: e02b:ffffffff81663ed8 EFLAGS: 00000246 RAX: 0000000000000000 RBX: ffffffff81662000 RCX: ffffffff810093aa RDX: ffffffff8100f23f RSI: 0000000000000000 RDI: 0000000000000001 RBP: ffffffff81663ef0 R08: 0000000000000000 R09: ffff880028092e08 R10: ffff880159558000 R11: 0000000000000246 R12: 0000000000000000 R13: 6db6db6db6db6db7 R14: ffffffff817d73a0 R15: 0000000000000000 FS: 00007f50215db6e0(0000) GS:ffff88002802c000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f449672d000 CR3: 00000000a7b0a000 CR4: 0000000000002660 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: [] ? xen_safe_halt+0x10/0x1a [] xen_idle+0x3c/0x46 [] cpu_idle+0x5d/0x8c [] rest_init+0x66/0x68 [] start_kernel+0x3ef/0x3fb [] x86_64_start_reservations+0xae/0xb2 [] xen_start_kernel+0x4c0/0x4c7 divide error: 0000 [#1] SMP last sysfs file: /sys/class/net/d342dd/address CPU 0 Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure serio_raw bnx2 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd pata_acpi soundcore ata_generic snd_page_alloc pcspkr iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table] Pid: 0, comm: swapper Not tainted 2.6.32.10xen #1 Tecal RH2285 RIP: e030:[] [] find_busiest_group+0x37d/0x721 RSP: e02b:ffff88002802fc90 EFLAGS: 00010246 RAX: 0000000000003c00 RBX: 0000000000000000 RCX: ffff880028041501 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040 RBP: ffff88002802fdf0 R08: 0000000000000000 R09: ffff88002803be08 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000040 R13: ffff88002803bdf0 R14: ffff88002803bce0 R15: 0000000000000000 FS: 00007f50215db6e0(0000) GS:ffff88002802c000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f449672d000 CR3: 00000000a7b0a000 CR4: 0000000000002660 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper (pid: 0, threadinfo ffffffff81662000, task ffffffff816e8980) Stack: 00001a0e95f9ec56 ffff88002803ba48 ffff88002802fe4c 000000008100eb79 <0> ffff88002802fe40 000000008100f252 0000000000003c00 0000000000000001 <0> ffff88002803be00 0000000100000010 ffff88002803bdf0 ffffffff8102edf9 Call Trace: [] ? pvclock_clocksource_read+0x47/0x80 [] ? xen_restore_fl_direct_end+0x0/0x1 [] rebalance_domains+0x17b/0x45b [] ? wake_up_process+0x15/0x17 [] run_rebalance_domains+0x40/0xc5 [] __do_softirq+0xd2/0x194 [] call_softirq+0x1c/0x30 [] do_softirq+0x46/0x87 [] irq_exit+0x3b/0x7a [] xen_evtchn_do_upcall+0x156/0x172 [] xen_do_hypervisor_callback+0x1e/0x30 [] ? hypercall_page+0x3aa/0x1000 [] ? xen_restore_fl_direct_end+0x0/0x1 [] ? hypercall_page+0x3aa/0x1000 [] ? xen_safe_halt+0x10/0x1a [] ? xen_idle+0x3c/0x46 [] ? cpu_idle+0x5d/0x8c [] ? rest_init+0x66/0x68 [] ? start_kernel+0x3ef/0x3fb [] ? x86_64_start_reservations+0xae/0xb2 [] ? xen_start_kernel+0x4c0/0x4c7 Code: 83 7d 10 00 74 0c 48 8b 5d 10 c7 03 00 00 00 00 eb 70 41 8b 55 08 48 8b 45 a8 48 89 d3 48 c1 a5 d0 fe ff ff 0a 48 c1 e0 0a 31 d2 <48> f7 f3 48 89 45 a0 48 8b 85 08 ff ff ff 48 29 85 00 ff ff ff RIP [] find_busiest_group+0x37d/0x721 RSP ---[ end trace 7c3e3b64ca341f0a ]--- divide error: 0000 [#2] SMP last sysfs file: /sys/class/net/d342dd/address CPU 2 Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure serio_raw bnx2 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd pata_acpi soundcore ata_generic snd_page_alloc pcspkr iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table] Pid: 0, comm: swapper Tainted: G D 2.6.32.10xen #1 Tecal RH2285 RIP: e030:[] [] find_busiest_group+0x37d/0x721 RSP: e02b:ffff880028069c90 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff880028041500 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040 RBP: ffff880028069df0 R08: 0000000000000000 R09: ffff88002803be08 R10: ffff880028069d28 R11: ffff88015f8f7e48 R12: 0000000000000040 R13: ffff88002803bdf0 R14: ffff880028058ce0 R15: 0000000000000001 FS: 00007faad3ba8730(0000) GS:ffff880028066000(0000) knlGS:0000000000000000 CS: e033 DS: 002b ES: 002b CR0: 000000008005003b CR2: 00007fd1e7ed9000 CR3: 000000015510a000 CR4: 0000000000002660 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper (pid: 0, threadinfo ffff88015f8f6000, task ffff88015f8e4410) Stack: 00001a0e95fdb97b ffff880028075a48 ffff880028069e4c 000000008100eb79 <0> ffff880028069e40 000000008100f252 0000000000003c00 ffffffff00000000 <0> ffff88002803be00 0000000100000010 ffff880028058df0 0000000000000002 Call Trace: [] ? tick_dev_program_event+0x2f/0xa1 [] rebalance_domains+0x17b/0x45b [] run_rebalance_domains+0x75/0xc5 [] __do_softirq+0xd2/0x194 [] call_softirq+0x1c/0x30 [] do_softirq+0x46/0x87 [] irq_exit+0x3b/0x7a [] xen_evtchn_do_upcall+0x156/0x172 [] xen_do_hypervisor_callback+0x1e/0x30 [] ? hypercall_page+0x3aa/0x1000 [] ? hypercall_page+0x3aa/0x1000 [] ? xen_safe_halt+0x10/0x1a [] ? xen_idle+0x3c/0x46 [] ? cpu_idle+0x5d/0x8c [] ? cpu_bringup_and_idle+0x13/0x15 Code: 83 7d 10 00 74 0c 48 8b 5d 10 c7 03 00 00 00 00 eb 70 41 8b 55 08 48 8b 45 a8 48 89 d3 48 c1 a5 d0 fe ff ff 0a 48 c1 e0 0a 31 d2 <48> f7 f3 48 89 45 a0 48 8b 85 08 ff ff ff 48 29 85 00 ff ff ff RIP [] find_busiest_group+0x37d/0x721 RSP ---[ end trace 7c3e3b64ca341f0b ]--- Kernel panic - not syncing: Fatal exception in interrupt Pid: 0, comm: swapper Tainted: G D 2.6.32.10xen #1 Call Trace: [] ? ftrace_profile_enable_sched_process_exit+0x10/0x17 [] panic+0xe0/0x198 [] ? xen_force_evtchn_callback+0xd/0xf [] ? check_events+0x12/0x20 [] ? xen_restore_fl_direct_end+0x0/0x1 [] ? ftrace_profile_enable_sched_process_exit+0x10/0x17 [] ? print_oops_end_marker+0x23/0x25 [] ? ftrace_profile_enable_sched_process_exit+0x10/0x17 [] oops_end+0xb6/0xc6 [] die+0x5a/0x63 [] do_trap+0x115/0x124 [] do_divide_error+0x96/0x9f [] ? find_busiest_group+0x37d/0x721 [] ? xen_restore_fl_direct_end+0x0/0x1 [] ? _spin_unlock_irqrestore+0x15/0x17 [] ? skb_release_data+0xab/0xb0 [] ? xen_force_evtchn_callback+0xd/0xf [] ? check_events+0x12/0x20 [] divide_error+0x1b/0x20 [] ? find_busiest_group+0x37d/0x721 [] ? tick_dev_program_event+0x2f/0xa1 [] rebalance_domains+0x17b/0x45b [] run_rebalance_domains+0x75/0xc5 [] __do_softirq+0xd2/0x194 [] call_softirq+0x1c/0x30 [] do_softirq+0x46/0x87 [] irq_exit+0x3b/0x7a [] xen_evtchn_do_upcall+0x156/0x172 [] xen_do_hypervisor_callback+0x1e/0x30 [] ? hypercall_page+0x3aa/0x1000 [] ? hypercall_page+0x3aa/0x1000 [] ? xen_safe_halt+0x10/0x1a [] ? xen_idle+0x3c/0x46 [] ? cpu_idle+0x5d/0x8c [] ? cpu_bringup_and_idle+0x13/0x15 Kernel panic - not syncing: Fatal exception in interrupt Pid: 0, comm: swapper Tainted: G D 2.6.32.10xen #1 Call Trace: [] panic+0xe0/0x198 [] ? megaraid_probe_one+0xf25/0x116d [] ? xen_force_evtchn_callback+0xd/0xf [] ? check_events+0x12/0x20 [] ? xen_restore_fl_direct_end+0x0/0x1 [] ? print_oops_end_marker+0x23/0x25 [] oops_end+0xb6/0xc6 [] die+0x5a/0x63 [] do_trap+0x115/0x124 [] do_divide_error+0x96/0x9f [] ? find_busiest_group+0x37d/0x721 [] ? __kfree_skb+0x79/0x7d [] ? xen_restore_fl_direct_end+0x0/0x1 [] ? kmem_cache_free+0x88/0xbb [] ? __kfree_skb+0x79/0x7d [] divide_error+0x1b/0x20 [] ? find_busiest_group+0x37d/0x721 [] ? pvclock_clocksource_read+0x47/0x80 [] ? xen_restore_fl_direct_end+0x0/0x1 [] rebalance_domains+0x17b/0x45b [] ? wake_up_process+0x15/0x17 [] run_rebalance_domains+0x40/0xc5 [] __do_softirq+0xd2/0x194 [] call_softirq+0x1c/0x30 [] do_softirq+0x46/0x87 [] irq_exit+0x3b/0x7a [] xen_evtchn_do_upcall+0x156/0x172 [] xen_do_hypervisor_callback+0x1e/0x30 [] ? hypercall_page+0x3aa/0x1000 [] ? xen_restore_fl_direct_end+0x0/0x1 [] ? hypercall_page+0x3aa/0x1000 [] ? xen_safe_halt+0x10/0x1a [] ? xen_idle+0x3c/0x46 [] ? cpu_idle+0x5d/0x8c [] ? rest_init+0x66/0x68 [] ? start_kernel+0x3ef/0x3fb [] ? x86_64_start_reservations+0xae/0xb2 [] ? xen_start_kernel+0x4c0/0x4c7 =========================crash log for machine two in 2.6.32.10 ================================================ blktap_sysfs_create: adding attributes for dev ffff88010b668c00 __ratelimit: 4 callbacks suppressed blktap_sysfs_create: adding attributes for dev ffff8800bd384200 __ratelimit: 6 callbacks suppressed INIT: Id "s0" respawning too fast: disabled for 5 minutes blktap_sysfs_destroy blktap_sysfs_create: adding attributes for dev ffff8800bf07d800 INIT: Id "s0" respawning too fast: disabled for 5 minutes divide error: 0000 [#1] SMP last sysfs file: /sys/hypervisor/type CPU 2 Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss bnx2 snd_pcm serio_raw snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core pcspkr pata_acpi ata_generic iTCO_wdt iTCO_vendor_support ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table] Pid: 19632, comm: xenstore-list Not tainted 2.6.32.10xen #1 Tecal RH2285 RIP: e030:[] [] find_busiest_group+0x37d/0x721 RSP: e02b:ffff8800b52a1c08 EFLAGS: 00010046 RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88002807b501 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040 RBP: ffff8800b52a1d68 R08: 0000000000000000 R09: ffff880028075e08 R10: ffff8800bf0cf6c0 R11: ffff8800b52a1dd8 R12: 0000000000000040 R13: ffff880028075df0 R14: ffff880028075ce0 R15: 0000000000000002 FS: 00007fcb228146e0(0000) GS:ffff880028066000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000003581d180e0 CR3: 00000000b83b5000 CR4: 0000000000002660 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xenstore-list (pid: 19632, threadinfo ffff8800b52a0000, task ffff88010c080000) Stack: ffff8800b52a1c20 ffff880028075a48 ffff8800b52a1dbc 00000002b52a1c30 <0> ffff8800b52a1db0 0000000000000004 0000000000000000 0000000200000000 <0> ffff880028075e00 0000000000000001 ffff880028075df0 ffffffff2807b5c0 Call Trace: [] schedule+0x27a/0x736 [] ? _spin_unlock_irqrestore+0x15/0x17 [] read_reply+0x86/0x104 [] ? autoremove_wake_function+0x0/0x3d [] ? _cond_resched+0xe/0x22 [] xenbus_dev_request_and_reply+0x58/0x89 [] xenbus_file_write+0x16a/0x469 [xenfs] [] vfs_write+0xb0/0x10a [] sys_write+0x4c/0x72 [] system_call_fastpath+0x16/0x1b Code: 83 7d 10 00 74 0c 48 8b 5d 10 c7 03 00 00 00 00 eb 70 41 8b 55 08 48 8b 45 a8 48 89 d3 48 c1 a5 d0 fe ff ff 0a 48 c1 e0 0a 31 d2 <48> f7 f3 48 89 45 a0 48 8b 85 08 ff ff ff 48 29 85 00 ff ff ff RIP [] find_busiest_group+0x37d/0x721 RSP ---[ end trace 13509d88f5b8918c ]--- divide error: 0000 [#2] SMP last sysfs file: /sys/hypervisor/type CPU 1 Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss bnx2 snd_pcm serio_raw snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core pcspkr pata_acpi ata_generic iTCO_wdt iTCO_vendor_support ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table] Pid: 3429, comm: kipmi0 Tainted: G D 2.6.32.10xen #1 Tecal RH2285 RIP: e030:[] [] find_busiest_group+0x37d/0x721 RSP: e02b:ffff880155e2dc70 EFLAGS: 00010046 RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88002807b500 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040 RBP: ffff880155e2ddd0 R08: 0000000000000000 R09: ffff880028075e08 R10: ffff880158339400 R11: ffff880028040000 R12: 0000000000000040 R13: ffff880028075df0 R14: ffff880028058ce0 R15: 0000000000000001 FS: 00007fde594896e0(0000) GS:ffff880028049000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000469000 CR3: 0000000001001000 CR4: 0000000000002660 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kipmi0 (pid: 3429, threadinfo ffff880155e2c000, task ffff880155cf2d60) Stack: ffff880155e2dc88 ffff880028058a48 ffff880155e2de24 0000000255e2dc98 <0> ffff880155e2de18 0000000000000004 0000000000000000 ffffffff00000000 <0> ffff880028075e00 0000000028040000 ffff880028058df0 000000002803be08 Call Trace: [] schedule+0x27a/0x736 [] ? _spin_unlock_irqrestore+0x15/0x17 [] schedule_timeout+0x9d/0xc4 [] ? process_timeout+0x0/0x10 [] schedule_timeout_interruptible+0x1e/0x20 [] ipmi_thread+0x6a/0x7e [ipmi_si] [] ? ipmi_thread+0x0/0x7e [ipmi_si] [] kthread+0x6e/0x76 [] child_rip+0xa/0x20 [] ? int_ret_from_sys_call+0x7/0x1b [] ? retint_restore_args+0x5/0x6 [] ? child_rip+0x0/0x20 Code: 83 7d 10 00 74 0c 48 8b 5d 10 c7 03 00 00 00 00 eb 70 41 8b 55 08 48 8b 45 a8 48 89 d3 48 c1 a5 d0 fe ff ff 0a 48 c1 e0 0a 31 d2 <48> f7 f3 48 89 45 a0 48 8b 85 08 ff ff ff 48 29 85 00 ff ff ff RIP [] find_busiest_group+0x37d/0x721 RSP ---[ end trace 13509d88f5b8918d ]---