| |
|
 |
|
 |
|
|
|
| |
|
|
xen-devel
Re: [Xen-devel] Network dies and kernel errors
On Mon, Jul 25, 2011 at 02:18:21PM -0500, John McMonagle wrote:
> Have a new amd 6100 based server.
> http://www.supermicro.com/Aplus/system/2U/2022/AS-2022G-URF.cfm
> Running debian squeeze with debian 2.6.32 xen kernel
> Running xen 4.1.1 built from source from xen.org
>
> I'm seeing 2 errors.
> during boot get this:
>
> [ 0.004823] ------------[ cut here ]------------
> [ 0.004833] WARNING:
> at
> /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/build/source_amd64_xen/arch/x86/xen/enlighten.c:726
>
> init_hw_perf_events+0x32d/0x3cd()
> [ 0.004838] Hardware name: H8DGU
> [ 0.004841] Modules linked in:
> [ 0.004847] Pid: 0, comm: swapper Not tainted 2.6.32-5-xen-amd64 #1
> [ 0.004850] Call Trace:
> [ 0.004857] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> [ 0.004862] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> [ 0.004870] [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
> [ 0.004875] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> [ 0.004881] [<ffffffff813044dc>] ? identify_cpu+0x2f7/0x300
> [ 0.004888] [<ffffffff8100eccf>] ? xen_restore_fl_direct_end+0x0/0x1
> [ 0.004895] [<ffffffff810e81d5>] ? kmem_cache_alloc+0x8c/0xf0
> [ 0.004900] [<ffffffff81510a16>] ? identify_boot_cpu+0x15/0x3e
> [ 0.004904] [<ffffffff81510baa>] ? check_bugs+0x9/0x2e
> [ 0.004910] [<ffffffff81509cce>] ? start_kernel+0x3cd/0x3e8
> [ 0.004915] [<ffffffff8150bc93>] ? xen_start_kernel+0x586/0x58a
You can ignore that one. It just means that you can't do profiling which we
haven't
yet up-ported.
..
> Then next one may not be xen but I only had the problem after running a domu.
> After a while I get kernel error and networking stops.
And some other user with a bnx2 driver seems to see a similar problem. Let me
CC them here.
> This is the error:
> [ 1411.813376] ------------[ cut here ]------------
> [ 1411.813398] WARNING:
> at
> /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/build/source_amd64_xen/net/sched/s
> ch_generic.c:261 dev_watchdog+0xe2/0x194()
OK, this is one is more worrysome.
> [ 1411.813410] Hardware name: H8DGU
> [ 1411.813417] NETDEV WATCHDOG: peth0 (igb): transmit queue 1 timed out
> [ 1411.813424] Modules linked in: xt_physdev iptable_filter tun ip_tables
> x_tables bridge stp sg sr_mod cdrom xfs exportfs ipmi_si i
> pmi_devintf ipmi_watchdog ipmi_msghandler xen_evtchn blktap xenfs loop
> snd_pcm
> snd_timer snd soundcore snd_page_alloc pcspkr psmouse joydev evdev serio_raw
> i2c_piix
> 4 edac_core k10temp edac_mce_amd i2c_core processor button acpi_processor
> ext4
> mbcache jbd2 crc16 usbhid hid dm_mod raid1 md_mod sd_mod crc_t10dif
> ata_generic usb_s
> torage pata_atiixp ahci ohci_hcd libata ehci_hcd usbcore nls_base scsi_mod
> igb
> dca thermal thermal_sys [last unloaded: scsi_wait_scan]
> [ 1411.813656] Pid: 4, comm: ksoftirqd/0 Tainted: G W
> 2.6.32-5-xen-amd64 #1
> [ 1411.813664] Call Trace:
> [ 1411.813671] <IRQ> [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> [ 1411.813697] [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> [ 1411.813711] [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
> [ 1411.813724] [<ffffffff81272d60>] ? dev_watchdog+0x0/0x194
> [ 1411.813736] [<ffffffff8104ef88>] ? warn_slowpath_fmt+0x51/0x59
> [ 1411.813751] [<ffffffff8130d42a>] ? _spin_unlock_irqrestore+0xd/0xe
> [ 1411.813762] [<ffffffff8104b41e>] ? try_to_wake_up+0x289/0x29b
> [ 1411.813778] [<ffffffff81272d34>] ? netif_tx_lock+0x3d/0x69
> [ 1411.813791] [<ffffffff8125d7da>] ? netdev_drivername+0x3b/0x40
> [ 1411.813803] [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> [ 1411.813816] [<ffffffff8100ece2>] ? check_events+0x12/0x20
> [ 1411.813827] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
> [ 1411.813841] [<ffffffff8105b5ef>] ? run_timer_softirq+0x1c9/0x268
> [ 1411.813855] [<ffffffff81054c9b>] ? __do_softirq+0xdd/0x1a6
> [ 1411.813867] [<ffffffff81012cac>] ? call_softirq+0x1c/0x30
> [ 1411.813873] <EOI> [<ffffffff8101422b>] ? do_softirq+0x3f/0x7c
> [ 1411.813893] [<ffffffff810548c2>] ? ksoftirqd+0x5f/0xd3
> [ 1411.813905] [<ffffffff81054863>] ? ksoftirqd+0x0/0xd3
> [ 1411.813915] [<ffffffff81065c39>] ? kthread+0x79/0x81
> [ 1411.813926] [<ffffffff81012baa>] ? child_rip+0xa/0x20
> [ 1411.813937] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
> [ 1411.813948] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
> [ 1411.813958] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
> [ 1411.813966] ---[ end trace a7919e7f17c0a727 ]---
> [ 1412.052253] eth0: port 1(peth0) entering disabled state
> [ 1635.796207] frontend_changed: backend/vbd/3/768: prepare for reconnect
> [ 1647.137513] eth0: port 3(vif3.0) entering disabled state
> [ 1647.157527] eth0: port 3(vif3.0) entering disabled state
> Kernel logging (proc) stopped.
>
> In this case dom0 locked up. Some times just networking stops and some times
> networking recovers.
>
> Looks like it uses msi-x interrupts.
>
> Concerning igb error I have tried the following one at a time:
> New igb driver from Intel site.
> kernel parameter pcie_aspm=off
> ethtool -K eth0 tx off on dom0
> ethtool -K eth0 gro off on dom0
>
OK.
> It has never died doing iperf from dom0 or domu <> external.
> Never died during network backup.
>
> Usually takes a least a few hours and has never made it a day running a domu.
> Wish I could get it to die faster :-)
> Any ideas?
> I'm pretty much down to trying different network cards
Did you try that? Did that make any difference?
>
> Any ideas?
There is a Xen parameter called 'noirqbalance' . Try that. Also see if you can
limit the CPUs in the dom0 using these two arguments on Xen hypervisor:
dom0_vcpus=2 dom0_vcpus_pin=1
It would be interesting to narrow down _when_ you trigger this failure. B/c we
can pull Xen to see what the MSI's are 'xl debug-keys M' _before_ and _after_
your
failure to see if something is amiss.
Mainly to figure out if the vectors are moving around the CPUs (or not)
(XEN) MSI 29 vec=21 lowest edge assert log lowest dest=00000001
mask=0/0/-1
and also 'xl debug-keys i' to see if the domain has ACK-ed the interrupt:
(XEN) IRQ: 29 affinity:00000000,00000000,00000000,00000001 vec:21
type=PCI-MSI status=00000010 in-flight=0 domain-list=0:275(----),
(the last '----' might have something else in in them - if so that is a sign
that
dom0 hasn't picked up the event/vector).
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
|
|
| |
|
Home