I've seen the same problem with my xen 3.1.0 setup. What
the Xen gurus are telling us is that this is a symptom of Xen dom0
being busy and not servicing the network interrupts of the domu's
promptly. Their advice to us was to shift an application that
had been running on dom0 to another Xen instance to see if that
would help. We are in the process of implementing that solution now.
By the way my system (Dell poweredge2950) has got broadcomm
inbuilt network cards, not Intel E1000 so it is unlikely that
it is a network driver specific issue.
During these episodes of non-network connectivity, by the way,
it was not unusual to see the following kernel dump in dom0
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel: Call Trace:
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel: <IRQ>
[<ffffffff8025
8269>] softlockup_tick+0xcc/0xde
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel:
[<ffffffff8020e84d>]
timer_interrupt+0x3a3/0x401
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel:
[<ffffffff80258898>]
handle_IRQ_event+0x4b/0x93
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel:
[<ffffffff8025897e>]
__do_IRQ+0x9e/0x100
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel:
[<ffffffff8020cc97>]
do_IRQ+0x63/0x71
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel:
[<ffffffff8034b347>]
evtchn_do_upcall+0xee/0x165
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel:
[<ffffffff8020abca>]
do_hypervisor_callback+0x1e/0x2c
2008-02-05T18:35:16-06:00 s_sys@xxxxxxxxxxxxxxxxxxx kernel: <EOI>
or
Feb 25 10:32:39 fermigrid6 kernel: BUG: soft lockup detected on CPU#0!
Feb 25 10:32:39 fermigrid6 kernel:
Feb 25 10:32:39 fermigrid6 kernel: Call Trace:
Feb 25 10:32:39 fermigrid6 kernel: <IRQ> [<ffffffff80258269>]
softlockup_tick+0xcc/0xde
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8020e84d>]
timer_interrupt+0x3a3/0x401
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff80258898>]
handle_IRQ_event+0x4b/0x93
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8025897e>]
__do_IRQ+0x9e/0x100
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8020cc97>] do_IRQ+0x63/0x71
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8034b347>]
evtchn_do_upcall+0xee/0x165
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8020abca>]
do_hypervisor_callback+0x1e/0x2c
Feb 25 10:32:39 fermigrid6 kernel: <EOI> [<ffffffff8020622a>]
hypercall_page+0x22a/0x1000
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8020622a>]
hypercall_page+0x22a/0x1000
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8034b258>]
force_evtchn_callback+0xa/0xb
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff803f2272>]
thread_return+0xdf/0x119
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8020622a>]
hypercall_page+0x22a/0x1000
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff80228a25>]
__cond_resched+0x1c/0x44
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff803f25df>]
cond_resched+0x37/0x42
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff802343c4>]
ksoftirqd+0x0/0xbf
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff80234432>]
ksoftirqd+0x6e/0xbf
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff802422d7>] kthread+0xc8/0xf1
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8020ae1c>]
child_rip+0xa/0x12
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8024220f>] kthread+0x0/0xf1
Feb 25 10:32:39 fermigrid6 kernel: [<ffffffff8020ae12>]
child_rip+0x0/0x12
----------------
One of our dom0's was running an LVS server, the other one on identical
hardware was not. We moved the LVS server from one to the other and
the network problems and kernel panics followed it.
Steve Timm
On Mon, 3 Mar 2008, Marc Teichgraeber wrote:
Hi all,
I have a strange network problem with some domU's on three XEN-Hosts.
They are loosing their network connectivity. I do bridged networking.
* It happens randomly and could happen right after bootup of the domU
or anytime later.
* The domU is not reachable from another host on the LAN.
* The domU is always reachable from the dom0 (ssh, ping).
* I can 'repair' the connection when attaching to the console and
ping out from the domU. First nothings happens, then the machine gets
back their network. (And thats also my momentary workaround, pinging all
the time from the console)
* Pinging from another host at the same time helps too.
* It could be that I can ping continously from one host and another
hosts gets only every 10th packet or so back.
* The interfaces could come back from their sleep by itself.
* When the networks has fallen asleep, ssh on the domU from another
host hangs, it does not come back with "no route to host" or something.
I'm suspicious about the network controllers, they are the same on all
hosts: "Intel Corporation 80003ES2LAN Gigabit Ethernet Controller
(Copper)"(lspci) some kind of "Intel® PRO/1000 EB Network Connection
with I/O Acceleration"(Intel website). I've tried the latest e1000
driver from Intel but it does'nt helped.
I've checked all MAC Adresses, they are unique, also the IP Adresses.
Any ideas are welcome :)
-------------------------------------------------------------------------
"xm info" from host1, openSUSE 10.2 (X86-64):
release : 2.6.18.8-0.9-xen
version : #1 SMP Sun Feb 10 22:48:05 UTC 2008
machine : x86_64
nr_cpus : 4
nr_nodes : 1
sockets_per_node : 2
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2327
hw_caps :
bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
total_memory : 32766
free_memory : 21607
max_free_memory : 21607
max_para_memory : 21603
max_hvm_memory : 21544
xen_major : 3
xen_minor : 0
xen_extra : .3_11774-23
xen_caps : xen-3.0-x86_64
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : 11774
cc_compiler : gcc version 4.1.2 20061115 (prerelease) (SUSE
Linux)
cc_compile_by : abuild
cc_compile_domain : suse.de
cc_compile_date : Thu Jan 10 21:22:54 UTC 2008
xend_config_format : 2
-------------------------------------------------------------------------
"xm info" output on host2, openSUSE 10.3 (X86-64)
release : 2.6.22.13-0.3-xen
version : #1 SMP 2007/11/19 15:02:58 UTC
machine : x86_64
nr_cpus : 8
nr_nodes : 1
sockets_per_node : 2
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 3000
hw_caps :
bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
total_memory : 16382
free_memory : 591
max_free_memory : 591
max_para_memory : 587
max_hvm_memory : 577
xen_major : 3
xen_minor : 1
xen_extra : .0_15042-51
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : 15042
cc_compiler : gcc version 4.2.1 (SUSE Linux)
cc_compile_by : abuild
cc_compile_domain : suse.de
cc_compile_date : Tue Sep 25 21:16:06 UTC 2007
xend_config_format : 4
--
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
timm@xxxxxxxx http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
| 
Home