[Xen-users] Network traffic stalling between domUs on different servers

[Adam Wilbraham <adam.wilbraham@xxxxxxxxxxxxxxxx>]

Up until now I've been using a single DL380 running Xen 3.3.1 on top of 
Debian Lenny (using the Debian Etch Xen kernel) to host a number of 
domUs, however I have had to install a second dom0 (this time a DL360, 
again Lenny + Etch kernel, this time Xen 3.4.2) to take over some of the 
domUs due to load issues.
Network connectivity between domUs on the same host has always been 
absolutely fine, however I am having a problem with network traffic 
between domUs on the new dom0 when connecting to domUs on the original 
dom0. Basically, connections can be established but as soon as data 
starts to flow then the connection stalls and eventually drops.
The network on each dom0 is configured as a series of VLAN bridges, eg:

 auto eth1.38
 iface eth1.38 inet manual

 auto vlan38
 iface vlan38 inet manual
        pre-up ip link set eth1.38 down
        pre-up brctl addbr vlan38
        pre-up brctl addif vlan38 eth1.38
        pre-up ip link set eth1.38 up
        pre-up ip link set vlan38 up
        post-down ip link set vlan38 down
        post-down brctl delbr vlan38

My head keeps telling me that this issue is related to TCP checksum 
offloading, but I've already turned that off and set the MTU to 1496 in 
case the network card doesn't cope with the VLAN header properly:
        pre-up ethtool -K eth1 tx off
        pre-up ifconfig eth1 mtu 1496

The NICs in both servers are Broadcom ones, bnx2 driver.

I managed to get an NFS export mounted and vaguely stable between domUs 
when forcing the it to use UDP and an rsize and wsize of 1024, however 
thats hardly workable and only for NFS. If I try and SCP a file between 
boxes then only a few k gets transferred before it stalls, similar from 
other services.
Does anyone have any experience of similar issues?

--
Adam Wilbraham - Senior Systems Administrator
TechnoPhobia Limited
The Workstation
15 Paternoster Row
SHEFFIELD
England
S1 2BX
t: +44 (0)114 2212123
f: +44 (0)114 2212124
e: adam.wilbraham@xxxxxxxxxxxxxxxx
w: http://www.technophobia.com/

Registered in England and Wales Company No. 3063669
VAT registration No. 598 7858 42
ISO 9001:2000 Accredited Company No. 21227
ISO 14001:2004 Accredited Company No. E997
ISO 27001:2005 (BS7799) Accredited Company No. IS 508906
Investor in People Certified No. 101507

The contents of this email are confidential to the addressee
and are intended solely for the recipients use. If you are not
the addressee, you have received this email in error.
Any disclosure, copying, distribution or action taken in
reliance on it is prohibited and may be unlawful.

Any opinions expressed in this email are those of the author
personally and not TechnoPhobia Limited who do not accept
responsibility for the contents of the message.

All email communications, in and out of TechnoPhobia,
are recorded for monitoring purposes.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users