WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] GPLPV drivers block when copying huge files

Hi all,

I have the same problem here, I tried to find some help with google with no luck, finnally I received this message on the list with the solution! :-D

Well, what I could trace is that the problem occurs only when communicating between two DomUs and when using samba/CIFS. I made a test transfer with HTTP and if my memory don't fail, it worked.

I was facing this problem and I was needing a solution, the only that I could find was to put samba on Dom0 (ok, this is not good), and finally everything worked for me.

Some tcpdump showed me that I was having a lot of checksum errors when communicating from domU to domU, but I could not find a reason, but James explanation sounds great for me.

An additional information is that the problem also happen when transferring lot of files in a roll, not just with big files.

What is still missing for me is why the thing works just fine between dom0 and domU!?

Oh, one more thing, what is the best way to disable Large Send Offload? Is it possible to disable it just for a domU with some xen config parameter or do I have to do something like ethtook -k <bridge??or hwinterface> gso off, I'm a little confused here.

Thanks in advance.

Catunda.


James Harper escreveu:
Good news: Disabling "Large Send Offload" fixes the problem.

How does enabled LSO in the GPLPV driver interact with the LSO implemented
by
the NIC driver in Linux? The mentioned problem could be reproduced on
three
machines with Broadcom NICs (BCM5708 and BCM715), but wasn't triggered on
a
system with an onboard Nvidia (MCP51).


It's all a bit of a mystery... but I think it goes something like this:

LSO allows the operating system to send a large (>MTU) TCP packet to the 
hardware. The hardware then breaks the packet up into header+MSS sized chunks, 
recalculates the sequence numbers and checksums, and sends them. Efficiency gains 
are to be had in doing this.

In a DomU, if a packet is sent to the virtual adapter, where it flows onto the 
Dom0 Bridge interface, and then onto a real hardware interface, then it is also 
more efficient to keep the packet 'large' all the way to the physical nic where 
the hardware then finally breaks it up and sends it.

If, instead of going to a real hardware interface, the packet goes to another 
DomU, then it is also more efficient to keep the packet 'large' all the way to 
the DomU, and never bother breaking it up. Also, because the originating DomU 
marked the flag as 'checksum correct', the destination DomU never bothers 
checking the checksum, so you save doing the original checksum calculation, and 
all the inbetween checksum validation.

There's a lot that could go wrong there isn't there? Particularly if you start 
doing NAT or something funny on the bridge!

Windows support Large Send Offload in a mostly compatible way, at least under 
Windows 2003 server. XP has a known problem with interaction with the firewall 
service, but still seems to work okay once that is disabled. Unfortunately 
Windows likes to break the packet up into more pieces than the Linux backend 
can handle, so in the GPLPV drivers I allocate some memory pages and copy the 
packet data into them. This reduces the number of scatter-gather segments to a 
maximum of 16, which also reduces the amount of space each packet takes up on 
the ring.

What Windows doesn't support though is the Large Receive Offload, which the xen 
network backend driver assumes is supported of LSO is advertised as supported. 
So I have to fudge this - if Dom0 sends me a 'large' packet, I break it up 
before giving it to Windows.

Also, although Windows says that it supports RX checksum offload, where the 
network adapter (or xennet in this case) calculates the checksum and reports on 
it's correctness, it turns out that it doesn't really. If the checksum is never 
calculated because the packet originated on a Linux DomU, and I just tell 
Windows 'the checksum is correct, just trust me', it goes and checks anyway and 
then drops the packet when it turns out to not be correct.

The above two paragraphs are why the TX speed from the GPLPV drivers is much 
faster (~2x) than RX.

So there is plenty of room for me to have made errors there...

Is anything useful reported if you run debugview from sysinternals?

James

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users