|
xen-users
Re: [Xen-users] GPLPV drivers block when copying huge files
Hi all,
I have the same problem here, I tried to find some help with google with
no luck, finnally I received this message on the list with the solution! :-D
Well, what I could trace is that the problem occurs only when
communicating between two DomUs and when using samba/CIFS. I made a test
transfer with HTTP and if my memory don't fail, it worked.
I was facing this problem and I was needing a solution, the only that I
could find was to put samba on Dom0 (ok, this is not good), and finally
everything worked for me.
Some tcpdump showed me that I was having a lot of checksum errors when
communicating from domU to domU, but I could not find a reason, but
James explanation sounds great for me.
An additional information is that the problem also happen when
transferring lot of files in a roll, not just with big files.
What is still missing for me is why the thing works just fine between
dom0 and domU!?
Oh, one more thing, what is the best way to disable Large Send Offload?
Is it possible to disable it just for a domU with some xen config
parameter or do I have to do something like ethtook -k <bridge??or
hwinterface> gso off, I'm a little confused here.
Thanks in advance.
Catunda.
James Harper escreveu:
Good news: Disabling "Large Send Offload" fixes the problem.
How does enabled LSO in the GPLPV driver interact with the LSO implemented
by
the NIC driver in Linux? The mentioned problem could be reproduced on
three
machines with Broadcom NICs (BCM5708 and BCM715), but wasn't triggered on
a
system with an onboard Nvidia (MCP51).
It's all a bit of a mystery... but I think it goes something like this:
LSO allows the operating system to send a large (>MTU) TCP packet to the
hardware. The hardware then breaks the packet up into header+MSS sized chunks,
recalculates the sequence numbers and checksums, and sends them. Efficiency gains
are to be had in doing this.
In a DomU, if a packet is sent to the virtual adapter, where it flows onto the
Dom0 Bridge interface, and then onto a real hardware interface, then it is also
more efficient to keep the packet 'large' all the way to the physical nic where
the hardware then finally breaks it up and sends it.
If, instead of going to a real hardware interface, the packet goes to another
DomU, then it is also more efficient to keep the packet 'large' all the way to
the DomU, and never bother breaking it up. Also, because the originating DomU
marked the flag as 'checksum correct', the destination DomU never bothers
checking the checksum, so you save doing the original checksum calculation, and
all the inbetween checksum validation.
There's a lot that could go wrong there isn't there? Particularly if you start
doing NAT or something funny on the bridge!
Windows support Large Send Offload in a mostly compatible way, at least under
Windows 2003 server. XP has a known problem with interaction with the firewall
service, but still seems to work okay once that is disabled. Unfortunately
Windows likes to break the packet up into more pieces than the Linux backend
can handle, so in the GPLPV drivers I allocate some memory pages and copy the
packet data into them. This reduces the number of scatter-gather segments to a
maximum of 16, which also reduces the amount of space each packet takes up on
the ring.
What Windows doesn't support though is the Large Receive Offload, which the xen
network backend driver assumes is supported of LSO is advertised as supported.
So I have to fudge this - if Dom0 sends me a 'large' packet, I break it up
before giving it to Windows.
Also, although Windows says that it supports RX checksum offload, where the
network adapter (or xennet in this case) calculates the checksum and reports on
it's correctness, it turns out that it doesn't really. If the checksum is never
calculated because the packet originated on a Linux DomU, and I just tell
Windows 'the checksum is correct, just trust me', it goes and checks anyway and
then drops the packet when it turns out to not be correct.
The above two paragraphs are why the TX speed from the GPLPV drivers is much
faster (~2x) than RX.
So there is plenty of room for me to have made errors there...
Is anything useful reported if you run debugview from sysinternals?
James
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
|
|
Home