http://bugzilla.xensource.com/cgi-bin/bugzilla/show_bug.cgi?id=143
Summary: Xen checksum optimization fails to handle IPSec, etc
Product: Xen
Version: unstable
Platform: All
OS/Version: Linux
Status: NEW
Severity: major
Priority: P2
Component: Guest-OS
AssignedTo: xen-bugs@xxxxxxxxxxxxxxxxxxx
ReportedBy: niv@xxxxxxxxxx
CC: hohnbaum@xxxxxxxxxx
Entering bug from post on xen-devel by Jonathan McCune.
See thread at:
http://marc.theaimsgroup.com/?l=xen-devel&m=112308642221935&w=2
System setup:
We have a domU sending packets through DOM0 to an external
host machine. We have setup an IPSEC tunnel between DOM0 and
the host machine through which the packets between domU and
the host machine are routed. We switched off MAC bridging for
this purpose, and configured all interfaces statically.
+ xen-unstable.hg from 7.28.2005.
+ Static IP configuration of DOM0 and one domU (NO MAC bridging).
+ IPsec enabled in DOM0 by editing linux-2.6.12-xen0/.config as
appropriate and recompiling.
+ IPsec using ESP in tunnel mode configured to tunnel all traffic
betweem domU and a particular external host (running linux-2.6.13-rc3
with IPsec enabled).
Problem:
Code for a checksum optimization imposed by Xen (basically, don't
checksum packets between domU and DOM0 since there is no real wire on
which they can become garbled) is not placed correctly. As-is, ESP
packets encapsulating IP packets from domU will be silently dropped by
DOM0. Using tcpdump on DOM0, the packets from domU can be seen arriving
in DOM0, but no ESP packet leaves DOM0 for the external host. It turns
out that an ESP packet is being created, but it gets dropped in
net/core/dev.c:dev_queue_xmit() in the switch(skb->nh.iph->protocol)
statement. It gets dropped here because the protocol is IPPROTO_ESP,
and that switch statement can only handle IPPROTO_[TCP|UDP]. The errno
returned is -ENOMEM. Debugging would have been significantly easier
with a more specific errno.
More info:
Xen gives its virtual network interfaces in domU domains the
NETIF_F_IP_CSUM feature flag, which is defined in
include/linux/netdevice.h to mean the interface is capable only of
checksumming TCP/UDP over IPv4. The expectation is that one can then
get away with not checksumming TCP/UDP packets at all as they pass
between domU and DOM0. This looks to me like a common-case optimization
and saves CPU cycles. Some code is then inserted in
net/core/dev.c:dev_queue_xmit() on DOM0 which puts in the checksum for
packets that are actually going on to the rest of the world. This
manifested itself as a problem for us in two ways:
1. The code in net/core/dev.c:dev_queue_xmit() (activated when
skb->proto_csum_blank == 1) can only handle TCP and UDP packets destined
for the rest of the world. ESP packets activate the `default:` case in
the switch() statement, and thus fail with the default errno in that
function: -ENOMEM.
2. I changed net/core/dev.c:dev_queue_xmit() to allow ESP through
unmolested just because I was curious. The ESP packets then went all
the way from DOM0 to the external host, where they were decrypted. Once
the tunneled packet was exposed, it was dropped on the remote system
because it did not have a valid checksum. In other words, the logic in
DOM0 (switch() statement in net/core/dev.c:dev_queue_xmit()) that is
supposed to insert the needed checksum into the original packet from
domU is too late.
Problem summarized:
The original packet from domU did not get the checksum it needed, and
the ESP packet created in DOM0 wasn't allowed to leave because the
too-late-code doesn't know how to handle ESP packets.
Temporary Solution:
We fixed this by removing the addition of flag NETIF_F_IP_CSUM in
drivers/xen/netfront/netfront.c:create_netdev(). I believe this tells
the kernel to just always do the checksum in software. Thus, the broken
optimization for TCP/UDP packets gets bypassed.
Permanent Solution:
???
That's why I posted this message... :-)
Cheers,
-Jonathan McCune
jonmccune@xxxxxxx
Keir's response:
"I suspect the ESP code would need to be made aware of the csum_blank
field, and fill in before forwarding. There are doubtless other paths
that may need similar tweaks (e.g., NAT IP masquerading is untested I
think, although there's a fair chance it'll just work).
Apart from the above 'proper fix', simple not-so-hacky solutions
include:
* Run 'ethtool -K tx off' in each domU
* Add an option to netback in domain0 to fill in checksums itself if
not done by domU.
* Allow netback to advertise to domUs whether it accepts
non-checksummed packets, and have an option to set this advertisement
when you start netback.
-- Keir"
---
We preferably want to do the first solution. Extended testing over IPSec, VLAN,
etc needed. IBM will shortly (in 2 weeks) be doing this, and expect to run into
the same bug..We'll try and get to this one soon.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
_______________________________________________
Xen-bugs mailing list
Xen-bugs@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-bugs
|
Home