[Xen-devel] Xen checksumming bug with IPsec ESP packets
We have a domU sending packets through DOM0 to an external
host machine. We have setup an IPSEC tunnel between DOM0 and
the host machine through which the packets between domU and
the host machine are routed. We switched off MAC bridging for
this purpose, and configured all interfaces statically.
+ xen-unstable.hg from 7.28.2005.
+ Static IP configuration of DOM0 and one domU (NO MAC bridging).
+ IPsec enabled in DOM0 by editing linux-2.6.12-xen0/.config as
appropriate and recompiling.
+ IPsec using ESP in tunnel mode configured to tunnel all traffic
betweem domU and a particular external host (running linux-2.6.13-rc3
with IPsec enabled).
Code for a checksum optimization imposed by Xen (basically, don't
checksum packets between domU and DOM0 since there is no real wire on
which they can become garbled) is not placed correctly. As-is, ESP
packets encapsulating IP packets from domU will be silently dropped by
DOM0. Using tcpdump on DOM0, the packets from domU can be seen arriving
in DOM0, but no ESP packet leaves DOM0 for the external host. It turns
out that an ESP packet is being created, but it gets dropped in
net/core/dev.c:dev_queue_xmit() in the switch(skb->nh.iph->protocol)
statement. It gets dropped here because the protocol is IPPROTO_ESP,
and that switch statement can only handle IPPROTO_[TCP|UDP]. The errno
returned is -ENOMEM. Debugging would have been significantly easier
with a more specific errno.
Xen gives its virtual network interfaces in domU domains the
NETIF_F_IP_CSUM feature flag, which is defined in
include/linux/netdevice.h to mean the interface is capable only of
checksumming TCP/UDP over IPv4. The expectation is that one can then
get away with not checksumming TCP/UDP packets at all as they pass
between domU and DOM0. This looks to me like a common-case optimization
and saves CPU cycles. Some code is then inserted in
net/core/dev.c:dev_queue_xmit() on DOM0 which puts in the checksum for
packets that are actually going on to the rest of the world. This
manifested itself as a problem for us in two ways:
1. The code in net/core/dev.c:dev_queue_xmit() (activated when
skb->proto_csum_blank == 1) can only handle TCP and UDP packets destined
for the rest of the world. ESP packets activate the `default:` case in
the switch() statement, and thus fail with the default errno in that
2. I changed net/core/dev.c:dev_queue_xmit() to allow ESP through
unmolested just because I was curious. The ESP packets then went all
the way from DOM0 to the external host, where they were decrypted. Once
the tunneled packet was exposed, it was dropped on the remote system
because it did not have a valid checksum. In other words, the logic in
DOM0 (switch() statement in net/core/dev.c:dev_queue_xmit()) that is
supposed to insert the needed checksum into the original packet from
domU is too late.
The original packet from domU did not get the checksum it needed, and
the ESP packet created in DOM0 wasn't allowed to leave because the
too-late-code doesn't know how to handle ESP packets.
We fixed this by removing the addition of flag NETIF_F_IP_CSUM in
drivers/xen/netfront/netfront.c:create_netdev(). I believe this tells
the kernel to just always do the checksum in software. Thus, the broken
optimization for TCP/UDP packets gets bypassed.
That's why I posted this message... :-)
Xen-devel mailing list