This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


[Xen-devel] Xen checksumming bug with IPsec ESP packets

To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] Xen checksumming bug with IPsec ESP packets
From: "Jonathan M. McCune" <jonmccune@xxxxxxx>
Date: Wed, 03 Aug 2005 12:27:00 -0400
Cc: caceres@xxxxxxxxxx, jaegert@xxxxxxxxxx, sailer@xxxxxxxxxx
Delivery-date: Wed, 03 Aug 2005 16:25:05 +0000
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla Thunderbird 1.0.2 (Windows/20050317)
System setup:

We have a domU sending packets through DOM0 to an external
host machine. We have setup an IPSEC tunnel between DOM0 and
the host machine through which the packets between domU and
the host machine are routed. We switched off MAC bridging for
this purpose, and configured all interfaces statically.

+ xen-unstable.hg from 7.28.2005.
+ Static IP configuration of DOM0 and one domU (NO MAC bridging).
+ IPsec enabled in DOM0 by editing linux-2.6.12-xen0/.config as appropriate and recompiling. + IPsec using ESP in tunnel mode configured to tunnel all traffic betweem domU and a particular external host (running linux-2.6.13-rc3 with IPsec enabled).


Code for a checksum optimization imposed by Xen (basically, don't checksum packets between domU and DOM0 since there is no real wire on which they can become garbled) is not placed correctly. As-is, ESP packets encapsulating IP packets from domU will be silently dropped by DOM0. Using tcpdump on DOM0, the packets from domU can be seen arriving in DOM0, but no ESP packet leaves DOM0 for the external host. It turns out that an ESP packet is being created, but it gets dropped in net/core/dev.c:dev_queue_xmit() in the switch(skb->nh.iph->protocol) statement. It gets dropped here because the protocol is IPPROTO_ESP, and that switch statement can only handle IPPROTO_[TCP|UDP]. The errno returned is -ENOMEM. Debugging would have been significantly easier with a more specific errno.

More info:

Xen gives its virtual network interfaces in domU domains the NETIF_F_IP_CSUM feature flag, which is defined in include/linux/netdevice.h to mean the interface is capable only of checksumming TCP/UDP over IPv4. The expectation is that one can then get away with not checksumming TCP/UDP packets at all as they pass between domU and DOM0. This looks to me like a common-case optimization and saves CPU cycles. Some code is then inserted in net/core/dev.c:dev_queue_xmit() on DOM0 which puts in the checksum for packets that are actually going on to the rest of the world. This manifested itself as a problem for us in two ways:

1. The code in net/core/dev.c:dev_queue_xmit() (activated when skb->proto_csum_blank == 1) can only handle TCP and UDP packets destined for the rest of the world. ESP packets activate the `default:` case in the switch() statement, and thus fail with the default errno in that function: -ENOMEM. 2. I changed net/core/dev.c:dev_queue_xmit() to allow ESP through unmolested just because I was curious. The ESP packets then went all the way from DOM0 to the external host, where they were decrypted. Once the tunneled packet was exposed, it was dropped on the remote system because it did not have a valid checksum. In other words, the logic in DOM0 (switch() statement in net/core/dev.c:dev_queue_xmit()) that is supposed to insert the needed checksum into the original packet from domU is too late.

Problem summarized:

The original packet from domU did not get the checksum it needed, and the ESP packet created in DOM0 wasn't allowed to leave because the too-late-code doesn't know how to handle ESP packets.

Temporary Solution:

We fixed this by removing the addition of flag NETIF_F_IP_CSUM in drivers/xen/netfront/netfront.c:create_netdev(). I believe this tells the kernel to just always do the checksum in software. Thus, the broken optimization for TCP/UDP packets gets bypassed.

Permanent Solution:


That's why I posted this message... :-)

-Jonathan McCune

Xen-devel mailing list

<Prev in Thread] Current Thread [Next in Thread>