[Xen-devel] Massive UDP packet dropping

[TRABE Patrick <trabe@xxxxxxxxxxxxxxx>]

Hi all,

We encounter an important problem with Xen3.1:
UDP packets are massively dropped when they are forwarded by a virtual
machine (the VM is running linux fedora FC5).

My tests are done as following :
We create a UDP traffic (using Iperf) between a client and a server (2
different equipments).
IPerf generates between 10000 to 50000 UDP pps.
The client and the server are connected to a third computer running Xen.
Note that client, server, and Xen are running on P4 with 512MB memory.
The computer runnung Xen owns 2 physical interfaces, one (called peth0
on the next scheme)connected to the server, and one (called peth1 on the
next scheme)connected to the client.


server (eth0: 10.0.0.1)<--->peth0 [ (eth0: 10.0.0.2) XEN VM1 LINUX FC5
(eth1: 10.0.1.2) ] (eth0: 10.0.1.1) peth1<---> client


When the traffic is generated and exceeds a threshold of 12000 pps,
around 20% (this number fluctuates +- 5%) of UDP packets are dropped.
Of course we have made similar tests without Xen, and measures show a
correct behavior of our testbed (ie no packets are dropped).
Thus virtualization is the cause of this massive packet dropping.

We are currently trying to fix this problem.
We have traced out that the buffer of the IO ring is involved. IO is a
well-known bottleneck with Xen as explained by the literature (e.g. HP
and IBM papers).
We have identified that in the file "drivers/xen/netback/netback.c, the
packets are dropped by the fonction "netif_be_start_xmit" since
netif->rx_req_cons_peek==netif->rx.sring->req_prod

We have searched where netif->rx_req_cons_peek &
netif->rx.sring->req_prod are defined.
netif->rx_req_cons_peek is defined in common.h such as "allow to peek
ahead in the request ring" .
netif->rx.sring->req_prod is the limit on request indices from
netif->rx.sring and is declared and initialized in
include/xen/interface/io/ring.h.
netif->rx.sring->req_prod is initialized with the value 0. However, we
have noticed that packets are dropped before netif->rx.sring->req_prod
is modified. This constitutes a surprising result.


Currently we have difficulties to locate where and how theses values are
modified.
Has someone a response for this problem ?


Thanks for your help.






_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel