[Xen-users] internal network performance problem

["Tibor Veres" <tibor.veres@xxxxxxxxx>]

i installed xen3.0.3 on debian sarge with 2 vms (also sarge), but i
was getting very bad inter-vm network performance (like 15-20Mbps max)

i converted the network setup from bridged to routed, and applied the
following kernel parameter tweaks on dom0 and domUs too:

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/wmem_default
echo 2569600 > /proc/sys/net/core/rmem_max
echo 2569600 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling
echo "40960 873800 1747600" > /proc/sys/net/ipv4/tcp_rmem
echo "40960 163840 1310720" > /proc/sys/net/ipv4/tcp_wmem


netperf-tcp output:

65:   12288 bytes    419 times -->    606.15 Mbps in     154.67 usec
66:   12291 bytes    431 times -->    624.73 Mbps in     150.10 usec

performance is acceptable until here, xm top shows full cpu
utilization, but after this point it drops badly. cpu utilization also
drops to <10%

67:   16381 bytes    222 times -->     28.96 Mbps in    4315.32 usec
68:   16384 bytes     11 times -->    583.25 Mbps in     214.32 usec
69:   16387 bytes    233 times -->     22.51 Mbps in    5553.60 usec
70:   24573 bytes      9 times -->      5.64 Mbps in   33220.50 usec
71:   24576 bytes      3 times -->    670.06 Mbps in     279.83 usec
72:   24579 bytes    238 times -->     23.08 Mbps in    8123.88 usec
73:   32765 bytes      4 times -->    714.72 Mbps in     349.76 usec
74:   32768 bytes    142 times -->     22.70 Mbps in   11011.61 usec
75:   32771 bytes      4 times -->     47.53 Mbps in    5259.99 usec
76:   49149 bytes      9 times -->    814.98 Mbps in     460.11 usec
77:   49152 bytes    144 times -->    145.25 Mbps in    2581.81 usec
78:   49155 bytes     25 times -->    887.54 Mbps in     422.54 usec
79:   65533 bytes     78 times -->     30.32 Mbps in   16489.71 usec
80:   65536 bytes      3 times -->     18.78 Mbps in   26617.32 usec
81:   65539 bytes      3 times -->     10.72 Mbps in   46664.85 usec
82:   98301 bytes      3 times -->      7.50 Mbps in   99982.16 usec
83:   98304 bytes      3 times -->      9.38 Mbps in   79990.15 usec

dom0 vif interfaces show some dropped packets, domUs do not.

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
         inet addr:192.168.100.128  Bcast:0.0.0.0  Mask:255.255.255.255
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:1457121 errors:0 dropped:0 overruns:0 frame:0
         TX packets:1458240 errors:0 dropped:970 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:2051710970 (1.9 GiB)  TX bytes:2069975753 (1.9 GiB)

vif2.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
         inet addr:192.168.99.128  Bcast:0.0.0.0  Mask:255.255.255.255
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:1459850 errors:0 dropped:0 overruns:0 frame:0
         TX packets:1457278 errors:0 dropped:924 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:2052283308 (1.9 GiB)  TX bytes:2069732297 (1.9 GiB)

dom0 has 512, domUs have 256M ram allocated, there is now swapping
involved, host is dual-xeon 3200, 4G ram, high-mem enabled


--
Tibor Veres
 tibor.veres@xxxxxxxxx

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users