WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] internal network performance problem

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] internal network performance problem
From: "Tibor Veres" <tibor.veres@xxxxxxxxx>
Date: Wed, 6 Dec 2006 10:30:43 +0100
Delivery-date: Wed, 06 Dec 2006 01:30:50 -0800
Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=BDAaKk5mEFJ1fdFD4DMy8fmJgWb039mr3ZRYwRiIlz+LnTmIsUZgVDlpnql0njBqwpXoTSFDq1iSGpTzOWewPy7O2MajsOOtrYFR9Mx/aC2b3OUqdQSOJUJoFGGRUq6J3J/Lt2s/xxWVtfE7D5I1b2cdEtmbcRW9hGFJl1bOFOI=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
i installed xen3.0.3 on debian sarge with 2 vms (also sarge), but i
was getting very bad inter-vm network performance (like 15-20Mbps max)

i converted the network setup from bridged to routed, and applied the
following kernel parameter tweaks on dom0 and domUs too:

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/wmem_default
echo 2569600 > /proc/sys/net/core/rmem_max
echo 2569600 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling
echo "40960 873800 1747600" > /proc/sys/net/ipv4/tcp_rmem
echo "40960 163840 1310720" > /proc/sys/net/ipv4/tcp_wmem


netperf-tcp output:

65:   12288 bytes    419 times -->    606.15 Mbps in     154.67 usec
66:   12291 bytes    431 times -->    624.73 Mbps in     150.10 usec

performance is acceptable until here, xm top shows full cpu
utilization, but after this point it drops badly. cpu utilization also
drops to <10%

67:   16381 bytes    222 times -->     28.96 Mbps in    4315.32 usec
68:   16384 bytes     11 times -->    583.25 Mbps in     214.32 usec
69:   16387 bytes    233 times -->     22.51 Mbps in    5553.60 usec
70:   24573 bytes      9 times -->      5.64 Mbps in   33220.50 usec
71:   24576 bytes      3 times -->    670.06 Mbps in     279.83 usec
72:   24579 bytes    238 times -->     23.08 Mbps in    8123.88 usec
73:   32765 bytes      4 times -->    714.72 Mbps in     349.76 usec
74:   32768 bytes    142 times -->     22.70 Mbps in   11011.61 usec
75:   32771 bytes      4 times -->     47.53 Mbps in    5259.99 usec
76:   49149 bytes      9 times -->    814.98 Mbps in     460.11 usec
77:   49152 bytes    144 times -->    145.25 Mbps in    2581.81 usec
78:   49155 bytes     25 times -->    887.54 Mbps in     422.54 usec
79:   65533 bytes     78 times -->     30.32 Mbps in   16489.71 usec
80:   65536 bytes      3 times -->     18.78 Mbps in   26617.32 usec
81:   65539 bytes      3 times -->     10.72 Mbps in   46664.85 usec
82:   98301 bytes      3 times -->      7.50 Mbps in   99982.16 usec
83:   98304 bytes      3 times -->      9.38 Mbps in   79990.15 usec

dom0 vif interfaces show some dropped packets, domUs do not.

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
         inet addr:192.168.100.128  Bcast:0.0.0.0  Mask:255.255.255.255
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:1457121 errors:0 dropped:0 overruns:0 frame:0
         TX packets:1458240 errors:0 dropped:970 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:2051710970 (1.9 GiB)  TX bytes:2069975753 (1.9 GiB)

vif2.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
         inet addr:192.168.99.128  Bcast:0.0.0.0  Mask:255.255.255.255
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:1459850 errors:0 dropped:0 overruns:0 frame:0
         TX packets:1457278 errors:0 dropped:924 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:2052283308 (1.9 GiB)  TX bytes:2069732297 (1.9 GiB)

dom0 has 512, domUs have 256M ram allocated, there is now swapping
involved, host is dual-xeon 3200, 4G ram, high-mem enabled


--
Tibor Veres
 tibor.veres@xxxxxxxxx

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>