WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 		  
Home Products Support Community News
 
   
 

xen-users

[Top] [All Lists]
<prev [Date] next>
[Advanced]
 <prev [Thread] next>

Re: [Xen-users] losing network packets on heavy load, xen 3.0.3

from [Daniel Nielsen] [Permanent Link][Original]
To: Luis Vinay <luisvinay@xxxxxxxxx>, <xen-users@xxxxxxxxxxxxxxxxxxx>
Subject: Re: [Xen-users] losing network packets on heavy load, xen 3.0.3
From: Daniel Nielsen <djn@xxxxxxxxxx>
Date: Mon, 27 Nov 2006 10:29:21 +0100
Delivery-date: Mon, 27 Nov 2006 01:29:36 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <93c69af10611240809y179d148l36b5fbdba585d652@xxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AccSBoTFwyurNn35EduDggAUUWMFRg==
Thread-topic: [Xen-users] losing network packets on heavy load, xen 3.0.3
User-agent: Microsoft-Entourage/11.2.5.060620 
Hi Luis.

It sounds very similar to our issues with Xen and bridged networking. In our
case it is also a problem when NFS servers have high disk I/O (I've posted
to this list about it, but no response yet).

Just to verify that it indeed is a similar issue, then if you monitor the
number of dropped packets on the vif in dom0, does the number of dropped
packets increase when running this test? Just use 'ifconfig vifX.X' to see
the status of a vif.

I really like the way Xen makes us think about server management, but if
these network issues isn't resolved one way or the other, then we might need
to rethink our virtualization strategy.

Are there other options for configuring networking in Xen than using the
bridges? /me goes googling.

/Daniel
Portalen 


> From: Luis Vinay <luisvinay@xxxxxxxxx>
> Date: Fri, 24 Nov 2006 13:09:08 -0300
> To: <xen-users@xxxxxxxxxxxxxxxxxxx>
> Subject: [Xen-users] losing network packets on heavy load, xen 3.0.3
> 
> I have made this test many times with similar results all times:
> 
> Two virtual machines, both with Debian 3.1 r3, the firs is running "iperf -s
> -d" and the second runs in a infite loop "iperf -c [ip addr of machine 1]
> -t 60 -N -P 10 -d" whit this creates 10(+10) parallel bidirectional
> connections for 60 seconds on the server side I get this:
> 
> vm-debian-B:~#  iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> [  4] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39537
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> write2 failed: Broken pipe
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  6] local 0.0.0.0 port 46381 connected with 10.0.41.129 port 5001
> [  6]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [  9] local 0.0.0.0 port 46383 connected with 10.0.41.129 port 5001
> [  9]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 13] local 10.0.41.131 port 46387 connected with 10.0.41.129 port 5001
> [ 12] local 10.0.41.131 port 46386 connected with 10.0.41.129 port 5001
> [ 14] local 10.0.41.131 port 46388 connected with 10.0.41.129 port 5001
> [ 10] local 10.0.41.131 port 46384 connected with 10.0.41.129 port 5001
> [ 16] local 10.0.41.131 port 46390 connected with 10.0.41.129 port 5001
> [ 11] local 10.0.41.131 port 46385 connected with 10.0.41.129 port 5001
> [  8] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39538
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  7] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39539
> connect failed: Connection refused
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> [ 32] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39540
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 35] local 0.0.0.0 port 46411 connected with 10.0.41.129 port 5001
> [ 35]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 39] local 0.0.0.0 port 46415 connected with 10.0.41.129 port 5001
> [ 39]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> [ 41] local 0.0.0.0 port 46417 connected with 10.0.41.129 port 5001
> write2 failed: Broken pipe
> [ 41]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 40] local 0.0.0.0 port 46416 connected with 10.0.41.129 port 5001
> [ 40]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> [ 44] local 0.0.0.0 port 46419 connected with 10.0.41.129 port 5001
> [ 44]  0.0- 0.1 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 43] local 0.0.0.0 port 46418 connected with 10.0.41.129 port 5001
> [ 43]  0.0- 0.2 sec  0.00 Bytes  0.00 bits/sec
> [ 34] local 0.0.0.0 port 46410 connected with 10.0.41.129 port 5001
> [ 34]  0.0- 0.2 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.2 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> [ 42] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39541
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> connect failed: Connection refused
> write2 failed: Broken pipe
> [ 30] local 0.0.0.0 port 46407 connected with 10.0.41.129 port 5001
> [ 30]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 31] local 0.0.0.0 port 46408 connected with 10.0.41.129 port 5001
> [ 31]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 22] local 10.0.41.131 port 46399 connected with 10.0.41.129 port 5001
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Connection reset by peer
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 23] local 0.0.0.0 port 46400 connected with 10.0.41.129 port 5001
> [ 23]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 27] local 0.0.0.0 port 46404 connected with 10.0.41.129 port 5001
> [ 27]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> 
> write2 failed: Broken pipe
> write2 failed: Connection reset by peer
> write2 failed: Connection reset by peer
> write2 failed: Connection reset by peer
> write2 failed: Broken pipe
> [ 28] local 0.0.0.0 port 46405 connected with 10.0.41.129 port 5001
> [ 28]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 19] local 10.0.41.131 port 46396 connected with 10.0.41.129 port 5001
> [ 51] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39542
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 47] local 0.0.0.0 port 46422 connected with 10.0.41.129 port 5001
> [ 47]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 48] local 0.0.0.0 port 46423 connected with 10.0.41.129 port 5001
> [ 48]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 37] local 0.0.0.0 port 46428 connected with 10.0.41.129 port 5001
> [ 37]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 35] local 0.0.0.0 port 46426 connected with 10.0.41.129 port 5001
> [ 35]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ 18] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39543
> [ 33] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39544
> [ 50] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39546
> [ 61] local 10.0.41.131 port 5001 connected with 10.0.41.129 port 39545
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Connection reset by peer
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> [ 22]  0.0- 0.1 sec  11.3 KBytes  1.40 Mbits/sec
> [SUM]  0.0- 0.1 sec  11.3 KBytes  1.40 Mbits/sec
> write2 failed: Connection reset by peer
> [ 19]  0.0- 1.2 sec  11.3 KBytes  78.0 Kbits/sec
> connect failed: Connection refused
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 54] local 0.0.0.0 port 46459 connected with 10.0.41.129 port 5001
> [ 54]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 53] local 0.0.0.0 port 46458 connected with 10.0.41.129 port 5001
> [ 53]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 65] local 0.0.0.0 port 46469 connected with 10.0.41.129 port 5001
> [ 65]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 64] local 0.0.0.0 port 46468 connected with 10.0.41.129 port 5001
> [ 64]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 75] local 0.0.0.0 port 46479 connected with 10.0.41.129 port 5001
> [ 75]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 67] local 0.0.0.0 port 46471 connected with 10.0.41.129 port 5001
> [ 67]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 68] local 0.0.0.0 port 46472 connected with 10.0.41.129 port 5001
> [ 68]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 74] local 0.0.0.0 port 46478 connected with 10.0.41.129 port 5001
> [ 74]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> ------------------------------------------------------------
> Client connecting to 10.0.41.129, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> connect failed: Connection refused
> connect failed: Connection refused
> connect failed: Connection refused
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 17] local 0.0.0.0 port 46438 connected with 10.0.41.129 port 5001
> [ 17]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [SUM]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 40] local 0.0.0.0 port 46432 connected with 10.0.41.129 port 5001
> [ 40]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> [ ... ]
> [ 23] local 0.0.0.0 port 46437 connected with 10.0.41.129 port 5001
> [ 23]  0.0- 0.0 sec  0.00 Bytes  0.00 bits/sec
> write2 failed: Broken pipe
> write2 failed: Broken pipe
> [ 20] local 0.0.0.0 port 46439 connected with 10.0.41.129 port 5001
> [ ... ]
> [  5]  0.0-57.9 sec  11.1 MBytes  1.61 Mbits/sec
> 
> the client side shows this:
> Debian31r1-vm1:~# iperf -c 10.0.41.131 -t 60 -N -P 10 -d
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> ------------------------------------------------------------
> Client connecting to 10.0.41.131, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  4] local 10.0.41.129 port 39538 connected with 10.0.41.131 port 5001
> [  6] local 10.0.41.129 port 39540 connected with 10.0.41.131 port 5001
> [  8] local 10.0.41.129 port 39542 connected with 10.0.41.131 port 5001
> [  7] local 10.0.41.129 port 39541 connected with 10.0.41.131 port 5001
> [  9] local 10.0.41.129 port 39543 connected with 10.0.41.131 port 5001
> [ 10] local 10.0.41.129 port 39544 connected with 10.0.41.131 port 5001
> [  3] local 10.0.41.129 port 39537 connected with 10.0.41.131 port 5001
> [ 13] local 10.0.41.129 port 39546 connected with 10.0.41.131 port 5001
> [  5] local 10.0.41.129 port 39539 connected with 10.0.41.131 port 5001
> [ 14] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46384
> [ 15] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46385
> [ 16] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46386
> [ 17] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46387
> [ 18] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46388
> [ 19] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46389
> [ 11] local 10.0.41.129 port 39545 connected with 10.0.41.131 port 5001
> [ 20] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46391
> [ 22] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46392
> [ 23] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46393
> [ 21] local 10.0.41.129 port 5001 connected with 10.0.41.131 port 46390
> [  4]  0.0-60.0 sec  11.6 MBytes  1.62 Mbits/sec
> [  7]  0.0-60.0 sec  11.8 MBytes  1.65 Mbits/sec
> [  6]  0.0-60.0 sec  12.2 MBytes  1.71 Mbits/sec
> [ 14]  0.0-60.0 sec  18.1 MBytes  2.53 Mbits/sec
> [ 16]  0.0-60.0 sec  14.1 MBytes  1.97 Mbits/sec
> [ 18]  0.0-60.0 sec  14.3 MBytes  2.00 Mbits/sec
> [ 19]  0.0-60.0 sec  12.5 MBytes  1.75 Mbits/sec
> [ 21]  0.0-60.0 sec  13.2 MBytes  1.85 Mbits/sec
> [ 15]  0.0-60.0 sec  14.4 MBytes  2.01 Mbits/sec
> [ 13]  0.0-60.3 sec  10.1 MBytes  1.40 Mbits/sec
> [  9]  0.0-60.3 sec  8.66 MBytes  1.20 Mbits/sec
> [ 17]  0.0-60.2 sec  12.2 MBytes  1.71 Mbits/sec
> [  8]  0.0-60.5 sec  4.96 MBytes    688 Kbits/sec
> [ 11]  0.0-60.5 sec  11.0 MBytes  1.53 Mbits/sec
> [ 23]  0.0-60.6 sec  17.6 MBytes  2.44 Mbits/sec
> [  5]  0.0-60.7 sec  12.8 MBytes  1.77 Mbits/sec
> [ 22]  0.0-60.6 sec  14.0 MBytes  1.93 Mbits/sec
> [  3]  0.0-60.7 sec  12.8 MBytes  1.77 Mbits/sec
> [ 10]  0.0-61.1 sec  9.49 MBytes  1.30 Mbits/sec
> [SUM]  0.0-61.1 sec    105 MBytes  14.5 Mbits/sec
> [ 20]  0.0-61.7 sec  11.1 MBytes  1.52 Mbits/sec
> [SUM]  0.0-61.7 sec    142 MBytes  19.3 Mbits/sec
> 
> As we can see there is a lot of connection errors and the speed measurement
> reveals a very poor performance
> 
> Test platform detail:
> 
> Dom0
>    Dell PE1950, 4 Xeon @1.6G, 8Gig memory
>    RedHat Enterprise Server 4 update 4
>    Xen 3.0.3, kernel 2.6.16.29
> 
> Virtual Machines
>    1 CPU, 300Mb memory
>    Debian 3.1r3
>    Kernel 2.6.8-3-386
> 
> Both machines are on the same Dom0 and the use bridged networking
> 
> Any ideas or comments?
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
[More with this subject...]
<Prev in Thread] Current Thread [Next in Thread>
Previous by Date:  Re: [Xen-users] Why the unmodified guest os can run on xen while hardware supports VT?, Ulrich Windl
Next by Date:  Re: [Xen-users] Time/clock issues with Xen 3.0.3?, marek cervenka
Previous by Thread:  [Xen-users] losing network packets on heavy load, xen 3.0.3, Luis Vinay
Next by Thread:  [Xen-users] Xen 3.0.3 serial domU amd64, Fabrice Coudert
Indexes:  [Date] [Thread] [Top] [All Lists]
 
   

Copyright , Citrix Systems Inc. All rights reserved. Legal and Privacy