WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] Release 0.8.0 of GPL PV Drivers for Windows

To: Pasi Kärkkäinen <pasik@xxxxxx>, xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] Release 0.8.0 of GPL PV Drivers for Windows
From: jim burns <jim_burn@xxxxxxxxxxxxx>
Date: Sat, 1 Mar 2008 22:25:18 -0500
Delivery-date: Sat, 01 Mar 2008 19:26:00 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <20080301223854.GV21162@xxxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <AEC6C66638C05B468B556EA548C1A77D0131AEF2@trantor> <200803010921.24969.jim_burn@xxxxxxxxxxxxx> <20080301223854.GV21162@xxxxxxxxxxxxxxx> (sfid-20080301_175431_675497_80ABD03F)
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: KMail/1.9.9
On Saturday 01 March 2008 05:38:55 pm you wrote:
> What NIC you have?

The numbers for my physical nic were just included for comparison purposes, to 
show what the range of possible rates could be. I don't really care about 
those numbers because all my active vms are on file backed vbds on the xen 
server's (fc8) disk. All the other, more significant #s were for software 
nics, with no intermediate hardware nics. However:

> What driver and version of the driver? 
> "ethtool -i device"

On the client side (SuSE 10.3):

[815] > ethtool -i eth0
driver: e100
version: 3.5.17-k4-NAPI
firmware-version: N/A
bus-info: 0000:02:08.0
[816] > lspci|grep 02:08.0
02:08.0 Ethernet controller: Intel Corporation 82801DB PRO/100 VE (LOM) 
Ethernet Controller (rev 81)

and on the server side (fc8):
[717] > ethtool -i peth0
driver: b44
version: 1.01
firmware-version:
bus-info: 0000:03:00.0
jimb@Insp6400 03/01/08  9:58PM:~
[718] > lspci|grep 03:00.0
03:00.0 Ethernet controller: Broadcom Corporation BCM4401-B0 100Base-TX (rev 
02)

> Did you try disabling checksum offloading? "ethtool -K ethX tx off"
> Try that on dom0 and/or on domU. Maybe also "ethtool -K ethX tso off"

Why would I do that? That's not how I operate normally. Doesn't that take 
checksumming out of the hardware, and put it in software, slowing things 
down? What are the advantages here? However, my current settings are:

SuSE:
[817] > ethtool -k eth0
Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
Cannot get device tx csum settings: Operation not supported
Cannot get device scatter-gather settings: Operation not supported
Cannot get device tcp segmentation offload settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off

fc8:
[720] > ethtool -k peth0
Offload parameters for peth0:
Cannot get device rx csum settings: Operation not supported
Cannot get device tx csum settings: Operation not supported
Cannot get device scatter-gather settings: Operation not supported
Cannot get device tcp segmentation offload settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off

> Does your ethX interface have errors? Check with "ifconfig ethX".
>
> Do you have tcp retransmits? Check with "netstat -s".

Before a test, on the fc8 side:
[721] > ifconfig peth0; netstat -s
peth0     Link encap:Ethernet  HWaddr 00:15:C5:04:7D:4F
          inet6 addr: fe80::215:c5ff:fe04:7d4f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1492  Metric:1
          RX packets:10566824 errors:0 dropped:219 overruns:0 frame:0
          TX packets:12540392 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4070940466 (3.7 GiB)  TX bytes:3443253043 (3.2 GiB)
          Interrupt:22

Ip:
    16735946 total packets received
    5 with invalid headers
    1 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    16516069 incoming packets delivered
    16444246 requests sent out
    8 dropped because of missing route
    217 fragments dropped after timeout
    363681 reassemblies required
    163328 packets reassembled ok
    37025 packet reassembles failed
    11 fragments received ok
    22 fragments created
Icmp:
    125737 ICMP messages received
    232 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 7181
        echo requests: 59164
        echo replies: 59160
    59195 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 31
        echo replies: 59164
Tcp:
    6370 active connections openings
    278 passive connection openings
    5616 failed connection attempts
    19 connection resets received
    9 connections established
    15612021 segments received
    16291429 segments send out
    296 segments retransmited
    0 bad segments received.
    5834 resets sent
Udp:
    758744 packets received
    29 packets to unknown port received.
    10611 packet receive errors
    29592 packets sent
    RcvbufErrors: 10611
UdpLite:
TcpExt:
    5 packets pruned from receive queue because of socket buffer overrun
    422 TCP sockets finished time wait in fast timer
    241 packets rejects in established connections because of timestamp
    234266 delayed acks sent
    1245 delayed acks further delayed because of locked socket
    Quick ack mode was activated 720 times
    2278561 packets directly queued to recvmsg prequeue.
    69369816 packets directly received from backlog
    3065751728 packets directly received from prequeue
    5402811 packets header predicted
    2206227 packets header predicted and directly queued to user
    418685 acknowledgments not containing data received
    7621128 predicted acknowledgments
    5 times recovered from packet loss due to SACK data
    Detected reordering 3 times using FACK
    1 congestion windows fully recovered
    3 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 3
    37 congestion windows recovered after partial ack
    0 TCP data loss events
    3 fast retransmits
    6 forward retransmits
    1 retransmits in slow start
    113 other TCP timeouts
    1 sack retransmits failed
    307 times receiver scheduled too late for direct processing
    113 packets collapsed in receive queue due to low socket buffer
    730 DSACKs sent for old packets
    37 DSACKs received
    10 connections reset due to unexpected data
    160 connections reset due to early user close
    19 connections aborted due to timeout

and after:
[722] > iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.100 port 5001 connected with 192.168.1.101 port 26433
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.1 sec   539 MBytes  75.3 Mbits/sec
jimb@Insp6400 03/01/08 10:08PM:~
[723] > ifconfig peth0; netstat -s
peth0     Link encap:Ethernet  HWaddr 00:15:C5:04:7D:4F
          inet6 addr: fe80::215:c5ff:fe04:7d4f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1492  Metric:1
          RX packets:10962754 errors:0 dropped:237 overruns:0 frame:0
          TX packets:12715673 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:369386256 (352.2 MiB)  TX bytes:3459549868 (3.2 GiB)
          Interrupt:22

Ip:
    17132793 total packets received
    5 with invalid headers
    1 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    16912908 incoming packets delivered
    16620447 requests sent out
    8 dropped because of missing route
    217 fragments dropped after timeout
    363681 reassemblies required
    163328 packets reassembled ok
    37025 packet reassembles failed
    11 fragments received ok
    22 fragments created
Icmp:
    125741 ICMP messages received
    232 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 7185
        echo requests: 59164
        echo replies: 59160
    59195 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 31
        echo replies: 59164
Tcp:
    6382 active connections openings
    279 passive connection openings
    5628 failed connection attempts
    19 connection resets received
    9 connections established
    16008836 segments received
    16467614 segments send out
    296 segments retransmited
    0 bad segments received.
    5846 resets sent
Udp:
    758760 packets received
    29 packets to unknown port received.
    10611 packet receive errors
    29606 packets sent
    RcvbufErrors: 10611
UdpLite:
TcpExt:
    5 packets pruned from receive queue because of socket buffer overrun
    422 TCP sockets finished time wait in fast timer
    286 packets rejects in established connections because of timestamp
    234432 delayed acks sent
    1270 delayed acks further delayed because of locked socket
    Quick ack mode was activated 765 times
    2539167 packets directly queued to recvmsg prequeue.
    79852248 packets directly received from backlog
    3460281584 packets directly received from prequeue
    5505468 packets header predicted
    2487426 packets header predicted and directly queued to user
    418832 acknowledgments not containing data received
    7624029 predicted acknowledgments
    5 times recovered from packet loss due to SACK data
    Detected reordering 3 times using FACK
    1 congestion windows fully recovered
    3 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 3
    37 congestion windows recovered after partial ack
    0 TCP data loss events
    3 fast retransmits
    6 forward retransmits
    1 retransmits in slow start
    113 other TCP timeouts
    1 sack retransmits failed
    354 times receiver scheduled too late for direct processing
    113 packets collapsed in receive queue due to low socket buffer
    775 DSACKs sent for old packets
    37 DSACKs received
    10 connections reset due to unexpected data
    160 connections reset due to early user close
    19 connections aborted due to timeout

Which shows a modest increase in drops in ifconfig, and no real significant 
differences in netstat, except for the TcpExt: section.

> I think it might be a good idea to "force" good/big tcp window size to get
> comparable results..

I did a couple of 32k window size tests, with not much significant difference.

> Some things to check:
>
> - txqueuelen of ethX device. I guess 1000 is the default nowadays.. try
> with bigger values too. This applies to dom0 and to linux domU.
>
> - txqueuelen of vifX.Y devices on dom0. Default has been really small, so
>   make sure to configure that bigger too.. This applies to both linux
>   and windows vm's.

I guess this is done on ifconfig, since it appears in an ifconfig output. Is 
it done in ipconfig for windows?

> - Check sysctl net.core.netdev_max_backlog setting.. it should be at least
>   1000, possibly even more.. this applies to dom0 and linux domU.

Where is this set, and what do I have to restart to make it 
effective? /etc/sysctl.conf?

In general, are there any downsides in changing these values?

Thanx for your interest.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users