WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] MPI benchmark performance gap between native linux anddo

To: "xuehai zhang" <hai@xxxxxxxxxxxxxxx>, <m+Ian.Pratt@xxxxxxxxxxxx>
Subject: RE: [Xen-devel] MPI benchmark performance gap between native linux anddomU
From: "Santos, Jose Renato G (Jose Renato Santos)" <joserenato.santos@xxxxxx>
Date: Tue, 5 Apr 2005 17:37:06 -0700
Cc: "Turner, Yoshio" <yoshio_turner@xxxxxx>, Aravind Menon <aravind.menon@xxxxxxx>, Xen-devel@xxxxxxxxxxxxxxxxxxx, G John Janakiraman <john@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Wed, 06 Apr 2005 00:37:59 +0000
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AcU6LhxujPVJCagoTPKbDyinXKneVQAElBcA
Thread-topic: [Xen-devel] MPI benchmark performance gap between native linux anddomU
  Xuehai,

  Thanks for posting your new results. In fact it seems that your
problem is not the same as the one we encountered.
  
  I believe your problem is due to a higher network latency in Xen. Your
formula to compute throughput uses the inverse of round trip latency (if
I understood it correctly). This probably means that your application is
sensitive to the round trip latency. Your latency mesurements show a
higher value for domainU and this is the reason for the lower
throughput.  I am not sure but it is possible that network interrupts or
event notifications in the inter-domain channel are being coalesced and
causing longer latency. Keir, do event notifications get coalesced in
the inter-domain I/O channel for networking?

  Renato
  

>> -----Original Message-----
>> From: xuehai zhang [mailto:hai@xxxxxxxxxxxxxxx] 
>> Sent: Tuesday, April 05, 2005 3:23 PM
>> To: Santos, Jose Renato G (Jose Renato Santos); 
>> m+Ian.Pratt@xxxxxxxxxxxx
>> Cc: Xen-devel@xxxxxxxxxxxxxxxxxxx; Aravind Menon; Turner, 
>> Yoshio; G John Janakiraman
>> Subject: Re: [Xen-devel] MPI benchmark performance gap 
>> between native linux anddomU
>> 
>> 
>> Hi Ian and Jose,
>> 
>> Based on your suggestions, I did two more experiments: one 
>> (with tag "domU-B" in table below) is 
>> changing the TCP advertised window of domU to -2 (the 
>> default is 2) and the other (with tag "dom0" 
>> in table below) is to repeat the experiment in dom0 (only 
>> dom0 is running). The following table 
>> contains the results from these two new experiments plus two 
>> old ones (with tags "native-linux" and 
>> "domU-A" in table below) in my previous email.
>> 
>> I have the following observation from the results:
>> 
>> 1. Decreasing the scaling of TCP window ("domU-B") doesn't 
>> buy any good to the  performance but 
>> slightly slowdown the performance (comparing with "domU-A").
>> 
>> 2. Generally, the performance of running the experiments in 
>> dom0 ("dom0" column) is very close 
>> (slightly less) to the performance on native linux 
>> ("native-linux" column). However, in certain 
>> situations, it outperforms the performance on native linux. 
>> For example, throughput values when 
>> message size is 64KB and latency values when message size is 
>> 1 , or 2, or 4, or 8 bytes.
>> 
>> 3. The performance gap between domU and dom0 is big, 
>> similarly as domU and native linux.
>> 
>> BTW, each reported data point in the following table is the 
>> average of over 10 runs of the same 
>> experiments. I forget to mention that in experiment using 
>> user domains, the 8 domU forms a private 
>> network and each domU is assigned a private network IP (for 
>> example, 192.168.254.X).
>> 
>> Xuehai
>> 
>> *********************************
>> *SendRecv Throughput(Mbytes/sec)*
>> *********************************
>> 
>> Msg Size(bytes)  native-linux        dom0          domU-A    
>>       domU-B
>>          0         0              0.00          0              0.00
>>          1         0              0.01          0              0.00
>>          2         0              0.01          0              0.00
>>          4         0              0.03          0              0.00
>>          8         0.04           0.05          0.01           0.01
>>         16         0.16           0.11          0.01           0.01
>>         32         0.34           0.21          0.02           0.02
>>         64         0.65           0.42          0.04           0.04
>>        128         1.17           0.79          0.09           0.10
>>        256         2.15           1.44          0.59           0.58
>>        512         3.4            2.39          1.23           1.22
>>       1024         5.29           3.79          2.57           2.50
>>       2048         7.68           5.30          3.5            3.44
>>       4096         10.7           8.51          4.96           5.23
>>       8192         13.35          11.06         7.07           6.00
>>      16384         14.9           13.60         3.77           4.62
>>      32768         9.85           11.13         3.68           4.34
>>      65536         5.06           9.06          3.02           3.14
>>     131072         7.91           7.61          4.94           5.04
>>     262144         7.85           7.65          5.25           5.29
>>     524288         7.93           7.77          6.11           5.40
>>    1048576         7.85           7.82          6.5            5.62
>>    2097152         8.18           7.35          5.44           5.32
>>    4194304         7.55           6.88          4.93           4.92
>> 
>> *********************************
>> *   SendRecv Latency(millisec)  *
>> *********************************
>> 
>> Msg Size(bytes)  native-linux        dom0           domU-A   
>>       domU-B                                 
>>          0         1979.6         1920.83       3010.96      
>>   3246.71                    
>>          1         1724.16        397.27        3218.88      
>>   3219.63
>>          2         1669.65        297.58        3185.3       
>>   3298.86
>>          4         1637.26        285.27        3055.67      
>>   3222.34
>>          8         406.77         282.78        2966.17      
>>   3001.24
>>         16         185.76         283.87        2777.89      
>>   2761.90
>>         32         181.06         284.75        2791.06      
>>   2798.77
>>         64         189.12         293.93        2940.82      
>>   3043.55
>>        128         210.51         310.47        2716.3       
>>   2495.83
>>        256         227.36         338.13        843.94         853.86
>>        512         287.28         408.14        796.71         805.51
>>       1024         368.72         515.59        758.19         786.67
>>       2048         508.65         737.12        1144.24      
>>   1150.66
>>       4096         730.59         917.97        1612.66      
>>   1516.35
>>       8192         1170.22        1411.94       2471.65      
>>   2650.17
>>      16384         2096.86        2297.19       8300.18      
>>   6857.13
>>      32768         6340.45        5619.56       17017.99     
>>   14392.36
>>      65536         24640.78       13787.31      41264.5      
>>   39871.19
>>     131072         31709.09       32797.52      50608.97     
>>   49533.68
>>     262144         63680.67       65174.67      94918.13     
>>   94157.30
>>     524288         125531.7       128116.73     162168.47    
>>   189307.05
>>    1048576         251566.94      252257.55     321451.02    
>>   361714.44
>>    2097152         477431.32      527432.60     707981       
>>   728504.38
>>    4194304         997768.35      1108898.61    1503987.61   
>>   1534795.56
>> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>