RE: [Xen-devel] MPI benchmark performance gap between native linux anddomU

["Santos, Jose Renato G (Jose Renato Santos)" <joserenato.santos@xxxxxx>]

  Xuehai,

  Thanks for posting your new results. In fact it seems that your
problem is not the same as the one we encountered.
  
  I believe your problem is due to a higher network latency in Xen. Your
formula to compute throughput uses the inverse of round trip latency (if
I understood it correctly). This probably means that your application is
sensitive to the round trip latency. Your latency mesurements show a
higher value for domainU and this is the reason for the lower
throughput.  I am not sure but it is possible that network interrupts or
event notifications in the inter-domain channel are being coalesced and
causing longer latency. Keir, do event notifications get coalesced in
the inter-domain I/O channel for networking?

  Renato
  

>> -----Original Message-----
>> From: xuehai zhang [mailto:hai@xxxxxxxxxxxxxxx] 
>> Sent: Tuesday, April 05, 2005 3:23 PM
>> To: Santos, Jose Renato G (Jose Renato Santos); 
>> m+Ian.Pratt@xxxxxxxxxxxx
>> Cc: Xen-devel@xxxxxxxxxxxxxxxxxxx; Aravind Menon; Turner, 
>> Yoshio; G John Janakiraman
>> Subject: Re: [Xen-devel] MPI benchmark performance gap 
>> between native linux anddomU
>> 
>> 
>> Hi Ian and Jose,
>> 
>> Based on your suggestions, I did two more experiments: one 
>> (with tag "domU-B" in table below) is 
>> changing the TCP advertised window of domU to -2 (the 
>> default is 2) and the other (with tag "dom0" 
>> in table below) is to repeat the experiment in dom0 (only 
>> dom0 is running). The following table 
>> contains the results from these two new experiments plus two 
>> old ones (with tags "native-linux" and 
>> "domU-A" in table below) in my previous email.
>> 
>> I have the following observation from the results:
>> 
>> 1. Decreasing the scaling of TCP window ("domU-B") doesn't 
>> buy any good to the  performance but 
>> slightly slowdown the performance (comparing with "domU-A").
>> 
>> 2. Generally, the performance of running the experiments in 
>> dom0 ("dom0" column) is very close 
>> (slightly less) to the performance on native linux 
>> ("native-linux" column). However, in certain 
>> situations, it outperforms the performance on native linux. 
>> For example, throughput values when 
>> message size is 64KB and latency values when message size is 
>> 1 , or 2, or 4, or 8 bytes.
>> 
>> 3. The performance gap between domU and dom0 is big, 
>> similarly as domU and native linux.
>> 
>> BTW, each reported data point in the following table is the 
>> average of over 10 runs of the same 
>> experiments. I forget to mention that in experiment using 
>> user domains, the 8 domU forms a private 
>> network and each domU is assigned a private network IP (for 
>> example, 192.168.254.X).
>> 
>> Xuehai
>> 
>> *********************************
>> *SendRecv Throughput(Mbytes/sec)*
>> *********************************
>> 
>> Msg Size(bytes)  native-linux        dom0          domU-A    
>>       domU-B
>>          0         0              0.00          0              0.00
>>          1         0              0.01          0              0.00
>>          2         0              0.01          0              0.00
>>          4         0              0.03          0              0.00
>>          8         0.04           0.05          0.01           0.01
>>         16         0.16           0.11          0.01           0.01
>>         32         0.34           0.21          0.02           0.02
>>         64         0.65           0.42          0.04           0.04
>>        128         1.17           0.79          0.09           0.10
>>        256         2.15           1.44          0.59           0.58
>>        512         3.4            2.39          1.23           1.22
>>       1024         5.29           3.79          2.57           2.50
>>       2048         7.68           5.30          3.5            3.44
>>       4096         10.7           8.51          4.96           5.23
>>       8192         13.35          11.06         7.07           6.00
>>      16384         14.9           13.60         3.77           4.62
>>      32768         9.85           11.13         3.68           4.34
>>      65536         5.06           9.06          3.02           3.14
>>     131072         7.91           7.61          4.94           5.04
>>     262144         7.85           7.65          5.25           5.29
>>     524288         7.93           7.77          6.11           5.40
>>    1048576         7.85           7.82          6.5            5.62
>>    2097152         8.18           7.35          5.44           5.32
>>    4194304         7.55           6.88          4.93           4.92
>> 
>> *********************************
>> *   SendRecv Latency(millisec)  *
>> *********************************
>> 
>> Msg Size(bytes)  native-linux        dom0           domU-A   
>>       domU-B                                 
>>          0         1979.6         1920.83       3010.96      
>>   3246.71                    
>>          1         1724.16        397.27        3218.88      
>>   3219.63
>>          2         1669.65        297.58        3185.3       
>>   3298.86
>>          4         1637.26        285.27        3055.67      
>>   3222.34
>>          8         406.77         282.78        2966.17      
>>   3001.24
>>         16         185.76         283.87        2777.89      
>>   2761.90
>>         32         181.06         284.75        2791.06      
>>   2798.77
>>         64         189.12         293.93        2940.82      
>>   3043.55
>>        128         210.51         310.47        2716.3       
>>   2495.83
>>        256         227.36         338.13        843.94         853.86
>>        512         287.28         408.14        796.71         805.51
>>       1024         368.72         515.59        758.19         786.67
>>       2048         508.65         737.12        1144.24      
>>   1150.66
>>       4096         730.59         917.97        1612.66      
>>   1516.35
>>       8192         1170.22        1411.94       2471.65      
>>   2650.17
>>      16384         2096.86        2297.19       8300.18      
>>   6857.13
>>      32768         6340.45        5619.56       17017.99     
>>   14392.36
>>      65536         24640.78       13787.31      41264.5      
>>   39871.19
>>     131072         31709.09       32797.52      50608.97     
>>   49533.68
>>     262144         63680.67       65174.67      94918.13     
>>   94157.30
>>     524288         125531.7       128116.73     162168.47    
>>   189307.05
>>    1048576         251566.94      252257.55     321451.02    
>>   361714.44
>>    2097152         477431.32      527432.60     707981       
>>   728504.38
>>    4194304         997768.35      1108898.61    1503987.61   
>>   1534795.56
>> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel