On 01/29/11 18:15, Pasi Kärkkäinen wrote:
> On Sat, Jan 29, 2011 at 06:12:34PM +0100, Bart Coninckx wrote:
>
>> On 01/29/11 16:30, Pasi Kärkkäinen wrote:
>>
>>> On Sat, Jan 29, 2011 at 04:27:52PM +0100, Bart Coninckx wrote:
>>>
>>>
>>>> On 01/29/11 16:09, Pasi Kärkkäinen wrote:
>>>>
>>>>
>>>>> On Thu, Jan 27, 2011 at 09:35:38AM +0100, Adi Kriegisch wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> iSCSI tipically has a quite big overhead due to the protocol, FC, SAS,
>>>>>>>> native infiniband, AoE have very low overhead.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> For iSCSI vs AoE, that isn't as true as you might think. TCP offload can
>>>>>>> take care of a lot of the overhead. Any server class network adapter
>>>>>>> these days should allow you to send 60kb packets to the network adapter
>>>>>>> and it will take care of the segmentation, while AoE would be limited to
>>>>>>> MTU sized packets. With AoE you need to checksum every packet yourself
>>>>>>> while with iSCSI it is taken care of by the network adapter.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> What AoE actually does is sending a frame per block. Block size is 4K --
>>>>>> so
>>>>>> no need for fragmentation. The overhead is pretty low, because we're
>>>>>> talking about Ethernet frames.
>>>>>> Most iSCSI issues I have seen are with reordering of packages due to
>>>>>> transmission across several interfaces. So what most people recommend is
>>>>>> to
>>>>>> keep the number of interfaces to two. To keep performance up this means
>>>>>> you
>>>>>> have to use 10G, FC or similar which is quite expensive -- especially if
>>>>>> you'd like to have a HA SAN network (HSRP and stuff like that is
>>>>>> required).
>>>>>>
>>>>>> AoE does not suffer from those issues: Using 6 GBit interfaces is no
>>>>>> problem at all, load balancing will happen automatically, as the load is
>>>>>> distributed equally across all available interfaces. HA is very simple:
>>>>>> just use two switches and connect one half of the interfaces to one
>>>>>> switch
>>>>>> and the other half to the other switch. (It is recommended to use
>>>>>> switches
>>>>>> that can do jumbo frames and flow control)
>>>>>> IMHO most of the current recommendations and practises surrounding iSCSI
>>>>>> are there to overcome the shortcomings of the protocol. AoE is way more
>>>>>> robust and easier to handle.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> iSCSI does not have problems using multiple gige interfaces.
>>>>> Just setup multipathing properly.
>>>>>
>>>>> -- Pasi
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Xen-users mailing list
>>>>> Xen-users@xxxxxxxxxxxxxxxxxxx
>>>>> http://lists.xensource.com/xen-users
>>>>>
>>>>>
>>>>>
>>>> On this subject: am using multipathing to iSCSI too, hoping to have
>>>> aggregated speed on top of path redundancy but the speed seems not to
>>>> surpass the one of a single interface.
>>>>
>>>> Is anyone successful at doing this?
>>>>
>>>>
>>>>
>>> You're benchmarking sequential/linear IO, using big blocksizes, right?
>>>
>>> Some questions:
>>> - Are you using multipath round robin path policy?
>>> - After how many IOs do you switch paths? You might need to lower the
>>> min_ios.
>>>
>>>
>>> -- Pasi
>>>
>>>
>>>
>> Hi Pasi,
>>
>> the benchmarking was intuitively done, with just dd and bonnie++.
>>
>>
> with dd, use "bs=1024k" or so to force big blocks.
>
>
>> It is indeed rr, this is a part of my multipath.conf:
>>
>> defaults {
>> udev_dir /dev
>> polling_interval 10
>> selector "round-robin 0"
>> path_grouping_policy multibus
>> getuid_callout "/lib/udev/scsi_id --whitelisted
>> --device=/dev/%n"
>> prio const
>> path_checker directio
>> rr_min_io 100
>> max_fds 8192
>> rr_weight priorities
>> failback immediate
>> no_path_retry 5
>> user_friendly_names no
>> }
>>
>> should the "100" go down a bit?
>>
>>
> Yeah, for example Equallogic enterprise iSCSI storage
> manuals recommends rr_min_io of "3".
>
> That should allow utilizing both paths.
>
> -- Pasi
>
>
Excellent, will try this and start measuring again ...
cheers,
B.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
Home