WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] iscsi vs nfs for xen VMs

On Sat, Jan 29, 2011 at 06:12:34PM +0100, Bart Coninckx wrote:
> On 01/29/11 16:30, Pasi Kärkkäinen wrote:
> > On Sat, Jan 29, 2011 at 04:27:52PM +0100, Bart Coninckx wrote:
> >   
> >> On 01/29/11 16:09, Pasi Kärkkäinen wrote:
> >>     
> >>> On Thu, Jan 27, 2011 at 09:35:38AM +0100, Adi Kriegisch wrote:
> >>>   
> >>>       
> >>>> Hi!
> >>>>
> >>>>     
> >>>>         
> >>>>>> iSCSI tipically has a quite big overhead due to the protocol, FC, SAS,
> >>>>>> native infiniband, AoE have very low overhead.
> >>>>>>
> >>>>>>         
> >>>>>>             
> >>>>> For iSCSI vs AoE, that isn't as true as you might think. TCP offload can
> >>>>> take care of a lot of the overhead. Any server class network adapter
> >>>>> these days should allow you to send 60kb packets to the network adapter
> >>>>> and it will take care of the segmentation, while AoE would be limited to
> >>>>> MTU sized packets. With AoE you need to checksum every packet yourself
> >>>>> while with iSCSI it is taken care of by the network adapter.
> >>>>>       
> >>>>>           
> >>>> What AoE actually does is sending a frame per block. Block size is 4K -- 
> >>>> so
> >>>> no need for fragmentation. The overhead is pretty low, because we're
> >>>> talking about Ethernet frames.
> >>>> Most iSCSI issues I have seen are with reordering of packages due to
> >>>> transmission across several interfaces. So what most people recommend is 
> >>>> to
> >>>> keep the number of interfaces to two. To keep performance up this means 
> >>>> you
> >>>> have to use 10G, FC or similar which is quite expensive -- especially if
> >>>> you'd like to have a HA SAN network (HSRP and stuff like that is 
> >>>> required).
> >>>>
> >>>> AoE does not suffer from those issues: Using 6 GBit interfaces is no
> >>>> problem at all, load balancing will happen automatically, as the load is
> >>>> distributed equally across all available interfaces. HA is very simple:
> >>>> just use two switches and connect one half of the interfaces to one 
> >>>> switch
> >>>> and the other half to the other switch. (It is recommended to use 
> >>>> switches
> >>>> that can do jumbo frames and flow control)
> >>>> IMHO most of the current recommendations and practises surrounding iSCSI
> >>>> are there to overcome the shortcomings of the protocol. AoE is way more
> >>>> robust and easier to handle.
> >>>>
> >>>>     
> >>>>         
> >>> iSCSI does not have problems using multiple gige interfaces.
> >>> Just setup multipathing properly.
> >>>
> >>> -- Pasi
> >>>
> >>>
> >>> _______________________________________________
> >>> Xen-users mailing list
> >>> Xen-users@xxxxxxxxxxxxxxxxxxx
> >>> http://lists.xensource.com/xen-users
> >>>   
> >>>       
> >> On this subject: am using multipathing to iSCSI too, hoping to have
> >> aggregated speed on top of path redundancy but the speed seems not to
> >> surpass the one of a single interface.
> >>
> >> Is anyone successful at doing this?
> >>
> >>     
> > You're benchmarking sequential/linear IO, using big blocksizes, right?
> >
> > Some questions:
> >     - Are you using multipath round robin path policy?
> >     - After how many IOs do you switch paths? You might need to lower the 
> > min_ios.
> >
> >
> > -- Pasi
> >
> >   
> Hi Pasi,
> 
> the benchmarking was intuitively done, with just dd and bonnie++.
> 

with dd, use "bs=1024k" or so to force big blocks.

> It is indeed rr, this is a part of my multipath.conf:
> 
> defaults {
>         udev_dir                /dev
>         polling_interval        10
>         selector                "round-robin 0"
>         path_grouping_policy    multibus
>         getuid_callout          "/lib/udev/scsi_id --whitelisted
> --device=/dev/%n"
>         prio                    const
>         path_checker            directio
>         rr_min_io               100
>         max_fds                 8192
>         rr_weight               priorities
>         failback                immediate
>         no_path_retry           5
>         user_friendly_names     no
> }
> 
> should the "100" go down a bit?
> 

Yeah, for example Equallogic enterprise iSCSI storage
manuals recommends rr_min_io of "3".

That should allow utilizing both paths.

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users