[Xen-users] XCP NFS SR issues

[Ciaran Kendellen <ciaran@xxxxxxxxxxxxxxx>]

Hello all,

We have 2 x compute nodes running XCP and 2 x storage nodes with 
GlusterFS exposed via NFS to the compute nodes.
Roughly every week we're experiencing failure of VMs due to tapdisk 
errors like the below :
Aug 31 17:00:51 xen02 tapdisk[15678]: ERROR: errno -116 at vhd_complete: 
/var/run/sr-mount/0554f47c-1a12-9bd1-ea9b-d2b68984f0ed/75af4999-3447-4fa4-bdf0-b89580807a7c.vhd: 
op: 1, lsec: 3464605, secs: 8, nbytes: 4096, blk: 845, blk_offset: 2725095
Aug 31 17:00:51 xen02 tapdisk[15678]: ERROR: errno -116 at 
__tapdisk_vbd_complete_td_request: req 1: read 0x0008 secs to 0x0034dd9d
The standard fix is always to reboot the XCP host, start the VMs and 
repair filesystems, and so far, I'm happy/lucky to say they have 
repaired successfully and come back up. However, I believe it is only a 
matter of time before I experience an unrecoverable failure....
We have introduced NIC bonding on the storage nodes (All NICs are Gb), 
we have only ~8 VMs  running on the compute nodes currently, but was 
reluctant to add NIC bonding to them as it means reconfiguring the VMs' 
VIFs etc.
Can anybody advise what fine tuning of NFS may have worked for you? Are 
there any "Usual suspects" I should be looking at here? We're currently 
using vanilla NFS out of the box...
The main reason we're using Gluster is so we can expose volumes greater 
than the size of a single physical disc and have built in redundancy. We 
achieved good direct I/O results on the storage nodes, hence why I'm 
suspecting NFS...
Any feedback would be very much appreciated - the powers that be *may* 
use this as an excuse to test Hyper-V as a solution which I want to 
avoid at all costs ;)
Many thanks in advance for any trouble taken.

Kind regards,

Ciaran Kendellen.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users