RE: [Xen-devel] disk io errors possibly caused by high network load?

["James Harper" <james.harper@xxxxxxxxxxxxxxxx>]

> 
> We rebooted the machines really quickly because it was a productive
> system, so I didn't have the time to copy the logs, and on the disks I
> see nothing about this in the logfiles, propably because the IO was
> already down.
> 
> The machines are Supermicro, Intel Xeon Quad or Dual-Quadcore, 8 to 32
> GB RAM, and some have a mdraid setup with two SATA drives with the on
> board sata controller (intel ICH), other have a dedicated 3ware / AMCC
> 9660 or similar.
> 
> The machines that crashed were on different power lines and connected to
> different switches, although on the same network segment. Also there
> were no physical interferences.
> 
> The error was reported by domU and dom0 - both saying the disk would
> give a I/O error, but no specific information.
> 
> Network card is intel e1000.

The error wasn't a timeout was it? We had a similar problem under Windows (no 
Xen involved at all) where the switch the server was plugged into was looped 
back to itself one evening. Any broadcast packet sent to the switch would just 
circulate around the switch indefinitely, until there were enough broadcast 
packets looping around that everything ground to a halt.

The server was a HP DL380, so a more than capable machine, but there were 
enough interrupts occurring due to a completely saturated network that 
everything was reporting timeouts. In this case the server didn't require a 
reboot. It sat in that state the whole night, reporting disk timeouts etc but 
the moment we rectified the cabling fault in the morning it instantly bounced 
back to life.

It could be that Linux treats timeout errors a little more severely?

Can anyone say if the layer above blkfront in the Linux kernel will report 
timeouts? Or would the errors have been coming through from Dom0?

Anyway, do you have a test environment you can reproduce the problem on? If the 
problem is as simple as a looped switch then it shouldn't be too hard to 
reproduce...

James


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel