This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


[Xen-users] strange i/o hangs on debian lenny guests

To: "xen-users@xxxxxxxxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxxxxxxxx>
Subject: [Xen-users] strange i/o hangs on debian lenny guests
From: John Madden <jmadden@xxxxxxxxxxx>
Date: Wed, 18 Nov 2009 14:48:59 -0500
Delivery-date: Wed, 18 Nov 2009 11:49:38 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird (X11/20090817)
I haven't seen this on the list before but I'm wondering if anyone else has experienced this or has any ideas on what I'm finding to be very strange (and bad) i/o behavior. I have soft hangs on four Debian lenny guests backed by LVM that I can trigger reliably with bonnie++ by running it and then firing off something in another session that isn't cached and thus has to go to disk (like running 'who' instead of 'w').

The second command will eventually come back but it takes about 90 seconds. Further testing showed that no i/o was actually going on during those 90 seconds. So as the scenario goes, start up `vmstat 1` in a shell and then `bonnie++ -r 1024` in another. Let that bonnie++ run for 15 seconds or so so you run out your buffer cache, then try to launch something else that has to read from disk.

The vmstat counters will drop to 0 on bi and bo and anything requiring disk activity will hang for exactly 90 seconds. After 90 seconds, i/o picks up where it left off and everything will appear to be ok.

I've played with i/o schedulers and various kernel versions (even running etch's kernel) to no avail. Any ideas?

This is on a CentOS 5.4 hypervisor, Debian 5 domU with 1GB memory, 2 vcpu's, single LVM phy: disk.


John Madden
Sr UNIX Systems Engineer
Ivy Tech Community College of Indiana

Xen-users mailing list

<Prev in Thread] Current Thread [Next in Thread>