This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] domU oom -> xvda1 read-only without any notice?

To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] domU oom -> xvda1 read-only without any notice?
From: Josip Rodin <joy@xxxxxxxxxxxxxx>
Date: Wed, 31 Mar 2010 11:17:11 +0200
Delivery-date: Wed, 31 Mar 2010 02:19:43 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20100331085245.GA32106@xxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20100331074950.GA18028@xxxxxxxxxxxxxxx> <C7D8CA1D.F509%keir.fraser@xxxxxxxxxxxxx> <20100331085245.GA32106@xxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/1.5.18 (2008-05-17)
On Wed, Mar 31, 2010 at 10:52:45AM +0200, Josip Rodin wrote:
> I'm going to restart it now because we need the machine in a
> not-so-useless state.

After the shutdown and create:

[    0.488419] blkfront: xvda1: barriers enabled
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[    0.670172] EXT3-fs: INFO: recovery required on readonly filesystem.
[    0.670186] EXT3-fs: write access will be enabled during recovery.
[    9.996563] kjournald starting.  Commit interval 5 seconds
[    9.996577] EXT3-fs warning (device xvda1): ext3_clear_journal_err: 
Filesystem error recorded from previous mount: IO failure
[    9.996585] EXT3-fs warning (device xvda1): ext3_clear_journal_err: Marking 
fs in need of filesystem check.
[    9.996991] EXT3-fs: recovery complete.
[    9.997318] EXT3-fs: mounted filesystem with ordered data mode.
Will now check root file system:fsck 1.41.3 (12-Oct-2008)
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a -C0 /dev/xvda1
lastovo-root contains a file system with errors, check forced.
Deleted inode 1704618 has zero dtime.  FIXED.
lastovo-root: ***** REBOOT LINUX *****
lastovo-root: 1046964/3276800 files (1.1% non-contiguous), 10147312/13107200 
fsck died with exit status 3
The file system check corrected errors on the root partition but requested that 
the system be restarted. failed!
The system will be restarted in 5 seconds. (warning).
Will now restart.

Then it booted fine.

I examined the graphs of the machine from the time of the incident, and it
seems that everything was fine until around 2:30 when a large network
operation started - Legato nsrexecd was backing it up - its remote logs say
it transferred 4 GB of data in around 11 minutes, and finished successfully.
At that point, the graphs on the machine recorded a huge spike in both
Apache and PostgreSQL connections, and then soon after the whole thing
went AWOL. If necessary I can attach the entire graph snapshot, which also
includes the approximated state of /proc/interrupts and /proc/meminfo.

     2. That which causes joy or happiness.

Xen-devel mailing list