WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

[Xen-devel] XCP - FYI - An easy way to wedge (and fix) a Cloud

To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] XCP - FYI - An easy way to wedge (and fix) a Cloud
From: "dwight at supercomputer.org" <dwight@xxxxxxxxxxxxxxxxx>
Date: Tue, 8 Jun 2010 09:04:31 -0700
Delivery-date: Tue, 08 Jun 2010 09:07:01 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
This is mostly FYI. I know someone else is going to run into this.

It turns out that it's real easy to wedge an entire Cloud with
the default configurations in XCP 0.1.1. We saw this recently
with our Development Cloud.

It turns out that /var/log had filled up the root filesystem on
the master.  500M+ worth of messages in there. After I tracked 
down the problem, and freed this space up, everything started  
working again.

When this happens, various things either fail mysteriously 
(including a failure of the slaves and master to reboot),
xsconsole wedging (on the master and slaves), and OpenXenCenter 
not being able to connect, and at best messages that aren't
helpful.

I would recommend, at the very least, that compression of the
logs in logrotate.conf be turned on. I'd also strongly  recommend
that this be the default in release 0.5.

Myself, I've taken this further, by putting logrotate into the
hourly cronjob. And we're going to change our automatic 
installation scripts to put /var on a separate, large disk 
volume, not on the root filesystem.

Having /var separate from the root filesystem is generally
a wise move for servers, so that /var doesn't impact the root.

I'd also add that having grub available would've been helpful.

   -dwight-

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel