RE: [Xen-devel] XCP - FYI - An easy way to wedge (and fix) a Cloud

["Roger Cruz" <roger.cruz@xxxxxxxxxxxxxxxxxxx>]

With XenServer, which uses XAPI, I have encountered a similar problem where the /var/log partition gets full.  In my case, it was xensource.log that stopped being rotated.  These logs are automatically rotated by XAPI and up to 20 files of 3MB (can't recall exactly now) each are kept.  The problem occured when I changed the system time backwards (adjusting timezones), it caused the periodic (5mins I think) checks to now be a lot longer and during that time, the partition filled up because the files grew past the 3MB.  When this happens, the only way I got the system running again was to boot with a rescue CD and remove the large files.  I reported the problem to Citrix a while back so this is likely already fixed, so I'm not sure how your xensource.logs could have grown to 500+ MB

 

Roger R. Cruz 

---

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx on behalf of dwight at supercomputer.org
Sent: Wed 6/9/2010 12:58 PM
To: Daniel Stodden
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] XCP - FYI - An easy way to wedge (and fix) a Cloud

On Tuesday 08 June 2010 01:36:53 pm Daniel Stodden wrote:
> On Tue, 2010-06-08 at 12:04 -0400, dwight at supercomputer.org
wrote:
> > It turns out that /var/log had filled up the root filesystem on
> > the master.  500M+ worth of messages in there. After I tracked
> > down the problem, and freed this space up, everything started
> > working again.
>
> Which ones were the files growing too big? I recently caused
> potential trouble with blktap. But there may be more. Both xapi
> and storage management can get quite chatty, although I think this
> improved with xs5.x.
>
> Daniel

I'm going from memory here, as the main impetus was on triage, and
not proper debug/fix/testing. But if memory serves, it was
xensource.log.

It's unlikely that any recent change was the culprit, as this was
stock XCP 0.1.1.

I have to say that it's something else to reboot and debug an entire
Cloud. I've dealt with wedged/crashed systems before on
microcontrollers, small embedded devices, PC's, Servers, Mainfraimes
and Supercomputers, including Virtualized Systems. This is the first
time I've had to debug and reboot an entire Cloud before.

The main lesson for me is that the debugging interface could be
improved. This is one of the most critical aspects of any
Development environment.

Being able to get to a single user shell prompt easily from
the "boot:" prompt would go a long way here.

    -dwight-




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel