WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] Re: domU corrupt after server crash, help needed trying to r

To: xen-users <xen-users@xxxxxxxxxxxxxxxxxxx>
Subject: [Xen-users] Re: domU corrupt after server crash, help needed trying to recover domU
From: Rudi Ahlers <rudiahlers@xxxxxxxxx>
Date: Fri, 8 May 2009 15:43:00 +0200
Delivery-date: Fri, 08 May 2009 06:43:59 -0700
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=9pC+i6krt4/+FlfZ/pWtVIgXtCOMwQXrc57EqMcG8GI=; b=b79lU5DeXjbJ0+B/8tq1zbuL9mvuti98yUzh01JNZ8GQBzxkm2zX/kbr7MCtX4Gt9T 8znSVBtxwjM1hRCKrmYomeWtY1gX67ov9cqAm+NQ2utZIfjL7PO9T2V+NKSkAVH5Sx3d WgoS6ZBvUSeshzES+XD+2/QwZpuymwLvjeQDE=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=leoRYDkcZdGtkVPDDygDR+dmSVRoYcoTGIVRV0QR1xNGW3Ag5yRIcxiM95c43znQ1E UwoY24oJ9MJvHGFUPULJ5KS/IiUoBQ2IKLqzbbUPB0k+jy6C0qQmMQY5vHJHgrRt6dhd KcZIWwYjCgfj5bJ/q9IjA0i5ojJ+CAFqETNMU=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <695200da0905080529i1891ecd0jd402f8a42a4498e4@xxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <695200da0905080529i1891ecd0jd402f8a42a4498e4@xxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx


On Fri, May 8, 2009 at 2:29 PM, Rudi Ahlers <rudiahlers@xxxxxxxxx> wrote:
Hi all,

One of our Dell servers has failed badly, and one of the domU's has been corrupted in the process. It boots up to a point and then gives me a kernel panic:

Loading dm-zero.ko module
Loading dm-snapshot.ko module
Scanning and configuring dmraid supported devices
Scanning logical volumes
  Reading all physical volumes.  This may take a while...
  No volume groups found
Activating logical volumes
  Volume group "VolGroup00" not found
Creating root device.
Mounting root filesystem.
mount: could not find filesystem '/dev/root'
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!


It shows up as a Zombie:

[root@xen ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     1439     1 r-----    329.0
Zombie-hfserver2                          15     1024     1 ----c-      0.5
hfdns02                                   10      519     2 r-----   1552.8


I can't mount either:

root@xen ~]# mount /dev/data/hf
hfdns02    hfserver2
[root@xen ~]# mount /dev/data/hfserver2 /mnt/cpanel/
mount: you must specify the filesystem type
[root@xen ~]# mount -o loop /dev/data/hfserver2 /mnt/cpanel/
mount: you must specify the filesystem type

Here's the output of the LVM partitions:

[root@xen ~]# lvscan
  ACTIVE            '/dev/data/cpanel002' [100.00 GB] inherit
  ACTIVE            '/dev/data/windows2003_web' [30.00 GB] inherit
  ACTIVE            '/dev/data/storage' [50.00 GB] inherit
  ACTIVE   Original '/dev/data/hfserver2' [30.00 GB] inherit
  ACTIVE            '/dev/data/hfdns02' [30.00 GB] inherit
  ACTIVE            '/dev/data/pluto' [30.00 GB] inherit
  ACTIVE   Snapshot '/dev/data/pluto_s' [30.00 GB] inherit
  ACTIVE            '/dev/system/root' [39.06 GB] inherit
  ACTIVE            '/dev/system/swap' [9.75 GB] inherit
[root@xen ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "data" using metadata type lvm2
  Found volume group "system" using metadata type lvm2
[root@xen ~]#           


Does anyone know how to fix a LVM like this?                                                          

--



 Here's what I've done so far:

[root@xen ~]# losetup /dev/loop4  /dev/data/hfserver2

# This mounts the LVM partition as imaginary physical to /dev/loop4
# data is the virtual group (VG) name

[root@xen ~]#  kpartx -va /dev/loop4
add map loop4p1 : 0 208782 linear /dev/loop4 63
add map loop4p2 : 0 62701695 linear /dev/loop4 208845

# This creates a device map with partitions in /dev/data/hfserver in /dev/mapper 

[root@xen ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup00" using metadata type lvm2
  Found volume group "data" using metadata type lvm2
  Found volume group "system" using metadata type lvm2


[root@xen ~]# lvscan
  inactive          '/dev/VolGroup00/LogVol00' [27.94 GB] inherit
  inactive          '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
  ACTIVE            '/dev/data/cpanel002' [100.00 GB] inherit
  ACTIVE            '/dev/data/windows2003_web' [30.00 GB] inherit
  ACTIVE            '/dev/data/storage' [50.00 GB] inherit
  ACTIVE   Original '/dev/data/hfserver2' [30.00 GB] inherit
  ACTIVE            '/dev/data/hfdns02' [30.00 GB] inherit
  ACTIVE            '/dev/data/pluto' [30.00 GB] inherit
  ACTIVE   Snapshot '/dev/data/pluto_s' [30.00 GB] inherit
  ACTIVE            '/dev/system/root' [39.06 GB] inherit
  ACTIVE            '/dev/system/swap' [9.75 GB] inherit

[root@xen ~]# lvchange -ay VolGroup00
[root@xen ~]# lvscan
  ACTIVE            '/dev/VolGroup00/LogVol00' [27.94 GB] inherit
  ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
  ACTIVE            '/dev/data/cpanel002' [100.00 GB] inherit
  ACTIVE            '/dev/data/windows2003_web' [30.00 GB] inherit
  ACTIVE            '/dev/data/storage' [50.00 GB] inherit
  ACTIVE   Original '/dev/data/hfserver2' [30.00 GB] inherit
  ACTIVE            '/dev/data/hfdns02' [30.00 GB] inherit
  ACTIVE            '/dev/data/pluto' [30.00 GB] inherit
  ACTIVE   Snapshot '/dev/data/pluto_s' [30.00 GB] inherit
  ACTIVE            '/dev/system/root' [39.06 GB] inherit
  ACTIVE            '/dev/system/swap' [9.75 GB] inherit

[root@xen ~]# e2fsck /dev/VolGroup00/LogVol00
e2fsck 1.39 (29-May-2006)
/dev/VolGroup00/LogVol00: clean, 631982/7325696 files, 4512772/7323648 blocks


At first it found a whole lot of damages inodes which I repaired.

Then, I reversed the process:

[root@xen ~]# lvchange -an VolGroup00
[root@xen ~]# lvscan
  inactive          '/dev/VolGroup00/LogVol00' [27.94 GB] inherit
  inactive          '/dev/VolGroup00/LogVol01' [1.94 GB] inherit
  ACTIVE            '/dev/data/cpanel002' [100.00 GB] inherit
  ACTIVE            '/dev/data/windows2003_web' [30.00 GB] inherit
  ACTIVE            '/dev/data/storage' [50.00 GB] inherit
  ACTIVE   Original '/dev/data/hfserver2' [30.00 GB] inherit
  ACTIVE            '/dev/data/hfdns02' [30.00 GB] inherit
  ACTIVE            '/dev/data/pluto' [30.00 GB] inherit
  ACTIVE   Snapshot '/dev/data/pluto_s' [30.00 GB] inherit
  ACTIVE            '/dev/system/root' [39.06 GB] inherit
  ACTIVE            '/dev/system/swap' [9.75 GB] inherit



[root@xen ~]# vgchange -an VolGroup00
  0 logical volume(s) in volume group "VolGroup00" now active
[root@xen ~]# kpartx -d /dev/loop4
[root@xen ~]# losetup -d /dev/loop4
[root@xen ~]#

[root@xen ~]# xm create -c /etc/xen/hfserver2


And then it dies:


 Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup00" using metadata type lvm2
Activating logical volumes
  2 logical volume(s) in volume group "VolGroup00" now active
Creating root device.
Mounting root filesystem.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
exec of init (/sbin/init) failed!!!: No such file or directory
Kernel panic - not syncing: Attempted to kill init!
 [root@xen ~]# 


--
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users