Re: [Xen-users] I upgraded, now vms won't start

[Chris <cjdl01@xxxxxxxxxxxxxxxxxx>]

Quoting Steve Allison <xen-users@xxxxxxxxxxxxxxxxxx>:

> On 13/11/2011 00:23, Chris wrote:
> > Hi,
> >
> > I have been going over and over this for about 8 hours straight,   
> > and I can't see what I'm doing wrong.
> > I have just upgraded from Debian lenny to Squeeze, using their Xen.
> >
> > When I was running lenny, I made my domUs with the debian tool   
> > utility.  It set the drives to sda*, instead of xvda*.
> > I know that with Xen 4, they like you to use xvda*, rather than   
> > sda*.  I'm cool with that, so I changed it in my configs, and even   
> > mounted the domu in question, and set it correctly in /etc/fstab.    
> > However, it still will not boot.  In the console, I see it complain  
> >  that it wasn't able to assemble all the raid devices -- why that   
> > matters, I don't know, Dom0 assembled them just fine, I don't   
> > understand why they would need to be reassembled... but anyway...    
> > Here is what I am getting at the xm console:
> > dom0box:/etc/xen/configs# xm console domubox
> > [    0.000000] Initializing cgroup subsys cpuset
> > [    0.000000] Initializing cgroup subsys cpu
> > [    0.000000] Linux version 2.6.32-5-xen-amd64 (Debian 2.6.32-38)   
> > (ben@xxxxxxxxxxxxxxx) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP   
> > Mon Oct 3 07:53:54 UTC 2011
> > [    0.000000] Command line: root=/dev/xvda2 ro   
> > ip=:127.0.255.255::::eth0:dhcp clocksource=jiffies
> > [    0.000000] KERNEL supported cpus:
> > [    0.000000]   Intel GenuineIntel
> > [    0.000000]   AMD AuthenticAMD
> > [    0.000000]   Centaur CentaurHauls
> > [    0.000000] ACPI in unprivileged domain disabled
> > [    0.000000] released 0 pages of unused memory
> > [    0.000000] BIOS-provided physical RAM map:
> > [    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
> > [    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
> > [    0.000000]  Xen: 0000000000100000 - 0000000020000000 (usable)
> > [    0.000000] DMI not present or invalid.
> > [    0.000000] last_pfn = 0x20000 max_arch_pfn = 0x400000000
> > [    0.000000] init_memory_mapping: 0000000000000000-0000000020000000
> > [    0.000000] RAMDISK: 016b9000 - 01f21000
> > [    0.000000] No NUMA configuration found
> > [    0.000000] Faking a node at 0000000000000000-0000000020000000
> > [    0.000000] Bootmem setup node 0 0000000000000000-0000000020000000
> > [    0.000000]   NODE_DATA [0000000000008000 - 000000000000ffff]
> > [    0.000000]   bootmap [0000000000010000 -  0000000000013fff] pages 4
> > [    0.000000] (7 early reservations) ==> bootmem [0000000000 - 0020000000]
> > [    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==>   
> > [0000000000 - 0000001000]
> > [    0.000000]   #1 [0002024000 - 0002039000]   XEN PAGETABLES ==>   
> > [0002024000 - 0002039000]
> > [    0.000000]   #2 [0000006000 - 0000008000]       TRAMPOLINE ==>   
> > [0000006000 - 0000008000]
> > [    0.000000]   #3 [0001000000 - 0001698b14]    TEXT DATA BSS ==>   
> > [0001000000 - 0001698b14]
> > [    0.000000]   #4 [00016b9000 - 0001f21000]          RAMDISK ==>   
> > [00016b9000 - 0001f21000]
> > [    0.000000]   #5 [0001f21000 - 0002024000]   XEN START INFO ==>   
> > [0001f21000 - 0002024000]
> > [    0.000000]   #6 [0000100000 - 00001ea000]          PGTABLE ==>   
> > [0000100000 - 00001ea000]
> > [    0.000000] Zone PFN ranges:
> > [    0.000000]   DMA      0x00000000 -> 0x00001000
> > [    0.000000]   DMA32    0x00001000 -> 0x00100000
> > [    0.000000]   Normal   0x00100000 -> 0x00100000
> > [    0.000000] Movable zone start PFN for each node
> > [    0.000000] early_node_map[2] active PFN ranges
> > [    0.000000]     0: 0x00000000 -> 0x000000a0
> > [    0.000000]     0: 0x00000100 -> 0x00020000
> > [    0.000000] SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
> > [    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
> > [    0.000000] No local APIC present
> > [    0.000000] APIC: disable apic facility
> > [    0.000000] PM: Registered nosave memory: 00000000000a0000 -   
> > 0000000000100000
> > [    0.000000] Allocating PCI resources starting at 20000000 (gap:   
> > 20000000:e0000000)
> > [    0.000000] Booting paravirtualized kernel on Xen
> > [    0.000000] Xen version: 4.0.1 (preserve-AD)
> > [    0.000000] NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:1 nr_node_ids:1
> > [    0.000000] PERCPU: Embedded 30 pages/cpu @ffff88000206f000   
> > s90328 r8192 d24360 u122880
> > [    0.000000] pcpu-alloc: s90328 r8192 d24360 u122880 alloc=30*4096
> > [    0.000000] pcpu-alloc: [0] 0
> > [    0.000000] Xen: using vcpu_info placement
> > [    0.000000] Built 1 zonelists in Node order, mobility grouping   
> > on.  Total pages: 128947
> > [    0.000000] Policy zone: DMA32
> > [    0.000000] Kernel command line: root=/dev/xvda2 ro   
> > ip=:127.0.255.255::::eth0:dhcp clocksource=jiffies
> > [    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
> > [    0.000000] Initializing CPU#0
> > [    0.000000] Checking aperture...
> > [    0.000000] No AGP bridge found
> > [    0.000000] Memory: 498868k/524288k available (3149k kernel   
> > code, 384k absent, 25036k reserved, 1906k data, 604k init)
> > [    0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3,   
> > MinObjects=0, CPUs=1, Nodes=1
> > [    0.000000] Hierarchical RCU implementation.
> > [    0.000000] NR_IRQS:4352 nr_irqs:512
> > [    0.000000] Console: colour dummy device 80x25
> > [    0.000000] console [tty0] enabled
> > [    0.000000] console [hvc0] enabled
> > [    0.000000] installing Xen timer for CPU 0
> > [    0.000000] Detected 2800.168 MHz processor.
> > [    0.004000] Calibrating delay loop (skipped), value calculated   
> > using timer frequency.. 5600.33 BogoMIPS (lpj=11200672)
> > [    0.004000] Security Framework initialized
> > [    0.004000] SELinux:  Disabled at boot.
> > [    0.004000] Dentry cache hash table entries: 65536 (order: 7,   
> > 524288 bytes)
> > [    0.004000] Inode-cache hash table entries: 32768 (order: 6,   
> > 262144 bytes)
> > [    0.004000] Mount-cache hash table entries: 256
> > [    0.004000] Initializing cgroup subsys ns
> > [    0.004000] Initializing cgroup subsys cpuacct
> > [    0.004000] Initializing cgroup subsys devices
> > [    0.004000] Initializing cgroup subsys freezer
> > [    0.004000] Initializing cgroup subsys net_cls
> > [    0.004000] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K   
> > (64 bytes/line)
> > [    0.004000] CPU: L2 Cache: 1024K (64 bytes/line)
> > [    0.004000] CPU 0/0x0 -> Node 0
> > [    0.004000] CPU: Physical Processor ID: 1
> > [    0.004000] CPU: Processor Core ID: 0
> > [    0.004000] Performance Events: AMD PMU driver.
> > [    0.004000] ... version:                0
> > [    0.004000] ... bit width:              48
> > [    0.004000] ... generic registers:      4
> > [    0.004000] ... value mask:             0000ffffffffffff
> > [    0.004000] ... max period:             00007fffffffffff
> > [    0.004000] ... fixed-purpose events:   0
> > [    0.004000] ... event mask:             000000000000000f
> > [    0.004000] SMP alternatives: switching to UP code
> > [    0.004000] Freeing SMP alternatives: 28k freed
> > [    0.004277] Brought up 1 CPUs
> > [    0.004436] devtmpfs: initialized
> > [    0.007292] Grant table initialized
> > [    0.007299] regulator: core version 0.5
> > [    0.007362] NET: Registered protocol family 16
> > [    0.008000] PCI: setting up Xen PCI frontend stub
> > [    0.008435] bio: create slab <bio-0> at 0
> > [    0.008520] ACPI: Interpreter disabled.
> > [    0.008563] xen_balloon: Initialising balloon driver with page order 0.
> > [    0.008624] vgaarb: loaded
> > [    0.008704] PCI: System does not support PCI
> > [    0.008710] PCI: System does not support PCI
> > [    0.010231] pnp: PnP ACPI: disabled
> > [    0.010484] NET: Registered protocol family 2
> > [    0.010572] IP route cache hash table entries: 4096 (order: 3,   
> > 32768 bytes)
> > [    0.010861] TCP established hash table entries: 16384 (order: 6,  
> >  262144 bytes)
> > [    0.011043] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
> > [    0.011157] TCP: Hash tables configured (established 16384 bind 16384)
> > [    0.011165] TCP reno registered
> > [    0.011235] NET: Registered protocol family 1
> > [    0.011295] Unpacking initramfs...
> > [    0.022974] Freeing initrd memory: 8608k freed
> > [    0.028269] platform rtc_cmos: registered platform RTC device   
> > (no PNP device found)
> > [    0.028524] audit: initializing netlink socket (disabled)
> > [    0.032013] type=2000 audit(1321142933.822:1): initialized
> > [    0.034410] HugeTLB registered 2 MB page size, pre-allocated 0 pages
> > [    0.035999] VFS: Disk quotas dquot_6.5.2
> > [    0.036039] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> > [    0.036147] msgmni has been set to 991
> > [    0.036367] alg: No test for stdrng (krng)
> > [    0.036429] Block layer SCSI generic (bsg) driver version 0.4   
> > loaded (major 253)
> > [    0.036439] io scheduler noop registered
> > [    0.036445] io scheduler anticipatory registered
> > [    0.036451] io scheduler deadline registered
> > [    0.036487] io scheduler cfq registered (default)
> > [    0.042810] registering netback
> > [    0.044510] Linux agpgart interface v0.103
> > [    0.044551] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
> > [    0.044778] input: Macintosh mouse button emulation as   
> > /devices/virtual/input/input0
> > [    0.044836] PNP: No PS/2 controller found. Probing ports directly.
> > [    0.045659] i8042.c: No controller found.
> > [    0.045741] mice: PS/2 mouse device common for all mice
> > [    0.045846] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
> > [    0.045906] cpuidle: using governor ladder
> > [    0.045914] cpuidle: using governor menu
> > [    0.045927] No iBFT detected.
> > [    0.046192] TCP cubic registered
> > [    0.046317] NET: Registered protocol family 10
> > [    0.046993] Mobile IPv6
> > [    0.047003] NET: Registered protocol family 17
> > [    0.047095] registered taskstats version 1
> > [    0.047112] XENBUS: Device with no driver: device/vbd/51713
> > [    0.047119] XENBUS: Device with no driver: device/vbd/51714
> > [    0.047126] XENBUS: Device with no driver: device/vif/0
> > [    0.047132] XENBUS: Device with no driver: device/console/0
> > [    0.047152]   
> > /build/buildd-linux-2.6_2.6.32-38-amd64-bk66e4/linux-2.6-2.6.32/debian/build/source_amd64_xen/drivers/rtc/hctosys.c: unable to open rtc device   
> > (rtc0)
> > [    0.047191] Initalizing network drop monitor service
> > [    0.047267] Freeing unused kernel memory: 604k freed
> > [    0.047549] Write protecting the kernel read-only data: 4332k
> > Loading, please wait...
> > [    0.088230] udev[45]: starting version 164
> > Begin: Loading essential drivers ... [    0.165910] SCSI subsystem   
> > initialized
> > done.
> > Begin: Running /scripts/init-premount ... done.
> > Begin: Mounting root file system ... Begin: Running   
> > /scripts/local-top ... Begin: Loading[    0.202908] md: raid1   
> > personality registered for level 1
> > Success: loaded module raid1.
> > [    0.205951] async_tx: api initialized (async)
> > [    0.206720] xor: automatically using best checksumming function:  
> >  generic_sse
> > [    0.224008]    generic_sse:  3660.000 MB/sec
> > [    0.224016] xor: using function: generic_sse (3660.000 MB/sec)
> > [    0.292026] raid6: int64x1   2503 MB/s
> > [    0.360020] raid6: int64x2   3307 MB/s
> > [    0.428014] raid6: int64x4   2505 MB/s
> > [    0.496025] raid6: int64x8   2286 MB/s
> > [    0.564012] raid6: sse2x1     767 MB/s
> > [    0.632030] raid6: sse2x2    1508 MB/s
> > [    0.700022] raid6: sse2x4    2062 MB/s
> > [    0.700030] raid6: using algorithm sse2x4 (2062 MB/s)
> > [    0.707225] md: raid6 personality registered for level 6
> > [    0.707237] md: raid5 personality registered for level 5
> > [    0.707244] md: raid4 personality registered for level 4
> > Success: loaded module raid456.
> > done.
> > Begin: Assembling all MD arrays ... mdadm: No devices listed in   
> > conf file were found.
> > Failure: failed to assemble all arrays.
> > done.
> > [    0.722826] device-mapper: uevent: version 1.0.3
> > [    0.723476] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01)   
> > initialised: dm-devel@xxxxxxxxxx
> > done.
> > Begin: Waiting for root file system ... done.
> > Gave up waiting for root device.  Common problems:
> > - Boot args (cat /proc/cmdline)
> >   - Check rootdelay= (did the system wait long enough?)
> >   - Check root= (did the system wait for the right device?)
> > - Missing modules (cat /proc/modules; ls /dev)
> > ALERT!  /dev/xvda2 does not exist.  Dropping to a shell!
> >
> >
> > BusyBox v1.17.1 (Debian 1:1.17.1-8) built-in shell (ash)
> > Enter 'help' for a list of built-in commands.
> >
> > /bin/sh: can't access tty; job control turned off
> > (initramfs)
> >
> >
> > Here is my config:
> > kernel      = '/boot/vmlinuz-2.6.32-5-xen-amd64'
> > ramdisk     = '/boot/initrd.img-2.6.32-5-xen-amd64'
> > memory      = '512'
> > root        = '/dev/xvda2 ro'
> > disk        = [
> >                  'phy:/dev/vg/domubox-swap,xvda1,w',
> >                  'phy:/dev/vg/domubox-disk,xvda2,w'
> >              ]
> > vcpus = 1
> > cpus = "0"
> > name        = 'domubox'
> > dhcp        = 'dhcp'
> > vif         = [ 'mac=00:16:3E:xx:xx:xx' ]
> > on_poweroff = 'destroy'
> > on_reboot   = 'restart'
> > on_crash    = 'restart'
> > extra = "console=hvc0 xencons=tty"
> > extra="clocksource=jiffies
> >
> > I have checked (on Dom0) my /etc/mdadm/mdadm.conf file, and the   
> > UUIDs are correct.  I have been looking at scores of posts on the   
> > list and elsewhere for help, and one had stated that changing   
> > level=raidx to level=x, but that didn't help either.
> > Interestingly enough, my hvm domus DO start up fine.  They were   
> > made using xvda* instead of sda*... I don't know which part here is  
> >  significant (hvm or being made with xvda).
> > I also noticed that the debian wiki says: "If your domU kernel   
> > happens to miss support for the xvda* disk devices (the   
> > xen-blkfront driver), use the --scsi option that makes the VM use   
> > normal SCSI HD names like sda*. "
> > However, they do not say where to use the --scsi... the xm command   
> > doesn't accept it... so what else is there?
> > I've been over this a million times now, I just can't see what I'm missing.
> >
> > Can someone please give me a hand here?
> >
> > Thank you.
> >
> >
> >
> >
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-users
> Oops, apologies for sending this off-list :/
>
> Is it something silly like the order of your disks in the config?
>
> Try:
> disk        = [
>                   'phy:/dev/vg/domubox-disk,xvda2,w',
>                   'phy:/dev/vg/domubox-swap,xvda1,w'
>               ]
>
>
> Have you tried booting with pygrub? (this would depend on a valid
> /boot/grub/menu.lst, which you could create manually easily enough)
>
> #kernel      = '/boot/vmlinuz-2.6.32-5-xen-amd64'
> #ramdisk     = '/boot/initrd.img-2.6.32-5-xen-amd64'
> bootloader      = '/usr/lib/xen-default/bin/pygrub'
>
> Extra edit: If you have updated the dom0 did you copy the modules from
> /lib/modules for the new kernel to the guest?
Thanks for the response, however, I'm still not having any luck.

I tried reversing the disk lines as you mentioned, I get the same  
results.  This domu has been running for three years.  I know the  
config is good.  In fact, I updated another xen server just the other  
day, and did not have these problems.  The only difference I see is  
the /dev/sda vs. /dev/xvda thing.  That machine had domUs created with  
xvda*, mine has domUs created with /dev/sda*.  According to all the  
posts I read, switching the sdx to xvdx should be enough, but it  
isn't.  Something is missing... a driver, a library, some sort of  
emulation layer... something....
I have read some on pygrub, I understand what it is supposed to  
accomplish, but I am very confused on how it is supposed to work.  I  
am running Squeeze, which uses grub 2, which does NOT use a menu.lst.   
All the documentation I see on pygrub refers to the old style grub.   
So, I don't know if my domU here will run under pygrub... but I really  
think that we are barking up the wrong tree...  even if I get it  
running under pygrub, I don't know what that would prove.  I need it  
running normally through xen.
Why the heck is it trying to rebuild the array anyway?

I did copy over the modules from /lib/modules on dom0 to domU, but it  
still does not boot...


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users