WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

[Xen-devel] Re: 2.6.37-rc1 mainline domU - BUG: unable to handle kernel

To: xen-devel@xxxxxxxxxxxxxxxxxxx, Jeremy Fitzhardinge <jeremy@xxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Subject: [Xen-devel] Re: 2.6.37-rc1 mainline domU - BUG: unable to handle kernel paging request
From: Bruce Edge <bruce.edge@xxxxxxxxx>
Date: Thu, 11 Nov 2010 07:08:03 -0800
Cc:
Delivery-date: Thu, 11 Nov 2010 07:09:38 -0800
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=/SlolHAhZnfkfoizketiPtkdikdnZ4JvOEnuFCY41BM=; b=sJJnpkJWUCu2c/AYfFC3ug/aUENknQVY0XqT9TUivt6ZgdTROb4/quT1DapXxQTEpH sc9tCRLpHG3jDc+wEz05iVgcwD5/CNmJ5wTida0qBzQHBmMM0cI7pODMZ/a/yeRz2797 BDH738ROkhiT7krOagTbhFq8M50hpMDt5mkgg=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=BBJ0ionDLeA5M+4GxUxRnI6Je7bOOV9po6uxeTz2HhSfGC/AwKiwWzL52iSNZva+hG ANqo2lvNQQw4I1Kt2KZI7aFN6XRqb1DTD+cczpjjtX5bYOdU0UdI6vBYrxxbJTVSkFc6 ouWNCayaxG+EFL1oDWOXXXHfTaABYjomprxSw=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <AANLkTikemdbCx8yzvZ+m_f_zmyoW1SxjSAWZKi_XMteJ@xxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <AANLkTimP7+osxudXViX60JrsKhteMhP2YXBQkz2r69Cn@xxxxxxxxxxxxxx> <AANLkTikemdbCx8yzvZ+m_f_zmyoW1SxjSAWZKi_XMteJ@xxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx

On Wed, Nov 10, 2010 at 2:30 PM, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:
On Wed, Nov 10, 2010 at 2:15 PM, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:
>
> This is a mainline 2.6.37-rc1 pvops domU on top of a xen-unstable build 22241 and xen/stable-2.6.32.x 2.6.32.24 pvops dom0.
>
> The domU console output hung up prior to spitting out a prompt post root login. I logged into the domU via ssh and ran an 'ls' of an NFS mounted dir and
> this immediately came out on the domU console:
>
> [  480.467525] BUG: unable to handle kernel paging request at ffff880077e9f030
> [  480.467547] IP: [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
> [  480.467564] PGD 1a04067 PUD 3dd7067 PMD 3f97067 PTE 8010000077e9f065
> [  480.467587] Oops: 0003 [#1] SMP
> [  480.467601] last sysfs file: /sys/kernel/uevent_seqnum
> [  480.467611] CPU 3
> [  480.467616] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
> [  480.467676]
> [  480.467684] Pid: 3957, comm: zsh Not tainted 2.6.37-rc1-kernel.org-domu-6.0.bedge-debug #1 /
> [  480.467695] RIP: e030:[<ffffffff810355f3>]  [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
> [  480.467710] RSP: e02b:ffff880077ebfbd8  EFLAGS: 00010202
> [  480.467719] RAX: 800000004d686167 RBX: ffff8800763ddb80 RCX: 800000004d686167
> [  480.467729] RDX: ffff880077e9f030 RSI: 00007f5cb26069d0 RDI: ffff8800763ddb80
> [  480.467739] RBP: ffff880077ebfc08 R08: 0000000000000001 R09: ffffea0001a3b2d8
> [  480.467748] R10: 0000000000000100 R11: 0000000000000000 R12: 00007f5cb26069d0
> [  480.467759] R13: 0000000000000001 R14: 0000000000000000 R15: ffff8800763ddb80
> [  480.467772] FS:  00007f5cb2606700(0000) GS:ffff88007ff94000(0000) knlGS:0000000000000000
> [  480.467783] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  480.467792] CR2: ffff880077e9f030 CR3: 0000000077dbc000 CR4: 0000000000002660
> [  480.467802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  480.467812] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [  480.467822] Process zsh (pid: 3957, threadinfo ffff880077ebe000, task ffff880078b80000)
> [  480.467832] Stack:
> [  480.467838]  0000000000000100 ffffea0001a3b2d8 7000000000000000 0000000000000000
> [  480.467862]  ffff880077e9f030 ffffea0001a3b2d8 ffff880077ebfca8 ffffffff810e2200
> [  480.467886]  ffff880077ebfc38 ffffffff81007c09 ffffea0001a3b2d8 0000000000000001
> [  480.467910] Call Trace:
> [  480.467921]  [<ffffffff810e2200>] do_wp_page+0x2d8/0x6be
> [  480.467934]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
> [  480.467946]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
> [  480.467958]  [<ffffffff810e2e60>] handle_mm_fault+0x87a/0x8c1
> [  480.467972]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
> [  480.467984]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
> [  480.467996]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
> [  480.468009]  [<ffffffff810072a2>] ? check_events+0x12/0x20
> [  480.468020]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
> [  480.468034]  [<ffffffff810a6fa4>] ? __call_rcu+0x123/0x12c
> [  480.468045]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
> [  480.468059]  [<ffffffff810072a2>] ? check_events+0x12/0x20
> [  480.468072]  [<ffffffff81443615>] page_fault+0x25/0x30
> [  480.468085]  [<ffffffff81206add>] ? __put_user_4+0x1d/0x30
> [  480.468096]  [<ffffffff810498a6>] ? schedule_tail+0x5c/0x60
> [  480.468108]  [<ffffffff8100a9c3>] ret_from_fork+0x13/0x80
> [  480.468116] Code: 41 5d c9 c3 55 48 89 e5 41 55 45 31 ed 41 54 49 89 f4 53 48 89 fb 48 83 ec 18 48 39 0a 41 0f 95 c5 45 85 c0 74 1d 45 85 ed 74 18 <48> 89 0a 48 8b 3f 0f 1f 80 00 00 00 00 4c 89 e6 48 89 df e8 b2
> [  480.468323] RIP  [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
> [  480.468323]  RSP <ffff880077ebfbd8>
> [  480.468323] CR2: ffff880077e9f030
> [  480.468323] ---[ end trace 6d1623321ecd50ac ]---
>
> The domU config is attached as config.2.6.37_domU_debug
>
>
> -Bruce

Here's another attempt from the same config with the full domU boot logs:

Started domain dpm (id=4)
                        [    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version
2.6.37-rc1-kernel.org-domu-6.0.bedge-debug (root@opal) (gcc version
4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #1 SMP Tue Nov 9 20:26:03 PST 2010
[    0.000000] Command line:
root=UUID=4d87e4cb-8317-41d4-aea2-ac2ebee699f7 ro   swiotlb=force
noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r
ip=:127.0.255.255::::eth0:dhcp
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000080800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x80800 max_arch_pfn = 0x400000000
[    0.000000] Scanning 0 areas for low memory corruption
[    0.000000] init_memory_mapping: 0000000000000000-0000000080800000
[    0.000000] RAMDISK: 01d95000 - 02fa9000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-0000000080800000
[    0.000000] Initmem setup node 0 0000000000000000-0000000080800000
[    0.000000]   NODE_DATA [000000007fffb000 - 000000007fffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00080800
[    0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 80800000 (gap:
80800000:7f800000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.1-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32
nr_cpu_ids:6 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88007ff40000 s82752
r8192 d23744 u114688
[    0.000000] pcpu-alloc: s82752 r8192 d23744 u114688 alloc=28*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.
Total pages: 519022
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line:
root=UUID=4d87e4cb-8317-41d4-aea2-ac2ebee699f7 ro   swiotlb=force
noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r
ip=:127.0.255.255::::eth0:dhcp
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Placing 64MB software IO TLB between ffff880079400000 -
ffff88007d400000
[    0.000000] software IO TLB at phys 0x79400000 - 0x7d400000
[    0.000000] Memory: 1959896k/2105344k available (4398k kernel code,
448k absent, 145000k reserved, 6327k data, 756k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0,
CPUs=6, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:320 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2660.064 MHz processor.
[    0.000999] Calibrating delay loop (skipped), value calculated
using timer frequency.. 5320.12 BogoMIPS (lpj=2660064)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001408] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001635] Mount-cache hash table entries: 256
[    0.002195] Initializing cgroup subsys ns
[    0.002208] ns_cgroup deprecated: consider using the
'clone_children' flag without the ns_cgroup.
[    0.002220] Initializing cgroup subsys cpuacct
[    0.002231] Initializing cgroup subsys freezer
[    0.002356] CPU: Unsupported number of siblings 16
[    0.002362] Performance Events: unsupported p6 CPU model 26 no PMU
driver, software events only.
[    0.002483] SMP alternatives: switching to UP code
[    0.004119] cpu 0 spinlock event irq 318
[    0.004765] NMI watchdog failed to create perf event on cpu0:
fffffffffffffffe
[    0.005050] installing Xen timer for CPU 1
[    0.005157] cpu 1 spinlock event irq 312
[    0.005194] SMP alternatives: switching to SMP code
[    0.000999] CPU: Unsupported number of siblings 16
[    0.006158] NMI watchdog failed to create perf event on cpu1:
fffffffffffffffe
[    0.006474] installing Xen timer for CPU 2
[    0.006585] cpu 2 spinlock event irq 306
[    0.000999] CPU: Unsupported number of siblings 16
[    0.006816] NMI watchdog failed to create perf event on cpu2:
fffffffffffffffe
[    0.006999] installing Xen timer for CPU 3
[    0.007037] cpu 3 spinlock event irq 300
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007300] NMI watchdog failed to create perf event on cpu3:
fffffffffffffffe
[    0.007534] installing Xen timer for CPU 4
[    0.007636] cpu 4 spinlock event irq 294
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007874] NMI watchdog failed to create perf event on cpu4:
fffffffffffffffe
[    0.008041] installing Xen timer for CPU 5
[    0.008083] cpu 5 spinlock event irq 288
[    0.000999] CPU: Unsupported number of siblings 16
[    0.008344] NMI watchdog failed to create perf event on cpu5:
fffffffffffffffe
[    0.008356] Brought up 6 CPUs
[    0.009181] kworker/u:0 used greatest stack depth: 5696 bytes left
[    0.012051] Grant table initialized
[    0.030978] Time: 165:165:165  Date: 165/165/65
[    0.031229] NET: Registered protocol family 16
[    0.036672] PCI: setting up Xen PCI frontend stub
[    0.037225] kworker/u:0 used greatest stack depth: 5528 bytes left
[    0.074302] bio: create slab <bio-0> at 0
[    0.075385] ACPI: Interpreter disabled.
[    0.077056] xen_balloon: Initialising balloon driver.
[    0.079323] vgaarb: loaded
[    0.081270] usbcore: registered new interface driver usbfs
[    0.081283] usbcore: registered new interface driver hub
[    0.081283] usbcore: registered new device driver usb
[    0.083183] PCI: System does not support PCI
[    0.084007] PCI: System does not support PCI
[    0.085236] NetLabel: Initializing
[    0.085236] NetLabel:  domain hash size = 128
[    0.085236] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.085313] NetLabel:  unlabeled traffic allowed by default
[    0.086544] Switching to clocksource xen
[    0.106256] pnp: PnP ACPI: disabled
[    0.129799] NET: Registered protocol family 2
[    0.130174] IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.131882] TCP established hash table entries: 262144 (order: 10,
4194304 bytes)
[    0.132891] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.133179] TCP: Hash tables configured (established 262144 bind 65536)
[    0.133191] TCP reno registered
[    0.133301] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[    0.133407] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[    0.133914] NET: Registered protocol family 1
[    0.134582] RPC: Registered udp transport module.
[    0.134590] RPC: Registered tcp transport module.
[    0.134595] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.134892] Trying to unpack rootfs image as initramfs...
[    0.178824] Freeing initrd memory: 18512k freed
[    0.268798] DMA-API: preallocated 32768 debug entries
[    0.268810] DMA-API: debugging enabled by kernel config
[    0.271181] platform rtc_cmos: registered platform RTC device (no
PNP device found)
[    0.285410] Machine check injector initialized
[    0.294732] microcode: CPU0 sig=0x106a4, pf=0x2, revision=0xa
[    0.294803] microcode: CPU1 sig=0x106a4, pf=0x2, revision=0xa
[    0.294893] microcode: CPU2 sig=0x106a4, pf=0x2, revision=0xa
[    0.294944] microcode: CPU3 sig=0x106a4, pf=0x2, revision=0xa
[    0.294998] microcode: CPU4 sig=0x106a4, pf=0x2, revision=0xa
[    0.295043] microcode: CPU5 sig=0x106a4, pf=0x2, revision=0xa
[    0.295590] microcode: Microcode Update Driver: v2.00
<tigran@xxxxxxxxxxxxxxxxxxxx>, Peter Oruba
[    0.295607] Scanning for low memory corruption every 60 seconds
[    0.298813] audit: initializing netlink socket (disabled)
[    0.298940] type=2000 audit(1289403082.795:1): initialized
[    0.333762] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.376985] VFS: Disk quotas dquot_6.5.2
[    0.377853] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.383121] NTFS driver 2.1.29 [Flags: R/W].
[    0.385014] msgmni has been set to 3864
[    0.389953] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 253)
[    0.389970] io scheduler noop registered
[    0.389979] io scheduler deadline registered
[    0.390666] io scheduler cfq registered (default)
[    0.392486] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.406238] pcifront pci-0: Installing PCI frontend
[    0.406967] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/3
[    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/4
[    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/0
[    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/2
[    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/3
[    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/4
[    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/0
[    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/2
[    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/3
[    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/4
[    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/0
[    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/2
[    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/3
[    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/4
[    0.731775] Non-volatile memory driver v1.3
[    0.731790] Linux agpgart interface v0.103
[    0.733731] [drm] Initialized drm 1.1.0 20060810
[    0.733752] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.827202] brd: module loaded
[    0.840788] loop: module loaded
[    0.841965] Fixed MDIO Bus: probed
[    0.844025] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.844651] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.845194] uhci_hcd: USB Universal Host Controller Interface driver
[    0.846985] usbcore: registered new interface driver usblp
[    0.847763] usbcore: registered new interface driver libusual
[    0.849433] PNP: No PS/2 controller found. Probing ports directly.
[    0.850279] i8042.c: No controller found.
[    0.850883] mice: PS/2 mouse device common for all mice
[    0.853446] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.854066] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.855928] cpuidle: using governor ladder
[    0.855941] cpuidle: using governor menu
[    0.856029] Netfilter messages via NETLINK v0.30.
[    0.856136] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.857888] ctnetlink v0.93: registering with nfnetlink.
[    0.860285] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.860677] TCP cubic registered
[    0.860687] Initializing XFRM netlink socket
[    0.863286] NET: Registered protocol family 10
[    0.867216] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.867612] IPv6 over IPv4 tunneling driver
[    0.871326] NET: Registered protocol family 17
[    0.871566] Registering the dns_resolver key type
[    0.873668] registered taskstats version 1
[    0.873912] kmemleak: Kernel memory leak detector initialized
[    0.873928] kmemleak: Automatic memory scanning thread started
[    0.875057] XENBUS: Device with no driver: device/vbd/51712
[    0.875057] XENBUS: Device with no driver: device/vif/0
[    0.875057] XENBUS: Device with no driver: device/console/0
[    0.875057]   Magic number: 1:252:3141
[   13.388958] Freeing unused kernel memory: 756k freed
[   13.389190] Write protecting the kernel read-only data: 10240k
[   13.398774] Freeing unused kernel memory: 1724k freed
[   13.400663] Freeing unused kernel memory: 1992k freed
[   13.494573] all_generic_ide used greatest stack depth: 5472 bytes left
[   13.519388] udevd (1111): /proc/1111/oom_adj is deprecated, please
use /proc/1111/oom_score_adj instead.
Begin: Loading essential drivers... ...
[   13.559435] blkid used greatest stack depth: 5408 bytes left
[   13.568391] blkfront: xvda: barriers disabled
[   13.574762] blkid used greatest stack depth: 5264 bytes left
[   13.580813]  xvda: xvda1 xvda2 < xvda5 >
[   13.645562] modprobe used greatest stack depth: 5088 bytes left
[   13.653160] Initialising Xen virtual ethernet driver.
[   13.659540] blkid used greatest stack depth: 4960 bytes left
[   13.735467] blkid used greatest stack depth: 4752 bytes left
Done.
Begin: Running /scripts/init-premount ...
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
Begin: Running /scripts/local-premount ...
Done.
[   13.903774] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
[   13.903799] EXT4-fs (xvda1): write access will be enabled during recovery
[   14.033719] EXT4-fs (xvda1): recovery complete
[   14.034966] EXT4-fs (xvda1): mounted filesystem with ordered data
mode. Opts: (null)
[   14.035510] exe used greatest stack depth: 4112 bytes left
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.
init: ureadahead main process (1214) terminated with status 5
fsck from util-linux-ng 2.17.2
e2fsck 1.41.11 (14-Mar-2010)
/dev/xvda1: clean, 56314/288576 files, 287743/1153280 blocks
 * Enable login on console                                               [ OK ]
 * Check for hostname changes...                                         [ OK ]
  ...done.
 * Setting sensors limits
        No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
                                                                        [ OK ]
 * Initializing random number generator...                               [ OK ]
 * Setting up X server socket directory /tmp/.X11-unix...                [ OK ]
 * Setting up ICE socket directory /tmp/.ICE-unix...                     [ OK ]
 * Not starting fancontrol; run pwmconfig first.
 * Starting internet superserver inetd                                   [ OK ]
Starting openntpd: ntpd.
 * Starting network management services:
 * Setting core file location...                                         [ OK ]
 * Restoring resolver state...                                           [ OK ]
 * Checking tomcat configuration
  ...done.
 * Starting Tomcat servlet engine tomcat6
        Using CATALINA_BASE:   /var/lib/tomcat6
Using CATALINA_HOME:   /usr/share/tomcat6
Using CATALINA_TMPDIR: /tmp/tomcat6-tmp
Using JRE_HOME:        /usr/lib/jvm/java-6-openjdk
Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
                                                                        [ OK ]
collectd has been disabled - see /etc/default/collectd.
 * Recording successful boot for GRUB
  ...done.
 * Running local boot scripts (/etc/rc.local)                            [ OK ]
mountall: Plymouth command failed
mountall: Disconnected from Plymouth

6.0.16-01
kaan-04-dpm login: root
Password:
[   42.646538] ------------[ cut here ]------------
[   42.646556] kernel BUG at arch/x86/xen/mmu.c:1831!
[   42.646566] invalid opcode: 0000 [#1] SMP
[   42.646582] last sysfs file: /sys/kernel/uevent_seqnum
[   42.646592] CPU 0
[   42.646598] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront
fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront
xen_netfront xen_blkfront
[   42.646665]
[   42.646674] Pid: 3916, comm: lsb_release Not tainted
2.6.37-rc1-kernel.org-domu-6.0.bedge-debug #1 /
[   42.646688] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>]
pin_pagetable_pfn+0x31/0x37
[   42.646711] RSP: e02b:ffff88007595bce8  EFLAGS: 00010282
[   42.646721] RAX: 00000000ffffffea RBX: 000000000007751e RCX: 0000000000000001
[   42.646732] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[   42.646744] RBP: ffff88007595bd08 R08: 00003ffffffff000 R09: ffff880000000000
[   42.646756] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[   42.646768] R13: ffff880075c90150 R14: ffff8800781d9398 R15: ffff880075161d40
[   42.646784] FS:  00007fee63b01700(0000) GS:ffff88007ff40000(0000)
knlGS:0000000000000000
[   42.646797] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   42.646807] CR2: 00007faac54182b8 CR3: 00000000771dd000 CR4: 0000000000002660
[   42.646819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   42.646832] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   42.646844] Process lsb_release (pid: 3916, threadinfo
ffff88007595a000, task ffff880075161d40)
[   42.646857] Stack:
[   42.646864]  ffff880000000000 00000000000484f0 ffff8800777c6d80
000000000007751e
[   42.646892]  ffff88007595bd28 ffffffff810066a8 000000000007751e
ffff8800777c6d80
[   42.646919]  ffff88007595bd38 ffffffff810066db ffff88007595bd78
ffffffff810e048f
[   42.646946] Call Trace:
[   42.646959]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[   42.646973]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[   42.646989]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[   42.647003]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
[   42.647017]  [<ffffffff810e6e36>] ? mmap_region+0x391/0x4bf
[   42.647033]  [<ffffffff8100eb84>] ? arch_get_unmapped_area_topdown+0x4a/0x28a
[   42.647052]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[   42.647059]  [<ffffffff8102b78a>] ? pvclock_clocksource_read+0x4b/0xb4
[   42.647059]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
[   42.647059]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
[   42.647059]  [<ffffffff81443615>] page_fault+0x25/0x30
[   42.647059] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d
e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff
85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48
8b 3d
[   42.647059] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   42.647059]  RSP <ffff88007595bce8>
[   42.647059] ---[ end trace 3fbbffee414a5166 ]---


Here's another full boot that I just let sit at the login prompt. This is using the pcifront-fixes kernel:

Started domain dpm (id=1)
                         [    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug (root@opal) (gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #1 SMP Wed Nov 10 17:11:41 PST 2010
[    0.000000] Command line: root=UUID=1e67e680-4108-4e11-8bc4-844dad448902 ro   swiotlb=force noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r ip=:127.0.255.255::::eth0:dhcp
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000080800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x80800 max_arch_pfn = 0x400000000
[    0.000000] Scanning 0 areas for low memory corruption
[    0.000000] init_memory_mapping: 0000000000000000-0000000080800000
[    0.000000] RAMDISK: 01d95000 - 02db9000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-0000000080800000
[    0.000000] Initmem setup node 0 0000000000000000-0000000080800000
[    0.000000]   NODE_DATA [000000007fffb000 - 000000007fffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00080800
[    0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 80800000 (gap: 80800000:7f800000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.1-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:6 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88007ff40000 s82752 r8192 d23744 u114688
[    0.000000] pcpu-alloc: s82752 r8192 d23744 u114688 alloc=28*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 519022
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=UUID=1e67e680-4108-4e11-8bc4-844dad448902 ro   swiotlb=force noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r ip=:127.0.255.255::::eth0:dhcp
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Placing 64MB software IO TLB between ffff880079400000 - ffff88007d400000
[    0.000000] software IO TLB at phys 0x79400000 - 0x7d400000
[    0.000000] Memory: 1961880k/2105344k available (4398k kernel code, 448k absent, 143016k reserved, 6327k data, 756k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:320 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2660.054 MHz processor.
[    0.000999] Calibrating delay loop (skipped), value calculated using timer frequency.. 5320.10 BogoMIPS (lpj=2660054)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001410] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001641] Mount-cache hash table entries: 256
[    0.002191] Initializing cgroup subsys ns
[    0.002204] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.002216] Initializing cgroup subsys cpuacct
[    0.002227] Initializing cgroup subsys freezer
[    0.002354] CPU: Unsupported number of siblings 16
[    0.002361] Performance Events: unsupported p6 CPU model 26 no PMU driver, software events only.
[    0.002524] SMP alternatives: switching to UP code
[    0.004910] cpu 0 spinlock event irq 318
[    0.005490] NMI watchdog failed to create perf event on cpu0: fffffffffffffffe
[    0.005817] installing Xen timer for CPU 1
[    0.005857] cpu 1 spinlock event irq 312
[    0.005893] SMP alternatives: switching to SMP code
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007158] NMI watchdog failed to create perf event on cpu1: fffffffffffffffe
[    0.007474] installing Xen timer for CPU 2
[    0.007585] cpu 2 spinlock event irq 306
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007909] NMI watchdog failed to create perf event on cpu2: fffffffffffffffe
[    0.008078] installing Xen timer for CPU 3
[    0.008193] cpu 3 spinlock event irq 300
[    0.000999] CPU: Unsupported number of siblings 16
[    0.008450] NMI watchdog failed to create perf event on cpu3: fffffffffffffffe
[    0.008685] installing Xen timer for CPU 4
[    0.008789] cpu 4 spinlock event irq 294
[    0.000999] CPU: Unsupported number of siblings 16
[    0.008999] NMI watchdog failed to create perf event on cpu4: fffffffffffffffe
[    0.009210] installing Xen timer for CPU 5
[    0.009252] cpu 5 spinlock event irq 288
[    0.000999] CPU: Unsupported number of siblings 16
[    0.009530] NMI watchdog failed to create perf event on cpu5: fffffffffffffffe
[    0.009541] Brought up 6 CPUs
[    0.010295] kworker/u:0 used greatest stack depth: 5696 bytes left
[    0.013138] Grant table initialized
[    0.032051] Time: 165:165:165  Date: 165/165/65
[    0.032378] NET: Registered protocol family 16
[    0.037699] PCI: setting up Xen PCI frontend stub
[    0.039132] kworker/u:0 used greatest stack depth: 5528 bytes left
[    0.067272] bio: create slab <bio-0> at 0
[    0.068203] ACPI: Interpreter disabled.
[    0.069064] xen_balloon: Initialising balloon driver.
[    0.070332] vgaarb: loaded
[    0.072248] usbcore: registered new interface driver usbfs
[    0.072279] usbcore: registered new interface driver hub
[    0.073080] usbcore: registered new device driver usb
[    0.075072] PCI: System does not support PCI
[    0.075137] PCI: System does not support PCI
[    0.076112] NetLabel: Initializing
[    0.076120] NetLabel:  domain hash size = 128
[    0.076126] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.076223] NetLabel:  unlabeled traffic allowed by default
[    0.077209] Switching to clocksource xen
[    0.093954] pnp: PnP ACPI: disabled
[    0.138850] NET: Registered protocol family 2
[    0.139329] IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.141816] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
[    0.143253] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.143474] TCP: Hash tables configured (established 262144 bind 65536)
[    0.143487] TCP reno registered
[    0.143537] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[    0.143593] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[    0.144245] NET: Registered protocol family 1
[    0.145189] RPC: Registered udp transport module.
[    0.145203] RPC: Registered tcp transport module.
[    0.145213] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.145645] Trying to unpack rootfs image as initramfs...
[    0.186054] Freeing initrd memory: 16528k freed
[    0.270158] DMA-API: preallocated 32768 debug entries
[    0.270170] DMA-API: debugging enabled by kernel config
[    0.272584] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.285338] Machine check injector initialized
[    0.294148] microcode: CPU0 sig=0x106a4, pf=0x2, revision=0xa
[    0.294217] microcode: CPU1 sig=0x106a4, pf=0x2, revision=0xa
[    0.294308] microcode: CPU2 sig=0x106a4, pf=0x2, revision=0xa
[    0.294359] microcode: CPU3 sig=0x106a4, pf=0x2, revision=0xa
[    0.294427] microcode: CPU4 sig=0x106a4, pf=0x2, revision=0xa
[    0.294473] microcode: CPU5 sig=0x106a4, pf=0x2, revision=0xa
[    0.294990] microcode: Microcode Update Driver: v2.00 <tigran@xxxxxxxxxxxxxxxxxxxx>, Peter Oruba
[    0.295008] Scanning for low memory corruption every 60 seconds
[    0.297474] audit: initializing netlink socket (disabled)
[    0.297559] type=2000 audit(1289430805.716:1): initialized
[    0.331826] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.375119] VFS: Disk quotas dquot_6.5.2
[    0.375896] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.381318] NTFS driver 2.1.29 [Flags: R/W].
[    0.383311] msgmni has been set to 3864
[    0.387906] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.387924] io scheduler noop registered
[    0.387933] io scheduler deadline registered
[    0.388724] io scheduler cfq registered (default)
[    0.390543] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.403338] pcifront pci-0: Installing PCI frontend
[    0.404082] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.406134] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.406134] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.406134] pcifront pci-0: claiming resource 0000:00:00.0/3
[    0.406134] pcifront pci-0: claiming resource 0000:00:00.0/4
[    0.406134] pcifront pci-0: claiming resource 0000:00:01.0/0
[    0.406134] pcifront pci-0: claiming resource 0000:00:01.0/2
[    0.406134] pcifront pci-0: claiming resource 0000:00:01.0/3
[    0.406134] pcifront pci-0: claiming resource 0000:00:01.0/4
[    0.406134] pcifront pci-0: claiming resource 0000:00:02.0/0
[    0.406134] pcifront pci-0: claiming resource 0000:00:02.0/2
[    0.406134] pcifront pci-0: claiming resource 0000:00:02.0/3
[    0.406134] pcifront pci-0: claiming resource 0000:00:02.0/4
[    0.406134] pcifront pci-0: claiming resource 0000:00:03.0/0
[    0.406134] pcifront pci-0: claiming resource 0000:00:03.0/2
[    0.406134] pcifront pci-0: claiming resource 0000:00:03.0/3
[    0.406134] pcifront pci-0: claiming resource 0000:00:03.0/4
[    0.734074] Non-volatile memory driver v1.3
[    0.734089] Linux agpgart interface v0.103
[    0.735535] [drm] Initialized drm 1.1.0 20060810
[    0.735556] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.829048] brd: module loaded
[    0.842342] loop: module loaded
[    0.843465] Fixed MDIO Bus: probed
[    0.845827] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.846398] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.846882] uhci_hcd: USB Universal Host Controller Interface driver
[    0.848574] usbcore: registered new interface driver usblp
[    0.849103] usbcore: registered new interface driver libusual
[    0.850786] PNP: No PS/2 controller found. Probing ports directly.
[    0.851626] i8042.c: No controller found.
[    0.852258] mice: PS/2 mouse device common for all mice
[    0.855068] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.855717] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.857602] cpuidle: using governor ladder
[    0.857619] cpuidle: using governor menu
[    0.857713] Netfilter messages via NETLINK v0.30.
[    0.857800] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.859562] ctnetlink v0.93: registering with nfnetlink.
[    0.861913] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.862342] TCP cubic registered
[    0.862354] Initializing XFRM netlink socket
[    0.864995] NET: Registered protocol family 10
[    0.869020] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.869455] IPv6 over IPv4 tunneling driver
[    0.873134] NET: Registered protocol family 17
[    0.873381] Registering the dns_resolver key type
[    0.875443] registered taskstats version 1
[    0.875615] kmemleak: Kernel memory leak detector initialized
[    0.875634] kmemleak: Automatic memory scanning thread started
[    0.877640] XENBUS: Device with no driver: device/vbd/51712
[    0.877654] XENBUS: Device with no driver: device/vif/0
[    0.877664] XENBUS: Device with no driver: device/console/0
[    0.877692]   Magic number: 1:252:3141
[   13.391082] Freeing unused kernel memory: 756k freed
[   13.391322] Write protecting the kernel read-only data: 10240k
[   13.401075] Freeing unused kernel memory: 1724k freed
[   13.402500] Freeing unused kernel memory: 1992k freed
Loading, please wait...
mount: mounting none on /dev failed: No such device
[   13.498289] all_generic_ide used greatest stack depth: 5472 bytes left
[   13.523488] udevd (1111): /proc/1111/oom_adj is deprecated, please use /proc/1111/oom_score_adj instead.
Begin: Loading essential drivers... ...
[   13.546062] udevadm used greatest stack depth: 5088 bytes left
[   13.580639] blkfront: xvda: barriers disabled
[   13.594779]  xvda: xvda1 xvda2 < xvda5 >
[   13.669872] Initialising Xen virtual ethernet driver.
[   13.672323] blkid used greatest stack depth: 4960 bytes left
[   13.756667] blkid used greatest stack depth: 4752 bytes left
Done.
Begin: Running /scripts/init-premount ...
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
Begin: Running /scripts/local-premount ...
Done.
[   13.903338] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
[   13.903363] EXT4-fs (xvda1): write access will be enabled during recovery
[   14.190114] EXT4-fs (xvda1): recovery complete
[   14.191367] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
[   14.191951] exe used greatest stack depth: 4112 bytes left
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.
init: ureadahead main process (1208) terminated with status 5
fsck from util-linux-ng 2.17.2
e2fsck 1.41.11 (14-Mar-2010)
/dev/xvda1: clean, 56107/288576 files, 296111/1153280 blocks
 * Enable login on console                                               [ OK ]
 * Check for hostname changes...                                         [ OK ]
   ...done.
 * Setting sensors limits                                                       No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
                                                                         [ OK ]
 * Initializing random number generator...                               [ OK ]
 * Setting up X server socket directory /tmp/.X11-unix...                [ OK ]
 * Setting up ICE socket directory /tmp/.ICE-unix...                     [ OK ]
 * Not starting fancontrol; run pwmconfig first.
 * Starting internet superserver inetd                                   [ OK ]
Starting openntpd: ntpd.
 * Starting network management services:                                         * Starting the system activity data collector sadc                                                                       Cannot open /var/log/sysstat/sa10: No such file or directory
                                                                         [fail]
 * Setting core file location...                                         [ OK ]
 * Restoring resolver state...                                           [ OK ]
 * Checking tomcat configuration
   ...done.
 * Starting Tomcat servlet engine tomcat6                                       Using CATALINA_BASE:   /var/lib/tomcat6
Using CATALINA_HOME:   /usr/share/tomcat6
Using CATALINA_TMPDIR: /tmp/tomcat6-tmp
Using JRE_HOME:        /usr/lib/jvm/java-6-openjdk
Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
                                                                         [ OK ]
collectd has been disabled - see /etc/default/collectd.
 * Recording successful boot for GRUB
   ...done.
 * Running local boot scripts (/etc/rc.local)                            [ OK ]
mountall: Plymouth command failed
mountall: Disconnected from Plymouth

6.0.16-01
kaan-04-dpm login: [   47.548041] ------------[ cut here ]------------
[   47.548078] kernel BUG at arch/x86/xen/mmu.c:1831!
[   47.548093] invalid opcode: 0000 [#1] SMP
[   47.548114] last sysfs file: /sys/kernel/uevent_seqnum
[   47.548124] CPU 0
[   47.548130] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[   47.548199]
[   47.548208] Pid: 3815, comm: java Not tainted 2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[   47.548222] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   47.548245] RSP: e02b:ffff880077fadce8  EFLAGS: 00010282
[   47.548255] RAX: 00000000ffffffea RBX: 0000000000078312 RCX: 0000000000000001
[   47.548266] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[   47.548278] RBP: ffff880077fadd08 R08: 00003ffffffff000 R09: ffff880000000000
[   47.548290] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[   47.548302] R13: ffff880077f5e838 R14: ffff88007629ab80 R15: ffff880078288750
[   47.548319] FS:  00007f2e8058c710(0000) GS:ffff88007ff40000(0000) knlGS:0000000000000000
[   47.548332] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   47.548342] CR2: 00007f2e60fe8728 CR3: 0000000077ec9000 CR4: 0000000000002660
[   47.548354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   47.548366] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   47.548379] Process java (pid: 3815, threadinfo ffff880077fac000, task ffff880078288750)
[   47.548391] Stack:
[   47.548398]  ffff880000000000 00000000000476fc ffff88007603ea00 0000000000078312
[   47.548426]  ffff880077fadd28 ffffffff810066a8 0000000000078312 ffff88007603ea00
[   47.548454]  ffff880077fadd38 ffffffff810066db ffff880077fadd78 ffffffff810e048f
[   47.548481] Call Trace:
[   47.548494]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[   47.548509]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[   47.548525]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[   47.548539]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
[   47.548554]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
[   47.548571]  [<ffffffff81442ef2>] ? _raw_spin_unlock_irqrestore+0x19/0x1c
[   47.548588]  [<ffffffff81048583>] ? try_to_wake_up+0x207/0x219
[   47.548603]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[   47.548617]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
[   47.548632]  [<ffffffff81442ef2>] ? _raw_spin_unlock_irqrestore+0x19/0x1c
[   47.548648]  [<ffffffff81202bce>] ? rwsem_wake+0x39/0x43
[   47.548661]  [<ffffffff81206bc7>] ? call_rwsem_wake+0x17/0x30
[   47.548677]  [<ffffffff8106aa3d>] ? up_write+0x18/0x1a
[   47.548690]  [<ffffffff810e7378>] ? sys_mmap_pgoff+0x126/0x159
[   47.548705]  [<ffffffff81443615>] page_fault+0x25/0x30
[   47.548715] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48 8b 3d
[   47.548794] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   47.548794]  RSP <ffff880077fadce8>
[   47.548794] ---[ end trace ceae70ee7c69e40a ]---
[   63.942338] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[   92.806778] ------------[ cut here ]------------
[   92.806802] kernel BUG at arch/x86/xen/mmu.c:1831!
[   92.806817] invalid opcode: 0000 [#2] SMP
[   92.806841] last sysfs file: /sys/kernel/uevent_seqnum
[   92.806856] CPU 0
[   92.806865] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[   92.806967]
[   92.806981] Pid: 3914, comm: rsyslogd Tainted: G      D     2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[   92.807005] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   92.807036] RSP: e02b:ffff880077869ce8  EFLAGS: 00010282
[   92.807053] RAX: 00000000ffffffea RBX: 0000000000077c61 RCX: 0000000000000001
[   92.807063] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[   92.807063] RBP: ffff880077869d08 R08: 00003ffffffff000 R09: ffff880000000000
[   92.807063] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[   92.807063] R13: ffff8800760e44e0 R14: ffff880075ce8958 R15: ffff880077015f10
[   92.807063] FS:  00007ffd53e39700(0000) GS:ffff88007ff40000(0000) knlGS:0000000000000000
[   92.807063] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   92.807063] CR2: 00007ffd53801330 CR3: 000000007821e000 CR4: 0000000000002660
[   92.807063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   92.807063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   92.807063] Process rsyslogd (pid: 3914, threadinfo ffff880077868000, task ffff880077015f10)
[   92.807063] Stack:
[   92.807063]  ffff880000000000 0000000000047dad ffff880076138700 0000000000077c61
[   92.807063]  ffff880077869d28 ffffffff810066a8 0000000000077c61 ffff880076138700
[   92.807063]  ffff880077869d38 ffffffff810066db ffff880077869d78 ffffffff810e048f
[   92.807063] Call Trace:
[   92.807063]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[   92.807063]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[   92.807063]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[   92.807063]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
[   92.807063]  [<ffffffff8102b78a>] ? pvclock_clocksource_read+0x4b/0xb4
[   92.807063]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[   92.807063]  [<ffffffff81007bc9>] ? xen_spin_unlock+0x11/0x30
[   92.807063]  [<ffffffff8105b253>] ? do_sigaction+0x175/0x18d
[   92.807063]  [<ffffffff811039a7>] ? fput+0x1ed/0x1fc
[   92.807063]  [<ffffffff8105d855>] ? sys_rt_sigaction+0x5e/0x9d
[   92.807063]  [<ffffffff81443615>] page_fault+0x25/0x30
[   92.807063] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48 8b 3d
[   92.807063] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   92.807063]  RSP <ffff880077869ce8>
[   92.807063] ---[ end trace ceae70ee7c69e40b ]---
[  693.101054] kmemleak: 86 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[  695.855546] ------------[ cut here ]------------
[  695.855557] kernel BUG at arch/x86/xen/mmu.c:1831!
[  695.855563] invalid opcode: 0000 [#3] SMP
[  695.855574] last sysfs file: /sys/kernel/uevent_seqnum
[  695.855581] CPU 1
[  695.855585] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[  695.855629]
[  695.855635] Pid: 3946, comm: cron Tainted: G      D     2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[  695.855645] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[  695.855664] RSP: e02b:ffff880067a29ce8  EFLAGS: 00010282
[  695.855671] RAX: 00000000ffffffea RBX: 0000000000077c58 RCX: 0000000000000001
[  695.855680] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[  695.855689] RBP: ffff880067a29d08 R08: 00003ffffffff000 R09: ffff880000000000
[  695.855696] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[  695.855703] R13: ffff880067a2cb40 R14: ffff880077c7e678 R15: ffff8800682395f0
[  695.855716] FS:  00007fa1afde67a0(0000) GS:ffff88007ff5c000(0000) knlGS:0000000000000000
[  695.855725] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  695.855734] CR2: 00007fa1ad1ac758 CR3: 0000000067890000 CR4: 0000000000002660
[  695.855742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  695.855749] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  695.855759] Process cron (pid: 3946, threadinfo ffff880067a28000, task ffff8800682395f0)
[  695.855767] Stack:
[  695.855774]  ffff880000000000 0000000000047db6 ffff8800760e3480 0000000000077c58
[  695.855793]  ffff880067a29d28 ffffffff810066a8 0000000000077c58 ffff8800760e3480
[  695.855814]  ffff880067a29d38 ffffffff810066db ffff880067a29d78 ffffffff810e048f
[  695.855834] Call Trace:
[  695.855842]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[  695.855852]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[  695.855865]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[  695.855877]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
[  695.855887]  [<ffffffff810e6e36>] ? mmap_region+0x391/0x4bf
[  695.855900]  [<ffffffff8100eb84>] ? arch_get_unmapped_area_topdown+0x4a/0x28a
[  695.855915]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[  695.855925]  [<ffffffff8102b78a>] ? pvclock_clocksource_read+0x4b/0xb4
[  695.855937]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
[  695.855949]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
[  695.855959]  [<ffffffff81443615>] page_fault+0x25/0x30
[  695.855967] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48 8b 3d
[  695.856056] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[  695.856056]  RSP <ffff880067a29ce8>
[  695.856056] ---[ end trace ceae70ee7c69e40c ]---
[  696.704202] cron[3928] general protection ip:7fa1af4ce588 sp:7ffff1875200 error:0 in libc-2.11.1.so[7fa1af424000+17a000]
[  697.853416] ------------[ cut here ]------------
[  697.853427] kernel BUG at arch/x86/xen/mmu.c:1831!
[  697.853433] invalid opcode: 0000 [#4] SMP
[  697.853443] last sysfs file: /sys/kernel/uevent_seqnum
[  697.853450] CPU 4
[  697.853453] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[  697.853497]
[  697.853503] Pid: 3947, comm: sh Tainted: G      D     2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[  697.853512] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[  697.853526] RSP: e02b:ffff880067aa5b38  EFLAGS: 00010282
[  697.853532] RAX: 00000000ffffffea RBX: 0000000000075203 RCX: 0000000000000001
[  697.853540] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[  697.853547] RBP: ffff880067aa5b58 R08: 00003ffffffff000 R09: ffff880000000000
[  697.853554] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[  697.853561] R13: ffff880075204000 R14: ffff880075204000 R15: ffff880078a26ff8
[  697.853571] FS:  00007fa1afde67a0(0000) GS:ffff88007ffb0000(0000) knlGS:0000000000000000
[  697.853580] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  697.853586] CR2: 00007fa1af7bd560 CR3: 0000000077a98000 CR4: 0000000000002660
[  697.853594] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  697.853602] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  697.853609] Process sh (pid: 3947, threadinfo ffff880067aa4000, task ffff880068239d40)
[  697.853616] Stack:
[  697.853621]  ffff880000000000 000000000004a80b ffff8800760e1500 0000000000075203
[  697.853640]  ffff880067aa5b78 ffffffff810066a8 0000000000075203 ffff8800760e1500
[  697.853658]  ffff880067aa5b88 ffffffff810066db ffff880067aa5bc8 ffffffff810e048f
[  697.853676] Call Trace:
[  697.853685]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[  697.853693]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[  697.853704]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[  697.853713]  [<ffffffff810e7faa>] move_page_tables+0x1cb/0x4c1
[  697.853722]  [<ffffffff81007bc9>] ? xen_spin_unlock+0x11/0x30
[  697.853732]  [<ffffffff8110870a>] setup_arg_pages+0x1cb/0x355
[  697.853743]  [<ffffffff8113ce7a>] load_elf_binary+0x418/0x16ec
[  697.853752]  [<ffffffff8113b319>] ? load_misc_binary+0x5c/0x33c
[  697.853761]  [<ffffffff810e3336>] ? get_user_pages+0x44/0x46
[  697.853772]  [<ffffffff81107355>] ? get_arg_page+0x50/0xa7
[  697.853781]  [<ffffffff81107793>] search_binary_handler+0xbb/0x24d
[  697.853790]  [<ffffffff8113ca62>] ? load_elf_binary+0x0/0x16ec
[  697.853799]  [<ffffffff81108e2d>] do_execve+0x19e/0x28d
[  697.853809]  [<ffffffff810118dd>] sys_execve+0x3e/0x55
[  697.853818]  [<ffffffff8100af1c>] stub_execve+0x6c/0xc0
[  697.853824] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48 8b 3d
[  697.854013] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[  697.854024]  RSP <ffff880067aa5b38>
[  697.854031] ---[ end trace ceae70ee7c69e40d ]---
[  755.706246] BUG: unable to handle kernel paging request at ffff880075202b80
[  755.706263] IP: [<ffffffff810e24b3>] do_wp_page+0x58b/0x6be
[  755.706278] PGD 1a04067 PUD 39d9067 PMD 3b83067 PTE 8010000075202065
[  755.706294] Oops: 0003 [#5] SMP
[  755.706305] last sysfs file: /sys/kernel/uevent_seqnum
[  755.706312] CPU 4
[  755.706316] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[  755.706364]
[  755.706370] Pid: 3948, comm: cron Tainted: G      D     2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[  755.706379] RIP: e030:[<ffffffff810e24b3>]  [<ffffffff810e24b3>] do_wp_page+0x58b/0x6be
[  755.706391] RSP: e02b:ffff880067439ce8  EFLAGS: 00010246
[  755.706397] RAX: 0000000000000000 RBX: ffffea00016f7d98 RCX: 0000000000000001
[  755.706404] RDX: 6db6db6db6db6db7 RSI: ffffffff81b460c0 RDI: ffff8800760a1f80
[  755.706411] RBP: ffff880067439d78 R08: 0000000000000000 R09: 0000000000000041
[  755.706419] R10: 0000000000000100 R11: ffff880078277030 R12: 80000000407c3067
[  755.706426] R13: ffffea0001bd0068 R14: ffff880075202b80 R15: ffff880067a77508
[  755.706436] FS:  00007fa1afde67a0(0000) GS:ffff88007ffb0000(0000) knlGS:0000000000000000
[  755.706444] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  755.706451] CR2: ffff880075202b80 CR3: 0000000077ea9000 CR4: 0000000000002660
[  755.706458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  755.706466] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  755.706474] Process cron (pid: 3948, threadinfo ffff880067438000, task ffff88006823a490)
[  755.706481] Stack:
[  755.706486]  ffff880067439d08 ffffffff81007c09 ffffea000199f080 0000000000000000
[  755.706504]  01ff880067439d58 ffffea000199f080 ffff8800760a1f80 0000000000d703b0
[  755.706523]  00007fa1afde6a70 ffff880078277030 ffff880067a77508 ffff8800760a1f80
[  755.706542] Call Trace:
[  755.706552]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
[  755.706562]  [<ffffffff810e2e60>] handle_mm_fault+0x87a/0x8c1
[  755.706571]  [<ffffffff8103ed4d>] ? enqueue_entity+0x2ac/0x2b8
[  755.706582]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[  755.706590]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
[  755.706600]  [<ffffffff81004274>] ? xen_mc_flush+0x1cd/0x1f2
[  755.706609]  [<ffffffff8103ed4d>] ? enqueue_entity+0x2ac/0x2b8
[  755.706618]  [<ffffffff81007bc9>] ? xen_spin_unlock+0x11/0x30
[  755.706628]  [<ffffffff81045b20>] ? finish_task_switch+0x4f/0x94
[  755.706637]  [<ffffffff81443615>] page_fault+0x25/0x30
[  755.706644] Code: 48 0f af fa 48 c1 e7 0c 48 09 c7 ff 14 25 28 13 a1 81 49 89 c4 48 83 c8 42 49 8b 3f 49 83 cc 40 41 f6 47 30 02 4c 0f 45 e0 31 c0 <49> 87 06 48 8b 75 a8 4c 89 f2 0f 1f 80 00 00 00 00 48 8b 75 a8
[  755.706830] RIP  [<ffffffff810e24b3>] do_wp_page+0x58b/0x6be
[  755.706841]  RSP <ffff880067439ce8>
[  755.706847] CR2: ffff880075202b80
[  755.706854] ---[ end trace ceae70ee7c69e40e ]---

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
<Prev in Thread] Current Thread [Next in Thread>