This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split

To: Juergen Gross <juergen.gross@xxxxxxxxxxxxxx>
Subject: Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split
From: Andre Przywara <andre.przywara@xxxxxxx>
Date: Fri, 28 Jan 2011 12:07:09 +0100
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>, Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Delivery-date: Fri, 28 Jan 2011 03:08:41 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4D426673.7020200@xxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4D41FD3A.5090506@xxxxxxx> <4D426673.7020200@xxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird (X11/20090820)
Juergen Gross wrote:
On 01/28/11 00:18, Andre Przywara wrote:

when I boot my machine without restricting Dom0 (dom0_mem=
dom0_max_vcpus=) I get an _hypervisor_ crash when I run
# xl cpupool-numa-split
If Dom0's resources are limited on the Xen cmdline, everything works fine.
The crashdump points to a scheduling problem with weights, so I assume
the NUMA distribution algorithm some fools the hypervisor completely.

I will investigate this further tomorrow, but maybe someone has some
good idea.

I've seen this once with an older cpupool version on a 24 processor machine.
It was NOT related to NUMA, but did occur only on reboot after a Dom0 panic.
The machine had an init script creating a cpupool and populating it with
cpus. The machine was in a panic loop due to the BUG in sched_acct then until
it was resetted manually. After the reset the problem was gone.

As I was never able to reproduce the problem later (the same software is
running on dozens of machines!), I assumed there was a problem related to
the first Dom0 panic, may be some destroyed BIOS tables.

Can the crash be reproduced easily?
If I don't specify dom0_max_vcpus= and dom0_mem= on the Xen cmdline, I can reliably trigger the crash with xl cpupool-numa-split.
Omitting dom0_max_vcpus only does not suffice.

Will continue after lunch-break ;-)




root@dosorca:/data/images# xl cpupool-numa-split
(XEN) Xen BUG at sched_credit.c:990
(XEN) ----[ Xen-4.1.0-rc2-pre x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff82c4801180f8>] csched_acct+0x11f/0x419
(XEN) RFLAGS: 0000000000010006 CONTEXT: hypervisor
(XEN) rax: 0000000000000010 rbx: 0000000000000f00 rcx: 0000000000000100
(XEN) rdx: 0000000000001000 rsi: ffff830437ffa600 rdi: 0000000000000010
(XEN) rbp: ffff82c480297e10 rsp: ffff82c480297d80 r8: 0000000000000100
(XEN) r9: 0000000000000006 r10: ffff82c4802d4100 r11: 000000afc7df0edf
(XEN) r12: ffff830437ffa5e0 r13: ffff82c480117fd9 r14: ffff830437f9f2e8
(XEN) r15: ffff830434321ec0 cr0: 000000008005003b cr4: 00000000000006f0
(XEN) cr3: 000000080df4e000 cr2: ffff88179af79618
(XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008
(XEN) Xen stack trace from rsp=ffff82c480297d80:
(XEN) 0000000000000282 fffffed4802d3f80 0000000000000eff ffff830437ffa5e0
(XEN) ffff830437ffa5e8 ffff830437ffa870 ffff830437ffa5e0 0000000000000282
(XEN) ffff830437ffa5e8 00002a3037ffa870 00000f0000000f00 0000000000000000
(XEN) ffff82c400000000 ffff82c4802d3f80 ffff830437ffa5e0 ffff82c480117fd9
(XEN) ffff830437f9f2e8 ffff830437f9f2e0 ffff82c480297e40 ffff82c480125f34
(XEN) 0000000000000002 ffff830437ffa600 ffff82c4802d3f80 000000afb6f8667f
(XEN) ffff82c480297e90 ffff82c480126259 ffff82c48024ae20 ffff82c4802d3f80
(XEN) ffff830437f9f2e0 0000000000000000 0000000000000000 ffff82c4802b0880
(XEN) ffff82c480297f18 ffffffffffffffff ffff82c480297ed0 ffff82c480123327
(XEN) ffff82c4802d4a00 ffff82c480297f18 ffff82c48024ae20 ffff82c480297f18
(XEN) 000000afb6abd652 ffff82c4802d3ec0 ffff82c480297ee0 ffff82c4801233a2
(XEN) ffff82c480297f10 ffff82c4801563f5 0000000000000000 ffff8300c7cd6000
(XEN) 0000000000000000 ffff8300c7ad4000 ffff82c480297d48 0000000000000000
(XEN) 0000000000000000 0000000000000000 ffffffff81a69060 ffff8817a8503f10
(XEN) ffff8817a8503fd8 0000000000000246 ffff8817a8503e80 ffff880000000001
(XEN) 0000000000000000 0000000000000000 ffffffff810093aa 000000aafab2f86e
(XEN) 00000000deadbeef 00000000deadbeef 0000010000000000 ffffffff810093aa
(XEN) 000000000000e033 0000000000000246 ffff8817a8503ef8 000000000000e02b
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 ffff8300c7cd6000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82c4801180f8>] csched_acct+0x11f/0x419
(XEN) [<ffff82c480125f34>] execute_timer+0x4e/0x6c
(XEN) [<ffff82c480126259>] timer_softirq_action+0xf2/0x245
(XEN) [<ffff82c480123327>] __do_softirq+0x88/0x99
(XEN) [<ffff82c4801233a2>] do_softirq+0x6a/0x7a
(XEN) [<ffff82c4801563f5>] idle_loop+0x6a/0x6f
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at sched_credit.c:990
(XEN) ****************************************
(XEN) Reboot in five seconds...

Andre Przywara
AMD-OSRC (Dresden)
Tel: x29712

Xen-devel mailing list