Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split

To:	Andre Przywara <andre.przywara@xxxxxxx>
Subject:	Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split
From:	Juergen Gross <juergen.gross@xxxxxxxxxxxxxx>
Date:	Mon, 07 Feb 2011 14:32:02 +0100
Cc:	George Dunlap <George.Dunlap@xxxxxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, "Diestelhorst, Stephan" <Stephan.Diestelhorst@xxxxxxx>
Delivery-date:	Mon, 07 Feb 2011 05:32:37 -0800
Dkim-signature:	v=1; a=rsa-sha256; c=simple/simple; d=ts.fujitsu.com; i=juergen.gross@xxxxxxxxxxxxxx; q=dns/txt; s=s1536b; t=1297085525; x=1328621525; h=message-id:date:from:mime-version:to:cc:subject: references:in-reply-to; z=Message-ID:=20<4D4FF452.6060508@xxxxxxxxxxxxxx>\|Date:=20 Mon,=2007=20Feb=202011=2014:32:02=20+0100\|From:=20Juergen =20Gross=20<juergen.gross@xxxxxxxxxxxxxx>\|MIME-Version: =201.0\|To:=20Andre=20Przywara=20<andre.przywara@xxxxxxx> \|CC:=20George=20Dunlap=20<George.Dunlap@xxxxxxxxxxxxx>, =20=0D=0A=20"xen-devel@xxxxxxxxxxxxxxxxxxx"=20<xen-devel@ lists.xensource.com>,=0D=0A=20"Diestelhorst,=20Stephan" =20<Stephan.Diestelhorst@xxxxxxx>\|Subject:=20Re:=20[Xen-d evel]=20Hypervisor=20crash(!)=20on=20xl=20cpupool-numa-sp lit\|References:=20<4D41FD3A.5090506@xxxxxxx>=09<201102021 539.06664.stephan.diestelhorst@xxxxxxx>=09<4D4974D1.10805 03@xxxxxxxxxxxxxx>=09<201102021701.05665.stephan.diestelh orst@xxxxxxx>=09<4D4A43B7.5040707@xxxxxxxxxxxxxx>=09<4D4A 72D8.3020502@xxxxxxxxxxxxxx>=20<4D4C08B6.30600@xxxxxxx> =20<4D4FE7E2.9070605@xxxxxxx>\|In-Reply-To:=20<4D4FE7E2.90 70605@xxxxxxx>; bh=xxJZSHFIJTa35/9TudIqMUKpopawvYDvZKEY51/gJ2U=; b=osWvlJ093r03DWojs0QrjpE2JOAKnZ7GWicVwoYtUiyaaMeL51+JQTb8 sQX5qQ5l5YFvnSssLijK5njUk2M2rs9Wxg/R/NqFiphpBCNvKzwqHc4fK BJLx8O7lswJAgdLtta1e7cFbpjKrQ1VsIZFU0AkHjz7urhd+D1kCNm29D x8+EyJGk9LpYiDWe/03E0kgtvbSzXYh/oi3A2xPzvej3GpKLDsDELSe/2 j3ciknncLmAJ/WKV+vZYQFV2Gg2go;
Domainkey-signature:	s=s1536a; d=ts.fujitsu.com; c=nofws; q=dns; h=X-SBRSScore:X-IronPort-AV:Received:X-IronPort-AV: Received:Received:Message-ID:Date:From:Organization: User-Agent:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type; b=UaCV4NG3EUCnrYwfJKkejdmRWKCC2REdz+nW2FOt2ZyJrmLuuPD1JYU4 CTboE7/gfyWvEuz/TXRini3fjvrU8KCuGAxtqtc5Kf+uKSXnJwUhchROL z35wUlkZABmfLXUNPMwJgOI689Yv4ogHuJRAkbhS+MU3JtpV2XDBk5P6G F+ynrhdNZGWwoBwyBr0NjK+XqeFrt0LW8PYgu4IpCJ75FFj5NPEEnVGhf nNSHrfQ32LSkVSCJrpmXlhF8IQ+z7;
Envelope-to:	www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to:	<4D4FE7E2.9070605@xxxxxxx>
List-help:	<mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id:	Xen developer discussion <xen-devel.lists.xensource.com>
List-post:	<mailto:xen-devel@lists.xensource.com>
List-subscribe:	<http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe:	<http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Organization:	Fujitsu Technology Solutions
References:	<4D41FD3A.5090506@xxxxxxx> <201102021539.06664.stephan.diestelhorst@xxxxxxx> <4D4974D1.1080503@xxxxxxxxxxxxxx> <201102021701.05665.stephan.diestelhorst@xxxxxxx> <4D4A43B7.5040707@xxxxxxxxxxxxxx> <4D4A72D8.3020502@xxxxxxxxxxxxxx> <4D4C08B6.30600@xxxxxxx> <4D4FE7E2.9070605@xxxxxxx>
Sender:	xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent:	Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.16) Gecko/20101226 Iceowl/1.0b1 Icedove/3.0.11

On 02/07/11 13:38, Andre Przywara wrote:

Juergen,

as promised some more debug data. This is from c/s 22858 with Stephans
debug patch (attached).
We get the following dump when the hypervisor crashes, note that the
first lock is different from the second and subsequent ones:

(XEN) sched_credit.c, 572: prv: ffff831836df2970 &prv->lock:
ffff831836df2970 prv->weight: 256 sdom->active_vcpu_count: 3
sdom->weight: 256
(XEN) sched_credit.c, 572: prv: ffff830437ffa5e0 &prv->lock:
ffff830437ffa5e0 prv->weight: 768 sdom->active_vcpu_count: 4
sdom->weight: 256
(XEN) sched_credit.c, 572: prv: ffff830437ffa5e0 &prv->lock:
ffff830437ffa5e0 prv->weight: 1024 sdom->active_vcpu_count: 5
sdom->weight: 256
(XEN) sched_credit.c, 572: prv: ffff830437ffa5e0 &prv->lock:
ffff830437ffa5e0 prv->weight: 1280 sdom->active_vcpu_count: 6
sdom->weight: 256

....

Hope that gives you an idea. I attach the whole log for your reference.


Hmm, could it be your log wasn't created with the attached patch? I'm missing
Dom-Id and VCPU from the printk() above, which would be interesting (at least
I hope so)...
Additionally printing the local pcpu number would help, too.
And could you add a printk for the new prv address in csched_init()?

It would be nice if you could enable cpupool diag output. Please use the
attached patch (includes the previous patch for executing the cpu move on the
cpu to be moved, plus some diag printk corrections).


Juergen

--
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@xxxxxxxxxxxxxx
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

diag.patch
Description: Text Data

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

WARNING - OLD ARCHIVES

xen-devel

Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split