WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split

To: Stephan Diestelhorst <stephan.diestelhorst@xxxxxxx>
Subject: Re: [Xen-devel] Hypervisor crash(!) on xl cpupool-numa-split
From: Juergen Gross <juergen.gross@xxxxxxxxxxxxxx>
Date: Thu, 03 Feb 2011 06:57:11 +0100
Cc: George Dunlap <George.Dunlap@xxxxxxxxxxxxx>, "Przywara, Andre" <Andre.Przywara@xxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Keir Fraser <keir@xxxxxxx>, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
Delivery-date: Wed, 02 Feb 2011 21:58:38 -0800
Dkim-signature: v=1; a=rsa-sha256; c=simple/simple; d=ts.fujitsu.com; i=juergen.gross@xxxxxxxxxxxxxx; q=dns/txt; s=s1536b; t=1296712634; x=1328248634; h=message-id:date:from:mime-version:to:cc:subject: references:in-reply-to:content-transfer-encoding; z=Message-ID:=20<4D4A43B7.5040707@xxxxxxxxxxxxxx>|Date:=20 Thu,=2003=20Feb=202011=2006:57:11=20+0100|From:=20Juergen =20Gross=20<juergen.gross@xxxxxxxxxxxxxx>|MIME-Version: =201.0|To:=20Stephan=20Diestelhorst=20<stephan.diestelhor st@xxxxxxx>|CC:=20George=20Dunlap=20<George.Dunlap@xxxxxx rix.com>,=20=0D=0A=20"Przywara,=20Andre"=20<Andre.Przywar a@xxxxxxx>,=0D=0A=20Keir=20Fraser=20<keir@xxxxxxx>,=20=0D =0A=20"xen-devel@xxxxxxxxxxxxxxxxxxx"=20<xen-devel@lists. xensource.com>,=0D=0A=20Ian=20Jackson=20<Ian.Jackson@xxxx itrix.com>|Subject:=20Re:=20[Xen-devel]=20Hypervisor=20cr ash(!)=20on=20xl=20cpupool-numa-split|References:=20<4D41 FD3A.5090506@xxxxxxx>=09<201102021539.06664.stephan.diest elhorst@xxxxxxx>=09<4D4974D1.1080503@xxxxxxxxxxxxxx>=20<2 01102021701.05665.stephan.diestelhorst@xxxxxxx> |In-Reply-To:=20<201102021701.05665.stephan.diestelhorst@ amd.com>|Content-Transfer-Encoding:=207bit; bh=CmUaqmg/O+hh7Oqimm+GTBAVOs8SSmQmSLcbjVxgTkE=; b=KmuKvIcIng6TbwnLSC/+/n2b5Wooah2TgZ7bl5H7qcbFhf6V2xFAxrPS rMOv2D/P5Yd5FcofAezjgii5mlKywleUGM2Mvey0VM6xccZ6+9QK/l9QL WGizV0eeKqtGYc0EiunTmwe3ujL5JF52Kngv4asJ+yDziZFOUP633Ab5f 9YEgLxb9SuaCj9mWRIdbVGzOtSMqiZVAEdJIrRuLwga09dV/gTuFZd0b1 L7AHP8OA6ACMbfv/yXxvKrKsiI1KL;
Domainkey-signature: s=s1536a; d=ts.fujitsu.com; c=nofws; q=dns; h=X-SBRSScore:X-IronPort-AV:Received:X-IronPort-AV: Received:Received:Message-ID:Date:From:Organization: User-Agent:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=ckctJYYyvc2uRCuHZFbKlScDd5vM4yP9IcHnLwmE5W1we4FKQ71Js74C MTHYDereyFSdhRYMDV1ovp3xhZoplpksICaQc9UHy62v58pyva7ymDwrg cWiiEVUncSw8497TeOESiPlDqTwsCiwsixwnVpCHwCQvsUcftkcbxTeWq 6gmoVaLRAYJjIZ5HxQ4SQQ//Tm4u5zjIcWPe3UjiF/wrStjQ9rA3N6y8F Xahki+jqFC0drSf6uKx0ZOHnVSkjm;
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <201102021701.05665.stephan.diestelhorst@xxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Organization: Fujitsu Technology Solutions
References: <4D41FD3A.5090506@xxxxxxx> <201102021539.06664.stephan.diestelhorst@xxxxxxx> <4D4974D1.1080503@xxxxxxxxxxxxxx> <201102021701.05665.stephan.diestelhorst@xxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.16) Gecko/20101226 Iceowl/1.0b1 Icedove/3.0.11
On 02/02/11 17:01, Stephan Diestelhorst wrote:
On Wednesday 02 February 2011 16:14:25 Juergen Gross wrote:
On 02/02/11 15:39, Stephan Diestelhorst wrote:
We have the following theory of what happens:
* some vcpus of a particular domain are currently in the process of
    being moved to the new pool

The only _vcpus_ to be moved between pools are the idle vcpus. And those
never contribute to accounting in credit scheduler.

We are moving _pcpus_ only (well, moving a domain between pools actually
moves vcpus as well, but then the domain is paused).

How do you ensure that the domain is paused and stays that way? Pausing
the domain was what I had in mind, too...

Look at sched_move_domain() in schedule.c: I'm calling domain_pause()
before moving the vcpus and domain_unpause() after that.


Despite the rant, it is amazing to see the ability to move running
things around through this remote continuation trick! In my (ancient)
balancer experiments I added hypervisor-threads just for side-
stepping this issue..

I think the easiest way to solve the problem would be to move the cpu to the
new pool in a tasklet. This is possible now, because tasklets are always
executed in the idle vcpus.

Yep. That was exactly what I build. At the time stuff like that did
not exist (2005).

OTOH I'd like to understand what is wrong with my current approach...

Nothing, in fact I like it. In my rant I complained about the fact
that splitting the critical section accross this continuation looks
scary, basically causing some generic red lights to turn on :-) And
making reasoning about the correctness a little complicated, but that
may well be a local issue ;-)

Perhaps you can help solving the miracle:

Could you replace the BUG_ON in sched_credit.c:389 with something like this:

if (!is_idle_vcpu(per_cpu(schedule_data, cpu).curr)) {
  extern void dump_runq(unsigned char key);
  struct vcpu *vc = per_cpu(schedule_data, cpu).curr;

  printk("+++ (%d.%d) instead idle vcpu on cpu %d\n", vc->domain->domain_id,
          vc->vcpu_id, cpu);
  dump_runq('q');
  BUG();
}


Juergen

--
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@xxxxxxxxxxxxxx
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>