This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] Xen 3.4.1 NUMA support

To: Dulloor <dulloor@xxxxxxxxx>
Subject: Re: [Xen-devel] Xen 3.4.1 NUMA support
From: George Dunlap <George.Dunlap@xxxxxxxxxxxxx>
Date: Mon, 9 Nov 2009 12:29:42 +0000
Cc: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Keir Fraser <Keir.Fraser@xxxxxxxxxxxxx>, Papagiannis Anastasios <apapag@xxxxxxxxxxxx>
Delivery-date: Mon, 09 Nov 2009 04:30:02 -0800
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type; bh=9RbJdMZp/+0z36V8EbsJO9fwIv7B0Uw1z/hJpDTb5jw=; b=VjzuATYIQSaemBp1CeYabXC865FtUCWhaEpnSf0BdU4rv3NKJwwtiH6hxh4tuTwaMm KaEM1UNem8untAGzijE48OhoCOTW5XvfvsNnOmWm3trOW3XRKVYvRnwPS4U14pgDbYUu 1GQGLrmQX5T9X+R1TooMMWOR6TBcIceOCjFsA=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=mk8wM04t4GFmAUzvJ5rnoSLGgZhfN66EKb4oJ1yqjB393ICfP6FDxY94H4cOSLDEO0 1Aly4t9hSmA1C0eHhd/VWKRT8tQw0jBoeq+A6Y6bhZcyCWs2/r4No6S1Sc3NfjuTMMxC 3Eux+pTP8wQMJ6EqjxiLNBhFrr4WaSDtI4cas=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <940bcfd20911090339m199e3893ke8e855e2c0ed3b9b@xxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <bd4f4a54-5269-42d8-b16d-cbdfaeeba361@default> <4AF7FE15.6070503@xxxxxxxxxxxxx> <940bcfd20911090339m199e3893ke8e855e2c0ed3b9b@xxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
On Mon, Nov 9, 2009 at 11:39 AM, Dulloor <dulloor@xxxxxxxxx> wrote:
> What's the current scope and status of your scheduler work ? Is it
> going to look similar to the Linux scheduler (with scheduling domains,
> et al). In that case, topology is already accounted for, to a large
> extent. It would be good to know so that I can work on something that
> doesn't overlap.

My plan was to do something similar to Linux, but with this
difference: Instead of having one runqueue per logical processor (as
both Xen and Linux currently do), and having "domains" all the way up
(as Linux currently does), I had planned on having one runqueue per L2
processor cache.  The main reason to avoid migration is to preserve a
warm cache; but since L1's are replaced so quickly, there should be
little impact to a VM migrating between different threads and cores
which share the same L2.

Above the L2s I was planning on having an idea similar to the Linux
"domains" (although obviously it would need a different name to avoid
confusion), and doing explicit load-balancing between them.  But as I
have not had a chance to test this kind of load balancing yet, the
plan may change somewhate before then.

Problems to solve wrt NUMA, as I understand it, are to balance the
performance cost of sharing a busy local CPU, vs the performance cost
of non-local memory accesses.  This would involve adding the NUMA
logic to the load balancing algorithm.  Which I guess would depend in
part on having a load balancing algorithm to begin with. :-)

Once I have the basic credit patches in working order, would you be
interested in working on the load-balancing between runqueues?  I can
then work on further testing of the credit algorithm.  My ultimate
goal would be to have a basic regression test that people could use to
measure how their changes to the scheduler affect a wide variety of


Xen-devel mailing list