This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] comment request: dom0 dma on large memory systems

To: "Tian, Kevin" <kevin.tian@xxxxxxxxx>
Subject: Re: [Xen-devel] comment request: dom0 dma on large memory systems
From: "Scott Parish" <srparish@xxxxxxxxxx>
Date: Sat, 4 Jun 2005 03:59:53 +0000
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, Scott Parish <srparish@xxxxxxxxxx>
Delivery-date: Sat, 04 Jun 2005 04:39:51 +0000
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <571ACEFD467F7749BC50E0A98C17CDD806E9F77F@pdsmsx403>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <571ACEFD467F7749BC50E0A98C17CDD806E9F77F@pdsmsx403>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/
On Sat, Jun 04, 2005 at 11:48:16AM +0800, Tian, Kevin wrote:

> ----Original Message-----
> >From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
> >[mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Scott
> Parish
> >Sent: Friday, June 03, 2005 3:35 PM
> >
> >On x86_64 with 6gig ram, dom0's initial allocation is from memory
> >above the pci hole (referred to as "high memory" in this email) if
> >dom0_mem is set to 2g or higher. The only problem is that most io/dma
> >devices (non-dac) can only dma to the first 32bits worth of machine
> >addresses--thus for some configurations, dom0 has no memory which is
> >dma-able.
> IIRC, 2 or 3 months ago, Keir said that default memory allocation for
> Dom0 is all available memory. And then CP has to decrease by balloon
> interface before creating other domains. If this still holds true, I'm
> not sure whether above problem still exists, since all avail memory
> including both <4G and >4G belonging to Dom0 then. (XEN itself only
> consumes a small trunk). However after looking at your patch and then
> the source, it seems that only the max available order, meaning must be
> continuous, is allocated to Dom0 currently. So did I misunderstand this
> concept? If it really only means maximum continuous trunk, then you
> patch definitely shoots straight on the real problem on all 64bit
> platform. ;-)

Right, there are several hacks around this problem, a couple i've
thought of are:

  + enforce dom0 take all memory
  + drop the max order size for MEMZONEs to 18 (in which case
    alloc_largest should always allocate from the lower memory)
  + prealloc X amount of low memory (128M for instance) and add
    it into the dom0 allocation

You nailed it when you mentioned driver domains (next email); the long
term goal is to make sure we're able to support them and hopefully avoid
the hogging of that memory unnecessarily for non-dma uses. Thanks for
noticing ;^)  

(i was also glad Ian brought up numa, i had forgotten about it and this
is probably a good time to think about that while i'm tearing up this


Scott Parish
Signed-off-by: srparish@xxxxxxxxxx

Xen-devel mailing list