This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] The mfn of the frame, that holds a mlock-ed PV domU user

To: Rafal Wojtczuk <rafal@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [Xen-devel] The mfn of the frame, that holds a mlock-ed PV domU usermode page, can change
From: Jeremy Fitzhardinge <jeremy@xxxxxxxx>
Date: Mon, 12 Apr 2010 13:01:54 -0700
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, joanna@xxxxxxxxxxxxxxxxxxxxxx
Delivery-date: Mon, 12 Apr 2010 13:03:03 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20100412185454.GC3671@xxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20100412185454.GC3671@xxxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20100330 Fedora/3.0.4-1.fc12 Lightning/1.0b2pre Thunderbird/3.0.4
On 04/12/2010 11:54 AM, Rafal Wojtczuk wrote:
> Hello,
> Would someone on the list enlight me on the following issue, possibly related 
> to mfn management in the PV guest.
> Environment: xen-3.4.3, pvops in dom0 and domU, all 64bit.
> Usermode code
> (if you are interested, at 
> http://gitweb.qubes-os.org/gitweb/?p=mainstream/gui.git;a=blob;f=vchan/vchan/init.c;h=cb2fb851c3b97804b115dbf58fd47a30d6d0a8a3;hb=HEAD)
> in PV domU does the following:
> 1) gets a page via
> ring_ref_v=mmap(0, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE |MAP_ANON,-1, 0);
> 2) mlock(ring_ref_v, 4096)
> 3) determines the mfn number of the frame holding ring_ref_v via a call to 
> u2mfn driver
> (http://gitweb.qubes-os.org/gitweb/?p=mainstream/gui.git;a=blob;f=vchan/u2mfn/u2mfn.c;h=6ff113c07c50ef078ab04d9e61d2faab338357e7;hb=HEAD)
> the driver basically does get_user_pages+kmap+virt_to_mfn (and later
> kunmap+put_page for cleanup).

Yeah, this looks fundamentally suspect.  Using mlock in this way is
going to be fragile.

> Then, the
> usermode code in dom0 does xc_map_foreign_range on the returned mfn, and can
> communicate with the code in domU over this shared page...
> ...but sometimes, apparently the page that backs ring_ref_v changes: if the 
> domU application calls u2mfn ioctl again with ring_ref_v argument, it 
> returns a different mfn.
> And naturally the code in dom0 reads garbage from the address returned by
> its pevious call to xc_map_foreign_range.
> I find it puzzling. Is this behaviour normal/expected ?


> Mlock man pages say "All pages that contain a part of the specified address 
> range are guaranteed to be resident in RAM", not "be resident at the same
> RAM location", but why would a frame backing a mlock-ed memory be changed ?
> Is there some memory defragmentation going on ?

Yes, the kernel can move usermode memory around to defrag memory.  This
is done to allow higher-order memory allocations to keep working even on
a long-running system which would otherwise fragment the address space. 
Ideally it allows 2M page allocations to succeed.

>  Or maybe only frame->frame
> number function changes (but why would it) ?
> Anyway, this behaviour causes problems, as you can see in
> http://www.qubes-os.org/trac/ticket/16#comment:4
> It would be nice if the mfn of a frame that holds a given mlock-ed usermode 
> page could be made constant. 
> If you can offer some insight, that would be helpful, particularly:
> 1) Why this does not happen to pages allocated in kernel mode (if it did, it
> would break the split drivers model) ?

No, kernel allocations are not movable by default.

> 2) Can this frame-changing behaviour be switched off at Xen/kernel level?

Not that I know of, and it wouldn't be desirable if it could be.

> 3) Would using grant tables (instead of brutal xc_map_foreign_range) change 
> the situation ?
> BTW, for Qubes it is necessary to map PV domU usermode pages in dom0; 
> particularly, map X server composition buffers.

Why is it necessary to map usermode pages?  It just seems like asking
for trouble.  Why not make it so that the domU X server gets the memory
from the kernel (via some kind of driver), and then map that through to


Xen-devel mailing list