Re: [Xen-devel] Error restoring DomU when using GPLPV

[Mukesh Rathor <mukesh.rathor@xxxxxxxxxx>]

Keir Fraser wrote:
> On 15/09/2009 03:25, "Mukesh Rathor" <mukesh.rathor@xxxxxxxxxx> wrote:
>
> > Ok, I've been looking at this and figured what's going on. Annie's problem
> > lies in not remapping the grant frames post migration. Hence the leak,
> > tot_pages goes up every time until migration fails. On linux, remapping
> > is where the frames created by restore (for heap pfn's), get freed back to
> > the dom heap, is what I found.  So that's a fix to be made on win
> > pv driver side.
> Although obviosuly that is a bug, I'm not sure why it would cause this
> particular issue? The domheap pages do not get freed and replaced with
> xenheap pages, but why does that affect the next save/restore cycle? After
> all, xc_domain_save does not distinguish between Xenheap and domheap pages?
xc_domain_save doesn't distinguish is actually the problem, as
xc_domain_restore then backs xenheap pfn's for shinfo/gnt frames with dom
heap pages. These dom heap pages do get freed and replaced by xenheap pages
on target host (upon guest remap in gnttab_map()) in following code:

arch_memory_op():
        /* Remove previously mapped page if it was present. */
        prev_mfn = gmfn_to_mfn(d, xatp.gpfn);
        if ( mfn_valid(prev_mfn) )
        {
                .....
                guest_remove_page(d, xatp.gpfn);  <=======
        }

Eg. my guest with 128M gets created with tot_pages=0x83eb
max_pages:0x8400. Now xc_domain_save saves all, 0x83eb+shinfo+gnt
frames(2), so I see tot_pages on target go upto 0x83ee. Now, guest
remaps() shinfo and gnt frames. The dom heap pages are returned in
guest_remove_page(), tot_pages goes back to 0x83eb. In Annie's case,
driver forgets to remap the 2 gnt frames, so dom heap pages are wrongly
mapped and tot_pages remains at 0x83ed, and after few more when it reaches
0x83ff, migration fails as save is not be able to create
0x83ff+shinfo+gntframes temporarily, max_page being 0x8400.

Hope that makes sense.


> > 1. Always balloon down, shinfo+gnttab frames: This needs to be done just
> >     once during load, right? I'm not sure how it would work tho if mem gets
> >     ballooned up subsequently. I suppose the driver will have to intercept
> >     every increase in reservation and balloon down everytime?
> Well, it is the same driver that is doing the ballooning, so it's kind of
> easy to intercept, right? Just need to track how many Xenheap pages are
> mapped and maintain that amount of 'balloon down'.
Yup, that's what I thought, but just wanted to make sure.

> >     Also, balloon down during suspend call would prob be too late, right?
> Indeed it would. Need to do it during boot. It's only a few pages though, so
> noone will miss them.
>
> > 2. libxc fix: I wonder how much work this will be. Good thing here is,
> >     it'll take care of both linux and PV HVM guests avoiding driver
> >     updates in many versions, and hence appealing to us. Can we somehow
> >     mark the frames special to be skipped? Looking at biiig xc_domain_save
> >     function, not sure in case of HVM, how pfn_type gets set. May be before
> > the
> >     outer loop, it could ask hyp for all xen heap page list, but then what if
> > a
> >     new page gets added to the list in between.....
> It's a pain. Pfn_type[] I think doesn't really get used. Xc_domain_save()
> just tries to map PFNs and saves all the ones it successfully maps. So the
> problem is it is allowed to map Xenheap pages. But we can't always disallow
> that because sometimes the tools have good reason to map Xenheap pages. So
> we'd need a new hypercall, or a flag, or something, and that would need dom0
> kernel changes as well as Xen and toolstack changes. So it's rather a pain.
Ok got it, I think driver change is the way to go.

> > Also, unfortunately, the failure case is not handled properly sometimes.
> > If migration fails after suspend, then no way to get the guest
> > back. I even noticed, the guest disappeared totally from both source and
> > target when failed, couple times of several dozen migrations I did.
> That shouldn't happen since there is a mechanism to cancel the suspension of
> a suspended guest. Possibly xend doesn't get it right every time, as it's
> error handling is pretty poor in general. I trust the underlying mechanisms
> below xend pretty well however.
>  -- Keir

thanks a lot,
Mukesh

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel