This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] live saving of domU

To: Andres Lagar Cavilla <andreslc@xxxxxxxxxxxxxx>
Subject: Re: [Xen-devel] live saving of domU
From: Anthony Liguori <aliguori@xxxxxxxxxx>
Date: Wed, 10 May 2006 14:40:53 -0500
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Wed, 10 May 2006 12:41:21 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <44623B92.7050300@xxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <E1FdtIP-0000Id-VQ@host-192-168-0-1-bcn-london> <44623B92.7050300@xxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mail/News 1.5 (X11/20060309)
Andres Lagar Cavilla wrote:
Anthony Liguori wrote

Moreover, you cannot dump the state of a domain after a pause and expect it to ever run again.

Guests are aware of the physical addresses of the memory that's been allocated to them. Because of this, to save a domain's state in a restorable way you need the guest to "canonicalize" itself. The only way to do this today is through a suspend operation which happens to be a subop of shutdown. Shutdowns are non-recoverable so you cannot use this as a snapshotting mechanism.
My understanding is that the guest only canonicalizes the store and console mfn's and places them on the shared info frame which is passed to the suspend hypercall. The rest of the canonicalizations are done by dom0 user-space code (xc_linux_save).

Sort of. When you pause a domain, it could be doing something like a PTE update in which case it has a PFN in a register (or on the stack somewhere). Part of the reason for having a suspend entry point in the kernel is to ensure that we're in a consistent state.

The guest never really shuts down: it issues the suspend hypercall and waits for it to return. This could happen months later when the domain is resumed :) The suspend hypercall executing in xen is the one that pauses all vcpus and kills the domain.

Actually, take a look at what HYPERVISOR_suspend is:

static inline int
   unsigned long srec)
   struct sched_shutdown sched_shutdown = {
       .reason = SHUTDOWN_suspend

   int rc = _hypercall3(int, sched_op, SCHEDOP_shutdown,
                &sched_shutdown, srec);

   if (rc == -ENOSYS)
       rc = _hypercall3(int, sched_op_compat, SCHEDOP_shutdown,
                SHUTDOWN_suspend, srec);

   return rc;

It's just a shutdown op. It's the same hypercall as reboot/halt and the hypervisor doesn't do anything differently for these calls. What happens next is that the hypervisor stops scheduling the domain and sets the 's' flag in the domain's state. The userspace tools will see that the domain has now "suspended" by checking the shutdown reason and begin the teardown process.

This of course implies that the userspace tools have to remember which domains have issued which requests (so they know who to check these things for). This is why we have the Xend daemon--to keep track of this information.

Is it feasible to use a different hypercall that pauses the domain but doesn't kill it, and once xc_linux_save is done checkpointing have it issue a dom0_op that unpauses the domain?

A domain is "killed" with a dom0_op of domain_destroy which is invoked by Xend. The problem with checkpointing is that once the 's' bit has been set on a domain, there's no way to unset that bit.

For filesystem corruption you're gonna have to hack up your own thing. Probably a CoW solution, where you begin a new "epoch" when resuming from the checkpoint.

Especially with something like dm-userspace on the horizon, this would (hopefully) soon be a moot issue. It just leaves us with this pesky problem of suspend.


Anthony Liguori


The closest thing you can achieve is a localhost migration. There are some caveats to this, of course. The first is that you need to have as much memory as the domain has available since you'll have a copy of the domain created briefly while the migration takes place. Migrations are also quite intrusive since they involve tearing down and bringing up all the devices.

I've gotten a lot of requests for light weight checkpointing. AFAIK, noone is actually working on it though.

Xen-devel mailing list

Xen-devel mailing list