[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] blkif migration problem

Ewan Mellor wrote:
On Thu, Dec 07, 2006 at 03:47:39PM +0000, Cristian Zamfir wrote:


I am trying to live migrate blkif devices backed by drbd devices and I am struggling with a problem for a few days now. The problem is that after migration, the domU machine cannot load any new programs into memory. The ssh connection survives migration and I can run programs that are already in the memory but not something that needs to be loaded from the disk.

I am currently testing with an almost idle machine and I am triggering the drive migration after the domain is suspended, in step 2, from: XendCheckpoint.py: dominfo.migrateDevices(network, dst, DEV_MIGRATE_STEP2, domain_name).

However, I also tried before the domain is suspended from step 1 (dominfo.migrateDevices(network, dst, DEV_MIGRATE_STEP1, domain_name)) and everything works fine, except that there is the obvious possibility of loosing some writes to the disk because the domain is not suspended yet.

After migration, when I reattach a console I get this message:
"vbd vbd-769: 16 Device in use; refusing to close"
This is from the blkfront.c backend_changed() function but I cannot figure out why this error occurs.

I believe that this means that the frontend has seen that the backend is
tearing down, but since the device is still mounted inside the guest, it's
refusing.  I don't think that the frontend ought to see the backend tear down
at all -- the guest ought to be suspended before you tear down the backend

I am triggering the migration in DEV_MIGRATE_STEP2, which is right after the domain was suspended, as far as I can tell from the python code in XendCheckpoint.py:

dominfo.migrateDevices(network, dst, DEV_MIGRATE_STEP1, domain_name)
def saveInputHandler(line, tochild):
           log.debug("In saveInputHandler %s", line)
           if line == "suspend":
                log.debug("Suspending %d ...", dominfo.getDomid())
                dominfo.migrateDevices(network, dst, DEV_MIGRATE_STEP2,
                log.info("Domain %d suspended.", dominfo.getDomid())
                dominfo.migrateDevices(network, dst, DEV_MIGRATE_STEP3,

"Triggering the migration" involves dominfo.migrateDevices(..) calling my script in /etc/xen/scripts. This script checks that the drive at the source and the replica at the destination are in sync and then switches their roles (the one on the source becomes secondary and the one on the destination becomes primary). But since the guest is suspended at this point, I don't understand why should the frontend see any change.

I found that DRBD drives are not quite usable when they are in secondary state, only the primary one should be mounted. For instance, when trying to mount a drbd device in secondary state I get this error:
#mount -r -t reiserfs /dev/drbd1 /mnt/vm
mount: /dev/drbd1 already mounted or /mnt/vm busy

Therefore, could this error happen on the destination, during restore while waiting for backends to set up, if the drive is in secondary state?

I also don't understand why everything works if I migrate the hard drive in DEV_MIGRATE_STEP1. The only error I get in this case is reiserfs complainig about some writes that failed, but everything besides this seems ok.

I cannot really try localhost migration because I think drbd only works with two machines, but I have tested most of my code outside xen and it worked.

Thank you very much for your help.

When you say that you are "triggering the drive migration", what does that
involve?  Why would the frontend see the store contents change at all at this

Have you tried a localhost migration?  This would be easier, because you don't
actually need to move the disk of course, so you can get half your signalling
tested before moving on to the harder problem.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.