[Xen-users] Live migration fails (ERROR Restore failed)

[Hitesh Khandelwal <hitesh.iitk@xxxxxxxxx>]

I compiled xen-4.0.1 (kernel�2.6.32.27) on Ubuntu 10.10 as Dom0. Ubuntu 10.10 with stock kernel is used as DomU. I tried live migration by following instructions on�http://www.virtuatopia.com/index.php/Migrating_Xen_domainU_Guests_Between_Host_Systems�, but it failed with the following logs.

Both the machines have same configuration and same amount of memory, and there is plenty of memory available on target host. VM guest�runs successfully on both the source and target hosts individually.�Relevant bug report:�http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1653

Thanks,

Hitesh

Log on source Dom0:

[2011-01-31 14:20:24 1008] DEBUG (XendCheckpoint:124) [xc_save]: /usr/lib/xen/bin/xc_save 46 10 0 0 1

[2011-01-31 14:20:24 1008] INFO (XendCheckpoint:423) /usr/lib/xen/bin/xc_save: symbol lookup error: /usr/lib/xen/bin/xc_save: undefined symbol: xs_suspend_evtchn_port

[2011-01-31 14:20:24 1008] ERROR (XendCheckpoint:178) Save failed on domain cloud-vm01 (10) - resuming.

Traceback (most recent call last):

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 146, in save

�� �forkHelper(cmd, fd, saveInputHandler, False)

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 409, in forkHelper

�� �raise XendError("%s failed: popen failed" % string.join(cmd))

XendError: /usr/lib/xen/bin/xc_save 46 10 0 0 1 failed: popen failed

[2011-01-31 14:20:24 1008] DEBUG (XendDomainInfo:3117) XendDomainInfo.resumeDomain(10)

[2011-01-31 14:20:27 1008] ERROR (xmlrpclib2:178) Internal error handling xend.domain.migrate

Traceback (most recent call last):

��File "/usr/local/lib/python2.6/dist-packages/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch

�� �response = self._dispatch(method, params)

��File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 418, in _dispatch

�� �return func(*params)

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1366, in domain_migrate

�� �self._domain_migrate(dominfo, dst, live, port, node)

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1458, in _domain_migrate

�� �dsterr = sock.recv(1024)

timeout: timed out

Log on�target�Dom0:

[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:235) XendDomainInfo.restore([\047domain\047, [\047domid\047, \04710\047], [\047cpu_weight\047, \047256\047], [\047cpu_cap\047, \0470\047], [\047bootloader\047, \047/usr/bin/pygrub\047], [\047on_crash\047, \047restart\047], [\047uuid\047, \047f32d195c-1fed-a9ad-5d7e-ff0456f79505\047], [\047bootloader_args\047], [\047vcpus\047, \0471\047], [\047description\047], [\047name\047, \047cloud-vm01\047], [\047cpus\047, [[]]], [\047on_reboot\047, \047restart\047], [\047on_poweroff\047, \047destroy\047], [\047maxmem\047, \047512\047], [\047memory\047, \047512\047], [\047shadow_memory\047, \0470\047], [\047vcpu_avail\047, \0471\047], [\047features\047], [\047on_xend_start\047, \047ignore\047], [\047on_xend_stop\047, \047ignore\047], [\047start_time\047, \0471296428689.01\047], [\047cpu_time\047, \04711.77212655\047], [\047online_vcpus\047, \0471\047], [\047image\047, [\047linux\047, [\047kernel\047], [\047args\047, \047root=/dev/sda2 ro \047], [\047superpages\047, \0470\047], [\047videoram\047, \0474\047], [\047pci\047, []], [\047nomigrate\047, \0470\047], [\047tsc_mode\047, \0470\047], [\047notes\047, [\047HV_START_LOW\047, \0474118806528\047], [\047FEATURES\047, \047!writable_page_tables|pae_pgdir_above_4gb\047], [\047VIRT_BASE\047, \0473221225472\047], [\047GUEST_VERSION\047, \0472.6\047], [\047PADDR_OFFSET\047, \0470\047], [\047GUEST_OS\047, \047linux\047], [\047HYPERCALL_PAGE\047, \0473222282240\047], [\047LOADER\047, \047generic\047], [\047SUSPEND_CANCEL\047, \0471\047], [\047PAE_MODE\047, \047yes\047], [\047ENTRY\047, \0473229974528\047], [\047XEN_VERSION\047, \047xen-3.0\047]]]], [\047status\047, \0472\047], [\047state\047, \047-b----\047], [\047store_mfn\047, \0472262985\047], [\047console_mfn\047, \0472262984\047], [\047device\047, [\047vif\047, [\047bridge\047, \047eth2\047], [\047uuid\047, \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047], [\047script\047, \047/etc/xen/scripts/vif-bridge\047], [\047ip\047, \047128.10.130.186\047], [\047mac\047, \04700:16:3E:C3:D2:A8\047], [\047backend\047, \0470\047]]], [\047device\047, [\047console\047, [\047protocol\047, \047vt100\047], [\047location\047, \0472\047], [\047uuid\047, \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047]]], [\047device\047, [\047vbd\047, [\047protocol\047, \047x86_32-abi\047], [\047uuid\047, \047f522f311-68bf-4358-216d-b8f81488a31b\047], [\047bootable\047, \0471\047], [\047dev\047, \047sda2:disk\047], [\047uname\047, \047file:/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047], [\047mode\047, \047w\047], [\047backend\047, \0470\047], [\047VDI\047]]], [\047device\047, [\047vbd\047, [\047protocol\047, \047x86_32-abi\047], [\047uuid\047, \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047], [\047bootable\047, \0470\047], [\047dev\047, \047sda1:disk\047], [\047uname\047, \047file:/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047], [\047mode\047, \047w\047], [\047backend\047, \0470\047], [\047VDI\047]]], [\047change_home_server\047, \047False\047]])

[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain

[2011-01-31 14:20:23 12024] DEBUG (balloon:220) Balloon: 1125176 KiB free; need 16384; done.

[2011-01-31 14:20:23 12024] DEBUG (XendDomain:464) Adding Domain: 8

[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:3400) Storing VM details: {\047on_xend_stop\047: \047ignore\047, \047shadow_memory\047: \0470\047, \047uuid\047: \047f32d195c-1fed-a9ad-5d7e-ff0456f79505\047, \047on_reboot\047: \047restart\047, \047start_time\047: \0471296428689.01\047, \047on_poweroff\047: \047destroy\047, \047bootloader_args\047: \047\047, \047on_xend_start\047: \047ignore\047, \047on_crash\047: \047restart\047, \047xend/restart_count\047: \0470\047, \047vcpus\047: \0471\047, \047vcpu_avail\047: \0471\047, \047bootloader\047: \047/usr/bin/pygrub\047, \047image\047: "(linux (kernel ) (args \047root=/dev/sda2 ro \047) (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES \047!writable_page_tables|pae_pgdir_above_4gb\047) (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE 3222282240) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3229974528) (XEN_VERSION xen-3.0)))", \047name\047: \047cloud-vm01\047}

[2011-01-31 14:20:23 12024] INFO (XendDomainInfo:2367) createDevice: console : {\047protocol\047: \047vt100\047, \047location\047: \0472\047, \047uuid\047: \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047}

[2011-01-31 14:20:23 12024] DEBUG (DevController:95) DevController: writing {\047state\047: \0471\047, \047backend-id\047: \0470\047, \047backend\047: \047/local/domain/0/backend/console/8/0\047} to /local/domain/8/device/console/0.

[2011-01-31 14:20:23 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/console/0\047, \047uuid\047: \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047, \047frontend-id\047: \0478\047, \047state\047: \0471\047, \047location\047: \0472\047, \047online\047: \0471\047, \047protocol\047: \047vt100\047} to /local/domain/0/backend/console/8/0.

[2011-01-31 14:20:23 12024] INFO (XendDomainInfo:2367) createDevice: vbd : {\047protocol\047: \047x86_32-abi\047, \047uuid\047: \047f522f311-68bf-4358-216d-b8f81488a31b\047, \047bootable\047: 1, \047driver\047: \047paravirtualised\047, \047dev\047: \047sda2:disk\047, \047uname\047: \047file:/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047, \047mode\047: \047w\047, \047backend\047: \0470\047}

[2011-01-31 14:20:23 12024] DEBUG (DevController:95) DevController: writing {\047virtual-device\047: \0472050\047, \047protocol\047: \047x86_32-abi\047, \047device-type\047: \047disk\047, \047backend-id\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vbd/8/2050\047} to /local/domain/8/device/vbd/2050.

[2011-01-31 14:20:23 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/vbd/2050\047, \047uuid\047: \047f522f311-68bf-4358-216d-b8f81488a31b\047, \047bootable\047: \0471\047, \047dev\047: \047sda2\047, \047state\047: \0471\047, \047params\047: \047/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047, \047mode\047: \047w\047, \047online\047: \0471\047, \047frontend-id\047: \0478\047, \047type\047: \047file\047} to /local/domain/0/backend/vbd/8/2050.

[2011-01-31 14:20:24 12024] INFO (XendDomainInfo:2367) createDevice: vbd : {\047protocol\047: \047x86_32-abi\047, \047uuid\047: \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047, \047bootable\047: 0, \047driver\047: \047paravirtualised\047, \047dev\047: \047sda1:disk\047, \047uname\047: \047file:/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047, \047mode\047: \047w\047, \047backend\047: \0470\047}

[2011-01-31 14:20:24 12024] DEBUG (DevController:95) DevController: writing {\047virtual-device\047: \0472049\047, \047protocol\047: \047x86_32-abi\047, \047device-type\047: \047disk\047, \047backend-id\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vbd/8/2049\047} to /local/domain/8/device/vbd/2049.

[2011-01-31 14:20:24 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/vbd/2049\047, \047uuid\047: \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047, \047bootable\047: \0470\047, \047dev\047: \047sda1\047, \047state\047: \0471\047, \047params\047: \047/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047, \047mode\047: \047w\047, \047online\047: \0471\047, \047frontend-id\047: \0478\047, \047type\047: \047file\047} to /local/domain/0/backend/vbd/8/2049.

[2011-01-31 14:20:24 12024] INFO (XendDomainInfo:2367) createDevice: vif : {\047bridge\047: \047eth2\047, \047uuid\047: \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047, \047script\047: \047/etc/xen/scripts/vif-bridge\047, \047ip\047: \047128.10.130.186\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047backend\047: \0470\047}

[2011-01-31 14:20:24 12024] DEBUG (DevController:95) DevController: writing {\047backend-id\047: \0470\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047handle\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vif/8/0\047} to /local/domain/8/device/vif/0.

[2011-01-31 14:20:24 12024] DEBUG (DevController:97) DevController: writing {\047bridge\047: \047eth2\047, \047domain\047: \047cloud-vm01\047, \047handle\047: \0470\047, \047uuid\047: \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047, \047script\047: \047/etc/xen/scripts/vif-bridge\047, \047ip\047: \047128.10.130.186\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047frontend-id\047: \0478\047, \047state\047: \0471\047, \047online\047: \0471\047, \047frontend\047: \047/local/domain/8/device/vif/0\047} to /local/domain/0/backend/vif/8/0.

[2011-01-31 14:20:24 12024] DEBUG (XendDomainInfo:1804) Storing domain details: {\047image/entry\047: \0473229974528\047, \047console/port\047: \0472\047, \047image/loader\047: \047generic\047, \047vm\047: \047/vm/f32d195c-1fed-a9ad-5d7e-ff0456f79505\047, \047control/platform-feature-multiprocessor-suspend\047: \0471\047, \047image/hv-start-low\047: \0474118806528\047, \047image/guest-os\047: \047linux\047, \047image/virt-base\047: \0473221225472\047, \047memory/target\047: \047524288\047, \047image/guest-version\047: \0472.6\047, \047image/pae-mode\047: \047yes\047, \047description\047: \047\047, \047console/limit\047: \0471048576\047, \047image/paddr-offset\047: \0470\047, \047image/hypercall-page\047: \0473222282240\047, \047image/suspend-cancel\047: \0471\047, \047cpu/0/availability\047: \047online\047, \047image/features/pae-pgdir-above-4gb\047: \0471\047, \047image/features/writable-page-tables\047: \0470\047, \047console/type\047: \047xenconsoled\047, \047name\047: \047cloud-vm01\047, \047domid\047: \0478\047, \047image/xen-version\047: \047xen-3.0\047, \047store/port\047: \0471\047}

[2011-01-31 14:20:24 12024] DEBUG (XendCheckpoint:286) restore:shadow=0x0, _static_max=0x20000000, _static_min=0x0,�

[2011-01-31 14:20:24 12024] DEBUG (XendCheckpoint:305) [xc_restore]: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0

[2011-01-31 14:20:26 12024] INFO (XendCheckpoint:423) ERROR Internal error: read: p2m_size

[2011-01-31 14:20:26 12024] INFO (XendCheckpoint:423) Restore exit with rc=1

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:3053) XendDomainInfo.destroy: domid=8

[2011-01-31 14:20:26 12024] ERROR (XendDomainInfo:3067) XendDomainInfo.destroy: domain destruction failed.

Traceback (most recent call last):

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomainInfo.py", line 3060, in destroy

�� �xc.domain_pause(self.domid)

Error: (3, \047No such process\047)

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2416) No device model

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2418) Releasing devices

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2424) Removing vif/0

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2424) Removing console/0

[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0

[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:2424) Removing vbd/2050

[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2050

[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:2424) Removing vbd/2049

[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2049

[2011-01-31 14:20:27 12024] INFO (XendDomain:1107) Domain cloud-vm01 (f32d195c-1fed-a9ad-5d7e-ff0456f79505) deleted.

[2011-01-31 14:20:27 12024] ERROR (XendCheckpoint:357) /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed

Traceback (most recent call last):

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 309, in restore

�� �forkHelper(cmd, fd, handler.handler, True)

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 411, in forkHelper

�� �raise XendError("%s failed" % string.join(cmd))

XendError: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed

[2011-01-31 14:20:27 12024] ERROR (XendDomain:1175) Restore failed

Traceback (most recent call last):

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1159, in domain_restore_fd

�� �dominfo = XendCheckpoint.restore(self, fd, paused=paused, relocating=relocating)

��File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 358, in restore

�� �raise exn

XendError: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed

Hitesh Khandelwal | Graduate Student | Purdue University |�http://www.hiteshkhandelwal.com

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users