Re: [Xen-devel] live migration fails (assert in shadow_hash_delete)

[Ashish Bijlani <ashish.bijlani@xxxxxxxxx>]

xen barfs while live migrating a 32-bit VM (on 64-bit platform):

(XEN) Assertion '__mfn_valid(mfn_x(smfn))' failed at multi.c:2561
(XEN) ----[ Xen-4.0.0-rc4  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    2
(XEN) RIP:    e008:[<ffff82c4801e0639>]
sh_map_and_validate_gl4e__guest_4+0x6d/0x1d4
(XEN) RFLAGS: 0000000000010246   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff830213187000   rcx: 00000000000000d3
(XEN) rdx: 0000000049ba6ceb   rsi: 00000000000000d3   rdi: ffffffffffffffff
(XEN) rbp: ffff83022ff2fb68   rsp: ffff83022ff2faf8   r8:  0000000000213187
(XEN) r9:  007fffffffffffff   r10: ffff82c480207e90   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000213187   r14: 0000000000000008
(XEN) r15: 0000000000000008   cr0: 000000008005003b   cr4: 00000000000006f4
(XEN) cr3: 000000020ee46000   cr2: 00000000c1829248
(XEN) ds: 007b   es: 007b   fs: 00d8   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff83022ff2faf8:
(XEN)    ffff83022ff2fb68 ffff82c4801bbe32 ffff830004060000 ffffffffffffffff
(XEN)    ffff83022ff2ff28 ffff83022ff2ff28 ffff8301f5330000 0000000000000000
(XEN)    ffff83022ff2fb78 ffff82f6042630e0 0000000000000000 0000000000213187
(XEN)    ffff830004060000 0000000000000008 ffff83022ff2fbb8 ffff82c4801c785b
(XEN)    ffff83022ff2fbc8 ffff830213187000 3000000000000000 ffff830004060000
(XEN)    00000001dc092027 ffff83022ff2fc60 0000000000000000 ffff830213187000
(XEN)    ffff83022ff2fc08 ffff82c4801c79a8 0000000000213187 00000001dbaa5027
(XEN)    ffff830004060000 00000001dc092027 ffff830004060000 00000001dc092027
(XEN)    0000000000000000 00000001dbaa5027 ffff83022ff2fc98 ffff82c480163091
(XEN)    ffff83022ff2fc88 ffff82c4801e1180 ffff830100000000 ffff83022ff2fc60
(XEN)    0000000000213187 00000001dbaa5027 00000001dbaa5027 ffff830213187000
(XEN)    ffff83022ff2ff28 00000001dc092027 ffff83022ff2ff28 ffff830004060000
(XEN)    ffff8301f5330000 00000000001dbaa5 0000000000000005 ffff83022ff2ff28
(XEN)    ffff83022ff2fcc8 ffff82c480163242 ffff8301f5330000 0000000000000000
(XEN)    ffff83022ff24000 0000000000000005 ffff83022ff2fdc8 ffff82c480163ba7
(XEN)    ffff8301f5330018 00007ff0d8c3c148 0000000000000000 ffff82c480265db0
(XEN)    ffff82c480265db8 ffff83022ff2ff28 ffff83022ff2ff28 ffff8301f5330218
(XEN)    000000200000007b ffff81800060c148 ffff830004060000 ffff8301f5330000
(XEN)    ffff818000000000 00000001001d8462 0000000000000000 00000006cfd24000
(XEN)    80000001d9582021 ffff830000000001 ffff83022ff2fd78 0000000004060060
(XEN) Xen call trace:
(XEN)    [<ffff82c4801e0639>] sh_map_and_validate_gl4e__guest_4+0x6d/0x1d4
(XEN)    [<ffff82c4801c785b>] sh_validate_guest_entry+0x17e/0x1c6
(XEN)    [<ffff82c4801c79a8>] shadow_cmpxchg_guest_entry+0x105/0x189
(XEN)    [<ffff82c480163091>] mod_l4_entry+0x2fd/0x3e3
(XEN)    [<ffff82c480163242>] new_guest_cr3+0xcb/0x269
(XEN)    [<ffff82c480163ba7>] do_mmuext_op+0x7c7/0x14b8
(XEN)    [<ffff82c4801f2248>] compat_mmuext_op+0x217/0x3a9
(XEN)    [<ffff82c4801309b9>] compat_multicall+0x269/0x404
(XEN)    [<ffff82c4801ff580>] compat_hypercall+0xc0/0x119
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 2:
(XEN) Assertion '__mfn_valid(mfn_x(smfn))' failed at multi.c:2561
(XEN) ****************************************
(XEN)

Any ideas how to fix this prob?

Is live migration not stable enough with xen-4.0 (rc4) yet?

Thanks,
Ashish

On Tue, Feb 23, 2010 at 12:05 PM, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx> wrote:
> Keir Fraser writes ("Re: [Xen-devel] live migration fails (assert in 
> shadow_hash_delete)"):
>> Localhost migrations were just added to the automated tests. But I think
>> maybe they are trivially failing due to trying to do them via the 'xl'
>> interface, which doesn't support it(!). Ian?
>
> Localhost migration does work in most combinations in our tests.  It
> was only recently added and there are a few teething troubles with it
> so I don't have a full slate of results.
>
> It doesn't work at all with libxl because it's not implemented.
>
> Keir:
>> A quick manual test indicates it's very easy to get Xen to blow up. I got
>> the following on my first localhost live migration attempt, which is a
>> different looking crash in the shadow code. This is with 2.6.18 dom0 and
>> domU by the way, so it's not pv_ops tickling the hypervisor in an unexpected
>> way...
>
> 2.6.18 doesn't boot on my test hardware so I'm just building it, not
> running it.  So I haven't reproduced your test, which explains the
> different results.
>
> Ian.
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel