Re: [Xen-devel] RE: hypervisor crash in map_vtd_domain_page

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Fri, 2011-01-14 at 08:57 +0000, Ian Campbell wrote: 
> On Thu, 2011-01-13 at 18:38 +0000, Kay, Allen M wrote:
> > Ian/Keir,
> > 
> > The attached patch should fix the crash issue.  Somehow the check for 
> > !iommu_hap_pt_share in function iommu_set_pgd() got dropped in changeset 
> > 22692.
> > 
> > Signed-off-by: Allen Kay <allen.m.kay@xxxxxxxxx>
> 
> Works for me. Thanks.

However I'm now seeing this when passing through devices to a HVM guest
with a stub domain.

I don't know if it relates to the recent changes, I seem to be seeing it
reliably since Friday both with your fix and with the reverts I was
using yesterday but my reverts were working OK for me on Thursday so my
mileage seems variable in some way I've not yet put my finger on.

It seems to be triggered specifically by HVM PCI passthrough + stub
domain. HVM passthrough without stubdom seems to be ok.

The error seems to indicate that a lock has been leaked. 

        (XEN) HVM1: pci dev 02:0 bar 10 size 02000000: f0000008
        (XEN) HVM1: pci dev 03:0 bar 14 size 01000000: f2000008
        (XEN) HVM1: pci dev 04:0 bar 10 size 00020000: f3000000
        (XEN) HVM1: pci dev 02:0 bar 14 size 00001000: f3020000
        (XEN) HVM1: pci dev 05:0 bar 20 size 00001000: f3021000
        (XEN) Assertion '!(((*({ unsigned long __ptr; __asm__ ("" : "=r"(__ptr) 
: "0"(&per_cpu____preempt_count)); (typeof(&per_cpu____preempt_count)) \
        (__ptr + (get_cpu_info()->per_cpu_offset)); }))) || 
((irq_stat[(((get_cpu_info()->processor_id)))].__local_irq_count) != 0) || 
!local_irq_is_enabled())' \
        failed at softirq.c:61                                                  
                                                                                
                              
        (XEN) ----[ Xen-4.1-unstable  x86_64  debug=y  Not tainted ]----
        (XEN) CPU:    14
        (XEN) RIP:    e008:[<ffff82c4801233a9>] do_softirq+0x51/0x7a
        (XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
        (XEN) rax: ffff82c4802d3ea0   rbx: ffff8300bf5e0000   rcx: 
ffff82c4802b0880
        (XEN) rdx: 00000041b0e28180   rsi: 0000000000000000   rdi: 
0000000000000001
        (XEN) rbp: ffff8306310f7f08   rsp: ffff8306310f7f08   r8:  
00000000deadbeef
        (XEN) r9:  00000000deadbeef   r10: 00000000deadbeef   r11: 
0000000000000297
        (XEN) r12: 0000002002002fb0   r13: 00000013deb7a92d   r14: 
000000118aabc52d
        (XEN) r15: 0000002002002fb0   cr0: 000000008005003b   cr4: 
00000000000026f0
        (XEN) cr3: 00000006217ad000   cr2: 00000020025b9000
        (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
        (XEN) Xen stack trace from rsp=ffff8306310f7f08:
        (XEN)    00007cf9cef080c7 ffff82c480207986 0000002002002fb0 
000000118aabc52d
        (XEN)    00000013deb7a92d 0000002002002fb0 00000000005df228 
00000013deb7a92d
        (XEN)    0000000000000297 0000000000000009 00000000005df41f 
0000000000000005
        (XEN)    0000000000000000 00000000000023aa 0000000000000002 
00000000deadbeef
        (XEN)    00000000deadbeef 0000010000000000 00000000000023aa 
000000000000e033
        (XEN)    0000000000000297 00000000005df1f0 000000000000e02b 
0000000000000000
        (XEN)    0000000000000000 0000000000000000 0000000000000000 
000000000000000e
        (XEN)    ffff8300bf5e0000 00000041b0e28180 0000000000000000
        (XEN) Xen call trace:
        (XEN)    [<ffff82c4801233a9>] do_softirq+0x51/0x7a
        (XEN)    
        (XEN)
        (XEN) ****************************************
        (XEN) Panic on CPU 14:
        (XEN) Assertion '!(((*({ unsigned long __ptr; __asm__ ("" : "=r"(__ptr) 
: "0"(&per_cpu____preempt_count)); 
(typeof(&per_cpu____preemp****************************************

Other than what is already decoded none of the stack or register values
seem to contain anything useful.

> 
> Tested-by: Ian Campbell <ian.campbell@xxxxxxxxxx>
> 
> > 
> > -----Original Message-----
> > From: Ian Campbell [mailto:Ian.Campbell@xxxxxxxxxx] 
> > Sent: Thursday, January 13, 2011 2:12 AM
> > To: Kay, Allen M
> > Cc: Keir Fraser; xen-devel
> > Subject: hypervisor crash in map_vtd_domain_page
> > 
> > Hi Allen,
> > 
> > When I try and start an HVM guest with a passed through PCI device I
> > immediately get the hypervisor crash below.
> > 
> > Domain 0 is xen/stable-2.6.32.x if that matters. The two devices are USB
> > controllers.
> > 
> > Based on the hg log of xen/drivers/passthrough I reverted
> > 22697:1ae74f060a39+22696:f5b3fc5a5220+22695:55450858f414 and reapplied
> > the build fix bit of 22695:55450858f414 and that still failed.
> > 
> > I then reverted 22692:66e806289464 and now my guest boots fine.
> > 
> > Ian.
> > 
> > (XEN) Assertion 'ma < ((((((262UL) >> 8) * 0xffff000000000000UL) | (262UL 
> > << 39))) + ((1UL << 39)*10)) - (((((262UL) >> 8) * 0xffff000000000000UL) | 
> > (262UL << 39)))' failed at 
> > /local/scratch/ianc/devel/xen-unstable.hg/xen/incl:98
> > (XEN) ----[ Xen-4.1-unstable  x86_64  debug=y  Not tainted ]----
> > (XEN) CPU:    5
> > (XEN) RIP:    e008:[<ffff82c48014725d>] map_vtd_domain_page+0x1d/0x58
> > (XEN) RFLAGS: 0000000000010212   CONTEXT: hypervisor
> > (XEN) rax: 000004ffffffffff   rbx: ffff8306221da000   rcx: 0000000000000027
> > (XEN) rdx: 1c000006221d9000   rsi: 0000000000000000   rdi: 1c000006221d9007
> > (XEN) rbp: ffff830631167b68   rsp: ffff830631167b68   r8:  ffff82c48014261a
> > (XEN) r9:  0000000000000001   r10: 0000000000000000   r11: 0000000000000000
> > (XEN) r12: 0000000000000012   r13: 0000000000000004   r14: 0000000000000027
> > (XEN) r15: ffff8306221da000   cr0: 000000008005003b   cr4: 00000000000026f0
> > (XEN) cr3: 0000000630e85000   cr2: 00000000b75ab3f9
> > (XEN) ds: 007b   es: 007b   fs: 00d8   gs: 0033   ss: 0000   cs: e008
> > (XEN) Xen stack trace from rsp=ffff830631167b68:
> > (XEN)    ffff830631167bb8 ffff82c4801423a6 0000000131167ba8 ffff830622338000
> > (XEN)    0000000000000000 00000000000000e8 0000000000000000 0000000000000003
> > (XEN)    ffff830622338dc0 ffff830622338000 ffff830631167c18 ffff82c48014268d
> > (XEN)    ffff830631167bd8 ffff82c480144a04 000000000023368b 0000000000000000
> > (XEN)    ffff830622338000 00000000000000e8 ffff830622338000 ffff82f60466d160
> > (XEN)    0000000000000000 00007d0a00000000 ffff830631167c58 ffff82c48013e6be
> > (XEN)    ffff830622338018 00000000000000e8 ffff830622338000 0000000000000000
> > (XEN)    00000000000000e8 0000000000000000 ffff830631167da8 ffff82c480151905
> > (XEN)    0000000000000092 ffff830631167cd0 0000000000000001 ffff82c4802b0880
> > (XEN)    ffff82c4802c27c0 0000000000000282 ffff830631167ca8 ffff82c480176456
> > (XEN)    ffff830631167cb8 ffff82c480176601 ffff830631167cf8 ffff82c4801232b4
> > (XEN)    ffff8306310a0018 ffff830631167e38 0000000000000000 ffff830631167d78
> > (XEN)    ffff830630e92db0 ffff8300bf4f8000 0000000031167d48 5000000000000000
> > (XEN)    ffff82f6046e84e0 0000000000237427 5400000000000002 ffff82f6046e84f0
> > (XEN)    0000000000000002 0000004844bbac9c ffff82c4802d3ec0 ffff8300bf4f8000
> > (XEN)    ffff830631167d98 0000000000000282 00000000c8bd98ec 0000000000000001
> > (XEN)    0000000000000025 0000000000000000 0000000000000202 fffffffffffffff3
> > (XEN)    0000000008064004 0000000000000000 0000000000000000 0000000000000000
> > (XEN)    ffff830631167ef8 ffff82c480103e41 ffff830631167dc8 0000000000000000
> > (XEN)    ffff830631167e28 ffff82c48016cc4c ffff830631167e08 0000000180106918
> > (XEN) Xen call trace:
> > (XEN)    [<ffff82c48014725d>] map_vtd_domain_page+0x1d/0x58
> > (XEN)    [<ffff82c4801423a6>] addr_to_dma_page_maddr+0x151/0x1b7
> > (XEN)    [<ffff82c48014268d>] intel_iommu_map_page+0x73/0x1db
> > (XEN)    [<ffff82c48013e6be>] assign_device+0x1e7/0x26a
> > (XEN)    [<ffff82c480151905>] arch_do_domctl+0x1e9f/0x38aa
> > (XEN)    [<ffff82c480103e41>] do_domctl+0x1983/0x1a12
> > (XEN)    [<ffff82c4802111c1>] compat_hypercall+0xa1/0xfa
> > (XEN)    
> > (XEN) 
> > (XEN) ****************************************
> > (XEN) Panic on CPU 5:
> > (XEN) Assertion 'ma < ((((((262UL) >> 8) * 0xffff000000000000UL) | (262UL 
> > << 39))) + ((1UL << 39)*10)) - (((((262UL) >> 8) * 
> > 0xffff00****************************************
> > (XEN) 
> > (XEN) Manual reset required ('noreboot' specified)
> > 
> > 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel