By messing a bit with printk's and debug settings a warn_on in the hypervisor
is being triggered when starting the videograbbing domU:
mapping kernel into physical memory
about to get started...
(XEN) [2010-10-13 13:30:44] Xen WARN at msi.c:636
(XEN) [2010-10-13 13:30:44] ----[ Xen-4.1-unstable x86_64 debug=y Tainted:
C ]----
(XEN) [2010-10-13 13:30:44] CPU: 2
(XEN) [2010-10-13 13:30:44] RIP: e008:[<ffff82c48015d797>]
pci_enable_msi+0x48a/0x9d5
(XEN) [2010-10-13 13:30:44] RFLAGS: 0000000000010216 CONTEXT: hypervisor
(XEN) [2010-10-13 13:30:44] rax: 0000000000000004 rbx: 00000000fe5fe000
rcx: 0000000000000001
(XEN) [2010-10-13 13:30:44] rdx: 0000000000000004 rsi: 0000000000000282
rdi: ffff82c48024e940
(XEN) [2010-10-13 13:30:44] rbp: ffff830237e57dc8 rsp: ffff830237e57cf8 r8:
0000000000000009
(XEN) [2010-10-13 13:30:44] r9: 000000000000003a r10: 0000000000000092
r11: 0000000000000213
(XEN) [2010-10-13 13:30:44] r12: 0000000000000000 r13: ffff830237e57ea8
r14: ffff83020211ed10
(XEN) [2010-10-13 13:30:44] r15: 0000000000000008 cr0: 000000008005003b
cr4: 00000000000006f0
(XEN) [2010-10-13 13:30:44] cr3: 0000000225f0e000 cr2: ffff880004e93d68
(XEN) [2010-10-13 13:30:44] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss:
e010 cs: e008
(XEN) [2010-10-13 13:30:44] Xen stack trace from rsp=ffff830237e57cf8:
(XEN) [2010-10-13 13:30:44] ffff830237e57d38 ffff82c480126b66
ffff830237e57e18 0700000000000010
(XEN) [2010-10-13 13:30:44] 0000000000001000 0000000000000030
00000000fe5ff000 00000000fe5ff000
(XEN) [2010-10-13 13:30:44] 0000009000077d68 ffff83014601ad10
0000000700000246 0000000000000000
(XEN) [2010-10-13 13:30:44] 0000000700000092 0000000000000000
ffff83020211eda8 00000000000fe5ff
(XEN) [2010-10-13 13:30:44] 00000000000fe5ff ffff8301622fde28
0000000000000202 ffff830237e57da8
(XEN) [2010-10-13 13:30:44] ffff82c480120680 ffff830237e57ea8
00000000ffffffed ffff830146a24000
(XEN) [2010-10-13 13:30:44] 0000000000000057 0000000000000048
ffff830237e57e48 ffff82c48015f16e
(XEN) [2010-10-13 13:30:44] 0000000025dfc910 000000000000015c
0000000000000048 0000000000000120
(XEN) [2010-10-13 13:30:44] ffff830237e82480 0000000000000282
ffff83020211ed10 ffff830237e57e28
(XEN) [2010-10-13 13:30:44] ffff82c480120680 ffff88002df4bb30
0000000000000057 ffff830146a24000
(XEN) [2010-10-13 13:30:44] 0000000000000048 ffff830146a24190
ffff830237e57ef8 ffff82c480172806
(XEN) [2010-10-13 13:30:44] 0000000180196b1a ffff830237e5a020
ffff830200000004 ffff830237e57ea8
(XEN) [2010-10-13 13:30:44] 000000000000000b ffffffffffffffff
0000000000000007 0000000000000000
(XEN) [2010-10-13 13:30:44] 00000000fe5fe000 aaaaaaaaaaaaaaaa
0000000000000007 0000000000000048
(XEN) [2010-10-13 13:30:44] 00000000fe5fe000 0000000000000000
0000000000000246 ffff8300c7e88000
(XEN) [2010-10-13 13:30:44] 000000000000000b ffff8800278c4400
0000000000000011 ffff88002ffea700
(XEN) [2010-10-13 13:30:44] 00007cfdc81a80c7 ffff82c480202a82
ffffffff8100942a 0000000000000021
(XEN) [2010-10-13 13:30:44] ffff88002ffea700 0000000000000011
ffff8800278c4400 000000000000000b
(XEN) [2010-10-13 13:30:44] ffff88002df4bbd0 00000000000006a1
0000000000000213 ffff88002fc20200
(XEN) [2010-10-13 13:30:44] ffffffff810df6ea 0000000000000011
0000000000000021 ffffffff8100942a
(XEN) [2010-10-13 13:30:44] Xen call trace:
(XEN) [2010-10-13 13:30:44] [<ffff82c48015d797>] pci_enable_msi+0x48a/0x9d5
(XEN) [2010-10-13 13:30:44] [<ffff82c48015f16e>] map_domain_pirq+0x275/0x363
(XEN) [2010-10-13 13:30:44] [<ffff82c480172806>] do_physdev_op+0x826/0x10b0
(XEN) [2010-10-13 13:30:44] [<ffff82c480202a82>] syscall_enter+0xf2/0x14c
(XEN) [2010-10-13 13:30:44]
(XEN) [2010-10-13 13:30:44] SEIK bus: 7 slot: 0 func:0 msi->table_base:
fe5fe000 read_pci_mem_bar: 4
(XEN) [2010-10-13 13:30:44] SEIK pba_paddr: 4
it's this one:
WARN_ON(msi->table_base != read_pci_mem_bar(bus, slot, func, bir));
I have added some printk's .. and read_pci_mem_bar seems to return a bogus
value .. the pba_addr is used later in the function, but i can't oversee if and
when this could have implications.
This also occurs when disabling the pci_resource_align on the kernel line.
lspci on dom0 shows:
07:00.0 USB Controller: NEC Corporation Device 0194 (rev 03) (prog-if 30)
Subsystem: ASUSTeK Computer Inc. Device 8413
Flags: bus master, fast devsel, latency 0, IRQ 33
Memory at fe5fe000 (64-bit, non-prefetchable) [size=8K]
Capabilities: [50] Power Management version 3
Capabilities: [70] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Capabilities: [90] MSI-X: Enable+ Mask- TabSize=8
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting <?>
Capabilities: [140] Device Serial Number ff-ff-ff-ff-ff-ff-ff-ff
Capabilities: [150] #18
Kernel driver in use: pciback
In the same function it seems to trigger
if ( d )
{
/* XXX How to deal with existing mappings? */
}
Which seems to be a bit odd for a freshly booted system with no domU restarts ?
grub menu.lst:
title xen-4.1-unstable.gz / Debian GNU/Linux,
2.6.32.23-xen-next-2.6.32.x-generaldebug-20101002
root (hd0,0)
kernel /xen-4.1-unstable.gz dom0_mem=768M loglvl=all loglvl_guest=all
com1=115200,8n1 sync_console console_to_ring console_timestamps
console=vga,com1 iommu=off debug lapic=debug apic_verbosity=debug apic=debug
noirqbalance
module /vmlinuz-2.6.32.24-xen-next-2.6.32.x-tracing-20101013
root=/dev/mapper/serveerstertje-root ro earlyprintk=xen max_loop=255
loop_max_part=63 libata.noacpi=1 debug loglevel=10 noirqbalance irqbalance=off
iommu=soft xen-pciback.hide=(03:06.0)(07:00.0)(09:01.0)(09:01.1)(09:01.2)
pci=resource_alignment=03:06.0;07:00.0;09:01.0;09:01.1;09:01.2;
module /initrd.img-2.6.32.24-xen-next-2.6.32.x-tracing-20101013
--
Sander
Tuesday, October 12, 2010, 6:44:33 PM, you wrote:
> On Tue, Oct 12, 2010 at 06:28:13PM +0200, Sander Eikelenboom wrote:
>> Hi Keir,
>>
>> Does xen and/or the xen console depend on physical cpu 0 ?
> Usually the console for Dom0, and I think all other domains go
> through CPU0. Let me CC Ian here, who has been mucking in this
> area and found some bugs (and produced fixes).
> Ian, that bug you found with not clearing the eventchannel - that
> wouldn't have an impact here, right?
>>
>> I'm still trying to solve the mystery of my machine freezing when doing:
>>
>> - videograbbing in a domU with a usb3 pci-express controller passed through
>> (seems to cause quite a few interrupts)
>> - compiling a linux kernel with "make -j 6"
>>
>> It's a 6 core AMD phenom x6.
>>
>> Without cpu pinning:
>> I can freeze the machine easily within a minute after starting the compile,
>> at first xen serial console also slows down under the load (slow updates).
>> When the machine freezes i can't do anything with xen serial console.
>>
>> With cpu pinning:
>> By not using the pcpu 0 at all for any domain, and pinning the domain with
>> the videograbber to it's own pcpu (pcpu 5) it seems the machine keeps
>> running after 20 "make -j6" iterations of kernel compilation.
>> Xen serial console stays responsive and doesn't slow down during the kernel
>> compilation. The videograbber shows no problem grabbing video.
>>
> AHA! So finally closer to the mystery.
> Can you provide the /proc/interrupts of the Dom0?
> I wonder if this is related to the isseu I had some time ago, and never got
> to look at. The problem was that during heavy compilation (this is a 2 Nehelem
> socket box, just running Dom0 - no guests), the keyboard and USB driver would
> stop getting interrupts. So the drivers would start polling which is quite
> slow,
> albeit servicable, and then at some point it would pick up again.
> The weirdness was that the /proc/interrupts showed absolutly _no_ interrupts
> on CPU0
> during that time - as if Xen just forgot to update them. Jeremy suggested I
> try to
> disable Xen IRQ balance (noirqbalance on Xen command line) in case that is
> it, and to my
> emberrasement I haven't tried that yet.
> Did you try that? I think somebody suggested that but I can't recall whether
> it
> was for this issue?
>>
>> Name ID VCPU CPU State Time(s) CPU
>> Affinity
>> Domain-0 0 0 3 r-- 2169.7 1-4
>> Domain-0 0 1 1 -b- 2339.3 1-4
>> Domain-0 0 2 2 -b- 2358.9 1-4
>> Domain-0 0 3 3 -b- 2298.2 1-4
>> Domain-0 0 4 1 -b- 2221.9 1-4
>> Domain-0 0 5 4 -b- 2287.7 1-4
>> backup 9 0 4 -b- 10.6 1-4
>> database 1 0 4 -b- 45.3 1-4
>> davical 5 0 3 -b- 8.7 1-4
>> git 8 0 2 -b- 7.9 1-4
>> mail 2 0 4 -b- 8.0 1-4
>> samba 3 0 3 -b- 11.1 1-4
>> security 7 0 5 r-- 1433.2 5
>> www 4 0 1 -b- 10.2 1-4
>> zabbix 6 0 3 -b- 21.2 1-4
>>
>>
>> Is there a way a deadlock could occur between hypervisor <-> dom0 <-> domU
>> especially related to passthrough/interrupts in the context of pcpu 0 ?
> I don't know, but I do know that the IRQ handling in Xen 4.0 changed
> significantly compared
> to 3.4. I don't remember if you ever ran this setup under 3.4?
>>
>> --
>> Sander
--
Best regards,
Sander mailto:linux@xxxxxxxxxxxxxx
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home