[Xen-devel] PCI Pass-through Difficulty

["Sandwell, James" <James.Sandwell@xxxxxxx>]

            We’re running into a problem with receiving interrupts from a PCI pass-through device (namely a 1068 SAS Host Board Adapter). The device has been added to be PCI pass-through in the grub.conf and a cfg script for starting the domain. We can read/write to the config registers on the PCI device but we never receive the doorbell interrupt during initialization.  We’re having trouble determining root cause of this issue.

 

          We're using a Dell T7400 with VT-D and VT-X enabled and running Xen 3.4-unstable with patches (c/s 18430 and c/s 4761) with CentOS. We’re currently loading a VxWorks kernel as a guest operating system and can successfully execute our initialization procedures.

 

 

Various outputs (hopefully helpful – if more detailed is needed like entire logs let me know)

 

uname –a

Linux 2.6.18.8-xen #3 SMP x86_64 x86_64 x86_64 GNU/Linux

 

.cfg file

pci=['06:00.0']

 

grub.conf

title XEN (whargharbl)

        root (hd0,0)

        kernel /xen-3.4-unstable.gz acpi=force apic=on vtd=1 iommu=1

        module /vmlinuz-2.6.18.8-xen ro root=/dev/VolGroup00/LogVol00 pciback.hide=(06:00.0)

        module /initrd-2.6.18.8-xen.img

 

lspci –b -v

06:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)

        Subsystem: LSI Logic / Symbios Logic Unknown device 30a0

        Flags: fast devsel, IRQ 5

        I/O ports at bc00 [disabled]

        Memory at 00000000f7cec000 (64-bit, non-prefetchable) [disabled]

        Memory at 00000000f7cf0000 (64-bit, non-prefetchable) [disabled]

        Expansion ROM at f7a00000 [disabled]

        Capabilities: [50] Power Management version 2

        Capabilities: [68] Express Endpoint IRQ 0

        Capabilities: [98] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-

        Capabilities: [b0] MSI-X: Enable- Mask- TabSize=1

 

 

dmesg

GSI 26 sharing vector 0x51 and IRQ 26

ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 70 (level, low) -> IRQ 26

 

pciback 0000:06:00.0: seizing device

 

ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 70 (level, low) -> IRQ 26

ACPI: PCI interrupt for device 0000:06:00.0 disabled

 

 

xm dmesg (after we didn’t receive the doorbell interrupt we were expecting)

 

(XEN) domain_context_mapping: DEV_TYPE_PCIe_ENDPOINT

(XEN) Xen WARN at irq.c:514

(XEN) ----[ Xen-3.4-unstable  x86_64  debug=n  Not tainted ]----

(XEN) CPU:    0

(XEN) RIP:    e008:[<ffff828c8013cf0e>] pirq_guest_bind+0x2e/0x240

(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor

(XEN) rax: 0000000000000001   rbx: ffff83007ee18080   rcx: 0000000000000000

(XEN) rdx: 0000000000000001   rsi: 000000000000001a   rdi: ffff83007eff6080

(XEN) rbp: 0000000000000002   rsp: ffff828c80247ca8   r8:  000000000000001a

(XEN) r9:  ffff83007ee18490   r10: 0000000000000008   r11: 0000000000000008

(XEN) r12: 0000000000000000   r13: ffff83007eff6080   r14: 000000000000001a

(XEN) r15: 0000000000000001   cr0: 000000008005003b   cr4: 00000000000026b0

(XEN) cr3: 000000006c646000   cr2: 000000000046c1f0

(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008

(XEN) Xen stack trace from rsp=ffff828c80247ca8:

(XEN)    0000000000000000 ffff828c80247e28 ffff828c8011b422 ffff83007ee18080

(XEN)    0000000000000002 0000000000000000 0000000000000006 000000000000001a

(XEN)    ffff83007effc080 ffff828c80125b9c 0000000000000cfc fffffffffffffff3

(XEN)    fffffffffffffffd 00007fffc86bb1d0 ffff828c80247e28 ffff828c80247e28

(XEN)    00000000004c5398 ffff828c801317a4 0000000000000cfc 0000000000000000

(XEN)    00000000800600b0 ffff83006ca8ed28 0000000000800227 0000000000800227

(XEN)    ffff83006ca8ed28 0000000000000000 0000000000000000 ffff828c80145261

(XEN)    ffff828c80247eb8 0000000000000000 0000000000000000 0000000080247f28

(XEN)    000000000006ca8e ffff828c80111eb0 ffff8284010fa630 fffffffffffffff3

(XEN)    00007fffc86bb1d0 0000000000305000 ffff828c80247e28 00000000ffffffda

(XEN)    00000000004c5398 ffff828c80104cdb 0000000020000000 0000000000000051

(XEN)    0000000000000202 ffff828c80271424 0000000000000000 0000000000000202

(XEN)    0000000500000026 0000000000d40001 000000000000001a 0000060000000001

(XEN)    0000000000000020 00000000f7cee000 0000000000000000 00000032bf209e25

(XEN)    0000000000000000 00007fffc86bb270 0000000000000006 0000000000d5b650

(XEN)    00000000004c5398 00000032bf20975c 0000000000000021 000000000000000d

(XEN)    00007fffc86bb270 0000003815b4e9a0 0000000000000000 0000000000000206

(XEN)    ffffffffffffffff 0000000000000000 00000038158cee57 0000000000000033

(XEN)    0000000000000206 ffff83007fdea080 00007fffc86bb260 0000000000305000

(XEN)    0000000000000006 00000000ffffffda 00000000004c5398 ffff828c801a9169

(XEN) Xen call trace:

(XEN)    [<ffff828c8013cf0e>] pirq_guest_bind+0x2e/0x240

(XEN)    [<ffff828c8011b422>] maybe_split+0x32/0x60

(XEN)    [<ffff828c80125b9c>] pt_irq_create_bind_vtd+0x22c/0x290

(XEN)    [<ffff828c801317a4>] arch_do_domctl+0xec4/0x1540

(XEN)    [<ffff828c80145261>] mod_l1_entry+0x971/0x990

(XEN)    [<ffff828c80111eb0>] rangeset_add_range+0x120/0x180

(XEN)    [<ffff828c80104cdb>] do_domctl+0xcdb/0xd80

(XEN)    [<ffff828c801a9169>] syscall_enter+0xa9/0xae

 

dmesg (after we didn’t receive the doorbell interrupt we were expecting)

 

PM: Adding info for xen-backend:vkbd-1-0

PM: Adding info for xen-backend:vfb-1-0

PM: Adding info for xen-backend:vbd-1-768

PM: Adding info for xen-backend:vif-1-0

PM: Adding info for xen-backend:pci-1-0

device vif1.0 entered promiscuous mode

eth0: port 2(vif1.0) entering learning state

eth0: topology change detected, propagating

eth0: port 2(vif1.0) entering forwarding state

ip_tables: (C) 2000-2006 Netfilter Core Team

tun: Universal TUN/TAP device driver, 1.6

tun: (C) 1999-2004 Max Krasnyansky <maxk@xxxxxxxxxxxx>

device tap1.0 entered promiscuous mode

eth0: port 3(tap1.0) entering learning state

eth0: topology change detected, propagating

eth0: port 3(tap1.0) entering forwarding state

PM: Adding info for xen-backend:console-1-0

vif1.0: no IPv6 routers present

tap1.0: no IPv6 routers present

 

            Another peculiarity is that we’re not seeing the vmx flag in /proc/cpuinfo while running Xen (but booting w/o Xen it does show up).

 

Thanks,

James Sandwell

 

 

 

 

 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel