This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] PCI Passthrough Problems/Questions

To: Nick Couchman <Nick.Couchman@xxxxxxxxx>
Subject: Re: [Xen-devel] PCI Passthrough Problems/Questions
From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Date: Mon, 25 Oct 2010 13:40:33 -0400
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Mon, 25 Oct 2010 10:41:33 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4CC55D5C02000099000BA698@xxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4CC55D5C02000099000BA698@xxxxxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/1.5.20 (2009-06-14)
On Mon, Oct 25, 2010 at 10:35:08AM -0600, Nick Couchman wrote:
> This is a message I posted to xen-users without getting any good leads...I'm 
> hoping someone on the devel list has some ideas as to what might be going on 
> here.
> I've asked a couple of these questions, already, but haven't really had
> any response, so I thought I'd try, again, and try to provide a little
> more detail.  I need to pass through a PCIe 4-port Serial card.  I've
> done this successfully multiple times with different serial cards, but
> the current one, a PCIe Perle quad-port RJ45, is causing some problems.
> I believe the problem stems from the fact that the card is a PCI card
> and that Perle just built a PCIe-PCI bridge so that they could reuse an
> existing PCI design.  The following three PCI devices are all on the
> same card:
> # lspci
> ...
> 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI
> Express-to-PCI Bridge (rev aa)
> 02:00.0 Serial controller: Perle Systems Ltd Device 0331
> 02:00.1 Bridge: Perle Systems Ltd Device 0331
> I've tried various combinations of hiding 01:00.0, 02:00.0, and/or
> 02:00.1 from dom0 and passing those through.  First, if I try all three,
> I put the following in my dom0 kernel arguments:
> pciback.hide=(01:00.0)(02:00.0)(02:00.1)
> Before someone suggests that maybe I have my kernel arguments wrong,
> this is a SuSE kernel, so it is not pvops, and that is the correct
> syntax - I can add other PCI(e) devices on the same exact host (like a
> graphics card) and pass those through without any problem at all.  So
> syntax is correct.  After I boot with these arguments, I check the dmesg
> output for pciback messages:
> # dmesg|grep pciback
> [    0.000000] Command line: root=/dev/local/dom0 resume=/dev/local/swap
> pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts
> [    0.000000] Kernel command line: root=/dev/local/dom0
> resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1)
> splash=silent quiet showopts
> [    3.646529] pciback 0000:01:00.0: seizing device
> [    3.646536] pciback 0000:02:00.0: seizing device
> [    3.646540] pciback 0000:02:00.1: seizing device
> [    3.646594] pciback 0000:02:00.1: PCI INT A -> GSI 16 (level, low) ->
> IRQ 16
> [    3.646599] pciback 0000:02:00.1: PCI INT A disabled
> [    3.646637] pciback 0000:02:00.0: PCI INT A -> GSI 16 (level, low) ->
> IRQ 16
> [    3.646642] pciback 0000:02:00.0: PCI INT A disabled
> So, everything points to the devices having been removed from dom0's
> visibility and placed under the control of pciback.  A check with "xm"
> confirms this:
> # xm pci-list-assignable-devices
> 0000:02:00.0
> 0000:02:00.1
> 0000:01:00.0
> Next, I set up the pci pass through in my HVM domU configuration.  By
> the way, VTd is enabled and working fine - xm dmesg shows it as enabled,
> and, as already mentioned, I can forward through other PCI(e) devices
> without problem.  I set the following in my domU configuration:
> pci=['01:00.0','02:00.0','02:00.1']
> and then I try to start the domU:
> # xm start XP
> Error: pci: PCI Backend and pci-stub don't own device 0000:01:00.0
> Usage: xm start <DomainName>
> ???  Ummm, actually, the device is owned by pciback, so why is Xen
> telling me it's not??  I can confirm via lspci -v:
> # lspci -v
> ...
> 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI
> Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
>       Flags: bus master, fast devsel, latency 0
>       Memory at d0000000 (64-bit, prefetchable) [size=64K]
>       Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
>       I/O behind bridge: 0000d000-0000dfff
>       Memory behind bridge: fe500000-fe5fffff
>       Capabilities: [40] Power Management version 2
>       Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>       Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00
>       Capabilities: [100] Power Budgeting <?>
>       Kernel driver in use: pciback
> Any hints on this particular part of the problem would be appreciated.
> My next step was to leave this one out and just try forwarding through
> 02:00.0 and 02:00.1.  However, when I modify the domU configuration and
> remove the 01:00.0 device and try to start the domU:
> # xm start XP
> Error: Failed to assign device to IOMMU
> (0000:02:00.0@100,msitranslate=1,power_mgmt=0)
> Usage: xm start <DomainName>
> I've been able to find very, very little useful information on this
> particular error - most of the issues are with trying to forward through
> one or two of the devices on a card to one domU and the others to
> another or not at all - splitting the devices on a card between domains.
> I'm not really trying to do this, but I suspect that there's some sort
> of tie between the PCIe-PCI bridge device (01:00.0) and the actual
> serial card (02:00.0) that is causing Xen to need to have both those
> devices present in the same domain.  Of course, because of the problems
> I'm having forwarding the bridge through, that's not really working for
> me.

What do you see on your Xen serial output? I presume you cranked up logging:
loglevel=all guest_lvl=all iommu=verbose on your Xen command line.

Is there anything that shows up when you get the 'Failed to assign.." ?

Xen-devel mailing list