Hi Jean / Ross,
I have been spending a lot of time trying to make this work.
At the moment I am booting with the XenClientInitiative (2.6.18) all downloaded
and compiled from source. (took some time to understand the buildroot and
toolchain due to limited documentation) but I'm there now !
Scenarios tried:
Under XCI:
echo -n "sbr=0000:03:00.0" >/sys/bus/pci/drivers/pciback/resets
echo -n "d3r=0000:03:00.0" >/sys/bus/pci/drivers/pciback/resets
echo -n "noflr=0000:03:00.0" >/sys/bus/pci/drivers/pciback/resets
The output (on tty3) each time shows "PCI FLR not performed for device 03:00.0"
when I try:
echo -n "0000:03:00.0" >/sys/bus/pci/drivers/pciback/do_flr
Also get "PCI FLR not performed for device 03:00.0"
pciback and qemu all seems to map all i/o memory and irq through correctly.
Please see the lspci at earlier post:
http://lists.xensource.com/archives/html/xen-devel/2009-04/msg00218.html
(same scenario in both XCI and xen-unstable)
I have also read through the pci.py and pciiy.py files under xen-unstable. The
function do_FLR() seems to attempt all the FLR methods when the device is
disconnected, so I tried attach/detach under xen-unstable but there is no
indication in DEBUG messages about the FLR being performed or failing at all !?
Do you know how I can dump out the pci dev structure to determine if the FLR
for this card is being detected/enumerated on the PCIe capabilties ? Maybe this
would be a good addition to the DEBUG output for the future anyway, to help
diagnose FLR capabilities...
Thanks,
Tim
P.S I have now also purchased a Nvidia Geforce 9500GT and attempted to setup
the configuration from here:
http://www.nabble.com/Successful-PCIe-Graphics-VT-d-Passthrough-to-Win32-DomU,-Q35-chipset-td21671745.html
I think the 8800 GTS has a better chance of working, although they both show
FLReset- in the lspci devcap.
-----Original Message-----
From: Ross Philipson [mailto:Ross.Philipson@xxxxxxxxxx]
Sent: 30 March 2009 17:43
To: Tim Moore; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough
to <any> DomU
FLRs allow you to do a full reset of the device at the slot/function level
putting the device is an uninitialized and quiescent state. For PCI devices it
should show up as PCI capability 0x13. PCIe support for it is indicated in the
PCI Express Capabilities -> Device Capabilities Register (bit 28). For both
types you write the appropriate FLR command bit to do the reset. This is more
or less a requirement for passing devices between domains (from what I have
been told). The code to perform the actions is in the xend python tools if you
want more details on it.
Thanks
Ross
-----Original Message-----
From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
[mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Tim Moore
Sent: Monday, March 30, 2009 11:34 AM
To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough
to <any> DomU
I agree, if any I was expecting the Nvidia card to work the most ..
One interesting thing is that there is a fan on the Nvidia that spins down when
the bios is loaded, with Windows it is evident when the bios is reset/loaded -
just before the GUI is displayed.
When booting Xen the fan stays high / fast as the pciback is loaded to prevent
Dom0 taking ownership.. boot the DomU (WinXP) and the fan spins down the same
as booting a native Win32 host .. aka I assume Bios/Card is loaded/Reset here ..
Pls can you explain FLR's ?
-----Original Message-----
From: Jean Guyader [mailto:jean.guyader@xxxxxxxxx]
Sent: 30 March 2009 15:57
To: Tim Moore
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough
to <any> DomU
Hi Tim,
In the native case there is only one mapping, there isn't any
remapping ones the bios has been executed. It is different in the hvm
scenario.
Anyway, the nvidia graphic card should works fine as a secondary, side
by side with the cirrus. I don't know if this card support FLR's but
that would be good to have a look.
Jean
2009/3/30 Tim Moore <timothy.moore@xxxxxxxxxxx>:
>
> Hi Jean,
>
> I can understand the issue with the graphics card drivers, and I can see how
> that could be able problem. I am using a very modern nvidia and ati card for
> testing, Alex is also using a couple of other different varieties so we are
> able to test with a few different scenarios...
>
> How would this work under an MS kernel? these manufacturers do support dual
> cards and I have also tested my ATI/Nvidia host with a Windows OS and both
> cards are loaded and work fine. This says to me that the drivers can deal
> with PCI remapping of some sorts?
>
> Cheers,
> Tim
>
> -----Original Message-----
> From: Jean Guyader [mailto:jean.guyader@xxxxxxxxx]
> Sent: 30 March 2009 15:38
> To: Tim Moore
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
> Subject: Re: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough
> to <any> DomU
>
> Hi Tim,
>
> Most of the time the problam doesn't come from the VT-d code but from
> the graphic card device drivers. Most of them doesn't really like PCI
> Bar remapping and try to use the old one.
> For instance the intel igfx has never been designed to be used as a
> secondary display adapter or even with another Host bridge than the
> one on the motherboard.
>
> Jean
>
> 2009/3/30 Tim Moore <timothy.moore@xxxxxxxxxxx>:
>> Hi Alex,
>>
>>
>>
>> When you say you had success with the onboard igfx, how much success? Have
>> you ever had a passthrough VGA display from a DomU ?
>>
>>
>>
>> I have also seen the write-up too although a lot of work has been done on
>> the VT-d support lately and I`m hoping to raise this again on the developer
>> list and start to document some of the problems/issues so as we can find a
>> way do solve it !
>>
>>
>>
>> Thanks for the reply! It still leaves this issue unsolved though ;(
>>
>>
>>
>> Tim
>>
>>
>>
>> ________________________________
>>
>> Subject: Re: [Xen-devel] Memory mapping for PEG/PCIe Graphics Passthrough to
>>
>> <any> DomU
>>
>> From: Alexia Benington <alexbenington@xxxxxxxxx>
>>
>> To: Tim Moore <timothy.moore@xxxxxxxxxxx>
>>
>> Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
>>
>>
>>
>> Hi,
>>
>>
>>
>> I've been trying to get this to work for some months, albeit not
>>
>> full-time. I concur that many would like to see it working. Some have
>>
>> posted that they managed to get it to work correctly, however, they
>>
>> almost always never reply to queries on how they got it to work, or
>>
>> their observations.
>>
>>
>>
>> I did manage to get rid of the yellow exclamation mark. You'll need to
>>
>> disable the emulated VGA bios, pretty much as described here
>>
>> http://lists.xensource.com/archives/html/xen-devel/2008-12/msg00474.html.
>>
>>
>>
>> Even after doing so, some of the registers are not directly passed
>>
>> through. See http://lists.xensource.com/archives/html/xen-devel/2009-02/msg=
>>
>> 01131.html.
>>
>> I'm not sure if that may affect anything.
>>
>>
>>
>> I've experimented with an Intel DQ45CB, nVidia Quadro NVS 290, nVidia
>>
>> GF9500, ATI Radion HD 2600XT. The best results I get are when using
>>
>> the onboard igfx as the primary, which is also being passed through.
>>
>>
>>
>> There is also a writeup which you might want to take a look
>>
>> http://staff.science.uva.nl/~delaat/sne-2008-2009/p22/report.pdf.
>>
>>
>>
>> I'm also trying to get DomU's output to the display.
>>
>>
>>
>> -Alex
>>
>>
>>
>> 2009/3/30 Tim Moore <timothy.moore@xxxxxxxxxxx>:
>>
>>> Hi
>>
>>>
>>
>>>
>>
>>>
>>
>>> I have been testing xen-unstable.hg (29/03/09) over the past few days ..
>>
>>> trying to enable passthrough for VGA.
>>
>>>
>>
>>>
>>
>>>
>>
>>> I am using Debian 5.0 (lenny) x64 and compiling from source each time. Tr=
>>
>> ied
>>
>>> both linux-2.6.18.8 and linux-2.6.27.5 from Xenbits =85
>>
>>>
>>
>>>
>>
>>>
>>
>>> Hardware is Nvidia Geforce 8800 512mb GTS PCIe on Intel X58 (ASUS P6T
>>
>>> Deluxe) with Core i7 920 CPU.
>>
>>>
>>
>>> (had to hack the xen/drviers/pci/passthrough/vtd/dmar.c (line 388) and
>>
>>> remove the DMAR bail because of bad RMRR tables.
>>
>>>
>>
>>> I also have ATI X300 PCIe for Dom0 Console.
>>
>>>
>>
>>>
>>
>>>
>>
>>> Using Xen parameter: iommu=3Dpassthrough,pv
>>
>>>
>>
>>> Using Dom) parameter: pciback.hide=3D(=91=85=92)
>>
>>>
>>
>>>
>>
>>>
>>
>>> I compiled pciback as embedded in the kernel and I have tried many differ=
>>
>> ent
>>
>>> scenarios:
>>
>>>
>>
>>>
>>
>>>
>>
>>> 1) Nvidia Primary (hidden with pciback.hide=3D kernel param) (no ATI card
>>
>>> installed)
>>
>>>
>>
>>> 2) ATI Primary, with Nvidia hidden (with pciback.hide=3D kernel param)
>>
>>>
>>
>>> 3) all scenarios with pciback as module and using echo devn >
>>
>>> /sys/bus/pci/drivers/pciback/new_slot and bind
>>
>>>
>>
>>>
>>
>>>
>>
>>> In all scenarios I can successfully see the correct PCI device in the
>>
>>> DomU(s), I have tried primarily Windows XP 32bit SP3 and also:
>>
>>> linux-2.6.18.8 and linux-2.6.27.5 PV and non PV kernels as DomU and
>>
>>> (although I didn=92t load a driver) the lspci =96vvv showed the devices.
>>
>>>
>>
>>>
>>
>>>
>>
>>> Now, I progressed to installing drivers under the Win32 DomU against the
>>
>>> Nvidia card, windows correctly identified the device and installed the
>>
>>> driver! (latest from nvidia.com). As we all know =96 Windows wanted a reb=
>>
>> oot-
>>
>>> beforehand a quick check of the device and it has a Yellow exclamation
>>
>>> indicating reboot required and resources unavailable.
>>
>>>
>>
>>> Reboot and XP starts up =85 BSOD .. nv4 driver fails .. L
>>
>>>
>>
>>>
>>
>>>
>>
>>> Tried with the ATI X300 Primary (no NVidia connected) and went through th=
>>
>> e
>>
>>> same steps. Windows XP now boots without crash although the ATI device st=
>>
>> ill
>>
>>> shows a yellow exclamation and insufficient resources.
>>
>>>
>>
>>>
>>
>>>
>>
>>> To summarise, pciback functions as expected for PCIe in Dom0 and DomU see=
>>
>> s
>>
>>> the device. All tools work fine (xm pci-attach / pci-detach / pci-list) a=
>>
>> nd
>>
>>> devid show and are managed correctly.
>>
>>>
>>
>>>
>>
>>>
>>
>>> I have read all that there is on this subject and have come to the
>>
>>> conclusion that this is a problem with qemu-dm and the memory mapping for
>>
>>> the new Video device, the Cirrus or stdvga card is always present which I
>>
>>> believe may be causing the problem. I have tried nographics=3D1 but qemu-=
>>
>> dm
>>
>>> still maps the Video BIOS.
>>
>>>
>>
>>>
>>
>>>
>>
>>> Then I noticed that qemu-dm has the new options for =93-vga none=94 .. so=
>>
>> I
>>
>>> built a wrapper script for qemu-dm to launch with this param. Unfortuntat=
>>
>> ely
>>
>>> the DomU doesn=92t startup and the qemu log does not show any mention of =
>>
>> the
>>
>>> Video memory ! so I believe the switch works, but the DomU then fails to
>>
>>> boot L
>>
>>>
>>
>>>
>>
>>>
>>
>>> To solve this and make it usable I think some development of qemu with
>>
>>> regards to the Cirrus VGA and memory management when using more than 1 vg=
>>
>> a
>>
>>> controller is necessary in order to remap the Cirrus or secondary vga
>>
>>> adapters in memory correctly. (its also difficult to configure a Windows
>>
>>> machine for a second vga when the primary is disabled lol.)
>>
>>>
>>
>>>
>>
>>>
>>
>>> Also, I have run strace using a qemu-dm wrapper (very verbose) and analys=
>>
>> ed
>>
>>> the results =96nothing seems obvious, but i`m not an expert by far =85.
>>
>>>
>>
>>>
>>
>>>
>>
>>> Windows has been able to support multiple VGA cards for a long time; I
>>
>>> assume that MS is remapping these somehow?
>>
>>>
>>
>>>
>>
>>>
>>
>>> I have read in 2 different places that this has worked for others before
>>
>>> (without the legacy patch for igfx) and with xen-unstable.hg .. why does =
>>
>> it
>>
>>> not work for me ?
>>
>>>
>>
>>>
>>
>>>
>>
>>> Any help / direction to further debug would be helpful .. I think many
>>
>>> people would like to get this one working once and for all!
>>
>>>
>>
>>>
>>
>>>
>>
>>> Tim
>>
>>>
>>
>>> _______________________________________________
>>
>>> Xen-devel mailing list
>>
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>
>>> http://lists.xensource.com/xen-devel
>>
>>>
>>
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-devel
>>
>>
>
>
>
> --
> Jean Guyader
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>
--
Jean Guyader
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home