WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] Re: HVM DomU, msi_translate=0, MSI/MSI-X PCI passthrough fails.
From: Dietmar Hahn <dietmar.hahn@xxxxxxxxxxxxxx>
Date: Wed, 8 Dec 2010 14:48:20 +0100
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>, yunhong.jiang@xxxxxxxxx, "JBeulich@xxxxxxxxxx" <JBeulich@xxxxxxxxxx>, yuan.b.liu@xxxxxxxxx, anthony.perard@xxxxxxxxxx, Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>, Mark Adams <mark@xxxxxxxxxxxxxxxxxx>
Delivery-date: Wed, 08 Dec 2010 05:49:59 -0800
Dkim-signature: v=1; a=rsa-sha256; c=simple/simple; d=ts.fujitsu.com; i=dietmar.hahn@xxxxxxxxxxxxxx; q=dns/txt; s=s1536b; t=1291816104; x=1323352104; h=from:to:subject:date:cc:references:in-reply-to: mime-version:message-id; z=From:=20Dietmar=20Hahn=20<dietmar.hahn@xxxxxxxxxxxxxx> |To:=20xen-users@xxxxxxxxxxxxxxxxxxx|Subject:=20Re:=20[Xe n-users]=20Re:=20HVM=20DomU,=20msi_translate=3D0,=20MSI/M SI-X=20PCI=20passthrough=20fails.|Date:=20Wed,=208=20Dec =202010=2014:48:20=20+0100|Cc:=20Mark=20Adams=20<mark@cam pbell-lange.net>,=0D=0A=20Konrad=20Rzeszutek=20Wilk=20<ko nrad.wilk@xxxxxxxxxx>,=0D=0A=20"xen-devel@xxxxxxxxxxxxxxx .com"=20<xen-devel@xxxxxxxxxxxxxxxxxxx>,=0D=0A=20Stefano =20Stabellini=20<stefano.stabellini@xxxxxxxxxxxxx>,=0D=0A =20yunhong.jiang@xxxxxxxxx,=0D=0A=20"JBeulich@xxxxxxxxxx" =20<JBeulich@xxxxxxxxxx>,=0D=0A=20yuan.b.liu@xxxxxxxxx, =0D=0A=20anthony.perard@xxxxxxxxxx|References:=20<2010111 2222215.GC26189@xxxxxxxxxxxx>=20<20101129163635.GA20417@d umpdata.com>=20<20101208125855.GA26222@xxxxxxxxxxxxxxxxxx >|In-Reply-To:=20<20101208125855.GA26222@xxxxxxxxxxxxxxxx et>|MIME-Version:=201.0|Message-Id:=20<201012081448.20935 .dietmar.hahn@xxxxxxxxxxxxxx>; bh=rBiHJbK3A1JVbg6hy6grfHQFSOWV9BnciBQj310UcOU=; b=ATd8rGskjQKAVSs01SBhzB9XvXPOyAw/+O/AkRVZr/SU4TzgDV/R8+Hl vfdDmaFiOZtfSS2pNCyPWIflmP9S9AHvh7a40K49kjZcOje5JAEgQOpBH pMmT8CwycvmByXMfQKDZDXOSzI2uIppoaCaBoR5fkfDo16Mf8zTqF2apg 75AtbQx7gpQ1pZlIokzfix2wFwFfDHCkDR4BuTuNG4Wreppk6d6Zwiy47 G8jplgezKC1B7/6SgwytKJz/iLCjU;
Domainkey-signature: s=s1536a; d=ts.fujitsu.com; c=nofws; q=dns; h=X-SBRSScore:X-IronPort-AV:Received:X-IronPort-AV: Received:Received:From:To:Subject:Date:User-Agent:Cc: References:In-Reply-To:X-KMail-Markup:MIME-Version: Content-Type:Message-Id; b=b62iH2c3ritpWJR5A6DqNWLyGxGvL129upxBg9x3C0go69dWpX53E8Xs v3WkYBNqaHY060KsLNBQ75tubh8gZkv88Ct6murh+OSdYN6jxwNzkZaCA bIk9MQu9rBxC/E+yqbWL4ME5UdlAH4KrQJC1WUC3TpWBnDWRECqd7x2iJ gB1gqmOhJAlm2NVCj3V8kBhW4wgX/Awb8gey/t3KUPxJ3IhAk7Qynby0Z hIX2zRGugqsLULkAgtP1iRIYbXEnS;
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20101208125855.GA26222@xxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <20101112222215.GC26189@xxxxxxxxxxxx> <20101129163635.GA20417@xxxxxxxxxxxx> <20101208125855.GA26222@xxxxxxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: KMail/1.13.5 (Linux/2.6.34.7-0.5-xen; KDE/4.5.4; x86_64; ; )

Hi,

Am 08.12.2010 schrieb "Mark Adams <mark@xxxxxxxxxxxxxxxxxx>":

> Hi - Apologies to top post this, but after alot of testing, I believe

> there must be an issue with IRQ's going missing between domU and dom0.

> Unfortunately I have no data to prove this!

>

> With msitranslate=0 as detailed below, and pci=nomsi in the guest kernel

> grub config, all 3 NIC's appear OK in the domU however I still had

> issues with the red-fone ISDN box. The interrupts were showing correctly

> (2000/s) in the domU but communication to the device via the NIC was

> still being interrupted (as shown in the asterisk console)Note that to

> get the igb driver to allow this many interrupts, the

> InterruptThrottleRate was set to 0. The same config (red-fone box,

> asterisk etc) works fine with a physical server.

>

> There is also the additional issue that I could not get the passthrough

> NIC's to show correctly when I also had a bridge setup.

>

> Throughout my testing however, I could not get the machine to crash.

>

> Not sure where to go with this one. For now we are keeping our VoIP

> servers physical when ISDN connections are required.

Today I did some tests with xen-unstable and found these problems too.

I tried to passthrough 2 pci cards and got some error messages on the xen

xonsole and in the qemu logs.

With msitranslate=0 and pci=nomsi I got the soundcard working in a domU linux

but it doesn't help on windows.

I attached the logs from the xen serial console and the qemu logs.

Thanks!

Dietmar.

>

> Regards,

> Mark

>

> On Mon, Nov 29, 2010 at 11:36:35AM -0500, Konrad Rzeszutek Wilk wrote:

> > >

> > > In my new test setup, I have seen some strange behaviour. 1 of the HVM's

> > > (with identical config in dom0 and domU) suddenly would not allow the

> > > igb driver to be loaded in domU, even though the device was visible in

> >

> > Let's create a new thread for this other issue.

> >

> > > lspci. Shutting the machine down, removing the power cord, waiting 5

> > > seconds then plugging it in again corrected that issue - Is this

> > > possibly a motherboard bug? I have also disabled the SR-IOV

> > > functionality in the BIOS incase this is causing any issues.

> > >

> > > In addition, to try to correct the MSI issue noted above, I have changed

> > > my pci= line to the following:

> > >

> > > pci=[ '08:00.0,msitranslate=0', '08:00.1,msitranslate=0' ]

> >

> > With the msi_translate=1 turned on the DomU HVM guests did work, right?

> >

> > >

> > > This has stopped the "already in use on device" log, and the devices

> > > appear to show correctly in the domU. Is it safe to disable

> > > msitranslate? as I understand it, its for allowing multifunction devices

> > > to be seen as such in domU. Is that correct?

> > >

> > > I haven't been able to reproduce the dropped raid issue yet, but I am

> > > awaiting delivery of the Red-Fone boxes (ISDN VoIP) which seem to cause

> > > this due to their very high interrupt usage (2000 per second).

> >

> > OK.

> > >

> > > In the mean time, I can see the following in the qemu-dm logs now with

> > > the msitranslate=0 enabled. Is it anything to worry about?

> >

> > Well, the "Error" ones are pretty bad, thought I am having a hard time

> > understanding what it means. Lets copy some of the QEMU folks on this.

> >

> > > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:05.0][Offset:14h][Length:4]

> > > pt_ioport_map: e_phys=ffff pio_base=e880 len=32 index=2 first_map=0

> > > pt_ioport_map: e_phys=c220 pio_base=e880 len=32 index=2 first_map=0

> > > pt_pci_write_config: Warning: Guest attempt to set address to unused Base Address Register. [00:06.0][Offset:14h][Length:4]

> > > pt_ioport_map: e_phys=ffff pio_base=ec00 len=32 index=2 first_map=0

> > > pt_ioport_map: e_phys=c240 pio_base=ec00 len=32 index=2 first_map=0

> > > pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59

> > > pt_msix_update_one: Update msix entry 1 with pirq 4e gvec 61

> > > pt_msix_update_one: Update msix entry 2 with pirq 4d gvec 69

> > > pt_msix_update_one: Update msix entry 3 with pirq 4c gvec 71

> > > pt_msix_update_one: Update msix entry 4 with pirq 4b gvec 79

> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 0 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 1 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 2 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 3 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.

> > > pci_msix_writel: Error: Can't update msix entry 4 since MSI-X is already function.

> > >

> > > >

> > > > Not yet. Need to serial log of the Linux kernel and the Xen hypervisor when your

> > > > machine is toast. I mentioned in the previous email the key sequences - look on Google

> > > > on how to pass in SysRQ if you are using a serial concentrator.

> > >

> > > I will do this when I can get the machine to crash.

> > >

> > > Best Regards,

> > > Mark

> > >

> > > _______________________________________________

> > > Xen-devel mailing list

> > > Xen-devel@xxxxxxxxxxxxxxxxxxx

> > > http://lists.xensource.com/xen-devel

>

> _______________________________________________

> Xen-users mailing list

> Xen-users@xxxxxxxxxxxxxxxxxxx

> http://lists.xensource.com/xen-users

>

>

--

Company details: http://ts.fujitsu.com/imprint.html

Attachment: xen-serial-log.txt
Description: Text document

Attachment: qemu-dm-OpenSuseHVM.log
Description: Text Data

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users