WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

To: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem
From: Bruce Edge <bruce.edge@xxxxxxxxx>
Date: Wed, 13 Oct 2010 15:08:14 -0700
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Wed, 13 Oct 2010 15:09:04 -0700
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=BuFHt9my3Zg3skS/0sj8zKiNFrrYrzSitke9CMIvu7Y=; b=EbUIyRJPqGreEMqvxfxgxgnfl7kA/mF6nax32DSVbjN2vZ79+Ld4vQDlzvMAtpNouE DERFjbvDfYY+eYpTB5SR0NDyX19rU/JAYQwUuOTFtVvgfQ94egnPgYeVqRabm7b1d4sk DfVRrAV0I5uYDTJZX6EYr4bo0NsPrs9Q47tZ4=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=hAjjCL63Yd2HvYiqfl07P7ZeDrYtcgsE6W85oQ3KLirf7T1sU9NvR1YrHsjrbsWTKE cD9zOIruQk6aR/YZ8649REqHWPDEiszYHAiVCjsAUfkTQOUbX/Sby8xuZQcJuQbpwy9q Lz7I5wrvMVfAPXgnhSRGUolrDYPjw4SK2U84k=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <AANLkTimKEBQcC4nPqAHHcorxL+ygLOrkWZVyC6gDC7+x@xxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <AANLkTi=G7CTuVP2=b18JjwFsCb_fvk8hoMWAngYUKB1Y@xxxxxxxxxxxxxx> <20101001211111.GA18244@xxxxxxxxxxxx> <AANLkTinMf=BL2MoXT1CiCkp85Y8WKPTc_Jgubdg7gmMK@xxxxxxxxxxxxxx> <20101011214605.GA18201@xxxxxxxxxxxx> <AANLkTimYYXUtkf3w32fDMdJq8teE5HLiE+LND5v59iKK@xxxxxxxxxxxxxx> <20101013214613.GA7153@xxxxxxxxxxxx> <AANLkTimKEBQcC4nPqAHHcorxL+ygLOrkWZVyC6gDC7+x@xxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
On Wed, Oct 13, 2010 at 3:00 PM, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:
> On Wed, Oct 13, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@xxxxxxxxxx> wrote:
>> On Wed, Oct 13, 2010 at 02:36:47PM -0700, Bruce Edge wrote:
>>> On Mon, Oct 11, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@xxxxxxxxxx> wrote:
>>> > On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge wrote:
>>> >> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
>>> >> <konrad.wilk@xxxxxxxxxx> wrote:
>>> >> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>>> >> >> One of our developers who is working on a tachyon driver is
>>> >> >> complaining that the pvops domU kernel is not working for these MSI
>>> >> >> interrupts.
>>> >> >> This is using the current head of xen/2.6.32.x on both a single
>>> >> >> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>>> >> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
>>> >> >
>>> >> >
>>> >> > I just checked on my SuperMicro X8DTN, this combination
>>> >> >  - For Dom0, git commit fe999249 (2.6.32.18)
>>> >> >  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
>>> >> >  - For Hypervisor I used cs 21976, but found that the latest (22155) 
>>> >> > works too
>>> >> >
>>> >> > with which where I passed in PCI devices with legacy IRQ, MSI, and 
>>> >> > MSI-X. I tried
>>> >> > a combination of doing this with IOMMU (VT-d) and without - both cases 
>>> >> > these devices:
>>> >> >
>>> >> > 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB 
>>> >> > UHCI Controller #1 (rev 02)
>>> >> > 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB 
>>> >> > UHCI Controller #2 (rev 02)
>>> >> > 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB 
>>> >> > UHCI Controller #3 (rev 02)
>>> >> > 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 
>>> >> > EHCI Controller #1 (rev 02)
>>> >> > 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit 
>>> >> > Ethernet Controller (Copper) (rev 06)
>>> >> > 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network 
>>> >> > Connection (rev 02)
>>> >> >
>>> >> > worked just fine (either defining pci=["..."] or just using 
>>> >> > pci-attach).
>>> >> >
>>> >> > But if I use the latest xen/next or xen/stable-2.6.32.x it does not 
>>> >> > look
>>> >> > that happy :-(
>>> >> >
>>> >> >
>>> >>
>>> >> Konrad,
>>> >> To try eliminate the remaining differences here, could you post your
>>> >> dom0/domU config files?
>>> >
>>> > Sure. See attached
>>>
>>> Konrad,
>>> That made a big difference. Looks much better now. It's been kicked
>>> over to several developers who have each got our tachyon driver
>>> working a little bit better.
>>
>> Hmm, I didn't really try to do anything fancy with the configs. Any
>> inklings of what config option might have caused all this headache?
>
> I had pruned out a lot of stuff I didn't need from the kernel. I may
> have been a bit overzealous. Also, we were using the same kernel for
> dom0 and domU. It may be that having all the dom0 baggage in the domU
> has some unexpected consequences. Very vague I know, sorry. If I have
> time I'll try narrow down the diffs that break it.
>
>>
>>>
>>> Now the sticking point is an apparent limitation on the amount of
>>> memory one can request using pci_map_single. It appears that we can
>>> only ask for 256K or less.  We need a 2MB DMA buffer.
>>
>> You can't use SG buffers? And chain them together and provide them
>> to the device? A lot of other drivers do this ...
>
> Perhaps. I don't know. I'm not the driver author. I think that it
> worked using 1 giant buffer before so there was no need to use SG
> buffers.

Got some more info. It's a hardware requirement for the tachyon chip.
It uses a 2 MB block for the FC SEST index table. It's one place where
we can't use SG.
It really not a matter of laziness.

-Bruce

>
>>
>>> Is there some alternate mechanism for getting a larger physically
>>> contiguous buffer under pvops?
>>
>> Why the contiguous requirement?
>
> It may not be a  hard requirement and was just a matter of
> convenience. I'll kick that over the the tachyon guys and see what
> they say. I think it's a non-trivial exercise to switch over. All the
> internal buffer mgmt would have to change. If that's the only option,
> well, that's it. ...is it?
>
> Thanks again for your help. You've been instrumental it getting this up.
>
> -Bruce
>
>>>
>>> Thanks
>>>
>>> -Bruce
>>>
>>> >> I'd like to build the same kernels to get an apples->apples comparison.
>>> >>
>>> >> Also, could you include your grub info and domU cfg file?
>>> >>
>>> >> These may eliminate some of the remaining diffs in the configs and
>>> >> show why your's works while mine does not.
>>> >>
>>> >> Thanks
>>> >>
>>> >> -Bruce
>>> >
>>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel