WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] InfiniBand device in Xen

To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] InfiniBand device in Xen
From: Lukas Hejtmanek <xhejtman@xxxxxxxxxxx>
Date: Tue, 26 Jun 2007 13:43:19 +0200
Delivery-date: Tue, 26 Jun 2007 04:42:05 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <20070625134020.GH3867@xxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20070625134020.GH3867@xxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/1.5.13 (2006-08-11)
On Mon, Jun 25, 2007 at 03:40:20PM +0200, Lukas Hejtmanek wrote:
> I'm trying to get working infiniband device in Xen. I have Xen v3 with 2.6.18
> kernel. I export PCI device from Dom0 into DomU. I've enabled permisive device
> in Dom0.
> 
> Do you have any advices what should I try? The device running in Dom0 works as
> expected. The problem is in DomU with exported PCI device.
> 
> It seems that the iomapped adrea returns all ones instead of reasonable 
> values.

I have found that this code is problematic in DomU. In Dom0, it works ok.

After the reset, the device respondes so it continues on the good label, but it 
seems that PCI config is not restored as it should be.

hca_header = kmalloc(256, GFP_KERNEL);
if (!hca_header) {
        err = -ENOMEM;
        mthca_err(mdev, "Couldn't allocate memory to save HCA "
                     "PCI header, aborting.\n");
        goto out;
}

for (i = 0; i < 64; ++i) {
        if (i == 22 || i == 23)
                continue;
        if (pci_read_config_dword(mdev->pdev, i * 4, hca_header + i)) {
                err = -ENODEV;
                mthca_err(mdev, "Couldn't save HCA "
                           "PCI header, aborting.\n");
                goto out;
        }
}

/* actually hit reset */
{
        void __iomem *reset = ioremap(pci_resource_start(mdev->pdev, 0) +
                                 MTHCA_RESET_OFFSET, 4);
        if (!reset) {
                err = -ENOMEM;
                mthca_err(mdev, "Couldn't map HCA reset register, "
                           "aborting.\n");
                goto out;
        }

        writel(MTHCA_RESET_VALUE, reset);
        iounmap(reset);
}

/* Docs say to wait one second before accessing device */
msleep(1000);

/* Now wait for PCI device to start responding again */
{
        u32 v;
        int c = 0;

        for (c = 0; c < 100; ++c) {
                if (pci_read_config_dword(bridge ? bridge : mdev->pdev, 0, &v)) 
{
                        err = -ENODEV;
                        mthca_err(mdev, "Couldn't access HCA after reset, "
                                        "aborting.\n");
                        goto out;
                }

                if (v != 0xffffffff)
                        goto good;

                msleep(100);
        }

        err = -ENODEV;
        mthca_err(mdev, "PCI device did not come back after reset, "
        "aborting.\n");
}

good:
devctl = hca_header[(hca_pcie_cap + PCI_EXP_DEVCTL) / 4];
if (pci_write_config_word(mdev->pdev, hca_pcie_cap + PCI_EXP_DEVCTL, devctl)) {
        err = -ENODEV;
        mthca_err(mdev, "Couldn't restore HCA PCI Express "
                           "Device Control register, aborting.\n");
        goto out;
}
linkctl = hca_header[(hca_pcie_cap + PCI_EXP_LNKCTL) / 4];
if (pci_write_config_word(mdev->pdev, hca_pcie_cap + PCI_EXP_LNKCTL, linkctl)) {
        err = -ENODEV;
        mthca_err(mdev, "Couldn't restore HCA PCI Express "
                                  "Link control register, aborting.\n");
        goto out;
}

for (i = 0; i < 16; ++i) {
        if (i * 4 == PCI_COMMAND)
                continue;

        if (pci_write_config_dword(mdev->pdev, i * 4, hca_header[i])) {
                err = -ENODEV;
                mthca_err(mdev, "Couldn't restore HCA reg %x, "
                                  "aborting.\n", i);
                goto out;
        }
}

if (pci_write_config_dword(mdev->pdev, PCI_COMMAND, hca_header[PCI_COMMAND / 
4])) {
        err = -ENODEV;
        mthca_err(mdev, "Couldn't restore HCA COMMAND, "
                          "aborting.\n");
        goto out;
}

-- 
Lukáš Hejtmánek

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>