Re: [Xen-devel] InfiniBand device in Xen

[Lukas Hejtmanek <xhejtman@xxxxxxxxxxx>]

On Mon, Jun 25, 2007 at 03:40:20PM +0200, Lukas Hejtmanek wrote:
> I'm trying to get working infiniband device in Xen. I have Xen v3 with 2.6.18
> kernel. I export PCI device from Dom0 into DomU. I've enabled permisive device
> in Dom0.
> 
> Do you have any advices what should I try? The device running in Dom0 works as
> expected. The problem is in DomU with exported PCI device.
> 
> It seems that the iomapped adrea returns all ones instead of reasonable 
> values.

I have found that this code is problematic in DomU. In Dom0, it works ok.

After the reset, the device respondes so it continues on the good label, but it 
seems that PCI config is not restored as it should be.

hca_header = kmalloc(256, GFP_KERNEL);
if (!hca_header) {
        err = -ENOMEM;
        mthca_err(mdev, "Couldn't allocate memory to save HCA "
                     "PCI header, aborting.\n");
        goto out;
}

for (i = 0; i < 64; ++i) {
        if (i == 22 || i == 23)
                continue;
        if (pci_read_config_dword(mdev->pdev, i * 4, hca_header + i)) {
                err = -ENODEV;
                mthca_err(mdev, "Couldn't save HCA "
                           "PCI header, aborting.\n");
                goto out;
        }
}

/* actually hit reset */
{
        void __iomem *reset = ioremap(pci_resource_start(mdev->pdev, 0) +
                                 MTHCA_RESET_OFFSET, 4);
        if (!reset) {
                err = -ENOMEM;
                mthca_err(mdev, "Couldn't map HCA reset register, "
                           "aborting.\n");
                goto out;
        }

        writel(MTHCA_RESET_VALUE, reset);
        iounmap(reset);
}

/* Docs say to wait one second before accessing device */
msleep(1000);

/* Now wait for PCI device to start responding again */
{
        u32 v;
        int c = 0;

        for (c = 0; c < 100; ++c) {
                if (pci_read_config_dword(bridge ? bridge : mdev->pdev, 0, &v)) 
{
                        err = -ENODEV;
                        mthca_err(mdev, "Couldn't access HCA after reset, "
                                        "aborting.\n");
                        goto out;
                }

                if (v != 0xffffffff)
                        goto good;

                msleep(100);
        }

        err = -ENODEV;
        mthca_err(mdev, "PCI device did not come back after reset, "
        "aborting.\n");
}

good:
devctl = hca_header[(hca_pcie_cap + PCI_EXP_DEVCTL) / 4];
if (pci_write_config_word(mdev->pdev, hca_pcie_cap + PCI_EXP_DEVCTL, devctl)) {
        err = -ENODEV;
        mthca_err(mdev, "Couldn't restore HCA PCI Express "
                           "Device Control register, aborting.\n");
        goto out;
}
linkctl = hca_header[(hca_pcie_cap + PCI_EXP_LNKCTL) / 4];
if (pci_write_config_word(mdev->pdev, hca_pcie_cap + PCI_EXP_LNKCTL, linkctl)) {
        err = -ENODEV;
        mthca_err(mdev, "Couldn't restore HCA PCI Express "
                                  "Link control register, aborting.\n");
        goto out;
}

for (i = 0; i < 16; ++i) {
        if (i * 4 == PCI_COMMAND)
                continue;

        if (pci_write_config_dword(mdev->pdev, i * 4, hca_header[i])) {
                err = -ENODEV;
                mthca_err(mdev, "Couldn't restore HCA reg %x, "
                                  "aborting.\n", i);
                goto out;
        }
}

if (pci_write_config_dword(mdev->pdev, PCI_COMMAND, hca_header[PCI_COMMAND / 
4])) {
        err = -ENODEV;
        mthca_err(mdev, "Couldn't restore HCA COMMAND, "
                          "aborting.\n");
        goto out;
}

-- 
Lukáš Hejtmánek

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel