WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] [Q] How to access a Mellanox InfiniBand HCA from domU direct

To: xen users <xen-users@xxxxxxxxxxxxxxxxxxx>
Subject: [Xen-users] [Q] How to access a Mellanox InfiniBand HCA from domU directly
From: Alexander Petry <petry@xxxxxxxxxxx>
Date: Tue, 31 Jul 2007 16:55:26 +0200
Delivery-date: Tue, 31 Jul 2007 07:53:11 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Organization: Fraunhofer ITWM
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 2.0.0.5 (Macintosh/20070716)
Hi list!

I would like to access my InfiniBand HCA from a domU, so I first tried
the smart-io.hg Xen tree but it did not build right away. As a temporary
workaround I tried to let the domU directly access the HCA. The steps I
have taken have been:

1. In dom0:

# lspci -v

03:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
HCA] (rev a0)
       Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
       Flags: fast devsel, IRQ 16
       Memory at f0800000 (64-bit, non-prefetchable) [size=1M]
       Memory at f0000000 (64-bit, prefetchable) [size=8M]
       Capabilities: [40] Power Management version 2
       Capabilities: [48] Vital Product Data
       Capabilities: [90] Message Signalled Interrupts: Mask- 64bit+
Queue=0/5 Enable-
       Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
       Capabilities: [60] Express Endpoint IRQ 0

With that information I edited my grub config to exclude this particular
PCI device:

module /boot/vmlinuz-2.6.19-4-server root=/dev/sda1 ro console=tty0
console=ttyS0,9600n8 max_loop=64 pciback.hide=(0000:03:00.0)

2. After a reboot, I edited the domU config so that it includes the line:

pci = ['0000:03:00.0']

The output of lspci -v executed in the user domain is:

0000:00:00.0 InfiniBand: Mellanox Technology: Unknown device 6274 (rev a0)
       Subsystem: Mellanox Technology: Unknown device 6274
       Flags: fast devsel, IRQ 16
       Memory at f0800000 (64-bit, non-prefetchable) [size=1M]
       Memory at f0000000 (64-bit, prefetchable) [size=8M]
       Capabilities: [40] Power Management version 2
       Capabilities: [48] Vital Product Data
       Capabilities: [90] Message Signalled Interrupts: 64bit+
Queue=0/5 Enable-
       Capabilities: [84] #11 [001f]
       Capabilities: [60] #10 [0001]

The module cannot be loaded and prints the following error:

xen-domU0:~# modprobe ib_mthca
[90558.241638] ib_mthca: Mellanox InfiniBand HCA driver v0.08 (February
14, 2006)
[90558.241651] ib_mthca: Initializing 0000:00:00.0
[90559.245730] ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
[90559.245792] ib_mthca: probe of 0000:00:00.0 failed with error -11


The output of 'xm info' is:

host                   : xen-o-matic
release                : 2.6.19-4-server
version                : #2 SMP Thu Apr 5 06:07:34 UTC 2007
machine                : i686
nr_cpus                : 2
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2009
hw_caps                :
178bfbff:ebd3fbff:00000000:00000010:00002001:00000000:0000001f
total_memory           : 2030
free_memory            : 1367
xen_major              : 3
xen_minor              : 0
xen_extra              : .3-0
xen_caps               : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)
cc_compile_by          : buildd
cc_compile_domain      : buildd
cc_compile_date        : Sat Mar 24 16:01:08 UTC 2007
xend_config_format     : 2


Does anybody have an idea?

Best regards,
   Alexander Petry

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>