This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] Xen and Infiniband as dom0 (only)

To: Pasi Kärkkäinen <pasik@xxxxxx>
Subject: Re: [Xen-devel] Xen and Infiniband as dom0 (only)
From: Xavier Beaudouin <xb@xxxxxxxxxxx>
Date: Fri, 26 Mar 2010 11:48:35 +0100
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Fri, 26 Mar 2010 03:48:58 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20100316155643.GP1878@xxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4286181A-AA18-411C-9E16-6B95FD54A57E@xxxxxxxxxxx> <20100316155643.GP1878@xxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Hi there,

Le 16 mars 2010 à 16:56, Pasi Kärkkäinen a écrit :

> On Mon, Mar 15, 2010 at 01:29:54PM +0100, Xavier Beaudouin wrote:
>> Hello there,
>> I am trying to make infiniband working on a dom0.  To allow me fast access 
>> to storage, and fast migration between hypervisor.
>> I still don't want to do Ib as Hypervisor Accelerator and specialy what is 
>> shown on slide 12 of this paper : 
>> http://xen.org/files/Xen_Virtualization_InfiniBand_Mellanox_MKagan.pdf, but 
>> I need rdma to get fast access to storage :
>> - iSer (eg iscsi over infiniband)
>> - RDMA for NFS over RDMA or stuff like this.
>> On my host, without Xen (F12, with myoung kernel backported, xen 4.0.0-rc6 
>> and OFED 1.5.1-RC4), I can use correctly ipoib (IP over IB), but rdma is 
>> still broken.
>> Is there any people that has made dom0 IB working that can me some hints 
>> about what I have to fix ?
> Well what actually is the problem? 
> What kind of errors do you get? 

Simple :

- is the server (filer) with IB and OFED 1.5.1-rc5

  $ ibv_rc_pingpong 
  local address:  LID 0x0007, QPN 0x590406, PSN 0x1adab0
  remote address: LID 0x000c, QPN 0x0a0406, PSN 0x877182

- client is xen-6 ( :
  $ ibv_rc_pingpong
  local address:  LID 0x000c, QPN 0x0a0406, PSN 0x877182
  remote address: LID 0x0007, QPN 0x590406, PSN 0x1adab0

Message from syslogd@xen-6 at Mar 26 12:41:56 ...
 kernel:Bad pagetable: 000f [#1] SMP 

Message from syslogd@xen-6 at Mar 26 12:41:56 ...
 kernel:last sysfs file: 

dmesg on the host : 
ibv_rc_pingpong: Corrupted page table at address 7fa3eccc1010
PGD 58f1d067 PUD 58927067 PMD 53e50067 PTE fffffffffffff237
Bad pagetable: 000f [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:0f.0/0000:04:00.0/infiniband/mt
CPU 0 
Modules linked in: tun bridge stp llc nfs lockd fscache nfs_acl auth_rpcgss sunr
pc rdma_ucm ib_sdp rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa ipv6 ib_uverbs ib_
umad iw_nes libcrc32c iw_cxgb3 cxgb3 mlx4_ib mlx4_en mlx4_core microcode xen_evt
chn xenfs ib_mthca ib_mad forcedeth amd64_edac_mod i2c_nforce2 ib_core edac_core
 i2c_core edac_mce_amd xfs exportfs dm_multipath pata_acpi ata_generic pata_amd 
[last unloaded: scsi_wait_scan]
Pid: 4402, comm: ibv_rc_pingpong Tainted: G        W
c12.x86_64 #1 empty
RIP: e033:[<00007fa3ec4a3f42>]  [<00007fa3ec4a3f42>] 0x7fa3ec4a3f42
RSP: e02b:00007fff80f16ca0  EFLAGS: 00010202
RAX: 000000000a000001 RBX: 0000000000000000 RCX: 00007fa3eccc1000
RDX: 0206040a0a000001 RSI: 00007fa3eccbb008 RDI: 0000000000000000
RBP: 0000000001bdc980 R08: 0000000000000000 R09: 0000000000000007
R10: 0000000000590406 R11: 0000000000000246 R12: 00007fa3ec49f010
R13: 0000000001be3170 R14: 0000000000000001 R15: 00007fff80f17000
FS:  00007fa3eccb8700(0000) GS:ffff880003e7d000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fa3eccc1010 CR3: 0000000002a12000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ibv_rc_pingpong (pid: 4402, threadinfo ffff88004f9ea000, task ffff88005b

RIP  [<00007fa3ec4a3f42>] 0x7fa3ec4a3f42
 RSP <00007fff80f16ca0>
---[ end trace 4eaa2a86a8e2da24 ]---

Note that I have same problems with intel platform as well...

ipoib works, but I really like to get rid of these errors....


Xavier Beaudouin - xb@xxxxxxxxxxx
GPG Fingerprints : A6B2 D563 F93B A3AF C08A  CBAC 6BC6 79EB DCC9 9867

Xen-devel mailing list

<Prev in Thread] Current Thread [Next in Thread>