WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] using ipoib with xcp

To: Xen <xen-users@xxxxxxxxxxxxxxxxxxx>
Subject: [Xen-users] using ipoib with xcp
From: Trygve Sanne Hardersen <trygve@xxxxxxxxxxxxx>
Date: Thu, 8 Apr 2010 00:01:38 +0200
Delivery-date: Wed, 07 Apr 2010 15:02:53 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Hello,

I have been playing with the XCP for a while now, and must say I'm very exited about the technology. I had no prior experience with Xen so it has taken me a while to understand the concepts, but now I feel most important issues are solved and I've purchased some hardware to build my (tiny) cloud on.

The box is a Supermicro 1026TT-IBXF, so I have 2 x Ethernet and 1 x Infiniband (IB) NICs per node. I want to use the IB NIC to provide fast connectivity between the domUs, while the Ethernet NICs will be used for the XCP management interface and ISP connectivity.

I have successfully built OFED 1.5.1 in the XCP DDK VM and installed OFED in the XPC 0.1.1 dom0. From there I can bring up the IB network, but I'm having problems getting this to work properly within XCP virtual machines. This is what happens:

Starting out I have 2 nodes in a pool; both are clean with only lo, eth0/xenbr0 and eth1/xenbr1 configured. I run the following commands to add the IB NICs to the pool:

xe pif-scan host-uuid=NODE1
xe pif-plug uuid=NODE1_IB0
xe pif-scan host-uuid=NODE2
xe pif-plug uuid=NODE2_IB0

As expected this adds ib0/brib0 on both nodes and a single pool-wide network, but there is no connectivity between the hosts after I give brib0 an IP:

xe pif-reconfigure-ip uuid=NODE1_IB0 IP=10.1.2.2 netmask=255.255.255.0 mode=static
xe pif-reconfigure-ip uuid=NODE2_IB0 IP=10.1.2.3 netmask=255.255.255.0 mode=static
ping 10.1.2.2 --> reply
ping 10.1.2.3 --> destination host unavailable

However if I also give ib0 an IP and use this as gateway for brib0, connectivity is achieved:

ifconfig ib0 10.1.2.22 netmask 255.255.255.0
xe pif-reconfigure-ip uuid= NODE1_IB0 IP=10.1.2.2 netmask=255.255.255.0 gateway=10.1.2.22 mode=static
ifconfig ib0 10.1.2.33 netmask 255.255.255.0
xe pif-reconfigure-ip uuid= NODE2_IB0 IP=10.1.2.3 netmask=255.255.255.0 gateway=10.1.2.33 mode=static
ping 10.1.2.2 --> reply
ping 10.1.2.3 --> reply
ping 10.1.2.22 --> reply
ping 10.1.2.33 --> reply

This is very well, but when I add a VIF on the IB network to a VM it is not able to communicate through it:

xe vif-create device=2 mac=random network-uuid=IB_NET vm-uuid=NODE1_IBVM
ifconfig eth2 10.1.2.122 netmask 255.255.255.0
xe vif-create device=2 mac=random network-uuid=IB_NET vm-uuid=NODE2_IBVM
ifconfig eth2 10.1.2.133 netmask 255.255.255.0
ping 10.1.2.122 --> reply
ping 10.1.2.22 --> destination host unavailable
ping 10.1.2.2 --> destination host unavailable
ping 10.1.2.133 --> destination host unavailable
ping 10.1.2.33 --> destination host unavailable
ping 10.1.2.3 --> destination host unavailable

I believe that the problem lies somewhere in the routing table configuration. This setup gives the following routing table:

10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 ib0
10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 brib0

If I delete and then add the brib0 route, the route order is changed:

10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 brib0
10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 ib0

Using this the VM can talk to the host (and visa versa), but hot across the network. Connectivity between ib0/brib0 over the network is also broken.

I've also noticed that the same MAC is added to /etc/ovs-vswitchd.conf multiple times for brib0:

bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

I've tried removing some of these but that does not seem to have any effect. My experience with IP routing and especially vswitch is limited and I'm not sure what to try from here. I've tried various configurations but no luck so far.

Note that I'm testing with 2 XCP nodes configured in a pool. I've also checked that the PIFs are in the same order on both nodes (the reference mentions this). The MTU (1500) of brib0 differs from that of ib0 (2044), but changing this does not solve the problem.

Any help is much appreciated. Thanks!

Trygve

--
HypoBytes Ltd.
Trygve Sanne Hardersen
Akersveien 24F
0177 Oslo
Norway

hypobytes.com
+47 40 55 30 25
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
<Prev in Thread] Current Thread [Next in Thread>