[Xen-users] using ipoib with xcp

[Trygve Sanne Hardersen <trygve@xxxxxxxxxxxxx>]

Hello,

I have been playing with the XCP for a while now, and must say I'm very exited about the technology. I had no prior experience with Xen so it has taken me a while to understand the concepts, but now I feel most important issues are solved and I've purchased some hardware to build my (tiny) cloud on.

The box is a Supermicro 1026TT-IBXF, so I have 2 x Ethernet and 1 x Infiniband (IB) NICs per node. I want to use the IB NIC to provide fast connectivity between the domUs, while the Ethernet NICs will be used for the XCP management interface and ISP connectivity.

I have successfully built OFED 1.5.1 in the XCP DDK VM and installed OFED in the XPC 0.1.1 dom0. From there I can bring up the IB network, but I'm having problems getting this to work properly within XCP virtual machines. This is what happens:

Starting out I have 2 nodes in a pool; both are clean with only lo, eth0/xenbr0 and eth1/xenbr1 configured. I run the following commands to add the IB NICs to the pool:

xe pif-scan host-uuid=NODE1

xe pif-plug uuid=NODE1_IB0

xe pif-scan host-uuid=NODE2

xe pif-plug uuid=NODE2_IB0

As expected this adds ib0/brib0 on both nodes and a single pool-wide network, but there is no connectivity between the hosts after I give brib0 an IP:

xe pif-reconfigure-ip uuid=NODE1_IB0 IP=10.1.2.2 netmask=255.255.255.0 mode=static

xe pif-reconfigure-ip uuid=NODE2_IB0 IP=10.1.2.3 netmask=255.255.255.0 mode=static

ping 10.1.2.2 --> reply

ping 10.1.2.3 --> destination host unavailable

However if I also give ib0 an IP and use this as gateway for brib0, connectivity is achieved:

ifconfig ib0 10.1.2.22 netmask 255.255.255.0

xe pif-reconfigure-ip uuid= NODE1_IB0 IP=10.1.2.2 netmask=255.255.255.0 gateway=10.1.2.22 mode=static

ifconfig ib0 10.1.2.33 netmask 255.255.255.0

xe pif-reconfigure-ip uuid= NODE2_IB0 IP=10.1.2.3 netmask=255.255.255.0 gateway=10.1.2.33 mode=static

ping 10.1.2.2 --> reply

ping 10.1.2.3 --> reply

ping 10.1.2.22 --> reply

ping 10.1.2.33 --> reply

This is very well, but when I add a VIF on the IB network to a VM it is not able to communicate through it:

xe vif-create device=2 mac=random network-uuid=IB_NET vm-uuid=NODE1_IBVM

ifconfig eth2 10.1.2.122 netmask 255.255.255.0

xe vif-create device=2 mac=random network-uuid=IB_NET vm-uuid=NODE2_IBVM

ifconfig eth2 10.1.2.133 netmask 255.255.255.0

ping 10.1.2.122 --> reply

ping 10.1.2.22 --> destination host unavailable

ping 10.1.2.2 --> destination host unavailable

ping 10.1.2.133 --> destination host unavailable

ping 10.1.2.33 --> destination host unavailable

ping 10.1.2.3 --> destination host unavailable

I believe that the problem lies somewhere in the routing table configuration. This setup gives the following routing table:

10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 ib0

10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 brib0

If I delete and then add the brib0 route, the route order is changed:

10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 brib0

10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 ib0

Using this the VM can talk to the host (and visa versa), but hot across the network. Connectivity between ib0/brib0 over the network is also broken.

I've also noticed that the same MAC is added to /etc/ovs-vswitchd.conf multiple times for brib0:

bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

I've tried removing some of these but that does not seem to have any effect. My experience with IP routing and especially vswitch is limited and I'm not sure what to try from here. I've tried various configurations but no luck so far.

Note that I'm testing with 2 XCP nodes configured in a pool. I've also checked that the PIFs are in the same order on both nodes (the reference mentions this). The MTU (1500) of brib0 differs from that of ib0 (2044), but changing this does not solve the problem.

Any help is much appreciated. Thanks!

Trygve

--
HypoBytes Ltd.
Trygve Sanne Hardersen
Akersveien 24F
0177 Oslo
Norway

hypobytes.com
+47 40 55 30 25

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users