Re: [Xen-users] using ipoib with xcp

I've finally got to spend some time looking further into this.

I now believe the underlaying problem is that Open vSwitch is unable to connect the brib0 bridge interface to the ib0 physical interface. I suspect the cause of this to be the long MAC address of the Infiniband NICs, but so far I have not found a workaround for the issue.

These are the relevant devices for my setup:

[root@hypoxcp1 ~]# ifconfig

brib0 Link encap:Ethernet HWaddr 80:00:00:48:FE:80

inet addr:10.1.2.2 Bcast:10.1.2.255 Mask:255.255.255.0

inet6 addr: fe80::8200:ff:fe48:fe80/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:12 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:0 (0.0 b) TX bytes:720 (720.0 b)

eth0 Link encap:Ethernet HWaddr 00:30:48:CC:5C:A4

inet6 addr: fe80::230:48ff:fecc:5ca4/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:31755 errors:0 dropped:0 overruns:0 frame:0

TX packets:10544 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:4284224 (4.0 MiB) TX bytes:1433336 (1.3 MiB)

ib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00

inet addr:10.1.2.102 Bcast:10.1.2.255 Mask:255.255.255.0

UP BROADCAST MULTICAST MTU:2044 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:128

RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

xenbr0 Link encap:Ethernet HWaddr 00:30:48:CC:5C:A4

inet addr:10.1.1.2 Bcast:10.1.1.255 Mask:255.255.255.0

inet6 addr: fe80::230:48ff:fecc:5ca4/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:25892 errors:0 dropped:0 overruns:0 frame:0

TX packets:10538 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:3586527 (3.4 MiB) TX bytes:1432868 (1.3 MiB)

The ifconfig command reports the wrong (or truncated) MAC address for the ib0 device. The real address can be found using other commands:

[root@hypoxcp1 ~]# cat /sys/class/net/ib0/address

80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

[root@hypoxcp1 ~]# ip link show ib0

4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast qlen 128

link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

As mentioned earlier in this thread I've had issues with duplicate MAC addresses in /etc/ovs-vswitchd.conf, but a clean install somehow fixed that issue, so the proper MAC address is now added to the file:

[root@hypoxcp1 ~]# cat /etc/ovs-vswitchd.conf

bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

bridge.brib0.port=brib0

bridge.brib0.port=ib0

bridge.brib0.port=vif1.2

bridge.brib0.xs-network-uuids=6455dd7f-4a61-43b8-a49d-656f749c4ac6

bridge.xenbr0.mac=00:30:48:cc:5c:a4

bridge.xenbr0.port=eth0

bridge.xenbr0.port=vif1.1

bridge.xenbr0.port=xenbr0

bridge.xenbr0.xs-network-uuids=528d85a4-f582-c181-54eb-acf09ac7dcf4

bridge.xenbr1.mac=00:30:48:cc:5c:a5

bridge.xenbr1.port=eth1

bridge.xenbr1.port=vif1.0

bridge.xenbr1.port=xenbr1

bridge.xenbr1.xs-network-uuids=4f033ff5-5a56-629c-1c27-0765ba7c03bb

I'm no expert on XCP and Open vSwitch, but I believe it works something like this:

XAPI writes /etc/ovs-vswitchd.conf based on the XCP DB
XAPI starts up Open vSwitch
Open vSwitch creates the interfaces defined in /etc/ovs-vswitchd.conf

To me it seems like the MAC address for the brib0 interface is truncated, and I believe this causes Open vSwitch to not bind brib0 and ib0 together:

[root@hypoxcp1 ~]# ovs-ofctl show brib0

Apr 14 15:38:31|00001|ofctl|INFO|connecting to unix:/var/run/brib0.mgmt

features_reply (xid=0x6bb27f3f): ver:0x97, dpid:32f493d6e290

n_tables:2, n_buffers:256

features: capabilities:0x17, actions:0x3ff

LOCAL(brib0): addr:80:00:00:48:fe:80, config: 0, state:0

Apr 14 15:38:31|00002|ofctl|INFO|connecting to unix:/var/run/brib0.mgmt

get_config_reply (xid=0x9b99aaf1): miss_send_len=0

[root@hypoxcp1 ~]# ovs-ofctl show xenbr0

Apr 14 15:38:19|00001|ofctl|INFO|connecting to unix:/var/run/xenbr0.mgmt

features_reply (xid=0x836b0867): ver:0x97, dpid:f68bde598f51

n_tables:2, n_buffers:256

features: capabilities:0x17, actions:0x3ff

1(eth0): addr:00:30:48:cc:5c:a4, config: 0, state:0

current: 1GB-FD COPPER AUTO_NEG

advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG

supported: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG

LOCAL(xenbr0): addr:00:30:48:cc:5c:a4, config: 0, state:0

Apr 14 15:38:19|00002|ofctl|INFO|connecting to unix:/var/run/xenbr0.mgmt

get_config_reply (xid=0x2b665ea): miss_send_len=0

As you see the binding to ib0 is missing, and the MAC of brib0 is different from that in /etc/ovs-vswitchd.conf.

As previously stated I can communicate between XCP hosts on both brib0 and ib0 using this setup. The problem is that VIFs on the brib0 network are not reachable. I have the following IB interfaces on a single host:

ib0 - 10.1.2.2/24

brib0 - 10.1.2.102/24

vif1.2 - 10.1.2.202/24

From within the VM that uses vif1.3 I try to ping brib0 and ib0 and watch the traffic on the XCP host:

[root@hypoxcp1 ~]# tcpdump -i vif1.2

tcpdump: WARNING: vif1.2: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on vif1.2, link-type EN10MB (Ethernet), capture size 96 bytes

15:54:59.948660 arp who-has 10.1.2.102 tell 10.1.2.202

15:55:00.948643 arp who-has 10.1.2.102 tell 10.1.2.202

15:55:01.948645 arp who-has 10.1.2.102 tell 10.1.2.202

[root@hypoxcp1 ~]# tcpdump -i brib0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on brib0, link-type EN10MB (Ethernet), capture size 96 bytes

15:54:22.612723 arp who-has 10.1.2.102 tell 10.1.2.202

15:54:23.612643 arp who-has 10.1.2.102 tell 10.1.2.202

15:54:24.612642 arp who-has 10.1.2.102 tell 10.1.2.202

[root@hypoxcp1 ~]# tcpdump -i ib0

tcpdump: WARNING: arptype 32 not supported by libpcap - falling back to cooked socket

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on ib0, link-type LINUX_SLL (Linux cooked), capture size 96 bytes

The packets never reach ib0.

This setup adds the follow IP routes to the XCP host:

[root@hypoxcp1 ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 xenbr0

10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0

10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 brib0

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 brib0

0.0.0.0 10.1.1.1 0.0.0.0 UG 0 0 0 xenbr0

If I remove the ib0 route I can talk to brib0 and ib0 from vif1.3, but only on the same physical machine. Inter-host and inter-vm over network communication breaks without that route.

I also tried using "bridge" networking instead of "vswitch", but the system behaves the same way AFAICT, though the configuration is of course different.

I'm not sure what to try next. I could use the IB network for the management interface and not run any VMs on it, but please let me know if you have any idea what's wrong.

Thanks!

Trygve

On Thu, Apr 8, 2010 at 11:40 AM, Trygve Sanne Hardersen <trygve@xxxxxxxxxxxxx> wrote:

Hi

Yes, I believe the packets are lost between brib0 and ib0, so they are never sent across the network but it works on a single host.

I'll do some more testing and let you know what I find.

Thanks!

Trygve

On Thu, Apr 8, 2010 at 10:40 AM, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote:

Hi,

Is it true that you have managed to get VM <-> Host connectivity working but not VM <-> VM (across host) connectivity working?

If so then it would be interesting to use something like tcpdump to find out where the packets are going missing. If they’re entering the vswitch and then getting lost then it would be worth talking about this on the openvswitch mailing list.

Another possibility is to revert to non-vswitch based networking in dom0: try writing “bridge” to /etc/xensource/network.conf and rebooting.

Cheers,

Dave

From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Trygve Sanne Hardersen
Sent: 07 April 2010 23:02
To: Xen
Subject: [Xen-users] using ipoib with xcp

Hello,

I have been playing with the XCP for a while now, and must say I'm very exited about the technology. I had no prior experience with Xen so it has taken me a while to understand the concepts, but now I feel most important issues are solved and I've purchased some hardware to build my (tiny) cloud on.

The box is a Supermicro 1026TT-IBXF, so I have 2 x Ethernet and 1 x Infiniband (IB) NICs per node. I want to use the IB NIC to provide fast connectivity between the domUs, while the Ethernet NICs will be used for the XCP management interface and ISP connectivity.

I have successfully built OFED 1.5.1 in the XCP DDK VM and installed OFED in the XPC 0.1.1 dom0. From there I can bring up the IB network, but I'm having problems getting this to work properly within XCP virtual machines. This is what happens:

Starting out I have 2 nodes in a pool; both are clean with only lo, eth0/xenbr0 and eth1/xenbr1 configured. I run the following commands to add the IB NICs to the pool:

xe pif-scan host-uuid=NODE1

xe pif-plug uuid=NODE1_IB0

xe pif-scan host-uuid=NODE2

xe pif-plug uuid=NODE2_IB0

As expected this adds ib0/brib0 on both nodes and a single pool-wide network, but there is no connectivity between the hosts after I give brib0 an IP:

xe pif-reconfigure-ip uuid=NODE1_IB0 IP=10.1.2.2 netmask=255.255.255.0 mode=static

xe pif-reconfigure-ip uuid=NODE2_IB0 IP=10.1.2.3 netmask=255.255.255.0 mode=static

ping 10.1.2.2 --> reply

ping 10.1.2.3 --> destination host unavailable

However if I also give ib0 an IP and use this as gateway for brib0, connectivity is achieved:

ifconfig ib0 10.1.2.22 netmask 255.255.255.0

xe pif-reconfigure-ip uuid= NODE1_IB0 IP=10.1.2.2 netmask=255.255.255.0 gateway=10.1.2.22 mode=static

ifconfig ib0 10.1.2.33 netmask 255.255.255.0

xe pif-reconfigure-ip uuid= NODE2_IB0 IP=10.1.2.3 netmask=255.255.255.0 gateway=10.1.2.33 mode=static

ping 10.1.2.2 --> reply

ping 10.1.2.3 --> reply

ping 10.1.2.22 --> reply

ping 10.1.2.33 --> reply

This is very well, but when I add a VIF on the IB network to a VM it is not able to communicate through it:

xe vif-create device=2 mac=random network-uuid=IB_NET vm-uuid=NODE1_IBVM

ifconfig eth2 10.1.2.122 netmask 255.255.255.0

xe vif-create device=2 mac=random network-uuid=IB_NET vm-uuid=NODE2_IBVM

ifconfig eth2 10.1.2.133 netmask 255.255.255.0

ping 10.1.2.122 --> reply

ping 10.1.2.22 --> destination host unavailable

ping 10.1.2.2 --> destination host unavailable

ping 10.1.2.133 --> destination host unavailable

ping 10.1.2.33 --> destination host unavailable

ping 10.1.2.3 --> destination host unavailable

I believe that the problem lies somewhere in the routing table configuration. This setup gives the following routing table:

10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0

10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 brib0

If I delete and then add the brib0 route, the route order is changed:

10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 brib0

10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0

Using this the VM can talk to the host (and visa versa), but hot across the network. Connectivity between ib0/brib0 over the network is also broken.

I've also noticed that the same MAC is added to /etc/ovs-vswitchd.conf multiple times for brib0:

bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25

I've tried removing some of these but that does not seem to have any effect. My experience with IP routing and especially vswitch is limited and I'm not sure what to try from here. I've tried various configurations but no luck so far.

Note that I'm testing with 2 XCP nodes configured in a pool. I've also checked that the PIFs are in the same order on both nodes (the reference mentions this). The MTU (1500) of brib0 differs from that of ib0 (2044), but changing this does not solve the problem.

Any help is much appreciated. Thanks!

Trygve

--
HypoBytes Ltd.
Trygve Sanne Hardersen
Akersveien 24F
0177 Oslo
Norway

hypobytes.com
+47 40 55 30 25

--
HypoBytes Ltd.
Trygve Sanne Hardersen
Akersveien 24F
0177 Oslo
Norway

hypobytes.com
+47 40 55 30 25

--
HypoBytes Ltd.
Trygve Sanne Hardersen
Akersveien 24F
0177 Oslo
Norway

hypobytes.com
+47 40 55 30 25

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

WARNING - OLD ARCHIVES

xen-users

Re: [Xen-users] using ipoib with xcp