WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] using ipoib with xcp

On Fri, Apr 16, 2010 at 01:14:46PM +0200, Trygve Sanne Hardersen wrote:
>    I've looked a bit into the Open vSwtich source code and it seems to me
>    like MAC addresses can only be 6 bytes, but the IB addresses are 20 bytes.
>    I'm also seeing this in the Open vSwitch log:
>    |00043|bridge|INFO|created port ib0 on bridge brib0
>    |00044|dpif|WARN|dp0: failed to add ib0 as port: Invalid argument
>    |00045|bridge|ERR|failed to add ib0 interface to dp0: Invalid argument
>    |00046|bridge|ERR|ib0 interface not in dp0, dropping
>    |00047|bridge|ERR|ib0 port has no interfaces, dropping
>    I've tried to report this on the Open vSwitch discuss list, but my
>    messages do not seem to get through.

Did you subscribe to the list? 

-- Pasi

>    Thanks!
>    Trygve
>    On Wed, Apr 14, 2010 at 6:23 PM, Trygve Sanne Hardersen
>    <[1]trygve@xxxxxxxxxxxxx> wrote:
> 
>      I've finally got to spend some time looking further into this.
>      I now believe the underlaying problem is that Open vSwitch is unable to
>      connect the brib0 bridge interface to the ib0 physical interface. I
>      suspect the cause of this to be the long MAC address of the Infiniband
>      NICs, but so far I have not found a workaround for the issue.
>      These are the relevant devices for my setup:
>      [root@hypoxcp1 ~]# ifconfig
>      brib0     Link encap:Ethernet  HWaddr 80:00:00:48:FE:80
>                inet addr:10.1.2.2  Bcast:10.1.2.255  Mask:255.255.255.0
>                inet6 addr: fe80::8200:ff:fe48:fe80/64 Scope:Link
>                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>                RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>                TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
>                collisions:0 txqueuelen:0
>                RX bytes:0 (0.0 b)  TX bytes:720 (720.0 b)
>      eth0      Link encap:Ethernet  HWaddr 00:30:48:CC:5C:A4
>                inet6 addr: fe80::230:48ff:fecc:5ca4/64 Scope:Link
>                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>                RX packets:31755 errors:0 dropped:0 overruns:0 frame:0
>                TX packets:10544 errors:0 dropped:0 overruns:0 carrier:0
>                collisions:0 txqueuelen:1000
>                RX bytes:4284224 (4.0 MiB)  TX bytes:1433336 (1.3 MiB)
>      ib0       Link encap:InfiniBand  HWaddr
>      80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
>                inet addr:10.1.2.102  Bcast:10.1.2.255  Mask:255.255.255.0
>                UP BROADCAST MULTICAST  MTU:2044  Metric:1
>                RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>                TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>                collisions:0 txqueuelen:128
>                RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>      xenbr0    Link encap:Ethernet  HWaddr 00:30:48:CC:5C:A4
>                inet addr:10.1.1.2  Bcast:10.1.1.255  Mask:255.255.255.0
>                inet6 addr: fe80::230:48ff:fecc:5ca4/64 Scope:Link
>                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>                RX packets:25892 errors:0 dropped:0 overruns:0 frame:0
>                TX packets:10538 errors:0 dropped:0 overruns:0 carrier:0
>                collisions:0 txqueuelen:0
>                RX bytes:3586527 (3.4 MiB)  TX bytes:1432868 (1.3 MiB)
>      The ifconfig command reports the wrong (or truncated) MAC address for
>      the ib0 device. The real address can be found using other commands:
>      [root@hypoxcp1 ~]# cat /sys/class/net/ib0/address
>      80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
>      [root@hypoxcp1 ~]# ip link show ib0
>      4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast qlen
>      128
>          link/infiniband
>      80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25 brd
>      00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
>      As mentioned earlier in this thread I've had issues with duplicate MAC
>      addresses in /etc/ovs-vswitchd.conf, but a clean install somehow fixed
>      that issue, so the proper MAC address is now added to the file:
>      [root@hypoxcp1 ~]# cat /etc/ovs-vswitchd.conf
>      
> bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
>      bridge.brib0.port=brib0
>      bridge.brib0.port=ib0
>      bridge.brib0.port=vif1.2
>      bridge.brib0.xs-network-uuids=6455dd7f-4a61-43b8-a49d-656f749c4ac6
>      bridge.xenbr0.mac=00:30:48:cc:5c:a4
>      bridge.xenbr0.port=eth0
>      bridge.xenbr0.port=vif1.1
>      bridge.xenbr0.port=xenbr0
>      bridge.xenbr0.xs-network-uuids=528d85a4-f582-c181-54eb-acf09ac7dcf4
>      bridge.xenbr1.mac=00:30:48:cc:5c:a5
>      bridge.xenbr1.port=eth1
>      bridge.xenbr1.port=vif1.0
>      bridge.xenbr1.port=xenbr1
>      bridge.xenbr1.xs-network-uuids=4f033ff5-5a56-629c-1c27-0765ba7c03bb
>      I'm no expert on XCP and Open vSwitch, but I believe it works something
>      like this:
> 
>       1. XAPI writes /etc/ovs-vswitchd.conf based on the XCP DB
>       2. XAPI starts up Open vSwitch
>       3. Open vSwitch creates the interfaces defined
>          in /etc/ovs-vswitchd.conf
> 
>      To me it seems like the MAC address for the brib0 interface
>      is truncated, and I believe this causes Open vSwitch to not bind brib0
>      and ib0 together:
>      [root@hypoxcp1 ~]# ovs-ofctl show brib0
>      Apr 14 15:38:31|00001|ofctl|INFO|connecting to unix:/var/run/brib0.mgmt
>      features_reply (xid=0x6bb27f3f): ver:0x97, dpid:32f493d6e290
>      n_tables:2, n_buffers:256
>      features: capabilities:0x17, actions:0x3ff
>       LOCAL(brib0): addr:80:00:00:48:fe:80, config: 0, state:0
>      Apr 14 15:38:31|00002|ofctl|INFO|connecting to unix:/var/run/brib0.mgmt
>      get_config_reply (xid=0x9b99aaf1): miss_send_len=0
>      [root@hypoxcp1 ~]# ovs-ofctl show xenbr0
>      Apr 14 15:38:19|00001|ofctl|INFO|connecting to unix:/var/run/xenbr0.mgmt
>      features_reply (xid=0x836b0867): ver:0x97, dpid:f68bde598f51
>      n_tables:2, n_buffers:256
>      features: capabilities:0x17, actions:0x3ff
>       1(eth0): addr:00:30:48:cc:5c:a4, config: 0, state:0
>           current:    1GB-FD COPPER AUTO_NEG
>           advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER
>      AUTO_NEG
>           supported:  10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER
>      AUTO_NEG
>       LOCAL(xenbr0): addr:00:30:48:cc:5c:a4, config: 0, state:0
>      Apr 14 15:38:19|00002|ofctl|INFO|connecting to unix:/var/run/xenbr0.mgmt
>      get_config_reply (xid=0x2b665ea): miss_send_len=0
>      As you see the binding to ib0 is missing, and the MAC of brib0 is
>      different from that in /etc/ovs-vswitchd.conf.
>      As previously stated I can communicate between XCP hosts on both brib0
>      and ib0 using this setup. The problem is that VIFs on the brib0 network
>      are not reachable. I have the following IB interfaces on a single host:
>      ib0 - [2]10.1.2.2/24
>      brib0 - [3]10.1.2.102/24
>      vif1.2 - [4]10.1.2.202/24
>      From within the VM that uses vif1.3 I try to ping brib0 and ib0 and
>      watch the traffic on the XCP host:
>      [root@hypoxcp1 ~]# tcpdump -i vif1.2
>      tcpdump: WARNING: vif1.2: no IPv4 address assigned
>      tcpdump: verbose output suppressed, use -v or -vv for full protocol
>      decode
>      listening on vif1.2, link-type EN10MB (Ethernet), capture size 96 bytes
>      15:54:59.948660 arp who-has 10.1.2.102 tell 10.1.2.202
>      15:55:00.948643 arp who-has 10.1.2.102 tell 10.1.2.202
>      15:55:01.948645 arp who-has 10.1.2.102 tell 10.1.2.202
>      [root@hypoxcp1 ~]# tcpdump -i brib0
>      tcpdump: verbose output suppressed, use -v or -vv for full protocol
>      decode
>      listening on brib0, link-type EN10MB (Ethernet), capture size 96 bytes
>      15:54:22.612723 arp who-has 10.1.2.102 tell 10.1.2.202
>      15:54:23.612643 arp who-has 10.1.2.102 tell 10.1.2.202
>      15:54:24.612642 arp who-has 10.1.2.102 tell 10.1.2.202
>      [root@hypoxcp1 ~]# tcpdump -i ib0
>      tcpdump: WARNING: arptype 32 not supported by libpcap - falling back to
>      cooked socket
>      tcpdump: verbose output suppressed, use -v or -vv for full protocol
>      decode
>      listening on ib0, link-type LINUX_SLL (Linux cooked), capture size 96
>      bytes
>      The packets never reach ib0.
>      This setup adds the follow IP routes to the XCP host:
>      [root@hypoxcp1 ~]# route -n
>      Kernel IP routing table
>      Destination     Gateway       Genmask         Flags  Metric  Ref     Use
>      Iface
>      10.1.1.0          0.0.0.0         255.255.255.0   U       0          0
>           0 xenbr0
>      10.1.2.0          0.0.0.0         255.255.255.0   U       0          0
>           0 ib0
>      10.1.2.0          0.0.0.0         255.255.255.0   U       0          0
>           0 brib0
>      169.254.0.0     0.0.0.0         255.255.0.0      U       0          0
>           0 brib0
>      0.0.0.0           10.1.1.1        0.0.0.0             UG     0
>       0        0 xenbr0
>      If I remove the ib0 route I can talk to brib0 and ib0 from vif1.3, but
>      only on the same physical machine. Inter-host and inter-vm over network
>      communication breaks without that route.
>      I also tried using "bridge" networking instead of "vswitch", but the
>      system behaves the same way AFAICT, though the configuration is of
>      course different.
>      I'm not sure what to try next. I could use the IB network for the
>      management interface and not run any VMs on it, but please let me know
>      if you have any idea what's wrong.
>      Thanks!
>      Trygve
>      On Thu, Apr 8, 2010 at 11:40 AM, Trygve Sanne Hardersen
>      <[5]trygve@xxxxxxxxxxxxx> wrote:
> 
>        Hi
>        Yes, I believe the packets are lost between brib0 and ib0, so they are
>        never sent across the network but it works on a single host.
>        I'll do some more testing and let you know what I find.
>        Thanks!
>        Trygve
> 
>        On Thu, Apr 8, 2010 at 10:40 AM, Dave Scott
>        <[6]Dave.Scott@xxxxxxxxxxxxx> wrote:
> 
>          Hi,
> 
> 
> 
>          Is it true that you have managed to get VM <-> Host connectivity
>          working but not VM <-> VM (across host) connectivity working?
> 
> 
> 
>          If so then it would be interesting to use something like tcpdump to
>          find out where the packets are going missing. If they*re entering
>          the vswitch and then getting lost then it would be worth talking
>          about this on the openvswitch mailing list.
> 
> 
> 
>          Another possibility is to revert to non-vswitch based networking in
>          dom0: try writing *bridge* to /etc/xensource/network.conf and
>          rebooting.
> 
> 
> 
>          Cheers,
> 
>          Dave
> 
> 
> 
>          From: [7]xen-users-bounces@xxxxxxxxxxxxxxxxxxx
>          [mailto:[8]xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of
>          Trygve Sanne Hardersen
>          Sent: 07 April 2010 23:02
>          To: Xen
>          Subject: [Xen-users] using ipoib with xcp
> 
> 
> 
>          Hello,
> 
> 
> 
>          I have been playing with the XCP for a while now, and must say I'm
>          very exited about the technology. I had no prior experience with Xen
>          so it has taken me a while to understand the concepts, but now I
>          feel most important issues are solved and I've purchased some
>          hardware to build my (tiny) cloud on.
> 
> 
> 
>          The box is a Supermicro 1026TT-IBXF, so I have 2 x Ethernet and 1 x
>          Infiniband (IB) NICs per node. I want to use the IB NIC to provide
>          fast connectivity between the domUs, while the Ethernet NICs will be
>          used for the XCP management interface and ISP connectivity.
> 
> 
> 
>          I have successfully built OFED 1.5.1 in the XCP DDK VM and
>          installed OFED in the XPC 0.1.1 dom0. From there I can bring up the
>          IB network, but I'm having problems getting this to work properly
>          within XCP virtual machines. This is what happens:
> 
> 
> 
>          Starting out I have 2 nodes in a pool; both are clean with only lo,
>          eth0/xenbr0 and eth1/xenbr1 configured. I run the following commands
>          to add the IB NICs to the pool:
> 
> 
> 
>          xe pif-scan host-uuid=NODE1
> 
>          xe pif-plug uuid=NODE1_IB0
> 
>          xe pif-scan host-uuid=NODE2
> 
>          xe pif-plug uuid=NODE2_IB0
> 
> 
> 
>          As expected this adds ib0/brib0 on both nodes and a single pool-wide
>          network, but there is no connectivity between the hosts after I give
>          brib0 an IP:
> 
> 
> 
>          xe pif-reconfigure-ip uuid=NODE1_IB0 IP=10.1.2.2
>          netmask=255.255.255.0 mode=static
> 
>          xe pif-reconfigure-ip uuid=NODE2_IB0 IP=10.1.2.3
>          netmask=255.255.255.0 mode=static
> 
>          ping 10.1.2.2 --> reply
> 
>          ping 10.1.2.3 --> destination host unavailable
> 
> 
> 
>          However if I also give ib0 an IP and use this as gateway for brib0,
>          connectivity is achieved:
> 
> 
> 
>          ifconfig ib0 10.1.2.22 netmask 255.255.255.0
> 
>          xe pif-reconfigure-ip uuid= NODE1_IB0 IP=10.1.2.2
>          netmask=255.255.255.0 gateway=10.1.2.22 mode=static
> 
>          ifconfig ib0 10.1.2.33 netmask 255.255.255.0
> 
>          xe pif-reconfigure-ip uuid= NODE2_IB0 IP=10.1.2.3
>          netmask=255.255.255.0 gateway=10.1.2.33 mode=static
> 
>          ping 10.1.2.2 --> reply
> 
>          ping 10.1.2.3 --> reply
> 
>          ping 10.1.2.22 --> reply
> 
>          ping 10.1.2.33 --> reply
> 
> 
> 
>          This is very well, but when I add a VIF on the IB network to a VM it
>          is not able to communicate through it:
> 
> 
> 
>          xe vif-create device=2 mac=random network-uuid=IB_NET
>          vm-uuid=NODE1_IBVM
> 
>          ifconfig eth2 10.1.2.122 netmask 255.255.255.0
> 
>          xe vif-create device=2 mac=random network-uuid=IB_NET
>          vm-uuid=NODE2_IBVM
> 
>          ifconfig eth2 10.1.2.133 netmask 255.255.255.0
> 
>          ping 10.1.2.122 --> reply
> 
>          ping 10.1.2.22 --> destination host unavailable
> 
>          ping 10.1.2.2 --> destination host unavailable
> 
>          ping 10.1.2.133 --> destination host unavailable
> 
>          ping 10.1.2.33 --> destination host unavailable
> 
>          ping 10.1.2.3 --> destination host unavailable
> 
> 
> 
>          I believe that the problem lies somewhere in the routing table
>          configuration. This setup gives the following routing table:
> 
> 
> 
>          10.1.2.0        0.0.0.0         255.255.255.0   U     0      0
>           0 ib0
> 
>          10.1.2.0        0.0.0.0         255.255.255.0   U     0      0
>           0 brib0
> 
> 
> 
>          If I delete and then add the brib0 route, the route order is
>          changed:
> 
> 
> 
>          10.1.2.0        0.0.0.0         255.255.255.0   U     0      0
>           0 brib0
> 
>          10.1.2.0        0.0.0.0         255.255.255.0   U     0      0
>           0 ib0
> 
> 
> 
>          Using this the VM can talk to the host (and visa versa), but hot
>          across the network. Connectivity between ib0/brib0 over the network
>          is also broken.
> 
> 
> 
>          I've also noticed that the same MAC is added to
>          /etc/ovs-vswitchd.conf multiple times for brib0:
> 
> 
> 
>          
> bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
> 
>          
> bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
> 
>          
> bridge.brib0.mac=80:00:00:48:fe:80:00:00:00:00:00:00:00:30:48:ff:ff:cc:0b:25
> 
> 
> 
>          I've tried removing some of these but that does not seem to have any
>          effect. My experience with IP routing and especially vswitch is
>          limited and I'm not sure what to try from here. I've tried various
>          configurations but no luck so far.
> 
> 
> 
>          Note that I'm testing with 2 XCP nodes configured in a pool. I've
>          also checked that the PIFs are in the same order on both nodes (the
>          reference mentions this). The MTU (1500) of brib0 differs from that
>          of ib0 (2044), but changing this does not solve the problem.
> 
> 
> 
>          Any help is much appreciated. Thanks!
> 
> 
> 
>          Trygve
> 
>          --
>          HypoBytes Ltd.
>          Trygve Sanne Hardersen
>          Akersveien 24F
>          0177 Oslo
>          Norway
> 
>          [9]hypobytes.com
>          +47 40 55 30 25
> 
>        --
>        HypoBytes Ltd.
>        Trygve Sanne Hardersen
>        Akersveien 24F
>        0177 Oslo
>        Norway
> 
>        [10]hypobytes.com
>        +47 40 55 30 25
> 
>      --
>      HypoBytes Ltd.
>      Trygve Sanne Hardersen
>      Akersveien 24F
>      0177 Oslo
>      Norway
> 
>      [11]hypobytes.com
>      +47 40 55 30 25
> 
>    --
>    HypoBytes Ltd.
>    Trygve Sanne Hardersen
>    Akersveien 24F
>    0177 Oslo
>    Norway
> 
>    [12]hypobytes.com
>    +47 40 55 30 25
> 
> References
> 
>    Visible links
>    1. mailto:trygve@xxxxxxxxxxxxx
>    2. http://10.1.2.2/24
>    3. http://10.1.2.102/24
>    4. http://10.1.2.202/24
>    5. mailto:trygve@xxxxxxxxxxxxx
>    6. mailto:Dave.Scott@xxxxxxxxxxxxx
>    7. mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx
>    8. mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx
>    9. http://hypobytes.com/
>   10. http://hypobytes.com/
>   11. http://hypobytes.com/
>   12. http://hypobytes.com/

> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>