WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] quick question about bonding with vlans and Xen

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] quick question about bonding with vlans and Xen
From: Donny Brooks <dbrooks@xxxxxxxxxxxxxxxx>
Date: Wed, 16 Jun 2010 10:14:09 -0500
Delivery-date: Wed, 16 Jun 2010 08:15:51 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4C18D1F8.5060602@xxxxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <521d-4bfaef00-41-65e8a500@153514511> <AANLkTikqBNHAXjlibasTT3xAm_hAj_d99tFFXXaW3VK_@xxxxxxxxxxxxxx> <4C1679EC.8010907@xxxxxxxxxxxxxxxx> <4C18D1F8.5060602@xxxxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
On 6/16/2010 8:30 AM, Donny Brooks wrote:
On 6/14/2010 1:50 PM, Donny Brooks wrote:
On 5/24/2010 11:16 PM, Fajar A. Nugraha wrote:
On Tue, May 25, 2010 at 4:25 AM, Donny Brooks<dbrooks@xxxxxxxxxxxxxxxx> wrote:
Ok, so far I have followed up till this point with eth2 and eth3 (third and fourth ports in the server) leaving eth0 and eth1 alone for now.
... which you can add to the bridge as well later, if you like.

So when I pass the domu "bridge=br1" (changed it to br1 from 9) and it is a dhcp host, how does it know where to get it's IP?
I think you're still missing the basics. Here's an analogy to make
things easier. When using the above setup (bonding + vlan + bridge),
Xen dom0 behaves the same way as an L2 switch with vlans enabled and
multiple bonded ports for uplink trunk.

For simplicity sake lets assume the bridge names is the same as vlan
numbers. I wouldn't recommend using vlan1 (and thus br1), as vlan1 is
usually the default management vlan/default vlan for untagged traffic.
So lets assume we'll be using vlan9/br9. eth2 and eth3 (which is the
uplink trunk interface) must be connected to a switch as trunk,
possibly requiring special bonding setup on the switch side as well
(depends on which bonding mode you use). The switch must already have
an existing vlan9, which is connected to an existing network with an
existing DHCP server.

So in that sense, a domU connected to br9 behaves just like another
physical machine connected to the switch directly and assigned vlan9.
It can get IP address from an existing DHCP server on that vlan. If no
DHCP server exists, you have to create one first :D


I have 18 VLAN's I need to pass to Xen, VLAN 2-19. Is it that br1 needs only setup with one vlan? So if I did a ifcfg-bond0.2 I would make a br2 and point the domu to that? Just trying to clarify since I think that's how I would need to do it.
You'd need 18 vlan interface, from bond0.2 to bond0.19, and 18
bridges, from br2 to br19.


But do I have to assign an IP to each "interface"? Would I need to designate one specific NIC to handle Dom0 and it's static IP or would Dom0 still be able to have a static IP with all 4 bonded together with vlans on them?
Again, think L2 switch. Usually it will only have one management IP
address, no matter how many vlans it has. Let's assume your dom0
management IP address will be on vlan9, so on
/etc/sysconfig/network-scripts/ifcfg-br9 you can put IP address
settings, and leave all other bridges WITHOUT IP address setting. This
setup has the advantage that dom0 will also benefit from the bonding
setup.

Another approach that you can use is to have a dedicated NIC for dom0
management IP address. Let's assume eth0. You can connect it to a
different switch. This setup has the advantage that you'd get an
out-of-band management network (think HP's ILO or Sun's rsc), but you
don't have the benefit of bonding.

For simplicity sake, I might just leave dom0 on eth0 for now and just use eth1 thru eth3 for the bond. That way if I foobar something I can still get to dom0 remotely. Thank you for taking the time to explain this. Now I just need to figure out the switch portion of the bond :)
In my example above, "mode=balance-alb" means you don't have to do
much on the switch side. Just make sure the ports eth1-3 uses are set
as trunk, not access. If you use  "mode=802.3ad" or 4, you'd need to
setup bonding on the switch side as well. This page has a good
explanation: http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking


Just writing back to let everyone know that I was able to get this working. It took me a bit as I was mis-reading what Fajar was saying. But his examples were spot on. Below is the exact steps I did to accomplish this:

1. On the switch (Cisco 4705R in my case) I simply put the 3 interfaces into trunk mode with the following commands:

interface GigabitEthernet6/30
 description to xen bond0 interface 1
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet6/31
 description to xen bond0 interface 2
 switchport trunk encapsulation dot1q
 switchport mode trunk
!
interface GigabitEthernet6/32
 description to xen bond0 interface 3
 switchport trunk encapsulation dot1q
 switchport mode trunk
!

The encapsulation part was needed on my switch but may not on others. It griped about it so I added it.

2. On the Xen dom0 I commented out the "(network-script network-bridge)" line in /etc/xen/xend-config.sxp
3. I created the bond device on Xen dom0:

cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none

4. Created the config file:

cat /etc/modprobe.d/bond0
alias bond0 bonding
options bond0 mode=balance-alb miimon=100

5. Added eth1-eth3 to the bond:

cat /etc/sysconfig/network-scripts/ifcfg-eth1
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
HWADDR=00:00:00:00:00:c5
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

cat /etc/sysconfig/network-scripts/ifcfg-eth2
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth2
HWADDR=00:00:00:00:00:c7
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

cat /etc/sysconfig/network-scripts/ifcfg-eth3
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth3
HWADDR=00:00:00:00:00:c9
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

6. Created the bridges br2 through br19 (one for each vlan you want accesss to) Just first one for example:

cat /etc/sysconfig/network-scripts/ifcfg-br2
DEVICE=br2
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes

7. Created vlans on top of the bond, and assign it to the bridge. Again bond0.2 through bond0.19 one for each vlan. Just first as example:

cat /etc/sysconfig/network-scripts/ifcfg-bond0.2
DEVICE=bond0.2
VLAN=yes
BRIDGE=br2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet

8. I then "virsh edit domuname" and changed the "bridge" line to this:

<interface type='bridge'>
<mac address='00:00:00:00:00:ee'/>
<source bridge='br2'/>

where br2 is the bridge you want that machine to have access to.

In a nutshell I was able to bond 3 gigabit interfaces together (soon to be 4) and have all of our vlans pass across that bond. What threw me for a loop is I misread Fajar's part about doing the switch side stuff according to the http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking site. You don't need that unless you are doing the 802.3ad bonding (which is a PAIN IN THE REAR!!!). So for simplicity just do the above steps and you will be rewarded. Now just to get the gigabit drivers for my HVM windows 2003 machine.

Donny B.



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


And I just THOUGHT I had it working. It seems that no machine on the same vlan as my domu can connect to it unless the domu first pings the machine trying to connect to it. That and the domu has about 80% packet loss! It has to be something I am missing. I did not have this issue before on a basic bridge interface. Anyone have any ideas where to even begin? Here are the system specs:

Centos 5.5 x86_64
Xen 4.0 from gitco repo

Donny B.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

This is most likely not directly related to xen but here goes hoping someone has had similar issues that they resolved!

Let me clarify this: Anything NOT on the same vlan has no issues getting to the domu. I have tried various domu's on various vlans and all exhibit the same behavior. Anything outside the vlan of the domu can connect to it. While anything on the same vlan cannot connect to it unless the domu first pings the machine in question. Even with that it has about 80% packet loss.

Is anyone else doing a balance-alb style bond0 with various vlans passing over it and having success?

Donny B.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users