This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-users] network-bridge breaks networking when eth0:1 is added

 Nugraha wrote on Tue, 28 Apr 2009 19:58:26 +0700:

> Seriously?

Yes. I would love to give you all the details. The problem is that what 
I'm working with is not the final setup but some "semblance". I had to 
improvise to get something that *should* behave similar than the final 
The final setup will be machine a4 with three NICs attached to two other 
machines (say a1 and a2, both with more than one NIC) via crossover cable 
and a4 pulling from the domUs on a1 and a2 data. I want to go thru the 
crossover link and not the switch, via a private network where all 
additional NICs and the domUs are on.
The problem is that a1 and a2 are not here in the office, but in 
production in the datacenter. So, I have a4 for testing and maybe can 
attach another box if need be. But most boxes I have here are in use and 
have only one NIC. So I can use them only for a short time and have to 
reconfigure them each time. I did this once and tested from another box 
(name it a3) to domUs on a4 with the network setup I mentioned in my first 
posting, so everythign done by the standard network-bridge script. I was 
happy with that as long as I didn't reboot and noticed the initial running 
of network-bridge breaks the eth0/peth0 bridge.
Now, as I cannot always attach another machine for testing (and I don't 
have one with two NICs anyway) my base assumption is that what works from 
a3 -> a4 -> domU should also work without a3. e.g. I don't ping from a3 
but from a4 dom0 -> a4 domUs, but using the interface/IP number that is on 
the private subnet ("ping" for instance, or "ping -I xenbr1" which would ping from the xenbr1 interface which is bridging 
to eth1, where the external machine comes in).
I'm using this simplified test scenario for testing. ping the private IP 
on domU from dom0. If it works, fine. If it doesn't, use -I xenbr1 for the 
ping. If that works, fine, but not so comfortable. If that doesn't work: 
bad :-(

Does this make sense so far?

Now I created two bridges as you suggested on a4 and two NICs on the dom0. 
That works but only in one way (dom0 -> domU) and only if I attach the 
ping to xenbr1. I just played around with routes and have found that 
adding a static route (on dom0) solves this so I can ping from either side 
and without specifying an interface. This solution is much more 
comfortable. The private subnet in question is
The output of this setup is now as follows:

brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.001ec9fefbab       no              eth0
xenbr1          8000.001ec9fefbac       no              eth1

ip addr list | grep "inet "
    inet scope host lo
    inet brd scope global eth2
    inet brd scope global xenbr0
    inet brd scope global xenbr1

ip route via dev xenbr1  scope link dev eth2  proto kernel  scope link  src dev xenbr1  proto kernel  scope link  src dev xenbr0  proto kernel  scope link  src
default via dev xenbr0

domU is straight forward:
no bridge

ip addr list | grep "inet "
    inet scope host lo
    inet brd scope global eth0
    inet brd scope global eth0:1
    inet brd scope global eth1

ip route dev eth0  proto kernel  scope link  src dev eth1  proto kernel  scope link  src dev eth0  proto kernel  scope link  src
default via dev eth0  src
default via dev eth0

I had forwarding enabled in the meantime, but it works without it as well, 
so I disabled that again.

So, just to make it clear, *this* setup with the additional route is 
working now in all directions. I now realize that my best option is 
probably to use a different subnet each for eth1 and eth2 and different 
subnets on the two machines it goes out to (remember a4 will not be the 
target, but the source). If I do that I can use a route for each subnet, 
otherwise I have to use a route per single IP address.

Oh, and I just find that using different nets for eth1 and eth2 solves the 
problem, anyway, without a static route. Like so:
ip route dev eth2  proto kernel  scope link  src dev xenbr1  proto kernel  scope link  src dev xenbr0  proto kernel  scope link  src
default via dev xenbr0

The big problem obviously was eth2 being on the same subnet as eth1/xenbr1 
and thus catching the packets to nowhere.

Good. I had just tried to add the static route with sysconfig/route-xenbr1 
and all variations I tried failed with obscure errors. I'm going to 
follow-up on this on the CentOS list.

Thanks for your answers, they were all really very helpful!

I think I have found a solution now, although I still would like to know 
why the original method works on some machines but not this one. 
Interestingly, one of the errors I got when trying to use route-xenbr1 was 
the same as when using the original setup and the network-bridge script.

If you are still interested in the original data I can revert this machine 
to the original setup and grab the data from there or from the other 
machine I mentioned where it works fine.


Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Xen-users mailing list