[Xen-users] Bridging in initrd with xen and aoe

[Tracy R Reed <treed@xxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I am trying to setup a cluster using Xen and AoE. Being the purist that
I am, my Xen boxes are completely diskless. Fast cpu's, lots of RAM, no
disk. Then I have several disk nodes. Lower end P4 cpu's, 512M RAM, 4
300G disks in a RAID5. All of the boxes have dual gig ethernet. One for
the IP network to serve our application and for management and the other
for the AoE SAN. Currently I have the AoE ports on a xen node and a disk
node directly connected but eventually it will all be connected with a
gig-e switch.

I have made an initrd that allows the xen boxes to PXE boot and mount
the root from AoE. Works great. Until I start up xend. As soon as xend
tries to configure its bridge it momentarily takes down the network
which means the root drive goes away which means the whole machine
hangs. This means I need to get the bridging established while the
machine is running off of the initrd.

I am having real problems getting the AoE disks to be discovered after
setting up a bridge in the initrd. It seems like I should be able to
create a bridge for the disk interfaces (I'll call it aoebr) and then I
should be able to add eth1 (direct connected from the xen node to the
disk node) to the aoebr bridge and echo eth1 into the
/dev/etherd/devices file. And then it should Just Work. At least as far
as the disk side of things goes. Some people have told me that I should
put the name of the bridge (aoebr) into /dev/etherd/devices file. But
that makes no sense to me. The bridge is just a bridge and you can only
talk to an actual interface, right?

My current initrd init script can be found here: http://pastebin.ca/248153

With this setup the AoE disks are not discovered. But I know the AoE
side of things works because if I don't set up any bridging and just let
the machine boot normally the AoE disks are detected and work perfectly.
Right up until xend tries to set up bridging anyway. I have disabled
xend on start and used the machine for a while and root over AoE works
great. I can even boot domainU's but they only get as far as the initrd
before they need to access AoE disk themselves.

So, any ideas as to why it cannot detect the AoE disk when setting up
bridging in the initrd?

I have even run a tcpdump on the interface of the directly attached disk
and it sees nothing.

What I would really like to do is put eth1 on aoebr and the eth1 from
each of my domainU's on aoebr also and use that as the SAN bridge and
then once I get the disk side of things working I'll create another
bridge and put the eth0 of domain0 on it as well as the eth0's from my
domainU's so they can access the IP network.

I am using CentOS 4.3, Xen stable checked out from mercurial a couple
days ago, aoe6-30, and vblade-10.

- --
Tracy R Reed
http://ultraviolet.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFFWYm09PIYKZYVAq0RAtLIAJ9xr6+xCNGuRz3HeWeLSy/Bd5rI9QCeKodG
x1oAF8cH3xfucgEkEKCYtio=
=cP5j
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users