1

About a month ago, I asked this question on Superuser. The answer given there certainly works, but I've run into an issue I haven't thought of before.

The setup I had running (based on the previously mentioned answer) was

auto eth0
iface eth0 inet static
    address a.b.c.d
    netmask 255.255.255.0
    network a.b.c.0
    broadcast a.b.c.255
    gateway a.b.c.254

auto xenbr0
iface xenbr0 inet static
    bridge_ports none
    address e.f.g.1
    netmask 255.255.255.0

Then, I would configure the domUs to use an address in the range that was assigned to me by the datacenter. In particular that range would be e.f.g.24/29, giving me 8 IP addresses available for domUs. The gateway would be the bridge IP, from where the packets get routed through eth0 to the outside world. This worked perfectly.

The issue I realised a couple days ago is that, the domUs will never be able to reach addresses such as e.f.g.1, because that's the bridge itself. But, what if one of the domUs actually needs to access e.f.g.1 (the actual outside IP)?

So, I started moving everything inside of the networks I was assigned. Problem then was that, with a /29 block, I would only keep 5 IP address available for a domU:

auto xenbr0
iface xenbr0 inet static
    bridge_ports none
    address e.f.g.25
    netmask 255.255.255.248

I always loose three addresses in my network (one for "the network", one for the bridge itself, and one for the broadcasting address). I actually didn't define the network and broadcasting address, but ifconfig still shows the broadcast address as 31 (last IP in the network) and I can't place the bridge on address 24 in any working manner. The issue gets even worse with a /30 network, where I would only have one out four IPs available for the domUs (yes, I have such a network as well).

Because I'm paying per IP address, I want to make as efficient use of them as possible. I then started researching Xen's routing. What I'm trying to do now is to use vif-route and network-route in /etc/xen/xend-config.sxp to make a setup like:

     +-----------------------------+        +-------------+
     |            dom0             |        |     domU    |
     |                             |        |             |
-------eth0                vif1.0--------------eth0       |
     | a.b.c.d              ????   |        |  e.f.g.24   |
     |                             |        |             |
     +-----------------------------+        +-------------+

This way the datacenter would send packets for my IPs to dom0's eth0, dom0 would route it to the correct vif which would hand it off to domU's eth0.

No matter what I try, I can't seem to get it working, without any meaningful error messages to start debugging with. Two days of constant Googling hasn't helped me and I'm kind of staring blank at the moment.

Am I looking at this the wrong way, or am I just configuring things wrongfully? Is such a setup even feasible? If so, what am I doing wrong?

Community
  • 1
Simon
  • 317

1 Answers1

1

I managed to get it working. It's not 100% as clean as I want it to be, but at least it's functional.

My solution basically consists of starting up the bridge with a /24 netmask and without an IP address. When the bridge is up, I manually add a route for the /29 IP block I got assigned to my server. In the domUs, I start the interface with the correct IP and a /24 netmask, without a gateway. When the interface is up, I manually add a route to my dom0's eth0 address and make that the default gateway. Of course, dom0 is set up to route IP packets between the networks.

dom0's /etc/network/interfaces

### The primary adapter ###
# These settings are provided by the dServer host
auto eth0
iface eth0 inet static
        address a.b.c.d
        netmask 255.255.255.0
        network a.b.c.0
        broadcast a.b.c.255
        gateway a.b.c.254

### Network bridges ###
auto xenbr0
iface xenbr0 inet manual
        bridge_ports none
        network e.f.g.0
        netmask 255.255.255.0
        gateway a.b.c.254

### Static routes ###
# No route for e.f.g.0/24 was automatically set
# I assume that's due to the bridge not having an IP address assigned
up ip route add e.f.g.h/29 dev xenbr0
down ip route delete e.f.g.h/29

domU's /etc/network/interfaces

# The primary network interface
auto eth0
iface eth0 inet static
        address e.f.g.h
        netmask 255.255.255.0

# Static routing
up ip route delete e.f.g.0/24
up ip route add a.b.c.d dev eth0
up ip route add default via a.b.c.d
down ip route delete default via a.b.c.d
dow ip route delete a.b.c.d

Enable packet routing by making sure this is set in /etc/sysctl.d/xen-routing.conf (the file name is arbitrary, but must end in .conf)

net.ipv4.ip_forward=1

After you've done that, you can enable the configuration by rebooting or do it online with

sysctl -w net.ipv4.ip_forward=1
Simon
  • 317