6

We have recently bought a server from the Hetzner server auction and want to migrate our previous root server. Previously we used only windows server, but now we want to set it up a bit more modular with proxmox.

We have a single public IP address, so we chose to use NAT with a 10.0.0.0/24 internal subnet.

The containers can access the internet, so this works fine.

But we can't connect to servers which run inside the containers over custom ports.

How can I properly forward the ports? We tried the routing and port forwarding as listed below. Something went wrong, the ports arent open, as shown by portscans. There is currently only a firewall in the Proxmox host, the server host firewall is disabled, so I dont know why traffic should be blocked.

This is the current setup of the proxmox network interface:

auto lo
iface lo inet loopback

iface lo inet6 loopback


auto eno1
iface eno1 inet static
        address public.ipv4/26
        gateway public.gateway
        up route add -net public.ipv4 netmask 255.255.255.192 gw public.ipv4 dev eno1


route public.ipv4.range.start/26 via public.ipv4.range.start+1


iface eno1 inet6 static
        address public.ipv6/128
        gateway fe80::1


auto vmbr0
iface vmbr0 inet static
        address 10.0.0.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0


    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o eno1 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o eno1 -j MASQUERADE
    post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 28967 -j DNAT --to 10.0.0.4:28967
    post-up iptables -t nat -A PREROUTING -i vmbr0 -p udp --dport 28967 -j DNAT --to 10.0.0.4:28967
    post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 28967 -j DNAT --to 10.0.0.4:28967
    post-down iptables -t nat -D PREROUTING -i vmbr0 -p udp --dport 28967 -j DNAT --to 10.0.0.4:28967




iface vmbr0 inet6 static
        address public.ipv6+1/64

iptables -L -t nat:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             anywhere             tcp dpt:28967 to:10.0.0.4:28967
DNAT       udp  --  anywhere             anywhere             udp dpt:28967 to:10.0.0.4:28967

Chain INPUT (policy ACCEPT)
target     prot opt source               destination


Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

MASQUERADE  all  --  10.0.0.0/24          anywhere

The eno1 interface was autogenerated by their installation script. We added the bridge and the NAT etc.

Client config:

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eth0
iface eth0 inet static
        address 10.0.0.4/24
        gateway 10.0.0.1

Client container has currently no firewall.

We are relatively inexperienced with networking inside a KWM host, so any help is greatly appreciated!

Raphael
  • 202

1 Answers1

4

We figured something out and I'll post it here in case its useful for anyone. (Disclaimer: We put this together after hours of trial and error, this might only work for our hosting provider. I am not a Linux network expert by any means.)

/etc/network/interfaces:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback


auto eno1
iface eno1 inet static
    address <hostpublicip>/26
    gateway <hostgateway>
    up route add -net <hostgateway-1> netmask 255.255.255.192 gw <hostgateway> dev eno1
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward # add this line


auto vmbr0
iface vmbr0 inet static
    address 10.0.0.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0


post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -F
post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

The eno1 interface was already set up by the install script of our hosting provider (Hetzner), we added the post-up line to enable IP forwarding on boot.
We created the vmbr0 interface in proxmox and set up a subnet address, you can choose any address range you like. We went with 10.0.0.0/24. Then you need to change the file entry according to our example above. The lines are required as they are (except the IP of course). If there are any ipv6 interfaces, leave them alone.

Now when you create containers/VMs you need to set the IPv4 address to static and choose any that fits your subrange (10.0.0.xxx/24), and set the gateway as 10.0.0.1.

All the port forwards are defined in the file /etc/network/interfaces.d/port_forwards (needs to be created by the user, all files in that path are loaded into the above main file with the source line). The IP address in the port forwards is the IP address you assigned to the container during creation.

EDIT: I since created a small tool for myself ez-port-forward, which reads the port definitions from a YAML file. This also checks for port collisions etc. and is much more readable than the custom script below.

# your network interface
iface vmbr0 inet static
    # minecraft container as example
    post-up port_forward -t 10.0.0.201 -p tcp -d 22     -s 22201 # external port 22201 for direct ssh to container
    post-up port_forward -t 10.0.0.201 -p tcp -m -d 25566,25576
    post-up port_forward -t 10.0.0.201 -p udp -m -d 25566,25576
    # follow with all the port forwards you need

With a custom port forwarding script called port_forward which just abbreviates the iptables commands to keep the file readable. This file must be placed in or linked to /bin/ so it is loaded before the user partition is mounted.

#!/bin/bash
# abbr for iptables dnat

usage() { echo "Usage: $0 -t <container_ip> -p <tcp/udp/etc> -d <port> [-s <external port if different>] [-m flag if ports will be comma separated list, not compatible with -s]" 1>&2; exit 0; }


while getopts t:p:d:s::m flag
do
    case "${flag}" in
        t) target_ip=${OPTARG};;
        p) protocoll=${OPTARG};;
        d) dest_port=${OPTARG};;
        s) src_port=${OPTARG};;
        m) multiport=1 ;;
        *) usage ;;
    esac
done


if [[ -z "$src_port" ]]; then
    src_port="$dest_port"
fi


if [[ "$multiport" -eq 1 ]]; then
    iptables -t nat -A PREROUTING -i eno1 -p "$protocoll" -m multiport --dports "$dest_port" -j DNAT --to "$target_ip"
else
    iptables -t nat -A PREROUTING -i eno1 -p "$protocoll" --dport "$src_port" -j DNAT --to "$target_ip":"$dest_port"
fi

Everytime the files are changed, iptables needs to be reset like so:

iptables --flush
iptables -t nat --flush
iptables --delete-chain
ifreload --all

This can be put in a file as well, we named it reload_network and placed it in the /usr/local/bin/ folder to access as a command.

Raphael
  • 202