8

I have a KVM server (host) with multiple virtual machines (guests).

My goal is my host forward port 222 to port 22 of a guest running an ssh service.

This works...

iptables -I OUTPUT -d 0.0.0.0/0 -j ACCEPT
iptables -I FORWARD -d 0.0.0.0/0 -j ACCEPT
iptables -I INPUT -d 0.0.0.0/0 -j ACCEPT
iptables -t nat -I PREROUTING -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22

This don't work...

firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter OUTPUT 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter INPUT 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 nat PREROUTING 0 -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
firewall-cmd --reload

This don't work too...

firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload

QUESTION: Why rules set with firewall-cmd don't work?

NOTE I: The firewall-cmd is the CentOS 7's default firewall service. This seems to me to be a problem with no solution! I searched many, many forums and nothing works! I'm starting to believe this is a limitation or bug in firewall-cmd...

NOTE II: I know that ssh itself provides the means to make this possible, but I really want this process to be "transparent" for the user to access the guest directly.

[Refs.: https://serverfault.com/q/915257/276753 , https://serverfault.com/q/980223/276753 , https://sebastianblade.com/how-to-modify-ssh-port-in-centos7/ , https://www.rootusers.com/how-to-use-firewalld-rich-rules-and-zones-for-filtering-and-nat/ , https://www.centos.org/forums/viewtopic.php?f=50&t=71454 ]

SYMPTOM:

The command...

ssh root@[HOST_IP] -p 222

... returns me the following error...

ssh: connect to host 172.16.13.8 port 222: Connection refused

UPDATE I:

@mwfearnley iptables-save output...

iptables-save - WORKS...

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*nat
:PREROUTING ACCEPT [1:70]
:INPUT ACCEPT [1:70]
:OUTPUT ACCEPT [2:146]
:POSTROUTING ACCEPT [3:206]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -p tcp -m tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*mangle
:PREROUTING ACCEPT [672:77587]
:INPUT ACCEPT [610:68993]
:FORWARD ACCEPT [58:7886]
:OUTPUT ACCEPT [655:151604]
:POSTROUTING ACCEPT [713:159490]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*security
:INPUT ACCEPT [609:68793]
:FORWARD ACCEPT [58:7886]
:OUTPUT ACCEPT [660:152010]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*raw
:PREROUTING ACCEPT [672:77587]
:OUTPUT ACCEPT [655:151604]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j ACCEPT
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j ACCEPT
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:17 2019

iptables-save - DON'T WORK...

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*nat
:PREROUTING ACCEPT [1:72]
:INPUT ACCEPT [1:72]
:OUTPUT ACCEPT [5:371]
:POSTROUTING ACCEPT [5:371]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PREROUTING_direct -p tcp -m tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*mangle
:PREROUTING ACCEPT [12:1319]
:INPUT ACCEPT [11:1259]
:FORWARD ACCEPT [1:60]
:OUTPUT ACCEPT [12:1070]
:POSTROUTING ACCEPT [12:1070]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*security
:INPUT ACCEPT [11:1259]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12:1070]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*raw
:PREROUTING ACCEPT [12:1319]
:OUTPUT ACCEPT [12:1070]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FORWARD_direct -j ACCEPT
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A INPUT_direct -j ACCEPT
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A OUTPUT_direct -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:16 2019

iptables-save - DON'T WORK TOO...

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*nat
:PREROUTING ACCEPT [5:371]
:INPUT ACCEPT [1:67]
:OUTPUT ACCEPT [2:134]
:POSTROUTING ACCEPT [2:134]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
-A PRE_public_allow -p tcp -m mark --mark 0x64 -j DNAT --to-destination 10.1.0.9:22
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*mangle
:PREROUTING ACCEPT [17:1649]
:INPUT ACCEPT [12:1285]
:FORWARD ACCEPT [5:364]
:OUTPUT ACCEPT [10:3037]
:POSTROUTING ACCEPT [14:3341]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
-A PRE_public_allow -p tcp -m tcp --dport 222 -j MARK --set-xmark 0x64/0xffffffff
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*security
:INPUT ACCEPT [12:1285]
:FORWARD ACCEPT [4:304]
:OUTPUT ACCEPT [10:3037]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*raw
:PREROUTING ACCEPT [17:1649]
:OUTPUT ACCEPT [10:3037]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [8:2813]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDI_public_allow -m conntrack --ctstate NEW -m mark --mark 0x64 -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:36 2019

UPDATE II:

The first rule in FORWARD of "works" is ACCEPT. That allows every packet to be forwarded. The others have rules to accept DNATted packets, but later in the chain. So... We may be able to solve the problem if we find out why this works...

iptables -I FORWARD -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload

... and why this not...

firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload
Eduardo Lúcio
  • 1,314

2 Answers2

14

THE REAL PROBLEM:

The command...

firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload

... described in "UPDATE II" works just as well as firewalld (used by CentOS 7).

The problem is in the KVM (libvirt) which when starting the "network" virbr0 injects rules (probably via firewalld) which prevents (first match principle) any attempt to port forwarding by conventional ways using firewall-cmd. The KVM injected (libvirt) rules are at the top of FORWARD and POSTROUTING, before regular firewalld ruleset. Since the netfilter operates on first match principle, these rules.

SOLUTION:

Due to the problems generated by the rules injected by KVM (libvirt) into the firewall the solution is use a hack/workaround called libvirt-hook-qemu as we explained below...

Download and install libvirt-hook-qemu...

yum -y install git-core
cd /usr/local/src
git clone https://github.com/saschpe/libvirt-hook-qemu.git
cd libvirt-hook-qemu
make install
cd ..
rm -rf libvirt-hook-qemu

List your virtual machines (guests)...

virsh list --all

... and copy the name of the target virtual machine.

Turn off the virtual machine...

virsh shutdown "<VM_NAME>" --mode acpi

Open the file "hooks.json", delete its contents...

vi /etc/libvirt/hooks/hooks.json

... and enter your rules as the template below (other ways of setting up are possible)...

{
    "<VM_NAME>": {
        "interface": "virbr0",
        "private_ip": "<VM_IP>",
        "port_map": {
            "tcp": [[<PORT_ON_HOST>, <PORT_ON_GUEST>]]
        }
    }
}

Restart libvirtd...

systemctl restart libvirtd.service

... and start the virtual machine...

virsh start "<VM_NAME>"

Done! =D

FURTHER:

This KVM (libvirt) behavior - injects rules which prevents any attempt to port forwarding - is IMPOSITIVE, INVASIVE AND UNNECESSARY... First because since there is no port forwarding on the host no virtual machine will be reachable on the default network (virbr0) and second because it forces us to use a hack/workaround (or "gato"/"gambiarra" as we call it in Brazil =D ).

In our view makes no sense impose the user how he should build his infrastructure or not... This makes no sense! To put it bluntly, we think it would be best if these settings were optional (injects rules...) as other leading hypervisors do. And we go further... If is to KVM (libvirt) controlling this kind of thing then it should also offer ways to map ports to virtual machines.

The KVM (libvirt) is a GREAT tool, but not all users are willing - for various reasons - to place their machines directly on the host machine's network just as not all users are willing to give up all the flexibility/accessibility that the default network (virbr0) offers.

Thanks!

[Refs.: https://bugzilla.redhat.com/show_bug.cgi?id=846810 , https://libvirt.org/firewall.html , https://libvirt.org/formatnetwork.html#examplesRoute, https://saschpe.wordpress.com/2014/03/06/dynamic-iptables-port-forwarding-for-nat-ed-libvirt-networks/ , https://serverfault.com/q/170079/276753 , https://serverfault.com/q/915257/276753 , https://wiki.libvirt.org/page/VirtualNetworking#Routed_mode , https://www.centos.org/forums/viewtopic.php?f=50&t=71454&sid=7f5190a29eebbf755235ec3dc404a47f , https://www.cyberciti.biz/faq/kvm-forward-ports-to-guests-vm-with-ufw-on-linux/ ]

UPDATE:

injects rules which prevents any attempt to port forwarding

-> The current upstream of KVM (libvirt) has resolved the issue!

See this: ...current upstream of KVM (libvirt) has resolved the issue...

Eduardo Lúcio
  • 1,314
0

I have found another solution, which does not require any extra code. This was tested on Fedora 41; I am not sure about CentOS 7 which is rather old by this time.

All the commands in these instructions are to be run as root.

Edit the libvirt network (virsh net-edit default usually), find , change to . Then either just reboot your host or restart the network, usually like this:

virsh net-destroy default
virsh net-start default

At this point your guests have lost communication with the "outside world" because you have disabled libvirt's NAT, but now you add your own NAT.

First, take a look at your zones with firewall-cmd --get-active-zones. Here's what I got:

# firewall-cmd --get-active-zones
FedoraServer (default)
 interfaces: ens18
libvirt
 interfaces: virbr0

Your public zone can have a different name, adjust the rest of the commands accordingly (I think it's usually "public" in RHEL and derivatives). If you don't see virbr0 in the libvirt zone, add it: firewall-cmd --zone libvirt --add-interface virbr0. If virbr0 is in some other zone, use that zone name instead. But it should not be in your public zone.

Now we create the new NAT:

firewall-cmd --permanent --new-policy libvirt-nat
firewall-cmd --permanent --policy libvirt-nat --set-target ACCEPT
firewall-cmd --permanent --policy libvirt-nat --add-ingress-zone libvirt
firewall-cmd --permanent --policy libvirt-nat --add-egress-zone FedoraServer
firewall-cmd --permanent --policy libvirt-nat --add-masquerade
firewall-cmd --reload

Your guests should have their connectivity restored, and with this NAT your port forwarding is rather simple, for example:

firewall-cmd --permanent --zone FedoraServer --add-forward-port=port=8080:proto=tcp:toport=8080:toaddr=192.168.122.117
firewall-cmd --reload

Replace the port number and internal address as necessary.

There is a special case: when your guests have real IPv6 addresses and you have routing of these addresses configured through your host already . In this case you might want to allow the outside world to connect to your guests. If - and that is an if - you want to do so, add another policy:

firewall-cmd --permanent --new-policy libvirt-inbound
firewall-cmd --permanent --policy libvirt-inbound --set-target ACCEPT
firewall-cmd --permanent --policy libvirt-inbound --add-egress-zone libvirt
firewall-cmd --permanent --policy libvirt-inbound --add-ingress-zone FedoraServer
firewall-cmd --reload

Sometimes a cloud provider will give you IPv6 addresses but not route them. There is still a way to use them like this, it is something called "proxying NDP", but it is beyond the scope of this answer.

Mikhail Ramendik
  • 521