6

I have three nodes in my cluster who are behind a firewall I do not control. This firewall has a public IP connected to it and can forward traffic to my kubernetes node. It has port 80 and 443 opened to my node.

Initially, I used the public IP in the MetalLB config like this:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 186.xx.xx.xx-186.xx.xx.xx

But after reading this answer of another question I'm guessing it is invalid since the IP used by MetalLB needs to be on the same subnet as the nodes? And they are all using private IPs.

When I tested locally a HTTP server listening on port 80 and ran it on the actual node (not in the cluster) then I was able get a response on the public IP from outside the network.

So my question is: How do I make MetalLB or Nginx ingress controller listen on port 80 and 443 for incoming request?

When using curl 186.xx.xx.xx:80 on of the nodes in the cluster then I'm receiving a response from the nginx ingress controller. But not when doing it outside of the node.

TryingMyBest
  • 131
  • 1
  • 1
  • 7
  • 1
    Assuming that you can port forward traffic from your firewall, have you tried this? 1. Tell `metallb` about a **single internal ip** that it can assign. 2. Modify `nginx-ingress-controller` `Service` manifest with an annotation that will assign this **internal** ip address to a `Service`. 3. Use port-forwarding (port 80,443) on your firewall to this freshly assigned ip of `nginx-ingress-controller`. Run `$ curl firewall-ip:80` and see that your `nginx-ingress-controller` responds. – Dawid Kruk Feb 17 '21 at 08:50
  • This is exactly what I did! I managed to solve it. The main problem was that the global IP was not in the subnet as the other nodes. – TryingMyBest Feb 17 '21 at 16:03

1 Answers1

14

Answering the question:

How can I create a setup with Kubernetes cluster and separate firewall to allow users to connect to my Nginx Ingress controller which is exposing my application.

Assuming the setup is basing on Kubernetes cluster provisioned in the internal network and there is a firewall between the cluster and the "Internet", following points should be addressed (there could be some derivatives which I will address):

  • Metallb provisioned on Kubernetes cluster (assuming it's a bare metal self-managed solution)
  • Nginx Ingress controller with modified Service
  • Port-forwarding set on the firewall

Service of type Loadbalancer in the most part (there are some exclusions) is a resource that requires a cloud provider to assign an External IP address for your Service.

A side note!

More reference can be found here:

For solutions that are on premise based, there is a tool called metallb:

Kubernetes does not offer an implementation of network load-balancers (Services of type LoadBalancer) for bare metal clusters. The implementations of Network LB that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created.

Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem.

MetalLB aims to redress this imbalance by offering a Network LB implementation that integrates with standard network equipment, so that external services on bare metal clusters also “just work” as much as possible.

Metallb.universe.tf

Following the guide on the installation/configuration of metallb, there will be a configuration for a single internal IP address that the firewall will send the traffic to:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: single-ip # <-- HERE
      protocol: layer2
      addresses:
      - 10.0.0.100/32 # <-- HERE

This IP address will be associated with the Service of type LoadBalancer of Nginx Ingress controller.

The changes required with the Nginx Ingress manifest (Service part):

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.universe.tf/address-pool: single-ip # <-- IMPORTANT
  labels:
    # <-- OMMITED --> 
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    # <-- OMMITED -->

Above changes in the YAML manifest will ensure that the address that was configured in a metallb ConfigMap will be used with the Service.

A side note!

You can omit the metallb and use the Service of type Nodeport but this carries some disadvantages.

The last part is to set the port-forwarding on the firewall. The rule should be following:

  • FIREWALL_IP:80 -> SINGLE_IP:80
  • FIREWALL_IP:443 -> SINGLE_IP:443

After that you should be able to communicate with your Nginx Ingress controller by:

  • $ curl FIREWALL_IP:80

Additional resources:

Dawid Kruk
  • 8,982
  • 2
  • 22
  • 45
  • What about the other case, that every node has its own public IP? – Mohammed Noureldin Sep 05 '22 at 19:45
  • Then the LoadBalancer of some sort is still advisable to check on the health of the individual nodes. You could run it with a hostPort but this Is a single point of failure. A lot depends on exact infrastructure you have in your environment still. – Dawid Kruk Sep 06 '22 at 17:46