8

So I'm working on a project that involves managing many postgres instances inside of a k8s cluster. Each instance is managed using a Stateful Set with a Service for network communication. I need to expose each Service to the public internet via DNS on port 5432.

The most natural approach here is to use the k8s Load Balancer resource and something like external dns to dynamically map a DNS name to a load balancer endpoint. This is great for many types of services, but for databases there is one massive limitation: the idle connection timeout. AWS ELBs have a maximum idle timeout limit of 4000 seconds. There are many long running analytical queries/transactions that easily exceed that amount of time, not to mention potentially long-running operations like pg_restore.

So I need some kind of solution that allows me to work around the limitations of Load Balancers. Node IPs are out of the question since I will need port 5432 exposed for every single postgres instance in the cluster. Ingress also seems less than ideal since it's a layer 7 proxy that only supports HTTP/HTTPS. I've seen workarounds with nginx-ingress involving some configmap chicanery, but I'm a little worried about committing to hacks like that for a large project. ExternalName is intriguing but even if I can find better documentation on it I think it may end up having similar limitations as NodeIP.

Any suggestions would be greatly appreciated.

Lee Hampton
  • 410
  • 5
  • 13
  • Perhaps you should investigate a way of being able to resume a connection in case it gets dropped? It probably would make a good addition to your system in case connections get dropped due to maintenance, or bad connections anywhere locally, in the internet or at AWS. – Paul Annetts Feb 22 '19 at 07:53
  • I definitely agree that ensuring connections don't get dropped is a crucial client-side design choice. Unfortunately, in this system I won't have control over the client side, and there are a good deal of users whom I anticipate will just fire off 1hr+ transactions in psql and eventually get disconnected if the transaction in Postgres lasts longer than the timeout period. – Lee Hampton Feb 22 '19 at 13:31
  • Are your TCP streams encapsulated in TLS? – webwurst Feb 24 '19 at 15:13
  • @webwurst Not necessarily, but I would definitely need the option for them to be. That being said I wouldn't need to terminate the TLS as the ingress point, Postgres has the ability to do that. – Lee Hampton Feb 24 '19 at 20:43
  • One update: I'm also looking into the `tcp_keepalives_idle` setting (see [runtime configurations](https://www.postgresql.org/docs/current/runtime-config-connection.html)) on Postgres, which in theory should force the server to send a keepalive back to the client at a specified interval and update the Load Balancer's idle timeout. However, I'm not sure if there are any gotchas around client side TCP configurations, guarantees around postgres sending the keepalive even when under maximum load, etc., so could use some guidance on that path as well. – Lee Hampton Feb 25 '19 at 16:40
  • Can you elaborate on how you need port 5432 exposed for _every single_ postgres instance in the cluster? Do you need each instance to be available on port 5432 of a _different_ node? – Simon Tesar Feb 27 '19 at 15:24
  • @SimonTesar I'm building a product that needs to expose an instance of Postgres for every user of the product -- eventually thousands of users. The users expect the database to behave as a standard Postgres deployment, so giving them an exotic port is not ideal. I just need to provide each user some entrypoint at port 5432. The LoadBalancer resource provides this perfectly, if it weren't for the pesky idle timeout setting. – Lee Hampton Feb 27 '19 at 17:03
  • There's a way how to expose TCP services through the Ingress, in case that would make sense for you: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ – Bernard Halas Feb 28 '19 at 08:03

2 Answers2

2

The Kubernetes ingress controller implementation Contour from Heptio can proxy TCP streams when they are encapsulated in TLS. This is required to use the SNI handshake message to direct the connection to the correct backend service.

Contour can handle ingresses, but introduces additionally a new ingress API IngressRoute which is implemented via a CRD. The TLS connection can be terminated at your backend service. An IngressRoute might look like this:

apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: postgres
  namespace: postgres-one
spec:
  virtualhost:
    fqdn: postgres-one.example.com
    tls:
      passthrough: true
  tcpproxy:
    services:
    - name: postgres
      port: 5432
  routes:
  - match: /
    services:
    - name: dummy
      port: 80
webwurst
  • 4,830
  • 3
  • 23
  • 32
  • So this is really promising. The one issue, though, is that psql and many other postgres clients don't support SNI out of the box! :-( `psql` doesnt doesn’t set the servername using `SSL_set_tlsext_host_name` when connecting, rendering SNI not an option. Sucks not having control over the client... – Lee Hampton Mar 04 '19 at 19:47
0

ha proxy supports tcp load balancing. you can look at ha-proxy as a proxy and load balancer for postgres database. it can support both tls and non tls connections.

P Ekambaram
  • 15,499
  • 7
  • 34
  • 59
  • I would love to use HA Proxy for this. I just need to know a good way to do that within the kubernetes framework. I need to be able to tie a service endpoint to HA Proxy and have the K8s framework dynamically update this when a pod changes nodes, a node gets destroyed, etc. This is what the Load Balancer object in kubernetes does, but you only have the option to provision the cloud provider's load balancer, ELB in this case. The k8s Ingress object may make this possible, but it's currently built as an L7 proxy that only supports HTTP/HTTPS without some hackery. – Lee Hampton Feb 26 '19 at 16:45
  • you can look at ha proxy ingress controller if you want to update the endpoints automatically when a pod/node goes down. --> https://github.com/jcmoraisjr/haproxy-ingress – P Ekambaram Feb 26 '19 at 17:34
  • i havent used ha-proxy ingress controller though. you will have to check and see if it meets your requirement. but used ha-proxy to proxy database traffic to postgresdb that is running in k8s – P Ekambaram Feb 26 '19 at 17:36