1

I'm testing Gateway API with GKE (version 1.21.11-gke.1100). I'm using gatewayClassName: gke-l7-rilb for a Gateway with TLS between the client and the gateway. HTTPS is working perfectly between the client and the loadbalancer using a managed regional SSL certificate .

I have 2 Httproutes referencing 2 kube services (backendRefs). One service is accessible through HTTP and the other through HTTPS (argo-server service from Argo Workflows project if it may helps).

When I create the httproute referencing the service using HTTP, the GCP load balancer backend service is created and working without any problem (Healthy).

But when I create the httproute referencing argo-service, a GCP load balancer backend service is created but not working (not healthy) with an endpoint protocol set to HTTP rather than HTTPS. You should know that I made sure to add to argo-server service the annotation cloud.google.com/app-protocols: '{"web":"HTTPS"}' to enable HTTPS between the load balancer and argo-server application.

If I create the same geatway api configuration using ingress resource and the same argo service definition, endpoint protocol ( of the GCP load balancer backend service )is set correctly to HTTPS and is perfectly healthy and working.

As if httproute of the gateway API the GKE gateway controller are not taking into consideration the cloud.google.com/app-protocols service annotation although it is mentioned that it's relevant for the gateway API here.

Edit 1: added yaml files

  • Gateway:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"regional-internal-https","namespace":"exposition"},"spec":{"addresses":[{"type":"NamedAddress","value":"dev-gateway-internal-lb-static-ip"}],"gatewayClassName":"gke-l7-rilb","listeners":[{"allowedRoutes":{"kinds":[{"kind":"HTTPRoute"}],"namespaces":{"from":"Selector","selector":{"matchLabels":{"exposed":"true"}}}},"name":"https","port":443,"protocol":"HTTPS","tls":{"mode":"Terminate","options":{"networking.gke.io/pre-shared-certs":"plat-dev-europe-west1"}}}]}}
    networking.gke.io/addresses: ""
    networking.gke.io/backend-services: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
      gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
    networking.gke.io/firewalls: ""
    networking.gke.io/forwarding-rules: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
    networking.gke.io/health-checks: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
      gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
    networking.gke.io/last-reconcile-time: "2022-06-16T15:57:45Z"
    networking.gke.io/ssl-certificates: ""
    networking.gke.io/target-proxies: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
    networking.gke.io/url-maps: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
  creationTimestamp: "2022-06-15T08:28:20Z"
  finalizers:
  - gateway.finalizer.networking.gke.io
  generation: 1
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
  - apiVersion: gateway.networking.k8s.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:addresses: {}
        f:gatewayClassName: {}
        f:listeners:
          .: {}
          k:{"name":"https"}:
            .: {}
            f:allowedRoutes:
              .: {}
              f:kinds: {}
              f:namespaces:
                .: {}
                f:from: {}
                f:selector:
                  .: {}
                  f:matchLabels:
                    .: {}
                    f:exposed: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:tls:
              .: {}
              f:mode: {}
              f:options:
                .: {}
                f:networking.gke.io/pre-shared-certs: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-06-15T08:28:20Z"
  - apiVersion: gateway.networking.k8s.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:networking.gke.io/addresses: {}
          f:networking.gke.io/backend-services: {}
          f:networking.gke.io/firewalls: {}
          f:networking.gke.io/forwarding-rules: {}
          f:networking.gke.io/health-checks: {}
          f:networking.gke.io/last-reconcile-time: {}
          f:networking.gke.io/ssl-certificates: {}
          f:networking.gke.io/target-proxies: {}
          f:networking.gke.io/url-maps: {}
        f:finalizers:
          .: {}
          v:"gateway.finalizer.networking.gke.io": {}
      f:status:
        f:addresses: {}
    manager: GoogleGKEGatewayController
    operation: Update
    time: "2022-06-15T08:30:16Z"
  name: regional-internal-https
  namespace: exposition
  resourceVersion: "42337844"
  uid: 59333aea-1a79-4e9b-afbc-595ae9ccdfd7
spec:
  addresses:
  - type: NamedAddress
    value: dev-gateway-internal-lb-static-ip
  gatewayClassName: gke-l7-rilb
  listeners:
  - allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      namespaces:
        from: Selector
        selector:
          matchLabels:
            exposed: "true"
    name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        networking.gke.io/pre-shared-certs: plat-dev-europe-west1
status:
  addresses:
  - type: IPAddress
    value: 10.163.112.28
  conditions:
  - lastTransitionTime: "1970-01-01T00:00:00Z"
    message: Waiting for controller
    reason: NotReconciled
    status: Unknown
    type: Scheduled
  • Httproute:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"hostnames":["argo-server.plat.dev.df.gcp.corp.modified.com"],"parentRefs":[{"kind":"Gateway","name":"regional-internal-https","namespace":"exposition"}],"rules":[{"backendRefs":[{"name":"argo-server","port":2746}]}]}}
  creationTimestamp: "2022-06-15T12:27:04Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/managed-by: {}
        f:spec:
          .: {}
          f:hostnames: {}
          f:parentRefs: {}
          f:rules: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2022-06-15T12:27:04Z"
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:parents: {}
      manager: GoogleGKEGatewayController
      operation: Update
      time: "2022-06-15T12:29:02Z"
  name: argo-server
  namespace: argo
  resourceVersion: "42362026"
  uid: 981ce997-c574-4878-bec1-b03c7707838c
spec:
  hostnames:
    - argo-server.plat.dev.df.gcp.corp.modified.com
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: regional-internal-https
      namespace: exposition
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: argo-server
          port: 2746
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
status:
  parents:
    - conditions:
        - lastTransitionTime: "2022-06-16T17:00:11Z"
          message: ""
          reason: RouteAccepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2022-06-16T17:00:11Z"
          message: ""
          reason: ReconciliationSucceeded
          status: "True"
          type: Reconciled
      controllerName: networking.gke.io/gateway
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: regional-internal-https
        namespace: exposition
  • Service:
apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"web":"HTTPS"}'
    cloud.google.com/backend-config: '{"default": "argo-server-backendconfig"}'
    cloud.google.com/neg: '{"exposed_ports":{"2746":{}}}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"2746":"k8s1-f83345f9-argo-argo-server-2746-4d39c835"},"zones":["europe-west1-c"]}'
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/app-protocols":"{\"web\":\"HTTPS\"}","cloud.google.com/backend-config":"{\"default\": \"argo-server-backendconfig\"}","cloud.google.com/neg":"{\"ingress\": true}","cluster-autoscaler.kubernetes.io/safe-to-evict":"true"},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"ports":[{"name":"web","port":2746,"targetPort":2746}],"selector":{"app":"argo-server"}}}
  creationTimestamp: "2022-06-15T11:44:07Z"
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cloud.google.com/app-protocols: {}
          f:cloud.google.com/backend-config: {}
          f:cluster-autoscaler.kubernetes.io/safe-to-evict: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":2746,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-06-15T12:27:23Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg: {}
    manager: GoogleGKEGatewayController
    operation: Update
    time: "2022-06-15T12:28:06Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg-status: {}
    manager: glbc
    operation: Update
    time: "2022-06-15T12:28:06Z"
  name: argo-server
  namespace: argo
  resourceVersion: "41692832"
  uid: 25024d53-1d31-4165-8033-1843ec5d72ec
spec:
  clusterIP: 10.163.247.121
  clusterIPs:
  - 10.163.247.121
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web
    port: 2746
    protocol: TCP
    targetPort: 2746
  selector:
    app: argo-server
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
MBHA Phoenix
  • 2,029
  • 2
  • 11
  • 25

1 Answers1

3

I found a solution which I think of as a workaround.

  1. Use networking.gke.io/app-protocols: '{"web":"HTTPS"}' annotation rather than cloud.google.com/app-protocols: '{"web":"HTTPS"}'. This annotation is to use at service level, where web is name of the port. This will enable HTTPS between the load balancer and the application (Endpoint protocol of the backend service created for the specified HTTPRoute). This is working perfectly with gatewayClassName: gke-l7-rilb a Regional Internal Load Balancer.

  2. create a custom health check using cloud.google.com/v1 BackendConfig where you set the type to HTTPS and port to 2746. More details here https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#direct_health With ingresses, GCE ingress controller create automatically this health check from the application readiness probe but apparently this function is not yet implemented in the GKE Gateway controller.

  3. Make sure you have firewall rules allowing ingress traffic for Google Cloud health checks on 2746 port With ingresses, GCE ingress controller create automatically the required firewall rule but apparently this function is not yet implemented in the GKE Gateway controller.

Finally I said this is a workaround because I imagine and hope that future version of the GKE gateway controller will fix the 3 issues or points I mentioned above.

MBHA Phoenix
  • 2,029
  • 2
  • 11
  • 25