Google Cloud not Applying the BackendConfig on the Load Balancer Health Check

Question

I have a deployment. The pod container's have no readinessProbe(s), because the healtcheck will be configured using a BackendConfig.

The service is as follows:

---
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
  namespace: my-namespace
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/backend-config: '{"default": "my-app-backendconfig"}'
spec:
  type: NodePort  
  externalTrafficPolicy: Local  
  ports:
  - name: flower-nodeport-port
    port: 80
    targetPort: 5555
    protocol: TCP
  selector:
    app: my-app

And this is the BackendConfig to create the health check:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-app-backendconfig
  namespace: my-namespace
  labels:
    app: my-app
spec:
  healthCheck:
    checkIntervalSec: 20
    timeoutSec: 1
    healthyThreshold: 1
    unhealthyThreshold: 5
    type: TCP        

The problem is that Google Cloud is not applying the healtcheck as I described it. Check the screenshot below:

As you can see, values like "Unhealthy threshold" and "Timeout" are not being taken into account.

What am I doing wrong?

Answer 1

your type field on BackendConfig is TCP, docs say only http/https allowed https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-configuration#direct_health

Answer 2

If the Pod template for the Service's serving Pods does not have a container with a readiness probe whose attributes can be interpreted as health check parameters, the default values are used to create the health check. Both the Anthos Ingress controller and the GKE Ingress controller can create a health check using only the default values.

The ideal way to use the ‘BackendConfig’ is when the serving pods for your service contains multiple containers, if you're using the Anthos Ingress controller or if you need control over the port used for the load balancer's health checks, then you should use a BackendConfig CDR to define health check parameters. Refer to the health check for each backend service corresponding to a Kubernetes Service and Parameters from a readiness probe for more information.

Refer to the similar SO1,SO2 for more information.

Also go through the how health checks work for more information.

Finally refer to this How to troubleshoot unhealthy backends in Google Cloud Load Balancers may help to resolve your issue.