GCP load balancer backend status unknown

Question

I'm flabbergasted.

I have a staging and production environment. Both environments have the same deployments, services, ingress, firewall rules, and both serve a 200 on /.

However, after turning on the staging environment and provisioning the same ingress, the staging service fails with Some backend services are in UNKNOWN state. Production is still live.

Both the frontend and backend pods are ready on GKE. I've manually tested the health checks and they pass when I visit /.

I see nothing in the logs or gcp docs pointing in the right direction. What could I have possibly broken?

ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: fanout-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "STATIC-IP"
spec:
  backend:
    serviceName: frontend
    servicePort: 8080
  tls:
  - hosts:
    - <DOMAIN>
    secretName: staging-tls
  rules:
  - host: <DOMAIN>
    http:
      paths:
      - path: /*
        backend:
          serviceName: frontend
          servicePort: 8080
      - path: /backend/*
        backend:
          serviceName: backend
          servicePort: 8080

frontend.yaml:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: frontend
  name: frontend
  namespace: default
spec:
  ports:
  - nodePort: 30664
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: frontend
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  generation: 15
  labels:
    app: frontend
  name: frontend
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - image: <our-image>
        name: frontend
        ports:
        - containerPort: 8080
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 3
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 3

possibly related: https://github.com/kubernetes/kubernetes/issues/45438 — Martin Zeitler, Aug 23 '18 at 21:33
Thanks @MartinZeitler - I've seen that. Unfortunately, It didn't really help. — Mike, Aug 23 '18 at 22:34
it's a bug report, not meant to help; there also another one linked there. — Martin Zeitler, Aug 23 '18 at 22:44
It's closed with the gcp folks blaming it on a project config issue — Mike, Aug 23 '18 at 22:54

score 9 · Answer 1 · answered Aug 24 '18 at 06:54

Yesterday even this guide https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer

didn't work. Don't know what happened but even waiting 30minutes + the ingress was reporting UNKNOWN state for backends .

After 24 hours, things seem to be much better. L7 http ingress works but with big delay on reporting healthy backends.

score 3 · Answer 2 · answered Aug 24 '18 at 00:43

3

I think this is a bug. I created a new cluster and couldn't reproduce. If anyone hits this again, I would suggest trying a new cluster.

answered Aug 24 '18 at 00:43

Mike

1,180
3
15
28

score 1 · Answer 3 · answered Oct 26 '20 at 19:13

1

If it started happening after altering the scalability settings of your cluster:

Deleting and re-creating the Ingress resource might help - in my case it fixed it almost immediately.

Steps I followed:

kubectl delete ingress <faulty_ingress>
kubectl apply -f <my_ingress.yaml>

answered Oct 26 '20 at 19:13

JesusIniesta

10,412
1
35
28

1

That is usually what works when hitting this one. But who wants to delete their entire ingress on a prod system just to update one config... – Riscie Nov 06 '21 at 14:08

score 1 · Answer 4 · answered Jun 24 '21 at 17:14

1

What worked for me was deleting and recreating the BackendConfig.

answered Jun 24 '21 at 17:14

NickTheDev

209
4
10

score 0 · Answer 5 · answered Aug 27 '18 at 14:53

Are you still experiencing this issue?

I tried to reproduce following the Google public documentation on: Setting up HTTP Load Balancing with Ingressto deploy:

A web App using the sample web application container image that listens on a HTTP server on port 8080:

However, it seems to be working now. So if you are still having this issue, please consider filing a public issue against the kubernetes/ingress-gce using the Google issue-tracking tool. Include as much details as possible, including steps to reproduce for so that this issue can get a better visibility as well as more sampling.

Please note:

The Issue Tracker User Content and Conduct Policy details the types of information that are inappropriate for submitting to Issue Tracker which includes things like sensitive personal information and spam. Please do not submit inappropriate content in Issue Tracker.

Repo Output $ kubectl describe ing

sunny@test-dev:~$ kubectl describe ing basic-ingress
Name:             basic-ingress
Namespace:        default
Address:          xx.xxx.xxx.228
Default backend:  web:8080 (10.8.2.6:8080)
Rules:
Host  Path  Backends
----  ----  --------
*     *     web:8080 (10.8.2.6:8080)
Annotations:
target-proxy:     k8s-tp-default-basic-ingress--f5636f071d87exxx
url-map:          k8s-um-default-basic-ingress--f5636f071d87exxx
backends:         {"k8s-be-31544--f5636f071d87exxx":"HEALTHY"}
forwarding-rule:  k8s-fw-default-basic-ingress--f5636f071d87exxx
Events:
Type    Reason   Age                From                     Message
----    ------   ----               ----                     -------
Normal  Service  7m (x376 over 2d)  loadbalancer-controller  default backend set to web:31544

GCP load balancer backend status unknown

5 Answers5

Linked