How to fix ingress 404 default backend

Question

I started to use Kubernetes few month ago and I actually migrate my microServices to my rancher cluster (RKE). Everything works good, my deployments are good and services too. I would like use ingress.

Everything looks good, services are find by ingress and pods are find by services. However when I try to go to the website, I have a 404 error page from ingress controller.

You can see my configuration for juste two paths : one nginx and on grafana. Someone knows how can i fix it and use ingress to do my reverse proxy ?

Thank you so much for your help.

I trie to use rewrite-target without result and add-base-url is deprecated.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    field.cattle.io/creatorId: user-cg5r7
    field.cattle.io/ingressState: '{"bXktaW5ncmVzcy9kZWZhdWx0L3d3dy5zY29sLWVhLm92aC8vbmdpbngvNDI=":""}'
    field.cattle.io/publicEndpoints: '[{"addresses":["51.68.226.21"],"port":80,"protocol":"HTTP","serviceName":"default:nginx-services","ingressName":"default:my-ingress","hostname":"www.scol-ea.ovh","path":"/nginx","allNodes":true}]'
  creationTimestamp: "2019-08-31T10:54:25Z"
  generation: 2
  labels:
    cattle.io/creator: or antoine
  name: my-ingress
  namespace: default
  resourceVersion: "106239"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/my-ingress
  uid: b27b7b20-cbdd-11e9-b16b-fa163ea73397
spec:
  rules:
  - host: www.scol-ea.ovh
    http:
      paths:
      - backend:
          serviceName: nginx-sample
          servicePort: 80
        path: /nginx
      - backend:
          serviceName: prometheus-grafana
          servicePort: http
        path : /grafana

------------------------
apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/targetWorkloadIds: '["deployment:default:nginx-sample"]'
    workload.cattle.io/targetWorkloadIdNoop: "true"
    workload.cattle.io/workloadPortBased: "true"
  creationTimestamp: "2019-08-31T10:03:47Z"
  labels:
    cattle.io/creator: norman
  name: nginx-sample
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1beta2
    controller: true
    kind: deployment
    name: nginx-sample
    uid: 57af9603-cb2a-11e9-b16b-fa163ea73397
  resourceVersion: "102071"
  selfLink: /api/v1/namespaces/default/services/nginx-sample
  uid: 9fffe98c-cbd6-11e9-b16b-fa163ea73397
spec:
  clusterIP: 10.43.183.187
  ports:
  - name: 80tcp02
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    workload.user.cattle.io/workloadselector: deployment-default-nginx-sample
  sessionAffinity: None
  type: ClusterIP

----------------------------
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-08-30T13:44:00Z"
  labels:
    app: prometheus-grafana
    chart: grafana-0.0.31
    heritage: Tiller
    io.cattle.field/appId: prometheus
    release: prometheus
  name: prometheus-grafana
  namespace: default
  resourceVersion: "2536"
  selfLink: /api/v1/namespaces/default/services/prometheus-grafana
  uid: 38ebd878-cb2c-11e9-b16b-fa163ea73397
spec:
  clusterIP: 10.43.142.143
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app: prometheus-grafana
  sessionAffinity: None
  type: ClusterIP

Nick_Kh · Answer 1 · 2019-09-03T12:58:08.313

Actually, Nginx Ingress Controller handles all requests that could not reach target location and sending them to default backend, resulting each request in default backend - 404 page.

Since you've decided to achieve path based routing scenario, rewrite rule can do the trick via nginx.ingress.kubernetes.io/rewrite-target annotation:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  labels:
    cattle.io/creator: or antoine
  name: my-ingress
  namespace: default
spec:
  rules:
  - host: www.scol-ea.ovh
    http:
      paths:
      - backend:
          serviceName: nginx-sample
          servicePort: 80
        path: /nginx(/|$)(.*)
      - backend:
          serviceName: prometheus-grafana
          servicePort: http
        path : /grafana(/|$)(.*)

Once you apply above Ingress manifest, the path based routing can be afforded in the following rewrite examples:

www.scol-ea.ovh/nginx -> www.scol-ea.ovh/

www.scol-ea.ovh/grafana -> www.scol-ea.ovh/

I wouldn't recommend to share any user sensitive data across published manifests or any other data objects, like public endpoint in your case.

score 2 · Answer 2 · answered Apr 27 '20 at 19:55

If you are using nginx as deployment proxy in the kubernetes and facing the issue then the below configuration can resolve the redirection issue.

Either you can include the below variables in the grafan.ini file or can expose as env variables as shown below,

GF_SERVER_DOMAIN=abc.google.com
GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:/grafana

So the Grafana deployment will look like the below one.

        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: grafana
          namespace: monitoring
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: grafana
            spec:
              containers:
              - name: grafana
                image: grafana/grafana:latest
                env:
                - name: GF_SERVER_DOMAIN
                  value: "abc.google.com"
                - name: GF_SERVER_ROOT_URL
                  value: "%(protocol)s://%(domain)s:/grafana"

                ports:
                - name: grafana
                  containerPort: 3000
                resources:
                  limits:
                    memory: "2Gi"
                    cpu: "1000m"
                  requests: 
                    memory: "1Gi"
                    cpu: "500m"
                volumeMounts:
                  - mountPath: /var/lib/grafana
                    name: grafana-storage-volume
                  - mountPath: /etc/grafana/provisioning/datasources
                    name: grafana-datasources
                    readOnly: false

And you have to update the nginx server location block as below,

       location /grafana {
           proxy_pass http://grafana.monitoring.svc.cluster.local:3000;
           rewrite ^/grafana/(.*) /$1 break;
           proxy_set_header Host $host;
        }

score 0 · Answer 3 · answered Sep 04 '19 at 05:26

0

I believe you should use nginx.ingress.kubernetes.io/backend-protocol: HTTPS annotation in your yaml

answered Sep 04 '19 at 05:26

gkc

449
6
18

score 0 · Answer 4 · answered May 10 '23 at 10:24

One thing I stumbled upon, make sure that in your spec rules, backend service names are correct.

Instead of this:

      - path: /app-ui
        pathType: Prefix
        backend:
          service:
            name: app-ui-service
            port:
              number: 8080

I had this:

      - path: /app-ui
        pathType: Prefix
        backend:
          service:
            name: app-ui
            port:
              number: 8080

Another thing, make sure that if you are using Nginx Ingress, that you specify the ingress class name.

spec:
  ingressClassName: nginx

How to fix ingress 404 default backend

4 Answers4

Linked