Kibana not accessible with ingress path url after kubernetes deployment

Question

I am unable to access the Kibana server UI through ingress path url I have deployed the Kibana pod along with Elasticsearch on Kubernetes cluster. While access the UI it is stating as "503 service unavailable" and it is re-directing path as https://myserver.com/spaces/enter. Both the elasticsearch and kibana pods are running. And I am able to curl my elasticsearch pod through my ingress path url. Can someone help with the issue.

Kibana yaml files:

deployment.yaml

---
  apiVersion: "apps/v1"
  kind: "Deployment"
  metadata: 
    name: "kibana-development"
    namespace: "development"
  spec: 
    selector: 
      matchLabels: 
        app: "kibana-development"
    replicas: 1
    strategy: 
      type: "RollingUpdate"
      rollingUpdate: 
        maxSurge: 1
        maxUnavailable: 1
    minReadySeconds: 5
    template: 
      metadata: 
        labels: 
          app: "kibana-development"
      spec: 
        containers: 
          - 
            name: "kibana-development"
            image: "docker.elastic.co/kibana/kibana:7.10.2"
            imagePullPolicy: "Always"
            
            env:
             - name: "ELASTICSEARCH_HOSTS"
               value: "https://my-server.com/elasticsearch"
            
            ports: 
              - 
                containerPort: 5601
                protocol: TCP     
        imagePullSecrets: 
          - 
            name: "kibana"

service.yaml

---
  apiVersion: "v1"
  kind: "Service"
  metadata: 
    name: "kibana-development"
    namespace: "development"
    labels: 
      app: "kibana-development"
  spec: 
    ports: 
      - 
        port: 56976
        targetPort: 5601
    selector: 
      app: "kibana-development"

ingress.yaml

---
  apiVersion: "networking.k8s.io/v1beta1"
  kind: "Ingress"
  metadata: 
    name: "kibana-development-ingress"
    namespace: "development"
    annotations: 
      nginx.ingress.kubernetes.io/rewrite-target: "/$1"
  spec: 
    rules: 
      - 
        host: "my-server.com"
        http: 
          paths: 
            - 
              backend: 
                serviceName: "kibana-development"
                servicePort: 56976
              path: "/kibana/(.*)"

I am able to access Kibana through cliuster-ip:port, but not with ingress path url. Is there any annotations that I am missing? Or the version 7.10.2 for elasticsearch and kibana not stable. I checked my endpoint, it is showing my cluster-ip

Hello @SVD, first - how did you setup your cluster? Is it bare-metal or did you use some cloud providor? I can see in your ingress `host: "my-server"`. I think, you should have `host: myserver.com`. Try to change it. This error could be also caused, when Ingress config referencing the incorrect services name or when you have problem with certificates. — Mikołaj Głodziak, Jun 08 '21 at 11:30
Hi @MikołajGłodziak, instead of my-server.com in actual it is my dns name more precisely, I have deployed it on EKS cluster — SVD, Jun 08 '21 at 11:51
now you have still different names. In the ingress you have `my-server.com`, but in the deployment `myserver.com`. Try to remove the `-` character in the `ingress.yaml` — Mikołaj Głodziak, Jun 08 '21 at 11:57
Hi @MikołajGłodziak, I have edited that, the issue still persists — SVD, Jun 08 '21 at 12:02
A 503 can be an internal error caused by the application - you could try viewing logs for the pod `kubectl get pods`, `kubectl log myPodName`. Alternatively `kubectl describe pod myPodName` may give some insight. This can be caused by a missing endpoint on a service. Check that the service has an endpoint assigned that maps to the internal IP(s) of the pod(s). `kubectl get service`, `kubectl describe service myServiceName` To check the actual endpoint(s) - `kubectl get ep`. Please add this info to the question. — Mikołaj Głodziak, Jun 08 '21 at 12:06
@SVC, You can see also an explanation of how a service needs to map to endpoints to connect to the pod - https://stackoverflow.com/questions/52857825 — Mikołaj Głodziak, Jun 08 '21 at 12:06
@MikołajGłodziak, I describe the service and it is showing the endpoint as my pod cluster ip, through which I am able to access kibana. But I want to access it through ingress path url. There it is showing 503 service unavilable — SVD, Jun 08 '21 at 12:20
In the ingress, you are trying to reference `serviceName: "kibana-development"`. However, your service has a different name: `name: "kibana-development-service"`. Correct it and test it again. — Mikołaj Głodziak, Jun 08 '21 at 12:40
@MikołajGłodziak tried, by correcting the name, but still the same issue — SVD, Jun 08 '21 at 13:22
@MikołajGłodziak thanks for your help, issue resolved now, I have added the answer. — SVD, Jun 08 '21 at 13:39

score 1 · Accepted Answer · answered Jun 08 '21 at 13:37

Issue resolved now, needed to add the below two env variables in deployment.yaml file.

- 
  name: "SERVER_BASEPATH"
  value: "/kibana-development"
                
-
  name: "SERVER_REWRITEBASEPATH"
  value: "false"

Don't forget the "/" in SERVER_BASEPATH value

Kibana not accessible with ingress path url after kubernetes deployment

1 Answers1