Dashboard not running

Question

I have setup kubenertes on ubuntu server using this link.

Then I installed kubernetes dashboard using:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc6/aio/deploy/recommended.yaml

Then I changed the ClusterIP to NodePort 32323, the service to NodePort.

But container is not running.

uday@dockermaster:~$ kubectl -n kubernetes-dashboard get all
NAME                                             READY   STATUS             RESTARTS   AGE
pod/dashboard-metrics-scraper-779f5454cb-pqfrj   1/1     Running            0          50m
pod/kubernetes-dashboard-64686c4bf9-5jkwq        0/1     CrashLoopBackOff   14         50m

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.103.22.252   <none>        8000/TCP        50m
service/kubernetes-dashboard        NodePort    10.102.48.80    <none>        443:32323/TCP   50m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dashboard-metrics-scraper   1/1     1            1           50m
deployment.apps/kubernetes-dashboard        0/1     1            0           50m

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/dashboard-metrics-scraper-779f5454cb   1         1         1       50m
replicaset.apps/kubernetes-dashboard-64686c4bf9        1         1         0       50m

uday@dockermaster:~$ kubectl -n kubernetes-dashboard describe svc kubernetes-dashboard
Name:                     kubernetes-dashboard
Namespace:                kubernetes-dashboard
Labels:                   k8s-app=kubernetes-dashboard
Annotations:              Selector:  k8s-app=kubernetes-dashboard
Type:                     NodePort
IP:                       10.102.48.80
Port:                     <unset>  443/TCP
TargetPort:               8443/TCP
NodePort:                 <unset>  32323/TCP
Endpoints:
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Other apps are working fine with NodePort, whether Tomcat/nginx/databases.

But here, it is failing with container creation.

C:\Users\uday\Desktop>kubectl.exe get pods -n kubernetes-dashboard
NAME                                         READY   STATUS             RESTARTS   AGE
dashboard-metrics-scraper-779f5454cb-pqfrj   1/1     Running            1          20h
kubernetes-dashboard-64686c4bf9-g9z2k        0/1     CrashLoopBackOff   84         18h

C:\Users\uday\Desktop>kubectl.exe describe pod kubernetes-dashboard-64686c4bf9-g9z2k -n kubernetes-dashboard
Name:         kubernetes-dashboard-64686c4bf9-g9z2k
Namespace:    kubernetes-dashboard
Priority:     0
Node:         slave-node/10.0.0.6
Start Time:   Sat, 28 Mar 2020 14:16:54 +0000
Labels:       k8s-app=kubernetes-dashboard
              pod-template-hash=64686c4bf9
Annotations:  <none>
Status:       Running
IP:           182.244.1.12
IPs:
  IP:           182.244.1.12
Controlled By:  ReplicaSet/kubernetes-dashboard-64686c4bf9
Containers:
  kubernetes-dashboard:
    Container ID:  docker://470ee8c61998c3c3dda86c58ad17817468f55aa73cd4feecf3b018977ce13ca3
    Image:         kubernetesui/dashboard:v2.0.0-rc6
    Image ID:      docker-pullable://kubernetesui/dashboard@sha256:61f9c378c427a3f8a9643f83baa9f96db1ae1357c67a93b533ae7b36d71c69dc
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --auto-generate-certificates
      --namespace=kubernetes-dashboard
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Sun, 29 Mar 2020 09:01:31 +0000
      Finished:     Sun, 29 Mar 2020 09:02:01 +0000
    Ready:          False
    Restart Count:  84
    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-pzfbl (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kubernetes-dashboard-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-certs
    Optional:    false
  tmp-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kubernetes-dashboard-token-pzfbl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-token-pzfbl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                     From                 Message
  ----     ------   ----                    ----                 -------
  Warning  BackOff  4m49s (x501 over 123m)  kubelet, slave-node  Back-off restarting failed container

kubectl.exe logs kubernetes-dashboard-64686c4bf9-g9z2k -n kubernetes-dashboard
2020/03/29 09:01:31 Starting overwatch
2020/03/29 09:01:31 Using namespace: kubernetes-dashboard
2020/03/29 09:01:31 Using in-cluster config to connect to apiserver
2020/03/29 09:01:31 Using secret token for csrf signing
2020/03/29 09:01:31 Initializing csrf token from kubernetes-dashboard-csrf secret
panic: Get https://10.96.0.1:443/api/v1/namespaces/kubernetes-dashboard/secrets/kubernetes-dashboard-csrf: dial tcp 10.96.0.1:443: i/o timeout

goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/client/csrf.(*csrfTokenManager).init(0xc0004e2dc0)
    /home/travis/build/kubernetes/dashboard/src/app/backend/client/csrf/manager.go:40 +0x3b0
github.com/kubernetes/dashboard/src/app/backend/client/csrf.NewCsrfTokenManager(...)
    /home/travis/build/kubernetes/dashboard/src/app/backend/client/csrf/manager.go:65
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).initCSRFKey(0xc00043ae80)
    /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:499 +0xc6
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).init(0xc00043ae80)
    /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:467 +0x47
github.com/kubernetes/dashboard/src/app/backend/client.NewClientManager(...)
    /home/travis/build/kubernetes/dashboard/src/app/backend/client/manager.go:548
main.main()
    /home/travis/build/kubernetes/dashboard/src/app/backend/dashboard.go:105 +0x20d

Describe the deployment and add that output – Arghya Sadhu Mar 28 '20 at 13:26 — Arghya Sadhu, Mar 28 '20 at 13:26

TJ Zimmerman · Answer 1 · 2020-03-28T17:18:39.987

The Problem

The reason the application is not coming up is because the Dashboard container itself is not running. If you look at the output you provided you can see this:

pod/kubernetes-dashboard-64686c4bf9-5jkwq 0/1 CrashLoopBackOff 14

So how do we troubleshoot this? Well, there are three principle ways. One of which you'll probably be using more than the other two.

Describe

Describe is a is a command that allows you to fetch details about a resource in kubernetes. This could be metadata information, the amount of replicas you assigned, or even some events depicting why a resource is failing to start. For example, a referenced Container Image in your Pod manifest can not be found in the usable container registries. The syntax for using Describe is like so:

kubectl describe pod -n kubernetes-dashboard kubernetes-dashboard-64686c4bf9-5jkwq

Here are some great docs on the tool as well.

Logs

The next troubleshooting step you'll likely take advantage of in Kubernetes is using the logging architecture. As you're probably aware, when a Docker container is spawned it is common practice to have the logs that are produced by the application be redirected to STDOUT or STDERR for the process. Kubernetes them captures this log data for you and provides an API abstraction layer with which you can interact with it. Sometimes your Describe events won't have any indication of why a process isn't running. However, you can then proceed with grabbing logs from the process to determine what is going wrong. An example syntax might look like:

kubectl logs -f -n kubernetes-dashboard kubernetes-dashboard-64686c4bf9-5jkwq

Exec

The last common troubleshooting technique is Exec. Exec effectively allows you to attach to the a shell in a running container so that you can interact with the live environment to troubleshoot the application. This allows you to do things like see if configuration files were properly staged on the container's filesystem, determine if environment variables were properly expanded and set, etc. An example syntax for Exec might look like:

kubectl exec -it -n kubernetes-dashboard kubernetes-dashboard-64686c4bf9-5jkwq sh

In this case, however, your pod is in a CrashLoopBackoff state. This means that you will not be able to exec into it due to the fact that the container is not running. The Kubernetes API Server recognized a pattern of failures and automatically reduced its attempts at scheduling the workload accordingly.

Here is a good thread on how to troubleshoot pods that enter this state.

Summary

So, now that I've said all of this. How do we answer your question? Well, we can't answer it directly. But I sort of did with my summary above. Because the real answer you're looking for is how to properly troubleshoot linux containers running in Kubernetes. These issues will be a reoccurring theme in your experience with Kubernetes so it's essential to develop debugging skills in the ecosystem as soon as possible.

If the Describe, Logs, and Exec command are unable to help you find out why the Dashboard pod is failing to come up, feel free to add a comment on this answer requesting additional support and I'll be happy to help where I can!

Hey, sorry this is and old question but I am having the exact same issue. I followed these steps and have found the issue, but I am unclear as to how to resolve it. Would you be able to review my question https://stackoverflow.com/questions/70398496/kubernetes-dashboard-crashloopbackoff-timeout-error-on-raspberry-pi-cluster? — Matthew Vine, Dec 17 '21 at 20:21
Dunno why, but the last line in your logs before the failure is ` Initializing csrf token from kubernetes-dashboard-csrf secret` so it's probably related to that. I googled it and found a few Github issues discussing it. Maybe a problematic node? Try rebooting all your nodes, reprovisoning the pods, and trying again? — TJ Zimmerman, Dec 18 '21 at 21:13

Dashboard not running

1 Answers1