After having problems with connectivity in my old (working completely fine until it didn't) Rancher installation in my homelab, I decided to reinstall my single-node Rancher installation on k3s (running on Ubuntu server) from scratch, hoping that this would be the easiest way, but it still does not work.
The admin UI is not accessible, any requests to the configured domain are timing out (the same problem I had before). So here are all steps I used in order to set it up:
- uninstall k3s:
/usr/local/bin/k3s-uninstall.sh - reinstall supported k3s version:
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.25.11+k3s1 sh -s - server --cluster-init - add helm chart:
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest - copy k3s config in order to be able to run
kubectlcommands instead ofsudo k3s kubectl:sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown aalex:aalex ~/.kube/config - create cattle-system namespace:
kubectl create namespace cattle-system - install cert-manager:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.11.0
- install rancher:
helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.mydomain.tld --set replicas=1 --set bootstrapPassword=password
This then shows that within a couple of minutes, the Rancher backend should be accessible. I have configured the DNS entry to point to my public IP address and forwarded port 80, 443 and 32400 to point to the ubuntu server, so on the ingress side everything should be working. I suspect the problems to be internal because when I run
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-system helm-operation-2786k 1/2 Error 0 30m
cattle-system helm-operation-2bvsk 1/2 Error 0 20m
cattle-system helm-operation-2kbjh 1/2 Error 0 34m
cattle-system helm-operation-2szb7 1/2 Error 0 17m
cattle-system helm-operation-5bksl 1/2 Error 0 39m
cattle-system helm-operation-5rx7n 1/2 Error 0 21m
cattle-system helm-operation-75cwd 2/2 Running 0 4m3s
cattle-system helm-operation-7t5g8 2/2 Running 0 8m16s
cattle-system helm-operation-8dkxf 1/2 Error 0 24m
cattle-system helm-operation-9jpts 2/2 Running 0 9m19s
cattle-system helm-operation-dvg87 1/2 Error 0 11m
cattle-system helm-operation-gbv8w 1/2 Error 0 13m
cattle-system helm-operation-hql8h 1/2 Error 0 10m
cattle-system helm-operation-k4pcm 1/2 Error 0 28m
cattle-system helm-operation-kh7kf 1/2 Error 0 31m
cattle-system helm-operation-ldm9n 1/2 Error 0 15m
cattle-system helm-operation-lf7zt 1/2 Error 0 26m
cattle-system helm-operation-p9kmb 1/2 Error 0 18m
cattle-system helm-operation-ppmhg 1/2 Error 0 38m
cattle-system helm-operation-pz6rd 1/2 Error 0 14m
cattle-system helm-operation-q7f42 1/2 Error 0 19m
cattle-system helm-operation-qhzzn 1/2 Error 0 35m
cattle-system helm-operation-qkffm 2/2 Running 0 5m6s
cattle-system helm-operation-qlh5q 2/2 Running 0 116s
cattle-system helm-operation-r5vgm 2/2 Running 0 6m10s
cattle-system helm-operation-r8w2t 2/2 Running 0 54s
cattle-system helm-operation-rwrcn 1/2 Error 0 27m
cattle-system helm-operation-s4xgh 1/2 Error 0 12m
cattle-system helm-operation-sl8s2 1/2 Error 0 25m
cattle-system helm-operation-sptzf 1/2 Error 0 36m
cattle-system helm-operation-vcxsq 1/2 Error 0 16m
cattle-system helm-operation-vwz4t 2/2 Running 0 3m
cattle-system helm-operation-x2w9x 1/2 Error 0 32m
cattle-system helm-operation-x69qf 1/2 Error 0 33m
cattle-system helm-operation-x95fc 1/2 Error 0 22m
cattle-system helm-operation-z8z67 1/2 Error 0 29m
cattle-system helm-operation-zk5h9 2/2 Running 0 7m13s
cattle-system helm-operation-zk8j7 1/2 Error 0 37m
cattle-system rancher-669557c8fd-cmrb2 1/1 Running 0 43m
cert-manager cert-manager-59bf757d77-84gkv 1/1 Running 0 46m
cert-manager cert-manager-cainjector-547c9b8f95-lz9cd 1/1 Running 0 46m
cert-manager cert-manager-webhook-6787f645b9-b6xjf 1/1 Running 0 46m
kube-system coredns-8b9777675-qplrr 1/1 Running 0 47m
kube-system helm-install-traefik-8m7ck 0/1 Completed 1 47m
kube-system helm-install-traefik-crd-7q4f8 0/1 Completed 0 47m
kube-system local-path-provisioner-69dff9496c-sqx5q 1/1 Running 0 47m
kube-system metrics-server-854c559bd-2m7rd 1/1 Running 0 47m
kube-system svclb-traefik-2bcb49c9-vmk8h 2/2 Running 0 46m
kube-system traefik-66fd46ccd-54v59 1/1 Running 0 46m
As you can see, the helm-operations are erroring out. When taking a look at the logs, they do not say much:
aalex@macmini-ubuntu:~$ kubectl -n cattle-system logs helm-operation-2786k
Defaulted container "helm" out of: helm, proxy
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Waiting for Kubernetes API to be available
Timeout waiting for kubernetes
Can someone please help me? What can be the reason that the Kubernetes API is not available for some of these pods? Which commands should I run next in order to troubleshoot this problem?
I found a bunch of threads like this github issue but most of them have no answers how it could be fixed and I'm not experienced enough in Kubernetes to be able to
check that your cluster's k8s API is communicating properly.
on my own.
