How to restart kubernetes nodes?

Question

The status of nodes is reported as unknown

"conditions": [
          {
            "type": "Ready",
            "status": "Unknown",
            "lastHeartbeatTime": "2015-11-12T06:03:19Z",
            "lastTransitionTime": "2015-11-12T06:04:03Z",
            "reason": "Kubelet stopped posting node status."
          }

whle kubectl get nodes return a NOTReady status. What does this imply and how to fix this?

have exactly same problem here :( I was able to delete node in VirtualBox and then `kube-up.sh` will create a new one. it didn't really help in my situation, but maybe will help you — Igor Artamonov, Nov 12 '15 at 12:51
Is there an api to delete the node? I am not sure how the cluster was set up — user_mda, Nov 12 '15 at 13:06
oh, i didn't even ask what kind of setup you have, though it's local vagrant based on virtualbox — Igor Artamonov, Nov 12 '15 at 13:07
This is a physical linux vm, any info on how to either create a new node , or restart an existing one? — user_mda, Nov 12 '15 at 13:08
The only answer is how you delete a node. Everyone who comes to this question is going to be looking for how to restart one. Can we get an answer for that? — Bruno Bronosky, Oct 31 '16 at 15:11

score 58 · Answer 1 · edited May 15 '18 at 18:47

Get nodes

kubectl get nodes

Result:

NAME            STATUS     AGE
192.168.1.157   NotReady   42d
192.168.1.158   Ready      42d
192.168.1.159   Ready      42d

Describe node

Here is a NotReady on the node of 192.168.1.157. Then debugging this notready node, and you can read offical documents - Application Introspection and Debugging.

kubectl describe node 192.168.1.157

Partial Result:

Conditions:
Type          Status          LastHeartbeatTime                       LastTransitionTime                      Reason                  Message
----          ------          -----------------                       ------------------                      ------                  -------
OutOfDisk     Unknown         Sat, 28 Dec 2016 12:56:01 +0000         Sat, 28 Dec 2016 12:56:41 +0000         NodeStatusUnknown       Kubelet stopped posting node status.
Ready         Unknown         Sat, 28 Dec 2016 12:56:01 +0000         Sat, 28 Dec 2016 12:56:41 +0000         NodeStatusUnknown       Kubelet stopped posting node status.

There is a OutOfDisk on my node, then Kubelet stopped posting node status. So, I must free some disk space, using the command of df on my Ubuntu14.04 I can check the details of memory, and using the command of docker rmi image_id/image_name under the role of su I can remove the useless images.

Login in node

Login in 192.168.1.157 by using ssh, like ssh administrator@192.168.1.157, and switch to the 'su' by sudo su;

Restart kubelet

/etc/init.d/kubelet restart

Result:

stop: Unknown instance: 
kubelet start/running, process 59261

Get nodes again

On the master:

kubectl get nodes

Result:

NAME            STATUS    AGE
192.168.1.157   Ready     42d
192.168.1.158   Ready     42d
192.168.1.159   Ready     42d

Ok, that node works fine.

Here is a reference: Kubernetes

score 9 · Answer 2 · answered Dec 10 '19 at 09:27

9

I had an onpremises HA installation, a master and a worker stopped working returning a NOTReady status. Checking the kubelet logs on the nodes I found out this problem:

failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false

Disabling swap on nodes with

swapoff -a

and restarting the kubelet

systemctl restart kubelet

did the work.

answered Dec 10 '19 at 09:27

Andrea Scova

101
1
3

Your issue probably was that `swapoff -a` only disables the swap temporary until next server restart. To disable to permanently you should remove it from `/etc/fstab` – Tseng Aug 11 '22 at 07:18

score 8 · Answer 3 · answered Nov 12 '15 at 13:53

8

You can delete the node from the master by issuing:

kubectl delete node hostname.company.net

The NOTReady status probably means that the master can't access the kubelet service. Check if everything is OK on the client.

answered Nov 12 '15 at 13:53

cristi

2,019
1
22
31

6

You may have to use following command to delete a node from cluster gracefully. `kubectl drain --ignore-daemonsets --delete-local-data` then `kubectl delete node ` – Hari Ennekat Nov 26 '19 at 05:39

score 8 · Answer 4 · answered Feb 24 '19 at 20:52

If a node is so unhealthy that the master can't get status from it -- Kubernetes may not be able to restart the node. And if health checks aren't working, what hope do you have of accessing the node by SSH?

In this case, you may have to hard-reboot -- or, if your hardware is in the cloud, let your provider do it.

For example, the AWS EC2 Dashboard allows you to right-click an instance to pull up an "Instance State" menu -- from which you can reboot/terminate an unresponsive node.

Before doing this, you might choose to kubectl cordon node for good measure. And you may find kubectl delete node to be an important part of the process for getting things back to normal -- if the node doesn't automatically rejoin the cluster after a reboot.

Why would a node become unresponsive? Probably some resource has been exhausted in a way that prevents the host operating system from handling new requests in a timely manner. This could be disk, or network -- but the more insidious case is out-of-memory (OOM), which Linux handles poorly.

To help Kubernetes manage node memory safely, it's a good idea to do both of the following:

Reserve some memory for the system.
Be very careful with (avoid) opportunistic memory specifications for your pods. In other words, don't allow different values of requests and limits for memory.

The idea here is to avoid the complications associated with memory overcommit, because memory is incompressible, and both Linux and Kubernetes' OOM killers may not trigger before the node has already become unhealthy and unreachable.

Thanks for the detailed explanation. In short, if you are using aws ec2 nodes, go to the console and reboot them and your node status may change from NotReady to Ready if you already solved the causing issues. Worked for me. — Jonathan Lee, Nov 12 '20 at 02:34
Very nice explanation. That was the case for me after hammering my websockets server with Artillery. ~5K open connections. I rebooted the instance manually from the AWS console and after ~5mins it became ready again..pheww — Kostas Demiris, Mar 10 '23 at 14:58

score 6 · Answer 5 · answered Apr 15 '20 at 06:36

In my case I am running 3 nodes in VM's by using Hyper-V. By using the following steps I was able to "restart" the cluster after restarting all VM's.

(Optional) Swap off

$ swapoff -a
You have to restart all Docker containers

$ docker restart $(docker ps -a -q)
Check the nodes status after you performed step 1 and 2 on all nodes (the status is NotReady)

$ kubectl get nodes
Restart the node

$ systemctl restart kubelet
Check again the status (now should be in Ready status)

Note: I do not know if it does metter the order of nodes restarting, but I choose to start with the k8s master node and after with the minions. Also it will take a little bit to change the node state from NotReady to Ready

score 3 · Answer 6 · answered Jul 28 '21 at 08:26

3

In my case I was using EKS. Just needed to reboot it from the aws console.

answered Jul 28 '21 at 08:26

1shubhamjoshi1

546
7
16

May I know how you restarted it through aws console. ? I am facing similar issue now – sudharsan tk Feb 23 '23 at 00:58

score -3 · Answer 7 · answered Jan 27 '17 at 19:23

I had this problem too but it looks like it depends on the Kubernetes offering and how everything was installed. In Azure, if you are using acs-engine install, you can find the shell script that is actually being run to provision it at:

/opt/azure/containers/provision.sh

To get a more fine-grained understanding, just read through it and run the commands that it specifies. For me, I had to run as root:

systemctl enable kubectl
systemctl restart kubectl

I don't know if the enable is necessary and I can't say if these will work with your particular installation, but it definitely worked for me.

Please help me understand how removing/installing the service used to manage the resources within Kubernetes can cause a NODE to restart. This is playing havoc on my mind. — Steven K7FAQ, Jan 16 '20 at 18:34

How to restart kubernetes nodes?

7 Answers7

Get nodes

Describe node

Login in node

Restart kubelet

Get nodes again

Linked