Kubernetes Node Management, Maintenance, Delete And Evict PODs With Node Drain

I hope Kubernetes Node Management is a crucial part of a Kubernetes cluster. In terms of node maintenance, we all need to accept that the compute resources such as physical servers or cloud instances need to have a downtime for maintenance works such as hardware changes, Operating system upgrade or any other maintenance work. Same theory applies to the Kubernetes Cluster Nodes and lets see how we can perform the Kubernetes Node Management, Maintenance and editing PODs without interrupting to it’s services.

I have four node Kubernetes cluster on Oracle Ravello Cloud, one Master node and three worker nodes with running Nginx POD with three replicas.

Lets see how we can perform the Cluster Maintenance work without interrupting the running Services

Evict PODs and Safely Perform A Node Maintenance

To view the Kubernetes node status execute the below commands

kubectl get nodes

To check the running POD status and the respective nodes execute below commands

kubectl get pods -o wide

Executing below the command you can safely take a node out from the cluster and running PODs will be evicted from the node

kubectl drain [NODE_HOSTNAME] --ignore-daemonsets

In the below screen capture two Nginx replicas are running in “kube-02.localdomain” node and after executing the command PODs are running in “kube-04.localdomain” node.

Subjected node which is “Kubelet-02.localdomain” is disabled for POD scheduling after performing this command

After verifying the POD status you can bring down the node and perform the maintenance operation now.

Once, the operation completed we can “uncordon” the node and schooling will be enabled again

Complete Operation is explained in the below screen capture

Deleting A Node From The Cluster

If we are planning to delete a node from the Kubernetes cluster, we have to drain, evict the running PODs and disable the POD scheduling first.

To delete the POD below command is used

kubectl delete node [NODE_NAME]

Generate A New Token And Add A New Worker Node

Lets assume that after deleting a node we are planning to add a new node to our Kubernetes Cluster. We might don’t have the “kubeadm join” command and the token with us. We can follow the below procedure to generate a new token and add the worker node to the cluster

To view the token list execute the below command

kubeadm token list

Generate a token with below command, if you don’t have a listed token

kubeadm token generate 

Take the generated token and create a command to join the worker node with below command

kubeadm token create [GENERATED_TOKEN] --ttl 23h --print-join-command

Issue this command in the worker node and it will be joining to the cluster. If you need to see how to create a Kubernetes cluster from the scratch read my previous article.