How to bring down worker nodes to unhealthy state?

 To simulate an unhealthy state for Kubernetes worker nodes, you can manually induce conditions that make the node unfit for running workloads. This is often done to test the cluster's resilience, monitoring, or failover capabilities. Below are several methods to achieve this:

 

Methods to Simulate an Unhealthy Node

1. Simulate Resource Exhaustion

• CPU or Memory Stress: Use tools like stress to overload the node's resources: 

• stress --cpu 8 --timeout 300

• stress --vm 2 --vm-bytes 1G --timeout 300

• This will cause resource pressure, and the node may be marked as NotReady or fail to run pods efficiently.

2. Network Disruption

• Use iptables to block traffic between the worker node and the Kubernetes control plane: 

• iptables -A INPUT -p tcp --dport 6443 -j DROP

• This will prevent the node from communicating with the API server, causing it to appear as NotReady.

3. Stop Critical Kubernetes Services

• SSH into the worker node and stop services like kubelet or containerd: 

• systemctl stop kubelet

• systemctl stop containerd

• Without kubelet, the node will be marked as NotReady by the control plane.

4. Disk Space Exhaustion

• Fill up the disk on the worker node to simulate a lack of disk space: 

• dd if=/dev/zero of=/tmp/filldisk bs=1M count=102400

• Kubernetes monitors node disk usage, and exceeding thresholds will mark the node as unschedulable.

5. Cordon Without Draining

• Mark the node as unschedulable: 

• kubectl cordon <node-name>

• This doesn't technically make the node "unhealthy" but prevents new workloads from being scheduled on it.

6. Induce Pod Eviction Pressure

• Configure taints or simulate eviction by manually applying taints: 

• kubectl taint nodes <node-name> key=value:NoSchedule

7. Simulate Node Shutdown

• Power off the node from the underlying infrastructure (e.g., cloud console or virtual machine manager).

• Kubernetes will detect the node as NotReady after it fails health checks.

8. Misconfigure Node Networking

• Remove or misconfigure network interfaces on the node.

• Example: Change or disable the default route in the node's networking configuration.

9. Cluster Autoscaler Interaction

• If using a cluster autoscaler, you can mark the node for termination using labels or taints to see how the autoscaler reacts.

 

Precautions

1. Test in Non-Production Environments: Always perform these actions in a staging or test environment, not on a production cluster.

2. Monitor Logs: Keep a watch on Kubernetes logs and node-level logs to understand the impact of the induced failure.

3. Have a Recovery Plan: Ensure you know how to revert the changes (e.g., restarting services, freeing up resources) to restore the node's health.

 

By inducing such states, you can test the robustness of your Kubernetes setup, validate alerting mechanisms, and ensure proper failover configurations.