Why Failed or Evicted pods are not deleted and How to delete them

In Kubernetes, Evicted or Failed Pods are not cleared until the sum of total terminated pods reaches a certain number

This numeric value is by default set at 12500 on standard Kubernetes including managed services like EKS, AKS, GCP etc and it is set at the Control Plane level as a startup FLAG/Parameter for Controller-manager

The Parameter which controls this numeric value is --terminated-pod-gc-threshold

Terminated pod gc threshold is by default set to 12500 which instructs the Kube-controller-manager to clean up the terminated pods only when the total no of failed or evicted pod reach that number

The following screenshot taken from the documentation of Kubernetes shows this value

To refer other values and defaults for the kube-controller-manager visit here 

But we cannot really wait for 12500 pods to become Evicted. so what can we do

You can run any of the following  kubectl commands to delete the Evicted and Failed Pods

 

Delete based on the status.reason of the pod ( Need JQ)

This command deletes only the Evicted pods not the other failed Pods

kubectl get po -A – all-namespaces -o json | jq  '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | "kubectl delete po \(.metadata.name) -n \(.metadata.namespace)"' | xargs -n 1 bash -c

 

Find all Failed pods and delete including Evicted ( Need JQ)

This command detects the Failed pods across all namespaces and delete

kubectl get po -A – all-namespaces -o json | jq '.items[] | select(.status.phase | IN ("Failed"))|"kubectl delete po \(.metadata.name) -n \(.metadata.namespace)"' | xargs -n 1 bash -c

 

Find all Failed pods with Field selector  ( faster & no JQ required )

Previous commands require the jq command as a prerequisite. this command uses the inbuilt field selector and finds the failed pods and deletes them

kubectl delete pods -A – field-selector=status.phase=Failed

 

Cheers
Sarav AK

---