Lab: The Stuck Pod

The second lab. A pod that stays pending forever. It has not failed, it has not crashed, it simply never starts. This is a scheduling problem, and the events will tell us which rule is blocking it. We will fix it in two different ways, and discuss which one is better.

• Pending forever
• Scheduling is the suspect
• Two possible fixes

The stuck pod

Here is the pod. A database called fast db, running engine X, as a stand in. Look at the node selector. It demands a node labelled, disk equals nvme. Does our cluster have such a node? Let's ask the cluster, and see how it reacts.

apiVersion: v1
kind: Pod
metadata:
  name: fastdb
spec:
  nodeSelector:
    disk: nvme
  containers:
  - name: db
    image: nginx:1.27

Diagnose and fix

We apply the file, and list the pod. Pending. Describe shows the events at the bottom. Failed scheduling, zero of three nodes are available, and they did not match the pod's node selector. The pod demands a label, that no node has. There are two fixes. Either the pod is wrong, and we remove the demand. Or the cluster is missing something, and we label a suitable node. Here, we label the node, k eight s m zero two, as having fast disks. The scheduler retries, immediately, and the pod starts, on exactly that node. The right fix depends on the truth. Does the app really need fast disks, or not?

kubectl apply -f fastdb.yaml
pod/fastdb created
kubectl get pod fastdb
NAME     READY   STATUS    RESTARTS   AGE
fastdb   0/1     Pending   0          6s
kubectl describe pod fastdb | grep FailedScheduling | cut -c1-120
  Warning  FailedScheduling  6s    default-scheduler  0/3 nodes are available: 3 node(s) didn't match Pod's node affinit
kubectl label node k8s-m02 disk=nvme
node/k8s-m02 labeled
kubectl get pod fastdb -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName
NAME     STATUS    NODE
fastdb   Running   k8s-m02

apiVersion: v1
kind: Pod
metadata:
  name: fastdb
spec:
  nodeSelector:
    disk: nvme
  containers:
  - name: db
    image: nginx:1.27

A checklist for pending pods

Keep this checklist for pending pods. Describe the pod, and read the scheduling event. If it mentions insufficient resources, lower the requests, or add capacity. If it mentions taints, add a toleration, or untaint the node. If it mentions a node selector, or affinity, fix the labels, or the rule. If it mentions volume claims, check that the claim is bound. And if a quota is exceeded, ask for more quota, or clean up. Nine times out of ten, the event message names the exact reason.

Summary