Lab: Disruption Budgets

Updates are not the only danger. Maintenance is another. When an administrator drains a node, Kubernetes evicts its pods. If the application has few replicas, the drain could remove too many at once. A pod disruption budget is a promise, from the application, that limits how many pods may be down. Let's see it enforce that promise.

• Maintenance can remove pods
• A budget limits how many
• Kubernetes enforces it

The budget

Here is the budget. It selects the pods labelled app equals web. And it says, minimum available, three. We have exactly three replicas, so this budget allows no voluntary disruption at all. Real budgets are usually less strict, for example, at least two of three. But this strict version makes the lesson easy to see. When someone drains a node, Kubernetes will check the budget, before evicting each pod.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: web

Draining against the budget

We pin our three pods onto one worker node, with a label, and apply the budget. The budget reports, allowed disruptions, zero. Now we try to drain that node. Kubernetes starts to evict, but the first eviction is refused. The message says, cannot evict pod, as it would violate the pod's disruption budget. It will retry every few seconds, and the drain waits, instead of taking the application down. In real life, replacement pods start elsewhere, the budget allows the next eviction, and the drain proceeds, one pod at a time, with the service always available.

kubectl label node k8s-m03 lab=pdb
node/k8s-m03 labeled
kubectl get pdb
NAME      MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web-pdb   3               N/A               0                     2s
kubectl drain k8s-m03 --ignore-daemonsets --delete-emptydir-data --timeout=15s 2>&1 | grep -E 'evicting|Cannot' | cut -c1-125
evicting pod default/web-65bbf7bc54-jrj4c
evicting pod default/web-65bbf7bc54-dlxjv
evicting pod default/web-65bbf7bc54-j4wxs
error when evicting pods/"web-65bbf7bc54-dlxjv" -n "default" (will retry after 5s): Cannot evict pod as it would violate the
error when evicting pods/"web-65bbf7bc54-jrj4c" -n "default" (will retry after 5s): Cannot evict pod as it would violate the
error when evicting pods/"web-65bbf7bc54-j4wxs" -n "default" (will retry after 5s): Cannot evict pod as it would violate the
evicting pod default/web-65bbf7bc54-jrj4c
evicting pod default/web-65bbf7bc54-dlxjv
evicting pod default/web-65bbf7bc54-j4wxs
error when evicting pods/"web-65bbf7bc54-jrj4c" -n "default" (will retry after 5s): Cannot evict pod as it would violate the
error when evicting pods/"web-65bbf7bc54-j4wxs" -n "default" (will retry after 5s): Cannot evict pod as it would violate the
error when evicting pods/"web-65bbf7bc54-dlxjv" -n "default" (will retry after 5s): Cannot evict pod as it would violate the
evicting pod default/web-65bbf7bc54-jrj4c
evicting pod default/web-65bbf7bc54-dlxjv
evicting pod default/web-65bbf7bc54-j4wxs
error when evicting pods/"web-65bbf7bc54-j4wxs" -n "default" (will retry after 5s): Cannot evict pod as it would violate the

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  minReadySeconds: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      nodeSelector:
        lab: pdb
      containers:
      - name: web
        image: nginx:1.25
        readinessProbe:
          httpGet:
            path: /
            port: 80
          periodSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: web

Budgets in practice

A few practical notes. Set a budget for every production workload, with at least two replicas. Prefer, minimum available, as a number, or a percentage, that matches your capacity. Budgets protect against voluntary disruptions, like drains, and upgrades. They cannot save you from a sudden crash, or a node failure, which is why you also need replicas, spread across nodes and zones. And remember to test it, by draining a node in a staging cluster, before the real maintenance day.

Summary