Lab: Graceful Shutdown

The last piece. When Kubernetes stops a pod, during an update, or a scale down, it first asks the application to stop, politely. If the application ignores the request, Kubernetes waits, and then kills it. A well behaved application listens, finishes its current work, and exits. Let's watch both behaviours, and measure the difference.

• Stopping is a conversation
• Ask politely, then force
• A good app finishes its work

The shutdown sequence

Here is the sequence. Kubernetes marks the pod as terminating, and removes it from the service, so no new requests arrive. If a pre stop hook exists, it runs. Then the container receives the term signal, which means, please stop. The application now has a grace period, thirty seconds by default, to finish in flight requests, close connections, and exit. If it is still running, when the time is up, Kubernetes sends a kill signal, which cannot be ignored. Apps that handle the term signal, restart quickly, and lose no work.

1. Removed from the serviceno new requests
2. PreStop hookoptional, runs first
3. SIGTERMplease stop, finish your work
4. SIGKILL after the grace periodforced, cannot be ignored

Two pods, two behaviours

Here are two pods. Both have a grace period of eight seconds, shortened for this lab. The first just sleeps, and ignores the stop request. The second runs a small shell script, which traps the term signal. When the signal arrives, it prints a message, and exits cleanly. In a real application, that handler would finish the current requests, flush buffers, and close database connections.

apiVersion: v1
kind: Pod
metadata:
  name: no-handler
spec:
  terminationGracePeriodSeconds: 8
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep", "1000"]
---
apiVersion: v1
kind: Pod
metadata:
  name: with-handler
spec:
  terminationGracePeriodSeconds: 8
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "trap 'echo got SIGTERM, finishing work; exit 0' TERM; sleep 1000 & wait"]

Measuring the shutdown

Let's measure. We delete the first pod, which ignores the signal, and time it. It took about nine seconds. Kubernetes waited the whole grace period, and then killed it. Now we delete the second pod, which handles the signal. It took about one second. It received the term signal, finished, and exited at once. In a deployment, that difference decides whether a rolling update is quick, and whether in flight requests are completed, or lost.

kubectl apply -f grace.yaml
pod/no-handler created
pod/with-handler created
s=$(date +%s); kubectl delete pod no-handler > /dev/null; echo "no-handler took $(( $(date +%s) - s )) seconds"
no-handler took 9 seconds
s=$(date +%s); kubectl delete pod with-handler > /dev/null; echo "with-handler took $(( $(date +%s) - s )) seconds"
with-handler took 1 seconds

apiVersion: v1
kind: Pod
metadata:
  name: no-handler
spec:
  terminationGracePeriodSeconds: 8
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep", "1000"]
---
apiVersion: v1
kind: Pod
metadata:
  name: with-handler
spec:
  terminationGracePeriodSeconds: 8
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "trap 'echo got SIGTERM, finishing work; exit 0' TERM; sleep 1000 & wait"]

A zero downtime checklist

Let's gather the whole zero downtime story into one checklist. At least two replicas, spread across nodes. A readiness probe, so traffic only reaches ready pods. A rolling update, with max unavailable zero. A pod disruption budget, for maintenance. A handler for the term signal, and a sensible grace period. And a pre stop delay, if your load balancer needs a few seconds to stop sending traffic. With these six things, deployments and maintenance become uneventful, which is exactly the goal.

Summary