Health Probes

A running container is not always a healthy one. An app can freeze, or take a minute to warm up, while its process stays alive. Kubernetes cannot see inside, unless you let it ask. Probes are those questions. In this lesson, you will learn the three kinds, and what each one triggers.

• Running does not mean healthy
• Probes ask the app directly
• Three probes, three decisions

Three kinds of probe

There are three probes. The liveness probe asks, are you still alive? If it keeps failing, the kubelet restarts the container. The readiness probe asks, are you ready for traffic? If it fails, the pod stays running, but it is removed from the service, and gets no requests. And the startup probe is for slow starters. It holds the other probes back, until the app has finished starting. Use readiness to protect users, and liveness to recover from freezes.

How a probe checks

A probe can check in three ways. An H T T P probe calls a path, like slash health, and expects a success code. A T C P probe just tries to open a port. And an exec probe runs a command inside the container, and checks that it exits with zero. You also set the timing, how long to wait before the first check, how often to check, and how many failures to tolerate. Good probes are cheap, quick, and honest.

A liveness probe

This pod has a liveness probe, using the exec style. Every five seconds, it runs cat on a file. If the file exists, all is well. The container creates that file when it starts, and then deletes it after fifteen seconds, on purpose, to simulate a frozen app. After two failures in a row, the kubelet decides the container is unhealthy, and restarts it.

apiVersion: v1
kind: Pod
metadata:
  name: probed
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "touch /tmp/healthy; sleep 15; rm /tmp/healthy; sleep 600"]
    livenessProbe:
      exec:
        command: ["cat", "/tmp/healthy"]
      initialDelaySeconds: 3
      periodSeconds: 5
      failureThreshold: 2

Probes in action

Let's watch both probes. First, the liveness pod. After about half a minute, the events show the probe failing, and the kubelet killing, and restarting the container. Second, a web server with a readiness probe, that checks a path which does not exist. The pod is running, but the ready column says zero of one. The events show the probe failing, with a four oh four. Kubernetes keeps this pod out of every service, until it is truly ready.

kubectl apply -f probe.yaml -f ready.yaml
pod/probed created
pod/notready created
kubectl describe pod probed | grep -E 'Unhealthy|Killing' | cut -c1-120
  Warning  Unhealthy  20s (x2 over 25s)  kubelet            Liveness probe failed: cat: can't open '/tmp/healthy': No su
  Normal   Killing    20s                kubelet            Container app failed liveness probe, will be restarted
kubectl get pod notready
NAME       READY   STATUS    RESTARTS   AGE
notready   0/1     Running   0          45s
kubectl describe pod notready | grep 'Readiness probe failed' | head -1 | cut -c1-120
  Warning  Unhealthy  1s (x16 over 44s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 4

apiVersion: v1
kind: Pod
metadata:
  name: probed
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "touch /tmp/healthy; sleep 15; rm /tmp/healthy; sleep 600"]
    livenessProbe:
      exec:
        command: ["cat", "/tmp/healthy"]
      initialDelaySeconds: 3
      periodSeconds: 5
      failureThreshold: 2

apiVersion: v1
kind: Pod
metadata:
  name: notready
spec:
  containers:
  - name: web
    image: nginx:1.27
    readinessProbe:
      httpGet:
        path: /ready
        port: 80
      periodSeconds: 3

Summary