Lab: Prove Zero Downtime

We said that rolling updates cause no downtime. Let's prove it. In this lab, a load generator sends ten requests every second to a service, and counts every failure. Then we update the application, twice. Once with a careful rolling update, and once with the strategy that stops everything first. The numbers will speak for themselves.

• Claims need proof
• A load generator counts failures
• Rolling versus recreate

The careful rollout

Here is the careful configuration. Three replicas. A rolling update strategy, with max unavailable set to zero, which means, never remove a working pod, before a replacement is ready. Max surge of one, so there can be one extra pod during the change. Min ready seconds, so a new pod proves itself for three seconds. And a readiness probe, that checks the web page, every second. These few lines are the recipe, for updates that nobody notices.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  minReadySeconds: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:1.25
        readinessProbe:
          httpGet:
            path: /
            port: 80
          periodSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80

Rolling update under load

First, the careful rollout. The load generator is already running, in the background. We update the image, and wait for the rollout to finish. Then we count what the load generator saw, during the last twenty seconds. Every request was answered. The count of failures does not even appear, because there was none. While the pods were being replaced, a ready pod was always available, so every request found one.

kubectl set image deployment/web web=nginx:1.27
deployment.apps/web image updated
kubectl rollout status deployment/web
Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
deployment "web" successfully rolled out
kubectl logs loadgen --since=25s | sort | uniq -c
 170 OK

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  minReadySeconds: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:1.25
        readinessProbe:
          httpGet:
            path: /
            port: 80
          periodSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80

The Recreate strategy

Now the contrast. We switch to the recreate strategy, which stops all old pods first, and then starts the new ones. We run the same load generator, and update the image again. After the rollout, we count again. This time, some requests failed. For a few seconds, there were no pods at all, and the service had nothing to send traffic to. Same application, same update, a different strategy, and a measurable outage.

kubectl set image deployment/web web=nginx:1.27
deployment.apps/web image updated
kubectl rollout status deployment/web
Waiting for deployment "web" rollout to finish: 0 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "web" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "web" rollout to finish: 2 of 3 updated replicas are available...
deployment "web" successfully rolled out
kubectl logs loadgen --since=25s | sort | uniq -c
   3 FAIL
  71 OK
   1 wget: can't connect to remote host (10.109.195.175): Connection refused
   2 wget: download timed out

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:1.25
        readinessProbe:
          httpGet:
            path: /
            port: 80
          periodSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80

Summary