Lab: The Broken Deployment

Welcome to the bonus labs. Reading about debugging is useful, but fixing real, broken things is how the skill sticks. In each lab, I give you a broken application, and we follow the method from earlier. Get, describe, logs, and then fix. Pause the video before each fix, and try it yourself first.

• Real broken applications
• Follow the method
• Pause and try it yourself

The broken file

Here is the first broken application. A deployment called shop, with two replicas, of the engine X web server. Can you spot the problems before we run it? There are two. Look at the image tag, and look at the readiness probe. Do not worry if you cannot see them yet. The cluster will tell us, using exactly the method from the debugging lessons.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: shop
  template:
    metadata:
      labels:
        app: shop
    spec:
      containers:
      - name: web
        image: nginx:1.27.99
        readinessProbe:
          httpGet:
            path: /healthz
            port: 80
          periodSeconds: 3

Fault one: the image

We apply the file, and look at the pods. The status says, error image pull, then image pull back off. That means the image could not be downloaded. Describe tells us exactly why. Failed to pull image, not found. The tag one point twenty seven point ninety nine does not exist. So we fix it, with set image, to the real tag, one point twenty seven. The deployment rolls out new pods, with the right image.

kubectl apply -f shop.yaml
deployment.apps/shop created
kubectl get pods
NAME                    READY   STATUS         RESTARTS   AGE
shop-5b787bfbb4-fw64b   0/1     ErrImagePull   0          12s
shop-5b787bfbb4-jhvrv   0/1     ErrImagePull   0          12s
kubectl describe pod -l app=shop | grep -E 'Failed to pull' | head -1 | cut -c1-115
  Warning  Failed     10s   kubelet            Failed to pull image "nginx:1.27.99": Error response from daemon: ma
kubectl set image deployment/shop web=nginx:1.27
deployment.apps/shop image updated
kubectl get pods
NAME                    READY   STATUS             RESTARTS   AGE
shop-5b787bfbb4-fw64b   0/1     ImagePullBackOff   0          26s
shop-5b787bfbb4-jhvrv   0/1     ImagePullBackOff   0          26s
shop-7d76bc459b-lndbr   0/1     Running            0          14s

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: shop
  template:
    metadata:
      labels:
        app: shop
    spec:
      containers:
      - name: web
        image: nginx:1.27.99
        readinessProbe:
          httpGet:
            path: /healthz
            port: 80
          periodSeconds: 3

Fault two: the probe

The pods are running now, but look at the ready column. Zero of one. The containers are alive, yet Kubernetes does not consider them ready, so no traffic would reach them. Describe shows the reason in the events. Readiness probe failed, with an H T T P status of four oh four. The probe asks for the path slash health z, and engine X has no such page. We fix the probe, with a patch, to ask for the root path instead. After a few seconds, both pods are ready, and the application is healthy. Two faults, found, and fixed, with the commands you already know.

kubectl get pods
NAME                    READY   STATUS             RESTARTS   AGE
shop-5b787bfbb4-fw64b   0/1     ImagePullBackOff   0          26s
shop-5b787bfbb4-jhvrv   0/1     ImagePullBackOff   0          26s
shop-7d76bc459b-lndbr   0/1     Running            0          14s
kubectl describe pod -l app=shop | grep 'Readiness probe failed' | head -1 | cut -c1-110
  Warning  Unhealthy  2s (x6 over 13s)  kubelet            Readiness probe failed: HTTP probe failed with stat
kubectl patch deployment shop --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/"}]'
deployment.apps/shop patched
kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
shop-66bb57b86-79tgl   1/1     Running   0          16s
shop-66bb57b86-7k7d5   1/1     Running   0          15s

Summary