Events and Alerts

There are two more things to know. First, Kubernetes keeps a diary of what it does, the events. Second, watching numbers all day is impossible, so we need alerts that tap us on the shoulder. But bad alerts are worse than none. In this lesson, you will use events, and learn how to design alerts people trust.

• Events: Kubernetes' diary
• Dashboards for looking
• Alerts for acting

The events diary

Events are objects, created by Kubernetes components, every time something happens. A pod was scheduled. An image was pulled. A container was killed. Let's create some trouble, and then ask for the recent events, sorted by time. We see a story, scheduled, pulling image, failed, back off. Events only live for about an hour, so check them early, or send them to your log store.

kubectl run broken --image=nginx:no-such-tag
pod/broken created
kubectl get events --sort-by=.lastTimestamp | tail -5 | cut -c1-120
10s         Normal    Pulling                        pod/broken                          Pulling image "nginx:no-such-ta
8s          Warning   Failed                         pod/broken                          Failed to pull image "nginx:no-
8s          Warning   Failed                         pod/broken                          Error: ErrImagePull
8s          Normal    BackOff                        pod/broken                          Back-off pulling image "nginx:n
8s          Warning   Failed                         pod/broken                          Error: ImagePullBackOff

The golden signals

What should you watch? Google's engineers suggest four golden signals. Latency, how long requests take. Traffic, how many requests arrive. Errors, how many fail. And saturation, how full your resources are. If you track these four for each service, you will catch most problems. Dashboards, built in a tool like Grafana, put them on a single screen, so anyone can see the health of the system at a glance.

Good alerts and bad alerts

Now, alerts. A good alert is urgent, someone must act now. It is actionable, there is something to do. And it is about symptoms users feel, such as errors, and slow responses. A bad alert fires for every small blip, or for causes that nobody can fix at night, like a busy processor for one minute. Too many bad alerts cause alert fatigue, and people start ignoring all of them. Page people only for problems that hurt users, and send everything else to a dashboard, or a ticket.

Summary