How Services Route

Here is a puzzle. A service has a cluster I P, but no machine, and no pod, owns that address. Yet connections to it work. How? The answer is a clever trick, performed by kube proxy on every node. Let's reveal it.

• The cluster IP belongs to no one
• kube-proxy runs on every node
• It rewrites destinations

The virtual address trick

Here is the trick. A pod sends a request to the service address. On its way out, the packet passes through rules, installed on its node by kube proxy. The rules say, anything addressed to this service, should be rewritten, to the address of one of the real pods, chosen at random. So the packet is redirected, before it ever leaves the node, and then travels on to the real pod. The service address is a virtual address. It exists only in rules.

kube-proxy modes

Kube proxy can create those rules in several ways. The default mode uses iptables, the Linux firewall rules. It is simple, and works well for thousands of services. For very large clusters, I P V S mode uses a faster lookup table. And some network plugins, such as Cilium, replace kube proxy completely, with E B P F programs inside the kernel, which are faster still. Whatever the mode, you use services in exactly the same way.

Peeking at the rules

Let's look at the real rules. First, the number of service rule chains in the firewall of one node. Then, we print a few of them. Each line mentions a chain named after a service. Behind those names are the rules that choose between the pods, with probabilities, so each pod gets its fair share. You rarely need to read them, but it is satisfying to know that no magic is involved.

minikube -p k8s ssh -- sudo iptables-save -t nat | grep -c KUBE-SVC
48
minikube -p k8s ssh -- sudo iptables-save -t nat | grep KUBE-SVC | head -3 | cut -c1-100
:KUBE-SVC-CG5I4G2RS3ZVWGLK - [0:0]
:KUBE-SVC-EDNDUDH2C75GIR6O - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]

Summary