Models on Kubernetes

A single container is fine for a demo. In production, you want several copies, that heal themselves, scale with traffic, and update without downtime. That is exactly what Kubernetes does for web services, and a model service is just a web service. Let's deploy our model API the Kubernetes way.

• Several copies, always on
• Heal and scale automatically
• Update without downtime

The model deployment

Here is the deployment of our model service. Three replicas, so that one failure never takes the service down. The container uses our image, tagged with a version. A readiness probe calls the health endpoint, so traffic only reaches pods that have loaded the model. And resource requests and limits tell Kubernetes how much memory and processor the service needs. Models can be hungry, so these numbers matter. If you update the image tag, Kubernetes performs a rolling update, with no downtime.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: churn-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: churn-api
  template:
    metadata:
      labels:
        app: churn-api
    spec:
      containers:
      - name: api
        image: registry.example.com/churn-api:1
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
          limits:
            memory: 1Gi

Autoscaling the service

Traffic changes, so the number of pods should too. This autoscaler watches the average processor use of the pods. It keeps at least three pods, and at most twelve. When the average goes above sixty percent, it adds pods. When traffic falls, it removes them. For models, you may also scale on request latency, or on the length of a queue, instead of processor use, if those reflect the real load better.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: churn-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: churn-api
  minReplicas: 3
  maxReplicas: 12
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

ML-specific Kubernetes concerns

Models bring a few special concerns. Large models take a long time to load, so use readiness probes, and generous start up time. Models need memory, so set limits carefully. Some need graphics cards, which you request as a resource, and place with node selectors and taints, as you may remember. Keep the model file in a registry or storage, and load it at start, instead of baking huge files into images. And consider tools made for this, like KServe, which adds autoscaling to zero, canary releases, and standard model protocols.

Summary