ML Platforms

So far, we used separate tools for each job. Real teams often want them bundled together, with one login, and one way of working. These bundles are called M L platforms. They are not magic. They are the tools you now understand, packaged, and connected. In this lesson, we look at the main building blocks of a platform.

• Separate tools, bundled together
• Not magic, just connected
• Know the building blocks

The layers of an ML platform

Picture a platform as layers. At the bottom, compute, Kubernetes, or a cloud, with processors and graphics cards. Above it, data, storage, warehouses, and a feature store. Then the experiment layer, tracking, and the model registry. Above that, the pipeline layer, orchestration, and continuous delivery. Then serving, model servers, with autoscaling, and rollouts. And on top, monitoring, and governance. A platform team builds these layers once, so that every data scientist can use them, without becoming an infrastructure expert.

1. ComputeKubernetes, CPUs, GPUs
2. Data and featuresstorage, warehouse, feature store
3. Experiments and registrytracking, versions
4. Pipelines and deliveryorchestration, CI/CD
5. Servingmodel servers, autoscaling
6. Monitoring and governancedrift, quality, cards

Popular open source building blocks

Many layers have popular open source options. MLflow, for tracking, and the registry. Kubeflow, for pipelines, and notebooks, on Kubernetes. KServe, and Seldon, for serving, with autoscaling, and canary releases. Ray, for distributed training and serving. Feast, for features. Evidently, and others, for drift and quality reports. And Argo, for workflows, and Git ops delivery. None of them is mandatory. Choose the smallest set, that solves the problems you actually have.

Serving with KServe

Here is what serving looks like on such a platform. This one object, an inference service, replaces a deployment, a service, an autoscaler, and an ingress. It names the model format, scikit learn, and where the model is stored. It asks for at least one replica, and at most ten. And it scales on concurrency, aiming at twenty requests at once, per replica. The platform pulls the model from storage, wraps it in a server, exposes an endpoint, and scales it. You describe the model, and the platform does the plumbing.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: churn
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 10
    scaleMetric: concurrency
    scaleTarget: 20
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://models/churn/champion

Summary