Pipelines on Kubernetes

When pipelines grow, each step often needs its own environment. Data preparation needs one set of libraries. Training needs another. And evaluation, another still. Running each step as a container on Kubernetes solves this neatly. Let's see how a pipeline looks, when every step is a container.

• Each step is a container
• Own libraries per step
• Runs on Kubernetes

A Kubeflow pipeline

Here is a pipeline in the Kubeflow style. Each step is a Python function, marked as a component, with its own base image, and its own libraries. The train step receives a data path, and produces a model, as an output. The evaluate step receives that model, and returns a score. Then the pipeline function connects them. Training runs first, and its model output feeds the evaluation. Finally, the compiler turns the pipeline into a file, which the platform runs, every step in its own container, on the cluster, with logs, retries, and a visual graph.

from kfp import dsl, compiler

@dsl.component(base_image="python:3.12-slim", packages_to_install=["pandas", "scikit-learn"])
def train(data_path: str, model_out: dsl.Output[dsl.Model]):
    ...

@dsl.component(base_image="python:3.12-slim")
def evaluate(model: dsl.Input[dsl.Model]) -> float:
    ...

@dsl.pipeline(name="churn-training")
def pipeline(data_path: str = "s3://data/churn.csv"):
    trained = train(data_path=data_path)
    evaluate(model=trained.outputs["model_out"])

compiler.Compiler().compile(pipeline, "churn.yaml")

Build or buy?

Should you build a platform from open source, or buy a managed one? Building gives control, flexibility, and no licence fees. But it takes skilled people, time, and ongoing maintenance, and many teams underestimate that. Buying, from the cloud providers, or from specialist vendors, gets you started fast, with support, and less to operate. The costs are fees, and some lock in. A sensible path is to start with managed services, where they fit, keep your own code, and formats portable, and build custom parts only where they give you a real advantage.

Managed platforms in the clouds

Each major cloud has a full platform. Amazon has SageMaker. Google has Vertex A I. Microsoft has Azure Machine Learning. They cover notebooks, training jobs, registries, pipelines, endpoints, and monitoring. Their ideas map directly to what you have learned here, so skills transfer. Judge them on integration with your data, and security needs, the price of training and serving, and how easily you could leave. Keep your pipelines in code, and your models in open formats, and you keep your options open.

Summary