A Growing Team Setup

When a team grows to several data scientists, and several models, new problems appear. People duplicate work. Experiments are hard to compare. Everyone has their own way of deploying. At this point, a shared setup starts to pay off. Let's see what to share, and how to start small.

• Several people, several models
• Duplicated work
• A shared setup pays off

A shared tracking server

The first shared piece is usually a tracking server. Here is a compose file, that starts one. M L flow, as the tracking server, and registry. A Postgres database, that stores runs, and metadata. And Minio, an object store, compatible with the cloud storage protocol, that keeps the model files. Everyone logs to the same server, so experiments are comparable, and every model has a home. This setup runs on a single machine, and can later move to managed cloud services, without changing how people work.

services:
  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    command: mlflow server --host 0.0.0.0 --backend-store-uri postgresql://mlflow:secret@db/mlflow --artifacts-destination s3://mlflow
    ports: ["5000:5000"]
    environment:
      MLFLOW_S3_ENDPOINT_URL: http://minio:9000
    depends_on: [db, minio]
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: mlflow
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: mlflow
    volumes: ["db-data:/var/lib/postgresql/data"]
  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    ports: ["9000:9000", "9001:9001"]
    volumes: ["minio-data:/data"]
volumes:
  db-data:
  minio-data:

What a growing team shares

Here is what teams typically share, from the bottom up. A common compute, perhaps a Kubernetes cluster, with graphics cards, available to everyone. A shared data platform, with a warehouse, and versioned datasets. A tracking server, and a model registry. Standard pipeline templates, so that every project starts with tests, validation, and gates already in place. A standard way to serve models, so there is one deployment path, instead of five. And common monitoring dashboards, so every model is watched the same way.

1. Common computeKubernetes, shared GPUs
2. Data platformwarehouse, versioned datasets
3. Tracking and registryone server for everyone
4. Pipeline templatestests and gates built in
5. Standard serving pathone deployment route
6. Common monitoringevery model watched alike

The platform team

At this size, a small platform team often appears. Its job is to make the right thing the easy thing. It builds templates, runs the shared servers, and keeps the tooling current. It treats data scientists as customers, and measures its own success by how quickly a new model reaches production, safely. The best platform teams avoid forcing a single tool. They offer a paved road, that is easy to follow, and leave room for exceptions, when a project has real special needs.

Summary