CI/CD for ML

Remember maturity level two, where changes ship automatically and safely? This is how you get there. Continuous integration, and continuous delivery, applied to machine learning. The pipeline tests the code and the data, trains the model, checks its quality, and only then builds, and ships it. Let's read a real pipeline.

• Automate every step
• Gates stop bad models
• Ship only what passes

The ML delivery pipeline

An ML pipeline has more gates than a software pipeline. First, code checks, linting and unit tests. Second, data checks, the validation contract we wrote earlier. Third, training, with the pinned environment. Fourth, a quality gate. The new model must beat a baseline, and must not be worse than the current champion, on the same test data. Only then does the pipeline build the container image, push it to the registry, and update the deployment manifest, which a Git ops tool delivers. If any gate fails, the pipeline stops, and nothing reaches users.

1. Test code and dataunit tests, data contract
2. Trainpinned, reproducible
3. Quality gatebeat baseline and champion
4. Build and deployimage, registry, manifest

A pipeline in YAML

Here is such a pipeline, in the GitHub Actions format. It has two jobs. The first, validate and train, installs the libraries, runs the tests, trains the model, and then runs a quality gate script, which fails if the F one score is below the minimum. The second job, build and deploy, needs the first to succeed, and runs only on the main branch. It builds an image, tagged with the commit identifier, pushes it, and updates the manifest. Nothing is manual, and everything is traceable to a commit.

name: ml-pipeline
on:
  pull_request:
  push:
    branches: [main]
jobs:
  validate-and-train:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - run: pip install -r requirements.txt
    - name: Unit and data tests
      run: pytest -q
    - name: Train and evaluate
      run: python src/train.py
    - name: Quality gate
      run: python src/gate.py --candidate metrics.json --min-f1 0.35
  build-and-deploy:
    needs: validate-and-train
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Build and push image
      run: |
        docker build -t registry.example.com/churn-api:${{ github.sha }} .
        docker push registry.example.com/churn-api:${{ github.sha }}
    - name: Update the deployment manifest
      run: ./scripts/set-image.sh ${{ github.sha }}

Continuous training

There is one more idea, continuous training. The pipeline does not only run when code changes. It also runs when new data arrives, or on a schedule, or when monitoring detects drift. The same gates apply. A retrained model is only promoted, if it proves better than the champion. This closes the loop of the life cycle, monitoring triggers training, training passes the gates, and deployment updates the service, without a person touching anything.

Summary