Supply Chain Security

A typical machine learning project depends on hundreds of libraries, written by strangers, and each of those depends on more. A flaw in any one of them, is a flaw in your system. Supply chain security is the practice of knowing what you depend on, checking it for known problems, and keeping it up to date. It is easier than it sounds.

• Hundreds of dependencies
• A flaw anywhere is your flaw
• Know, check, update

A secure dependency routine

A good routine has four steps. First, pin your dependencies, to exact versions, so builds are repeatable. Second, scan them regularly, against databases of known vulnerabilities. Third, update, in small steps, with tests, rather than rarely, in giant jumps. Fourth, generate a software bill of materials, a list of every component in each image, so that when a new vulnerability is announced, you can answer, within minutes, whether you are affected.

1. Pinexact versions
2. Scanknown vulnerabilities
3. Update often, in small stepswith tests
4. Keep a bill of materialsknow where each library runs

Scanning for known problems

Let's scan a deliberately old requirements file. It pins an old version of the requests library, and an old version of Jinja. The audit tool checks each pinned version against a public vulnerability database. It reports dozens of known problems, with their identifiers, and the version that fixes each one. In a real pipeline, this command runs on every change, and fails the build, when a serious vulnerability appears, so nobody ships a known hole by accident.

cat requirements.txt
requests==2.19.0
jinja2==2.10
pip-audit -r requirements.txt --no-deps 2>/dev/null | head -9 | cut -c1-80
Name     Version ID              Fix Versions
-------- ------- --------------- -------------
requests 2.19.0  PYSEC-2018-28   2.20.0
requests 2.19.0  PYSEC-2023-74   2.31.0
requests 2.19.0  PYSEC-2026-1873 2.32.0
requests 2.19.0  PYSEC-2026-1872 2.32.4
requests 2.19.0  PYSEC-2026-2275 2.33.0

requests==2.19.0
jinja2==2.10

Securing the pipeline itself

The build pipeline is part of the supply chain, too. Give it the minimum permissions. Keep its secrets in a secret store, never in the code. Pin the versions of the actions, and base images, that it uses. Sign the images it produces, and verify the signatures, before deploying. And require reviews, for changes to the pipeline files, as strictly as for the application code. An attacker who controls your pipeline, controls everything it builds.

Summary