Team Practices

Tools do not make an M L Ops culture. People and habits do. In the last lesson of this part, we look at who does what, how teams work together, and the small routines that keep a machine learning system healthy for years. These habits cost almost nothing, and they matter more than any platform.

• Tools need habits
• Clear roles
• Small routines, big effect

Roles in an ML team

Several roles work together. Data scientists explore, model, and evaluate. Machine learning engineers turn models into reliable services, and pipelines. Data engineers build the pipelines, that feed everything. Platform engineers run the tools, and infrastructure. Product managers define the problem, and the success measure. And, in many companies, domain experts, labelers, and risk or legal reviewers. Small teams share these hats, but every hat must be worn by someone, or its work silently does not happen.

Habits of healthy teams

Healthy teams share habits. Everything is in Git, code, pipelines, configs, and model cards. Every change is reviewed. Pull requests include tests, and evaluation results. Experiments are tracked, so no result is lost. Models are promoted through a registry, with gates. Incidents lead to blameless reviews, and new tests. And knowledge is written down, in short documents, next to the code, so that people can join, and leave, without the system stopping.

A weekly rhythm

A simple weekly rhythm keeps a system healthy. Review the monitoring dashboards, quality, drift, cost, and latency. Look at the alerts that fired, and tune the noisy ones. Check the retraining runs, did they pass their gates. Skim a sample of real predictions, or conversations, with a critical eye. Update the golden test set with new cases. And look at the error budget. Thirty minutes a week, spent this way, prevents most of the nasty surprises.

1. Dashboardsquality, drift, cost, latency
2. Alertstune the noisy ones
3. Samplesread real predictions
4. Golden setadd new cases

Summary