FinOps for ML

Machine learning can be wonderfully useful, and wonderfully expensive. Graphics cards cost real money by the hour, and an idle one costs exactly as much as a busy one. Finance operations, or FinOps, is the practice of making costs visible, and keeping them in check, without slowing teams down. Let's see the numbers.

• Idle hardware costs full price
• Make costs visible
• Optimise without slowing down

Why utilisation is everything

Here is a quick calculation, with example prices. A graphics card costs two dollars fifty an hour, and can serve four hundred predictions a second, at full load. We compute the cost of a thousand predictions, at different levels of busyness. When the card is busy five percent of the time, a thousand predictions cost about three and a half cents. When it is eighty percent busy, the cost drops to a fraction of a cent. Same hardware, fifteen times cheaper, only by keeping it busy. Then three monthly figures. Running all month costs eighteen hundred dollars. Spot pricing cuts it to six hundred and thirty. And scaling to zero outside working hours, costs six hundred.

GPU_PER_HOUR = 2.50          # example price in dollars
PEAK_PER_SECOND = 400        # predictions per second at full load

def cost_per_1000(utilisation):
    served_per_hour = PEAK_PER_SECOND * 3600 * utilisation
    return GPU_PER_HOUR / served_per_hour * 1000

for utilisation in (0.05, 0.20, 0.50, 0.80):
    print(f"GPU {utilisation:4.0%} busy -> ${cost_per_1000(utilisation):.4f} per 1000 predictions")

monthly = GPU_PER_HOUR * 24 * 30
print(f"one GPU running all month: ${monthly:,.0f}")
print(f"same GPU on spot pricing (-65%): ${monthly * 0.35:,.0f}")
print(f"GPU only 8 busy hours a day, scaled to zero otherwise: ${GPU_PER_HOUR * 8 * 30:,.0f}")

GPU   5% busy -> $0.0347 per 1000 predictions
GPU  20% busy -> $0.0087 per 1000 predictions
GPU  50% busy -> $0.0035 per 1000 predictions
GPU  80% busy -> $0.0022 per 1000 predictions
one GPU running all month: $1,800
same GPU on spot pricing (-65%): $630
GPU only 8 busy hours a day, scaled to zero otherwise: $600

Where ML money goes

Where does the money go? Training, experiments, and tuning runs, often on expensive hardware, and often left running by accident. Serving, which runs all the time, and is usually the largest recurring cost. Data storage, and movement, because copies pile up, and transfers between regions are billed. Notebooks, and development machines, forgotten overnight. And, for language models, tokens. Each of these has a standard remedy, and the first step for all is the same, label every resource with a team and a project, so that you can see who spends what.

FinOps habits

A few habits keep bills sane. Tag everything, with team, project, and environment. Set budgets and alerts, so that a runaway job is caught in hours, not at month end. Review cost per prediction, and per experiment, regularly. Shut down idle resources automatically. Prefer spot instances for interruptible work. And celebrate savings the same way you celebrate features. Cost is a quality metric, like latency, not an afterthought.

Summary