Inference Spending Just Passed Training. Your Capacity Plan Hasn’t Caught Up

Global spending on AI inference has overtaken spending on model training, and the order flipped faster than most capacity plans did. Finance teams spotted the swap on the invoice before engineering spotted it on the dashboard.

The uncomfortable part isn't the number. Training was a project with a start and end date. Inference is a utility bill that arrives every month and grows with usage. That changes how capacity should be planned, financed, and audited, and most teams are still budgeting for the old shape of the workload.

Why Did the Inference Line Item Appear Overnight

Training spend concentrates. A team books a cluster for a few weeks, runs the job, and the bill ends. Inference spend accumulates instead. Every prompt, every retrieval, every background summarization job pulls tokens and GPU-seconds, and the meter never stops.

What changed in the last 18 months is that inference stopped being a rounding error on top of training and became the dominant cost of running an AI product. Pilots moved to production. Features that used to call a model once per session now call it a dozen times, chained together with retrieval and tool use.

A recent piece in TechBullion on the finance audit of AI inference costs walks through the exact question CFOs are now typing into search bars: why is the GPU bill suddenly larger than the data warehouse bill? The honest answer is that nobody budgeted for a workload that runs continuously at production scale.

What Is Finance Actually Looking For When They Audit

An inference audit isn't a hunt for waste in the abstract. It's a request for the three numbers finance needs to forecast next quarter without guessing.

• Tokens per feature. How many input and output tokens does each user-facing feature consume on average, and at the 95th percentile? Without this, you can't price a seat, a workflow, or a tier.
• GPU-hours per workload. Steady-state draw separated from spiky draw, so reserved capacity can cover the baseline and elastic capacity can absorb the peaks.
• Egress and storage. Model weights, embeddings, logs, and the data moving between regions. These line items hide inside cloud invoices and often grow faster than compute.

Once those three are visible, the audit stops being adversarial. It becomes a shared capacity plan.

Where Does Inference Run, and What Does Each Option Cost

There are three realistic homes for a production inference workload, and most companies end up with some mix of all three.

• Cloud API. Priced per token. Zero capital commitment, fastest path to production, and the right default for unpredictable or low-volume workloads. The cost per unit is the highest of the three, and the margin belongs to someone else.
• Reserved GPU in the cloud. Priced per GPU-hour under a commitment. Lower unit cost than on-demand, predictable budget, no hardware to own. You still pay for idle capacity when traffic dips, and egress keeps metering.
• On-prem or colocated hardware. Capital expense up front, then power, cooling, and amortization. Lowest unit cost when the boxes stay busy, full control of the data, and no per-token meter. Break-even depends almost entirely on how full the boxes stay.

Schneider Electric's guidance on inference is worth reading alongside this: the clusters built for training are overkill for serving, and operators are shifting toward GPU sharing, reservations, and scheduling tuned specifically for inference traffic patterns.

When Does Owning the Hardware Actually Pencil Out

On-prem hardware stops being a nostalgia play the moment a workload has three traits: a predictable baseline of traffic, a sensitivity to where the data lives, and a horizon longer than the depreciation schedule of a GPU. Hit those three and the math tilts fast.

The failure mode is buying hardware that then sits idle. Utilization is the whole game. Enterprise GPU utilization is often far lower than planners assume, not because the chips are slow, but because workloads are provisioned for peak and the peaks are rare. A capacity plan that promises sustained utilization and delivers a fraction of it is worse than paying per token.

The practical pattern is a core-and-flex split: own or reserve the baseline, burst to the cloud for spikes. That requires knowing the shape of the baseline, which is exactly what the finance audit above is designed to surface. For a deeper treatment of how to size that baseline, this capacity-planning guide walks through workload categorization and the utilization benchmarks production teams hit in practice.