Inference economics after the training boom

Training and inference are different businesses

Training is a large, scheduled project. It concentrates compute into long runs and tolerates specialized infrastructure. Inference is a continuous service. Demand arrives unevenly, latency targets vary by application and capacity must be available before the request appears.

This changes the operating model. An inference provider must balance response time, batch size, model quality and hardware utilization. Extra capacity protects service levels but reduces returns when it sits idle. Tight capacity improves utilization but creates queues during demand spikes.

The metric is cost per useful outcome

Cost per token is helpful but incomplete. A smaller model may generate more tokens because it needs additional steps or retries. A more capable model may cost more per request but finish the task with fewer interactions. The relevant denominator is a useful outcome at the required quality and latency.

For operators, this means routing requests across models, hardware types and regions. High-value, latency-sensitive tasks may justify premium accelerators. Routine workloads can move to optimized models or lower-cost silicon. The orchestration layer becomes an economic control plane.

Utilization is the hidden margin variable

Accelerators are fixed-cost assets with rapid performance obsolescence. Revenue depends on keeping them productive without degrading customer experience. Scheduling software, model quantization, caching and speculative decoding can therefore create as much economic impact as a modest hardware upgrade.

Contract structure matters as well. Reserved capacity produces predictable revenue but may cap upside. On-demand pricing captures peaks but transfers utilization risk to the operator. A resilient platform uses a portfolio of commitments rather than a single pricing model.

What to underwrite

A credible inference business should explain workload mix, gross margin by model class, hardware refresh policy, customer concentration and the path to higher utilization. Aggregate GPU hours conceal whether the platform is serving valuable applications or discounting capacity to fill the cluster.

• Revenue per accelerator hour after energy and networking costs.
• Utilization by workload class and time of day.
• Latency compliance during peak demand.
• Model-routing savings that are retained rather than passed through.