Why your on-prem GPUs lose to the API below 45% utilisation

Why on-prem utilisation decides profitability

As compute becomes scarce, the compute economy comes back into focus: how much of the throughput a deployment paid for is actually serving tokens decides whether the deployment makes money. On-prem makes this severe. The fixed cost is already sunk, every idle accelerator-hour is cost with no revenue against it, and the only way to serve more demand is another capex cycle. Sovereign and air-gapped sites feel it hardest because they cannot burst to a cloud, and in practice they have almost no visibility into the total cost of ownership or the return on the capital that bought the hardware.

Mintok gives on-prem operators that visibility at the token level. It takes a model, a target throughput and a chip, sizes the fleet from physics rather than from vendor box counts, prices every token the fleet can serve, and shows what each lever (precision, batch, utilisation, chip, financing) does to cost per million tokens and to return on invested capital.

The sizing kernel

Chip count is the largest of three constraints, and whichever wins is the binding constraint. Compute counts every token (prefill and decode); bandwidth counts decode only, because each decode step re-reads the weights once per batch; memory is the residency floor.

N = max( N_compute , N_bandwidth , N_memory , 1 )

N_compute   = ceil( 2 × T × P_active / (F_peak × s_prec × MFU × m_prec) )
N_bandwidth = ceil( T_out × P_active × b / (B × BW × MBU) )
N_memory    = ceil( (P_total × b + B × L × kv_tok) / C_mem )

T is target tokens per second (T_out for decode), P is parameters (active for MoE compute and bandwidth, total for memory residency), b is bytes per parameter, B is batch size, L is average context length, kv_tok is KV-cache bytes per token, F is peak TFLOPS, BW is memory bandwidth, C is memory per chip. MFU and MBU are the fractions of peak compute and peak bandwidth achieved in practice.

Per-chip throughput is the inverse of the same terms, and it is what turns a fleet into tokens:

tok/s per chip = min( B × BW × MBU / (P_active × b) ,
                      F_peak × s_prec × MFU × m_prec / (2 × P_active) )

Precision is one lever with three effects, applied from a single precision map:

Mixture-of-experts models use active parameters for compute and bandwidth and total parameters for memory. The KV cache grows with batch, context and layer count, so a long-context workload can flip a chip from bandwidth-bound to memory-bound without any change in demand. A latency mode adds time-to-first-token and tokens-per-second-per-user targets on top, which caps batch size and raises chip count.

TCO and ROIC

Annual cost of an owned fleet is depreciation plus power plus an operating overhead plus fixed labour and software; tokens served are capacity times utilisation; cost per million tokens is the ratio. Return on invested capital compares the market value of the tokens served with that annual cost, over the capital deployed.

Capex = N × c_chip + racks × c_rack
Dep   = Capex × (1 − r) / Y
Power = N × TDP × PUE × 8760 × p_kWh

Cost_yr   = (Dep + Power) × (1 + o) + Labour + Software
Tokens_yr = N × tok/s_chip × 31,536,000 × u

$/Mtok = Cost_yr / (Tokens_yr / 1,000,000)
ROIC   = (Tokens_yr × p_market − Cost_yr) / Capex

The market price is the blended API price for the same model from the catalog, so ROIC answers a plain question: does owning this fleet return more than buying the same tokens would cost? Lease and cloud paths use the same token denominator with the lease rate or the cloud hourly rate in the numerator. The finance parameters below are the live values the examples run on:

Worked example: Gemma 4 31B on one rack of 32 H200

The same 32 deployed H200s serve four times the tokens at FP4 as at BF16, and only the FP4 fleet above roughly 45 percent utilisation returns more than the tokens would cost from an API. Inputs: Gemma 4 31B (31B dense, 262k context, tier 3, API price $0.12 in and $0.35 out per million tokens), H200 SXM ($38K capex, 700 W, 141 GB, 4.8 TB/s), batch 32, 8k average context, 3 input tokens per output token, KV cache assumed at 0.25 MB per token at BF16.

The fleet's annual cost is the same at every precision and utilisation, because the boxes are deployed: capex $1.241M (32 x $38K plus one $25K rack), depreciation $279K after 10 percent residual, power $18K at 1.3 PUE, overhead 15 percent, labour and software $35K, total $377K a year. What changes is how many tokens that $377K buys.

Return on invested capital at the blended API price of $0.1775 per million tokens:

Two readings. First, a 31B model's API price is a commodity, so a rack of H200s at BF16 never beats it; the only route to a positive return is FP4 and high utilisation, or a cheaper chip. Second, the move from 10 to 20 percent utilisation halves cost per token but moves ROIC only a few points at BF16, because the value of the tokens is small against $377K of fixed cost. Utilisation is the lever, and precision is the multiplier on it.

From boxes to physics

Mintok takes telemetry from a deployment (requests, input and output tokens, latency per model) and converts it into a usage profile: average and peak tokens per second by model, context distribution, input ratio. That profile, not the number of boxes, is what the kernel sizes. For the Gemma 4 31B workload, if telemetry shows a peak of 5,000 output tokens per second, the requirement is 5 H200s at BF16 or 2 at FP4; at a 20,000 peak it is 17 or 5. A 32-chip rack bought on vendor guidance is 6 to 16 times the physics requirement, and that gap is the utilisation problem measured in dollars before the hardware is ordered.

The same kernel runs across every priced chip in the catalog. One rack of 32 at FP8 and 20 percent utilisation, Gemma 4 31B:

At this utilisation the cheapest chip per token is the one with the lowest capex per unit of bandwidth, not the fastest one. B200 doubles throughput over H200 and still costs more per token, because its capex per chip doubles too and the fleet is idle 80 percent of the time.

Acquisition mode is the last lever, and at low utilisation it is the biggest. H200 at FP8 and 20 percent utilisation, 500 billion tokens a year:

Owning only wins once utilisation is high enough that the fixed cost spreads thin; below that, the elastic path is cheaper even at list cloud rates. Mintok's capacity plan makes this a load-duration-curve decision: own the base layer whose capacity factor exceeds the crossover, rent the peaks. For an air-gapped site with no rent tier, the result collapses to one number, the utilisation the fleet must reach to pay back, and that number belongs in the business case before the purchase order.

The levers

Every input below is a field the user sets or the catalog supplies; the kernel recomputes chips, cost per million tokens and ROIC on each change.

Catalog coverage

The production catalog holds 54 chip profiles from 15 vendors, 154 model profiles from 16 vendors and 34 rack architectures, maintained by watcher agents that stage or auto-apply launches, price changes and deprecations. Browse the public pages at /silicon and /models.

Gemma 4 31B has no layer count in the catalog, so the KV cache per token in the worked example is an assumption (0.25 MB at BF16) rather than a derived value. MI355X carries no capex in the catalog and was priced at the MI350X row.