# AI Inference Cost Optimization: Calculating and Reducing Generative AI Serving Costs

> How to calculate generative AI inference cost per token and per task, and the main levers to reduce it: batching, caching, quantization and scaling.

Source: https://hyperlake.cloud/blog/ai-inference-cost-optimization-calculating-reducing-serving-costs-2026
By Hyperlake Team · Hyperlake

To calculate inference cost, divide what you pay for serving capacity by the useful tokens it serves, then turn that into a cost per completed task. For AI inference cost optimization, work in this order: raise GPU utilization (batching, scheduling), stop recomputing shared prompts (caching), shrink the model's footprint (quantization), and only then tune the model itself. Measure cost per task, not per token. Retries, long prompts and idle hardware all hide inside the per-token figure.

We make HyperLake, a sovereign AI operating platform. It comes up once near the end, along with the cases where it doesn't apply.

![Descending bars for batching, caching, quantization and right-sizing, labelled illustrative.](https://hyperlake.cloud/blog/img/production/57621ffbe4377b1c425fdb547c450c0b15f6ead1-1200x840.png?w=1600&fit=max&auto=format)

## How do you calculate generative AI inference cost?

It depends on how you serve. There are two cost bases.

- Hosted API. You pay per token, usually at different prices for input and output. Cost per request is input tokens times the input price, plus output tokens times the output price.
- Your own GPUs. You pay for capacity whether it's busy or not. Cost per million tokens is the hourly GPU cost divided by the millions of tokens they actually serve in that hour.

The second formula is where teams get caught out, because the denominator depends on utilization, not the datasheet. Here's a hypothetical example. Every number is made up to show the arithmetic, so swap in your own.

| Input | Hypothetical value |
| --- | --- |
| GPU cost | $4.00 per hour |
| Measured throughput (input plus output tokens) | 2,000 tokens per second |
| Tokens per hour at 100% busy | 2,000 x 3,600 = 7.2 million |
| Cost per million tokens at 100% busy | $4.00 / 7.2 = about $0.56 |
| Cost per million tokens at 40% busy | $4.00 / (7.2 x 0.4 = 2.88) = about $1.39 |

Same hardware, two and a half times the cost per token, just because it's 40% busy instead of fully busy (1.39 / 0.56 is about 2.5). That's why the first optimization is almost always about keeping GPUs working. Making the model smaller comes later.

```python
# Illustration only: replace every number with your own measurements.
def cost_per_million_tokens(gpu_cost_per_hour, tokens_per_second, utilization):
    tokens_per_hour = tokens_per_second * 3600 * utilization
    return gpu_cost_per_hour / (tokens_per_hour / 1_000_000)

print(round(cost_per_million_tokens(4.00, 2000, 1.0), 2))  # 0.56
print(round(cost_per_million_tokens(4.00, 2000, 0.4), 2))  # 1.39
```

Measure throughput on your own model, prompts and concurrency. The vLLM benchmark tool, for example, reports total token throughput (input plus output) and output token throughput separately, along with time to first token and inter-token latency ([vLLM benchmark documentation](https://docs.vllm.ai/en/latest/benchmarking/cli/)). Someone else's numbers from someone else's setup can send you the wrong way. For the buy-versus-API question in more depth, the [LLM API vs self-hosted GPU cost calculator](https://hyperlake.cloud/tools/llm-api-vs-self-hosted-cost-calculator) takes your own prices and throughput. It has no built-in prices.

## What should you measure: cost per token or cost per task?

Cost per task. A task is whatever your product counts as one unit of work: an answered question, a summarized document, a finished agent run. Per-token cost feeds into it. It isn't the answer.

Carry the hypothetical on. Say a task uses 1,500 input and 500 output tokens, so 2,000 tokens. At the 40%-busy rate of about $1.39 per million tokens, that's 2,000 x 1.39 / 1,000,000 = about $0.0028 per attempt. If 20% of attempts get retried once, you average 1.2 attempts, or about $0.0033 per completed task. Agent workflows multiply this again, since every step re-sends context. Quite often, retries, tool calls and context growth move the bill more than the per-token rate does.

For each workload, track at least:

- input and output tokens per task;
- attempts per completed task;
- time to first token and inter-token latency, because your latency targets limit how hard you can batch;
- GPU utilization, or for APIs, the share of spend on cached versus uncached input;
- cost by team or feature, which needs attribution (see [LLM Cost Attribution](https://hyperlake.cloud/blog/llm-cost-attribution-who-owns-which-part-of-the-ai-bill) and [AI Cost Observability](https://hyperlake.cloud/blog/ai-cost-observability-seeing-your-spend-before-the-bill-arrives)).

## Why is inference often the bigger long-run cost than training? (Inference vs training economics)

This section takes on inference versus training costs. The short answer is structural, not a statistic. Training is a bounded effort you run now and then. Inference is a recurring cost that grows with every user, request and agent step. Plenty of teams never train a model at all and only ever pay for inference. Adopt open models or a hosted API and training cost belongs to someone else. Your bill is serving.

They also load hardware differently, which changes what you optimize. Serving has to hold the model weights plus a per-request attention cache in memory while answering inside a latency target. For the hardware side, read [AI Inference vs Training: Understanding Infrastructure Requirements in 2026](https://hyperlake.cloud/blog/ai-inference-vs-training-infrastructure-requirements-2026). For what drives the bill, see [AI Inference Costs: Why Production Gets Expensive](https://hyperlake.cloud/blog/why-ai-inference-is-so-expensive) and [Economics of Large Language Models](https://hyperlake.cloud/blog/economics-of-large-language-models).

## How do you reduce inference costs? The main levers

These are roughly in the order they tend to pay off. Test each against your own quality and latency targets, because every one of them trades something away.

| Lever | What it cuts | What it costs you |
| --- | --- | --- |
| Batching and scheduling | Idle GPU time | Latency under load |
| Prefix or prompt caching | Repeated prompt processing | Needs shared, identical prompt prefixes |
| Quantization | Memory footprint | Some accuracy risk; hardware support varies |
| Speculative decoding | Time per output token | Extra complexity; gains vary with traffic |
| Batch (offline) APIs | Price per token | Results arrive later |
| Right-sizing and scale-to-zero | Paying for idle capacity | Cold-start delay |
| Shorter prompts and outputs | Tokens | Prompt engineering effort |

### Does batching reduce serving costs?

Yes. GPUs do more useful work when several requests share each pass. What usually caps how many requests fit at once is the memory holding each request's attention cache. The vLLM paper says this directly: the cache for each request is large and changes size during generation, and memory wasted by fragmentation and duplication limits batch size. Its authors report that vLLM, built on PagedAttention, improves throughput of popular LLMs by 2 to 4 times at the same latency compared with the systems they tested against, with larger gains for longer sequences and larger models ([vLLM and PagedAttention paper, arXiv 2309.06180](https://arxiv.org/abs/2309.06180)). That's a research result on specific baselines and workloads. It's not a promise about your setup. For how continuous batching works, see [Continuous Batching for High-Throughput AI APIs](https://hyperlake.cloud/blog/continuous-batching-how-ai-apis-serve-thousands-of-users-at-once) and [Triton Inference Server: Dynamic Batching Architecture](https://hyperlake.cloud/blog/triton-inference-server-deployment-and-dynamic-batching-architecture).

### How does caching lower cost?

When lots of requests begin with the same text (a long system prompt, tool definitions, a shared document), you can skip reprocessing it.

- Self-hosted. vLLM's automatic prefix caching stores the cache from earlier queries and reuses it when a new query starts with the same prefix. Its documentation says this helps with repeated questions on one long document and with multi-turn chats, and that it only speeds up the prefill phase, so it does little when long generated answers dominate ([vLLM prefix caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html)).
- Hosted. Anthropic documents prompt caching with exact, prefix-based matching, and states that cache reads are priced as a fraction of the base input price, with the fraction depending on the model ([Anthropic prompt caching docs](https://platform.claude.com/docs/en/docs/build-with-claude/prompt-caching)). Check your provider's current terms.

Either way, put stable content first and per-request content (timestamps, user IDs) last. Otherwise the prefix never matches. The cluster-level version of the idea is in [KV Cache Routing](https://hyperlake.cloud/blog/kv-cache-routing-the-cluster-level-optimisation).

### Does quantization reduce cost?

It cuts the memory the model needs, which can let you serve it on fewer or smaller GPUs, or fit a bigger batch. vLLM's documentation describes quantization as trading model precision for a smaller memory footprint, and notes that hardware support differs by method ([vLLM quantization](https://docs.vllm.ai/en/latest/features/quantization/)). Published results vary. The LLM.int8() paper reports halving inference memory while retaining full-precision performance in its experiments ([LLM.int8() paper](https://arxiv.org/abs/2208.07339)). The GPTQ paper reports negligible accuracy loss at 3 to 4 bits per weight on the models it studied, with end-to-end inference speedups of about 3.25 times on an A100 and 4.5 times on an A6000 versus FP16 ([GPTQ paper](https://arxiv.org/abs/2210.17323)). We read those as reasons to test, and then to evaluate on your own tasks before switching. See [Quantization Explained](https://hyperlake.cloud/blog/quantization-explained) and [AWQ vs. GPTQ vs. GGUF](https://hyperlake.cloud/blog/awq-gptq-and-gguf-quantization-methods-comparison).

### When does speculative decoding help?

With speculative decoding, a smaller model proposes several tokens and the large model checks them in parallel. The original paper reports 2 to 3 times speedups on T5-XXL with identical outputs ([speculative decoding paper](https://arxiv.org/abs/2211.17192)). vLLM's documentation describes it as a way to reduce inter-token latency under medium-to-low QPS, memory-bound workloads, and says real gains depend on your model family, traffic pattern, hardware and sampling settings, so measure in your own environment ([vLLM speculative decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/)). Its documented target is latency in lightly loaded, memory-bound serving. If your GPUs are already saturated with batched traffic, it may matter less for cost.

### Should you use batch APIs for work that can wait?

For jobs that aren't urgent, like evaluations, classifying large datasets or embedding a content repository, asynchronous batch endpoints can be cheaper. OpenAI's Batch API documentation, for instance, states a 24-hour completion window and a lower price than synchronous calls ([OpenAI Batch API guide](https://developers.openai.com/api/docs/guides/batch)). Other providers set their own terms, so confirm the current ones. On your own GPUs the same idea holds: run offline work when interactive traffic is quiet.

### What about scaling down idle capacity?

Bursty traffic plus always-on GPUs inflates the per-token cost in the table above. Autoscaling, scale-to-zero for rarely used endpoints and right-sizing instances all go after the utilization term. The catch is cold-start delay when an endpoint wakes up, which interactive users may not tolerate.

## How do you decide which optimization to try first?

1. Measure today's cost per completed task and your latency targets.
1. Check utilization. If it's low, fix scheduling, batching and scaling before you touch the model.
1. Look at prompt structure, and add caching if prefixes repeat.
1. If memory is the binding limit, test quantization against your own evaluation set.
1. If you rent GPUs, compare against buying: see [GPU Cluster Rental vs Buying On-Premises Compute](https://hyperlake.cloud/blog/gpu-cluster-rental-vs-buying-on-premises-compute-2026), [How to Build an On-Prem GPU Cluster for AI](https://hyperlake.cloud/blog/how-to-build-an-on-prem-gpu-cluster-for-ai-2026) and [How Many GPUs Do I Need to Serve an LLM?](https://hyperlake.cloud/answers/how-many-gpus-to-serve-an-llm).
1. Re-measure after every change. Saving 30% on one line item while breaking your quality bar isn't a saving.

To be honest about the limit: if your volume is low or spiky, a hosted API with caching and batch endpoints may well be the cheapest option you have, and no self-hosting trick changes that.

## Where does HyperLake fit?

HyperLake doesn't make a model cheaper on its own. Its product description says it lets teams assemble, deploy and govern models, data and applications in infrastructure they control, with direct visibility into infrastructure usage and cost, compute matched to demand, and eligible inference endpoints that can scale down when idle. It says there's no compute markup, and that savings come from reuse, reduced integration effort and workload-aware capacity, not from a fixed percentage. If all you do is call a hosted API, none of this applies to you.

## Frequently asked questions

### How much does it cost to run LLM inference?

There's no single number. It depends on model size, tokens per task, whether you pay per token or for GPUs, and how busy those GPUs are. Run the formulas above on your own measurements.

### Is inference more expensive than training?

Over a product's life it can be, since inference recurs with every request and training is a bounded effort. With a hosted API or an open model, you may pay no training cost at all. Work it out for your own case instead of trusting a general ratio.

### What should I check first to cut inference costs?

Utilization. If GPUs sit idle, better batching and scaling usually come before any model change. If you're on an API, see whether prompt caching and batch endpoints fit your traffic.

### Does a smaller model always lower cost?

Not always. A smaller or quantized model uses less memory, but if it fails more tasks and needs retries or human review, cost per completed task can go up. Measure it on your own evaluation set.

### How do I calculate break-even between an API and self-hosting?

Work out your cost per million tokens on owned GPUs at realistic utilization, then compare it with the API's blended price for your input and output mix. Include fixed costs such as staff time. The [calculator](https://hyperlake.cloud/tools/llm-api-vs-self-hosted-cost-calculator) does this with your inputs.

### What is the difference between inference cost and serving cost?

Inference cost usually means the compute to run the model. Serving cost is wider and adds networking, storage, observability, autoscaling overhead and the engineering to keep it all running. Track both.

## Related reading

- [AI Inference Costs: Why Production Gets Expensive](https://hyperlake.cloud/blog/why-ai-inference-is-so-expensive)
- [Economics of Large Language Models: What Drives Cost](https://hyperlake.cloud/blog/economics-of-large-language-models)
- [Quantization Explained for LLM Inference](https://hyperlake.cloud/blog/quantization-explained)
- [KV Cache Routing: Cluster-Level Inference Optimization](https://hyperlake.cloud/blog/kv-cache-routing-the-cluster-level-optimisation)
