Training is where a model learns from data. Inference is where the trained model produces outputs for new inputs. AI inference vs training infrastructure differs because training is a large, scheduled, compute-bound job that has to keep every accelerator in a cluster synchronised, while inference is a continuous, latency-sensitive service that's often limited by memory and by how many requests you can serve at once. Size training for total compute and interconnect. Size inference for memory, latency targets and utilisation.
We make HyperLake, a sovereign AI operating platform. It doesn't train foundation models, and the section on where it fits says what it does cover.
What is the difference between AI training and inference?
NVIDIA's own explainer describes training as the learning phase, where a network adjusts its weights from an error signal, and inference as the phase where a trained model generates new outputs. Inference depends on training. A model can't be applied usefully until it has learned from data. The same page puts fine-tuning in between: reuse a broad foundation model and train only a few layers or adapters on domain data.
It's a vendor blog, so read its performance framing with that in mind. The definitions are standard. The infrastructure consequences are what the rest of this article is about.
| Training | Inference | |
|---|---|---|
| Work pattern | Long batch jobs, restarted from checkpoints | Always-on service, bursty traffic |
| Main goal | Finish the run; maximise throughput | Meet latency targets at a sensible cost per request |
| Dominant constraint | Total compute, memory for model states, cluster networking | Memory capacity and bandwidth, request concurrency |
| Scaling unit | A synchronised group of accelerators | Replicas behind a router |
| Cost shape | Large, bounded, then done | Smaller per request, but continuous and growing with usage |
| Failure impact | Lost progress since the last checkpoint | Failed or slow user requests |
How do compute requirements differ?
Kaplan et al.'s scaling-law paper gives a widely used approximation. Training compute is about 6N operations per token (the paper counts add-multiply operations), where N is the number of non-embedding parameters and the factor of six covers the forward and backward passes. The same paper puts the forward pass alone at roughly 2N operations per token plus a context-dependent term, which it drops from the training estimate when the model dimension is large relative to context length (see the paper).
A hypothetical example gives a feel for scale. Take a 7-billion-parameter model and, to keep it simple, use 7 billion for N.
- Training: 6 x 7 x 10^9 = 4.2 x 10^10 operations per token. Across a hypothetical 1 trillion tokens, that's 4.2 x 10^10 x 10^12 = 4.2 x 10^22 operations.
- Inference: 2 x 7 x 10^9 = 1.4 x 10^10 operations per token, leaving out the context term.
So one generated token costs about a third of one training token (4.2 / 1.4 = 3). Volume is what changes the picture. Training touches each token a bounded number of times. Inference keeps going for as long as the product has users, and that's how serving cost can end up bigger than the training bill. For the cost side, see AI Inference Costs: Why Production Gets Expensive and Economics of Large Language Models.
How do memory requirements differ?
Training has to hold far more than the weights. The ZeRO paper counts, for mixed-precision training with Adam, 2 bytes per parameter for fp16 weights, 2 for fp16 gradients, and 12 for optimizer state (an fp32 copy of the weights plus momentum and variance). That's 16 bytes per parameter before activations. The paper treats activations separately. For its 1.5-billion-parameter example it reports roughly 60 GB of activations at sequence length 1K and batch size 32, down to about 8 GB with activation checkpointing.
Inference needs the weights plus a working cache. Hypothetically, for the same 7-billion-parameter model:
- Training model states: 16 bytes x 7 x 10^9 = 112 GB, before activations.
- Inference weights in 16-bit: 2 bytes x 7 x 10^9 = 14 GB. Hugging Face's optimisation guide reports about 13.74 GB for Mistral-7B in bfloat16, which lines up with that estimate, and about 6.87 GB for an 8-bit load.
Those are lower bounds for one copy. On the inference side, the extra memory goes to the key-value (KV) cache. Hugging Face's cache documentation explains that the cache stores earlier attention computations so they aren't repeated for every new token, and that it can take a significant share of memory and become a bottleneck for long-context generation. Every concurrent request carries its own cache. More simultaneous users and longer contexts both eat GPU memory. Quantization shrinks the weights; see Quantization Explained for LLM Inference.
Why do networking and parallelism matter more for training?
Data-parallel training copies the model across workers and has to combine their gradients every step. PyTorch's DistributedDataParallel notes describe grouping gradients into buckets and running an all-reduce on each bucket to average them across processes, overlapping that communication with the rest of the backward pass where it can. Our reading: synchronisation lets the slowest link in the cluster set the pace. That's why training clusters are usually designed around fast interconnects.
Bigger models add model-level splitting, such as tensor and pipeline parallelism, which puts more communication on the critical path (see Tensor, Pipeline and Sequence Parallelism).
Inference generally needs less coordination. If a model fits on one GPU or one node, you can replicate it behind a load balancer and let each replica serve requests on its own. You only split a model across GPUs for inference when it doesn't fit or latency demands it. For how one replica shares its capacity across requests, see Continuous Batching.
What is different about latency and throughput?
Training cares about throughput over hours or weeks, and a single step can take as long as it takes. With inference, a person or a program is waiting. Latency matters, and there are two phases. NVIDIA's explainer notes that the prefill step processes all input tokens together and is computationally heavy, while decoding generates output one token at a time, each depending on the ones before. Hugging Face's guide adds that the model weights are loaded again for the forward pass on every new token, which is slow when a model has billions of parameters.
That leads to two practical effects:
- Long prompts stress prefill, and long answers stress decode. The vLLM documentation says automatic prefix caching only reduces prefill time, so it doesn't help much when responses are long and decoding dominates, or when requests share no prefix.
- Speculative decoding, described in the Hugging Face guide, uses a smaller assistant model to propose tokens that the larger model verifies in one pass. You do some extra work and wait less, and the outputs don't change.
Where does fine-tuning sit?
Fine-tuning is training, but it acts like a small job. Adapter-based methods train only a small number of added parameters, so the training state is much smaller than in full training. You still need room for the base weights and for optimizer state on whatever you're training. A lot of teams start here: adapt an open model to domain data, evaluate it, serve it. If you're still deciding whether to fine-tune at all, see RAG vs. Fine-Tuning.
How should you size infrastructure for training and inference?
Use this as a starting checklist, then measure with your own model and traffic.
- State the workload. Pre-training, fine-tuning, batch scoring and an interactive service each need a different plan.
- Estimate memory first: weights, plus training states or a KV cache sized for your expected concurrency and context length.
- Set the target. Training has a completion date; inference has a response-time objective.
- Decide the utilisation pattern. A training run can keep rented capacity busy for a fixed stretch. Steady inference may justify owned or reserved capacity, and spiky inference rewards scaling down when idle.
- Check data and policy constraints. Where weights, prompts and outputs are allowed to live often picks the environment before cost gets a say.
For a worked GPU count, see How Many GPUs Do I Need to Serve an LLM? and the LLM API vs Self-Hosted GPU Cost Calculator.
Where does HyperLake fit?
HyperLake is a platform for assembling, deploying and governing the data, models and applications behind AI in a customer's own environment or a client's. It isn't a tool for pre-training foundation models. According to its product description, its AI services cover serving open models through KServe, plus fine-tuning and evaluation. The foundation runs on Kubernetes in the customer's own cloud account, on AWS and Google Cloud today, with private cloud and on-premises environments coming soon. It also says eligible inference endpoints can scale down when idle, which ties back to the utilisation point above. Whether that saves you money depends on your traffic; the company says savings come from reuse and workload-aware capacity, not a fixed percentage.
If what you need is a very large training run on a dedicated cluster, HyperLake isn't the main decision. It matters when you're deciding where fine-tuned and open models get served and who governs access to them.
Frequently asked questions
Is inference cheaper than training?
Per token, yes. By the Kaplan approximation, a forward pass is about a third of the compute of a training step on the same token. Inference runs continuously, though, so total serving cost over a product's life can exceed what training cost.
Do I need the same GPUs for training and inference?
You don't have to. Training favours large memory capacity and fast interconnect across many accelerators, while inference can run on smaller or fewer accelerators if the model fits, especially after quantization. Check memory first.
Why does inference need so much memory if it does no learning?
The weights have to stay loaded, and every active request keeps a KV cache that grows with context length. More concurrent users or longer prompts mean more cache.
What is the difference between prefill and decode?
Prefill processes the whole prompt at once and is compute-heavy. Decode produces one token at a time, reading the model weights and cache over and over. Different optimisations help each phase.
Can I fine-tune and serve on the same cluster?
Often, as long as scheduling stops a training job from starving live traffic. Many teams keep them apart or run fine-tuning in off-peak windows, so test it under your own load.
Is the 6N and 2N rule exact?
It isn't. It's an approximation from the Kaplan et al. paper that drops a context-dependent term and counts only non-embedding parameters. Fine for planning, not for billing.
