Continuous batching is an LLM inference scheduling method that updates the active request batch after each generation step. Completed requests leave immediately, and waiting requests take their places without waiting for every sequence to finish. This reduces idle GPU capacity, padding work, and queueing delays while increasing throughput under concurrent demand.
This matters because AI APIs must serve requests with different prompt lengths, response lengths, and arrival times using finite GPU capacity. Efficient scheduling helps providers process more requests without making every user wait for the longest response in a group. The video above walks through the core ideas.
What is continuous batching in LLM inference?
Continuous batching, also called in-flight batching, schedules requests at individual generation iterations rather than as an unchangeable group. It preserves the parallel processing benefits of batching while allowing the membership of the active batch to change during generation.
Autoregressive language models generate responses token by token. After each decode step, the server identifies completed requests, releases their slots, and admits waiting work when capacity permits. Longer requests stay active without preventing shorter ones from finishing or new ones from starting.
The Orca research system introduced this iteration-level scheduling approach, which is now common in major LLM serving frameworks. It fits generative inference because users submit prompts at different times and request outputs of unpredictable lengths.
Static batching works differently. It creates a fixed group and admits no new requests until every sequence in that group has completed. If one user requests 500 tokens while the others need 50, the shorter requests finish early, but their capacity remains unavailable until the longest response ends.
That behavior creates three kinds of inefficiency:
- Tail waste: Finished requests cannot release their positions while a long request continues.
- Padding waste: Compute may be spent filling shorter sequence positions with padding tokens.
- Queueing delay: Incoming requests wait for the fixed batch to complete even when some positions are no longer useful.
How does iteration-level scheduling work?
Iteration-level scheduling reevaluates the active batch after each token-generation step. It continuously matches waiting requests with newly available capacity instead of waiting for an entire batch boundary.
A simplified scheduling loop has four stages:
- The scheduler selects requests that fit available GPU memory and execution capacity.
- The model performs one generation step for each active sequence.
- Requests that reached a stopping condition leave the batch.
- Waiting requests enter available slots, and the cycle repeats.
The server still batches compatible work to exploit GPU parallelism. However, no request must remain tied to the same batch membership for its entire lifetime. This rolling structure keeps useful work flowing as requests arrive and finish.
A production scheduler must also consider memory limits, sequence lengths, priorities, and latency objectives. Related optimizations, including chunked prefill and FlashDecoding, address bottlenecks within prompt processing and token generation.
How does continuous batching improve throughput and latency?
Continuous batching improves throughput by keeping more GPU capacity occupied with useful request work. It can also lower user-facing latency because a new request does not have to wait for every sequence in an unrelated fixed group to finish.
Its main practical effects are:
- Higher GPU utilization: Completed sequences are replaced instead of leaving idle positions behind.
- Greater throughput: The server can progress more requests without being constrained by the longest sequence in each fixed batch.
- Lower queueing latency: Waiting requests can enter at iteration boundaries rather than full-batch boundaries.
- Better serving economics: More effective utilization can reduce infrastructure cost per completed request.
Actual results depend on the model, hardware, prompt distribution, output lengths, memory pressure, and traffic pattern. Continuous batching also cannot remove physical capacity limits. Requests will still queue if arrival rates exceed what the serving pool can process, so teams need workload and AI cost observability alongside efficient scheduling.
What causes prefill-decode interference?
Prefill-decode interference occurs when a newly admitted request begins its compute-intensive prompt-processing phase while existing requests are decoding tokens. The prefill work can delay ongoing decode iterations and create uneven token latency for users who are already receiving responses.
The two phases have different execution characteristics. Prefill processes many prompt tokens in parallel and tends to emphasize compute throughput. Decode repeatedly generates individual tokens and is sensitive to responsive memory access and scheduling.
Prefill-decode disaggregation addresses this limitation by placing the phases on dedicated hardware pools. Prefill workers process incoming prompts, then transfer the required model state to decode workers that continue generation. This isolation adds architectural and operational complexity, so its value depends on workload scale, latency requirements, hardware, and serving-engine capabilities.
Key takeaways
- Continuous batching updates the active request group after each generation step.
- It reduces tail waste, padding work, and delays caused by fixed batches.
- Better GPU utilization can improve throughput, latency, and serving economics.
- Prefill-decode interference can still disrupt ongoing token generation.
- Disaggregating prefill and decode may help when workload requirements justify the added complexity.
How Hyperlake helps
Hyperlake lets teams assemble, deploy, govern, and operate private AI services in infrastructure they or their clients control. It can serve open models through KServe, subject to the deployment and validated integration, while providing shared controls for identity, policies, observability, lifecycle management, and infrastructure usage. To discuss a model-serving architecture for your workload, talk to our team.
Frequently asked questions
Does continuous batching change an LLM’s output quality?
Continuous batching is a serving and scheduling technique, not a model-training method. It changes when requests receive GPU execution time but does not inherently alter model weights or generation settings. Output differences can still result from sampling parameters, nondeterministic execution, or framework implementation details rather than batching itself.
Is continuous batching the same as ordinary request batching?
No. Ordinary static batching creates a fixed request group and waits for every member to finish before admitting another group. Continuous batching updates membership between generation iterations, allowing completed requests to leave and waiting requests to enter while longer responses continue.
Can continuous batching eliminate AI API queues?
Continuous batching can shorten queues by using GPU capacity more efficiently, but it cannot eliminate them when demand exceeds serving capacity. Queueing also depends on model size, prompt and response lengths, available memory, scheduling policy, hardware, and latency objectives. Additional capacity or workload controls may still be necessary.
