---
title: "Continuous batching"
type: "glossary term"
url: "https://aifordevs.org/glossary#continuous-batching"
markdown: "https://aifordevs.org/glossary/continuous-batching.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#continuous-batching"
---

# Continuous batching

Continuous batching is a serving technique that schedules work one model iteration at a time, so new requests join a running batch and finished ones leave it immediately. Requests stop waiting for the slowest sequence in the batch.

The Orca paper (Yu et al., OSDI 2022) calls this iteration-level scheduling and pairs it with selective batching, which applies batching only to a chosen subset of operations. Static batching holds every slot until the longest sequence finishes, so short requests leave GPU capacity idle.

## Example

Orca reports a 36.9x throughput gain over NVIDIA FasterTransformer at the same latency on GPT-3 175B. That figure comes from the authors' own evaluation.

## Source

[Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)](https://www.usenix.org/conference/osdi22/presentation/yu)

## Related terms

- [Throughput](https://aifordevs.org/glossary#throughput)
- [Inference](https://aifordevs.org/glossary#inference)
- [PagedAttention](https://aifordevs.org/glossary#paged-attention)

## Read the article

[vLLM vs SGLang, plus Ollama and llama.cpp compared](https://aifordevs.org/articles/vllm-sglang-llama-cpp)

## Cite this term

AI for Devs. "Continuous batching." https://aifordevs.org/glossary#continuous-batching

- Page: https://aifordevs.org/glossary#continuous-batching
- Markdown: https://aifordevs.org/glossary/continuous-batching.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs