---
title: "Throughput"
type: "glossary term"
url: "https://aifordevs.org/glossary#throughput"
markdown: "https://aifordevs.org/glossary/throughput.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#throughput"
---

# Throughput

Throughput is how many tokens a system generates per second, either for one request or summed across all requests on a deployment. Batching raises total throughput while usually lowering speed per request.

Check which number a benchmark reports. Tokens per second per request is what a user feels, and total tokens per second is what sets your serving cost. Continuous batching and better memory management raise the total by keeping GPU slots busy.

## Example

Forty concurrent requests at 25 tokens per second each is 1,000 tokens per second in total, even though each user still sees 25.

## Related terms

- [TTFT](https://aifordevs.org/glossary#ttft)
- [Inference](https://aifordevs.org/glossary#inference)
- [Continuous batching](https://aifordevs.org/glossary#continuous-batching)
- [Batch API](https://aifordevs.org/glossary#batch-api)
- [PagedAttention](https://aifordevs.org/glossary#paged-attention)
- [Speculative decoding](https://aifordevs.org/glossary#speculative-decoding)

## Cite this term

AI for Devs. "Throughput." https://aifordevs.org/glossary#throughput

- Page: https://aifordevs.org/glossary#throughput
- Markdown: https://aifordevs.org/glossary/throughput.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs