---
title: "Inference"
type: "glossary term"
url: "https://aifordevs.org/glossary#inference"
markdown: "https://aifordevs.org/glossary/inference.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#inference"
---

# Inference

Inference is running a trained model to produce output. For an LLM that means processing the prompt, then generating tokens one at a time until a stop token or a length limit.

LLM inference has two phases with different hardware limits, prefill and decode. Serving engines such as vLLM, SGLang, and llama.cpp schedule these phases and manage the KV cache, so the engine you pick changes latency and cost on the same hardware.

## Example

Generating a 500-token answer at 25 ms per token takes about 12.5 seconds of decode time.

## Related terms

- [Prefill and decode](https://aifordevs.org/glossary#prefill-and-decode)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [Throughput](https://aifordevs.org/glossary#throughput)
- [TTFT](https://aifordevs.org/glossary#ttft)
- [Continuous batching](https://aifordevs.org/glossary#continuous-batching)
- [GGUF](https://aifordevs.org/glossary#gguf)
- [Reasoning model](https://aifordevs.org/glossary#reasoning-model)
- [Speculative decoding](https://aifordevs.org/glossary#speculative-decoding)

## Read the article

[vLLM vs SGLang, plus Ollama and llama.cpp compared](https://aifordevs.org/articles/vllm-sglang-llama-cpp)

## Cite this term

AI for Devs. "Inference." https://aifordevs.org/glossary#inference

- Page: https://aifordevs.org/glossary#inference
- Markdown: https://aifordevs.org/glossary/inference.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs