--- title: "Inference and serving" type: "topic" url: "https://aifordevs.org/topics/inference" markdown: "https://aifordevs.org/topics/inference.md" publisher: "AI for Devs" license: "https://aifordevs.org/reuse" attribution: "Credit AI for Devs and link to https://aifordevs.org/topics/inference" --- # Inference and serving What happens between a request and the last token. Engines, batching, the KV cache, quantization, and decoding tricks that change latency and cost. ## Articles - [Mixture of experts: how MoE works and what it costs](https://aifordevs.org/articles/mixture-of-experts.md): Mixture of experts LLMs load every expert but run only a few per token. Compare total and active parameters, and see what memory, batching, and routing cost. (2026-11-08) - [KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained.md): A KV cache holds attention keys and values during LLM decode. Size one from config.json, see what concurrency costs, and how prompt caching reuses it. (2026-11-08) - [Speculative decoding: how drafts speed up LLM inference](https://aifordevs.org/articles/speculative-decoding.md): Speculative decoding has a cheap draft propose tokens the target model verifies in one pass. See the math, the costs, and what vLLM and SGLang support. (2026-11-08) - [vLLM vs SGLang, plus Ollama and llama.cpp compared](https://aifordevs.org/articles/vllm-sglang-llama-cpp.md): vLLM vs SGLang compared on prefix caching, hardware, and constrained decoding, plus Ollama and llama.cpp. Every default checked November 8, 2026. (2026-11-08) ## Glossary terms - [Constrained decoding](https://aifordevs.org/glossary/constrained-decoding.md): Constrained decoding is generation in which tokens that would break a grammar or schema are masked at each step before sampling. The output follows the grammar by construction, unless generation stops early. - [Continuous batching](https://aifordevs.org/glossary/continuous-batching.md): Continuous batching is a serving technique that schedules work one model iteration at a time, so new requests join a running batch and finished ones leave it immediately. Requests stop waiting for the slowest sequence in the batch. - [GGUF](https://aifordevs.org/glossary/gguf.md): GGUF is the single-file model format used by llama.cpp and by tools that read it, such as Ollama. A GGUF file holds the weights, often quantized to 4 to 8 bits, along with the tokenizer and metadata needed to run the model. - [Grouped-query attention](https://aifordevs.org/glossary/grouped-query-attention.md): Grouped-query attention (GQA) is an attention variant in which several query heads share one key and value head. It uses more key-value heads than multi-query attention and fewer than multi-head attention. - [Inference](https://aifordevs.org/glossary/inference.md): Inference is running a trained model to produce output. For an LLM that means processing the prompt, then generating tokens one at a time until a stop token or a length limit. - [KV cache](https://aifordevs.org/glossary/kv-cache.md): A KV cache (key-value cache) is memory that stores the attention keys and values of every token already processed, so each new token computes only its own. It trades compute for memory, cutting per-step attention cost from quadratic to linear in sequence length while the cache grows with sequence length and batch size. - [PagedAttention](https://aifordevs.org/glossary/paged-attention.md): PagedAttention is the KV cache memory manager introduced with vLLM (Kwon et al., 2023). It stores the cache in fixed-size blocks, like pages in virtual memory, so requests do not need one contiguous allocation and fragmentation stays low. - [Prefill and decode](https://aifordevs.org/glossary/prefill-and-decode.md): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in parallel and is limited by compute, and decode generates one token per step and is limited by memory bandwidth. - [Prefix caching](https://aifordevs.org/glossary/prefix-caching.md): Prefix caching is a serving-engine technique that reuses KV cache blocks from earlier requests when a new prompt starts with the same tokens. Prefill skips the matched tokens, which lowers time to first token and frees GPU compute. - [Quantization](https://aifordevs.org/glossary/quantization.md): Quantization is storing model weights, and sometimes activations or the KV cache, in fewer bits, for example 8-bit or 4-bit instead of 16-bit. It cuts memory and often speeds up decoding, at some cost in quality that you should measure on your task. - [Speculative decoding](https://aifordevs.org/glossary/speculative-decoding.md): Speculative decoding is a decoding method where a fast draft step proposes several tokens and the large model verifies them in one forward pass, keeping the longest accepted prefix. With the standard acceptance rule, the output distribution matches the large model's. - [Throughput](https://aifordevs.org/glossary/throughput.md): Throughput is how many tokens a system generates per second, either for one request or summed across all requests on a deployment. Batching raises total throughput while usually lowering speed per request. - [TTFT](https://aifordevs.org/glossary/ttft.md): TTFT (time to first token) is the delay between sending a request and receiving the first output token. It covers queueing and prefill, so long prompts raise it and prompt caching lowers it. ## Platforms - [Hugging Face](https://aifordevs.org/platforms/hugging-face.md): The main hub for open-weight models and datasets, plus Inference Providers and dedicated Inference Endpoints. - [Together AI](https://aifordevs.org/platforms/together-ai.md): Serverless and dedicated inference for open-weight models, with fine-tuning and GPU clusters. - [Fireworks AI](https://aifordevs.org/platforms/fireworks-ai.md): Fast inference for open-weight models, with fine-tuning and on-demand deployments. - [Groq](https://aifordevs.org/platforms/groq.md): Inference API running open-weight models on Groq's own LPU chips, built for low latency. - [Baseten](https://aifordevs.org/platforms/baseten.md): Dedicated and shared inference for open-weight and custom models, plus training and the Truss packaging tool. - [Modal](https://aifordevs.org/platforms/modal.md): Serverless GPUs driven from Python code, used for inference, batch jobs, and fine-tuning. - [Replicate](https://aifordevs.org/platforms/replicate.md): Run community and official models through an API, and package your own with the open-source Cog tool. - [OpenRouter](https://aifordevs.org/platforms/openrouter.md): One OpenAI-compatible API and one bill for hundreds of models across many providers. - [vLLM](https://aifordevs.org/platforms/vllm.md): High-throughput serving engine built around PagedAttention, continuous batching, and an OpenAI-compatible server. - [SGLang](https://aifordevs.org/platforms/sglang.md): Serving engine with RadixAttention prefix caching and a fast runtime for structured and multi-call workloads. - [llama.cpp](https://aifordevs.org/platforms/llama-cpp.md): C/C++ inference for GGUF models on CPUs, Apple silicon, and consumer GPUs, with heavy quantization support. - [Ollama](https://aifordevs.org/platforms/ollama.md): Pull and run open-weight models locally with one command, behind a local REST API. - [TensorRT-LLM](https://aifordevs.org/platforms/tensorrt-llm.md): NVIDIA's library for optimized LLM inference on NVIDIA GPUs, with in-flight batching and quantization.