Deep dive
Mixture of experts: how MoE works and what it costs
Mixture of experts LLMs load every expert but run only a few per token. Compare total and active parameters, and see what memory, batching, and routing cost.
11 min readModel releases
Topic
Inference and serving
What happens between a request and the last token. Engines, batching, the KV cache, quantization, and decoding tricks that change latency and cost.
vLLM, SGLang, llama.cpp, Ollama04 articles13 glossary terms13 platforms
Articles
Read on this topic
Deep dives and guides filed under this topic, newest first.
All articlesDeep dive
KV cache explained: what it stores and what it costs
A KV cache holds attention keys and values during LLM decode. Size one from config.json, see what concurrency costs, and how prompt caching reuses it.
11 min readInference and serving
Deep dive
Speculative decoding: how drafts speed up LLM inference
Speculative decoding has a cheap draft propose tokens the target model verifies in one pass. See the math, the costs, and what vLLM and SGLang support.
9 min readInference and serving
Deep dive
vLLM vs SGLang, plus Ollama and llama.cpp compared
vLLM vs SGLang compared on prefix caching, hardware, and constrained decoding, plus Ollama and llama.cpp. Every default checked November 8, 2026.
12 min readInference and serving
Glossary
Terms you will meet
- Constrained decoding
- Constrained decoding is generation in which tokens that would break a grammar or schema are masked at each step before sampling. The output follows the grammar by construction, unless generation stops early.
- Continuous batching
- Continuous batching is a serving technique that schedules work one model iteration at a time, so new requests join a running batch and finished ones leave it immediately. Requests stop waiting for the slowest sequence in the batch.
- GGUF
- GGUF is the single-file model format used by llama.cpp and by tools that read it, such as Ollama. A GGUF file holds the weights, often quantized to 4 to 8 bits, along with the tokenizer and metadata needed to run the model.
- Grouped-query attentionGQA
- Grouped-query attention (GQA) is an attention variant in which several query heads share one key and value head. It uses more key-value heads than multi-query attention and fewer than multi-head attention.
- Inference
- Inference is running a trained model to produce output. For an LLM that means processing the prompt, then generating tokens one at a time until a stop token or a length limit.
- KV cacheKey-value cache
- A KV cache (key-value cache) is memory that stores the attention keys and values of every token already processed, so each new token computes only its own. It trades compute for memory, cutting per-step attention cost from quadratic to linear in sequence length while the cache grows with sequence length and batch size.
- PagedAttention
- PagedAttention is the KV cache memory manager introduced with vLLM (Kwon et al., 2023). It stores the cache in fixed-size blocks, like pages in virtual memory, so requests do not need one contiguous allocation and fragmentation stays low.
- Prefill and decode
- Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in parallel and is limited by compute, and decode generates one token per step and is limited by memory bandwidth.
- Prefix caching
- Prefix caching is a serving-engine technique that reuses KV cache blocks from earlier requests when a new prompt starts with the same tokens. Prefill skips the matched tokens, which lowers time to first token and frees GPU compute.
- Quantization
- Quantization is storing model weights, and sometimes activations or the KV cache, in fewer bits, for example 8-bit or 4-bit instead of 16-bit. It cuts memory and often speeds up decoding, at some cost in quality that you should measure on your task.
- Speculative decoding
- Speculative decoding is a decoding method where a fast draft step proposes several tokens and the large model verifies them in one forward pass, keeping the longest accepted prefix. With the standard acceptance rule, the output distribution matches the large model's.
- Throughput
- Throughput is how many tokens a system generates per second, either for one request or summed across all requests on a deployment. Batching raises total throughput while usually lowering speed per request.
- TTFTTime to first token
- TTFT (time to first token) is the delay between sending a request and receiving the first output token. It covers queueing and prefill, so long prompts raise it and prompt caching lowers it.
Platforms