Terms, defined plainly

A2A (Agent2Agent) is an open protocol, introduced by Google in April 2025, that lets agents built by different vendors discover each other, exchange tasks, and stream results. MCP connects an agent to tools and data, while A2A connects agents to other agents.

A remote agent publishes a JSON Agent Card at /.well-known/agent-card.json listing its skills, endpoint, and auth requirements. Work runs as a task with a defined lifecycle, so a long job can pause for input and report status. As of November 8, 2026, the spec is at version 1.0.0 with JSON-RPC, gRPC, and HTTP+JSON bindings, and A2A is a Growth Stage project at the Agentic AI Foundation.

SourceA2A Protocol specification

MCPAgentMarkdown

An agent is a program that calls a model in a loop, runs the tools the model requests, and feeds the results back until a stop condition ends the loop. The tools and the stop condition set what the agent can do.

Each pass sends the accumulated history, so cost and latency grow with every turn. Anthropic's guide describes agents as LLMs using tools based on environmental feedback in a loop, and it recommends a stopping condition such as a maximum iteration count. The same guide advises finding the simplest solution possible and adding complexity only when needed.

SourceAnthropic, Building effective agents (2024)

Agent loopAgentic AITool callingMCPA2ARead the articleMarkdown

An agent loop is the code that sends the conversation to a model, runs any tool calls in the response, appends the results, and calls the model again. The loop ends when a response contains no tool calls or a limit is reached.

A turn is one pass: model output, then tool execution if the output asked for it. Without a cap, the loop runs until the model returns text with no tool calls, so set a turn limit and a spend limit in production. In the Claude Agent SDK those limits are max_turns and max_budget_usd, and as of November 8, 2026, both default to no limit.

SourceClaude Agent SDK docs: How the agent loop works

AgentAgentic AITool callingRead the articleMarkdown

Agentic AI is software in which a model's output selects the next step, usually a tool call, instead of a fixed script setting the order of steps. The term covers single coding agents as well as multi-agent systems.

Anthropic's guide draws the line at control flow. A workflow runs LLMs and tools through predefined code paths, and an agent lets model output direct the process and its tool use. The guide favors workflows for well-defined tasks because they are more predictable.

SourceAnthropic, Building effective agents (2024)

AgentAgent loopRead the articleMarkdown

An AI gateway is a proxy between your application and one or more model providers that gives you one API for many models. It also adds routing, fallbacks, caching, rate limits, cost tracking, and logging in one place.

A gateway sits in the request path, so it adds a network hop and needs its own uptime plan. It pays off once you call more than one provider, need per-team budgets, or want failover without code changes. A gateway's response cache is separate from provider prompt caching, which discounts repeated prefixes on the provider's side.

Prompt cachingGuardrailsMarkdown

Attention is the transformer operation that lets each token weigh the tokens before it when computing its own representation. Its cost grows with sequence length, which is why long contexts are expensive and why the KV cache exists.

Each token gets a query, a key, and a value vector. Scaled dot-product attention computes softmax(QK^T / sqrt(d_k)) V, so queries score keys and the scores weight the values. A model runs this across several heads in every layer.

SourceVaswani et al., Attention Is All You Need (2017)

TransformerKV cacheGrouped-query attentionRead the articleMarkdown

A batch API is an asynchronous endpoint that takes many model requests at once, processes them within a time window, and bills them at a discount. You give up immediate responses in exchange for a lower price.

You submit a list or file of requests, poll for completion, then download results keyed by your own request IDs. As of November 8, 2026, OpenAI, Anthropic, and Google all bill batch work at 50% of standard rates, with a 24-hour window or target. Use it for evals, backfills, and classification jobs where nobody is waiting.

SourceAnthropic docs: Pricing

Prompt cachingThroughputRead the articleMarkdown

Chunking is splitting documents into smaller passages before you embed and index them for retrieval. Chunk size sets how much text a single retrieved passage can bring into the prompt.

Small chunks match queries closely but lose surrounding context, and large chunks keep context but blur the embedding. Overlapping neighbors keeps a sentence from being cut at a boundary. Anthropic's Contextual Retrieval post says RAG systems usually chunk to no more than a few hundred tokens, and that size, boundaries, and overlap all affect retrieval.

SourceAnthropic, Introducing Contextual Retrieval (2024)

RAGEmbeddingVector databaseRerankingHybrid searchRead the articleMarkdown

Constrained decoding is generation in which tokens that would break a grammar or schema are masked at each step before sampling. The output follows the grammar by construction, unless generation stops early.

Willard and Louf (2023) frame it as walking a finite-state machine: precompute which vocabulary tokens are legal in each state, then mask the rest. The schema has to be compiled first, so the first request with a new schema is slower. Anthropic's docs describe compiling JSON Schemas into grammars and caching them for 24 hours after last use.

SourceWillard and Louf, Efficient Guided Generation for Large Language Models (2023)

Structured outputsTool callingMarkdown

Context engineering is the practice of choosing what goes into a model's context window for each call, and in what order: instructions, examples, retrieved documents, tool results, and memory. You treat the window as a limited budget and spend it on what the next step needs.

Anthropic defines it as curating and maintaining the best set of tokens during inference, which extends prompt engineering to tools, external data, and message history. Its working rule is the smallest set of high-signal tokens that makes the desired outcome likely. Long agent runs need compaction or tool-result clearing to stay inside the window.

SourceAnthropic, Effective context engineering for AI agents (2025)

Context windowSystem promptRAGPrompt cachingRead the articleMarkdown

A context window is the maximum number of tokens a model can handle in one request, usually counting the prompt and the output together. Everything the model has to work with for that request has to fit inside it.

On the Claude API, the system prompt, messages, tool definitions, tool results, generated output, and thinking all count, and cached tokens still occupy the window. As of November 8, 2026, Google's docs list separate input and output budgets for Gemini 3 and 2.5, and one shared 1M window for Gemini 4. Anthropic's docs warn that accuracy and recall degrade as token count grows, so a larger window is no reason to fill it.

SourceAnthropic docs: Context windows

TokenContext engineeringReasoning modelRead the articleMarkdown

Continuous batching is a serving technique that schedules work one model iteration at a time, so new requests join a running batch and finished ones leave it immediately. Requests stop waiting for the slowest sequence in the batch.

The Orca paper (Yu et al., OSDI 2022) calls this iteration-level scheduling and pairs it with selective batching, which applies batching only to a chosen subset of operations. Static batching holds every slot until the longest sequence finishes, so short requests leave GPU capacity idle.

SourceYu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)

ThroughputInferencePagedAttentionRead the articleMarkdown

Distillation is training a smaller student model to reproduce the outputs of a larger teacher model. It is a common way to get a cheaper model that keeps much of the teacher's quality on a narrow task.

Hinton et al. (2015) train the student on soft targets: both models divide their logits by a temperature above 1, so the teacher's near-miss answers carry signal, and the student returns to a temperature of 1 after training. DeepSeek's R1 paper reports that reasoning patterns from large models can also improve smaller ones.

SourceHinton, Vinyals, and Dean, Distilling the Knowledge in a Neural Network (2015)

Fine-tuningSmall language modelMarkdown

An embedding is a fixed-length vector of numbers that represents a piece of text, an image, or code, produced by an embedding model. Items with similar meaning end up close together, which is what vector search relies on.

Embed queries and documents with the same model, because vectors from different models are not comparable. As of November 8, 2026, OpenAI's text-embedding-3-small returns 1,536 dimensions by default and accepts a dimensions parameter to shorten the vector. OpenAI recommends cosine similarity, which on its unit-length vectors reduces to a dot product.

SourceOpenAI docs: Vector embeddings

Vector databaseRAGChunkingHNSWRead the articleMarkdown

An eval is a repeatable test of a model or an agent on examples from your task, scored by code, by people, or by another model. Evals show whether a prompt change, a model swap, or a new tool made results better or worse.

Anthropic's vocabulary is a useful starting point: a task has defined inputs and success criteria, a grader scores one aspect of the result, and a trial is one attempt. Outputs vary between runs, so run each task several times and compare pass rates.

SourceAnthropic, Demystifying evals for AI agents (2026)

Golden datasetLLM-as-a-judgeSWE-benchHallucinationGuardrailsRead the articleMarkdown

Fine-tuning is continuing to train a pretrained model on your own examples so it adopts a format, a style, or a skill. Full fine-tuning updates every weight, while parameter-efficient methods such as LoRA update a small set of added weights.

Fine-tuning changes behavior, so it suits output format and style. Facts that change belong in retrieval, because updating a fine-tuned model means training it again. As of November 8, 2026, Anthropic's glossary says the Claude API does not offer fine-tuning.

SourceAnthropic docs: Glossary

LoRAQLoRARLHFDistillationRAGRead the articleMarkdown

GGUF is the single-file model format used by llama.cpp and by tools that read it, such as Ollama. A GGUF file holds the weights, often quantized to 4 to 8 bits, along with the tokenizer and metadata needed to run the model.

A GGUF file has a header, key-value metadata, tensor info, and tensor data, so a runtime needs no external files to load the model. The spec describes version 3 of the format. Its tokenizer.ggml keys hold the vocabulary, and the spec warns that those keys are not guaranteed to be standardized across models.

Sourceggml docs: GGUF specification

QuantizationInferenceSmall language modelRead the articleMarkdown

A golden dataset is a fixed, versioned set of inputs with known-good outputs or success criteria that you use to score a model or an agent. Scores stay comparable across runs because the set does not change between them.

Build it from real usage and real failures, then add a case each time a bug reaches users. Anthropic's eval guidance says 20 to 50 simple tasks drawn from real failures is a great start, and that teams often wait too long for hundreds. Keep it separate from the examples you tune prompts against, or the score stops predicting production.

SourceAnthropic, Demystifying evals for AI agents (2026)

EvalLLM-as-a-judgeSWE-benchRead the articleMarkdown

Grouped-query attention (GQA) is an attention variant in which several query heads share one key and value head. It uses more key-value heads than multi-query attention and fewer than multi-head attention.

The KV cache stores only key and value heads, so cutting them shrinks cache memory and the memory read on every decode step. Ainslie et al. (2023) report that GQA reaches quality close to multi-head attention at a speed comparable to multi-query attention.

SourceAinslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023)

AttentionKV cacheRead the articleMarkdown

Guardrails are checks that run on a model's input or output to block, rewrite, or flag content such as personal data, prompt injection, policy violations, or malformed responses. They run outside the model, so they still apply when the model's output is wrong.

Each check adds latency, and running input checks in parallel with the model call limits that cost. OWASP lists prompt injection as LLM01 and says it is unclear whether fool-proof prevention exists. Treat guardrails as one layer alongside least-privilege tools and human approval for risky actions.

SourceOWASP, LLM01:2025 Prompt Injection

EvalHallucinationAI gatewayMarkdown

A hallucination is model output that reads as confident and fluent but is false or unsupported by the input, such as an invented API, citation, or number. Retrieval and tool use reduce it, and evals measure it, but none of them remove it.

Kalai et al. (2025) trace it to incentives: pretraining produces errors when a model cannot separate true statements from false ones, and most benchmarks grade like exams where guessing can beat saying "I don't know." In your own evals, score a wrong answer worse than an abstention.

SourceKalai et al., Why Language Models Hallucinate (2025)

RAGEvalGuardrailsMarkdown

HNSW (Hierarchical Navigable Small World) is a graph index for approximate nearest neighbor search that stacks proximity graphs, with fewer points in each higher layer. Queries start at the top layer and descend, which gives logarithmic scaling with dataset size.

Each vector gets a random top layer with an exponentially decaying probability, so upper layers stay sparse. M limits links per node, ef_construction sets the candidate list at build time, and ef_search sets it at query time, which trades recall for latency. pgvector's docs list IVFFlat as faster to build and lighter on memory than HNSW, with lower query performance at the same recall.

SourceMalkov and Yashunin, Hierarchical Navigable Small World graphs (2016)

Vector databaseEmbeddingHybrid searchRead the articleMarkdown

Hybrid search combines keyword search, usually BM25, with vector search and merges the two ranked lists into one result. Keyword matching catches exact identifiers that embeddings miss, and vectors catch paraphrases that keywords miss.

Reciprocal rank fusion is a common merge step because it uses ranks and needs no score normalization. Each document scores the sum of 1 / (k + rank) across lists, and Elasticsearch defaults k to 60. In Anthropic's own test, adding contextual BM25 to contextual embeddings raised the cut in top-20 retrieval failures from 35% to 49%.

SourceElastic docs: Reciprocal rank fusion

Vector databaseHNSWRerankingRAGChunkingRead the articleMarkdown

Inference is running a trained model to produce output. For an LLM that means processing the prompt, then generating tokens one at a time until a stop token or a length limit.

LLM inference has two phases with different hardware limits, prefill and decode. Serving engines such as vLLM, SGLang, and llama.cpp schedule these phases and manage the KV cache, so the engine you pick changes latency and cost on the same hardware.

Prefill and decodeKV cacheThroughputTTFTContinuous batchingGGUFReasoning modelSpeculative decodingRead the articleMarkdown

A KV cache (key-value cache) is memory that stores the attention keys and values of every token already processed, so each new token computes only its own. It trades compute for memory, cutting per-step attention cost from quadratic to linear in sequence length while the cache grows with sequence length and batch size.

Per token, each layer stores one key and one value vector for each KV head, so bytes per token = 2 × layers × KV heads × head dim × bytes per value. The cache belongs to one sequence, so concurrent requests multiply the total. Servers ration that memory with managers such as PagedAttention.

SourceHugging Face Transformers docs: How caching works

AttentionPagedAttentionPrefill and decodeInferenceGrouped-query attentionQuantizationPrefix cachingPrompt cachingRead the articleMarkdown

LLM-as-a-judge is an evaluation method that uses one model to score the outputs of another against a rubric or a comparison prompt. It gives you a scorer for open-ended text that exact-match checks cannot grade.

Zheng et al. (2023) found that strong judges such as GPT-4 agreed with human preferences more than 80% of the time, about the level of agreement among humans. The same paper documents position, verbosity, and self-enhancement biases, so validate a judge against human labels on your own task before you trust it.

SourceZheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)

EvalGolden datasetRead the articleMarkdown

LoRA (low-rank adaptation) is a fine-tuning method from Hu et al. (2021) that freezes the base model and trains small low-rank matrices added to selected layers. An adapter is far smaller than the model it modifies, so one base model can serve many adapters.

Each adapted weight becomes W + (alpha / r) BA, where B and A have rank r and only they are trained. Rank r sets adapter size and capacity, and alpha scales the strength of the update. Hu et al. adapted only attention projections on GPT-3 175B and report training about 0.01% of the parameters.

SourceHu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)

Fine-tuningQLoRARead the articleMarkdown

MCP (Model Context Protocol) is an open protocol, introduced by Anthropic in November 2024, that standardizes how AI applications connect to tools and data. A host application runs MCP clients that talk to MCP servers over JSON-RPC, using stdio for local servers or Streamable HTTP for remote ones.

Servers offer resources, prompts, and tools. The 2026-07-28 revision makes requests stateless and self-contained and negotiates capabilities per request, replacing the earlier initialize handshake. Anthropic donated MCP to the Agentic AI Foundation, a Linux Foundation fund, in December 2025.

SourceModel Context Protocol specification, revision 2026-07-28

MCP serverTool callingA2AAgentRead the articleMarkdown

An MCP server is a program that exposes tools, resources, and prompts to any MCP-compatible client. Write one server for your API or database, and every MCP host, from coding agents to chat apps, can use it.

Tools are model-controlled functions, resources hold context data, and prompts are templates for users. A server declares each capability, then answers requests such as tools/list and tools/call. The spec says a human should be able to deny tool invocations, and that tool annotations are untrusted unless they come from a trusted server.

SourceModel Context Protocol specification: Tools (2026-07-28)

MCPTool callingRead the articleMarkdown

Mixture of experts (MoE) is a model architecture that replaces selected feed-forward layers with many expert sub-networks and routes each token to only a few of them. Total parameters can be very large while the compute per token stays close to that of a much smaller dense model.

Shazeer et al. (2017) introduced the sparsely gated MoE layer, where a trainable gating network selects experts for each input. Memory scales with total parameters because every expert must be loaded, while compute per token scales with the active ones.

SourceShazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017)

TransformerRead the articleMarkdown

PagedAttention is the KV cache memory manager introduced with vLLM (Kwon et al., 2023). It stores the cache in fixed-size blocks, like pages in virtual memory, so requests do not need one contiguous allocation and fragmentation stays low.

Equal-size blocks remove external fragmentation, and small blocks limit internal fragmentation. Kwon et al. measured that earlier systems used only 20.4% to 38.2% of KV cache memory for actual token state, and they report 2 to 4 times the throughput at the same latency.

SourceKwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)

KV cacheThroughputContinuous batchingPrefix cachingRead the articleMarkdown

Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in parallel and is limited by compute, and decode generates one token per step and is limited by memory bandwidth.

Patel et al. (Splitwise, 2023) call these prompt computation and token generation, and show that token generation underuses compute. Time to first token mostly reflects prefill and time between tokens reflects decode, so the two respond to different fixes.

SourcePatel et al., Splitwise: Efficient generative LLM inference using phase splitting (2023)

InferenceKV cacheTTFTRead the articleMarkdown

Prefix caching is a serving-engine technique that reuses KV cache blocks from earlier requests when a new prompt starts with the same tokens. Prefill skips the matched tokens, which lowers time to first token and frees GPU compute.

vLLM hashes each block from its parent's hash, its own tokens, and extras such as LoRA IDs or a cache salt, so a block matches only when its whole prefix matches, and only full blocks are cached. SGLang's RadixAttention keeps prefixes in a radix tree with LRU eviction. Prompt caching is the hosted API feature built on the same idea, with its own pricing.

SourcevLLM docs: Automatic Prefix Caching

Prompt cachingKV cacheTTFTPagedAttentionRead the articleMarkdown

Prompt caching is a hosted API feature that reuses the processed state of a repeated prompt prefix, such as a long system prompt or a document, across calls. Providers that offer it bill cached input tokens at a discount and return the first token sooner.

Hits need an exact match of everything before the cache point, so put stable content first and volatile content last. Anthropic hashes tools, then system, then messages, so changing a tool definition invalidates the whole cache. As of November 8, 2026, Claude charges 1.25 times the input price for a 5-minute write and 0.025 to 0.1 times for reads, while OpenAI caches automatically from 1,024 tokens on GPT-5.6 and later.

SourceAnthropic docs: Prompt caching

Prefix cachingKV cacheContext engineeringSystem promptAI gatewayBatch APITTFTRead the articleMarkdown

QLoRA is a fine-tuning method from Dettmers et al. (2023) that trains LoRA adapters on a base model quantized to 4 bits. It lowers the memory needed to fine-tune a large model.

The base weights use a 4-bit NormalFloat (NF4) format, double quantization compresses the quantization constants, and paged optimizers manage memory spikes during training. Reach for it when the model will not fit on your GPU at full precision.

SourceDettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)

LoRAFine-tuningQuantizationRead the articleMarkdown

Quantization is storing model weights, and sometimes activations or the KV cache, in fewer bits, for example 8-bit or 4-bit instead of 16-bit. It cuts memory and often speeds up decoding, at some cost in quality that you should measure on your task.

GPTQ (Frantar et al., 2022) brought weights down to 3 or 4 bits and ran a 175B-parameter model on a single GPU. Decode is limited by memory bandwidth, so smaller weights can raise tokens per second. Quality loss varies by model and task, so run your own evals before and after.

SourceFrantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022)

GGUFKV cacheQLoRASmall language modelMarkdown

RAG (retrieval-augmented generation) is a technique that fetches relevant documents at request time and puts them in the prompt so the model answers from them. The term comes from Lewis et al. (2020).

A typical pipeline chunks documents, embeds them, retrieves the nearest passages for each query, and often reranks them before the model call. If retrieval misses the right passage, the model cannot answer from it, so measure retrieval and generation separately. Use RAG for facts that change or need citations, and fine-tuning for format and style.

SourceLewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)

EmbeddingVector databaseChunkingRerankingHybrid searchContext engineeringHallucinationFine-tuningRead the articleMarkdown

A reasoning model is a model trained to generate intermediate reasoning tokens before its final answer. It trades latency and output tokens for better results on math, code, and multi-step problems.

OpenAI bills reasoning tokens as output tokens and counts them against the context window, even though the API does not return them. DeepSeek-R1 reports that reinforcement learning alone can produce this behavior, without human-labeled reasoning examples. OpenAI and Anthropic both expose an effort setting that trades reasoning depth for latency and token use.

SourceOpenAI docs: Reasoning models

TokenInferenceContext windowMarkdown

Reranking is a second retrieval pass that rescores a shortlist of candidates with a slower, more accurate model and keeps the best few. The first pass can favor recall because the reranker sets the final order.

A reranker scores the query and each passage together, so it catches relevance that separate embeddings miss, at the cost of added latency per candidate. Nogueira and Cho (2019) reported a 27% relative gain in MRR@10 on MS MARCO with BERT as the reranker.

SourceNogueira and Cho, Passage Re-ranking with BERT (2019)

RAGHybrid searchChunkingVector databaseRead the articleMarkdown

RLHF (reinforcement learning from human feedback) is training a model with a reward signal learned from human preference comparisons. It is one of the post-training steps that turns a pretrained model into an assistant that follows instructions.

People rank model outputs, a reward model learns those rankings, and reinforcement learning pushes the model toward outputs the reward model scores highly. Christiano et al. (2017) demonstrated learning from pairwise preferences on Atari and robot locomotion, and InstructGPT (Ouyang et al., 2022) applied it to language models.

SourceOuyang et al., Training language models to follow instructions with human feedback (2022)

Fine-tuningMarkdown

A small language model is a language model small enough to run on a single consumer GPU, a laptop, or a phone, usually under about 10 billion parameters. It suits narrow tasks, low latency, and keeping data on the device.

Belcak et al. (2025) define the term by device fit: a model that runs on a common consumer device at low enough latency to serve one user. They say they would count most models below 10 billion parameters, and they argue that many agent calls do not need a larger model.

SourceBelcak et al., Small Language Models are the Future of Agentic AI (2025)

QuantizationDistillationGGUFMarkdown

Speculative decoding is a decoding method where a fast draft step proposes several tokens and the large model verifies them in one forward pass, keeping the longest accepted prefix. With the standard acceptance rule, the output distribution matches the large model's.

Leviathan et al. (2023) accept each draft token with probability min(1, p / q), where p is the target model's probability and q is the draft's. A rejection discards the later drafts and resamples from an adjusted distribution. The target model never needs more serial steps than standard decoding, and the speedup grows with the acceptance rate.

SourceLeviathan et al., Fast Inference from Transformers via Speculative Decoding (2023)

InferenceThroughputRead the articleMarkdown

Structured outputs is an API feature that constrains a model's output to match a JSON Schema you supply, so successful responses parse against your schema. It is usually implemented with constrained decoding, which masks tokens that would break the schema.

Handle the exceptions: a refusal can return text outside the schema, and hitting a token limit can cut the JSON short. Check the refusal field or stop reason before parsing. The first request with a new schema adds latency while the API processes it.

SourceAnthropic docs: Structured outputs

Tool callingConstrained decodingMarkdown

SWE-bench is a benchmark that asks a model or agent to fix real GitHub issues by editing the repository, then runs the project's tests to check the patch. The original set has 2,294 problems from 12 Python repositories.

SWE-bench Verified is a 500-problem subset that engineers confirmed as solvable. Compare scores only within one dataset and one agent setup, because the code around the model changes the result.

SourceJimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023)

EvalGolden datasetRead the articleMarkdown

A system prompt is a set of instructions placed before the conversation that sets the model's role, rules, and output format for every turn. Keeping it stable makes it a good candidate for prompt caching.

APIs name it differently: Anthropic uses a top-level system parameter, and OpenAI's Responses API uses instructions or developer messages, which take priority over user messages. Keep secrets and permission checks out of it, and enforce those in the code that runs tools.

SourceOpenAI docs: Text generation

Context engineeringPrompt cachingRead the articleMarkdown

Temperature is a sampling setting that divides the model's logits before they are converted to token probabilities. Lower values make output more deterministic, and higher values make it more varied.

A temperature of 0 does not guarantee identical outputs, as Anthropic's glossary says of its own API. As of November 8, 2026, Claude 4.7 and later models accept only the default sampling values, and API version 2026-10-21 rejects any request that sends temperature, top_p, or top_k. On those models, steer output with the prompt and effort settings.

SourceAnthropic docs: Migrating to API version 2026-10-21

Top-pTokenMarkdown

Throughput is how many tokens a system generates per second, either for one request or summed across all requests on a deployment. Batching raises total throughput while usually lowering speed per request.

Check which number a benchmark reports. Tokens per second per request is what a user feels, and total tokens per second is what sets your serving cost. Continuous batching and better memory management raise the total by keeping GPU slots busy.

TTFTInferenceContinuous batchingBatch APIPagedAttentionSpeculative decodingMarkdown

A token is the unit of text a model takes in and generates: a word, part of a word, a symbol, or a byte sequence, depending on the tokenizer. Context limits, speed, and API prices are all counted in tokens.

Token counts differ by model, so use the provider's counting tool for anything that affects cost. Google says a Gemini token is about 4 characters. Anthropic says about 3.5 English characters on earlier Claude models, and that Claude 4.7 and later produce about 30% more tokens for the same text.

SourceAnthropic docs: Glossary

TokenizerContext windowTemperatureReasoning modelRead the articleMarkdown

A tokenizer is the component that splits text into tokens and maps them to integer IDs before the model processes them. Different model families use different tokenizers, so the same text can produce a different number of tokens.

Byte pair encoding is a common method: it builds a vocabulary by repeatedly merging frequent adjacent symbol pairs, and Sennrich et al. (2015) applied it to rare words in neural translation. Count with the tokenizer that matches the model you call, because a mismatch gives wrong numbers.

SourceOpenAI Cookbook: How to count tokens with tiktoken

TokenMarkdown

Tool calling is a model returning a structured request to run a function you defined, with arguments, instead of plain text. Your code runs the function and sends the result back, which is how an agent loop reaches systems outside the model.

The model only produces the request, and your code or the provider's servers do the execution. On the Claude API a client tool comes back as a tool_use block with stop_reason tool_use, and you reply with a tool_result block. Tool names, descriptions, and schemas all count as input tokens, so a large tool set shrinks your context budget.

SourceAnthropic docs: Tool use with Claude

AgentAgent loopMCPMCP serverStructured outputsConstrained decodingRead the articleMarkdown

Top-p, or nucleus sampling, is a sampling method that keeps only the smallest set of most likely tokens whose probabilities add up to at least p, then samples from that set. The candidate count adapts to how concentrated the distribution is at each step.

Holtzman et al. (2019) proposed it to trim the unreliable low-probability tail while keeping diversity. As of November 8, 2026, API version 2026-10-21 of the Claude API rejects requests that send top_p, so check your provider's docs before you depend on it.

SourceHoltzman et al., The Curious Case of Neural Text Degeneration (2019)

TemperatureMarkdown

A transformer is the neural network architecture behind modern LLMs, introduced in the 2017 paper Attention Is All You Need (Vaswani et al.). It stacks attention and feed-forward layers and processes all tokens of the input in parallel.

A transformer has no recurrence or convolution, so it adds positional encodings to the token embeddings to carry word order. Prefill can process a whole prompt at once because of this design, while generation still proceeds one token at a time.

SourceVaswani et al., Attention Is All You Need (2017)

AttentionMixture of expertsMarkdown

TTFT (time to first token) is the delay between sending a request and receiving the first output token. It covers queueing and prefill, so long prompts raise it and prompt caching lowers it.

TTFT sets how responsive a streaming interface feels, and it says nothing about how fast the rest of the reply arrives. Report it as a percentile under load, because queueing time only shows up when the server is busy.

SourcePatel et al., Splitwise: Efficient generative LLM inference using phase splitting (2023)

Prefill and decodeThroughputInferencePrefix cachingPrompt cachingMarkdown

A vector database is a database that indexes embeddings and returns the nearest neighbors to a query vector, usually with approximate search such as HNSW. It is a common retrieval layer in RAG systems.

Approximate indexes trade some recall for speed, and you tune that trade with parameters such as ef_search. pgvector runs exact search by default and switches to approximate search when you add an HNSW or IVFFlat index. Metadata filters interact with the index, so measure recall with your real filters on.

Sourcepgvector on GitHub

EmbeddingRAGHNSWHybrid searchRerankingChunkingRead the articleMarkdown