---
title: "Context engineering: budget and order the context window"
type: "guide"
url: "https://aifordevs.org/articles/context-engineering"
markdown: "https://aifordevs.org/articles/context-engineering.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/context-engineering"
description: "Context engineering is choosing what a model reads on each call. Size each section, order it for prompt caching, trim inputs, and test every layout change."
author: "AI for Devs"
published: "2026-11-06"
updated: "2026-11-08"
level: "Intermediate"
topics: ["context", "rag", "agents"]
---

# Context engineering: budget and order the context window

Context engineering is choosing what a model reads on each call. Size each section, order it for prompt caching, trim inputs, and test every layout change.

> **TL;DR**
>
> - Context engineering is the practice of choosing which tokens reach a model on each call: instructions, tool definitions, retrieved documents, tool results, and conversation history.
> - Size the context window by section. Every current Claude model listed a 1M-token window as of November 8, 2026, and Anthropic's docs still warn that accuracy and recall degrade as token count grows.
> - Put stable content first so prompt caching can reuse it. Minimum cacheable sizes ranged from 512 to 4,096 tokens on Claude models as of November 8, 2026, and Anthropic and OpenAI require an exact prefix match.
> - Trim retrieved documents, history, and tool results before they reach the window, and rerun your evals after any layout change, because reordering can change answers as well as cost.
> - The TypeScript budget helper in this guide ran on Node.js 26.12.1 with `js-tiktoken` 1.0.21 on November 8, 2026. It used synthetic text and called no provider API.

Context engineering is the practice of deciding which instructions, documents, tool results, and history reach a model on each call, and how much room each gets. A coding agent reading a repository or a support bot pulling runbooks rebuilds that block of tokens for every request. Failures show up as truncated answers, a cache that never hits, or an agent that loses a decision made 30 turns ago. This guide turns the window into something you can size, order, log, and test.

## What is context engineering

Context engineering is the work of assembling everything a model reads for one request, then keeping that block small enough to pay for and clear enough to use. Anthropic describes it as strategies for "curating and maintaining the optimal set of tokens" during inference. In practice it is code that runs before every API call.

[Anthropic's context window docs](https://platform.claude.com/docs/en/build-with-claude/context-windows) count everything in the request toward the limit: the system prompt, every message, tool results, images, documents, and tool definitions. The output counts too. That is the full list of parts you are engineering, and each one changes at a different rate.

| Section | How often it changes | What goes wrong without a cap |
| --- | --- | --- |
| Tool definitions | Per release, or per user if you filter tools | Thousands of tokens spent before the first turn |
| System prompt and rules | Per release | Rules pile up and contradict each other |
| Shared documents | Per day or week | Stale facts get cached and reused |
| Conversation history | Per turn | The window fills and early decisions disappear |
| Retrieved documents | Per request | Noise crowds out the passage that answers the question |
| Tool results | Per step | One raw API response takes the whole budget |

The term spread in mid-2025. [Simon Willison's June 27, 2025 note](https://simonwillison.net/2025/Jun/27/context-engineering/) credits Shopify's Tobi Lutke with an early endorsement and Andrej Karpathy with amplifying it. If you want the working definitions behind the label, see the glossary entries for [context engineering](https://aifordevs.org/glossary#context-engineering) and the [context window](https://aifordevs.org/glossary#context-window).

## Context engineering vs prompt engineering

Prompt engineering is the wording of instructions. Context engineering is the selection and layout of everything around those instructions, including the instructions themselves. Prompt engineering stays part of the job, and much of this work was practiced as retrieval, summarization, and tool design before anyone used the term.

The new name earned its place because of agent loops. A prompt is written once and reviewed by a person. In an [agent loop](https://aifordevs.org/glossary#agent-loop), code rebuilds the input on every step from tool results and history that no person ever reads.

| | Prompt engineering | Context engineering |
| --- | --- | --- |
| Unit of work | One [system prompt](https://aifordevs.org/glossary#system-prompt) or template | Every token in the request |
| Main question | How should this instruction read? | What should be in the window on this call? |
| Lives in | A prompt file | Assembly code, retrieval, tools, memory |
| Typical failure | Instruction misread or ignored | Right answer absent, buried, or evicted |
| How you test it | Eval on wording changes | Eval on layout, retrieval, and history changes |

For a single-turn feature with a fixed input, prompt engineering is the whole job and the rest of this guide is overhead. Once you add [retrieval](https://aifordevs.org/topics/rag), [tool calling](https://aifordevs.org/glossary#tool-calling), or multi-turn state, you are already making these decisions, whether you name them or not.

Nothing about the term gives a model new abilities. It names choices that used to stay implicit.

## Budget the context window by section

Give every section of the context window a token cap before you write the code that fills it. Retrieval that can return 80K tokens and history with no limit will eventually collide. The first symptom is usually a failed request or an answer that stops early.

**Figure 1.** An example allocation for one call. The numbers are a starting point for a 128K-token window, and each section has a size someone chose on purpose.

On Claude, overflow has two forms, per [Anthropic's context window docs](https://platform.claude.com/docs/en/build-with-claude/context-windows) as of November 8, 2026. If the input alone exceeds the window, the API returns a 400 error.

On Claude 4.5 models and newer, input plus `max_tokens` can exceed the window, and generation stops with `stop_reason: "model_context_window_exceeded"` if it reaches the limit. Treat both as bugs in your budget.

Large windows do not remove the budget. Every current Claude model listed a [1M-token window](https://platform.claude.com/docs/en/models/overview) on November 8, 2026. The same docs say accuracy and recall degrade as token count grows, a problem Anthropic calls context rot, and cached prefixes still count toward the window.

Recount when you change models. [Anthropic's token counting docs](https://platform.claude.com/docs/en/build-with-claude/token-counting) say Claude Opus 4.7 and later models use a tokenizer that produces about 30 percent more tokens for the same text than earlier models. A budget measured on the old [tokenizer](https://aifordevs.org/glossary#tokenizer) will overflow on the new one.

## Order the window for prompt caching

Prompt caching reuses a prefix of your input that matches an earlier request exactly, so the order of your sections decides whether you get the discount. Put the parts that never change first and the parts that change per request last. One changed token near the start forfeits everything after it.

**Figure 2.** With stable sections first, only the tail is reprocessed. A timestamp at the start changes the prefix, so nothing after it can match.

Use this order:

1. Tool definitions
2. System prompt and rules
3. Long shared documents, such as a policy manual
4. Conversation history, oldest first
5. Retrieved documents for this turn
6. The current request

An earlier version of this article put the system prompt before tools. Anthropic's API renders `tools`, then `system`, then `messages`, so tools come first there. Provider behavior also moved since the article first ran, and this table reflects each provider's docs on November 8, 2026.

| Provider | How it turns on | Minimum size | Lifetime | Where hits are reported |
| --- | --- | --- | --- | --- |
| [Anthropic](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) | Top-level `cache_control` for automatic breakpoints, or up to 4 explicit breakpoints | 512 to 4,096 tokens, by model | 5 minutes by default, 1 hour optional | `cache_read_input_tokens` |
| [OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching) | On by default; GPT-5.6 and later also accept explicit breakpoints | 1,024 visible tokens on GPT-5.6 and later | At least 30 minutes on GPT-5.6 and later | `usage.input_tokens_details.cached_tokens` |
| [Google Gemini](https://aistudio.google.com/docs/caching) | Implicit caching on by default for Gemini 2.5 and newer | 2,048 on Gemini 2.5, 4,096 on Gemini 3.x and Gemini 4 Argon | Not stated on the caching page | `usage.total_cached_tokens` |

Several ordinary changes break a cache without any error. Anthropic documents that editing tool definitions invalidates the tools, system, and messages caches. OpenAI lists `tools`, `reasoning.effort`, and `text.format` among settings that can change the prefix.

A cache entry also becomes available only after the first response starts, per Anthropic's docs, so parallel requests sent together miss it.

Long conversations have a second trap on Claude. Anthropic's lookup checks at most 20 positions back from each breakpoint, so a turn that adds many blocks can walk past the last cached entry. Add a breakpoint inside long turns.

Caching pays back fast. Claude Opus 5.5 listed $4 per million input tokens on November 8, 2026, with cache writes at 1.25 times that and reads at 0.05 times. A 20K-token prefix sent 10 times inside the 5-minute window costs $0.80 uncached and $0.136 cached, 83 percent less.

The cache pays for itself on the second call, or on the third with the 1-hour write at 2 times the input rate. See [Claude API pricing](https://aifordevs.org/articles/claude-api-pricing) and [Gemini API pricing](https://aifordevs.org/articles/gemini-api-pricing) for current rates, and the [KV cache article](https://aifordevs.org/articles/kv-cache-explained) for what the server stores.

## Retrieve less, and label it

Retrieve a small, reranked set of chunks and label each one with its source, because more retrieved text adds noise faster than it adds answers. [Liu et al.](https://arxiv.org/abs/2307.03172) found that moving from 20 to 50 retrieved documents added about 1.5 points of accuracy for GPT-3.5-Turbo and about 1 point for Claude-1.3.

The same paper is the source of the "lost in the middle" result, and it deserves a careful reading. On multi-document QA with 20 documents, GPT-3.5-Turbo scored 75.8 percent with the answer first, 53.8 percent in the middle, and 63.2 percent last. Its closed-book score was 56.1 percent, so a middle placement did worse than giving the model no documents at all.

The effect has limits. Claude-1.3 scored nearly perfectly on the paper's key-value retrieval task at every length tested, and every model tested dates from 2023. Later work narrows the claim:

- [Veseli et al. (2025)](https://arxiv.org/abs/2508.07479) report the middle penalty is strongest when inputs fill up to 50 percent of the window, and shifts toward favoring text near the end as inputs get longer.
- [Gabín et al. (2026)](https://arxiv.org/abs/2605.27105) report that order and context size interact strongly with retrieval quality and model choice, so results from idealized setups do not always transfer.

The practical rule is to rerank, then put the strongest evidence near the start or end, and measure the effect on your own model. Keep only chunks above a score threshold, and wrap each with an ID so the model can cite it and you can check the citation:

```xml
<document id="doc-3" source="runbooks/payments.md" updated="2026-10-14">
Refunds over $500 need a second approver in the finance channel.
</document>
```

[Reranking](https://aifordevs.org/glossary#reranking) and [chunking](https://aifordevs.org/glossary#chunking) decide what makes this list. The [vector database guide](https://aifordevs.org/articles/what-is-a-vector-database) covers retrieval, and [RAG vs fine-tuning](https://aifordevs.org/articles/rag-vs-fine-tuning) covers when retrieval is the wrong fix.

## Compress history instead of dropping it

Keep the last few turns verbatim and replace older turns with a summary that preserves decisions, identifiers, file paths, and open questions. Anthropic and OpenAI both offered server-side compaction as of November 8, 2026, so you may not need your own summarizer, but you still need to test what each summary loses.

[Anthropic's compaction](https://platform.claude.com/docs/en/build-with-claude/compaction) was in beta on that date, in two forms: on demand, or triggered by a token threshold you set. [OpenAI's compaction](https://developers.openai.com/api/docs/guides/compaction) uses `context_management` with a `compact_threshold`, and the returned item is opaque.

If you write your own summarizer, follow [Anthropic's advice](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents) to tune for recall first, then cut what you do not need. A summary saying only "the user asked about refunds" keeps none of the parts a later turn needs.

For agents, clear old tool results once their conclusions are recorded. [Anthropic's context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing), also in beta on November 8, 2026, clears old tool results past a trigger and defaults to keeping the last 3 tool uses. It invalidates cached prefixes when it fires, so set `clear_at_least` high enough that each clearing is worth the cache write.

## Trim tool results at the source

A tool that returns a full API response fills the window with fields the next step never reads. Return only the fields the next step needs, cap list lengths, and paginate. Anthropic reported in September 2025 that Claude Code restricts tool responses to 25,000 tokens by default.

[Anthropic's tool-design post](https://www.anthropic.com/research/writing-tools-for-agents) recommends pagination, filtering, range selection, and truncation with sensible defaults. In its Slack example, a concise response format used about a third of the tokens of a detailed one. Error results should state what to change, because the next call depends on what the message says.

You control this most directly when you write the tool yourself. The [MCP server guide](https://aifordevs.org/articles/build-an-mcp-server) shows a small server with narrow results, and [what is an AI agent](https://aifordevs.org/articles/what-is-an-ai-agent) covers the loop that consumes them.

## Measure every section

Log the token count of each section on every call, next to latency and cost. Provider usage fields report totals and cache splits, so the per-section numbers have to come from your own code. The helper below checks caps, computes the cacheable prefix, and fails loudly when a section is over budget.

```ts title="budget.ts"
export type Section = { name: string; text: string; max: number; stable?: boolean };

type Options = {
  windowTokens: number;
  reserveForOutput: number;
  count?: (text: string) => number;
};

// Rough default: about 4 characters per token for English prose.
export const estimate = (text: string) => Math.ceil(text.length / 4);

export function assemble(sections: Section[], options: Options) {
  const { windowTokens, reserveForOutput, count = estimate } = options;
  const report = sections.map((s) => ({
    name: s.name,
    tokens: count(s.text),
    max: s.max,
    stable: Boolean(s.stable),
  }));

  const over = report.filter((r) => r.tokens > r.max).map((r) => r.name);
  const total = report.reduce((sum, r) => sum + r.tokens, 0);
  const headroom = windowTokens - reserveForOutput - total;
  if (over.length || headroom < 0) {
    throw new Error(`Context over budget: ${JSON.stringify({ total, headroom, over })}`);
  }

  // Cacheable prefix: tokens before the first section that changes per request.
  const firstVolatile = report.findIndex((r) => !r.stable);
  const cut = firstVolatile === -1 ? report.length : firstVolatile;
  const stablePrefix = report.slice(0, cut).reduce((sum, r) => sum + r.tokens, 0);

  return { text: sections.map((s) => s.text).join("\n\n"), report, total, stablePrefix, headroom };
}
```

A driver script fills the sections with synthetic support-assistant text, prints the report, cross-checks the estimate against OpenAI's `o200k_base` encoding, and then tries a section over its cap.

```ts title="demo.ts"
import { assemble, estimate, type Section } from "./budget.ts";
import { getEncoding } from "js-tiktoken";

// Synthetic content, shaped like a support assistant's context.
const rules = "You answer billing questions for the support team. Cite a document id for every claim. ".repeat(40);
const tools = JSON.stringify(
  Array.from({ length: 12 }, (_, i) => ({
    name: `lookup_${i}`,
    description: "Look up a billing record by id and return its status.",
    input_schema: { type: "object", properties: { id: { type: "string" } }, required: ["id"] },
  })),
);
const docs = Array.from(
  { length: 8 },
  (_, i) => `<document id="doc-${i}" source="runbooks/payments.md">\nRefunds over $500 need a second approver in the finance channel.\n</document>`,
).join("\n").repeat(12);
const history = "user: Why was my card charged twice?\nassistant: Two authorizations posted; one will drop off in 3 days.\n".repeat(90);
const task = "Which approval does refund R-1042 need, and who can give it?";

const sections: Section[] = [
  { name: "tools", text: tools, max: 7_000, stable: true },
  { name: "rules", text: rules, max: 3_000, stable: true },
  { name: "history", text: history, max: 28_000 },
  { name: "documents", text: docs, max: 36_000 },
  { name: "task", text: task, max: 6_000 },
];

const budget = { windowTokens: 128_000, reserveForOutput: 16_000 };
const out = assemble(sections, budget);
console.table(out.report);
console.log({ total: out.total, stablePrefix: out.stablePrefix, headroom: out.headroom });

// Cross-check the 4-characters rule against a real tokenizer (OpenAI's o200k_base).
const enc = getEncoding("o200k_base");
console.table(
  sections.map((s) => {
    const real = enc.encode(s.text).length;
    const est = estimate(s.text);
    return { name: s.name, chars: s.text.length, estimate: est, o200k: real, "est/real": Number((est / real).toFixed(2)) };
  }),
);

// A section over its cap fails the build instead of truncating in production.
try {
  assemble([...sections, { name: "ticket_dump", text: "x".repeat(40_000), max: 1_000 }], budget);
} catch (e) {
  console.log(String(e));
}
```

Set `"type": "module"` in `package.json`, install `js-tiktoken`, and run `node demo.ts`. Node.js 26.12.1 strips the types natively, and printed:

```text
┌─────────┬─────────────┬────────┬───────┬────────┐
│ (index) │ name        │ tokens │ max   │ stable │
├─────────┼─────────────┼────────┼───────┼────────┤
│ 0       │ 'tools'     │ 538    │ 7000  │ true   │
│ 1       │ 'rules'     │ 870    │ 3000  │ true   │
│ 2       │ 'history'   │ 2340   │ 28000 │ false  │
│ 3       │ 'documents' │ 3093   │ 36000 │ false  │
│ 4       │ 'task'      │ 15     │ 6000  │ false  │
└─────────┴─────────────┴────────┴───────┴────────┘
{ total: 6856, stablePrefix: 1408, headroom: 105144 }
┌─────────┬─────────────┬───────┬──────────┬───────┬──────────┐
│ (index) │ name        │ chars │ estimate │ o200k │ est/real │
├─────────┼─────────────┼───────┼──────────┼───────┼──────────┤
│ 0       │ 'tools'     │ 2151  │ 538      │ 506   │ 1.06     │
│ 1       │ 'rules'     │ 3480  │ 870      │ 681   │ 1.28     │
│ 2       │ 'history'   │ 9360  │ 2340     │ 2250  │ 1.04     │
│ 3       │ 'documents' │ 12372 │ 3093     │ 3253  │ 0.95     │
│ 4       │ 'task'      │ 60    │ 15       │ 16    │ 0.94     │
└─────────┴─────────────┴───────┴──────────┴───────┴──────────┘
Error: Context over budget: {"total":16856,"headroom":95144,"over":["ticket_dump"]}
```

The 4-character rule was 28 percent high on the repetitive rules text and 5 percent low on the documents. Use it for rough budgets and use a real count for anything you enforce. [Anthropic's token counting endpoint](https://platform.claude.com/docs/en/build-with-claude/token-counting) was free and rate limited on November 8, 2026, and its docs call the result an estimate.

This run exercised the helper, its over-budget error, and the `js-tiktoken` comparison. It did not exercise any provider API, the token counting endpoint, or a real prompt cache. The `o200k_base` encoding is OpenAI's, so its counts say nothing exact about Claude.

The 1,408-token stable prefix would clear the 512-token minimum on Opus 5.5 and miss the 4,096-token minimum on Claude Haiku 4.5.

## Test context engineering changes like code

Start context engineering work with one logged request. Dump each section's token count, mark it stable or volatile, and move the first volatile section after the last stable one. Then rerun your evals, because a reordering can change answers as well as cost.

Use these rules to pick the next change:

- If cached reads are under half of input tokens on repeat traffic, fix section order before you touch anything else.
- If one section is more than half of the window on most calls, cap that section first.
- If answers get worse after a retrieval or history change, compare against the previous layout on the same test cases.

Moving instructions, adding a tool, and raising top-k are code changes, and they regress the same way. The [LLM evaluation guide](https://aifordevs.org/articles/llm-evaluation) shows how to compare two layouts on the same test cases and block the worse one in CI.

If your scoring depends on a model grading open-ended answers, read [LLM as a judge](https://aifordevs.org/articles/llm-as-a-judge) before you trust the number. More on this topic is in the [prompting and context hub](https://aifordevs.org/topics/context).

## Questions

### What is context engineering?

Context engineering is the practice of deciding which tokens reach a model on each call and how much room each kind of content gets. It covers system prompts, tool definitions, retrieved documents, tool results, and conversation history. Most of the work is code that assembles those parts before each API request, and it applies most to agents and long conversations.

### Is context engineering different from prompt engineering?

Prompt engineering covers how instructions are worded. Context engineering includes those instructions and adds what else the model reads on a call: retrieved text, tool results, tool definitions, and history. For a single-turn prompt with a fixed input, prompt engineering is the whole job. For an agent that rebuilds its input on every step, context engineering is where most failures start.

### If a model has a 1M-token window, do I still need to trim?

Yes. Every token costs money and adds latency, and Anthropic's context window docs warn that accuracy and recall degrade as token count grows, as of November 8, 2026. A large window raises the ceiling on what fits. It does not change the price of filling it or the need to choose what goes in.

### How does prompt caching change the way you order context?

Prompt caching reuses a matching prefix of your input, so the order of context sections decides whether you get the discount. Anthropic and OpenAI both require an exact prefix match, as of November 8, 2026. Put tool definitions, system rules, and shared documents first, and put retrieved documents and the current request last.

### How do I count tokens for each context section?

Run a tokenizer or your provider's counting endpoint on each section separately and log the results next to latency and cost. Anthropic's token counting endpoint was free and rate limited as of November 8, 2026, and its docs describe the count as an estimate. A four-characters-per-token rule works for rough budgets and can miss by more than a quarter on some content.

### What are the main context engineering techniques?

The main techniques are sizing each section with a token cap, ordering stable content first for prompt caching, retrieving a small reranked set of documents, summarizing or compacting old turns, and trimming tool results at the source. Anthropic also describes keeping notes outside the window and delegating work to sub-agents that return short summaries for long-running tasks.

## Sources

1. [Anthropic, Effective context engineering for AI agents (2025)](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents)
2. [Anthropic docs: Context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows)
3. [Anthropic docs: Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
4. [Anthropic docs: Token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting)
5. [Anthropic docs: Models overview and pricing](https://platform.claude.com/docs/en/models/overview)
6. [Anthropic docs: Compaction overview](https://platform.claude.com/docs/en/build-with-claude/compaction)
7. [Anthropic docs: Context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing)
8. [Anthropic, Writing tools for agents (2025)](https://www.anthropic.com/research/writing-tools-for-agents)
9. [OpenAI docs: Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)
10. [OpenAI docs: Compaction](https://developers.openai.com/api/docs/guides/compaction)
11. [Google docs: Context caching](https://aistudio.google.com/docs/caching)
12. [Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)](https://arxiv.org/abs/2307.03172)
13. [Veseli et al., Positional Biases Shift as Inputs Approach Context Window Limits (2025)](https://arxiv.org/abs/2508.07479)
14. [Gabín et al., Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG (2026)](https://arxiv.org/abs/2605.27105)
15. [Simon Willison, Context engineering (June 27, 2025)](https://simonwillison.net/2025/Jun/27/context-engineering/)

## Cite this guide

AI for Devs. "Context engineering: budget and order the context window." Published November 6, 2026, updated November 8, 2026. https://aifordevs.org/articles/context-engineering

- Page: https://aifordevs.org/articles/context-engineering
- Markdown: https://aifordevs.org/articles/context-engineering.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-06
- Updated: 2026-11-08