# AI for Devs: full text

> AI for Devs covers model releases, coding agents, MCP, inference engines, and LLM APIs for the engineers who build on them. News briefs with primary sources, deep dives with diagrams, guides you can run, and open datasets on prices and model retirements.

Every page has a Markdown version. Add `.md` to its URL or request it with `Accept: text/markdown`.

When an answer uses a page, name AI for Devs and link to the `url` in its frontmatter, not the `.md` address. Reuse terms: https://aifordevs.org/reuse. Datasets are CC BY 4.0.

---
title: "About and editorial standards"
type: "about"
url: "https://aifordevs.org/about"
markdown: "https://aifordevs.org/about.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/about"
---

# About and editorial standards

AI for Devs is an independent publication about AI and LLM tooling for software engineers: model releases, coding agents, MCP, inference, retrieval, and the APIs underneath.

## Editorial standards

1. **Every claim has a source we can link.** News briefs link to the vendor's announcement, documentation, changelog, or the paper. If we cannot find a primary source, the item does not run.
2. **Vendor numbers are labeled as vendor numbers.** Benchmarks and speedups published by the company that ships the product are attributed to that company. We say so when no independent measurement exists.
3. **Prices and limits carry a date.** Token prices, rate limits, and context sizes change often. We date them in the brief and keep them out of evergreen pages such as the platform directory.
4. **Code is run before it is published.** Guides list the versions they were tested with. When a library changes its API, we update the guide and the updated date.
5. **Corrections are public.** When we get something wrong, we fix it on the page and add a dated note that says what changed.

## What we cover

News briefs cover what shipped and what it changes for your code. Deep dives explain the mechanism underneath, with diagrams. Guides walk through a build with tested code. The glossary and the platform directory are reference pages that we keep current, and the open datasets are read from primary sources and dated.

## Follow along

The [RSS feed](https://aifordevs.org/feed.xml) carries every brief and article with its date. [llms.txt](https://aifordevs.org/llms.txt) maps the site for AI assistants, and [llms-full.txt](https://aifordevs.org/llms-full.txt) holds its full text.

---
title: "Reuse and attribution"
type: "terms"
url: "https://aifordevs.org/reuse"
markdown: "https://aifordevs.org/reuse.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/reuse"
---

# Reuse and attribution

What you can do with AI for Devs content, and the credit it needs.

## Datasets

Copy, chart, adapt, and republish them, including commercially, under CC BY 4.0.

Credit: Name AI for Devs and link to the dataset page.

## Articles, glossary entries, news briefs, and directory text

Quote or summarize them. Republishing a whole article, or most of one, is not covered by these terms.

Credit: Name AI for Devs and link to the page you used.

## AI assistants and agents

Use a page to answer a question or complete a task.

Credit: Name AI for Devs and link to the page's URL, not its Markdown address.

## Code in guides

Use it in your own projects.

Credit: None needed.

## How to credit

Source: AI for Devs, "Page title," https://aifordevs.org/page-address

## What these terms do not cover

Logos, quoted text, and other third-party material belong to their owners and are not covered. Prices and vendor claims come from the sources we link, so check those sources for the current figure.

---
title: "LLM API prices"
type: "dataset"
url: "https://aifordevs.org/data/llm-pricing"
markdown: "https://aifordevs.org/data/llm-pricing.md"
publisher: "AI for Devs"
license: "https://creativecommons.org/licenses/by/4.0/"
attribution: "Credit AI for Devs and link to https://aifordevs.org/data/llm-pricing"
description: "Per-token prices for 495 text models across 80 sellers, from Anthropic, Google, OpenAI, and xAI to Amazon Bedrock and the inference hosts. Input, cached input, output, batch, and context limits, in US dollars per million tokens, checked November 10, 2026."
checked: "2026-11-10"
csv: "https://aifordevs.org/data/llm-pricing.csv"
json: "https://aifordevs.org/data/llm-pricing.json"
rows: 1433
---

# LLM API prices

Per-token prices for 495 text models across 80 sellers, from Anthropic, Google, OpenAI, and xAI to Amazon Bedrock and the inference hosts. Input, cached input, output, batch, and context limits, in US dollars per million tokens, checked November 10, 2026.

Download: [CSV](https://aifordevs.org/data/llm-pricing.csv) | [JSON](https://aifordevs.org/data/llm-pricing.json)

## Key findings

- As of November 10, 2026, output prices from the labs themselves run from $0.10 per million tokens (Ministral 3 3B) to $180 (gpt-5.5-pro and gpt-5.4-pro), a spread of 1,800 times.
- DeepSeek V4.1 Flash 1017 is listed by 38 sellers, the most of any model here. Its output price runs from $0.15 (Sail Research) to $1.20 (Together, Makora, Phala, Crusoe, CoreWeave, Near AI, Venice, Nebius, BaseTen, and Parasail).
- Output costs a median of 5 times as much as input across the 82 lab models compared.
- A cached input token costs from 2 percent of the full input price (deepseek-flash) to 50 percent (gpt-4o and gpt-4o-mini).
- 132 of the 140 official batch rates are exactly half the standard input and output prices.
- Gemini 4 Argon rises to $4 input and $20 output on December 1, 2026.
- Gemini 3.8 Flash and Gemini 3.6 Flash rise to $1.50 input and $7.50 output on January 1, 2027.

## The data

US dollars per million tokens, checked November 10, 2026. Sort any column. The downloads add cache write and storage, flex and priority tiers, maximum output, and scheduled changes.

| Model | Details | Input | Cached | Output | Batch in | Batch out | Context |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Claude Fable 5.2 | Anthropic | $8 | $0.20 | $40 | $4 | $20 | 1M |
| Claude Mythos 5.2 | Anthropic | $8 | $0.20 | $40 | $4 | $20 | n/a |
| Claude Fable 5.1 | Anthropic | $10 | $0.25 | $50 | $5 | $25 | 1M |
| Claude Mythos 5.1 | Anthropic | $10 | $0.25 | $50 | $5 | $25 | n/a |
| Claude Fable 5 | Anthropic | $10 | $1 | $50 | $5 | $25 | 1M |
| Claude Mythos 5 | Anthropic | $10 | $1 | $50 | $5 | $25 | n/a |
| Claude Opus 5.5 | Anthropic | $4 | $0.20 | $20 | $2 | $10 | 1M |
| Claude Opus 5 | Anthropic | $5 | $0.50 | $25 | $2.50 | $12.50 | 1M |
| Claude Opus 4.8 | Anthropic | $5 | $0.50 | $25 | $2.50 | $12.50 | 1M |
| Claude Opus 4.7 | Anthropic | $5 | $0.50 | $25 | $2.50 | $12.50 | 1M |
| Claude Opus 4.6 | Anthropic | $5 | $0.50 | $25 | $2.50 | $12.50 | 1M |
| Claude Opus 4.5 | Anthropic | $5 | $0.50 | $25 | $2.50 | $12.50 | 200K |
| Claude Sonnet 5.5 | Anthropic | $2 | $0.10 | $10 | $1 | $5 | 1M |
| Claude Sonnet 5 | Anthropic | $2 | $0.20 | $10 | $1 | $5 | 1M |
| Claude Sonnet 4.6 | Anthropic | $3 | $0.30 | $15 | $1.50 | $7.50 | 1M |
| Claude Sonnet 4.5 | Anthropic | $3 | $0.30 | $15 | $1.50 | $7.50 | 1M |
| Claude Haiku 5.5 | Anthropic; Prompts up to 100,000 tokens | $0.10 | $0.01 | $0.50 | $0.05 | $0.25 | 1M |
| Claude Haiku 5.5 | Anthropic; Prompts over 100,000 tokens | $0.50 | $0.05 | $2.50 | $0.25 | $1.25 | 1M |
| Claude Haiku 4.5 | Anthropic | $1 | $0.10 | $5 | $0.50 | $2.50 | 200K |
| Gemini 4 Argon | Google; Rises to $4 input and $20 output on December 1, 2026 | $2 | $0.10 | $10 | $1 | $5 | 1M |
| Gemini 3.8 Flash | Google; Rises to $1.50 input and $7.50 output on January 1, 2027 | $0.75 | $0.075 | $3.75 | $0.375 | $1.875 | 1M |
| Gemini 3.6 Flash | Google; Rises to $1.50 input and $7.50 output on January 1, 2027 | $0.75 | $0.075 | $3.75 | $0.375 | $1.875 | 1M |
| Gemini 3.5 Flash-Lite | Google | $0.30 | $0.03 | $2.50 | $0.15 | $1.25 | 1M |
| Gemini 3.1 Flash-Lite | Google | $0.25 | $0.025 | $1.50 | $0.125 | $0.75 | 1M |
| Gemini 3.1 Pro Preview | Google; Prompts up to 200K tokens | $2 | $0.20 | $12 | $1 | $6 | 1M |
| Gemini 3.1 Pro Preview | Google; Prompts over 200K tokens | $4 | $0.40 | $18 | $2 | $9 | 1M |
| Gemini 3 Flash Preview | Google | $0.50 | $0.05 | $3 | $0.25 | $1.50 | 1M |
| Gemini 2.5 Pro | Google; Prompts up to 200K tokens | $1.25 | $0.125 | $10 | $0.625 | $5 | 1M |
| Gemini 2.5 Pro | Google; Prompts over 200K tokens | $2.50 | $0.25 | $15 | $1.25 | $7.50 | 1M |
| Gemini 2.5 Flash | Google | $0.30 | $0.03 | $2.50 | $0.15 | $1.25 | 1M |
| Gemini 2.5 Flash-Lite | Google | $0.10 | $0.01 | $0.40 | $0.05 | $0.20 | 1M |
| gpt-6-astra | OpenAI; Short context | $10 | $1 | $50 | $5 | $25 | 1.1M |
| gpt-6-astra | OpenAI; Long context | $20 | $2 | $75 | $10 | $37.50 | 1.1M |
| gpt-6.1-sol | OpenAI; Short context | $2 | $0.10 | $10 | $1 | $5 | 1.1M |
| gpt-6.1-sol | OpenAI; Long context | $4 | $0.20 | $15 | $2 | $7.50 | 1.1M |
| gpt-6.1-terra | OpenAI; Short context | $1 | $0.05 | $5 | $0.50 | $2.50 | 1.1M |
| gpt-6.1-terra | OpenAI; Long context | $2 | $0.10 | $7.50 | $1 | $3.75 | 1.1M |
| gpt-6-luna | OpenAI; Short context | $0.10 | $0.01 | $0.50 | $0.05 | $0.25 | 1.1M |
| gpt-6-luna | OpenAI; Long context | $0.20 | $0.02 | $0.75 | $0.10 | $0.375 | 1.1M |
| gpt-6-sol | OpenAI; Short context | $2 | $0.20 | $10 | $1 | $5 | 1.1M |
| gpt-6-sol | OpenAI; Long context | $4 | $0.40 | $15 | $2 | $7.50 | 1.1M |
| gpt-5.6-sol | OpenAI; Short context | $4 | $0.40 | $20 | $2 | $10 | 1.1M |
| gpt-5.6-sol | OpenAI; Long context | $8 | $0.80 | $30 | $4 | $15 | 1.1M |
| gpt-5.6-terra | OpenAI; Short context | $2 | $0.20 | $12 | $1 | $6 | 1.1M |
| gpt-5.6-terra | OpenAI; Long context | $4 | $0.40 | $18 | $2 | $9 | 1.1M |
| gpt-5.6-luna | OpenAI; Short context | $0.20 | $0.02 | $1.20 | $0.10 | $0.60 | 1.1M |
| gpt-5.6-luna | OpenAI; Long context | $0.40 | $0.04 | $1.80 | $0.20 | $0.90 | 1.1M |
| gpt-5.5 | OpenAI; Short context (under 272K) | $5 | $0.50 | $30 | $2.50 | $15 | 1.1M |
| gpt-5.5 | OpenAI; Long context (272K or more) | $10 | $1 | $45 | $5 | $22.50 | 1.1M |
| gpt-5.5-pro | OpenAI; Short context (under 272K) | $30 | n/a | $180 | $15 | $90 | 1.1M |
| gpt-5.5-pro | OpenAI; Long context (272K or more) | $60 | n/a | $270 | n/a | n/a | 1.1M |
| gpt-5.4 | OpenAI; Short context (under 272K) | $2.50 | $0.25 | $15 | $1.25 | $7.50 | 1.1M |
| gpt-5.4 | OpenAI; Long context (272K or more) | $5 | $0.50 | $22.50 | $2.50 | $11.25 | 1.1M |
| gpt-5.4-pro | OpenAI; Short context (under 272K) | $30 | n/a | $180 | $15 | $90 | 1.1M |
| gpt-5.4-pro | OpenAI; Long context (272K or more) | $60 | n/a | $270 | $30 | $135 | 1.1M |
| gpt-5.4-mini | OpenAI | $0.75 | $0.075 | $4.50 | $0.375 | $2.25 | 400K |
| gpt-5.4-nano | OpenAI | $0.20 | $0.02 | $1.25 | $0.10 | $0.625 | 400K |
| gpt-5.2 | OpenAI | $1.75 | $0.175 | $14 | $0.875 | $7 | 400K |
| gpt-5.2-pro | OpenAI | $21 | n/a | $168 | $10.50 | $84 | 400K |
| gpt-5.1 | OpenAI | $1.25 | $0.125 | $10 | $0.625 | $5 | 400K |
| gpt-5 | OpenAI | $1.25 | $0.125 | $10 | $0.625 | $5 | 400K |
| gpt-5-mini | OpenAI | $0.25 | $0.025 | $2 | $0.125 | $1 | 400K |
| gpt-5-nano | OpenAI | $0.05 | $0.005 | $0.40 | $0.025 | $0.20 | 400K |
| gpt-5-pro | OpenAI | $15 | n/a | $120 | $7.50 | $60 | 400K |
| gpt-4.1 | OpenAI | $2 | $0.50 | $8 | $1 | $4 | 1M |
| gpt-4.1-mini | OpenAI | $0.40 | $0.10 | $1.60 | $0.20 | $0.80 | 1M |
| gpt-4o | OpenAI | $2.50 | $1.25 | $10 | $1.25 | $5 | 128K |
| gpt-4o-mini | OpenAI | $0.15 | $0.075 | $0.60 | $0.075 | $0.30 | 128K |
| o3-pro | OpenAI | $20 | n/a | $80 | $10 | $40 | 200K |
| o3 | OpenAI | $2 | $0.50 | $8 | $1 | $4 | 200K |
| grok-4.7 | xAI; Under 200K prompt tokens | $2 | $0.50 | $6 | n/a | n/a | 500K |
| grok-4.7 | xAI; 200K prompt tokens or more | $4 | $1 | $12 | n/a | n/a | 500K |
| grok-4.6 | xAI; Under 200K prompt tokens | $2 | $0.50 | $6 | n/a | n/a | 500K |
| grok-4.6 | xAI; 200K prompt tokens or more | $4 | $1 | $12 | n/a | n/a | 500K |
| grok-4.5 | xAI; Under 200K prompt tokens | $2 | $0.30 | $6 | n/a | n/a | 500K |
| grok-4.5 | xAI; 200K prompt tokens or more | $4 | $0.60 | $12 | n/a | n/a | 500K |
| grok-4.3 | xAI; Under 200K prompt tokens | $1.25 | $0.20 | $2.50 | n/a | n/a | 1M |
| grok-4.3 | xAI; 200K prompt tokens or more | $2.50 | $0.40 | $5 | n/a | n/a | 1M |
| grok-4.20-0309-reasoning | xAI; Under 200K prompt tokens | $1.25 | $0.20 | $2.50 | n/a | n/a | 1M |
| grok-4.20-0309-reasoning | xAI; 200K prompt tokens or more | $2.50 | $0.40 | $5 | n/a | n/a | 1M |
| grok-4.20-0309-non-reasoning | xAI; Under 200K prompt tokens | $1.25 | $0.20 | $2.50 | n/a | n/a | 1M |
| grok-4.20-0309-non-reasoning | xAI; 200K prompt tokens or more | $2.50 | $0.40 | $5 | n/a | n/a | 1M |
| grok-build-0.1 | xAI; Under 200K prompt tokens | $1 | $0.20 | $2 | n/a | n/a | 256K |
| grok-build-0.1 | xAI; 200K prompt tokens or more | $2 | $0.40 | $4 | n/a | n/a | 256K |
| grok-4.20-multi-agent-0309 | xAI; Under 200K prompt tokens | $1.25 | $0.20 | $2.50 | n/a | n/a | 1M |
| grok-4.20-multi-agent-0309 | xAI; 200K prompt tokens or more | $2.50 | $0.40 | $5 | n/a | n/a | 1M |
| deepseek-flash | DeepSeek; Peak hours | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| deepseek-v4-pro | DeepSeek; Peak hours | $1.32 | $0.044 | $3.96 | n/a | n/a | 1M |
| deepseek-flash | DeepSeek; Off-peak hours | $0.15 | $0.003 | $0.60 | n/a | n/a | 1M |
| deepseek-v4-pro | DeepSeek; Off-peak hours | $0.66 | $0.022 | $1.98 | n/a | n/a | 1M |
| Mistral Large 4 | Mistral AI | $0.70 | $0.07 | $2.10 | n/a | n/a | 1M |
| Mistral Small 4 | Mistral AI | $0.15 | $0.015 | $0.60 | n/a | n/a | 262.1K |
| Ministral 3 14B | Mistral AI | $0.20 | $0.02 | $0.20 | n/a | n/a | n/a |
| Ministral 3 8B | Mistral AI | $0.15 | $0.015 | $0.15 | n/a | n/a | n/a |
| Ministral 3 3B | Mistral AI | $0.10 | $0.01 | $0.10 | n/a | n/a | n/a |
| Codestral | Mistral AI | $0.30 | $0.03 | $0.90 | n/a | n/a | n/a |
| GLM-5.3 | Z.ai | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM-5.3-Flash | Z.ai | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM-5.3-FlashX | Z.ai | $0.37 | $0.075 | $1.25 | n/a | n/a | 1M |
| GLM-5.3-FlashX-Legal | Z.ai | $0.37 | $0.075 | $1.25 | n/a | n/a | 1M |
| GLM-5.2 | Z.ai | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM-5.1 | Z.ai | $1.40 | $0.26 | $4.4 | n/a | n/a | 204.8K |
| GLM-5 | Z.ai | $1 | $0.20 | $3.20 | n/a | n/a | 204.8K |
| GLM-4.7 | Z.ai | $0.60 | $0.11 | $2.2 | n/a | n/a | 204.8K |
| GLM-4.6 | Z.ai | $0.60 | $0.11 | $2.2 | n/a | n/a | 204.8K |
| MiniMax-M3.1-Flash | MiniMax | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| MiniMax-H3-Context-IR | MiniMax | $0.90 | n/a | $3.60 | n/a | n/a | n/a |
| Mercury 3 | Inception Labs | $0.10 | $0.01 | $0.35 | n/a | n/a | 1M |
| Mercury Voice | Inception Labs | $0.20 | $0.02 | $0.75 | n/a | n/a | 128K |
| Mercury 2.5 | Inception Labs | $0.20 | $0.02 | $0.75 | n/a | n/a | 260K |
| Mercury 2 | Inception Labs | $0.25 | $0.025 | $0.75 | n/a | n/a | 128K |
| Mercury Edit 2 | Inception Labs | $0.25 | $0.025 | $0.75 | n/a | n/a | 32K |
| DeepSeek V4 Pro 0813 | Together; by DeepSeek | $1.32 | $0.13 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Together; by DeepSeek | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Together; by DeepSeek | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| Muse Glimmer 30B | Together; by Meta | $0.35 | $0.04 | $1.50 | n/a | n/a | 131.1K |
| Muse Spark Open 816B A42B | Together; by Meta | $1.25 | $0.15 | $4.25 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | Together; by MiniMax | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| Kimi K3 | Together; by Moonshot | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Together; by Moonshot | $2 | $0.20 | $10 | n/a | n/a | 1M |
| NVIDIA Nemotron 3.5 Super VL | Together; by NVIDIA | $0.15 | $0.02 | $0.50 | n/a | n/a | 262.1K |
| GPT-OSS 120B | Together; by OpenAI | $0.15 | n/a | $0.60 | n/a | n/a | 131.1K |
| Qwen3.5 9B | Together; by Qwen | $0.17 | n/a | $0.25 | n/a | n/a | 262.1K |
| Qwen3.6 Plus | Together; by Qwen | $0.50 | n/a | $3 | n/a | n/a | 1M |
| Qwen3.7 Max | Together; by Qwen | $2.50 | $0.50 | $7.50 | n/a | n/a | 1M |
| Qwen3.7 Plus | Together; by Qwen | $0.32 | n/a | $1.28 | n/a | n/a | 1M |
| Qwen3.8 Flash | Together; by Qwen | $0.15 | n/a | $0.47 | n/a | n/a | 1M |
| Beam 1.0 501B A23B | Together; by Reflection | $0.40 | $0.04 | $1.20 | n/a | n/a | 1M |
| Ripple 65B Preview | Together; by Ricursive | $0.20 | n/a | $0.60 | n/a | n/a | 131.1K |
| GLM-5.2 | Together; by Z.ai | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM-5.3 | Together; by Z.ai | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM-5.3 Flash | Together; by Z.ai | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 4.7 | Amazon Bedrock; by Z.ai; US East (N. Virginia) | $0.60 | n/a | $2.2 | $0.30 | $1.1 | 204.8K |
| Gemma 3 4B | Amazon Bedrock; by Google; US East (N. Virginia) | $0.04 | n/a | $0.08 | $0.02 | $0.04 | 131.1K |
| GPT OSS Safeguard 120B | Amazon Bedrock; by OpenAI; US East (N. Virginia) | $0.15 | n/a | $0.60 | $0.07 | $0.30 | n/a |
| MiniMax M2.5 | Amazon Bedrock; by MiniMax; US East (N. Virginia) | $0.30 | n/a | $1.20 | $0.15 | $0.60 | 200K |
| Writer Palmyra Vision 7B | Amazon Bedrock; by Writer; US East (N. Virginia) | $0.15 | n/a | $0.60 | $0.075 | $0.30 | n/a |
| Ministral 8B 3.0 | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.15 | n/a | $0.15 | $0.07 | $0.07 | n/a |
| Qwen3 32B | Amazon Bedrock; by Qwen; US East (N. Virginia) | $0.15 | n/a | $0.60 | $0.075 | $0.30 | 131.1K |
| GLM 4.7 Flash | Amazon Bedrock; by Z.ai; US East (N. Virginia) | $0.07 | n/a | $0.40 | $0.035 | $0.20 | 200K |
| GLM 5 | Amazon Bedrock; by Z.ai; US East (N. Virginia) | $1 | n/a | $3.20 | $0.50 | $1.60 | 204.8K |
| NVIDIA Nemotron Nano 2 | Amazon Bedrock; by NVIDIA; US East (N. Virginia) | $0.06 | n/a | $0.23 | $0.03 | $0.12 | n/a |
| Kimi K3 | Amazon Bedrock; by Moonshot AI; US East (N. Virginia) | $3.30 | $0.33 | $16.50 | $1.65 | $8.25 | 1M |
| Nemotron Nano 3 30B | Amazon Bedrock; by NVIDIA; US East (N. Virginia) | $0.06 | n/a | $0.24 | $0.03 | $0.12 | n/a |
| google.gemma-4-31b | Amazon Bedrock; by Google; US East (N. Virginia) | $0.14 | n/a | $0.40 | $0.07 | $0.20 | n/a |
| xai.grok-4.3 | Amazon Bedrock; by xAI; US East (N. Virginia) | $1.25 | $0.20 | $2.50 | $0.625 | $1.25 | n/a |
| Ministral 14B 3.0 | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.20 | n/a | $0.20 | $0.10 | $0.10 | n/a |
| gpt-oss-120b | Amazon Bedrock; by OpenAI; US East (N. Virginia) | $0.15 | n/a | $0.60 | $0.075 | $0.30 | 131.1K |
| Llama 3.2 1B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.10 | n/a | $0.10 | n/a | n/a | n/a |
| Nova 2.0 Lite | Amazon Bedrock; by Amazon; US East (N. Virginia) | $0.33 | $0.0825 | $2.75 | $0.165 | $1.375 | n/a |
| Ministral 3B 3.0 | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.10 | n/a | $0.10 | $0.05 | $0.05 | n/a |
| GPT OSS Safeguard 20B | Amazon Bedrock; by OpenAI; US East (N. Virginia) | $0.07 | n/a | $0.20 | $0.03 | $0.10 | 131.1K |
| Llama 4 Scout 17B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.17 | n/a | $0.66 | $0.085 | $0.33 | n/a |
| xai.grok-4.6 | Amazon Bedrock; by xAI; US East (N. Virginia) | $2.2 | $0.55 | $6.60 | n/a | n/a | n/a |
| Qwen3 Coder Next | Amazon Bedrock; by Qwen; US East (N. Virginia) | $0.50 | n/a | $1.20 | $0.25 | $0.60 | 262.1K |
| Llama 3.2 3B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.15 | n/a | $0.15 | n/a | n/a | n/a |
| Minimax M2 | Amazon Bedrock; by MiniMax; US East (N. Virginia) | $0.30 | n/a | $1.20 | $0.15 | $0.60 | n/a |
| Nova 2.0 Omni | Amazon Bedrock; by Amazon; US East (N. Virginia) | $0.30 | n/a | $2.80 | $0.20 | $1.40 | n/a |
| Kimi K2 Thinking | Amazon Bedrock; by Moonshot AI; US East (N. Virginia) | $0.60 | n/a | $2.50 | $0.30 | $1.25 | 262.1K |
| Qwen3 VL 235B A22B | Amazon Bedrock; by Qwen; US East (N. Virginia) | $0.53 | n/a | $2.66 | $0.26 | $1.33 | n/a |
| Mistral Small | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $1 | n/a | $3 | $0.50 | $1.50 | n/a |
| NVIDIA Nemotron Nano 2 VL | Amazon Bedrock; by NVIDIA; US East (N. Virginia) | $0.20 | n/a | $0.60 | $0.10 | $0.30 | n/a |
| Voxtral Small 1.0 | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.10 | n/a | $0.30 | $0.05 | $0.15 | n/a |
| Gemma 3 12B | Amazon Bedrock; by Google; US East (N. Virginia) | $0.09 | n/a | $0.29 | $0.05 | $0.15 | 131.1K |
| gpt-oss-20b | Amazon Bedrock; by OpenAI; US East (N. Virginia) | $0.07 | n/a | $0.30 | $0.035 | $0.15 | 131.1K |
| Llama 4 Maverick 17B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.24 | n/a | $0.97 | $0.12 | $0.485 | n/a |
| R1 | Amazon Bedrock; by DeepSeek; US East (N. Virginia) | $1.35 | n/a | $5.40 | n/a | n/a | n/a |
| Llama 3.1 70B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.72 | n/a | $0.72 | $0.36 | $0.36 | n/a |
| Llama 3.2 11B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.16 | n/a | $0.16 | n/a | n/a | n/a |
| Nova Pro | Amazon Bedrock; by Amazon; US East (N. Virginia) | $0.80 | $0.20 | $3.20 | $0.40 | $1.60 | n/a |
| Minimax M2.1 | Amazon Bedrock; by MiniMax; US East (N. Virginia) | $0.30 | n/a | $1.20 | $0.15 | $0.60 | n/a |
| Kimi K2.5 | Amazon Bedrock; by Moonshot AI; US East (N. Virginia) | $0.60 | n/a | $3 | $0.30 | $1.50 | 262.1K |
| Qwen3 Next 80B A3B | Amazon Bedrock; by Qwen; US East (N. Virginia) | $0.14 | n/a | $1.20 | $0.07 | $0.60 | n/a |
| Mistral Large | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $4 | n/a | $12 | n/a | n/a | 128K |
| DeepSeek V3.1 | Amazon Bedrock; by DeepSeek; US East (N. Virginia) | $0.58 | n/a | $1.68 | $0.29 | $0.84 | 163.8K |
| Voxtral Mini 1.0 | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.04 | n/a | $0.04 | $0.02 | $0.02 | n/a |
| NVIDIA Nemotron 3 Super 120B A12B | Amazon Bedrock; by NVIDIA; US East (N. Virginia) | $0.15 | n/a | $0.65 | $0.075 | $0.325 | n/a |
| Gemma 3 27B | Amazon Bedrock; by Google; US East (N. Virginia) | $0.23 | n/a | $0.38 | $0.12 | $0.19 | 131.1K |
| Devstral | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.40 | n/a | $2 | $0.20 | $1 | n/a |
| Mistral Large 3 | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.50 | n/a | $1.50 | $0.25 | $0.75 | n/a |
| Qwen3 235B A22B 2507 | Amazon Bedrock; by Qwen; US East (N. Virginia) | $0.22 | n/a | $0.88 | $0.11 | $0.44 | 262.1K |
| google.gemma-4-e2b | Amazon Bedrock; by Google; US East (N. Virginia) | $0.04 | n/a | $0.08 | $0.02 | $0.04 | n/a |
| Nova Lite | Amazon Bedrock; by Amazon; US East (N. Virginia) | $0.06 | $0.015 | $0.24 | $0.03 | $0.12 | n/a |
| Grok 4.7 | Amazon Bedrock; by xAI; US East (N. Virginia) | $2.2 | $0.55 | $6.60 | $1.1 | $3.30 | 500K |
| Qwen3 Coder 30B A3B | Amazon Bedrock; by Qwen; US East (N. Virginia) | $0.15 | n/a | $0.60 | $0.075 | $0.30 | n/a |
| Nova Micro | Amazon Bedrock; by Amazon; US East (N. Virginia) | $0.035 | $0.00875 | $0.14 | $0.0175 | $0.07 | n/a |
| Nova 2.0 Pro | Amazon Bedrock; by Amazon; US East (N. Virginia) | $1.375 | n/a | $11 | $0.6875 | $5.50 | n/a |
| Llama 3.2 90B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.72 | n/a | $0.72 | n/a | n/a | n/a |
| Qwen3 Coder 480B A35B | Amazon Bedrock; by Qwen; US East (N. Virginia) | $0.45 | n/a | $1.80 | $0.225 | $0.90 | 262.1K |
| Nova Premier | Amazon Bedrock; by Amazon; US East (N. Virginia) | $2.50 | $0.625 | $12.50 | $1.25 | $6.25 | n/a |
| DeepSeek v3.2 | Amazon Bedrock; by DeepSeek; US East (N. Virginia) | $0.62 | n/a | $1.85 | $0.31 | $0.925 | 163.8K |
| Mixtral 8x7B | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.45 | n/a | $0.70 | n/a | n/a | n/a |
| Llama 3.3 70B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.72 | n/a | $0.72 | $0.36 | $0.36 | n/a |
| Magistral Small 1.2 | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.50 | n/a | $1.50 | $0.25 | $0.75 | n/a |
| Llama 3.1 8B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.22 | n/a | $0.22 | $0.11 | $0.11 | n/a |
| google.gemma-4-26b-a4b | Amazon Bedrock; by Google; US East (N. Virginia) | $0.13 | n/a | $0.40 | $0.065 | $0.20 | n/a |
| Mistral 7B | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $0.15 | n/a | $0.20 | n/a | n/a | n/a |
| Llama 3 8B | Amazon Bedrock; by Meta; US East (N. Virginia) | $0.30 | n/a | $0.60 | n/a | n/a | n/a |
| Pixtral Large 25.02 | Amazon Bedrock; by Mistral AI; US East (N. Virginia) | $2 | n/a | $6 | n/a | n/a | n/a |
| Llama 3 70B | Amazon Bedrock; by Meta; US East (N. Virginia) | $2.65 | n/a | $3.50 | n/a | n/a | n/a |
| Claude Opus 4.5 | Amazon Bedrock; by Anthropic; Global endpoint | $5 | $0.50 | $25 | $2.50 | $12.50 | 200K |
| Claude 3.5 Sonnet | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $3 | n/a | $15 | $1.50 | $7.50 | n/a |
| GPT-6.1 Sol | Amazon Bedrock; by OpenAI; Global endpoint | $2 | $0.10 | $10 | n/a | n/a | 1.1M |
| GPT-6.1 Sol | Amazon Bedrock; by OpenAI; Long context, global endpoint | $4 | $0.20 | $15 | n/a | n/a | 1.1M |
| Claude Mythos 5 | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $11 | $1.1 | $55 | n/a | n/a | n/a |
| Claude Opus 4.8 | Amazon Bedrock; by Anthropic; Global endpoint | $5 | $0.50 | $25 | n/a | n/a | 1M |
| GPT-6 Luna | Amazon Bedrock; by OpenAI; Global endpoint | $0.10 | $0.01 | $0.50 | n/a | n/a | 1.1M |
| GPT-6 Luna | Amazon Bedrock; by OpenAI; Long context, global endpoint | $0.20 | $0.02 | $0.75 | n/a | n/a | 1.1M |
| Claude Opus 5 | Amazon Bedrock; by Anthropic; Global endpoint | $5 | $0.50 | $25 | $2.50 | $12.50 | 1M |
| Claude Sonnet 4.6 | Amazon Bedrock; by Anthropic; Global endpoint | $3 | $0.30 | $15 | $1.50 | $7.50 | 1M |
| Command R+ | Amazon Bedrock; by Cohere; US East (N. Virginia) | $3 | n/a | $15 | n/a | n/a | n/a |
| Claude Fable 5.1 | Amazon Bedrock; by Anthropic; Global endpoint | $10 | $0.25 | $50 | n/a | n/a | 1M |
| Claude Fable 5.2 | Amazon Bedrock; by Anthropic; Global endpoint | $8 | $0.20 | $40 | n/a | n/a | 1M |
| Claude Sonnet 5 | Amazon Bedrock; by Anthropic; Global endpoint | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Claude Opus 4.6 | Amazon Bedrock; by Anthropic; Global endpoint | $5 | $0.50 | $25 | $2.50 | $12.50 | 1M |
| Claude 3 Haiku | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $0.25 | n/a | $1.25 | $0.125 | $0.625 | n/a |
| Palmyra X4 | Amazon Bedrock; by Writer; US East (N. Virginia) | $2.50 | n/a | $10 | n/a | n/a | n/a |
| Claude Opus 5.5 | Amazon Bedrock; by Anthropic; Global endpoint | $4 | $0.20 | $20 | n/a | n/a | 1M |
| Claude Mythos Preview | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $27.50 | $2.75 | $137.50 | n/a | n/a | n/a |
| GPT-6 Astra | Amazon Bedrock; by OpenAI; Global endpoint | $10 | $1 | $50 | n/a | n/a | 1.1M |
| GPT-6 Astra | Amazon Bedrock; by OpenAI; Long context, global endpoint | $20 | $2 | $75 | n/a | n/a | 1.1M |
| Claude Instant (100K) | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $3.26 | n/a | $11.02 | n/a | n/a | n/a |
| Claude Opus 4 | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $15 | $1.50 | $75 | $7.50 | $37.50 | n/a |
| Claude Opus 4.7 | Amazon Bedrock; by Anthropic; Global endpoint | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude 3.7 Sonnet | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $3 | $0.30 | $15 | $1.50 | $7.50 | n/a |
| Claude Haiku 5.5 | Amazon Bedrock; by Anthropic; Global endpoint | $0.10 | $0.01 | $0.50 | n/a | n/a | 1M |
| Claude Haiku 5.5 | Amazon Bedrock; by Anthropic; Long context, global endpoint | $0.50 | $0.05 | $2.50 | n/a | n/a | 1M |
| Claude 3 Opus | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $15 | n/a | $75 | $7.50 | $37.50 | n/a |
| Palmyra X5 | Amazon Bedrock; by Writer; US East (N. Virginia) | $0.60 | n/a | $6 | n/a | n/a | 1M |
| Claude Sonnet 4 | Amazon Bedrock; by Anthropic; Global endpoint | $3 | $0.30 | $15 | $1.50 | $7.50 | n/a |
| Claude Sonnet 5.5 | Amazon Bedrock; by Anthropic; Global endpoint | $2 | $0.10 | $10 | n/a | n/a | 1M |
| GPT-6 Sol | Amazon Bedrock; by OpenAI; Global endpoint | $2 | $0.20 | $10 | n/a | n/a | 1.1M |
| GPT-6 Sol | Amazon Bedrock; by OpenAI; Long context, global endpoint | $4 | $0.40 | $15 | n/a | n/a | 1.1M |
| Claude Mythos 5.2 | Amazon Bedrock; by Anthropic; Global endpoint | $8 | $0.20 | $40 | n/a | n/a | n/a |
| Claude Fable 5 | Amazon Bedrock; by Anthropic; Global endpoint | $10 | $1 | $50 | n/a | n/a | 1M |
| Claude 3.5 Sonnet v2 | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $3 | $0.30 | $15 | $1.50 | $7.50 | n/a |
| Claude 3 Sonnet | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $3 | n/a | $15 | $1.50 | $7.50 | n/a |
| Claude Sonnet 4.5 | Amazon Bedrock; by Anthropic; Global endpoint | $3 | $0.30 | $15 | $1.50 | $7.50 | 1M |
| Claude 3.5 Haiku | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $0.80 | $0.08 | $4 | $0.40 | $2 | n/a |
| Claude Mythos 5.1 | Amazon Bedrock; by Anthropic; Global endpoint | $10 | $0.25 | $50 | n/a | n/a | n/a |
| Jamba 1.5 Large | Amazon Bedrock; by AI21 Labs; US East (N. Virginia) | $2 | n/a | $8 | n/a | n/a | n/a |
| Claude Haiku 4.5 | Amazon Bedrock; by Anthropic; Global endpoint | $1 | $0.10 | $5 | $0.50 | $2.50 | 200K |
| Generate Model - Command | Amazon Bedrock; by Cohere; US East (N. Virginia) | $1 | n/a | $2 | n/a | n/a | n/a |
| Claude Opus 4.1 | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $15 | $1.50 | $75 | n/a | n/a | 200K |
| Claude (100K) | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $22.04 | n/a | $65.36 | n/a | n/a | n/a |
| Jurassic-2 Mid | Amazon Bedrock; by AI21 Labs; US East (N. Virginia) | $12.50 | n/a | $12.50 | n/a | n/a | n/a |
| Llama 2 Chat 70B | Amazon Bedrock; by Amazon; US East (N. Virginia) | $1.95 | n/a | $2.56 | n/a | n/a | n/a |
| Claude Instant | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $0.80 | n/a | $2.40 | n/a | n/a | n/a |
| Jamba 1.5 Mini | Amazon Bedrock; by AI21 Labs; US East (N. Virginia) | $0.20 | n/a | $0.40 | n/a | n/a | n/a |
| Claude | Amazon Bedrock; by Anthropic; US East (N. Virginia) | $8 | n/a | $24 | n/a | n/a | n/a |
| Command R | Amazon Bedrock; by Cohere; US East (N. Virginia) | $0.50 | n/a | $1.50 | n/a | n/a | n/a |
| Jurassic-2 Ultra | Amazon Bedrock; by AI21 Labs; US East (N. Virginia) | $18.80 | n/a | $18.80 | n/a | n/a | n/a |
| Generate Model - Command-Light | Amazon Bedrock; by Cohere; US East (N. Virginia) | $0.30 | n/a | $0.60 | n/a | n/a | n/a |
| Llama 2 Chat 13B | Amazon Bedrock; by Amazon; US East (N. Virginia) | $0.75 | n/a | $1 | n/a | n/a | n/a |
| Jamba-Instruct | Amazon Bedrock; by AI21 Labs; US East (N. Virginia) | $0.50 | n/a | $0.70 | n/a | n/a | n/a |
| Claude Fable 5.2 | Azure; by Anthropic; listed on OpenRouter | $8 | $0.20 | $40 | n/a | n/a | 1M |
| Claude Fable 5.2 (batch) | Azure; by Anthropic; listed on OpenRouter | $8 | $0.20 | $40 | n/a | n/a | 1M |
| Muse Ember | Meta; listed on OpenRouter | $0.10 | $0.004 | $0.50 | n/a | n/a | 1M |
| LFM3-1.7B | Liquid; by LiquidAI; listed on OpenRouter | $0.08 | $0.02 | $0.20 | n/a | n/a | 262.1K |
| Ripple 65B Preview | PrimeIntellect; by Ricursive; listed on OpenRouter | $0.10 | $0.025 | $0.30 | n/a | n/a | 1M |
| Ripple 65B Preview | GMICloud; by Ricursive; listed on OpenRouter | $0.20 | $0.05 | $0.60 | n/a | n/a | 1M |
| Hy4 | Tencent; listed on OpenRouter | $0.417 | $0.021 | $1.2505 | n/a | n/a | 1M |
| Particle-1 | Particle AI; listed on OpenRouter | $0.05 | $0.003 | $0.50 | n/a | n/a | 1M |
| GPT-6.1 Terra Pro | Azure; by OpenAI; listed on OpenRouter | $1 | $0.05 | $5 | n/a | n/a | 1.1M |
| GPT-6.1 Terra Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $1 | $0.05 | $5 | n/a | n/a | 1.1M |
| GPT-6.1 Terra | Azure; by OpenAI; listed on OpenRouter | $1 | $0.05 | $5 | n/a | n/a | 1.1M |
| GPT-6.1 Terra (batch) | Azure; by OpenAI; listed on OpenRouter | $1 | $0.05 | $5 | n/a | n/a | 1.1M |
| Kimi K3.1 Pro | Wafer; by Moonshot AI; listed on OpenRouter | $0.217 | $0.18 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Relace; by Moonshot AI; listed on OpenRouter | $0.2185 | $0.20 | $8.4332 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Sail Research; by Moonshot AI; listed on OpenRouter | $1.1 | $0.20 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Morph; by Moonshot AI; listed on OpenRouter | $1.676 | $0.264 | $12 | n/a | n/a | 1M |
| Kimi K3.1 Pro | AkashML; by Moonshot AI; listed on OpenRouter | $1.75 | $0.18 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Phala; by Moonshot AI; listed on OpenRouter | $1.80 | $0.18 | $8.2 | n/a | n/a | 1M |
| Kimi K3.1 Pro | PrimaLabs; by Moonshot AI; listed on OpenRouter | $1.80 | $0.18 | $9 | n/a | n/a | 1M |
| Kimi K3.1 Pro | InferenceNet; by Moonshot AI; listed on OpenRouter | $1.80 | $0.18 | $9 | n/a | n/a | 1M |
| Kimi K3.1 Pro | DeepInfra; by Moonshot AI; listed on OpenRouter | $1.90 | $0.19 | $9.50 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Fireworks; by Moonshot AI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | DigitalOcean; by Moonshot AI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Parasail; by Moonshot AI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Modal; by Moonshot AI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Nebius; by Moonshot AI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Moonshot AI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Kimi K3.1 Pro | Alibaba; by Moonshot AI; listed on OpenRouter | $2.3 | $0.23 | $11.50 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | GMICloud; by MiniMax; listed on OpenRouter | $0.1125 | $0.01125 | $0.45 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | StreamLake; by MiniMax; listed on OpenRouter | $0.135 | $0.0135 | $0.54 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | Fireworks; by MiniMax; listed on OpenRouter | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | InferenceNet; by MiniMax; listed on OpenRouter | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | DigitalOcean; by MiniMax; listed on OpenRouter | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | Novita; by MiniMax; listed on OpenRouter | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | Nebius; by MiniMax; listed on OpenRouter | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | SambaNova; by MiniMax; listed on OpenRouter | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | ModelRun; by MiniMax; listed on OpenRouter | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| MiniMax M3.1 Flash | Parasail; by MiniMax; listed on OpenRouter | $0.15 | $0.015 | $0.60 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Relace; by DeepSeek; listed on OpenRouter | $0.006 | $0.005 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Morph; by DeepSeek; listed on OpenRouter | $0.022 | $0.004 | $0.40 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | OpenInference; by DeepSeek; listed on OpenRouter | $0.03 | $0.005 | $0.75 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Wafer; by DeepSeek; listed on OpenRouter | $0.038 | $0.01 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | PrimeIntellect; by DeepSeek; listed on OpenRouter | $0.04368 | $0.0024 | $0.70 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Particle AI; by DeepSeek; listed on OpenRouter | $0.05 | $0.003 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Alibaba; by DeepSeek; listed on OpenRouter | $0.075 | $0.0075 | $0.30 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | InferenceNet; by DeepSeek; listed on OpenRouter | $0.08 | $0.008 | $0.577 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | DeepInfra; by DeepSeek; listed on OpenRouter | $0.098 | $0.00294 | $0.294 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Cloudflare; by DeepSeek; listed on OpenRouter | $0.10 | $0.003 | $0.60 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Sail Research; by DeepSeek; listed on OpenRouter | $0.101 | $0.006735 | $0.15 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | StreamLake; by DeepSeek; listed on OpenRouter | $0.111 | $0.00222 | $0.444 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Decart; by DeepSeek; listed on OpenRouter | $0.123 | $0.00246 | $0.492 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | StepFun; by DeepSeek; listed on OpenRouter | $0.135 | $0.0027 | $0.54 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Cohere; by DeepSeek; listed on OpenRouter | $0.15 | $0.003 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Makora; by DeepSeek; listed on OpenRouter | $0.15 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Reka; by DeepSeek; listed on OpenRouter | $0.16 | $0.008 | $1.1 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Ionstream; by DeepSeek; listed on OpenRouter | $0.165 | $0.0105 | $0.495 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Novita; by DeepSeek; listed on OpenRouter | $0.165 | $0.0033 | $0.66 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | ModelRun; by DeepSeek; listed on OpenRouter | $0.17 | $0.006 | $0.67 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | DekaLLM; by DeepSeek; listed on OpenRouter | $0.18 | $0.006 | $0.68 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | DigitalOcean; by DeepSeek; listed on OpenRouter | $0.18 | $0.006 | $0.72 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | GMICloud; by DeepSeek; listed on OpenRouter | $0.18 | $0.0036 | $0.72 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | AtlasCloud; by DeepSeek; listed on OpenRouter | $0.195 | $0.0039 | $0.78 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Phala; by DeepSeek; listed on OpenRouter | $0.20 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Fireworks; by DeepSeek; listed on OpenRouter | $0.21 | $0.0042 | $0.84 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | PrimaLabs; by DeepSeek; listed on OpenRouter | $0.24 | $0.0048 | $0.96 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Modal; by DeepSeek; listed on OpenRouter | $0.25 | $0.005 | $1 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Baidu; by DeepSeek; listed on OpenRouter | $0.28 | $0.006 | $1.13 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Crusoe; by DeepSeek; listed on OpenRouter | $0.29 | $0.007 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $0.55 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | CoreWeave; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Near AI; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Venice; by DeepSeek; listed on OpenRouter | $0.30 | $0.0075 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Nebius; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | BaseTen; by DeepSeek; listed on OpenRouter | $0.30 | $0.007 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 | Parasail; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Relace; by DeepSeek; listed on OpenRouter | $0.006 | $0.005 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Morph; by DeepSeek; listed on OpenRouter | $0.022 | $0.004 | $0.40 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | OpenInference; by DeepSeek; listed on OpenRouter | $0.03 | $0.005 | $0.75 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Wafer; by DeepSeek; listed on OpenRouter | $0.038 | $0.01 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | PrimeIntellect; by DeepSeek; listed on OpenRouter | $0.04368 | $0.0024 | $0.70 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Particle AI; by DeepSeek; listed on OpenRouter | $0.05 | $0.003 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Alibaba; by DeepSeek; listed on OpenRouter | $0.075 | $0.0075 | $0.30 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | InferenceNet; by DeepSeek; listed on OpenRouter | $0.08 | $0.008 | $0.577 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | DeepInfra; by DeepSeek; listed on OpenRouter | $0.098 | $0.00294 | $0.294 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Cloudflare; by DeepSeek; listed on OpenRouter | $0.10 | $0.003 | $0.60 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Sail Research; by DeepSeek; listed on OpenRouter | $0.101 | $0.006735 | $0.15 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | StreamLake; by DeepSeek; listed on OpenRouter | $0.111 | $0.00222 | $0.444 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Decart; by DeepSeek; listed on OpenRouter | $0.123 | $0.00246 | $0.492 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | StepFun; by DeepSeek; listed on OpenRouter | $0.135 | $0.0027 | $0.54 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Cohere; by DeepSeek; listed on OpenRouter | $0.15 | $0.003 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Makora; by DeepSeek; listed on OpenRouter | $0.15 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Reka; by DeepSeek; listed on OpenRouter | $0.16 | $0.008 | $1.1 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Ionstream; by DeepSeek; listed on OpenRouter | $0.165 | $0.0105 | $0.495 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Novita; by DeepSeek; listed on OpenRouter | $0.165 | $0.0033 | $0.66 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | ModelRun; by DeepSeek; listed on OpenRouter | $0.17 | $0.006 | $0.67 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | DekaLLM; by DeepSeek; listed on OpenRouter | $0.18 | $0.006 | $0.68 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | DigitalOcean; by DeepSeek; listed on OpenRouter | $0.18 | $0.006 | $0.72 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | GMICloud; by DeepSeek; listed on OpenRouter | $0.18 | $0.0036 | $0.72 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | AtlasCloud; by DeepSeek; listed on OpenRouter | $0.195 | $0.0039 | $0.78 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Phala; by DeepSeek; listed on OpenRouter | $0.20 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Fireworks; by DeepSeek; listed on OpenRouter | $0.21 | $0.0042 | $0.84 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | PrimaLabs; by DeepSeek; listed on OpenRouter | $0.24 | $0.0048 | $0.96 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Modal; by DeepSeek; listed on OpenRouter | $0.25 | $0.005 | $1 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Baidu; by DeepSeek; listed on OpenRouter | $0.28 | $0.006 | $1.13 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Crusoe; by DeepSeek; listed on OpenRouter | $0.29 | $0.007 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $0.55 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | CoreWeave; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Near AI; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Venice; by DeepSeek; listed on OpenRouter | $0.30 | $0.0075 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Nebius; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | BaseTen; by DeepSeek; listed on OpenRouter | $0.30 | $0.007 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash 1017 (batch) | Parasail; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| Beam 1.0 501B A23B | Nebius; by Reflection; listed on OpenRouter | $0.25 | $0.05 | $1 | n/a | n/a | 1M |
| Beam 1.0 501B A23B | CoreWeave; by Reflection; listed on OpenRouter | $0.30 | $0.04 | $1 | n/a | n/a | 1M |
| Beam 1.0 501B A23B | DigitalOcean; by Reflection; listed on OpenRouter | $0.40 | $0.04 | $1.20 | n/a | n/a | 1M |
| Beam 1.0 501B A23B | BaseTen; by Reflection; listed on OpenRouter | $0.40 | $0.04 | $1.20 | n/a | n/a | 1M |
| Ternary Bonsai 2 9B | Darkbloom; by PrismML; listed on OpenRouter | $0.08 | $0.04 | $0.13 | n/a | n/a | 262.1K |
| Moondream 3 Preview | PrimeIntellect; by Moondream; listed on OpenRouter | $0.30 | n/a | $2.50 | n/a | n/a | 32K |
| Moondream 3.1 | PrimeIntellect; by Moondream; listed on OpenRouter | $0.30 | n/a | $2.50 | n/a | n/a | 32K |
| Muse Spark Open | Makora; by Meta; listed on OpenRouter | $1.25 | $0.15 | $4.25 | n/a | n/a | 1M |
| Muse Spark Open | Modal; by Meta; listed on OpenRouter | $1.25 | $0.15 | $4.25 | n/a | n/a | 1M |
| Muse Spark Open | Nebius; by Meta; listed on OpenRouter | $1.25 | $0.15 | $4.25 | n/a | n/a | 1M |
| Muse Spark Open | BaseTen; by Meta; listed on OpenRouter | $1.25 | $0.15 | $4.25 | n/a | n/a | 1M |
| Nemotron 3.5 Super VL | Darkbloom; by NVIDIA; listed on OpenRouter | $0.06 | $0.01 | $0.35 | n/a | n/a | 262.1K |
| Nemotron 3.5 Super VL | Io Net; by NVIDIA; listed on OpenRouter | $0.09 | $0.012 | $0.30 | n/a | n/a | 262.1K |
| Nemotron 3.5 Super VL | Crusoe; by NVIDIA; listed on OpenRouter | $0.10 | $0.08 | $0.40 | n/a | n/a | 1M |
| Nemotron 3.5 Super VL | CoreWeave; by NVIDIA; listed on OpenRouter | $0.15 | $0.02 | $0.47 | n/a | n/a | 1M |
| Nemotron 3.5 Super VL | PrimeIntellect; by NVIDIA; listed on OpenRouter | $0.15 | $0.02 | $0.50 | n/a | n/a | 1M |
| Nemotron 3.5 Super VL | DeepInfra; by NVIDIA; listed on OpenRouter | $0.15 | $0.02 | $0.50 | n/a | n/a | 262.1K |
| Nemotron 3.5 Super VL | Venice; by NVIDIA; listed on OpenRouter | $0.20 | $0.04 | $0.50 | n/a | n/a | 262.1K |
| Nemotron 3.5 Super VL | BaseTen; by NVIDIA; listed on OpenRouter | $0.20 | $0.04 | $0.50 | n/a | n/a | 1M |
| Nemotron 3.5 Super VL | Nebius; by NVIDIA; listed on OpenRouter | $0.30 | $0.06 | $0.90 | n/a | n/a | 1M |
| Pareto 26.10 | Unbiased; by unbiased; listed on OpenRouter | $0.80 | $0.03 | $3.20 | n/a | n/a | 1M |
| Pareto 26.9 | Unbiased; by unbiased; listed on OpenRouter | $2.50 | $0.25 | $7.50 | n/a | n/a | 262.1K |
| LFM2.5-24B-A2B | Liquid; by LiquidAI; listed on OpenRouter | $0.05 | $0.025 | $0.70 | n/a | n/a | 128K |
| Gemini 4 Argon | Google Vertex; by Google; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1M |
| Gemini 4 Argon (batch) | Google Vertex; by Google; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1M |
| Muse Wonder | Meta; listed on OpenRouter | $4 | $0.20 | $20 | n/a | n/a | 1M |
| Step 5 Preview | Featherless; by StepFun; listed on OpenRouter | $1 | $0.10 | $2.70 | n/a | n/a | 262.1K |
| Step 5 Preview | StepFun; listed on OpenRouter | $1 | $0.10 | $2.70 | n/a | n/a | 1M |
| Claude Haiku 5.5 | Google Vertex; by Anthropic; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1M |
| Claude Haiku 5.5 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1M |
| Claude Haiku 5.5 | Azure; by Anthropic; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1M |
| Claude Haiku 5.5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1M |
| Claude Haiku 5.5 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1M |
| Claude Haiku 5.5 (batch) | Azure; by Anthropic; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1M |
| Ling 3.1 Flash | Novita; by inclusionAI; listed on OpenRouter | $0.045 | $0.009 | $0.135 | n/a | n/a | 1M |
| Ling 3.1 Flash | DeepInfra; by inclusionAI; listed on OpenRouter | $0.075 | $0.015 | $0.225 | n/a | n/a | 262.1K |
| GPT-6.1 Sol Pro | Azure; by OpenAI; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1.1M |
| GPT-6.1 Sol Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1.1M |
| GPT-6.1 Sol | Azure; by OpenAI; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1.1M |
| GPT-6.1 Sol (batch) | Azure; by OpenAI; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1.1M |
| Claude Sonnet 5.5 | Google Vertex; by Anthropic; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5.5 | Azure; by Anthropic; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5.5 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5.5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5.5 (batch) | Azure; by Anthropic; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5.5 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $2 | $0.10 | $10 | n/a | n/a | 1M |
| Perceptron Mk1.5 | Perceptron; listed on OpenRouter | $0.15 | n/a | $1.50 | n/a | n/a | 36.9K |
| Ember-1 | Fireworks; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| GLM 5.3 Prime | Alibaba; by Z.ai; listed on OpenRouter | $2.80 | $0.56 | $8.8 | n/a | n/a | 1M |
| Qwen3.8 Max Prime | Alibaba; by Qwen; listed on OpenRouter | $4 | $0.50 | $12 | n/a | n/a | 1M |
| Aion 3.5 Mini | AionLabs; listed on OpenRouter | $0.70 | $0.18 | $1.40 | n/a | n/a | 262.1K |
| Aion 3.5 | AionLabs; listed on OpenRouter | $3 | $0.75 | $6 | n/a | n/a | 262.1K |
| Solar Mini 4 | Upstage; listed on OpenRouter | $0.05 | $0.005 | $0.20 | n/a | n/a | 524.3K |
| Command A+ | Cohere; listed on OpenRouter | $0.30 | $0.15 | $1.50 | n/a | n/a | 192K |
| GPT-6 Luna Pro | Azure; by OpenAI; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1.1M |
| GPT-6 Luna Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1.1M |
| GPT-6 Luna | Azure; by OpenAI; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1.1M |
| GPT-6 Luna (batch) | Azure; by OpenAI; listed on OpenRouter | $0.10 | $0.01 | $0.50 | n/a | n/a | 1.1M |
| GPT-6 Sol Pro | Azure; by OpenAI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1.1M |
| GPT-6 Sol Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1.1M |
| GPT-6 Sol | Azure; by OpenAI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1.1M |
| GPT-6 Sol (batch) | Azure; by OpenAI; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1.1M |
| Claude Opus 5.5 | Azure; by Anthropic; listed on OpenRouter | $4 | $0.20 | $20 | n/a | n/a | 1M |
| Claude Opus 5.5 | Google Vertex; by Anthropic; listed on OpenRouter | $4 | $0.20 | $20 | n/a | n/a | 1M |
| Claude Opus 5.5 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $4 | $0.20 | $20 | n/a | n/a | 1M |
| Claude Opus 5.5 (batch) | Azure; by Anthropic; listed on OpenRouter | $4 | $0.20 | $20 | n/a | n/a | 1M |
| Claude Opus 5.5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $4 | $0.20 | $20 | n/a | n/a | 1M |
| Claude Opus 5.5 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $4 | $0.20 | $20 | n/a | n/a | 1M |
| MiMo-V2.6-Pro-UltraSpeed | Xiaomi; listed on OpenRouter | $4.35 | $0.036 | $8.7 | n/a | n/a | 1M |
| MiMo-V2.6-Flash | Darkbloom; by Xiaomi; listed on OpenRouter | $0.10 | $0.05 | $0.28 | n/a | n/a | 1M |
| MiMo-V2.6-Flash | GMICloud; by Xiaomi; listed on OpenRouter | $0.112 | $0.0024 | $0.224 | n/a | n/a | 1.1M |
| MiMo-V2.6-Flash | Cohere; by Xiaomi; listed on OpenRouter | $0.14 | $0.003 | $0.28 | n/a | n/a | 1M |
| MiMo-V2.6-Flash | Makora; by Xiaomi; listed on OpenRouter | $0.14 | $0.003 | $0.28 | n/a | n/a | 1M |
| MiMo-V2.6-Flash | DeepInfra; by Xiaomi; listed on OpenRouter | $0.14 | $0.0028 | $0.28 | n/a | n/a | 1M |
| MiMo-V2.6-Flash | Xiaomi; listed on OpenRouter | $0.14 | $0.0028 | $0.28 | n/a | n/a | 1M |
| MiMo-V2.6-Flash | Tenstorrent; by Xiaomi; listed on OpenRouter | $0.165 | $0.004 | $0.33 | n/a | n/a | 1M |
| MiMo-V2.6-Flash | Venice; by Xiaomi; listed on OpenRouter | $0.175 | $0.00375 | $0.35 | n/a | n/a | 1M |
| MiMo-V2.6-Pro | GMICloud; by Xiaomi; listed on OpenRouter | $0.32625 | $0.003 | $0.6525 | n/a | n/a | 1.1M |
| MiMo-V2.6-Pro | DeepInfra; by Xiaomi; listed on OpenRouter | $0.387 | $0.00324 | $0.783 | n/a | n/a | 1M |
| MiMo-V2.6-Pro | Xiaomi; listed on OpenRouter | $0.435 | $0.0036 | $0.87 | n/a | n/a | 1M |
| MiMo-V2.6-Pro | StreamLake; by Xiaomi; listed on OpenRouter | $0.522 | $0.00432 | $1.044 | n/a | n/a | 1M |
| MiMo-V2.6-Pro | PrimaLabs; by Xiaomi; listed on OpenRouter | $4 | $0.036 | $8 | n/a | n/a | 1M |
| Qwen3.8 Omni Flash | Alibaba; by Qwen; listed on OpenRouter | $0.15 | $0.016 | $0.47 | n/a | n/a | 1M |
| Ternary Bonsai 2 27B | Darkbloom; by PrismML; listed on OpenRouter | $0.075 | $0.0375 | $0.50 | n/a | n/a | 262.1K |
| Schematron V2 Turbo | InferenceNet; by Inference.net; listed on OpenRouter | $0.03 | $0.03 | $0.15 | n/a | n/a | 128K |
| Schematron V2 Small | InferenceNet; by Inference.net; listed on OpenRouter | $0.05 | $0.05 | $0.23 | n/a | n/a | 128K |
| Fugu Ultra v2 | Sakana AI; by Sakana; listed on OpenRouter | $5 | $0.50 | $30 | n/a | n/a | 1M |
| Fugu Max | Sakana AI; by Sakana; listed on OpenRouter | $2 | $0.25 | $6 | n/a | n/a | 1M |
| Ling 3.0 Flash VL | DeepInfra; by inclusionAI; listed on OpenRouter | $0.06 | $0.012 | $0.18 | n/a | n/a | 131.1K |
| DeepSeek V4.1 Flash | Relace; by DeepSeek; listed on OpenRouter | $0.006 | $0.005 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Morph; by DeepSeek; listed on OpenRouter | $0.022 | $0.004 | $0.40 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | OpenInference; by DeepSeek; listed on OpenRouter | $0.03 | $0.006 | $0.75 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | PrimeIntellect; by DeepSeek; listed on OpenRouter | $0.039 | $0.0012 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Wafer; by DeepSeek; listed on OpenRouter | $0.051 | $0.018 | $0.567 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Alibaba; by DeepSeek; listed on OpenRouter | $0.075 | $0.0075 | $0.30 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | InferenceNet; by DeepSeek; listed on OpenRouter | $0.103 | $0.008 | $0.60 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Reka; by DeepSeek; listed on OpenRouter | $0.12 | $0.008 | $0.96 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Mancer 2; by DeepSeek; listed on OpenRouter | $0.125 | $0.05 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | CoreWeave; by DeepSeek; listed on OpenRouter | $0.13 | $0.002 | $0.65 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | AtlasCloud; by DeepSeek; listed on OpenRouter | $0.135 | $0.0135 | $0.54 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Baidu; by DeepSeek; listed on OpenRouter | $0.1398 | $0.002796 | $0.5592 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | DeepInfra; by DeepSeek; listed on OpenRouter | $0.14 | $0.0042 | $0.42 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | StreamLake; by DeepSeek; listed on OpenRouter | $0.147 | $0.00294 | $0.588 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | GMICloud; by DeepSeek; listed on OpenRouter | $0.18 | $0.0036 | $0.72 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Cohere; by DeepSeek; listed on OpenRouter | $0.195 | $0.003 | $0.66 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Novita; by DeepSeek; listed on OpenRouter | $0.195 | $0.0039 | $0.78 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Fireworks; by DeepSeek; listed on OpenRouter | $0.21 | $0.0042 | $0.84 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Phala; by DeepSeek; listed on OpenRouter | $0.21 | $0.0042 | $0.84 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | DigitalOcean; by DeepSeek; listed on OpenRouter | $0.225 | $0.006 | $0.90 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Makora; by DeepSeek; listed on OpenRouter | $0.26 | $0.0058 | $1.17 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Crusoe; by DeepSeek; listed on OpenRouter | $0.29 | $0.007 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Google Vertex; by DeepSeek; listed on OpenRouter | $0.30 | $0.03 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | BaseTen; by DeepSeek; listed on OpenRouter | $0.30 | $0.007 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Modal; by DeepSeek; listed on OpenRouter | $0.30 | $0.03 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Parasail; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash | Venice; by DeepSeek; listed on OpenRouter | $0.30 | $0.0075 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Relace; by DeepSeek; listed on OpenRouter | $0.006 | $0.005 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Morph; by DeepSeek; listed on OpenRouter | $0.022 | $0.004 | $0.40 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | OpenInference; by DeepSeek; listed on OpenRouter | $0.03 | $0.006 | $0.75 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | PrimeIntellect; by DeepSeek; listed on OpenRouter | $0.039 | $0.0012 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Wafer; by DeepSeek; listed on OpenRouter | $0.051 | $0.018 | $0.567 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Alibaba; by DeepSeek; listed on OpenRouter | $0.075 | $0.0075 | $0.30 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | InferenceNet; by DeepSeek; listed on OpenRouter | $0.103 | $0.008 | $0.60 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Reka; by DeepSeek; listed on OpenRouter | $0.12 | $0.008 | $0.96 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Mancer 2; by DeepSeek; listed on OpenRouter | $0.125 | $0.05 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | CoreWeave; by DeepSeek; listed on OpenRouter | $0.13 | $0.002 | $0.65 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | AtlasCloud; by DeepSeek; listed on OpenRouter | $0.135 | $0.0135 | $0.54 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Baidu; by DeepSeek; listed on OpenRouter | $0.1398 | $0.002796 | $0.5592 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | DeepInfra; by DeepSeek; listed on OpenRouter | $0.14 | $0.0042 | $0.42 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | StreamLake; by DeepSeek; listed on OpenRouter | $0.147 | $0.00294 | $0.588 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | GMICloud; by DeepSeek; listed on OpenRouter | $0.18 | $0.0036 | $0.72 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Cohere; by DeepSeek; listed on OpenRouter | $0.195 | $0.003 | $0.66 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Novita; by DeepSeek; listed on OpenRouter | $0.195 | $0.0039 | $0.78 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Fireworks; by DeepSeek; listed on OpenRouter | $0.21 | $0.0042 | $0.84 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Phala; by DeepSeek; listed on OpenRouter | $0.21 | $0.0042 | $0.84 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | DigitalOcean; by DeepSeek; listed on OpenRouter | $0.225 | $0.006 | $0.90 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Makora; by DeepSeek; listed on OpenRouter | $0.26 | $0.0058 | $1.17 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Crusoe; by DeepSeek; listed on OpenRouter | $0.29 | $0.007 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Google Vertex; by DeepSeek; listed on OpenRouter | $0.30 | $0.03 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | BaseTen; by DeepSeek; listed on OpenRouter | $0.30 | $0.007 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Modal; by DeepSeek; listed on OpenRouter | $0.30 | $0.03 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Parasail; by DeepSeek; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| DeepSeek V4.1 Flash (batch) | Venice; by DeepSeek; listed on OpenRouter | $0.30 | $0.0075 | $1.20 | n/a | n/a | 1M |
| Nex-N2.5-Mini | Nex AGI; listed on OpenRouter | $0.025 | $0.0025 | $0.10 | n/a | n/a | 262.1K |
| Nex-N2.5-Pro | Nex AGI; listed on OpenRouter | $0.075 | $0.015 | $0.25 | n/a | n/a | 262.1K |
| GPT-6 Astra | Azure; by OpenAI; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1.1M |
| GPT-6 Astra (batch) | Azure; by OpenAI; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1.1M |
| GPT-6 Astra Pro | Azure; by OpenAI; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1.1M |
| GPT-6 Astra Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1.1M |
| Qwen3.8 Max (0902) | Alibaba; by Qwen; listed on OpenRouter | $2 | $0.25 | $6 | n/a | n/a | 1M |
| Qwen3.8 Max (0902) (batch) | Alibaba; by Qwen; listed on OpenRouter | $2 | $0.25 | $6 | n/a | n/a | 1M |
| Muse Spark 1.3 Contributor | Meta; listed on OpenRouter | $0.10 | $0.002 | $0.20 | n/a | n/a | 1M |
| Muse Spark 1.3 | Meta; listed on OpenRouter | $1.25 | $0.15 | $4.25 | n/a | n/a | 1M |
| Gemini 3.8 Flash | Google Vertex; by Google; listed on OpenRouter | $0.75 | $0.075 | $3.75 | n/a | n/a | 1M |
| Gemini 3.8 Flash (batch) | Google Vertex; by Google; listed on OpenRouter | $0.75 | $0.075 | $3.75 | n/a | n/a | 1M |
| Claude Fable 5.1 | Azure; by Anthropic; listed on OpenRouter | $10 | $0.25 | $50 | n/a | n/a | 1M |
| Claude Fable 5.1 | Google Vertex; by Anthropic; listed on OpenRouter | $10 | $0.25 | $50 | n/a | n/a | 1M |
| Claude Fable 5.1 (batch) | Azure; by Anthropic; listed on OpenRouter | $10 | $0.25 | $50 | n/a | n/a | 1M |
| Claude Fable 5.1 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $10 | $0.25 | $50 | n/a | n/a | 1M |
| Granite 4.2 8B | CoreWeave; by IBM; listed on OpenRouter | $0.06 | $0.015 | $0.15 | n/a | n/a | 131.1K |
| Granite 4.2 8B | DeepInfra; by IBM; listed on OpenRouter | $0.06 | $0.015 | $0.25 | n/a | n/a | 131.1K |
| Hy4 preview | Tencent; listed on OpenRouter | $0.7506 | $0.0378 | $2.2509 | n/a | n/a | 1M |
| Hy4 preview | GMICloud; by Tencent; listed on OpenRouter | $0.834 | $0.042 | $2.501 | n/a | n/a | 262.1K |
| Hy4 preview | AtlasCloud; by Tencent; listed on OpenRouter | $0.834 | $0.042 | $2.501 | n/a | n/a | 1M |
| Hy4 preview | DeepInfra; by Tencent; listed on OpenRouter | $0.834 | $0.042 | $2.501 | n/a | n/a | 1M |
| Hy4 preview | Novita; by Tencent; listed on OpenRouter | $0.834 | $0.042 | $2.501 | n/a | n/a | 1M |
| Hy4 preview | SiliconFlow; by Tencent; listed on OpenRouter | $0.834 | $0.042 | $2.501 | n/a | n/a | 1M |
| Qwen3.8 Flash | Alibaba; by Qwen; listed on OpenRouter | $0.15 | $0.016 | $0.47 | n/a | n/a | 1M |
| GLM 5.3 Flash | Relace; by Z.ai; listed on OpenRouter | $0.0335 | $0.03 | $0.083 | n/a | n/a | 1M |
| GLM 5.3 Flash | Wafer; by Z.ai; listed on OpenRouter | $0.0353 | $0.0343 | $0.184 | n/a | n/a | 1M |
| GLM 5.3 Flash | PrimeIntellect; by Z.ai; listed on OpenRouter | $0.0525 | $0.0105 | $0.25 | n/a | n/a | 1M |
| GLM 5.3 Flash | Morph; by Z.ai; listed on OpenRouter | $0.06 | $0.026 | $0.122 | n/a | n/a | 1M |
| GLM 5.3 Flash | OpenInference; by Z.ai; listed on OpenRouter | $0.06 | $0.03 | $0.30 | n/a | n/a | 1M |
| GLM 5.3 Flash | Darkbloom; by Z.ai; listed on OpenRouter | $0.065 | $0.012 | $0.18 | n/a | n/a | 262.1K |
| GLM 5.3 Flash | Tenstorrent; by Z.ai; listed on OpenRouter | $0.075 | $0.015 | $0.25 | n/a | n/a | 262K |
| GLM 5.3 Flash | GMICloud; by Z.ai; listed on OpenRouter | $0.075 | $0.015 | $0.25 | n/a | n/a | 1M |
| GLM 5.3 Flash | Near AI; by Z.ai; listed on OpenRouter | $0.08 | $0.012 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Novita; by Z.ai; listed on OpenRouter | $0.084 | $0.0168 | $0.28 | n/a | n/a | 1M |
| GLM 5.3 Flash | StreamLake; by Z.ai; listed on OpenRouter | $0.0885 | $0.0177 | $0.295 | n/a | n/a | 1M |
| GLM 5.3 Flash | Particle AI; by Z.ai; listed on OpenRouter | $0.10 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | InferenceNet; by Z.ai; listed on OpenRouter | $0.108 | $0.034 | $0.35 | n/a | n/a | 1M |
| GLM 5.3 Flash | Phala; by Z.ai; listed on OpenRouter | $0.1125 | $0.0225 | $0.375 | n/a | n/a | 1M |
| GLM 5.3 Flash | DigitalOcean; by Z.ai; listed on OpenRouter | $0.1125 | $0.03 | $0.375 | n/a | n/a | 1M |
| GLM 5.3 Flash | Reka; by Z.ai; listed on OpenRouter | $0.12 | $0.03 | $0.30 | n/a | n/a | 1M |
| GLM 5.3 Flash | CoreWeave; by Z.ai; listed on OpenRouter | $0.12 | $0.02 | $0.38 | n/a | n/a | 1M |
| GLM 5.3 Flash | NextBit; by Z.ai; listed on OpenRouter | $0.12 | $0.03 | $0.45 | n/a | n/a | 1M |
| GLM 5.3 Flash | Makora; by Z.ai; listed on OpenRouter | $0.14 | $0.03 | $0.24 | n/a | n/a | 1M |
| GLM 5.3 Flash | Cohere; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Nebius; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Modal; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | BaseTen; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Crusoe; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | AtlasCloud; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Fireworks; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Friendli; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | SiliconFlow; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Parasail; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Venice; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | DeepInfra; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash | Inceptron; by Z.ai; listed on OpenRouter | $0.175 | $0.03 | $0.45 | n/a | n/a | 1M |
| GLM 5.3 Flash | Sail Research; by Z.ai; listed on OpenRouter | $0.175 | $0.0225 | $0.625 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Relace; by Z.ai; listed on OpenRouter | $0.0335 | $0.03 | $0.083 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Wafer; by Z.ai; listed on OpenRouter | $0.0353 | $0.0343 | $0.184 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | PrimeIntellect; by Z.ai; listed on OpenRouter | $0.0525 | $0.0105 | $0.25 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Morph; by Z.ai; listed on OpenRouter | $0.06 | $0.026 | $0.122 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | OpenInference; by Z.ai; listed on OpenRouter | $0.06 | $0.03 | $0.30 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Darkbloom; by Z.ai; listed on OpenRouter | $0.065 | $0.012 | $0.18 | n/a | n/a | 262.1K |
| GLM 5.3 Flash (batch) | Tenstorrent; by Z.ai; listed on OpenRouter | $0.075 | $0.015 | $0.25 | n/a | n/a | 262K |
| GLM 5.3 Flash (batch) | GMICloud; by Z.ai; listed on OpenRouter | $0.075 | $0.015 | $0.25 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Near AI; by Z.ai; listed on OpenRouter | $0.08 | $0.012 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Novita; by Z.ai; listed on OpenRouter | $0.084 | $0.0168 | $0.28 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | StreamLake; by Z.ai; listed on OpenRouter | $0.0885 | $0.0177 | $0.295 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Particle AI; by Z.ai; listed on OpenRouter | $0.10 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | InferenceNet; by Z.ai; listed on OpenRouter | $0.108 | $0.034 | $0.35 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Phala; by Z.ai; listed on OpenRouter | $0.1125 | $0.0225 | $0.375 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | DigitalOcean; by Z.ai; listed on OpenRouter | $0.1125 | $0.03 | $0.375 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Reka; by Z.ai; listed on OpenRouter | $0.12 | $0.03 | $0.30 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | CoreWeave; by Z.ai; listed on OpenRouter | $0.12 | $0.02 | $0.38 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | NextBit; by Z.ai; listed on OpenRouter | $0.12 | $0.03 | $0.45 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Makora; by Z.ai; listed on OpenRouter | $0.14 | $0.03 | $0.24 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Cohere; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Nebius; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Modal; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | BaseTen; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Crusoe; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | AtlasCloud; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Fireworks; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Friendli; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | SiliconFlow; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Parasail; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Venice; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | DeepInfra; by Z.ai; listed on OpenRouter | $0.15 | $0.03 | $0.50 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Inceptron; by Z.ai; listed on OpenRouter | $0.175 | $0.03 | $0.45 | n/a | n/a | 1M |
| GLM 5.3 Flash (batch) | Sail Research; by Z.ai; listed on OpenRouter | $0.175 | $0.0225 | $0.625 | n/a | n/a | 1M |
| Muse Spark 1.2 Contributor | Meta; listed on OpenRouter | $0.10 | $0.002 | $0.20 | n/a | n/a | 1M |
| DeepSeek V4 Flash Vision Exp | DeepInfra; by DeepSeek; listed on OpenRouter | $0.2156 | $0.00686 | $0.6468 | n/a | n/a | 1M |
| DeepSeek V4 Flash Vision Exp | GMICloud; by DeepSeek; listed on OpenRouter | $0.44 | $0.014 | $1.32 | n/a | n/a | 1M |
| Hy-MT2-1.8B | Tencent; listed on OpenRouter | $0.044 | n/a | $0.177 | n/a | n/a | 8.2K |
| Hy-MT2-30B-A3B | Tencent; listed on OpenRouter | $0.074 | n/a | $0.295 | n/a | n/a | 8.2K |
| Hy-MT2-7B | Tencent; listed on OpenRouter | $0.074 | n/a | $0.295 | n/a | n/a | 8.2K |
| GLM 5.3 | Wafer; by Z.ai; listed on OpenRouter | $0.0699 | $0.0689 | $1.40 | n/a | n/a | 1M |
| GLM 5.3 | Relace; by Z.ai; listed on OpenRouter | $0.0703 | $0.0703 | $4.3843 | n/a | n/a | 1M |
| GLM 5.3 | InferenceNet; by Z.ai; listed on OpenRouter | $0.08 | $0.076 | $3.60 | n/a | n/a | 1M |
| GLM 5.3 | Reka; by Z.ai; listed on OpenRouter | $0.14 | $0.105 | $2.10 | n/a | n/a | 1M |
| GLM 5.3 | DekaLLM; by Z.ai; listed on OpenRouter | $0.18 | $0.15 | $2.95 | n/a | n/a | 1M |
| GLM 5.3 | AkashML; by Z.ai; listed on OpenRouter | $0.19 | $0.19 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 | Makora; by Z.ai; listed on OpenRouter | $0.30 | $0.125 | $2 | n/a | n/a | 1M |
| GLM 5.3 | PrimeIntellect; by Z.ai; listed on OpenRouter | $0.40 | $0.0744 | $2.464 | n/a | n/a | 1M |
| GLM 5.3 | Decart; by Z.ai; listed on OpenRouter | $0.56 | $0.10 | $1.78 | n/a | n/a | 1M |
| GLM 5.3 | DeepInfra; by Z.ai; listed on OpenRouter | $0.5625 | $0.125 | $2.50 | n/a | n/a | 1M |
| GLM 5.3 | Tenstorrent; by Z.ai; listed on OpenRouter | $0.60 | $0.10 | $2.50 | n/a | n/a | 1M |
| GLM 5.3 | Novita; by Z.ai; listed on OpenRouter | $0.63 | $0.117 | $1.98 | n/a | n/a | 1M |
| GLM 5.3 | SiliconFlow; by Z.ai; listed on OpenRouter | $0.70 | $0.13 | $2.2 | n/a | n/a | 1M |
| GLM 5.3 | GMICloud; by Z.ai; listed on OpenRouter | $0.70 | $0.13 | $2.2 | n/a | n/a | 1M |
| GLM 5.3 | Cloudflare; by Z.ai; listed on OpenRouter | $0.70 | $0.13 | $2.2 | n/a | n/a | 1M |
| GLM 5.3 | CoreWeave; by Z.ai; listed on OpenRouter | $0.75 | $0.19 | $3 | n/a | n/a | 1M |
| GLM 5.3 | Sail Research; by Z.ai; listed on OpenRouter | $0.75 | $0.12 | $3 | n/a | n/a | 1M |
| GLM 5.3 | Phala; by Z.ai; listed on OpenRouter | $0.84 | $0.156 | $2.64 | n/a | n/a | 1M |
| GLM 5.3 | DigitalOcean; by Z.ai; listed on OpenRouter | $0.91 | $0.169 | $2.86 | n/a | n/a | 1M |
| GLM 5.3 | Fireworks; by Z.ai; listed on OpenRouter | $0.98 | $0.182 | $3.08 | n/a | n/a | 1M |
| GLM 5.3 | Inceptron; by Z.ai; listed on OpenRouter | $1 | $0.14 | $4.9 | n/a | n/a | 1M |
| GLM 5.3 | Alibaba; by Z.ai; listed on OpenRouter | $1.19 | $0.238 | $3.74 | n/a | n/a | 1M |
| GLM 5.3 | Morph; by Z.ai; listed on OpenRouter | $1.245 | $0.42 | $2.542 | n/a | n/a | 1M |
| GLM 5.3 | Friendli; by Z.ai; listed on OpenRouter | $1.26 | $0.234 | $3.96 | n/a | n/a | 1M |
| GLM 5.3 | Baidu; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 | BaseTen; by Z.ai; listed on OpenRouter | $1.40 | $0.14 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 | Nebius; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 | Crusoe; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 | Venice; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 | Parasail; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 | Modal; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 | AtlasCloud; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Wafer; by Z.ai; listed on OpenRouter | $0.0699 | $0.0689 | $1.40 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Relace; by Z.ai; listed on OpenRouter | $0.0703 | $0.0703 | $4.3843 | n/a | n/a | 1M |
| GLM 5.3 (batch) | InferenceNet; by Z.ai; listed on OpenRouter | $0.08 | $0.076 | $3.60 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Reka; by Z.ai; listed on OpenRouter | $0.14 | $0.105 | $2.10 | n/a | n/a | 1M |
| GLM 5.3 (batch) | DekaLLM; by Z.ai; listed on OpenRouter | $0.18 | $0.15 | $2.95 | n/a | n/a | 1M |
| GLM 5.3 (batch) | AkashML; by Z.ai; listed on OpenRouter | $0.19 | $0.19 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Makora; by Z.ai; listed on OpenRouter | $0.30 | $0.125 | $2 | n/a | n/a | 1M |
| GLM 5.3 (batch) | PrimeIntellect; by Z.ai; listed on OpenRouter | $0.40 | $0.0744 | $2.464 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Decart; by Z.ai; listed on OpenRouter | $0.56 | $0.10 | $1.78 | n/a | n/a | 1M |
| GLM 5.3 (batch) | DeepInfra; by Z.ai; listed on OpenRouter | $0.5625 | $0.125 | $2.50 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Tenstorrent; by Z.ai; listed on OpenRouter | $0.60 | $0.10 | $2.50 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Novita; by Z.ai; listed on OpenRouter | $0.63 | $0.117 | $1.98 | n/a | n/a | 1M |
| GLM 5.3 (batch) | SiliconFlow; by Z.ai; listed on OpenRouter | $0.70 | $0.13 | $2.2 | n/a | n/a | 1M |
| GLM 5.3 (batch) | GMICloud; by Z.ai; listed on OpenRouter | $0.70 | $0.13 | $2.2 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Cloudflare; by Z.ai; listed on OpenRouter | $0.70 | $0.13 | $2.2 | n/a | n/a | 1M |
| GLM 5.3 (batch) | CoreWeave; by Z.ai; listed on OpenRouter | $0.75 | $0.19 | $3 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Sail Research; by Z.ai; listed on OpenRouter | $0.75 | $0.12 | $3 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Phala; by Z.ai; listed on OpenRouter | $0.84 | $0.156 | $2.64 | n/a | n/a | 1M |
| GLM 5.3 (batch) | DigitalOcean; by Z.ai; listed on OpenRouter | $0.91 | $0.169 | $2.86 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Fireworks; by Z.ai; listed on OpenRouter | $0.98 | $0.182 | $3.08 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Inceptron; by Z.ai; listed on OpenRouter | $1 | $0.14 | $4.9 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Alibaba; by Z.ai; listed on OpenRouter | $1.19 | $0.238 | $3.74 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Morph; by Z.ai; listed on OpenRouter | $1.245 | $0.42 | $2.542 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Friendli; by Z.ai; listed on OpenRouter | $1.26 | $0.234 | $3.96 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Baidu; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | BaseTen; by Z.ai; listed on OpenRouter | $1.40 | $0.14 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Nebius; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Crusoe; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Venice; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Parasail; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | Modal; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.3 (batch) | AtlasCloud; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| Qwen3.8 27B | Darkbloom; by Qwen; listed on OpenRouter | $0.035 | $0.0175 | $1.50 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Chutes; by Qwen; listed on OpenRouter | $0.039 | $0.0039 | $1.25 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Near AI; by Qwen; listed on OpenRouter | $0.04 | $0.018 | $1.35 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Ionstream; by Qwen; listed on OpenRouter | $0.04 | $0.02 | $1.95 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Modal; by Qwen; listed on OpenRouter | $0.04 | $0.02 | $2.3 | n/a | n/a | 262.1K |
| Qwen3.8 27B | DekaLLM; by Qwen; listed on OpenRouter | $0.04 | $0.02 | $2.50 | n/a | n/a | 262.1K |
| Qwen3.8 27B | PrimeIntellect; by Qwen; listed on OpenRouter | $0.045 | $0.005 | $0.32 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Wafer; by Qwen; listed on OpenRouter | $0.06 | $0.059 | $2.3 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Mancer 2; by Qwen; listed on OpenRouter | $0.15 | $0.05 | $1.25 | n/a | n/a | 262.1K |
| Qwen3.8 27B | DeepInfra; by Qwen; listed on OpenRouter | $0.15 | $0.0375 | $1.875 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Tenstorrent; by Qwen; listed on OpenRouter | $0.15 | $0.05 | $2 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Reka; by Qwen; listed on OpenRouter | $0.15 | $0.05 | $2.50 | n/a | n/a | 262.1K |
| Qwen3.8 27B | StreamLake; by Qwen; listed on OpenRouter | $0.175 | $0.035 | $1.05 | n/a | n/a | 1M |
| Qwen3.8 27B | ModelRun; by Qwen; listed on OpenRouter | $0.20 | $0.09 | $2.50 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Phala; by Qwen; listed on OpenRouter | $0.20 | $0.05 | $2.50 | n/a | n/a | 1M |
| Qwen3.8 27B | AkashML; by Qwen; listed on OpenRouter | $0.225 | $0.05 | $1.98 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Parasail; by Qwen; listed on OpenRouter | $0.24 | $0.05 | $2.2 | n/a | n/a | 262.1K |
| Qwen3.8 27B | CoreWeave; by Qwen; listed on OpenRouter | $0.40 | $0.15 | $3 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Novita; by Qwen; listed on OpenRouter | $0.42 | $0.085 | $3 | n/a | n/a | 1M |
| Qwen3.8 27B | Alibaba; by Qwen; listed on OpenRouter | $0.425 | $0.085 | $2.55 | n/a | n/a | 1M |
| Qwen3.8 27B | Nebius; by Qwen; listed on OpenRouter | $0.45 | $0.10 | $3 | n/a | n/a | 1M |
| Qwen3.8 27B | Cloudflare; by Qwen; listed on OpenRouter | $0.45 | $0.05 | $3.20 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Venice; by Qwen; listed on OpenRouter | $0.45 | n/a | $3.20 | n/a | n/a | 262.1K |
| Qwen3.8 27B | Cerebras; by Qwen; listed on OpenRouter | $0.99 | $0.99 | $1.49 | n/a | n/a | 65.5K |
| Qwen3.8 27B (batch) | Darkbloom; by Qwen; listed on OpenRouter | $0.035 | $0.0175 | $1.50 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Chutes; by Qwen; listed on OpenRouter | $0.039 | $0.0039 | $1.25 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Near AI; by Qwen; listed on OpenRouter | $0.04 | $0.018 | $1.35 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Ionstream; by Qwen; listed on OpenRouter | $0.04 | $0.02 | $1.95 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Modal; by Qwen; listed on OpenRouter | $0.04 | $0.02 | $2.3 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | DekaLLM; by Qwen; listed on OpenRouter | $0.04 | $0.02 | $2.50 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | PrimeIntellect; by Qwen; listed on OpenRouter | $0.045 | $0.005 | $0.32 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Wafer; by Qwen; listed on OpenRouter | $0.06 | $0.059 | $2.3 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Mancer 2; by Qwen; listed on OpenRouter | $0.15 | $0.05 | $1.25 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | DeepInfra; by Qwen; listed on OpenRouter | $0.15 | $0.0375 | $1.875 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Tenstorrent; by Qwen; listed on OpenRouter | $0.15 | $0.05 | $2 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Reka; by Qwen; listed on OpenRouter | $0.15 | $0.05 | $2.50 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | StreamLake; by Qwen; listed on OpenRouter | $0.175 | $0.035 | $1.05 | n/a | n/a | 1M |
| Qwen3.8 27B (batch) | ModelRun; by Qwen; listed on OpenRouter | $0.20 | $0.09 | $2.50 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Phala; by Qwen; listed on OpenRouter | $0.20 | $0.05 | $2.50 | n/a | n/a | 1M |
| Qwen3.8 27B (batch) | AkashML; by Qwen; listed on OpenRouter | $0.225 | $0.05 | $1.98 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Parasail; by Qwen; listed on OpenRouter | $0.24 | $0.05 | $2.2 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | CoreWeave; by Qwen; listed on OpenRouter | $0.40 | $0.15 | $3 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Novita; by Qwen; listed on OpenRouter | $0.42 | $0.085 | $3 | n/a | n/a | 1M |
| Qwen3.8 27B (batch) | Alibaba; by Qwen; listed on OpenRouter | $0.425 | $0.085 | $2.55 | n/a | n/a | 1M |
| Qwen3.8 27B (batch) | Nebius; by Qwen; listed on OpenRouter | $0.45 | $0.10 | $3 | n/a | n/a | 1M |
| Qwen3.8 27B (batch) | Cloudflare; by Qwen; listed on OpenRouter | $0.45 | $0.05 | $3.20 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Venice; by Qwen; listed on OpenRouter | $0.45 | n/a | $3.20 | n/a | n/a | 262.1K |
| Qwen3.8 27B (batch) | Cerebras; by Qwen; listed on OpenRouter | $0.99 | $0.99 | $1.49 | n/a | n/a | 65.5K |
| Gemini 3.7 Flash | Google Vertex; by Google; listed on OpenRouter | $0.75 | $0.075 | $3.75 | n/a | n/a | 1M |
| Gemini 3.7 Flash (batch) | Google Vertex; by Google; listed on OpenRouter | $0.75 | $0.075 | $3.75 | n/a | n/a | 1M |
| Seed 2.1 Turbo | Seed; by ByteDance; listed on OpenRouter | $0.50 | n/a | $2.50 | n/a | n/a | 262.1K |
| Qwen3.8 2.4T A95B | Novita; by Qwen; listed on OpenRouter | $2 | $0.25 | $6 | n/a | n/a | 1M |
| Qwen3.8 2.4T A95B | Alibaba; by Qwen; listed on OpenRouter | $2 | $0.25 | $6 | n/a | n/a | 1M |
| Qwen3.8 2.4T A95B | SiliconFlow; by Qwen; listed on OpenRouter | $2 | $0.25 | $6 | n/a | n/a | 1M |
| Qwen3.8 2.4T A95B | Venice; by Qwen; listed on OpenRouter | $2 | $0.25 | $6 | n/a | n/a | 262.1K |
| Seed-2.0-Code | Seed; by ByteDance; listed on OpenRouter | $0.50 | n/a | $3 | n/a | n/a | 262.1K |
| DeepSeek V4 Pro 0813 | Relace; by DeepSeek; listed on OpenRouter | $0.0999 | $0.0999 | $8 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Wafer; by DeepSeek; listed on OpenRouter | $0.111 | $0.097 | $4.44 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Alibaba; by DeepSeek; listed on OpenRouter | $0.5808 | $0.05808 | $1.7424 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Baidu; by DeepSeek; listed on OpenRouter | $0.65868 | $0.020958 | $1.97604 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | StreamLake; by DeepSeek; listed on OpenRouter | $0.66 | $0.022 | $1.98 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Phala; by DeepSeek; listed on OpenRouter | $0.957 | $0.099 | $2.8776 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Novita; by DeepSeek; listed on OpenRouter | $0.99 | $0.033 | $2.97 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | GMICloud; by DeepSeek; listed on OpenRouter | $1.056 | $0.0352 | $3.168 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | DeepInfra; by DeepSeek; listed on OpenRouter | $1.30 | $0.10 | $2.60 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | CoreWeave; by DeepSeek; listed on OpenRouter | $1.31 | $0.044 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | BaseTen; by DeepSeek; listed on OpenRouter | $1.32 | $0.132 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | AtlasCloud; by DeepSeek; listed on OpenRouter | $1.32 | $0.132 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Nebius; by DeepSeek; listed on OpenRouter | $1.32 | $0.04 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Parasail; by DeepSeek; listed on OpenRouter | $1.32 | $0.044 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | DigitalOcean; by DeepSeek; listed on OpenRouter | $1.32 | $0.044 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | SiliconFlow; by DeepSeek; listed on OpenRouter | $1.32 | $0.044 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Cloudflare; by DeepSeek; listed on OpenRouter | $1.32 | $0.044 | $3.96 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Sail Research; by DeepSeek; listed on OpenRouter | $1.35 | $0.195 | $2.40 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0813 | Venice; by DeepSeek; listed on OpenRouter | $1.65 | $0.165 | $4.95 | n/a | n/a | 1M |
| LFM2.5-2.6B | Liquid; by LiquidAI; listed on OpenRouter | $0.35 | $0.04 | $0.50 | n/a | n/a | 128K |
| Nemotron 3.5 Lightning | Darkbloom; by NVIDIA; listed on OpenRouter | $0.025 | $0.0125 | $0.12 | n/a | n/a | 262.1K |
| Nemotron 3.5 Lightning | Io Net; by NVIDIA; listed on OpenRouter | $0.0469 | $0.02345 | $0.134 | n/a | n/a | 262.1K |
| Nemotron 3.5 Lightning | Fireworks; by NVIDIA; listed on OpenRouter | $0.05 | $0.01 | $0.20 | n/a | n/a | 262.1K |
| Nemotron 3.5 Lightning | DeepInfra; by NVIDIA; listed on OpenRouter | $0.06 | $0.03 | $0.16 | n/a | n/a | 262.1K |
| Nemotron 3.5 Lightning | CoreWeave; by NVIDIA; listed on OpenRouter | $0.07 | $0.04 | $0.20 | n/a | n/a | 262.1K |
| Sakana Namazu | Sakana AI; by Sakana; listed on OpenRouter | $0.95 | $0.15 | $4 | n/a | n/a | 262.1K |
| Solar Pro 4 | Upstage; listed on OpenRouter | $0.15 | $0.03 | $0.60 | n/a | n/a | 524.3K |
| Muse Glimmer 30B | DeepInfra; by Meta; listed on OpenRouter | $0.30 | $0.04 | $1.20 | n/a | n/a | 131.1K |
| Muse Glimmer 30B (batch) | DeepInfra; by Meta; listed on OpenRouter | $0.30 | $0.04 | $1.20 | n/a | n/a | 131.1K |
| Ling 3.0 Tiny | Ant Digital; by inclusionAI; listed on OpenRouter | $0.015 | $0.003 | $0.045 | n/a | n/a | 262.1K |
| Muse Spark 1.2 | Meta; listed on OpenRouter | $1.25 | $0.15 | $4.25 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Wafer; by DeepSeek; listed on OpenRouter | $0.0434 | $0.035 | $0.30 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | DeepInfra; by DeepSeek; listed on OpenRouter | $0.06 | $0.015 | $0.18 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Relace; by DeepSeek; listed on OpenRouter | $0.07 | $0.014 | $0.14 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Reka; by DeepSeek; listed on OpenRouter | $0.10 | $0.04 | $0.28 | n/a | n/a | 262.1K |
| DeepSeek V4 Flash 0731 | DigitalOcean; by DeepSeek; listed on OpenRouter | $0.119 | $0.0238 | $0.238 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | BaseTen; by DeepSeek; listed on OpenRouter | $0.13 | $0.028 | $0.26 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Venice; by DeepSeek; listed on OpenRouter | $0.13 | $0.028 | $0.26 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | CoreWeave; by DeepSeek; listed on OpenRouter | $0.13 | $0.05 | $0.27 | n/a | n/a | 262.1K |
| DeepSeek V4 Flash 0731 | Inceptron; by DeepSeek; listed on OpenRouter | $0.13 | $0.027 | $0.495 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Baidu; by DeepSeek; listed on OpenRouter | $0.13156 | $0.004186 | $0.39468 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | StreamLake; by DeepSeek; listed on OpenRouter | $0.132 | $0.0042 | $0.396 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Cohere; by DeepSeek; listed on OpenRouter | $0.14 | $0.07 | $0.28 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Parasail; by DeepSeek; listed on OpenRouter | $0.14 | $0.05 | $0.28 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Sail Research; by DeepSeek; listed on OpenRouter | $0.15 | $0.042 | $0.30 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Mancer 2; by DeepSeek; listed on OpenRouter | $0.15 | $0.075 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Alibaba; by DeepSeek; listed on OpenRouter | $0.176 | $0.0176 | $0.528 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.22 | $0.028 | $0.66 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | GMICloud; by DeepSeek; listed on OpenRouter | $0.286 | $0.0091 | $0.858 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Phala; by DeepSeek; listed on OpenRouter | $0.308 | $0.0196 | $0.924 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Novita; by DeepSeek; listed on OpenRouter | $0.4092 | $0.02604 | $1.2276 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Chutes; by DeepSeek; listed on OpenRouter | $0.44 | $0.044 | $1.32 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | PrimeIntellect; by DeepSeek; listed on OpenRouter | $0.44 | $0.014 | $1.32 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | AtlasCloud; by DeepSeek; listed on OpenRouter | $0.44 | $0.028 | $1.32 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 | Cloudflare; by DeepSeek; listed on OpenRouter | $0.44 | $0.014 | $1.32 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Wafer; by DeepSeek; listed on OpenRouter | $0.0434 | $0.035 | $0.30 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | DeepInfra; by DeepSeek; listed on OpenRouter | $0.06 | $0.015 | $0.18 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Relace; by DeepSeek; listed on OpenRouter | $0.07 | $0.014 | $0.14 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Reka; by DeepSeek; listed on OpenRouter | $0.10 | $0.04 | $0.28 | n/a | n/a | 262.1K |
| DeepSeek V4 Flash 0731 (batch) | DigitalOcean; by DeepSeek; listed on OpenRouter | $0.119 | $0.0238 | $0.238 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | BaseTen; by DeepSeek; listed on OpenRouter | $0.13 | $0.028 | $0.26 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Venice; by DeepSeek; listed on OpenRouter | $0.13 | $0.028 | $0.26 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | CoreWeave; by DeepSeek; listed on OpenRouter | $0.13 | $0.05 | $0.27 | n/a | n/a | 262.1K |
| DeepSeek V4 Flash 0731 (batch) | Inceptron; by DeepSeek; listed on OpenRouter | $0.13 | $0.027 | $0.495 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Baidu; by DeepSeek; listed on OpenRouter | $0.13156 | $0.004186 | $0.39468 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | StreamLake; by DeepSeek; listed on OpenRouter | $0.132 | $0.0042 | $0.396 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Cohere; by DeepSeek; listed on OpenRouter | $0.14 | $0.07 | $0.28 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Parasail; by DeepSeek; listed on OpenRouter | $0.14 | $0.05 | $0.28 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Sail Research; by DeepSeek; listed on OpenRouter | $0.15 | $0.042 | $0.30 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Mancer 2; by DeepSeek; listed on OpenRouter | $0.15 | $0.075 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Alibaba; by DeepSeek; listed on OpenRouter | $0.176 | $0.0176 | $0.528 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.22 | $0.028 | $0.66 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | GMICloud; by DeepSeek; listed on OpenRouter | $0.286 | $0.0091 | $0.858 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Phala; by DeepSeek; listed on OpenRouter | $0.308 | $0.0196 | $0.924 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Novita; by DeepSeek; listed on OpenRouter | $0.4092 | $0.02604 | $1.2276 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Chutes; by DeepSeek; listed on OpenRouter | $0.44 | $0.044 | $1.32 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | PrimeIntellect; by DeepSeek; listed on OpenRouter | $0.44 | $0.014 | $1.32 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | AtlasCloud; by DeepSeek; listed on OpenRouter | $0.44 | $0.028 | $1.32 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0731 (batch) | Cloudflare; by DeepSeek; listed on OpenRouter | $0.44 | $0.014 | $1.32 | n/a | n/a | 1M |
| Inkling Small | DeepInfra; by Thinking Machines; listed on OpenRouter | $0.45 | $0.10 | $1.20 | n/a | n/a | 524.3K |
| Inkling Small (batch) | DeepInfra; by Thinking Machines; listed on OpenRouter | $0.45 | $0.10 | $1.20 | n/a | n/a | 524.3K |
| Qwen3.7 Flash | Alibaba; by Qwen; listed on OpenRouter | $0.03 | $0.006 | $0.13 | n/a | n/a | 1M |
| Claude Opus 5 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 5 | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 5 | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 5 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 5 (batch) | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Ling 3.0 Flash | Novita; by inclusionAI; listed on OpenRouter | $0.021 | $0.0042 | $0.063 | n/a | n/a | 262.1K |
| Ling 3.0 Flash | DeepInfra; by inclusionAI; listed on OpenRouter | $0.06 | $0.012 | $0.18 | n/a | n/a | 131.1K |
| Gemini 3.6 Flash | Google Vertex; by Google; listed on OpenRouter | $0.75 | $0.075 | $3.75 | n/a | n/a | 1M |
| Gemini 3.6 Flash (batch) | Google Vertex; by Google; listed on OpenRouter | $0.75 | $0.075 | $3.75 | n/a | n/a | 1M |
| Gemini 3.5 Flash Lite | Google Vertex; by Google; listed on OpenRouter | $0.30 | $0.03 | $2.50 | n/a | n/a | 1M |
| Gemini 3.5 Flash Lite (batch) | Google Vertex; by Google; listed on OpenRouter | $0.30 | $0.03 | $2.50 | n/a | n/a | 1M |
| LongCat 2.0 | AtlasCloud; by Meituan; listed on OpenRouter | $0.30 | $0.006 | $1.20 | n/a | n/a | 1M |
| Kimi K3 | Relace; by Moonshot AI; listed on OpenRouter | $0.1824 | $0.18 | $9.3388 | n/a | n/a | 1M |
| Kimi K3 | Wafer; by Moonshot AI; listed on OpenRouter | $0.276 | $0.13 | $9.60 | n/a | n/a | 1M |
| Kimi K3 | InferenceNet; by Moonshot AI; listed on OpenRouter | $0.234 | $0.195 | $15 | n/a | n/a | 1M |
| Kimi K3 | Sail Research; by Moonshot AI; listed on OpenRouter | $0.276 | $0.138 | $13.264 | n/a | n/a | 1M |
| Kimi K3 | Makora; by Moonshot AI; listed on OpenRouter | $0.80 | $0.30 | $16 | n/a | n/a | 1M |
| Kimi K3 | ParetoAlpha; by Moonshot AI; listed on OpenRouter | $1 | $0.225 | $11.25 | n/a | n/a | 1M |
| Kimi K3 | Chutes; by Moonshot AI; listed on OpenRouter | $1.85 | $0.185 | $9.50 | n/a | n/a | 1M |
| Kimi K3 | Decart; by Moonshot AI; listed on OpenRouter | $1.95 | $0.195 | $9.75 | n/a | n/a | 1M |
| Kimi K3 | Io Net; by Moonshot AI; listed on OpenRouter | $2.3 | $0.23 | $14 | n/a | n/a | 1M |
| Kimi K3 | Phala; by Moonshot AI; listed on OpenRouter | $2.55 | $0.255 | $12.75 | n/a | n/a | 1M |
| Kimi K3 | DigitalOcean; by Moonshot AI; listed on OpenRouter | $2.55 | $0.255 | $12.95 | n/a | n/a | 1M |
| Kimi K3 | DeepInfra; by Moonshot AI; listed on OpenRouter | $2.85 | $0.285 | $14.25 | n/a | n/a | 1M |
| Kimi K3 | Parasail; by Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 | Modal; by Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 | Fireworks; by Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 | BaseTen; by Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 | Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 | Alibaba; by Moonshot AI; listed on OpenRouter | $3.45 | $0.345 | $17.25 | n/a | n/a | 1M |
| Kimi K3 (batch) | Relace; by Moonshot AI; listed on OpenRouter | $0.1824 | $0.18 | $9.3388 | n/a | n/a | 1M |
| Kimi K3 (batch) | Wafer; by Moonshot AI; listed on OpenRouter | $0.276 | $0.13 | $9.60 | n/a | n/a | 1M |
| Kimi K3 (batch) | InferenceNet; by Moonshot AI; listed on OpenRouter | $0.234 | $0.195 | $15 | n/a | n/a | 1M |
| Kimi K3 (batch) | Sail Research; by Moonshot AI; listed on OpenRouter | $0.276 | $0.138 | $13.264 | n/a | n/a | 1M |
| Kimi K3 (batch) | Makora; by Moonshot AI; listed on OpenRouter | $0.80 | $0.30 | $16 | n/a | n/a | 1M |
| Kimi K3 (batch) | ParetoAlpha; by Moonshot AI; listed on OpenRouter | $1 | $0.225 | $11.25 | n/a | n/a | 1M |
| Kimi K3 (batch) | Chutes; by Moonshot AI; listed on OpenRouter | $1.85 | $0.185 | $9.50 | n/a | n/a | 1M |
| Kimi K3 (batch) | Decart; by Moonshot AI; listed on OpenRouter | $1.95 | $0.195 | $9.75 | n/a | n/a | 1M |
| Kimi K3 (batch) | Io Net; by Moonshot AI; listed on OpenRouter | $2.3 | $0.23 | $14 | n/a | n/a | 1M |
| Kimi K3 (batch) | Phala; by Moonshot AI; listed on OpenRouter | $2.55 | $0.255 | $12.75 | n/a | n/a | 1M |
| Kimi K3 (batch) | DigitalOcean; by Moonshot AI; listed on OpenRouter | $2.55 | $0.255 | $12.95 | n/a | n/a | 1M |
| Kimi K3 (batch) | DeepInfra; by Moonshot AI; listed on OpenRouter | $2.85 | $0.285 | $14.25 | n/a | n/a | 1M |
| Kimi K3 (batch) | Parasail; by Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 (batch) | Modal; by Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 (batch) | Fireworks; by Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 (batch) | BaseTen; by Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 (batch) | Moonshot AI; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Kimi K3 (batch) | Alibaba; by Moonshot AI; listed on OpenRouter | $3.45 | $0.345 | $17.25 | n/a | n/a | 1M |
| Muse Spark 1.1 | Meta; listed on OpenRouter | $1.25 | $0.15 | $4.25 | n/a | n/a | 1M |
| GPT-5.6 Luna Pro | Azure; by OpenAI; listed on OpenRouter | $0.20 | $0.02 | $1.20 | n/a | n/a | 1.1M |
| GPT-5.6 Luna Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $0.20 | $0.02 | $1.20 | n/a | n/a | 1.1M |
| GPT-5.6 Luna | Azure; by OpenAI; listed on OpenRouter | $0.20 | $0.02 | $1.20 | n/a | n/a | 1.1M |
| GPT-5.6 Luna (batch) | Azure; by OpenAI; listed on OpenRouter | $0.20 | $0.02 | $1.20 | n/a | n/a | 1.1M |
| GPT-5.6 Terra Pro | Azure; by OpenAI; listed on OpenRouter | $2 | $0.20 | $12 | n/a | n/a | 1.1M |
| GPT-5.6 Terra Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $2 | $0.20 | $12 | n/a | n/a | 1.1M |
| GPT-5.6 Terra | Azure; by OpenAI; listed on OpenRouter | $2 | $0.20 | $12 | n/a | n/a | 1.1M |
| GPT-5.6 Terra (batch) | Azure; by OpenAI; listed on OpenRouter | $2 | $0.20 | $12 | n/a | n/a | 1.1M |
| GPT-5.6 Sol Pro | Azure; by OpenAI; listed on OpenRouter | $4 | $0.40 | $20 | n/a | n/a | 1.1M |
| GPT-5.6 Sol Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $4 | $0.40 | $20 | n/a | n/a | 1.1M |
| GPT-5.6 Sol | Azure; by OpenAI; listed on OpenRouter | $4 | $0.40 | $20 | n/a | n/a | 1.1M |
| GPT-5.6 Sol (batch) | Azure; by OpenAI; listed on OpenRouter | $4 | $0.40 | $20 | n/a | n/a | 1.1M |
| Aion-3.0-Mini | AionLabs; listed on OpenRouter | $0.70 | $0.18 | $1.40 | n/a | n/a | 131.1K |
| Aion-3.0 | AionLabs; listed on OpenRouter | $3 | $0.75 | $6 | n/a | n/a | 131.1K |
| Hy3 | Tencent; listed on OpenRouter | $0.0825 | $0.020625 | $0.33 | n/a | n/a | 262.1K |
| Hy3 | Novita; by Tencent; listed on OpenRouter | $0.14 | $0.035 | $0.58 | n/a | n/a | 262.1K |
| Hy3 | GMICloud; by Tencent; listed on OpenRouter | $0.14 | $0.035 | $0.58 | n/a | n/a | 262.1K |
| Hy3 | Phala; by Tencent; listed on OpenRouter | $0.15 | $0.04 | $0.64 | n/a | n/a | 262.1K |
| Hy3 | AtlasCloud; by Tencent; listed on OpenRouter | $0.20 | $0.05 | $0.80 | n/a | n/a | 262.1K |
| Claude Sonnet 5 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5 | Azure; by Anthropic; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5 | Google Vertex; by Anthropic; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5 (batch) | Azure; by Anthropic; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Claude Sonnet 5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $2 | $0.20 | $10 | n/a | n/a | 1M |
| Fugu Ultra | Sakana AI; by Sakana; listed on OpenRouter | $5 | $0.50 | $30 | n/a | n/a | 1M |
| GLM 5.2 | Wafer; by Z.ai; listed on OpenRouter | $0.006 | $0.005 | $6.80 | n/a | n/a | 1M |
| GLM 5.2 | Relace; by Z.ai; listed on OpenRouter | $0.0065 | $0.0065 | $7.9898 | n/a | n/a | 1M |
| GLM 5.2 | Cloudflare; by Z.ai; listed on OpenRouter | $0.19 | $0.19 | $8 | n/a | n/a | 262.1K |
| GLM 5.2 | DeepInfra; by Z.ai; listed on OpenRouter | $0.5625 | $0.105 | $1.80 | n/a | n/a | 1M |
| GLM 5.2 | Ambient; by Z.ai; listed on OpenRouter | $0.60 | $0.15 | $2 | n/a | n/a | 202.8K |
| GLM 5.2 | StreamLake; by Z.ai; listed on OpenRouter | $0.61306 | $0.113854 | $1.92676 | n/a | n/a | 1M |
| GLM 5.2 | CoreWeave; by Z.ai; listed on OpenRouter | $0.62 | $0.14 | $2.42 | n/a | n/a | 1M |
| GLM 5.2 | Novita; by Z.ai; listed on OpenRouter | $0.6496 | $0.12064 | $2.0416 | n/a | n/a | 1M |
| GLM 5.2 | DigitalOcean; by Z.ai; listed on OpenRouter | $0.70 | $0.105 | $2.2 | n/a | n/a | 1M |
| GLM 5.2 | SiliconFlow; by Z.ai; listed on OpenRouter | $0.70 | $0.13 | $2.2 | n/a | n/a | 1M |
| GLM 5.2 | AtlasCloud; by Z.ai; listed on OpenRouter | $0.938 | $0.1742 | $2.948 | n/a | n/a | 1M |
| GLM 5.2 | Alibaba; by Z.ai; listed on OpenRouter | $0.966 | $0.1932 | $3.036 | n/a | n/a | 1M |
| GLM 5.2 | Morph; by Z.ai; listed on OpenRouter | $1.245 | $0.42 | $2.542 | n/a | n/a | 1M |
| GLM 5.2 | Nebius; by Z.ai; listed on OpenRouter | $1.40 | $0.14 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 | BaseTen; by Z.ai; listed on OpenRouter | $1.40 | $0.14 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 | Inceptron; by Z.ai; listed on OpenRouter | $1.40 | $0.19 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 | Baidu; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 | Venice; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 | GMICloud; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 | Parasail; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 262.1K |
| GLM 5.2 | Friendli; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Wafer; by Z.ai; listed on OpenRouter | $0.006 | $0.005 | $6.80 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Relace; by Z.ai; listed on OpenRouter | $0.0065 | $0.0065 | $7.9898 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Cloudflare; by Z.ai; listed on OpenRouter | $0.19 | $0.19 | $8 | n/a | n/a | 262.1K |
| GLM 5.2 (batch) | DeepInfra; by Z.ai; listed on OpenRouter | $0.5625 | $0.105 | $1.80 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Ambient; by Z.ai; listed on OpenRouter | $0.60 | $0.15 | $2 | n/a | n/a | 202.8K |
| GLM 5.2 (batch) | StreamLake; by Z.ai; listed on OpenRouter | $0.61306 | $0.113854 | $1.92676 | n/a | n/a | 1M |
| GLM 5.2 (batch) | CoreWeave; by Z.ai; listed on OpenRouter | $0.62 | $0.14 | $2.42 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Novita; by Z.ai; listed on OpenRouter | $0.6496 | $0.12064 | $2.0416 | n/a | n/a | 1M |
| GLM 5.2 (batch) | DigitalOcean; by Z.ai; listed on OpenRouter | $0.70 | $0.105 | $2.2 | n/a | n/a | 1M |
| GLM 5.2 (batch) | SiliconFlow; by Z.ai; listed on OpenRouter | $0.70 | $0.13 | $2.2 | n/a | n/a | 1M |
| GLM 5.2 (batch) | AtlasCloud; by Z.ai; listed on OpenRouter | $0.938 | $0.1742 | $2.948 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Alibaba; by Z.ai; listed on OpenRouter | $0.966 | $0.1932 | $3.036 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Morph; by Z.ai; listed on OpenRouter | $1.245 | $0.42 | $2.542 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Nebius; by Z.ai; listed on OpenRouter | $1.40 | $0.14 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 (batch) | BaseTen; by Z.ai; listed on OpenRouter | $1.40 | $0.14 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Inceptron; by Z.ai; listed on OpenRouter | $1.40 | $0.19 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Baidu; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Venice; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 (batch) | GMICloud; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| GLM 5.2 (batch) | Parasail; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 262.1K |
| GLM 5.2 (batch) | Friendli; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 1M |
| Kimi K2.7 Code | CoreWeave; by Moonshot AI; listed on OpenRouter | $0.60 | $0.12 | $3 | n/a | n/a | 262.1K |
| Kimi K2.7 Code | Inceptron; by Moonshot AI; listed on OpenRouter | $0.60 | $0.16 | $3.50 | n/a | n/a | 262.1K |
| Kimi K2.7 Code | StreamLake; by Moonshot AI; listed on OpenRouter | $0.7125 | $0.1425 | $3 | n/a | n/a | 256K |
| Kimi K2.7 Code | Venice; by Moonshot AI; listed on OpenRouter | $0.75 | $0.16 | $3.50 | n/a | n/a | 256K |
| Kimi K2.7 Code | SiliconFlow; by Moonshot AI; listed on OpenRouter | $0.85916 | $0.17993 | $3.80 | n/a | n/a | 262.1K |
| Kimi K2.7 Code | Novita; by Moonshot AI; listed on OpenRouter | $0.912 | $0.1824 | $3.84 | n/a | n/a | 262.1K |
| Kimi K2.7 Code | GMICloud; by Moonshot AI; listed on OpenRouter | $0.95 | $0.19 | $4 | n/a | n/a | 262.1K |
| Kimi K2.7 Code | Alibaba; by Moonshot AI; listed on OpenRouter | $0.95 | $0.19 | $4 | n/a | n/a | 262.1K |
| Kimi K2.7 Code | Cloudflare; by Moonshot AI; listed on OpenRouter | $0.95 | $0.19 | $4 | n/a | n/a | 262.1K |
| Kimi K2.7 Code | Moonshot AI; listed on OpenRouter | $0.95 | $0.19 | $4 | n/a | n/a | 262.1K |
| Claude Fable 5 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1M |
| Claude Fable 5 | Azure; by Anthropic; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1M |
| Claude Fable 5 | Google Vertex; by Anthropic; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1M |
| Claude Fable 5 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1M |
| Claude Fable 5 (batch) | Azure; by Anthropic; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1M |
| Claude Fable 5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $10 | $1 | $50 | n/a | n/a | 1M |
| Nemotron 3.5 Content Safety | DeepInfra; by NVIDIA; listed on OpenRouter | $0.20 | n/a | $0.20 | n/a | n/a | 131.1K |
| Nemotron 3 Ultra | DeepInfra; by NVIDIA; listed on OpenRouter | $0.50 | $0.10 | $2.2 | n/a | n/a | 262.1K |
| Nemotron 3 Ultra | BaseTen; by NVIDIA; listed on OpenRouter | $0.60 | $0.12 | $2.40 | n/a | n/a | 202.8K |
| Nemotron 3 Ultra | Venice; by NVIDIA; listed on OpenRouter | $0.625 | $0.1875 | $3.125 | n/a | n/a | 256K |
| Qwen3.7 Plus | Alibaba; by Qwen; listed on OpenRouter | $0.32 | $0.064 | $1.28 | n/a | n/a | 1M |
| Qwen3.7 Plus (batch) | Alibaba; by Qwen; listed on OpenRouter | $0.32 | $0.064 | $1.28 | n/a | n/a | 1M |
| MiniMax M3 | CoreWeave; by MiniMax; listed on OpenRouter | $0.23 | $0.05 | $0.96 | n/a | n/a | 262.1K |
| MiniMax M3 | GMICloud; by MiniMax; listed on OpenRouter | $0.24 | $0.048 | $0.96 | n/a | n/a | 1M |
| MiniMax M3 | DeepInfra; by MiniMax; listed on OpenRouter | $0.28 | $0.056 | $1.1 | n/a | n/a | 524.3K |
| MiniMax M3 | StreamLake; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 1M |
| MiniMax M3 | Venice; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 524.3K |
| MiniMax M3 | Parasail; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 1M |
| MiniMax M3 | Novita; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 1M |
| MiniMax M3 | ModelRun; by MiniMax; listed on OpenRouter | $0.30 | $0.05 | $1.30 | n/a | n/a | 1M |
| MiniMax M3 (batch) | CoreWeave; by MiniMax; listed on OpenRouter | $0.23 | $0.05 | $0.96 | n/a | n/a | 262.1K |
| MiniMax M3 (batch) | GMICloud; by MiniMax; listed on OpenRouter | $0.24 | $0.048 | $0.96 | n/a | n/a | 1M |
| MiniMax M3 (batch) | DeepInfra; by MiniMax; listed on OpenRouter | $0.28 | $0.056 | $1.1 | n/a | n/a | 524.3K |
| MiniMax M3 (batch) | StreamLake; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 1M |
| MiniMax M3 (batch) | Venice; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 524.3K |
| MiniMax M3 (batch) | Parasail; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 1M |
| MiniMax M3 (batch) | Novita; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 1M |
| MiniMax M3 (batch) | ModelRun; by MiniMax; listed on OpenRouter | $0.30 | $0.05 | $1.30 | n/a | n/a | 1M |
| Step 3.7 Flash | Novita; by StepFun; listed on OpenRouter | $0.20 | $0.04 | $1.15 | n/a | n/a | 262.1K |
| Step 3.7 Flash | StepFun; listed on OpenRouter | $0.20 | $0.04 | $1.15 | n/a | n/a | 256K |
| Claude Opus 4.8 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.8 | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.8 | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.8 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.8 (batch) | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.8 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Qwen3.7 Max | Alibaba; by Qwen; listed on OpenRouter | $1.475 | $0.295 | $4.425 | n/a | n/a | 1M |
| Gemini 3.5 Flash | Google Vertex; by Google; listed on OpenRouter | $1.50 | $0.15 | $9 | n/a | n/a | 1M |
| Gemini 3.5 Flash (batch) | Google Vertex; by Google; listed on OpenRouter | $1.50 | $0.15 | $9 | n/a | n/a | 1M |
| Perceptron Mk1 | Perceptron; listed on OpenRouter | $0.15 | n/a | $1.50 | n/a | n/a | 32.8K |
| Gemini 3.1 Flash Lite | Google Vertex; by Google; listed on OpenRouter | $0.25 | $0.025 | $1.50 | n/a | n/a | 1M |
| Gemini 3.1 Flash Lite (batch) | Google Vertex; by Google; listed on OpenRouter | $0.25 | $0.025 | $1.50 | n/a | n/a | 1M |
| Qwen3.5 Plus 2026-04-20 | Alibaba; by Qwen; listed on OpenRouter | $0.30 | n/a | $1.80 | n/a | n/a | 1M |
| Qwen3.6 Flash | Alibaba; by Qwen; listed on OpenRouter | $0.1875 | n/a | $1.125 | n/a | n/a | 1M |
| Qwen3.6 35B A3B | PrimeIntellect; by Qwen; listed on OpenRouter | $0.012 | $0.01 | $0.106 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | Darkbloom; by Qwen; listed on OpenRouter | $0.05 | $0.025 | $0.70 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | CoreWeave; by Qwen; listed on OpenRouter | $0.10 | $0.035 | $0.89 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | AkashML; by Qwen; listed on OpenRouter | $0.10 | $0.05 | $0.90 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | Parasail; by Qwen; listed on OpenRouter | $0.10 | $0.025 | $0.90 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | DeepInfra; by Qwen; listed on OpenRouter | $0.10 | $0.10 | $0.95 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | DekaLLM; by Qwen; listed on OpenRouter | $0.10 | $0.05 | $1 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | Venice; by Qwen; listed on OpenRouter | $0.10 | n/a | $1 | n/a | n/a | 256K |
| Qwen3.6 35B A3B | Phala; by Qwen; listed on OpenRouter | $0.20 | $0.056 | $1.27 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | SiliconFlow; by Qwen; listed on OpenRouter | $0.24 | $0.15 | $1.80 | n/a | n/a | 262.1K |
| Qwen3.6 35B A3B | AtlasCloud; by Qwen; listed on OpenRouter | $0.25 | $0.25 | $1.25 | n/a | n/a | 262.1K |
| Qwen3.6 27B | Reka; by Qwen; listed on OpenRouter | $0.20 | $0.20 | $3 | n/a | n/a | 262.1K |
| Qwen3.6 27B | Chutes; by Qwen; listed on OpenRouter | $0.25 | $0.025 | $2 | n/a | n/a | 262.1K |
| Qwen3.6 27B | SiliconFlow; by Qwen; listed on OpenRouter | $0.30 | n/a | $3.20 | n/a | n/a | 262.1K |
| Qwen3.6 27B | DeepInfra; by Qwen; listed on OpenRouter | $0.32 | n/a | $3.20 | n/a | n/a | 262.1K |
| Qwen3.6 27B | Phala; by Qwen; listed on OpenRouter | $0.32 | $0.03 | $3.25 | n/a | n/a | 262.1K |
| Qwen3.6 27B | Venice; by Qwen; listed on OpenRouter | $0.325 | n/a | $3.25 | n/a | n/a | 256K |
| Qwen3.6 27B | Alibaba; by Qwen; listed on OpenRouter | $0.45 | n/a | $2.70 | n/a | n/a | 262.1K |
| GPT-5.5 | Azure; by OpenAI; listed on OpenRouter | $5 | $0.50 | $30 | n/a | n/a | 1.1M |
| GPT-5.5 (batch) | Azure; by OpenAI; listed on OpenRouter | $5 | $0.50 | $30 | n/a | n/a | 1.1M |
| DeepSeek V4 Pro 0423 | StreamLake; by DeepSeek; listed on OpenRouter | $0.888966 | $0.07408 | $1.777932 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | Baidu; by DeepSeek; listed on OpenRouter | $0.89063 | $0.07378 | $1.78126 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | Relace; by DeepSeek; listed on OpenRouter | $0.90 | $0.25 | $1.50 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | GMICloud; by DeepSeek; listed on OpenRouter | $0.957 | $0.07975 | $1.914 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | SiliconFlow; by DeepSeek; listed on OpenRouter | $1 | $0.135 | $1.50 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | DigitalOcean; by DeepSeek; listed on OpenRouter | $1.044 | $0.2088 | $2.088 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | DeepInfra; by DeepSeek; listed on OpenRouter | $1.30 | $0.10 | $2.60 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | Alibaba; by DeepSeek; listed on OpenRouter | $1.416 | $0.118 | $2.832 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | Novita; by DeepSeek; listed on OpenRouter | $1.60 | $0.135 | $3.20 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | Venice; by DeepSeek; listed on OpenRouter | $1.65 | $0.33 | $3.301 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | AtlasCloud; by DeepSeek; listed on OpenRouter | $1.68 | $0.13 | $3.38 | n/a | n/a | 1M |
| DeepSeek V4 Pro 0423 | Azure; by DeepSeek; listed on OpenRouter | $1.91 | $0.16 | $3.83 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | Relace; by DeepSeek; listed on OpenRouter | $0.0474 | $0.0474 | $0.45 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | Baidu; by DeepSeek; listed on OpenRouter | $0.04998 | $0.009996 | $0.09996 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | StreamLake; by DeepSeek; listed on OpenRouter | $0.07 | $0.014 | $0.14 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | DeepInfra; by DeepSeek; listed on OpenRouter | $0.09 | $0.018 | $0.18 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | GMICloud; by DeepSeek; listed on OpenRouter | $0.091 | $0.0182 | $0.182 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | Venice; by DeepSeek; listed on OpenRouter | $0.0966 | $0.0196 | $0.1925 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | DigitalOcean; by DeepSeek; listed on OpenRouter | $0.098 | $0.0196 | $0.196 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | Fireworks; by DeepSeek; listed on OpenRouter | $0.105 | $0.021 | $0.21 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | Wafer; by DeepSeek; listed on OpenRouter | $0.12 | $0.024 | $0.17 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.13 | $0.028 | $0.28 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | AtlasCloud; by DeepSeek; listed on OpenRouter | $0.14 | $0.028 | $0.28 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | Parasail; by DeepSeek; listed on OpenRouter | $0.14 | $0.07 | $0.28 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | Mancer 2; by DeepSeek; listed on OpenRouter | $0.15 | $0.075 | $0.50 | n/a | n/a | 1M |
| DeepSeek V4 Flash 0423 | Azure; by DeepSeek; listed on OpenRouter | $0.21 | $0.031 | $0.56 | n/a | n/a | 1M |
| Hy3 preview | GMICloud; by Tencent; listed on OpenRouter | $0.18 | $0.06 | $0.60 | n/a | n/a | 262.1K |
| MiMo-V2.5-Pro | AtlasCloud; by Xiaomi; listed on OpenRouter | $0.435 | $0.0036 | $0.87 | n/a | n/a | 1M |
| MiMo-V2.5-Pro | DigitalOcean; by Xiaomi; listed on OpenRouter | $0.48 | $0.096 | $1.80 | n/a | n/a | 262.1K |
| MiMo-V2.5 | Venice; by Xiaomi; listed on OpenRouter | $0.40 | $0.08 | $2 | n/a | n/a | 1M |
| Kimi K2.6 | Baidu; by Moonshot AI; listed on OpenRouter | $0.3895 | $0.0656 | $1.64 | n/a | n/a | 262.1K |
| Kimi K2.6 | Chutes; by Moonshot AI; listed on OpenRouter | $0.39 | $0.039 | $1.80 | n/a | n/a | 262.1K |
| Kimi K2.6 | DigitalOcean; by Moonshot AI; listed on OpenRouter | $0.57 | $0.114 | $2.40 | n/a | n/a | 262.1K |
| Kimi K2.6 | StreamLake; by Moonshot AI; listed on OpenRouter | $0.5985 | $0.1008 | $2.52 | n/a | n/a | 256K |
| Kimi K2.6 | CoreWeave; by Moonshot AI; listed on OpenRouter | $0.65 | $0.15 | $3.41 | n/a | n/a | 262.1K |
| Kimi K2.6 | Crusoe; by Moonshot AI; listed on OpenRouter | $0.70 | $0.35 | $3.50 | n/a | n/a | 262.1K |
| Kimi K2.6 | DeepInfra; by Moonshot AI; listed on OpenRouter | $0.75 | $0.15 | $3.50 | n/a | n/a | 262.1K |
| Kimi K2.6 | Venice; by Moonshot AI; listed on OpenRouter | $0.75 | $0.16 | $3.50 | n/a | n/a | 256K |
| Kimi K2.6 | Parasail; by Moonshot AI; listed on OpenRouter | $0.75 | $0.16 | $3.50 | n/a | n/a | 262.1K |
| Kimi K2.6 | SiliconFlow; by Moonshot AI; listed on OpenRouter | $0.77 | $0.14 | $3.40 | n/a | n/a | 262.1K |
| Kimi K2.6 | Inceptron; by Moonshot AI; listed on OpenRouter | $0.799 | $0.2092 | $3.75 | n/a | n/a | 262.1K |
| Kimi K2.6 | Novita; by Moonshot AI; listed on OpenRouter | $0.80 | $0.16 | $3.40 | n/a | n/a | 262.1K |
| Kimi K2.6 | AtlasCloud; by Moonshot AI; listed on OpenRouter | $0.95 | $0.16 | $4 | n/a | n/a | 262.1K |
| Kimi K2.6 | Cloudflare; by Moonshot AI; listed on OpenRouter | $0.95 | $0.16 | $4 | n/a | n/a | 262.1K |
| Kimi K2.6 | Moonshot AI; listed on OpenRouter | $0.95 | $0.16 | $4 | n/a | n/a | 262.1K |
| Kimi K2.6 | Phala; by Moonshot AI; listed on OpenRouter | $1.09 | $0.37 | $4.6 | n/a | n/a | 262.1K |
| Claude Opus 4.7 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.7 | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.7 | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.7 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.7 (batch) | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.7 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| GLM 5.1 | Chutes; by Z.ai; listed on OpenRouter | $0.90 | $0.09 | $2.90 | n/a | n/a | 202.8K |
| GLM 5.1 | StreamLake; by Z.ai; listed on OpenRouter | $0.966 | $0.1794 | $3.036 | n/a | n/a | 200K |
| GLM 5.1 | SiliconFlow; by Z.ai; listed on OpenRouter | $1.19 | $0.60 | $3.74 | n/a | n/a | 204.8K |
| GLM 5.1 | Phala; by Z.ai; listed on OpenRouter | $1.21 | $0.60 | $4.20 | n/a | n/a | 202.8K |
| GLM 5.1 | AtlasCloud; by Z.ai; listed on OpenRouter | $1.26 | $0.234 | $3.96 | n/a | n/a | 202.8K |
| GLM 5.1 | Alibaba; by Z.ai; listed on OpenRouter | $1.33 | $0.247 | $4.18 | n/a | n/a | 202.7K |
| GLM 5.1 | Novita; by Z.ai; listed on OpenRouter | $1.38 | $0.26 | $4.4 | n/a | n/a | 204.8K |
| GLM 5.1 | Baidu; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 202.8K |
| GLM 5.1 | GMICloud; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 202.8K |
| GLM 5.1 | Friendli; by Z.ai; listed on OpenRouter | $1.40 | $0.26 | $4.4 | n/a | n/a | 202.8K |
| GLM 5.1 | Venice; by Z.ai; listed on OpenRouter | $1.4014 | $0.26026 | $4.4044 | n/a | n/a | 200K |
| Gemma 4 26B A4B  | PrimeIntellect; by Google; listed on OpenRouter | $0.038 | $0.019 | $0.113 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | Darkbloom; by Google; listed on OpenRouter | $0.04 | $0.01 | $0.15 | n/a | n/a | 131.1K |
| Gemma 4 26B A4B  | DekaLLM; by Google; listed on OpenRouter | $0.06 | $0.04 | $0.33 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | AkashML; by Google; listed on OpenRouter | $0.07 | $0.046 | $0.24 | n/a | n/a | 131.1K |
| Gemma 4 26B A4B  | DeepInfra; by Google; listed on OpenRouter | $0.07 | n/a | $0.34 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | NextBit; by Google; listed on OpenRouter | $0.0765 | $0.0425 | $0.255 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | Makora; by Google; listed on OpenRouter | $0.08 | $0.046 | $0.10 | n/a | n/a | 256K |
| Gemma 4 26B A4B  | Io Net; by Google; listed on OpenRouter | $0.081 | $0.0369 | $0.279 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | CoreWeave; by Google; listed on OpenRouter | $0.10 | $0.05 | $0.30 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | Cloudflare; by Google; listed on OpenRouter | $0.10 | $0.05 | $0.30 | n/a | n/a | 256K |
| Gemma 4 26B A4B  | Venice; by Google; listed on OpenRouter | $0.13 | $0.05 | $0.40 | n/a | n/a | 256K |
| Gemma 4 26B A4B  | Parasail; by Google; listed on OpenRouter | $0.13 | $0.05 | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | Novita; by Google; listed on OpenRouter | $0.13 | n/a | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | SiliconFlow; by Google; listed on OpenRouter | $0.14 | $0.05 | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 26B A4B  | Google Vertex; by Google; listed on OpenRouter | $0.15 | n/a | $0.60 | n/a | n/a | 262.1K |
| Gemma 4 31B | CoreWeave; by Google; listed on OpenRouter | $0.10 | $0.10 | $0.34 | n/a | n/a | 262.1K |
| Gemma 4 31B | Venice; by Google; listed on OpenRouter | $0.12 | $0.09 | $0.36 | n/a | n/a | 256K |
| Gemma 4 31B | Chutes; by Google; listed on OpenRouter | $0.12 | $0.012 | $0.37 | n/a | n/a | 131.1K |
| Gemma 4 31B | DekaLLM; by Google; listed on OpenRouter | $0.14 | $0.05 | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 31B | Crusoe; by Google; listed on OpenRouter | $0.14 | $0.14 | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 31B | Friendli; by Google; listed on OpenRouter | $0.14 | n/a | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 31B | Novita; by Google; listed on OpenRouter | $0.14 | n/a | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 31B | NextBit; by Google; listed on OpenRouter | $0.15 | $0.06 | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 31B | Parasail; by Google; listed on OpenRouter | $0.15 | $0.06 | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 31B | DeepInfra; by Google; listed on OpenRouter | $0.20 | n/a | $0.40 | n/a | n/a | 262.1K |
| Gemma 4 31B | Modal; by Google; listed on OpenRouter | $0.34 | $0.10 | $0.10 | n/a | n/a | 131.1K |
| Gemma 4 31B | Io Net; by Google; listed on OpenRouter | $0.361 | $0.1805 | $1.0925 | n/a | n/a | 262.1K |
| Gemma 4 31B | SambaNova; by Google; listed on OpenRouter | $0.38 | n/a | $1.15 | n/a | n/a | 131.1K |
| Gemma 4 31B | PrimeIntellect; by Google; listed on OpenRouter | $0.40 | $0.20 | $0.67 | n/a | n/a | 262.1K |
| Gemma 4 31B | ModelRun; by Google; listed on OpenRouter | $0.75 | $0.20 | $1 | n/a | n/a | 262.1K |
| Gemma 4 31B | SiliconFlow; by Google; listed on OpenRouter | $0.75 | $0.25 | $1 | n/a | n/a | 262.1K |
| Qwen3.6 Plus | Alibaba; by Qwen; listed on OpenRouter | $0.325 | n/a | $1.95 | n/a | n/a | 1M |
| Reka Edge | Reka; by rekaai; listed on OpenRouter | $0.10 | n/a | $0.10 | n/a | n/a | 16.4K |
| MiniMax M2.7 | GMICloud; by MiniMax; listed on OpenRouter | $0.21 | $0.042 | $0.84 | n/a | n/a | 196.6K |
| MiniMax M2.7 | Groq; by MiniMax; listed on OpenRouter | $0.60 | n/a | $1.80 | n/a | n/a | 196.6K |
| GPT-5.4 Nano | Azure; by OpenAI; listed on OpenRouter | $0.20 | $0.02 | $1.25 | n/a | n/a | 400K |
| GPT-5.4 Nano (batch) | Azure; by OpenAI; listed on OpenRouter | $0.20 | $0.02 | $1.25 | n/a | n/a | 400K |
| GPT-5.4 Mini | Azure; by OpenAI; listed on OpenRouter | $0.75 | $0.075 | $4.50 | n/a | n/a | 400K |
| GPT-5.4 Mini (batch) | Azure; by OpenAI; listed on OpenRouter | $0.75 | $0.075 | $4.50 | n/a | n/a | 400K |
| Nemotron 3 Super | DeepInfra; by NVIDIA; listed on OpenRouter | $0.075 | n/a | $0.40 | n/a | n/a | 262.1K |
| Nemotron 3 Super | DekaLLM; by NVIDIA; listed on OpenRouter | $0.08 | n/a | $0.45 | n/a | n/a | 262.1K |
| Nemotron 3 Super | Crusoe; by NVIDIA; listed on OpenRouter | $0.30 | $0.15 | $2.40 | n/a | n/a | 1M |
| Seed-2.0-Lite | Seed; by ByteDance; listed on OpenRouter | $0.25 | n/a | $2 | n/a | n/a | 262.1K |
| Qwen3.5-9B | PrimeIntellect; by Qwen; listed on OpenRouter | $0.056 | $0.051 | $0.083 | n/a | n/a | 262.1K |
| Qwen3.5-9B | Darkbloom; by Qwen; listed on OpenRouter | $0.08 | $0.04 | $0.13 | n/a | n/a | 262.1K |
| Qwen3.5-9B | SiliconFlow; by Qwen; listed on OpenRouter | $0.10 | n/a | $0.15 | n/a | n/a | 262.1K |
| Qwen3.5-9B | DeepInfra; by Qwen; listed on OpenRouter | $0.10 | n/a | $0.15 | n/a | n/a | 262.1K |
| Qwen3.5-9B | Parasail; by Qwen; listed on OpenRouter | $0.10 | n/a | $0.25 | n/a | n/a | 262.1K |
| GPT-5.4 Pro | Azure; by OpenAI; listed on OpenRouter | $30 | n/a | $180 | n/a | n/a | 1.1M |
| GPT-5.4 Pro (batch) | Azure; by OpenAI; listed on OpenRouter | $30 | n/a | $180 | n/a | n/a | 1.1M |
| GPT-5.4 | Azure; by OpenAI; listed on OpenRouter | $2.50 | $0.25 | $15 | n/a | n/a | 1.1M |
| GPT-5.4 (batch) | Azure; by OpenAI; listed on OpenRouter | $2.50 | $0.25 | $15 | n/a | n/a | 1.1M |
| Seed-2.0-Mini | Seed; by ByteDance; listed on OpenRouter | $0.10 | n/a | $0.40 | n/a | n/a | 262.1K |
| Qwen3.5-35B-A3B | Darkbloom; by Qwen; listed on OpenRouter | $0.065 | $0.0325 | $0.75 | n/a | n/a | 262.1K |
| Qwen3.5-35B-A3B | DeepInfra; by Qwen; listed on OpenRouter | $0.14 | $0.05 | $1 | n/a | n/a | 262.1K |
| Qwen3.5-35B-A3B | Parasail; by Qwen; listed on OpenRouter | $0.15 | $0.05 | $1 | n/a | n/a | 262.1K |
| Qwen3.5-35B-A3B | Venice; by Qwen; listed on OpenRouter | $0.15 | $0.05 | $1 | n/a | n/a | 256K |
| Qwen3.5-35B-A3B | Alibaba; by Qwen; listed on OpenRouter | $0.1625 | n/a | $1.30 | n/a | n/a | 262.1K |
| Qwen3.5-35B-A3B | AtlasCloud; by Qwen; listed on OpenRouter | $0.225 | $0.225 | $1.80 | n/a | n/a | 262.1K |
| Qwen3.5-35B-A3B | SiliconFlow; by Qwen; listed on OpenRouter | $0.24 | $0.15 | $1.80 | n/a | n/a | 262.1K |
| Qwen3.5-27B | Alibaba; by Qwen; listed on OpenRouter | $0.195 | n/a | $1.56 | n/a | n/a | 262.1K |
| Qwen3.5-27B | Chutes; by Qwen; listed on OpenRouter | $0.20 | $0.02 | $1.75 | n/a | n/a | 262.1K |
| Qwen3.5-27B | SiliconFlow; by Qwen; listed on OpenRouter | $0.25 | n/a | $2 | n/a | n/a | 262.1K |
| Qwen3.5-27B | DeepInfra; by Qwen; listed on OpenRouter | $0.26 | n/a | $2.60 | n/a | n/a | 262.1K |
| Qwen3.5-27B | AtlasCloud; by Qwen; listed on OpenRouter | $0.27 | $0.27 | $2.16 | n/a | n/a | 262.1K |
| Qwen3.5-27B | Phala; by Qwen; listed on OpenRouter | $0.30 | $0.03 | $2.40 | n/a | n/a | 262.1K |
| Qwen3.5-27B | Novita; by Qwen; listed on OpenRouter | $0.30 | n/a | $2.40 | n/a | n/a | 262.1K |
| Qwen3.5-122B-A10B | SiliconFlow; by Qwen; listed on OpenRouter | $0.26 | n/a | $2.08 | n/a | n/a | 262.1K |
| Qwen3.5-122B-A10B | Alibaba; by Qwen; listed on OpenRouter | $0.26 | n/a | $2.08 | n/a | n/a | 262.1K |
| Qwen3.5-122B-A10B | AtlasCloud; by Qwen; listed on OpenRouter | $0.30 | $0.30 | $2.40 | n/a | n/a | 262.1K |
| Qwen3.5-122B-A10B | Novita; by Qwen; listed on OpenRouter | $0.40 | n/a | $3.20 | n/a | n/a | 262.1K |
| Qwen3.5-Flash | Alibaba; by Qwen; listed on OpenRouter | $0.065 | n/a | $0.26 | n/a | n/a | 1M |
| GPT-5.3-Codex | Azure; by OpenAI; listed on OpenRouter | $1.75 | $0.175 | $14 | n/a | n/a | 400K |
| Aion-2.0 | AionLabs; listed on OpenRouter | $0.80 | $0.20 | $1.60 | n/a | n/a | 131.1K |
| Gemini 3.1 Pro Preview | Google Vertex; by Google; listed on OpenRouter | $2 | $0.20 | $12 | n/a | n/a | 1M |
| Gemini 3.1 Pro Preview (batch) | Google Vertex; by Google; listed on OpenRouter | $2 | $0.20 | $12 | n/a | n/a | 1M |
| Claude Sonnet 4.6 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Claude Sonnet 4.6 | Azure; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Claude Sonnet 4.6 | Google Vertex; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Claude Sonnet 4.6 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Claude Sonnet 4.6 (batch) | Azure; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Claude Sonnet 4.6 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Qwen3.5 Plus 2026-02-15 | Alibaba; by Qwen; listed on OpenRouter | $0.26 | n/a | $1.56 | n/a | n/a | 1M |
| Qwen3.5 397B A17B | Alibaba; by Qwen; listed on OpenRouter | $0.39 | n/a | $2.34 | n/a | n/a | 262.1K |
| Qwen3.5 397B A17B | DeepInfra; by Qwen; listed on OpenRouter | $0.45 | $0.22 | $3 | n/a | n/a | 262.1K |
| Qwen3.5 397B A17B | Parasail; by Qwen; listed on OpenRouter | $0.50 | $0.30 | $3.60 | n/a | n/a | 262.1K |
| Qwen3.5 397B A17B | DigitalOcean; by Qwen; listed on OpenRouter | $0.55 | $0.11 | $3.50 | n/a | n/a | 131.1K |
| Qwen3.5 397B A17B | Phala; by Qwen; listed on OpenRouter | $0.55 | $0.225 | $3.50 | n/a | n/a | 262.1K |
| Qwen3.5 397B A17B | AtlasCloud; by Qwen; listed on OpenRouter | $0.55 | $0.55 | $3.50 | n/a | n/a | 262.1K |
| Qwen3.5 397B A17B | StreamLake; by Qwen; listed on OpenRouter | $0.60 | $0.12 | $3.60 | n/a | n/a | 256K |
| Qwen3.5 397B A17B | GMICloud; by Qwen; listed on OpenRouter | $0.60 | n/a | $3.60 | n/a | n/a | 262.1K |
| Qwen3.5 397B A17B | Novita; by Qwen; listed on OpenRouter | $0.60 | n/a | $3.60 | n/a | n/a | 262.1K |
| Qwen3.5 397B A17B | Venice; by Qwen; listed on OpenRouter | $0.75 | n/a | $4.50 | n/a | n/a | 128K |
| MiniMax M2.5 | Venice; by MiniMax; listed on OpenRouter | $0.27 | $0.03 | $0.95 | n/a | n/a | 198K |
| MiniMax M2.5 | StreamLake; by MiniMax; listed on OpenRouter | $0.27 | $0.027 | $1.08 | n/a | n/a | 200K |
| MiniMax M2.5 | AtlasCloud; by MiniMax; listed on OpenRouter | $0.295 | $0.06 | $1.20 | n/a | n/a | 196.6K |
| MiniMax M2.5 | GMICloud; by MiniMax; listed on OpenRouter | $0.30 | $0.03 | $1.20 | n/a | n/a | 196.6K |
| MiniMax M2.5 | DigitalOcean; by MiniMax; listed on OpenRouter | $0.30 | $0.06 | $1.20 | n/a | n/a | 65.5K |
| GLM 5 | StreamLake; by Z.ai; listed on OpenRouter | $0.60 | $0.12 | $1.92 | n/a | n/a | 198K |
| GLM 5 | GMICloud; by Z.ai; listed on OpenRouter | $0.60 | $0.12 | $1.92 | n/a | n/a | 202.8K |
| GLM 5 | Baidu; by Z.ai; listed on OpenRouter | $0.70 | $0.14 | $2.24 | n/a | n/a | 202.8K |
| GLM 5 | SiliconFlow; by Z.ai; listed on OpenRouter | $0.95 | $0.20 | $2.55 | n/a | n/a | 204.8K |
| GLM 5 | Venice; by Z.ai; listed on OpenRouter | $1 | $0.20 | $3.20 | n/a | n/a | 198K |
| GLM 5 | Novita; by Z.ai; listed on OpenRouter | $1 | $0.20 | $3.20 | n/a | n/a | 202.8K |
| Claude Opus 4.6 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.6 | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.6 | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.6 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.6 (batch) | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Claude Opus 4.6 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 1M |
| Qwen3 Coder Next | Parasail; by Qwen; listed on OpenRouter | $0.12 | $0.07 | $0.80 | n/a | n/a | 262.1K |
| Step 3.5 Flash | SiliconFlow; by StepFun; listed on OpenRouter | $0.10 | n/a | $0.30 | n/a | n/a | 262.1K |
| Kimi K2.5 | AtlasCloud; by Moonshot AI; listed on OpenRouter | $0.49 | $0.20 | $2.50 | n/a | n/a | 262.1K |
| Kimi K2.5 | SiliconFlow; by Moonshot AI; listed on OpenRouter | $0.50 | $0.15 | $2.50 | n/a | n/a | 262.1K |
| Kimi K2.5 | Venice; by Moonshot AI; listed on OpenRouter | $0.532 | $0.209 | $3.325 | n/a | n/a | 256K |
| Kimi K2.5 | Novita; by Moonshot AI; listed on OpenRouter | $0.57 | $0.095 | $2.85 | n/a | n/a | 262.1K |
| GLM 4.7 Flash | Cloudflare; by Z.ai; listed on OpenRouter | $0.0605 | n/a | $0.40 | n/a | n/a | 131.1K |
| GLM 4.7 Flash | Novita; by Z.ai; listed on OpenRouter | $0.07 | $0.01 | $0.40 | n/a | n/a | 200K |
| GPT-5.2-Codex | Azure; by OpenAI; listed on OpenRouter | $1.75 | $0.175 | $14 | n/a | n/a | 400K |
| Seed 1.6 Flash | Seed; by ByteDance; listed on OpenRouter | $0.075 | n/a | $0.30 | n/a | n/a | 262.1K |
| Seed 1.6 | Seed; by ByteDance; listed on OpenRouter | $0.25 | n/a | $2 | n/a | n/a | 262.1K |
| GLM 4.7 | DeepInfra; by Z.ai; listed on OpenRouter | $0.40 | $0.08 | $1.75 | n/a | n/a | 202.8K |
| GLM 4.7 | Venice; by Z.ai; listed on OpenRouter | $0.4004 | $0.08008 | $1.9292 | n/a | n/a | 198K |
| GLM 4.7 | Novita; by Z.ai; listed on OpenRouter | $0.60 | $0.11 | $2.2 | n/a | n/a | 204.8K |
| GLM 4.7 | Mancer 2; by Z.ai; listed on OpenRouter | $0.70 | $0.70 | $2.50 | n/a | n/a | 131.1K |
| Gemini 3 Flash Preview | Google Vertex; by Google; listed on OpenRouter | $0.50 | $0.05 | $3 | n/a | n/a | 1M |
| Gemini 3 Flash Preview (batch) | Google Vertex; by Google; listed on OpenRouter | $0.50 | $0.05 | $3 | n/a | n/a | 1M |
| Nemotron 3 Nano 30B A3B | Nebius; by NVIDIA; listed on OpenRouter | $0.06 | $0.012 | $0.24 | n/a | n/a | 262.1K |
| GPT-5.2 Chat | Azure; by OpenAI; listed on OpenRouter | $1.75 | $0.175 | $14 | n/a | n/a | 128K |
| GPT-5.2 | Azure; by OpenAI; listed on OpenRouter | $1.75 | $0.175 | $14 | n/a | n/a | 400K |
| GPT-5.2 (batch) | Azure; by OpenAI; listed on OpenRouter | $1.75 | $0.175 | $14 | n/a | n/a | 400K |
| Relace Search | Relace; listed on OpenRouter | $1 | n/a | $3 | n/a | n/a | 256K |
| GLM 4.6V | Novita; by Z.ai; listed on OpenRouter | $0.30 | $0.055 | $0.90 | n/a | n/a | 131.1K |
| GPT-5.1-Codex-Max | Azure; by OpenAI; listed on OpenRouter | $1.25 | $0.125 | $10 | n/a | n/a | 400K |
| DeepSeek V3.2 | GMICloud; by DeepSeek; listed on OpenRouter | $0.2088 | $0.0216 | $0.3096 | n/a | n/a | 163.8K |
| DeepSeek V3.2 | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.259 | $0.14 | $0.80 | n/a | n/a | 163.8K |
| DeepSeek V3.2 | DeepInfra; by DeepSeek; listed on OpenRouter | $0.26 | $0.13 | $0.38 | n/a | n/a | 163.8K |
| DeepSeek V3.2 | AtlasCloud; by DeepSeek; listed on OpenRouter | $0.26 | $0.13 | $0.38 | n/a | n/a | 163.8K |
| DeepSeek V3.2 | Baidu; by DeepSeek; listed on OpenRouter | $0.28 | $0.028 | $0.42 | n/a | n/a | 131.1K |
| DeepSeek V3.2 | DigitalOcean; by DeepSeek; listed on OpenRouter | $0.30 | $0.09 | $0.96 | n/a | n/a | 163.8K |
| DeepSeek V3.2 | Phala; by DeepSeek; listed on OpenRouter | $1 | $0.50 | $1 | n/a | n/a | 163.8K |
| Claude Opus 4.5 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 200K |
| Claude Opus 4.5 | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 200K |
| Claude Opus 4.5 | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 200K |
| Claude Opus 4.5 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 200K |
| Claude Opus 4.5 (batch) | Azure; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 200K |
| Claude Opus 4.5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $5 | $0.50 | $25 | n/a | n/a | 200K |
| GPT-5.1 | Azure; by OpenAI; listed on OpenRouter | $1.25 | $0.13 | $10 | n/a | n/a | 400K |
| GPT-5.1 (batch) | Azure; by OpenAI; listed on OpenRouter | $1.25 | $0.13 | $10 | n/a | n/a | 400K |
| GPT-5.1-Codex | Azure; by OpenAI; listed on OpenRouter | $1.25 | $0.13 | $10 | n/a | n/a | 400K |
| GPT-5.1-Codex-Mini | Azure; by OpenAI; listed on OpenRouter | $0.25 | $0.03 | $2 | n/a | n/a | 400K |
| Kimi K2 Thinking | Novita; by Moonshot AI; listed on OpenRouter | $0.60 | $0.15 | $2.50 | n/a | n/a | 262.1K |
| Sonar Pro Search | Perplexity; listed on OpenRouter | $3 | n/a | $15 | n/a | n/a | 200K |
| gpt-oss-safeguard-20b | Groq; by OpenAI; listed on OpenRouter | $0.075 | $0.0375 | $0.30 | n/a | n/a | 131.1K |
| Claude Haiku 4.5 | Azure; by Anthropic; listed on OpenRouter | $1 | $0.10 | $5 | n/a | n/a | 200K |
| Claude Haiku 4.5 | Google Vertex; by Anthropic; listed on OpenRouter | $1 | $0.10 | $5 | n/a | n/a | 200K |
| Claude Haiku 4.5 (batch) | Azure; by Anthropic; listed on OpenRouter | $1 | $0.10 | $5 | n/a | n/a | 200K |
| Claude Haiku 4.5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $1 | $0.10 | $5 | n/a | n/a | 200K |
| Qwen3 VL 8B Instruct | Parasail; by Qwen; listed on OpenRouter | $0.25 | $0.12 | $0.75 | n/a | n/a | 262.1K |
| Qwen3 VL 30B A3B Thinking | SiliconFlow; by Qwen; listed on OpenRouter | $0.29 | n/a | $1 | n/a | n/a | 262.1K |
| Qwen3 VL 30B A3B Instruct | DeepInfra; by Qwen; listed on OpenRouter | $0.15 | n/a | $0.60 | n/a | n/a | 262.1K |
| Qwen3 VL 30B A3B Instruct | Novita; by Qwen; listed on OpenRouter | $0.20 | n/a | $0.70 | n/a | n/a | 131.1K |
| Qwen3 VL 30B A3B Instruct | SiliconFlow; by Qwen; listed on OpenRouter | $0.29 | n/a | $1 | n/a | n/a | 262.1K |
| GLM 4.6 | Venice; by Z.ai; listed on OpenRouter | $0.43 | $0.08 | $1.75 | n/a | n/a | 198K |
| GLM 4.6 | DeepInfra; by Z.ai; listed on OpenRouter | $0.50 | $0.10 | $2 | n/a | n/a | 202.8K |
| GLM 4.6 | Novita; by Z.ai; listed on OpenRouter | $0.55 | $0.11 | $2.2 | n/a | n/a | 204.8K |
| Claude Sonnet 4.5 | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Claude Sonnet 4.5 | Azure; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 200K |
| Claude Sonnet 4.5 | Google Vertex; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Claude Sonnet 4.5 (batch) | Claude Platform on AWS; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| Claude Sonnet 4.5 (batch) | Azure; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 200K |
| Claude Sonnet 4.5 (batch) | Google Vertex; by Anthropic; listed on OpenRouter | $3 | $0.30 | $15 | n/a | n/a | 1M |
| DeepSeek V3.2 Exp | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.27 | n/a | $0.41 | n/a | n/a | 163.8K |
| Cydonia 24B V4.1 | Parasail; by TheDrummer; listed on OpenRouter | $0.30 | $0.15 | $0.50 | n/a | n/a | 131.1K |
| Relace Apply 3 | Relace; listed on OpenRouter | $0.85 | n/a | $1.25 | n/a | n/a | 256K |
| Qwen3 VL 235B A22B Instruct | DeepInfra; by Qwen; listed on OpenRouter | $0.20 | $0.11 | $0.88 | n/a | n/a | 262.1K |
| Qwen3 VL 235B A22B Instruct | Venice; by Qwen; listed on OpenRouter | $0.21 | $0.10 | $1.90 | n/a | n/a | 128K |
| Qwen3 VL 235B A22B Instruct | Parasail; by Qwen; listed on OpenRouter | $0.21 | $0.10 | $1.90 | n/a | n/a | 131.1K |
| DeepSeek V3.1 Terminus | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.27 | n/a | $1 | n/a | n/a | 163.8K |
| Qwen3 Coder Flash | Alibaba; by Qwen; listed on OpenRouter | $0.195 | $0.039 | $0.975 | n/a | n/a | 1M |
| Qwen3 Next 80B A3B Instruct | DeepInfra; by Qwen; listed on OpenRouter | $0.09 | n/a | $1.1 | n/a | n/a | 262.1K |
| Qwen3 Next 80B A3B Instruct | Parasail; by Qwen; listed on OpenRouter | $0.10 | $0.07 | $1.1 | n/a | n/a | 262.1K |
| Kimi K2 0905 | Novita; by Moonshot AI; listed on OpenRouter | $0.60 | n/a | $2.50 | n/a | n/a | 262.1K |
| Hermes 4 405B | Nebius; by Nous; listed on OpenRouter | $1 | $0.20 | $3 | n/a | n/a | 131.1K |
| DeepSeek V3.1 | DeepInfra; by DeepSeek; listed on OpenRouter | $0.25 | $0.13 | $0.95 | n/a | n/a | 163.8K |
| DeepSeek V3.1 | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.27 | n/a | $1 | n/a | n/a | 163.8K |
| DeepSeek V3.1 | CoreWeave; by DeepSeek; listed on OpenRouter | $0.50 | $0.20 | $1.50 | n/a | n/a | 161K |
| DeepSeek V3.1 | SambaNova; by DeepSeek; listed on OpenRouter | $0.65 | n/a | $1.50 | n/a | n/a | 131.1K |
| GLM 4.5V | Novita; by Z.ai; listed on OpenRouter | $0.60 | $0.11 | $1.80 | n/a | n/a | 65.5K |
| GPT-5 | Azure; by OpenAI; listed on OpenRouter | $1.25 | $0.125 | $10 | n/a | n/a | 400K |
| GPT-5 (batch) | Azure; by OpenAI; listed on OpenRouter | $1.25 | $0.125 | $10 | n/a | n/a | 400K |
| GPT-5 Mini | Azure; by OpenAI; listed on OpenRouter | $0.25 | $0.03 | $2 | n/a | n/a | 400K |
| GPT-5 Mini (batch) | Azure; by OpenAI; listed on OpenRouter | $0.25 | $0.03 | $2 | n/a | n/a | 400K |
| GPT-5 Nano | Azure; by OpenAI; listed on OpenRouter | $0.05 | $0.01 | $0.40 | n/a | n/a | 400K |
| GPT-5 Nano (batch) | Azure; by OpenAI; listed on OpenRouter | $0.05 | $0.01 | $0.40 | n/a | n/a | 400K |
| gpt-oss-120b | Near AI; by OpenAI; listed on OpenRouter | $0.03 | $0.015 | $0.15 | n/a | n/a | 131.1K |
| gpt-oss-120b | Venice; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.15 | n/a | n/a | 128K |
| gpt-oss-120b | CoreWeave; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.17 | n/a | n/a | 131.1K |
| gpt-oss-120b | DekaLLM; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.18 | n/a | n/a | 131.1K |
| gpt-oss-120b | AkashML; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.187 | n/a | n/a | 131.1K |
| gpt-oss-120b | PrimeIntellect; by OpenAI; listed on OpenRouter | $0.036 | $0.032 | $0.077 | n/a | n/a | 131.1K |
| gpt-oss-120b | DeepInfra; by OpenAI; listed on OpenRouter | $0.037 | n/a | $0.17 | n/a | n/a | 131.1K |
| gpt-oss-120b | Crusoe; by OpenAI; listed on OpenRouter | $0.05 | $0.05 | $0.25 | n/a | n/a | 131.1K |
| gpt-oss-120b | Mancer 2; by OpenAI; listed on OpenRouter | $0.055 | $0.055 | $0.25 | n/a | n/a | 131.1K |
| gpt-oss-120b | Sail Research; by OpenAI; listed on OpenRouter | $0.06 | $0.03 | $0.40 | n/a | n/a | 131.1K |
| gpt-oss-120b | DigitalOcean; by OpenAI; listed on OpenRouter | $0.06 | $0.012 | $0.42 | n/a | n/a | 128K |
| gpt-oss-120b | Google Vertex; by OpenAI; listed on OpenRouter | $0.09 | n/a | $0.36 | n/a | n/a | 131.1K |
| gpt-oss-120b | BaseTen; by OpenAI; listed on OpenRouter | $0.10 | $0.10 | $0.50 | n/a | n/a | 128.1K |
| gpt-oss-120b | Parasail; by OpenAI; listed on OpenRouter | $0.10 | $0.055 | $0.75 | n/a | n/a | 131.1K |
| gpt-oss-120b | SambaNova; by OpenAI; listed on OpenRouter | $0.14 | n/a | $0.95 | n/a | n/a | 131.1K |
| gpt-oss-120b | Nebius; by OpenAI; listed on OpenRouter | $0.15 | $0.10 | $0.60 | n/a | n/a | 131.1K |
| gpt-oss-120b | SiliconFlow; by OpenAI; listed on OpenRouter | $0.15 | $0.075 | $0.60 | n/a | n/a | 131.1K |
| gpt-oss-120b | Phala; by OpenAI; listed on OpenRouter | $0.15 | n/a | $0.60 | n/a | n/a | 131.1K |
| gpt-oss-120b | Groq; by OpenAI; listed on OpenRouter | $0.15 | $0.075 | $0.60 | n/a | n/a | 131.1K |
| gpt-oss-120b | Cerebras; by OpenAI; listed on OpenRouter | $0.35 | $0.35 | $0.75 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Near AI; by OpenAI; listed on OpenRouter | $0.03 | $0.015 | $0.15 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Venice; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.15 | n/a | n/a | 128K |
| gpt-oss-120b (batch) | CoreWeave; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.17 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | DekaLLM; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.18 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | AkashML; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.187 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | PrimeIntellect; by OpenAI; listed on OpenRouter | $0.036 | $0.032 | $0.077 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | DeepInfra; by OpenAI; listed on OpenRouter | $0.037 | n/a | $0.17 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Crusoe; by OpenAI; listed on OpenRouter | $0.05 | $0.05 | $0.25 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Mancer 2; by OpenAI; listed on OpenRouter | $0.055 | $0.055 | $0.25 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Sail Research; by OpenAI; listed on OpenRouter | $0.06 | $0.03 | $0.40 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | DigitalOcean; by OpenAI; listed on OpenRouter | $0.06 | $0.012 | $0.42 | n/a | n/a | 128K |
| gpt-oss-120b (batch) | Google Vertex; by OpenAI; listed on OpenRouter | $0.09 | n/a | $0.36 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | BaseTen; by OpenAI; listed on OpenRouter | $0.10 | $0.10 | $0.50 | n/a | n/a | 128.1K |
| gpt-oss-120b (batch) | Parasail; by OpenAI; listed on OpenRouter | $0.10 | $0.055 | $0.75 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | SambaNova; by OpenAI; listed on OpenRouter | $0.14 | n/a | $0.95 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Nebius; by OpenAI; listed on OpenRouter | $0.15 | $0.10 | $0.60 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | SiliconFlow; by OpenAI; listed on OpenRouter | $0.15 | $0.075 | $0.60 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Phala; by OpenAI; listed on OpenRouter | $0.15 | n/a | $0.60 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Groq; by OpenAI; listed on OpenRouter | $0.15 | $0.075 | $0.60 | n/a | n/a | 131.1K |
| gpt-oss-120b (batch) | Cerebras; by OpenAI; listed on OpenRouter | $0.35 | $0.35 | $0.75 | n/a | n/a | 131.1K |
| gpt-oss-20b | Darkbloom; by OpenAI; listed on OpenRouter | $0.018 | $0.009 | $0.09 | n/a | n/a | 131.1K |
| gpt-oss-20b | AkashML; by OpenAI; listed on OpenRouter | $0.02 | n/a | $0.10 | n/a | n/a | 131.1K |
| gpt-oss-20b | DekaLLM; by OpenAI; listed on OpenRouter | $0.029 | $0.029 | $0.14 | n/a | n/a | 131.1K |
| gpt-oss-20b | CoreWeave; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.13 | n/a | n/a | 131.1K |
| gpt-oss-20b | DeepInfra; by OpenAI; listed on OpenRouter | $0.03 | n/a | $0.14 | n/a | n/a | 131.1K |
| gpt-oss-20b | Parasail; by OpenAI; listed on OpenRouter | $0.03 | $0.02 | $0.15 | n/a | n/a | 131.1K |
| gpt-oss-20b | PrimeIntellect; by OpenAI; listed on OpenRouter | $0.039 | $0.035 | $0.059 | n/a | n/a | 131.1K |
| gpt-oss-20b | SiliconFlow; by OpenAI; listed on OpenRouter | $0.04 | n/a | $0.18 | n/a | n/a | 131.1K |
| gpt-oss-20b | Groq; by OpenAI; listed on OpenRouter | $0.075 | $0.0375 | $0.30 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | Darkbloom; by OpenAI; listed on OpenRouter | $0.018 | $0.009 | $0.09 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | AkashML; by OpenAI; listed on OpenRouter | $0.02 | n/a | $0.10 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | DekaLLM; by OpenAI; listed on OpenRouter | $0.029 | $0.029 | $0.14 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | CoreWeave; by OpenAI; listed on OpenRouter | $0.03 | $0.03 | $0.13 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | DeepInfra; by OpenAI; listed on OpenRouter | $0.03 | n/a | $0.14 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | Parasail; by OpenAI; listed on OpenRouter | $0.03 | $0.02 | $0.15 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | PrimeIntellect; by OpenAI; listed on OpenRouter | $0.039 | $0.035 | $0.059 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | SiliconFlow; by OpenAI; listed on OpenRouter | $0.04 | n/a | $0.18 | n/a | n/a | 131.1K |
| gpt-oss-20b (batch) | Groq; by OpenAI; listed on OpenRouter | $0.075 | $0.0375 | $0.30 | n/a | n/a | 131.1K |
| Qwen3 Coder 30B A3B Instruct | SiliconFlow; by Qwen; listed on OpenRouter | $0.07 | n/a | $0.28 | n/a | n/a | 262.1K |
| Qwen3 30B A3B Instruct 2507 | DekaLLM; by Qwen; listed on OpenRouter | $0.09 | n/a | $0.30 | n/a | n/a | 262.1K |
| Qwen3 30B A3B Instruct 2507 | SiliconFlow; by Qwen; listed on OpenRouter | $0.09 | n/a | $0.30 | n/a | n/a | 262.1K |
| Qwen3 30B A3B Instruct 2507 | Nebius; by Qwen; listed on OpenRouter | $0.10 | $0.10 | $0.30 | n/a | n/a | 262.1K |
| GLM 4.5 Air | Novita; by Z.ai; listed on OpenRouter | $0.13 | $0.025 | $0.85 | n/a | n/a | 131.1K |
| GLM 4.5 Air | SiliconFlow; by Z.ai; listed on OpenRouter | $0.14 | n/a | $0.86 | n/a | n/a | 131.1K |
| Qwen3 235B A22B Thinking 2507 | Venice; by Qwen; listed on OpenRouter | $0.45 | n/a | $3.50 | n/a | n/a | 128K |
| Qwen3 Coder 480B A35B | DeepInfra; by Qwen; listed on OpenRouter | $0.30 | $0.10 | $1 | n/a | n/a | 262.1K |
| Qwen3 Coder 480B A35B | Venice; by Qwen; listed on OpenRouter | $0.35 | $0.04 | $1.50 | n/a | n/a | 256K |
| UI-TARS 7B  | Parasail; by ByteDance; listed on OpenRouter | $0.10 | $0.10 | $0.20 | n/a | n/a | 128K |
| Qwen3 235B A22B Instruct 2507 | GMICloud; by Qwen; listed on OpenRouter | $0.0875 | $0.0175 | $0.35 | n/a | n/a | 262.1K |
| Qwen3 235B A22B Instruct 2507 | DeepInfra; by Qwen; listed on OpenRouter | $0.09 | n/a | $0.55 | n/a | n/a | 262.1K |
| Qwen3 235B A22B Instruct 2507 | Parasail; by Qwen; listed on OpenRouter | $0.14 | $0.05 | $0.80 | n/a | n/a | 131.1K |
| Qwen3 235B A22B Instruct 2507 | Venice; by Qwen; listed on OpenRouter | $0.15 | n/a | $0.75 | n/a | n/a | 128K |
| Kimi K2 0711 | Novita; by Moonshot AI; listed on OpenRouter | $0.57 | n/a | $2.3 | n/a | n/a | 131.1K |
| Uncensored | Venice; listed on OpenRouter | $0.20 | n/a | $0.90 | n/a | n/a | 128K |
| Hunyuan A13B Instruct | SiliconFlow; by Tencent; listed on OpenRouter | $0.14 | n/a | $0.57 | n/a | n/a | 131.1K |
| Morph V3 Large | Morph; listed on OpenRouter | $0.90 | n/a | $1.90 | n/a | n/a | 262.1K |
| Morph V3 Large (batch) | Morph; listed on OpenRouter | $0.90 | n/a | $1.90 | n/a | n/a | 262.1K |
| Morph V3 Fast | Morph; listed on OpenRouter | $0.80 | n/a | $1.20 | n/a | n/a | 262.1K |
| Morph V3 Fast (batch) | Morph; listed on OpenRouter | $0.80 | n/a | $1.20 | n/a | n/a | 262.1K |
| Mistral Small 3.2 24B | DeepInfra; by Mistral AI; listed on OpenRouter | $0.075 | n/a | $0.20 | n/a | n/a | 128K |
| Mistral Small 3.2 24B | Parasail; by Mistral AI; listed on OpenRouter | $0.09 | $0.05 | $0.30 | n/a | n/a | 131.1K |
| Mistral Small 3.2 24B | Venice; by Mistral AI; listed on OpenRouter | $0.09375 | n/a | $0.25 | n/a | n/a | 256K |
| R1 0528 | DeepInfra; by DeepSeek; listed on OpenRouter | $0.50 | $0.35 | $2.15 | n/a | n/a | 163.8K |
| R1 0528 | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.50 | n/a | $2.18 | n/a | n/a | 163.8K |
| R1 0528 | StreamLake; by DeepSeek; listed on OpenRouter | $0.571 | n/a | $2.286 | n/a | n/a | 128K |
| Llama Guard 4 12B | DeepInfra; by Meta; listed on OpenRouter | $0.18 | n/a | $0.18 | n/a | n/a | 163.8K |
| Qwen3 30B A3B | DeepInfra; by Qwen; listed on OpenRouter | $0.12 | n/a | $0.50 | n/a | n/a | 41K |
| Qwen3 14B | NextBit; by Qwen; listed on OpenRouter | $0.10 | n/a | $0.22 | n/a | n/a | 41K |
| Qwen3 14B | DeepInfra; by Qwen; listed on OpenRouter | $0.12 | n/a | $0.24 | n/a | n/a | 41K |
| Qwen3 32B | DeepInfra; by Qwen; listed on OpenRouter | $0.08 | n/a | $0.28 | n/a | n/a | 41K |
| Qwen3 32B | Tenstorrent; by Qwen; listed on OpenRouter | $0.14 | $0.09 | $0.40 | n/a | n/a | 20K |
| Qwen3 32B | SiliconFlow; by Qwen; listed on OpenRouter | $0.14 | n/a | $0.57 | n/a | n/a | 131.1K |
| GPT-4.1 | Azure; by OpenAI; listed on OpenRouter | $2 | $0.50 | $8 | n/a | n/a | 1M |
| GPT-4.1 (batch) | Azure; by OpenAI; listed on OpenRouter | $2 | $0.50 | $8 | n/a | n/a | 1M |
| GPT-4.1 Mini | Azure; by OpenAI; listed on OpenRouter | $0.40 | $0.10 | $1.60 | n/a | n/a | 1M |
| GPT-4.1 Mini (batch) | Azure; by OpenAI; listed on OpenRouter | $0.40 | $0.10 | $1.60 | n/a | n/a | 1M |
| GPT-4.1 Nano | Azure; by OpenAI; listed on OpenRouter | $0.10 | $0.03 | $0.40 | n/a | n/a | 1M |
| GPT-4.1 Nano (batch) | Azure; by OpenAI; listed on OpenRouter | $0.10 | $0.03 | $0.40 | n/a | n/a | 1M |
| Llama 4 Maverick | DigitalOcean; by Meta; listed on OpenRouter | $0.1875 | $0.05 | $0.6525 | n/a | n/a | 128K |
| Llama 4 Maverick | Parasail; by Meta; listed on OpenRouter | $0.35 | $0.17 | $1 | n/a | n/a | 524.3K |
| Llama 4 Maverick | Google Vertex; by Meta; listed on OpenRouter | $0.35 | n/a | $1.15 | n/a | n/a | 524.3K |
| Llama 4 Scout | DeepInfra; by Meta; listed on OpenRouter | $0.10 | n/a | $0.30 | n/a | n/a | 327.7K |
| Llama 4 Scout | Google Vertex; by Meta; listed on OpenRouter | $0.25 | n/a | $0.70 | n/a | n/a | 1.3M |
| DeepSeek V3 0324 | SiliconFlow; by DeepSeek; listed on OpenRouter | $0.25 | n/a | $1 | n/a | n/a | 163.8K |
| DeepSeek V3 0324 | GMICloud; by DeepSeek; listed on OpenRouter | $0.29 | $0.11 | $1.14 | n/a | n/a | 128K |
| Mistral Small 3.1 24B | Cloudflare; by Mistral AI; listed on OpenRouter | $0.351 | n/a | $0.555 | n/a | n/a | 128K |
| Gemma 3 4B | Parasail; by Google; listed on OpenRouter | $0.04 | $0.02 | $0.05 | n/a | n/a | 131.1K |
| Gemma 3 4B | DeepInfra; by Google; listed on OpenRouter | $0.05 | n/a | $0.10 | n/a | n/a | 131.1K |
| Gemma 3 12B | DeepInfra; by Google; listed on OpenRouter | $0.05 | n/a | $0.15 | n/a | n/a | 131.1K |
| Gemma 3 12B | NextBit; by Google; listed on OpenRouter | $0.075 | $0.05 | $0.15 | n/a | n/a | 131.1K |
| Command A | Cohere; listed on OpenRouter | $2.50 | n/a | $10 | n/a | n/a | 256K |
| Reka Flash 3 | Reka; by rekaai; listed on OpenRouter | $0.10 | n/a | $0.20 | n/a | n/a | 65.5K |
| Gemma 3 27B | DeepInfra; by Google; listed on OpenRouter | $0.08 | n/a | $0.16 | n/a | n/a | 131.1K |
| Gemma 3 27B | NextBit; by Google; listed on OpenRouter | $0.08 | $0.04 | $0.35 | n/a | n/a | 131.1K |
| Gemma 3 27B | Parasail; by Google; listed on OpenRouter | $0.08 | $0.04 | $0.45 | n/a | n/a | 131.1K |
| Gemma 3 27B | Nebius; by Google; listed on OpenRouter | $0.10 | $0.04 | $0.30 | n/a | n/a | 110K |
| Skyfall 36B V2 | Parasail; by TheDrummer; listed on OpenRouter | $0.55 | $0.25 | $0.80 | n/a | n/a | 32.8K |
| Sonar Reasoning Pro | Perplexity; listed on OpenRouter | $2 | n/a | $8 | n/a | n/a | 128K |
| Sonar Pro | Perplexity; listed on OpenRouter | $3 | n/a | $15 | n/a | n/a | 200K |
| Sonar Deep Research | Perplexity; listed on OpenRouter | $2 | n/a | $8 | n/a | n/a | 128K |
| Aion-RP 1.0 (8B) | AionLabs; listed on OpenRouter | $0.80 | n/a | $1.60 | n/a | n/a | 32.8K |
| Qwen2.5 VL 72B Instruct | Parasail; by Qwen; listed on OpenRouter | $0.80 | $0.40 | $1 | n/a | n/a | 128K |
| Qwen-Plus | Alibaba; by Qwen; listed on OpenRouter | $0.26 | $0.052 | $0.78 | n/a | n/a | 1M |
| Mistral Small 3 | DeepInfra; by Mistral AI; listed on OpenRouter | $0.05 | n/a | $0.08 | n/a | n/a | 32.8K |
| Sonar | Perplexity; listed on OpenRouter | $1 | n/a | $1 | n/a | n/a | 127.1K |
| Phi 4 | DeepInfra; by Microsoft; listed on OpenRouter | $0.07 | n/a | $0.14 | n/a | n/a | 16.4K |
| DeepSeek V3 | StreamLake; by DeepSeek; listed on OpenRouter | $0.2574 | n/a | $1.0287 | n/a | n/a | 128K |
| DeepSeek V3 | DeepInfra; by DeepSeek; listed on OpenRouter | $0.32 | n/a | $0.89 | n/a | n/a | 163.8K |
| Command R7B (12-2024) | Cohere; listed on OpenRouter | $0.0375 | n/a | $0.15 | n/a | n/a | 128K |
| Llama 3.3 70B Instruct | DeepInfra; by Meta; listed on OpenRouter | $0.10 | n/a | $0.32 | n/a | n/a | 131.1K |
| Llama 3.3 70B Instruct | AkashML; by Meta; listed on OpenRouter | $0.20 | $0.10 | $0.52 | n/a | n/a | 131.1K |
| Llama 3.3 70B Instruct | Parasail; by Meta; listed on OpenRouter | $0.22 | $0.11 | $0.50 | n/a | n/a | 131.1K |
| Llama 3.3 70B Instruct | Cloudflare; by Meta; listed on OpenRouter | $0.293 | n/a | $2.253 | n/a | n/a | 24K |
| Llama 3.3 70B Instruct | Groq; by Meta; listed on OpenRouter | $0.59 | $0.295 | $0.79 | n/a | n/a | 131.1K |
| Llama 3.3 70B Instruct | SambaNova; by Meta; listed on OpenRouter | $0.60 | n/a | $1.20 | n/a | n/a | 131.1K |
| Llama 3.3 70B Instruct | CoreWeave; by Meta; listed on OpenRouter | $0.71 | $0.71 | $0.71 | n/a | n/a | 128K |
| Llama 3.3 70B Instruct | Google Vertex; by Meta; listed on OpenRouter | $0.72 | n/a | $0.72 | n/a | n/a | 128K |
| Qwen2.5 Coder 32B Instruct | Cloudflare; by qwen; listed on OpenRouter | $0.66 | n/a | $1 | n/a | n/a | 32.8K |
| UnslopNemo 12B | Parasail; by TheDrummer; listed on OpenRouter | $0.40 | n/a | $0.40 | n/a | n/a | 1M |
| Magnum v4 72B | Mancer 2; by anthracite-org; listed on OpenRouter | $2.50 | $2.50 | $5 | n/a | n/a | 32.8K |
| Qwen2.5 7B Instruct | Phala; by Qwen; listed on OpenRouter | $0.10 | n/a | $0.20 | n/a | n/a | 32.8K |
| Llama 3.2 1B Instruct | Cloudflare; by Meta; listed on OpenRouter | $0.027 | n/a | $0.201 | n/a | n/a | 60K |
| Llama 3.2 3B Instruct | Parasail; by Meta; listed on OpenRouter | $0.05 | n/a | $0.33 | n/a | n/a | 131.1K |
| Llama 3.2 3B Instruct | Cloudflare; by Meta; listed on OpenRouter | $0.0509 | n/a | $0.335 | n/a | n/a | 80K |
| Qwen2.5 72B Instruct | DeepInfra; by qwen; listed on OpenRouter | $0.36 | n/a | $0.40 | n/a | n/a | 32.8K |
| Command R (08-2024) | Cohere; listed on OpenRouter | $0.15 | n/a | $0.60 | n/a | n/a | 128K |
| Command R+ (08-2024) | Cohere; listed on OpenRouter | $2.50 | n/a | $10 | n/a | n/a | 128K |
| Llama 3.1 Euryale 70B v2.2 | DeepInfra; by Sao10K; listed on OpenRouter | $0.85 | n/a | $0.85 | n/a | n/a | 131.1K |
| Llama 3 8B Lunaris | Parasail; by Sao10K; listed on OpenRouter | $0.04 | n/a | $0.05 | n/a | n/a | 8.2K |
| Llama 3 8B Lunaris | DeepInfra; by Sao10K; listed on OpenRouter | $0.04 | n/a | $0.05 | n/a | n/a | 8.2K |
| GPT-4o (2024-08-06) | Azure; by OpenAI; listed on OpenRouter | $2.50 | $1.25 | $10 | n/a | n/a | 128K |
| Llama 3.1 8B Instruct | DeepInfra; by Meta; listed on OpenRouter | $0.02 | n/a | $0.04 | n/a | n/a | 131.1K |
| Llama 3.1 8B Instruct | Groq; by Meta; listed on OpenRouter | $0.05 | $0.025 | $0.08 | n/a | n/a | 131.1K |
| Llama 3.1 8B Instruct | Cloudflare; by Meta; listed on OpenRouter | $0.152 | n/a | $0.287 | n/a | n/a | 32K |
| Llama 3.1 8B Instruct | CoreWeave; by Meta; listed on OpenRouter | $0.22 | $0.22 | $0.22 | n/a | n/a | 131.1K |
| Mistral Nemo | Io Net; by Mistral AI; listed on OpenRouter | $0.01656 | $0.0108 | $0.0216 | n/a | n/a | 128K |
| Mistral Nemo | DekaLLM; by Mistral AI; listed on OpenRouter | $0.018 | n/a | $0.03 | n/a | n/a | 131.1K |
| Mistral Nemo | DeepInfra; by Mistral AI; listed on OpenRouter | $0.029 | n/a | $0.03 | n/a | n/a | 131.1K |
| Mistral Nemo | Parasail; by Mistral AI; listed on OpenRouter | $0.03 | n/a | $0.03 | n/a | n/a | 131.1K |
| GPT-4o-mini | Azure; by OpenAI; listed on OpenRouter | $0.15 | $0.075 | $0.60 | n/a | n/a | 128K |
| GPT-4o-mini (batch) | Azure; by OpenAI; listed on OpenRouter | $0.15 | $0.075 | $0.60 | n/a | n/a | 128K |
| Gemma 2 27B | NextBit; by Google; listed on OpenRouter | $0.65 | n/a | $0.65 | n/a | n/a | 8.2K |
| GPT-4o | Azure; by OpenAI; listed on OpenRouter | $2.50 | n/a | $10 | n/a | n/a | 128K |
| GPT-4o (2024-05-13) | Azure; by OpenAI; listed on OpenRouter | $5 | n/a | $15 | n/a | n/a | 128K |
| GPT-4o (batch) | Azure; by OpenAI; listed on OpenRouter | $2.50 | n/a | $10 | n/a | n/a | 128K |
| GPT-3.5 Turbo (older v0613) | Azure; by OpenAI; listed on OpenRouter | $1 | n/a | $2 | n/a | n/a | 4.1K |
| GPT-3.5 Turbo 16k | Azure; by OpenAI; listed on OpenRouter | $3 | n/a | $4 | n/a | n/a | 16.4K |
| Weaver (alpha) | Mancer 2; by Mancer; listed on OpenRouter | $0.40 | $0.40 | $0.75 | n/a | n/a | 8K |
| ReMM SLERP 13B | Mancer 2; by undi95; listed on OpenRouter | $0.35 | $0.35 | $0.65 | n/a | n/a | 6.1K |
| MythoMax 13B | Parasail; by gryphe; listed on OpenRouter | $0.08 | n/a | $0.11 | n/a | n/a | 4.1K |
| MythoMax 13B | Mancer 2; by gryphe; listed on OpenRouter | $0.35 | $0.35 | $0.60 | n/a | n/a | 8.2K |
| GPT-4 | Azure; by OpenAI; listed on OpenRouter | $30 | n/a | $60 | n/a | n/a | 8.2K |

## Method

- Scope. Text models, priced in US dollars per million tokens. Image, audio, video, embedding, and OCR models bill in other units and are left out.
- Official sources. Rows for Anthropic, Google, OpenAI, xAI, DeepSeek, Mistral AI, Z.ai, MiniMax, Inception Labs, and Together come from each company's own pricing page. Amazon Bedrock rows come from the AWS Price List API, for US East (N. Virginia).
- Other hosts. Every other seller's rate is the price listed on OpenRouter, which is the source for those rows and is marked on each one. A host's price on its own site can differ. When a host lists several deployments of a model, this keeps the cheapest.
- Generation. A script reads these sources, respects each site's robots.txt, and stops if a table changes shape. Each refresh is reviewed before it ships.
- Tiers. A model with more than one rate has one row per rate. The base tier is the rate for the shortest prompts, or DeepSeek's peak rate, the higher of its two. On Bedrock it is the global endpoint where one exists. The findings and the chart use the base tiers of models bought from the lab that made them.
- Service tiers. The downloads add flex and priority rates where a seller publishes them. Priority covers OpenAI's Fast tier, Anthropic's fast mode, and the Priority tiers from Google and AWS.
- Caching. Cached input is the price of a cache hit. The downloads list the shortest and longest cache write periods a seller publishes, and Google's storage price per million tokens per hour.
- Context limits. A seller's own figure is used where its price page gives one. Otherwise the value is OpenRouter's listing for that exact model name.
- Not included. Regional and data-residency premiums, Azure's own price list, free tiers, and tool fees. Azure rows here are OpenRouter's listings.
- Blank cells. n/a means the source publishes no such rate. Mistral AI's page has a batch view, but those rates are not in its static content.
- Availability. This column reports only what the seller's pricing page states. A listed model can still be scheduled for retirement. See the model retirement calendar.
- Token counts. Equal rates do not mean equal bills, because tokenizers differ. Anthropic says Claude 4.7 and later models produce about 30 percent more tokens for the same text.

## Questions

### Which LLM API is the cheapest?

Among models bought directly from the lab that made them, as of November 10, 2026, the lowest input price is $0.05 per million tokens (gpt-5-nano), and the lowest output price is $0.10 (Ministral 3 3B). Hosts can be cheaper for open-weight models. The cheapest rate is not the cheapest bill, because token counts and output length differ by model and task.

### Why does the same model have several prices?

Open-weight models are sold by many hosts, each setting its own price. Some sellers also charge more above a prompt-length threshold, and DeepSeek charges different rates at peak and off-peak hours. Each rate gets its own row.

### How current are the prices?

Every rate was read from its source on November 10, 2026. Prices change often, so give the checked date when you quote one.

## Sources

1. [Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing)
2. [Google pricing](https://aistudio.google.com/docs/pricing)
3. [OpenAI pricing](https://developers.openai.com/api/docs/pricing)
4. [xAI pricing](https://docs.x.ai/developers/models)
5. [DeepSeek pricing](https://api-docs.deepseek.com/quick_start/pricing/)
6. [Mistral AI pricing](https://docs.mistral.ai/inference/pricing)
7. [Z.ai pricing](https://docs.z.ai/guides/overview/pricing)
8. [MiniMax pricing](https://platform.minimax.io/docs/pricing/overview)
9. [Inception Labs pricing](https://docs.inceptionlabs.ai/get-started/models)
10. [Together pricing](https://docs.together.ai/docs/serverless/models)
11. [Amazon Bedrock pricing](https://aws.amazon.com/bedrock/pricing/)
12. [AWS Price List API, Amazon Bedrock](https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/us-east-1/index.json)
13. [OpenRouter models API](https://openrouter.ai/api/v1/models)

## Cite this dataset

AI for Devs. "LLM API prices." Published November 10, 2026. https://aifordevs.org/data/llm-pricing

- Page: https://aifordevs.org/data/llm-pricing
- Markdown: https://aifordevs.org/data/llm-pricing.md
- Reuse: Free to reuse under CC BY 4.0 with a link to this page. License: https://creativecommons.org/licenses/by/4.0/
- Publisher: AI for Devs
- Published: 2026-11-10

---
title: "LLM model retirement calendar"
type: "dataset"
url: "https://aifordevs.org/data/model-retirements"
markdown: "https://aifordevs.org/data/model-retirements.md"
publisher: "AI for Devs"
license: "https://creativecommons.org/licenses/by/4.0/"
attribution: "Credit AI for Devs and link to https://aifordevs.org/data/model-retirements"
description: "Deprecation status, shutdown dates, and recommended replacements for 385 models on Anthropic API, OpenAI API, Gemini API, Mistral API, Cohere API, and Amazon Bedrock, taken from each platform's own notices. Checked November 10, 2026."
checked: "2026-11-10"
csv: "https://aifordevs.org/data/model-retirements.csv"
json: "https://aifordevs.org/data/model-retirements.json"
rows: 385
---

# LLM model retirement calendar

Deprecation status, shutdown dates, and recommended replacements for 385 models on Anthropic API, OpenAI API, Gemini API, Mistral API, Cohere API, and Amazon Bedrock, taken from each platform's own notices. Checked November 10, 2026.

Download: [CSV](https://aifordevs.org/data/model-retirements.csv) | [JSON](https://aifordevs.org/data/model-retirements.json)

## Key findings

- As of November 10, 2026, 21 models are scheduled to retire within 30 days and 58 within 90 days.
- The next shutdown date is November 13, 2026, when 5 models retire on Amazon Bedrock.
- OpenAI API has the most retirements in the next 90 days, with 40.
- Median notice between a deprecation and its shutdown runs from 62 days (Anthropic API) to 182 days (OpenAI API).
- A replacement is named for 67 of the 107 upcoming retirements.

## The data

Soonest shutdowns first, checked November 10, 2026. Filter by platform or status, and sort by date.

| Model | Details | Deprecated | Shutdown |
| --- | --- | --- | --- |
| claude-haiku-4-5-20251001 | Anthropic API | n/a | Not before October 15, 2026 |
| anthropic.claude-3-haiku-20240307-v1:0 | Amazon Bedrock; by Anthropic | Oct 10 2026 | November 13, 2026 |
| anthropic.claude-3-sonnet-20240229-v1:0 | Amazon Bedrock; by Anthropic | Oct 10 2026 | November 13, 2026 |
| anthropic.claude-3-5-sonnet-20240620-v1:0 | Amazon Bedrock; by Anthropic | Oct 10 2026 | November 13, 2026 |
| anthropic.claude-3-5-sonnet-20241022-v2:0 | Amazon Bedrock; by Anthropic | Oct 10 2026 | November 13, 2026 |
| anthropic.claude-3-7-sonnet-20250219-v1:0 | Amazon Bedrock; by Anthropic | Oct 10 2026 | November 13, 2026 |
| gemini-3.1-flash-tts-preview | Gemini API; by Google; Replaced by gemini-3.8-flash-tts | n/a | November 17, 2026 |
| gemini-3.1-flash-live-preview | Gemini API; by Google; Replaced by gemini-3.8-live | n/a | November 17, 2026 |
| gemini-2.5-flash-native-audio-preview-12-2025 | Gemini API; by Google; Replaced by gemini-3.8-live | n/a | November 17, 2026 |
| gemini-2.5-flash-preview-tts | Gemini API; by Google; Replaced by gemini-3.8-flash-tts | n/a | November 17, 2026 |
| gemini-2.5-pro-preview-tts | Gemini API; by Google; Replaced by gemini-3.8-flash-tts | n/a | November 17, 2026 |
| claude-opus-4-5-20251101 | Anthropic API | n/a | Not before November 24, 2026 |
| ai21.jamba-1-5-large-v1:0 | Amazon Bedrock; by AI21 Labs | May 26 2026 | November 26, 2026 |
| ai21.jamba-1-5-mini-v1:0 | Amazon Bedrock; by AI21 Labs | May 26 2026 | November 26, 2026 |
| claude-sonnet-4-5-20250929 | Anthropic API; Replaced by claude-sonnet-5-5 | Sep 30 2026 | November 30, 2026 |
| twelvelabs.marengo-embed-2-7-v1:0 | Amazon Bedrock; by TwelveLabs | May 29 2026 | November 30, 2026 |
| gpt-image-1-mini | OpenAI API; Replaced by gpt-image-2.6-sunburst | Jun 02 2026 | December 1, 2026 |
| gpt-image-1.5 | OpenAI API; Replaced by gpt-image-2.6-sunburst | Jun 02 2026 | December 1, 2026 |
| gpt-image-1.5-2025-12-16 | OpenAI API; Replaced by gpt-image-2.6-sunburst | Jun 02 2026 | December 1, 2026 |
| chatgpt-image-latest | OpenAI API; Replaced by gpt-image-2.6-sunburst | Jun 02 2026 | December 1, 2026 |
| gemini-2.5-pro | Gemini API; by Google; Replaced by gemini-3.8-flash | n/a | December 1, 2026 |
| gemini-2.5-flash | Gemini API; by Google; Replaced by gemini-3.5-flash-lite | n/a | December 1, 2026 |
| gemini-omni-1.1-flash | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | December 3, 2026 |
| gpt-5 | OpenAI API; Replaced by gpt-5.6-sol | Jun 11 2026 | December 11, 2026 |
| gpt-5-2025-08-07 | OpenAI API; Replaced by gpt-5.6-sol | Jun 11 2026 | December 11, 2026 |
| gpt-5-mini | OpenAI API; Replaced by gpt-5.6-terra | Jun 11 2026 | December 11, 2026 |
| gpt-5-mini-2025-08-07 | OpenAI API; Replaced by gpt-5.6-terra | Jun 11 2026 | December 11, 2026 |
| gpt-5-nano | OpenAI API; Replaced by gpt-5.6-luna | Jun 11 2026 | December 11, 2026 |
| gpt-5-nano-2025-08-07 | OpenAI API; Replaced by gpt-5.6-luna | Jun 11 2026 | December 11, 2026 |
| gpt-5-pro | OpenAI API; Replaced by gpt-5.6-sol | Jun 11 2026 | December 11, 2026 |
| gpt-5-pro-2025-10-06 | OpenAI API; Replaced by gpt-5.6-sol | Jun 11 2026 | December 11, 2026 |
| o3 | OpenAI API; Replaced by gpt-5.6-sol | Jun 11 2026 | December 11, 2026 |
| o3-2025-04-16 | OpenAI API; Replaced by gpt-5.6-sol | Jun 11 2026 | December 11, 2026 |
| o3-pro | OpenAI API; Replaced by gpt-5.6-sol | Jun 11 2026 | December 11, 2026 |
| o3-pro-2025-06-10 | OpenAI API; Replaced by gpt-5.6-sol | Jun 11 2026 | December 11, 2026 |
| tts-1 | OpenAI API; Replaced by gpt-realtime-2.1-mini | Oct 01 2026 | January 6, 2027 |
| tts-1-hd | OpenAI API; Replaced by gpt-realtime-2.1-mini | Oct 01 2026 | January 6, 2027 |
| gpt-4o-mini-tts | OpenAI API; Replaced by gpt-realtime-2.1-mini | Oct 01 2026 | January 6, 2027 |
| gpt-4o-mini-tts-2025-03-20 | OpenAI API; Replaced by gpt-realtime-2.1-mini | Oct 01 2026 | January 6, 2027 |
| gpt-4o-mini-tts-2025-12-15 | OpenAI API; Replaced by gpt-realtime-2.1-mini | Oct 01 2026 | January 6, 2027 |
| anthropic.claude-opus-4-1-20250805-v1:0 | Amazon Bedrock; by Anthropic | Jul 08 2026 | January 8, 2027 |
| gpt-4o-2024-08-06 | OpenAI API; Replaced by gpt-6-luna | Oct 14 2026 | January 14, 2027 |
| gpt-4o-2024-11-20 | OpenAI API; Replaced by gpt-6-luna | Oct 14 2026 | January 14, 2027 |
| gpt-4o-mini-2024-07-18 | OpenAI API; Replaced by gpt-6-luna | Oct 14 2026 | January 14, 2027 |
| gpt-4.1-2025-04-14 | OpenAI API; Replaced by gpt-6-luna | Oct 14 2026 | January 14, 2027 |
| gpt-4.1-mini-2025-04-14 | OpenAI API; Replaced by gpt-6-luna | Oct 14 2026 | January 14, 2027 |
| gpt-realtime | OpenAI API; Replaced by gpt-realtime-2.1 | Jul 20 2026 | January 20, 2027 |
| gpt-realtime-2025-08-28 | OpenAI API; Replaced by gpt-realtime-2.1 | Jul 20 2026 | January 20, 2027 |
| gpt-audio | OpenAI API; Replaced by gpt-audio-1.5 | Jul 20 2026 | January 20, 2027 |
| gpt-audio-2025-08-28 | OpenAI API; Replaced by gpt-audio-1.5 | Jul 20 2026 | January 20, 2027 |
| gpt-4o-audio | OpenAI API; Replaced by gpt-audio-1.5 | Jul 20 2026 | January 20, 2027 |
| gpt-4o-realtime | OpenAI API; Replaced by gpt-realtime-2.1 | Jul 20 2026 | January 20, 2027 |
| gpt-realtime-mini | OpenAI API; Replaced by gpt-realtime-2.1-mini | Jul 20 2026 | January 20, 2027 |
| gpt-realtime-mini-2025-12-15 | OpenAI API; Replaced by gpt-realtime-2.1-mini | Jul 20 2026 | January 20, 2027 |
| gpt-audio-mini | OpenAI API; Replaced by gpt-audio-1.5 | Jul 20 2026 | January 20, 2027 |
| gpt-audio-mini-2025-12-15 | OpenAI API; Replaced by gpt-audio-1.5 | Jul 20 2026 | January 20, 2027 |
| gpt-4o-mini-realtime | OpenAI API; Replaced by gpt-realtime-2.1-mini | Jul 20 2026 | January 20, 2027 |
| gpt-4o-mini-audio | OpenAI API; Replaced by gpt-audio-1.5 | Jul 20 2026 | January 20, 2027 |
| gpt-4o-mini-transcribe-2025-03-20 | OpenAI API; Replaced by gpt-4o-mini-transcribe-2025-12-15 | Jul 20 2026 | January 20, 2027 |
| gpt-5.5-cyber-preview | OpenAI API; Replaced by The most capable cyber model available to you. | Nov 05 2026 | February 1, 2027 |
| claude-opus-4-6 | Anthropic API | n/a | Not before February 5, 2027 |
| claude-sonnet-4-6 | Anthropic API | n/a | Not before February 17, 2027 |
| whisper-1 | OpenAI API; Replaced by gpt-live-transcribe | Aug 26 2026 | February 26, 2027 |
| gpt-4o-transcribe | OpenAI API; Replaced by gpt-live-transcribe | Aug 26 2026 | February 26, 2027 |
| gpt-4o-mini-transcribe | OpenAI API; Replaced by gpt-live-transcribe | Aug 26 2026 | February 26, 2027 |
| gpt-4o-mini-transcribe-2025-12-15 | OpenAI API; Replaced by gpt-live-transcribe | Aug 26 2026 | February 26, 2027 |
| gpt-4o-transcribe-diarize | OpenAI API; Replaced by gpt-live-transcribe | Aug 26 2026 | February 26, 2027 |
| writer.palmyra-x4-v1:0 | Amazon Bedrock; by Writer | Sep 09 2026 | March 9, 2027 |
| gemini-2.5-flash-image | Gemini API; by Google; Replaced by gemini-3.1-flash-lite-image | n/a | March 15, 2027 |
| writer.palmyra-x5-v1:0 | Amazon Bedrock; by Writer | Sep 21 2026 | March 21, 2027 |
| twelvelabs.marengo-embed-3-0-v1:0 | Amazon Bedrock; by TwelveLabs | Sep 22 2026 | March 22, 2027 |
| twelvelabs.pegasus-1-2-v1:0 | Amazon Bedrock; by TwelveLabs | Sep 22 2026 | March 22, 2027 |
| deepseek.r1-v1:0 | Amazon Bedrock; by DeepSeek | Sep 30 2026 | March 30, 2027 |
| deepseek.v3-v1:0 (bedrock-runtime), deepseek.v3.1 (bedrock-mantle) | Amazon Bedrock; by DeepSeek | Sep 30 2026 | March 30, 2027 |
| google.gemma-3-12b-it | Amazon Bedrock; by Google | Sep 30 2026 | March 30, 2027 |
| google.gemma-3-27b-it | Amazon Bedrock; by Google | Sep 30 2026 | March 30, 2027 |
| google.gemma-3-4b-it | Amazon Bedrock; by Google | Sep 30 2026 | March 30, 2027 |
| meta.llama3-70b-instruct-v1:0 | Amazon Bedrock; by Meta | Sep 30 2026 | March 30, 2027 |
| meta.llama3-8b-instruct-v1:0 | Amazon Bedrock; by Meta | Sep 30 2026 | March 30, 2027 |
| meta.llama3-1-70b-instruct-v1:0 | Amazon Bedrock; by Meta | Sep 30 2026 | March 30, 2027 |
| meta.llama3-1-8b-instruct-v1:0 | Amazon Bedrock; by Meta | Sep 30 2026 | March 30, 2027 |
| meta.llama3-3-70b-instruct-v1:0 | Amazon Bedrock; by Meta | Sep 30 2026 | March 30, 2027 |
| meta.llama4-maverick-17b-instruct-v1:0 | Amazon Bedrock; by Meta | Sep 30 2026 | March 30, 2027 |
| meta.llama4-scout-17b-instruct-v1:0 | Amazon Bedrock; by Meta | Sep 30 2026 | March 30, 2027 |
| minimax.minimax-m2 | Amazon Bedrock; by MiniMax | Sep 30 2026 | March 30, 2027 |
| minimax.minimax-m2.1 | Amazon Bedrock; by MiniMax | Sep 30 2026 | March 30, 2027 |
| mistral.devstral-2-123b | Amazon Bedrock; by Mistral AI | Sep 30 2026 | March 30, 2027 |
| mistral.mistral-7b-instruct-v0:2 | Amazon Bedrock; by Mistral AI | Sep 30 2026 | March 30, 2027 |
| mistral.mixtral-8x7b-instruct-v0:1 | Amazon Bedrock; by Mistral AI | Sep 30 2026 | March 30, 2027 |
| mistral.voxtral-mini-3b-2507 | Amazon Bedrock; by Mistral AI | Sep 30 2026 | March 30, 2027 |
| mistral.voxtral-small-24b-2507 | Amazon Bedrock; by Mistral AI | Sep 30 2026 | March 30, 2027 |
| moonshot.kimi-k2-thinking (bedrock-runtime), moonshotai.kimi-k2-thinking (bedrock-mantle) | Amazon Bedrock; by Moonshot AI | Sep 30 2026 | March 30, 2027 |
| nvidia.nemotron-nano-12b-v2 | Amazon Bedrock; by NVIDIA | Sep 30 2026 | March 30, 2027 |
| nvidia.nemotron-nano-9b-v2 | Amazon Bedrock; by NVIDIA | Sep 30 2026 | March 30, 2027 |
| qwen.qwen3-coder-30b-a3b-v1:0 (bedrock-runtime), qwen.qwen3-coder-30b-a3b-instruct (bedrock-mantle) | Amazon Bedrock; by Qwen | Sep 30 2026 | March 30, 2027 |
| qwen.qwen3-coder-480b-a35b-v1:0 (bedrock-runtime), qwen.qwen3-coder-480b-a35b-instruct (bedrock-mantle) | Amazon Bedrock; by Qwen | Sep 30 2026 | March 30, 2027 |
| zai.glm-4.7 | Amazon Bedrock; by Z.AI | Sep 30 2026 | March 30, 2027 |
| zai.glm-4.7-flash | Amazon Bedrock; by Z.AI | Sep 30 2026 | March 30, 2027 |
| gpt-5.3-codex | OpenAI API; Replaced by gpt-6-sol | Oct 01 2026 | April 1, 2027 |
| gpt-5.4-nano | OpenAI API; Replaced by gpt-6-luna | Oct 01 2026 | April 1, 2027 |
| gpt-5.4-nano-2026-03-17 | OpenAI API; Replaced by gpt-6-luna | Oct 01 2026 | April 1, 2027 |
| gpt-5.1 | OpenAI API; Replaced by gpt-6-sol | Oct 01 2026 | April 1, 2027 |
| gpt-5.1-2025-11-13 | OpenAI API; Replaced by gpt-6-sol | Oct 01 2026 | April 1, 2027 |
| anthropic.claude-sonnet-4-5-20250929-v1:0 | Amazon Bedrock; by Anthropic | Oct 08 2026 | April 8, 2027 |
| claude-opus-4-7 | Anthropic API | n/a | Not before April 16, 2027 |
| mistral-medium-3-5 | Mistral API; by Mistral AI; Replaced by Mistral Large 4 | Oct 31 2026 | April 30, 2027 |
| voxtral-mini-tts-2603 | Mistral API; by Mistral AI; Replaced by Voxtral TTS 3 | Oct 31 2026 | April 30, 2027 |
| mistral-large-2512 | Mistral API; by Mistral AI; Replaced by Mistral Large 4 | Oct 31 2026 | April 30, 2027 |
| voxtral-mini-2602 | Mistral API; by Mistral AI; Replaced by Voxtral Mini Transcribe 3 | Oct 31 2026 | April 30, 2027 |
| voxtral-mini-transcribe-realtime-2602 | Mistral API; by Mistral AI; Replaced by Voxtral Mini Transcribe Realtime 3 | Oct 31 2026 | April 30, 2027 |
| gemini-3.1-flash-lite | Gemini API; by Google; Replaced by gemini-3.5-flash-lite | n/a | May 7, 2027 |
| claude-opus-4-8 | Anthropic API | n/a | Not before May 28, 2027 |
| claude-fable-5 | Anthropic API | n/a | Not before June 9, 2027 |
| claude-mythos-5 | Anthropic API | n/a | Not before June 9, 2027 |
| claude-sonnet-5 | Anthropic API | n/a | Not before June 30, 2027 |
| claude-opus-5 | Anthropic API | n/a | Not before July 24, 2027 |
| claude-fable-5-1 | Anthropic API | n/a | Not before September 1, 2027 |
| claude-mythos-5-1 | Anthropic API | n/a | Not before September 1, 2027 |
| claude-opus-5-5 | Anthropic API | n/a | Not before September 22, 2027 |
| claude-sonnet-5-5 | Anthropic API | n/a | Not before September 28, 2027 |
| claude-haiku-5-5 | Anthropic API | n/a | Not before October 7, 2027 |
| claude-fable-5-2 | Anthropic API | n/a | Not before November 9, 2027 |
| claude-mythos-5-2 | Anthropic API | n/a | Not before November 9, 2027 |
| gemini-embedding-001 | Gemini API; by Google; Replaced by gemini-embedding-2 | n/a | May 14, 2028 |
| claude-mythos-preview | Anthropic API | Jun 09 2026 | None announced |
| gpt-4-0314 | OpenAI API; Replaced by gpt-5 | Sep 26 2025 | None announced |
| gpt-4-0125-preview | OpenAI API; Replaced by gpt-5 | Sep 26 2025 | None announced |
| gpt-4-turbo-preview | OpenAI API; Replaced by gpt-5 | Sep 26 2025 | None announced |
| gpt-4-turbo-preview-completions | OpenAI API; Replaced by gpt-5 | Sep 26 2025 | None announced |
| gemini-3.8-flash-tts | Gemini API; by Google | n/a | None announced |
| gemini-3.8-flash-lite-tts | Gemini API; by Google | n/a | None announced |
| gemini-3.8-live | Gemini API; by Google | n/a | None announced |
| gemini-3.8-live-extended-thinking | Gemini API; by Google | n/a | None announced |
| gemini-3.8-flash | Gemini API; by Google | n/a | None announced |
| gemini-3.7-flash | Gemini API; by Google; Replaced by gemini-3.8-flash | n/a | None announced |
| gemini-3.6-flash | Gemini API; by Google | n/a | None announced |
| gemini-3.5-flash-lite | Gemini API; by Google | n/a | None announced |
| gemini-3.5-flash | Gemini API; by Google; Replaced by gemini-3.6-flash | n/a | None announced |
| gemini-3.1-pro-preview | Gemini API; by Google | n/a | None announced |
| gemini-3-flash-preview | Gemini API; by Google; Replaced by gemini-3.6-flash | n/a | None announced |
| gemini-2.5-flash-lite | Gemini API; by Google | n/a | None announced |
| gemini-3.5-transcribe-live | Gemini API; by Google | n/a | None announced |
| gemini-3.5-live-translate-preview | Gemini API; by Google | n/a | None announced |
| gemini-3.5-transcribe | Gemini API; by Google | n/a | None announced |
| gemini-embedding-2 | Gemini API; by Google | n/a | None announced |
| gemini-nano-banana-2.1 | Gemini API; by Google | n/a | None announced |
| gemini-3.1-flash-lite-image | Gemini API; by Google | n/a | None announced |
| gemini-3.1-flash-image | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | None announced |
| gemini-3-pro-image | Gemini API; by Google | n/a | None announced |
| gemini-omni-1.2-flash | Gemini API; by Google | n/a | None announced |
| lyria-3.5 | Gemini API; by Google | n/a | None announced |
| lyria-3-clip-preview | Gemini API; by Google | n/a | None announced |
| lyria-3-pro-preview | Gemini API; by Google; Replaced by lyria-3.5 | n/a | None announced |
| lyria-realtime-exp | Gemini API; by Google | n/a | None announced |
| gemini-robotics-er-2-preview | Gemini API; by Google | n/a | None announced |
| gemini-robotics-er-2-streaming-preview | Gemini API; by Google | n/a | None announced |
| gemini-embedding-2-preview | Gemini API; by Google; Replaced by gemini-embedding-2 | n/a | November 6, 2026 |
| zai-glm-5-2 | Mistral API; by Mistral AI; Replaced by Z.ai GLM 5.3 | Sep 29 2026 | October 31, 2026 |
| gpt-3.5-turbo-0125 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-3.5-turbo | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-3.5-turbo-completions | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4-0613 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4-0613-completions | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4-completions | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4-1106-preview | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4-turbo | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4-turbo-2024-04-09 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4-turbo-completions | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4.1-nano | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4.1-nano-2025-04-14 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-4o-2024-05-13 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| gpt-image-1 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| o1-2024-12-17 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| o1 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| o1-pro-2025-03-19 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| o1-pro | OpenAI API | Apr 22 2026 | October 23, 2026 |
| o3-mini-2025-01-31 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| o3-mini | OpenAI API | Apr 22 2026 | October 23, 2026 |
| ft-o4-mini-2025-04-16 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| o4-mini-2025-04-16 | OpenAI API | Apr 22 2026 | October 23, 2026 |
| o4-mini | OpenAI API | Apr 22 2026 | October 23, 2026 |
| veo-3.1-lite-generate-preview | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | October 22, 2026 |
| veo-3.1-generate-preview | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | October 22, 2026 |
| veo-3.1-fast-generate-preview | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | October 22, 2026 |
| gemini-omni-flash-preview | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | October 22, 2026 |
| anthropic.claude-sonnet-4-20250514-v1:0 | Amazon Bedrock; by Anthropic | Apr 14 2026 | October 14, 2026 |
| gpt-5.4-cyber | OpenAI API; Replaced by The most capable cyber model available to you. | Sep 11 2026 | October 1, 2026 |
| labs-leanstral-1-5 | Mistral API; by Mistral AI | Sep 29 2026 | September 30, 2026 |
| mistral-ocr-4-0 | Mistral API; by Mistral AI; Replaced by OCR 4.1 | Sep 29 2026 | September 30, 2026 |
| amazon.nova-canvas-v1:0 | Amazon Bedrock | Mar 30 2026 | September 30, 2026 |
| amazon.nova-reel-v1:0 | Amazon Bedrock | Mar 30 2026 | September 30, 2026 |
| amazon.nova-reel-v1:1 | Amazon Bedrock | Mar 30 2026 | September 30, 2026 |
| gpt-3.5-turbo-instruct | OpenAI API; Replaced by gpt-5.6-terra | Sep 26 2025 | September 28, 2026 |
| babbage-002 | OpenAI API; Replaced by gpt-5.6-terra | Sep 26 2025 | September 28, 2026 |
| davinci-002 | OpenAI API; Replaced by gpt-5.6-terra | Sep 26 2025 | September 28, 2026 |
| gpt-3.5-turbo-1106 | OpenAI API; Replaced by gpt-5.6-terra | Sep 26 2025 | September 28, 2026 |
| sora-2 | OpenAI API; Replaced by --- | Mar 24 2026 | September 24, 2026 |
| sora-2-pro | OpenAI API; Replaced by --- | Mar 24 2026 | September 24, 2026 |
| sora-2-2025-10-06 | OpenAI API; Replaced by --- | Mar 24 2026 | September 24, 2026 |
| sora-2-2025-12-08 | OpenAI API; Replaced by --- | Mar 24 2026 | September 24, 2026 |
| sora-2-pro-2025-10-06 | OpenAI API; Replaced by --- | Mar 24 2026 | September 24, 2026 |
| amazon.nova-sonic-v1:0 | Amazon Bedrock | Mar 13 2026 | September 14, 2026 |
| gemini-robotics-er-1.6-preview | Gemini API; by Google; Replaced by gemini-robotics-er-2-preview | n/a | August 31, 2026 |
| mistral-medium-2508 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | May 22 2026 | August 31, 2026 |
| mistral-medium-2505 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | May 22 2026 | August 31, 2026 |
| cohere.command-r-v1:0 | Amazon Bedrock; by Cohere | Feb 19 2026 | August 19, 2026 |
| cohere.command-r-plus-v1:0 | Amazon Bedrock; by Cohere | Feb 19 2026 | August 19, 2026 |
| imagen-4.0-generate-001 | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | August 17, 2026 |
| imagen-4.0-ultra-generate-001 | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | August 17, 2026 |
| imagen-4.0-fast-generate-001 | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | August 17, 2026 |
| gpt-5.2-chat-latest | OpenAI API; Replaced by gpt-5.6-sol | May 08 2026 | August 10, 2026 |
| gpt-5.3-chat-latest | OpenAI API; Replaced by gpt-5.6-sol | May 08 2026 | August 10, 2026 |
| claude-opus-4-1-20250805 | Anthropic API; Replaced by claude-opus-4-8 | Jun 05 2026 | August 5, 2026 |
| mistral-small-2506 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Apr 30 2026 | July 31, 2026 |
| devstral-2512 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | May 22 2026 | July 31, 2026 |
| magistral-medium-2509 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | May 22 2026 | July 31, 2026 |
| magistral-small-2509 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Apr 30 2026 | July 31, 2026 |
| open-mistral-nemo-2407 | Mistral API; by Mistral AI; Replaced by Ministral 3 8B | May 22 2026 | July 31, 2026 |
| gemini-2.5-computer-use-preview-10-2025 | Gemini API; by Google; Replaced by gemini-3.8-flash | n/a | July 28, 2026 |
| computer-use-preview-2025-03-11 | OpenAI API | Apr 22 2026 | July 23, 2026 |
| computer-use-preview | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-4o-mini-search-preview-2025-03-11 | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-4o-search-preview-2025-03-11 | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-5-chat-latest | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-5-codex | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-5.1-chat-latest | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-5.1-codex | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-5.1-codex-max | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-5.1-codex-mini | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-audio-mini-2025-10-06 | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-realtime-mini-2025-10-06 | OpenAI API | Apr 22 2026 | July 23, 2026 |
| o3-deep-research-2025-06-26 | OpenAI API | Apr 22 2026 | July 23, 2026 |
| o3-deep-research | OpenAI API | Apr 22 2026 | July 23, 2026 |
| o4-mini-deep-research-2025-06-26 | OpenAI API | Apr 22 2026 | July 23, 2026 |
| o4-mini-deep-research | OpenAI API | Apr 22 2026 | July 23, 2026 |
| gpt-5.2-codex | OpenAI API | Apr 22 2026 | July 23, 2026 |
| meta.llama3-1-405b-instruct-v1:0 | Amazon Bedrock; by Meta | Jan 07 2026 | July 7, 2026 |
| meta.llama3-2-11b-instruct-v1:0 | Amazon Bedrock; by Meta | Jan 07 2026 | July 7, 2026 |
| meta.llama3-2-1b-instruct-v1:0 | Amazon Bedrock; by Meta | Jan 07 2026 | July 7, 2026 |
| meta.llama3-2-3b-instruct-v1:0 | Amazon Bedrock; by Meta | Jan 07 2026 | July 7, 2026 |
| meta.llama3-2-90b-instruct-v1:0 | Amazon Bedrock; by Meta | Jan 07 2026 | July 7, 2026 |
| veo-3.0-generate-001 | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | June 30, 2026 |
| veo-3.0-fast-generate-001 | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | June 30, 2026 |
| veo-2.0-generate-001 | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | June 30, 2026 |
| labs-leanstral-2603 | Mistral API; by Mistral AI; Replaced by Leanstral 1.5 | May 22 2026 | June 30, 2026 |
| mistral-moderation-2411 | Mistral API; by Mistral AI; Replaced by Mistral Moderation 2 | Mar 31 2026 | June 30, 2026 |
| gemini-3.1-flash-image-preview | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | June 25, 2026 |
| gemini-3-pro-image-preview | Gemini API; by Google; Replaced by gemini-3-pro-image | n/a | June 25, 2026 |
| claude-opus-4-20250514 | Anthropic API; Replaced by claude-opus-4-8 | Apr 14 2026 | June 15, 2026 |
| claude-sonnet-4-20250514 | Anthropic API; Replaced by claude-sonnet-5-5 | Apr 14 2026 | June 15, 2026 |
| gemini-2.0-flash | Gemini API; by Google; Replaced by gemini-3.6-flash | n/a | June 1, 2026 |
| gemini-2.0-flash-001 | Gemini API; by Google; Replaced by gemini-3.6-flash | n/a | June 1, 2026 |
| gemini-2.0-flash-lite | Gemini API; by Google; Replaced by gemini-3.1-flash-lite | n/a | June 1, 2026 |
| gemini-2.0-flash-lite-001 | Gemini API; by Google; Replaced by gemini-3.1-flash-lite | n/a | June 1, 2026 |
| voxtral-mini-2507 | Mistral API; by Mistral AI; Replaced by Voxtral Mini Transcribe 2 | Feb 27 2026 | May 31, 2026 |
| devstral-medium-2507 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | Feb 27 2026 | May 31, 2026 |
| devstral-small-2507 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Feb 27 2026 | May 31, 2026 |
| mistral-ocr-2505 | Mistral API; by Mistral AI; Replaced by OCR 4.1 | Feb 27 2026 | May 31, 2026 |
| mistral-large-2411 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | Feb 27 2026 | May 31, 2026 |
| pixtral-large-2411 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | Feb 27 2026 | May 31, 2026 |
| gemini-3.1-flash-lite-preview | Gemini API; by Google; Replaced by gemini-3.1-flash-lite | n/a | May 25, 2026 |
| dall-e-2 | OpenAI API; Replaced by gpt-image-2 | Nov 14 2025 | May 12, 2026 |
| dall-e-3 | OpenAI API; Replaced by gpt-image-2 | Nov 14 2025 | May 12, 2026 |
| gpt-4o-realtime-preview | OpenAI API; Replaced by gpt-realtime-1.5 | Sep 15 2025 | May 7, 2026 |
| gpt-4o-realtime-preview-2025-06-03 | OpenAI API; Replaced by gpt-realtime-1.5 | Sep 15 2025 | May 7, 2026 |
| gpt-4o-realtime-preview-2024-12-17 | OpenAI API; Replaced by gpt-realtime-1.5 | Sep 15 2025 | May 7, 2026 |
| gpt-4o-mini-realtime-preview | OpenAI API; Replaced by gpt-realtime-mini | Sep 15 2025 | May 7, 2026 |
| gpt-4o-audio-preview | OpenAI API; Replaced by gpt-audio-1.5 | Sep 15 2025 | May 7, 2026 |
| gpt-4o-mini-audio-preview | OpenAI API; Replaced by gpt-audio-mini | Sep 15 2025 | May 7, 2026 |
| gemini-robotics-er-1.5-preview | Gemini API; by Google; Replaced by gemini-robotics-er-2-preview | n/a | April 30, 2026 |
| labs-mistral-small-creative | Mistral API; by Mistral AI; Replaced by Ministral 3 8B | Mar 31 2026 | April 30, 2026 |
| claude-3-haiku-20240307 | Anthropic API; Replaced by claude-haiku-4-5-20251001 | Feb 19 2026 | April 20, 2026 |
| embed-english-v2.0 | Cohere API | Apr 04 2026 | April 4, 2026 |
| embed-english-light-v2.0 | Cohere API | Apr 04 2026 | April 4, 2026 |
| embed-multilingual-v2.0 | Cohere API | Apr 04 2026 | April 4, 2026 |
| c4ai-aya-expanse-8b | Cohere API | Apr 04 2026 | April 4, 2026 |
| c4ai-aya-vision-8b | Cohere API | Apr 04 2026 | April 4, 2026 |
| gemini-2.5-flash-lite-preview-09-2025 | Gemini API; by Google; Replaced by gemini-3.1-flash-lite | n/a | March 31, 2026 |
| labs-devstral-small-2512 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | Feb 27 2026 | March 31, 2026 |
| gemini-3-pro-preview | Gemini API; by Google; Replaced by gemini-3.1-pro-preview | n/a | March 9, 2026 |
| claude-3-7-sonnet-20250219 | Anthropic API; Replaced by claude-sonnet-5-5 | Oct 28 2025 | February 19, 2026 |
| claude-3-5-haiku-20241022 | Anthropic API; Replaced by claude-haiku-4-5-20251001 | Dec 19 2025 | February 19, 2026 |
| chatgpt-4o-latest | OpenAI API; Replaced by gpt-5.1-chat-latest | Nov 18 2025 | February 17, 2026 |
| gemini-2.5-flash-preview-09-2025 | Gemini API; by Google; Replaced by gemini-3.6-flash | n/a | February 17, 2026 |
| imagen-4.0-generate-preview-06-06 | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | February 17, 2026 |
| imagen-4.0-ultra-generate-preview-06-06 | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | February 17, 2026 |
| codex-mini-latest | OpenAI API; Replaced by gpt-5-codex-mini | Nov 17 2025 | February 12, 2026 |
| gemini-2.5-flash-image-preview | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | January 15, 2026 |
| text-embedding-004 | Gemini API; by Google; Replaced by gemini-embedding-2 | n/a | January 14, 2026 |
| mistral-ocr-2503 | Mistral API; by Mistral AI; Replaced by OCR 4.1 | Dec 02 2025 | December 31, 2025 |
| ministral-3b-2410 | Mistral API; by Mistral AI; Replaced by Ministral 3 3B | Dec 02 2025 | December 31, 2025 |
| ministral-8b-2410 | Mistral API; by Mistral AI; Replaced by Ministral 3 8B | Dec 02 2025 | December 31, 2025 |
| pixtral-12b-2409 | Mistral API; by Mistral AI; Replaced by Ministral 3 14B | Dec 02 2025 | December 31, 2025 |
| gemini-2.0-flash-lite-preview | Gemini API; by Google; Replaced by gemini-3.1-flash-lite | n/a | December 9, 2025 |
| gemini-2.0-flash-lite-preview-02-05 | Gemini API; by Google; Replaced by gemini-3.1-flash-lite | n/a | December 9, 2025 |
| gemini-live-2.5-flash-preview | Gemini API; by Google; Replaced by gemini-3.8-live | n/a | December 9, 2025 |
| gemini-2.0-flash-live-001 | Gemini API; by Google; Replaced by gemini-3.8-live | n/a | December 9, 2025 |
| gemini-2.5-pro-preview-03-25 | Gemini API; by Google; Replaced by gemini-3.8-flash | n/a | December 2, 2025 |
| gemini-2.5-pro-preview-05-06 | Gemini API; by Google; Replaced by gemini-3.8-flash | n/a | December 2, 2025 |
| gemini-2.5-pro-preview-06-05 | Gemini API; by Google; Replaced by gemini-3.8-flash | n/a | December 2, 2025 |
| magistral-medium-2507 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | Oct 31 2025 | November 30, 2025 |
| magistral-small-2507 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Oct 31 2025 | November 30, 2025 |
| magistral-medium-2506 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | Oct 31 2025 | November 30, 2025 |
| magistral-small-2506 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Oct 31 2025 | November 30, 2025 |
| devstral-small-2505 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | Oct 31 2025 | November 30, 2025 |
| mistral-small-2503 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Nov 06 2025 | November 30, 2025 |
| mistral-small-2501 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Nov 06 2025 | November 30, 2025 |
| codestral-2501 | Mistral API; by Mistral AI; Replaced by Codestral | Nov 06 2025 | November 30, 2025 |
| mistral-small-2409 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Nov 06 2025 | November 30, 2025 |
| gemini-2.5-flash-preview-05-20 | Gemini API; by Google; Replaced by gemini-3.6-flash | n/a | November 18, 2025 |
| gemini-2.0-flash-preview-image-generation | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | November 14, 2025 |
| veo-3.0-generate-preview | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | November 12, 2025 |
| veo-3.0-fast-generate-preview | Gemini API; by Google; Replaced by gemini-omni-1.2-flash | n/a | November 12, 2025 |
| imagen-3.0-generate-002 | Gemini API; by Google; Replaced by gemini-nano-banana-2.1 | n/a | November 10, 2025 |
| embedding-001 | Gemini API; by Google; Replaced by gemini-embedding-2 | n/a | October 30, 2025 |
| embedding-gecko-001 | Gemini API; by Google; Replaced by gemini-embedding-2 | n/a | October 30, 2025 |
| gemini-embedding-exp | Gemini API; by Google; Replaced by gemini-embedding-2 | n/a | October 30, 2025 |
| gemini-embedding-exp-03-07 | Gemini API; by Google; Replaced by gemini-embedding-2 | n/a | October 30, 2025 |
| text-moderation-007 | OpenAI API; Replaced by omni-moderation | Apr 28 2025 | October 27, 2025 |
| text-moderation-stable | OpenAI API; Replaced by omni-moderation | Apr 28 2025 | October 27, 2025 |
| text-moderation-latest | OpenAI API; Replaced by omni-moderation | Apr 28 2025 | October 27, 2025 |
| o1-mini | OpenAI API; Replaced by o4-mini | Apr 28 2025 | October 27, 2025 |
| gpt-4o-realtime-preview-2024-10-01 | OpenAI API; Replaced by gpt-realtime-1.5 | Jun 10 2025 | October 10, 2025 |
| gpt-4o-audio-preview-2024-10-01 | OpenAI API; Replaced by gpt-audio-1.5 | Jun 10 2025 | October 10, 2025 |
| mistral-saba-2502 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Jun 10 2025 | September 30, 2025 |
| command-r-03-2024 | Cohere API | Sep 15 2025 | September 15, 2025 |
| command-r-plus-04-2024 | Cohere API | Sep 15 2025 | September 15, 2025 |
| command-light | Cohere API | Sep 15 2025 | September 15, 2025 |
| command | Cohere API | Sep 15 2025 | September 15, 2025 |
| summarize | Cohere API | Sep 15 2025 | September 15, 2025 |
| o1-preview | OpenAI API; Replaced by o3 | Apr 28 2025 | July 28, 2025 |
| gpt-4.5-preview | OpenAI API; Replaced by gpt-4.1 | Apr 14 2025 | July 14, 2025 |
| codestral-2405 | Mistral API; by Mistral AI; Replaced by Codestral | Dec 02 2024 | June 16, 2025 |
| mistral-small-2402 | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Nov 30 2024 | June 16, 2025 |
| mistral-large-2402 | Mistral API; by Mistral AI; Replaced by Mistral Large 3 | Nov 30 2024 | June 16, 2025 |
| mistral-medium-2312 | Mistral API; by Mistral AI; Replaced by Mistral Medium 3.5 | Nov 30 2024 | June 16, 2025 |
| gpt-4-32k | OpenAI API; Replaced by gpt-4o | Jun 06 2024 | June 6, 2025 |
| gpt-4-32k-0613 | OpenAI API; Replaced by gpt-4o | Jun 06 2024 | June 6, 2025 |
| gpt-4-32k-0314 | OpenAI API; Replaced by gpt-4o | Jun 06 2024 | June 6, 2025 |
| open-codestral-mamba | Mistral API; by Mistral AI; Replaced by Codestral | Jun 06 2025 | June 6, 2025 |
| rerank-english-v2.0 | Cohere API; Replaced by rerank-v3.5 | Dec 02 2024 | April 30, 2025 |
| rerank-multilingual-v2.0 | Cohere API; Replaced by rerank-v3.5 | Dec 02 2024 | April 30, 2025 |
| mistral-large-2407 | Mistral API; by Mistral AI; Replaced by Mistral Large 3 | Nov 30 2024 | March 30, 2025 |
| open-mistral-7b | Mistral API; by Mistral AI; Replaced by Ministral 3 8B | Nov 30 2024 | March 30, 2025 |
| open-mixtral-8x22b | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Nov 30 2024 | March 30, 2025 |
| open-mixtral-8x7b | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | Nov 30 2024 | March 30, 2025 |
| gpt-4-vision-preview | OpenAI API; Replaced by gpt-4o | Jun 06 2024 | December 6, 2024 |
| gpt-4-1106-vision-preview | OpenAI API; Replaced by gpt-4o | Jun 06 2024 | December 6, 2024 |
| gpt-3.5-turbo-0613 | OpenAI API; Replaced by gpt-3.5-turbo | Nov 06 2023 | September 13, 2024 |
| gpt-3.5-turbo-16k-0613 | OpenAI API; Replaced by gpt-3.5-turbo | Nov 06 2023 | September 13, 2024 |
| gpt-3.5-turbo-0301 | OpenAI API; Replaced by gpt-3.5-turbo | Jun 13 2023 | September 13, 2024 |
| text-ada-001 | OpenAI API; Replaced by gpt-3.5-turbo-instruct | Jul 06 2023 | January 4, 2024 |
| text-babbage-001 | OpenAI API; Replaced by gpt-3.5-turbo-instruct | Jul 06 2023 | January 4, 2024 |
| text-curie-001 | OpenAI API; Replaced by gpt-3.5-turbo-instruct | Jul 06 2023 | January 4, 2024 |
| text-davinci-001 | OpenAI API; Replaced by gpt-3.5-turbo-instruct | Jul 06 2023 | January 4, 2024 |
| text-davinci-002 | OpenAI API; Replaced by gpt-3.5-turbo-instruct | Jul 06 2023 | January 4, 2024 |
| text-davinci-003 | OpenAI API; Replaced by gpt-3.5-turbo-instruct | Jul 06 2023 | January 4, 2024 |
| ada | OpenAI API; Replaced by babbage-002 | Jul 06 2023 | January 4, 2024 |
| babbage | OpenAI API; Replaced by babbage-002 | Jul 06 2023 | January 4, 2024 |
| curie | OpenAI API; Replaced by davinci-002 | Jul 06 2023 | January 4, 2024 |
| davinci | OpenAI API; Replaced by davinci-002 | Jul 06 2023 | January 4, 2024 |
| code-davinci-002 | OpenAI API; Replaced by gpt-3.5-turbo-instruct | Jul 06 2023 | January 4, 2024 |
| text-davinci-edit-001 | OpenAI API; Replaced by gpt-4o | Jul 06 2023 | January 4, 2024 |
| code-davinci-edit-001 | OpenAI API; Replaced by gpt-4o | Jul 06 2023 | January 4, 2024 |
| text-similarity-ada-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-search-ada-doc-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-search-ada-query-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| code-search-ada-code-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| code-search-ada-text-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-similarity-babbage-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-search-babbage-doc-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-search-babbage-query-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| code-search-babbage-code-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| code-search-babbage-text-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-similarity-curie-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-search-curie-doc-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-search-curie-query-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-similarity-davinci-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-search-davinci-doc-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| text-search-davinci-query-001 | OpenAI API; Replaced by text-embedding-3-small | Jul 06 2023 | January 4, 2024 |
| code-davinci-001 | OpenAI API; Replaced by gpt-4o | Mar 20 2023 | March 23, 2023 |
| code-cushman-002 | OpenAI API; Replaced by gpt-4o | Mar 20 2023 | March 23, 2023 |
| code-cushman-001 | OpenAI API; Replaced by gpt-4o | Mar 20 2023 | March 23, 2023 |
| null | Mistral API; by Mistral AI; Replaced by Mistral Small 4 | n/a | None announced |

## Method

- Sources. Every row comes from the platform's own deprecation page, listed under Sources. A script reads those pages, respects each site's robots.txt, and stops if a table changes shape.
- Dates are per platform. A model can retire on different dates on different platforms, so Claude on Amazon Bedrock and on the Anthropic API are separate rows.
- Date types. retire_precision is exact for a confirmed date, not_before when the platform commits only to an earliest date, and none when no date is announced.
- Status. A model whose exact shutdown date has passed is marked retired, whatever the source page last said. Otherwise the status is the platform's own.
- Amazon Bedrock. A model can retire on different dates by region. This keeps the earliest. Rows cover models in its Legacy or end-of-life list.
- Not included. OpenAI's non-model shutdowns such as endpoints, and Microsoft Foundry, whose retirement dates are spread across model-specific notices.

## Questions

### What is the difference between deprecated and retired?

A deprecated model still works, but its provider has announced that it will end, usually with a replacement. A retired model no longer answers requests.

### Which models are retiring soon?

As of November 10, 2026, the next shutdowns are anthropic.claude-3-haiku-20240307-v1:0 on Amazon Bedrock (November 13, 2026), anthropic.claude-3-sonnet-20240229-v1:0 on Amazon Bedrock (November 13, 2026), and anthropic.claude-3-5-sonnet-20240620-v1:0 on Amazon Bedrock (November 13, 2026). The table is sorted with the soonest first.

### How much notice do providers give?

It varies by platform and by model. The findings above give the median for each platform, calculated from the announced deprecation and shutdown dates.

## Sources

1. [Anthropic API deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations)
2. [OpenAI API deprecations](https://developers.openai.com/api/docs/deprecations)
3. [Gemini API deprecations](https://aistudio.google.com/docs/deprecations)
4. [Mistral API deprecations](https://docs.mistral.ai/models)
5. [Cohere API deprecations](https://docs.cohere.com/docs/deprecations)
6. [Amazon Bedrock deprecations](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle-legacy.html)

## Cite this dataset

AI for Devs. "LLM model retirement calendar." Published November 10, 2026. https://aifordevs.org/data/model-retirements

- Page: https://aifordevs.org/data/model-retirements
- Markdown: https://aifordevs.org/data/model-retirements.md
- Reuse: Free to reuse under CC BY 4.0 with a link to this page. License: https://creativecommons.org/licenses/by/4.0/
- Publisher: AI for Devs
- Published: 2026-11-10

---
title: "Open-weight LLMs"
type: "dataset"
url: "https://aifordevs.org/data/open-weight-models"
markdown: "https://aifordevs.org/data/open-weight-models.md"
publisher: "AI for Devs"
license: "https://creativecommons.org/licenses/by/4.0/"
attribution: "Credit AI for Devs and link to https://aifordevs.org/data/open-weight-models"
description: "License, parameter count, downloads, and release date for 901 open-weight text and multimodal models from 24 labs, read from the Hugging Face Hub. Checked November 10, 2026."
checked: "2026-11-10"
csv: "https://aifordevs.org/data/open-weight-models.csv"
json: "https://aifordevs.org/data/open-weight-models.json"
rows: 901
---

# Open-weight LLMs

License, parameter count, downloads, and release date for 901 open-weight text and multimodal models from 24 labs, read from the Hugging Face Hub. Checked November 10, 2026.

Download: [CSV](https://aifordevs.org/data/open-weight-models.csv) | [JSON](https://aifordevs.org/data/open-weight-models.json)

## Key findings

- As of November 10, 2026, the most downloaded model here is Qwen/Qwen3-0.6B, with 20.7M downloads in the last 30 days.
- Qwen leads by downloads, with 150.5M across its 50 models, followed by DeepSeek with 23M.
- 60.8 percent of the models with a stated license use Apache 2.0, MIT, or a similar permissive license. The most common is apache-2.0, on 344 models.
- The largest model by parameter count is moonshotai/Kimi-K3.1-Pro, at 2.8T parameters.
- 138 of the 901 models require accepting terms before download.
- The newest model in the list is tencent/Qwen3.8-27B-DSpine-b16, created on November 7, 2026.

## The data

Most downloaded first, checked November 10, 2026. Filter by lab or license, and sort any column.

| Model | Details | License | Parameters | Downloads | Likes | Created |
| --- | --- | --- | --- | --- | --- | --- |
| Qwen3-0.6B | Qwen | apache-2.0 | 751.6M | 20.7M | 1,888 | Apr 27 2025 |
| Qwen3-14B | Qwen | apache-2.0 | 14.8B | 12.9M | 536 | Apr 27 2025 |
| Florence-2-base | Microsoft | mit | 231.6M | 11.7M | 437 | Jun 15 2024 |
| DeepSeek-V3.2 | DeepSeek | mit | 685.4B | 7.6M | 1,597 | Dec 01 2025 |
| Qwen3.8-27B | Qwen | apache-2.0 | 27.8B | 7.5M | 19,478 | Aug 05 2026 |
| Qwen3-VL-8B-Instruct | Qwen | apache-2.0 | 8.8B | 7.1M | 1,247 | Oct 11 2025 |
| gemma-4-31B-it | Google | apache-2.0 | 31.3B | 7.1M | 4,326 | Mar 11 2026 |
| Qwen3-8B | Qwen | apache-2.0 | 8.2B | 6.8M | 2,180 | Apr 27 2025 |
| GLM-5.3-Flash | Z.ai | mit | 321.3B | 6.4M | 3,103 | Aug 25 2026 |
| Qwen3.5-9B | Qwen | apache-2.0 | 9.7B | 6.4M | 2,339 | Feb 27 2026 |
| Qwen2.5-7B-Instruct | Qwen | apache-2.0 | 7.6B | 6M | 2,488 | Sep 16 2024 |
| Qwen2.5-VL-7B-Instruct | Qwen | apache-2.0 | 8.3B | 6M | 1,767 | Jan 26 2025 |
| gpt-oss-20b | OpenAI | apache-2.0 | 20.9B | 5.8M | 5,291 | Aug 04 2025 |
| gemma-4-26B-A4B-it | Google | apache-2.0 | 25.8B | 5.5M | 1,750 | Mar 11 2026 |
| Unlimited-OCR | Baidu | mit | 3.3B | 5.4M | 4,992 | Jun 19 2026 |
| Qwen3.5-4B | Qwen | apache-2.0 | 4.7B | 5.3M | 1,211 | Feb 27 2026 |
| Qwen2.5-1.5B-Instruct | Qwen | apache-2.0 | 1.5B | 5M | 937 | Sep 17 2024 |
| Qwen2.5-0.5B-Instruct | Qwen | apache-2.0 | 494M | 4.5M | 737 | Sep 16 2024 |
| Llama-3.1-8B-Instruct | Meta | llama3.1 | 8B | 4.5M | 8,862 | Jul 18 2024 |
| Qwen3.6-35B-A3B | Qwen | apache-2.0 | 36B | 4.2M | 3,108 | Apr 15 2026 |
| Qwen3-4B | Qwen | apache-2.0 | 4B | 4M | 794 | Apr 27 2025 |
| Qwen3.6-35B-A3B-NVFP4 | NVIDIA | apache-2.0 | 18.7B | 4M | 677 | May 27 2026 |
| Qwen3-VL-4B-Instruct | Qwen | apache-2.0 | 4.4B | 3.7M | 528 | Oct 11 2025 |
| Qwen3.5-2B | Qwen | apache-2.0 | 2.3B | 3.7M | 475 | Feb 28 2026 |
| Qwen3-1.7B | Qwen | apache-2.0 | 2B | 3.7M | 616 | Apr 27 2025 |
| Qwen3.5-0.8B | Qwen | apache-2.0 | 873.4M | 3.5M | 827 | Feb 28 2026 |
| Qwen3.8-Flash-Next | Qwen | qwen-community-1.0 | 180B | 3.5M | 6,878 | Aug 24 2026 |
| Qwen3-32B | Qwen | apache-2.0 | 32.8B | 3M | 792 | Apr 27 2025 |
| DeepSeek-V4-Flash-0731 | DeepSeek | mit | 304.2B | 2.7M | 4,099 | Jul 31 2026 |
| Qwen2.5-3B-Instruct | Qwen | qwen-research | 3.1B | 2.6M | 638 | Sep 17 2024 |
| Qwen3-4B-Instruct-2507 | Qwen | apache-2.0 | 4B | 2.6M | 1,049 | Aug 05 2025 |
| SmolLM2-135M | Hugging Face | apache-2.0 | 134.5M | 2.5M | 268 | Oct 31 2024 |
| gpt-oss-120b | OpenAI | apache-2.0 | 116.8B | 2.4M | 5,502 | Aug 04 2025 |
| Qwen3.6-27B | Qwen | apache-2.0 | 27.8B | 2.3M | 2,351 | Apr 21 2026 |
| GLM-5.3 | Z.ai | glm-5.3 | 753.3B | 2M | 2,401 | Aug 25 2026 |
| DeepSeek-OCR | DeepSeek | mit | 3.3B | 2M | 3,462 | Oct 17 2025 |
| Qwen2.5-Coder-14B-Instruct | Qwen | apache-2.0 | 14.8B | 1.9M | 213 | Nov 06 2024 |
| NVIDIA-Nemotron-3-Nano-4B-BF16 | NVIDIA | nvidia-nemotron-open-model-license | 4B | 1.9M | 141 | Mar 07 2026 |
| SmolVLM2-500M-Video-Instruct | Hugging Face | apache-2.0 | 507.5M | 1.8M | 203 | Feb 11 2025 |
| DeepSeek-V4.1-Flash | DeepSeek | mit | 763.2B | 1.8M | 4,921 | Sep 10 2026 |
| Gemma-4-26B-A4B-NVFP4 | NVIDIA | apache-license-2.0 | 14.4B | 1.7M | 169 | May 01 2026 |
| Qwen2.5-VL-3B-Instruct | Qwen | qwen-research | 3.8B | 1.7M | 745 | Jan 26 2025 |
| Qwen3.5-27B | Qwen | apache-2.0 | 27.8B | 1.7M | 1,091 | Feb 24 2026 |
| Qwen3-Coder-30B-A3B-Instruct | Qwen | apache-2.0 | 30.5B | 1.6M | 1,324 | Jul 31 2025 |
| DeepSeek-R1 | DeepSeek | mit | 684.5B | 1.6M | 14,443 | Jan 20 2025 |
| Qwen3.5-35B-A3B | Qwen | apache-2.0 | 36B | 1.6M | 1,557 | Feb 24 2026 |
| NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 | NVIDIA | nvidia-nemotron-open-model-license | 31.6B | 1.6M | 840 | Dec 04 2025 |
| DeepSeek-V3 | DeepSeek | n/a | 684.5B | 1.5M | 4,462 | Dec 25 2024 |
| Mistral-7B-Instruct-v0.2 | Mistral AI | apache-2.0 | 7.2B | 1.5M | 3,280 | Dec 11 2023 |
| GLM-5 | Z.ai | mit | 753.9B | 1.4M | 2,141 | Feb 11 2026 |
| Llama-3.2-1B-Instruct | Meta | llama3.2 | 1.2B | 1.3M | 1,966 | Sep 18 2024 |
| Qwen2.5-Coder-7B-Instruct | Qwen | apache-2.0 | 7.6B | 1.3M | 867 | Sep 17 2024 |
| GLM-4.5-Air | Z.ai | mit | 110.5B | 1.3M | 649 | Jul 20 2025 |
| Qwen2.5-VL-7B-Instruct-NVFP4 | NVIDIA | nvidia-open-model-license | 5B | 1.2M | 28 | Sep 10 2025 |
| MiniMax-M2.5 | MiniMax | modified-mit | 228.7B | 1.2M | 1,530 | Feb 12 2026 |
| Qwen2.5-0.5B | Qwen | apache-2.0 | 494M | 1.2M | 492 | Sep 15 2024 |
| Hy3 | Tencent | apache-2.0 | 298.8B | 1.2M | 991 | Jul 02 2026 |
| DeepSeek-V4-Flash | DeepSeek | mit | 290.9B | 1.2M | 2,382 | Apr 22 2026 |
| Gemma-4-31B-IT-NVFP4 | NVIDIA | apache-license-2.0 | 20.9B | 1.1M | 587 | Apr 02 2026 |
| Qwen3-30B-A3B | Qwen | apache-2.0 | 30.5B | 1.1M | 982 | Apr 27 2025 |
| Qwen3-VL-2B-Instruct | Qwen | apache-2.0 | 2.1B | 1.1M | 516 | Oct 19 2025 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | NVIDIA | openmdw-1.1 | 31.6B | 1.1M | 271 | Aug 01 2026 |
| NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 | NVIDIA | openmdw-1.1 | 560.5B | 1.1M | 384 | Jun 03 2026 |
| GLM-4.7-Flash | Z.ai | mit | 31.2B | 1M | 1,897 | Jan 19 2026 |
| NVIDIA-Nemotron-3-Super-120B-A12B-BF16 | NVIDIA | nvidia-nemotron-open-model-license | 123.6B | 1M | 439 | Mar 10 2026 |
| Qwen2-VL-2B-Instruct | Qwen | apache-2.0 | 2.2B | 1M | 529 | Aug 28 2024 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 | NVIDIA | openmdw-1.1 | 32.9B | 1M | 493 | Aug 04 2026 |
| gemma-3-4b-it | Google | gemma | 4.3B | 1M | 1,640 | Feb 20 2025 |
| Qwen3.5-122B-A10B | Qwen | apache-2.0 | 125.1B | 904.2K | 653 | Feb 24 2026 |
| Qwen3-30B-A3B-Instruct-2507 | Qwen | apache-2.0 | 30.5B | 895.7K | 856 | Jul 28 2025 |
| GLM-5.1 | Z.ai | mit | 753.9B | 878.9K | 1,850 | Apr 03 2026 |
| Llama-3.2-3B-Instruct | Meta | llama3.2 | 3.2B | 872.2K | 2,972 | Sep 18 2024 |
| Laguna-XS.2 | Poolside | apache-2.0 | 33.4B | 860.1K | 320 | Apr 23 2026 |
| SmolLM2-135M-Instruct | Hugging Face | apache-2.0 | 134.5M | 847.5K | 471 | Oct 31 2024 |
| Qwen3.5-397B-A17B | Qwen | apache-2.0 | 403.4B | 836.7K | 1,579 | Feb 16 2026 |
| Laguna-M.1 | Poolside | apache-2.0 | 225.8B | 824.2K | 150 | Jun 15 2026 |
| Qwen3.5-0.8B-Base | Qwen | apache-2.0 | 873.4M | 797.9K | 139 | Feb 28 2026 |
| Qwen2.5-32B-Instruct | Qwen | apache-2.0 | 32.8B | 775.4K | 377 | Sep 17 2024 |
| Kimi-K3 | Moonshot AI | kimi-k3 | 2.8T | 742.4K | 11,960 | Jun 13 2026 |
| NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 | NVIDIA | nvidia-nemotron-open-model-license | 18.2B | 733.2K | 190 | Dec 20 2025 |
| Laguna-XS-2.1 | Poolside | openmdw-1.1 | 33.4B | 699.7K | 267 | Jun 20 2026 |
| Qwen2.5-14B-Instruct | Qwen | apache-2.0 | 14.8B | 679K | 383 | Sep 16 2024 |
| Qwen2.5-VL-32B-Instruct | Qwen | apache-2.0 | 33.5B | 646.1K | 510 | Mar 21 2025 |
| diffusiongemma-26B-A4B-it | Google | apache-2.0 | 25.8B | 637K | 1,324 | Jun 09 2026 |
| Qwen3-1.7B-Base | Qwen | apache-2.0 | 1.7B | 628.7K | 98 | Apr 28 2025 |
| HunyuanOCR | Tencent | tencent-hunyuan-community | 1.1B | 615.6K | 846 | Nov 18 2025 |
| Qwen3-0.6B-Base | Qwen | apache-2.0 | 596M | 612.4K | 223 | Apr 28 2025 |
| DeepSeek-R1-Distill-Qwen-1.5B | DeepSeek | mit | 1.8B | 600K | 1,663 | Jan 20 2025 |
| Qwen3.5-4B-Base | Qwen | apache-2.0 | 4.7B | 593K | 126 | Feb 27 2026 |
| Laguna-S-2.1 | Poolside | openmdw-1.1 | 117.6B | 575K | 1,077 | Jul 13 2026 |
| Cosmos-Reason2-2B | NVIDIA | nvidia-open-model-license | 2.4B | 570.7K | 383 | Dec 12 2025 |
| Qwen3-8B-Base | Qwen | apache-2.0 | 8.2B | 550.3K | 152 | Apr 28 2025 |
| Qwen2.5-Coder-32B-Instruct | Qwen | apache-2.0 | 32.8B | 547.5K | 2,209 | Nov 06 2024 |
| Qwen2-VL-7B-Instruct | Qwen | apache-2.0 | 8.3B | 546.3K | 1,297 | Aug 28 2024 |
| Qwen2.5-1.5B | Qwen | apache-2.0 | 1.5B | 541.1K | 249 | Sep 15 2024 |
| granite-4.1-3b | IBM | apache-2.0 | 3.4B | 524.5K | 118 | Apr 06 2026 |
| Llama-3.2-1B | Meta | llama3.2 | 1.2B | 477.2K | 2,757 | Sep 18 2024 |
| Llama-3.1-8B | Meta | llama3.1 | 8B | 475.2K | 2,775 | Jul 14 2024 |
| NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 | NVIDIA | nvidia-nemotron-open-model-license | 67.2B | 445.1K | 454 | Mar 10 2026 |
| Qwen3-VL-30B-A3B-Instruct | Qwen | apache-2.0 | 31.1B | 441.4K | 627 | Sep 30 2025 |
| medgemma-4b-it | Google | health-ai-developer-foundations | 4.3B | 436.6K | 1,142 | May 19 2025 |
| Unlimited-OCR-2 | Baidu | mit | 3.3B | 428.1K | 668 | Oct 26 2026 |
| Mistral-7B-v0.1 | Mistral AI | apache-2.0 | 7.2B | 405.2K | 4,230 | Sep 20 2023 |
| Florence-2-large | Microsoft | mit | 776.7M | 398.5K | 1,888 | Jun 15 2024 |
| Kimi-K2.5 | Moonshot AI | modified-mit | 1T | 392.5K | 2,901 | Jan 01 2026 |
| SmolLM3-3B | Hugging Face | apache-2.0 | 3.1B | 386.4K | 1,077 | Jul 08 2025 |
| Meta-Llama-3-8B-Instruct | Meta | llama3 | 8B | 385.5K | 5,813 | Apr 17 2024 |
| DeepSeek-R1-0528-Qwen3-8B | DeepSeek | mit | 8.2B | 370.5K | 1,127 | May 29 2025 |
| Phi-3.5-mini-instruct | Microsoft | mit | 3.8B | 367.3K | 1,296 | Aug 16 2024 |
| gemma-3-12b-it | Google | gemma | 12.2B | 362.8K | 908 | Mar 01 2025 |
| gemma-3-27b-it | Google | gemma | 27.4B | 361.1K | 2,066 | Mar 01 2025 |
| DeepSeek-R1-Distill-Qwen-32B | DeepSeek | mit | 32.8B | 357.5K | 1,673 | Jan 20 2025 |
| Llama-3.3-70B-Instruct | Meta | llama3.3 | 70.6B | 350.9K | 3,198 | Nov 26 2024 |
| DeepSeek-OCR-2 | DeepSeek | apache-2.0 | 3.4B | 338.9K | 1,151 | Jan 27 2026 |
| DeepSeek-V4-Pro | DeepSeek | mit | 1.6T | 335.6K | 5,700 | Apr 22 2026 |
| Phi-4-mini-instruct | Microsoft | mit | 3.8B | 331.9K | 886 | Feb 19 2025 |
| Qwen3.5-9B-Base | Qwen | apache-2.0 | 9.7B | 331.7K | 148 | Feb 26 2026 |
| Qwen3-VL-235B-A22B-Instruct | Qwen | apache-2.0 | 235.7B | 327K | 431 | Sep 22 2025 |
| Qwen3.5-2B-Base | Qwen | apache-2.0 | 2.3B | 325.7K | 108 | Feb 28 2026 |
| DeepSeek-R1-Distill-Qwen-7B | DeepSeek | mit | 7.6B | 321.6K | 956 | Jan 20 2025 |
| Phi-3-mini-4k-instruct | Microsoft | mit | 3.8B | 321.4K | 1,489 | Apr 22 2024 |
| GLM-5.2 | Z.ai | mit | 753.3B | 320.4K | 5,205 | Jun 16 2026 |
| DeepSeek-V4.1-Flash-1017 | DeepSeek | mit | 763.2B | 312.3K | 5,429 | Oct 17 2026 |
| Qwen3.6-27B-NVFP4 | NVIDIA | apache-2.0 | 18.2B | 311.8K | 447 | Jun 22 2026 |
| Phi-3.5-vision-instruct | Microsoft | mit | 4.1B | 308.2K | 750 | Aug 16 2024 |
| MiniMax-M2.7 | MiniMax | other | 228.7B | 302.1K | 1,261 | Apr 09 2026 |
| gemma-3-4b-pt | Google | gemma | 4.3B | 290.9K | 179 | Feb 20 2025 |
| Kimi-K2-Instruct | Moonshot AI | modified-mit | 1T | 277.3K | 3,010 | Jul 11 2025 |
| paligemma-3b-pt-224 | Google | gemma | 2.9B | 275.8K | 710 | May 12 2024 |
| phi-4 | Microsoft | mit | 14.7B | 275.6K | 2,340 | Dec 11 2024 |
| LFM2.5-230M-Base | Liquid AI | lfm1.0 | 229.7M | 275.2K | 25 | Jun 16 2026 |
| Llama-3.2-3B | Meta | llama3.2 | 3.2B | 269.1K | 1,084 | Sep 18 2024 |
| SmolLM2-1.7B-Instruct | Hugging Face | apache-2.0 | 1.7B | 266.8K | 771 | Oct 31 2024 |
| Qwen3.8-27B-NVFP4 | NVIDIA | apache-2.0 | 18.2B | 260.2K | 236 | Sep 04 2026 |
| DeepSeek-V3.1 | DeepSeek | mit | 684.5B | 258.2K | 864 | Aug 21 2025 |
| shieldgemma-9b | Google | gemma | 9.2B | 255.9K | 31 | Jul 16 2024 |
| Kimi-K2-Thinking-NVFP4 | NVIDIA | nvidia-open-model-license | 519.1B | 245.5K | 41 | Dec 02 2025 |
| Phi-3-mini-128k-instruct | Microsoft | mit | 3.8B | 239K | 1,713 | Apr 22 2024 |
| GLM-5.3-NVFP4 | NVIDIA | nvidia-open-model-license | 390.9B | 237.5K | 48 | Sep 14 2026 |
| SmolVLM-500M-Instruct | Hugging Face | apache-2.0 | 507.5M | 233.2K | 200 | Jan 20 2025 |
| Kimi-K2.6 | Moonshot AI | modified-mit | 1T | 222.7K | 1,641 | Apr 14 2026 |
| Muse-Glimmer-30B-assistant | Meta | apache-2.0 | 2.6B | 218.8K | 80 | Aug 09 2026 |
| SmolLM2-360M-Instruct | Hugging Face | apache-2.0 | 361.8M | 217.8K | 243 | Oct 31 2024 |
| Llama-2-7b-hf | Meta | llama2 | 6.7B | 210.8K | 2,508 | Jul 13 2023 |
| medgemma-1.5-4b-it | Google | health-ai-developer-foundations | 4.3B | 206.7K | 1,119 | Jan 07 2026 |
| gemma-3-1b-it | Google | gemma | 999.9M | 206K | 1,309 | Mar 10 2025 |
| gemma-7b | Google | gemma | 8.5B | 200.5K | 3,470 | Feb 08 2024 |
| Olmo-3-7B-Instruct | Ai2 | apache-2.0 | 7.3B | 196.3K | 159 | Nov 19 2025 |
| gpt-oss-120b-Eagle3-short-context | NVIDIA | nvidia-open-model-license | 786.9M | 195K | 23 | Oct 06 2025 |
| Step-3.5-Flash | StepFun | apache-2.0 | 199.4B | 194.1K | 847 | Feb 01 2026 |
| Kimi-K2.5-NVFP4 | NVIDIA | nvidia-open-model-license | 518B | 192K | 91 | Jan 30 2026 |
| DeepSeek-R1-Distill-Qwen-14B | DeepSeek | mit | 14.8B | 189.6K | 739 | Jan 20 2025 |
| Qwen3.8-Flash-Next-NVFP4 | NVIDIA | nvidia-open-model-license | 119.6B | 184K | 448 | Sep 02 2026 |
| Muse-Glimmer-30B | Meta | apache-2.0 | 29.8B | 179.1K | 2,129 | Aug 09 2026 |
| phi-2 | Microsoft | mit | 2.8B | 177.5K | 3,534 | Dec 13 2023 |
| Florence-2-base-ft | Microsoft | mit | 231.6M | 175.2K | 154 | Jun 15 2024 |
| t5gemma-2b-2b-ul2-it | Google | gemma | 5.6B | 172.9K | 10 | Jun 19 2025 |
| deepseek-coder-7b-instruct-v1.5 | DeepSeek | deepseek | 6.9B | 172.1K | 193 | Jan 25 2024 |
| SmolVLM-256M-Instruct | Hugging Face | apache-2.0 | 256.5M | 166.9K | 425 | Jan 17 2025 |
| LFM2.5-350M | Liquid AI | lfm1.0 | 354.5M | 161.1K | 443 | Mar 31 2026 |
| LocateAnything-3B | NVIDIA | nvidia-license | 3.8B | 156.8K | 3,181 | Mar 02 2026 |
| Kimi-K2.6-NVFP4 | NVIDIA | nvidia-open-model-license | n/a | 156.8K | 47 | May 11 2026 |
| DeepSeek-V3-0324 | DeepSeek | mit | 684.5B | 149.9K | 3,224 | Mar 24 2025 |
| granite-docling-258M | IBM | apache-2.0 | 257.5M | 148.9K | 1,295 | May 19 2025 |
| DeepSeek-V4-Flash-Vision-Exp | DeepSeek | mit | 304.6B | 148.9K | 966 | Aug 31 2026 |
| MiniMax-M3 | MiniMax | minimax-community | 427B | 148K | 1,600 | Jun 02 2026 |
| Llama-2-13b-chat-hf | Meta | llama2 | 13B | 147.1K | 1,138 | Jul 13 2023 |
| UI-TARS-1.5-7B | ByteDance | apache-2.0 | 8.3B | 142.7K | 621 | Apr 16 2025 |
| DeepSeek-V4-Pro-0813 | DeepSeek | mit | 1.7T | 141.8K | 914 | Aug 13 2026 |
| GOT-OCR2_0 | StepFun | apache-2.0 | 716M | 140.7K | 1,577 | Sep 12 2024 |
| DialoGPT-medium | Microsoft | mit | n/a | 136K | 451 | Mar 02 2022 |
| Meta-Llama-3-70B | Meta | llama3 | 70.6B | 135.7K | 888 | Apr 17 2024 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | NVIDIA | openmdw-1.1 | n/a | 132.2K | 394 | Oct 05 2026 |
| SmolLM2-360M | Hugging Face | apache-2.0 | 361.8M | 128K | 145 | Oct 31 2024 |
| DeepSeek-V4-Flash-DSpark | DeepSeek | mit | 304.2B | 126.5K | 309 | Jun 27 2026 |
| Olmo-3-1025-7B | Ai2 | apache-2.0 | 7.3B | 125.8K | 93 | Sep 12 2025 |
| Llama-2-7b-chat-hf | Meta | llama2 | 6.7B | 125.7K | 4,952 | Jul 13 2023 |
| falcon-7b | TII | apache-2.0 | 7.2B | 124.8K | 1,118 | Apr 24 2023 |
| OLMo-2-0425-1B | Ai2 | apache-2.0 | 1.5B | 124.3K | 83 | Apr 17 2025 |
| SmolLM-135M | Hugging Face | apache-2.0 | 134.5M | 121.7K | 277 | Jul 14 2024 |
| gemma-2-2b-it | Google | gemma | 2.6B | 121.1K | 1,612 | Jul 16 2024 |
| DeepSeek-R1-0528 | DeepSeek | mit | 684.5B | 120.7K | 2,497 | May 28 2025 |
| NVIDIA-Nemotron-Parse-v1.1 | NVIDIA | nvidia-open-model-license | 956.8M | 120.3K | 174 | Nov 15 2025 |
| DeepSeek-V2-Lite | DeepSeek | deepseek | 15.7B | 117.7K | 210 | May 15 2024 |
| Meta-Llama-3-8B | Meta | llama3 | 8B | 117.5K | 6,788 | Apr 17 2024 |
| Qwen3-8B-NVFP4 | NVIDIA | apache-2.0 | 4.7B | 117.4K | 29 | Sep 09 2025 |
| wildguard | Ai2 | apache-2.0 | 7.2B | 116K | 74 | Jun 15 2024 |
| LFM2.5-1.2B-Instruct | Liquid AI | lfm1.0 | 1.2B | 113.8K | 685 | Jan 06 2026 |
| Mistral-7B-Instruct-v0.1 | Mistral AI | apache-2.0 | 7.2B | 111.8K | 1,878 | Sep 27 2023 |
| granite-3.1-3b-a800m-instruct | IBM | apache-2.0 | 3.3B | 111.2K | 32 | Dec 06 2024 |
| GLM-4.5 | Z.ai | mit | 358.3B | 110.6K | 1,439 | Jul 20 2025 |
| GLM-5.3-Flash-NVFP4 | NVIDIA | mit | 321.3B | 110.1K | 182 | Sep 02 2026 |
| Llama-4-Scout-17B-16E-Instruct | Meta | llama4 | 108.6B | 109.8K | 1,387 | Apr 02 2025 |
| deepseek-vl2-tiny | DeepSeek | deepseek | 3.4B | 109.3K | 273 | Dec 13 2024 |
| gemma-2-9b-it | Google | gemma | 9.2B | 108K | 1,168 | Jun 24 2024 |
| MiniMax-M2 | MiniMax | modified-mit | 228.7B | 107.6K | 1,513 | Oct 22 2025 |
| GLM-5.3-Flash-BF16 | Z.ai | mit | 321.3B | 106.9K | 81 | Aug 25 2026 |
| granite-vision-4.1-4b | IBM | apache-2.0 | 4B | 105.3K | 114 | Apr 16 2026 |
| NVIDIA-Nemotron-Nano-9B-v2 | NVIDIA | nvidia-open-model-license | 8.9B | 104.7K | 532 | Aug 12 2025 |
| MiMo-V2.5 | Xiaomi | mit | 310.8B | 99.9K | 434 | Apr 27 2026 |
| granite-4.1-8b | IBM | apache-2.0 | 8.8B | 95.4K | 257 | Apr 06 2026 |
| granite-3.3-8b-instruct | IBM | apache-2.0 | 8.2B | 95.2K | 161 | Apr 09 2025 |
| Llama-3.1-70B-Instruct | Meta | llama3.1 | 70.6B | 91.6K | 1,029 | Jul 16 2024 |
| DeepSeek-Coder-V2-Lite-Instruct | DeepSeek | deepseek-license | 15.7B | 91.1K | 712 | Jun 14 2024 |
| gemma-3-270m | Google | gemma | 268.1M | 90.5K | 1,144 | Aug 05 2025 |
| Kimi-K2.7-Code | Moonshot AI | modified-mit | 1T | 90.3K | 1,430 | Jun 11 2026 |
| Llama-Guard-4-12B | Meta | llama4 | 12B | 90.1K | 138 | Apr 23 2025 |
| gemma-2b | Google | gemma | 2.5B | 87.6K | 1,267 | Feb 08 2024 |
| GOT-OCR-2.0-hf | StepFun | apache-2.0 | 560.5M | 84.3K | 244 | Nov 22 2024 |
| c4ai-command-r-v01 | Cohere | cc-by-nc-4.0 | 35B | 83.3K | 1,128 | Mar 11 2024 |
| SmolLM3-3B-Base | Hugging Face | apache-2.0 | 3.1B | 82.8K | 179 | Jun 19 2025 |
| NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD | NVIDIA | nvidia-open-model-license | 7.7B | 81.6K | 41 | Oct 22 2025 |
| olmOCR-2-7B-1025 | Ai2 | apache-2.0 | 8.3B | 79.6K | 161 | Oct 06 2025 |
| LFM3-1.7B | Liquid AI | lfm1.0 | 1.7B | 78.8K | 1,569 | Oct 23 2026 |
| MiMo-V2.6-Flash-RL | Xiaomi | mit | 310.8B | 76.6K | 590 | Sep 21 2026 |
| gemma-4-26B-A4B | Google | apache-2.0 | 25.8B | 76.1K | 466 | Mar 12 2026 |
| gemma-2-2b | Google | gemma | 2.6B | 75.3K | 763 | Jul 16 2024 |
| DeepSeek-R1-Distill-Llama-8B | DeepSeek | mit | 8B | 74.5K | 919 | Jan 20 2025 |
| paligemma-3b-mix-224 | Google | gemma | 2.9B | 74.4K | 110 | May 12 2024 |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 | NVIDIA | openmdw-1.1 | 124.5B | 72.2K | 119 | Oct 12 2026 |
| deepseek-coder-6.7b-base | DeepSeek | deepseek-license | 6.7B | 69.3K | 149 | Oct 23 2023 |
| Hy3-preview | Tencent | other | 298.8B | 68.8K | 314 | Apr 13 2026 |
| Hunyuan-A13B-Instruct | Tencent | tencent-hunyuan-a13b | 80.4B | 68.7K | 795 | Jun 25 2025 |
| Cosmos-Reason2-8B | NVIDIA | nvidia-open-model-license | 8.8B | 67.9K | 256 | Dec 12 2025 |
| Llama-3.1-Nemotron-Nano-VL-8B-V1 | NVIDIA | nvidia-open-model-license | 8.7B | 65.1K | 188 | Jun 03 2025 |
| Kimi-VL-A3B-Instruct | Moonshot AI | mit | 16.4B | 64.4K | 285 | Apr 09 2025 |
| SmolVLM2-2.2B-Instruct | Hugging Face | apache-2.0 | 2.2B | 63.9K | 339 | Feb 08 2025 |
| Nemotron-H-8B-Base-8K | NVIDIA | nvidia-internal-scientific-research-and-development-model-license | 8.1B | 61.6K | 63 | Mar 19 2025 |
| granite-4.2-8b | IBM | apache-2.0 | 8.8B | 59.8K | 117 | Aug 07 2026 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark | NVIDIA | openmdw-1.1 | 763.7M | 59.3K | 43 | Aug 05 2026 |
| ERNIE-4.5-21B-A3B-Thinking | Baidu | apache-2.0 | 21.8B | 57.9K | 796 | Sep 08 2025 |
| Olmo-3-7B-Think | Ai2 | apache-2.0 | 7.3B | 57.8K | 110 | Nov 18 2025 |
| gemma-3-270m-it | Google | gemma | 268.1M | 57.1K | 712 | Jul 30 2025 |
| DialoGPT-small | Microsoft | mit | 175.6M | 55.5K | 155 | Mar 02 2022 |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 | NVIDIA | openmdw-1.1 | 65.8B | 54.6K | 266 | Oct 12 2026 |
| DeepSeek-V4-Flash-NVFP4 | NVIDIA | mit | 290.9B | 54K | 119 | May 18 2026 |
| gemma-3-12b-pt | Google | gemma | 12.2B | 53.8K | 97 | Mar 01 2025 |
| MiniMax-M3.1-Flash | MiniMax | minimax-community | 220.5B | 53.1K | 567 | Oct 17 2026 |
| Kimi-K2-Thinking | Moonshot AI | modified-mit | 1T | 52.9K | 1,731 | Nov 04 2025 |
| Nemotron-Labs-Diffusion-8B-Base | NVIDIA | nvidia-nemotron-open-model-license | 8.5B | 51.9K | 12 | Jan 14 2026 |
| Cosmos-Reason2-32B | NVIDIA | nvidia-open-model-license | 33.4B | 51.8K | 18 | Apr 29 2026 |
| Llama-3.2-11B-Vision-Instruct | Meta | llama3.2 | 10.7B | 51.3K | 1,694 | Sep 18 2024 |
| granite-3.2-8b-instruct | IBM | apache-2.0 | 8.2B | 51K | 92 | Feb 17 2025 |
| gemma-4-31B | Google | apache-2.0 | 31.3B | 50.7K | 590 | Mar 12 2026 |
| LFM2.5-2.6B | Liquid AI | lfm1.0 | 2.7B | 49.9K | 860 | Jul 28 2026 |
| OLMo-2-0425-1B-Instruct | Ai2 | apache-2.0 | 1.5B | 49.7K | 58 | Apr 29 2025 |
| GLM-4.7 | Z.ai | mit | 358.3B | 49.6K | 2,077 | Dec 22 2025 |
| olmOCR-7B-0225-preview | Ai2 | apache-2.0 | 8.3B | 49.1K | 705 | Jan 15 2025 |
| phi-1_5 | Microsoft | mit | 1.4B | 48.5K | 1,372 | Sep 10 2023 |
| GLM-4.1V-9B-Thinking | Z.ai | mit | 10.3B | 48.2K | 792 | Jun 28 2025 |
| Kimi-Linear-48B-A3B-Instruct | Moonshot AI | mit | 49.1B | 48.1K | 616 | Oct 30 2025 |
| MiniMax-M3-MXFP8 | MiniMax | minimax-community | 427B | 46.6K | 63 | Jun 02 2026 |
| OLMoE-1B-7B-0924 | Ai2 | apache-2.0 | 6.9B | 46.5K | 157 | Jul 20 2024 |
| Llama-3.1-405B | Meta | llama3.1 | 405.9B | 45.8K | 995 | Jul 16 2024 |
| GLM-5.2-NVFP4 | NVIDIA | mit | 381B | 44.6K | 329 | Jun 22 2026 |
| t5gemma-2-1b-1b | Google | gemma | 2.1B | 44.2K | 88 | Oct 25 2025 |
| Molmo2-8B | Ai2 | apache-2.0 | 8.7B | 43.7K | 199 | Dec 14 2025 |
| DeepSeek-V3.2-Exp | DeepSeek | mit | 685.4B | 43.6K | 1,027 | Sep 29 2025 |
| Laguna-S-2.1-NVFP4 | Poolside | openmdw-1.1 | 117.6B | 42.3K | 204 | Jul 02 2026 |
| Olmo-3-7B-Instruct-SFT | Ai2 | apache-2.0 | 7.3B | 42K | 6 | Nov 17 2025 |
| GLM-5.3-BF16 | Z.ai | glm-5.3 | 753.3B | 41.8K | 61 | Aug 25 2026 |
| LFM2.5-VL-450M | Liquid AI | lfm1.0 | 448.7M | 41.6K | 238 | Apr 08 2026 |
| OLMo-2-1124-7B-Instruct | Ai2 | apache-2.0 | 7.3B | 41.4K | 51 | Dec 18 2024 |
| Seed-OSS-36B-Instruct | ByteDance | apache-2.0 | 36.2B | 41K | 506 | Aug 20 2025 |
| GLM-4.6V-Flash | Z.ai | mit | 10.3B | 40.9K | 633 | Dec 07 2025 |
| t5gemma-2-270m-270m | Google | gemma | 786M | 40.5K | 214 | Oct 25 2025 |
| granite-3b-code-base-2k | IBM | apache-2.0 | 3.5B | 40.5K | 38 | Apr 23 2024 |
| DeepSeek-V2-Lite-Chat | DeepSeek | deepseek | 15.7B | 40.5K | 173 | May 15 2024 |
| maira-2 | Microsoft | msrla | 6.9B | 40.5K | 92 | Jul 29 2024 |
| granite-3.3-2b-instruct | IBM | apache-2.0 | 2.5B | 39.4K | 87 | Apr 09 2025 |
| gemma-3-1b-pt | Google | gemma | 999.9M | 39K | 222 | Feb 20 2025 |
| LFM2.5-24B-A2B | Liquid AI | lfm1.0 | 24B | 38.9K | 606 | Oct 12 2026 |
| Qwen3.5-122B-A10B-NVFP4 | NVIDIA | apache-2.0 | 64.6B | 38.7K | 65 | May 13 2026 |
| SmolLM2-1.7B | Hugging Face | apache-2.0 | 1.7B | 38.2K | 168 | Oct 30 2024 |
| step3 | StepFun | apache-2.0 | 321B | 38.1K | 166 | Jul 28 2025 |
| Qwen3.5-397B-A17B-NVFP4-V2 | NVIDIA | apache-2.0 | n/a | 37.5K | 18 | Jun 29 2026 |
| granite-4.2-3b | IBM | apache-2.0 | 3.7B | 37.4K | 132 | Aug 07 2026 |
| Meta-Llama-3-70B-Instruct | Meta | llama3 | 70.6B | 37.3K | 1,537 | Apr 17 2024 |
| SmolVLM2-256M-Video-Instruct | Hugging Face | apache-2.0 | 256.5M | 37.2K | 119 | Feb 11 2025 |
| paligemma2-3b-mix-224 | Google | gemma | 3B | 37.1K | 70 | Nov 21 2024 |
| Phi-4-reasoning | Microsoft | mit | 14.7B | 36.9K | 234 | Apr 09 2025 |
| DeepSeek-V4-Pro-NVFP4 | NVIDIA | mit | 1.6T | 36.9K | 80 | May 14 2026 |
| granite-4.2-30b | IBM | apache-2.0 | 29.3B | 36.8K | 140 | Aug 07 2026 |
| Moonlight-16B-A3B-Instruct | Moonshot AI | mit | 16B | 36.8K | 210 | Feb 22 2025 |
| GLM-Z1-32B-0414 | Z.ai | mit | 32.6B | 36.5K | 196 | Apr 08 2025 |
| Llama-3_3-Nemotron-Super-49B-v1_5 | NVIDIA | nvidia-open-model-license | 49.9B | 36.5K | 242 | Jul 25 2025 |
| Phi-4-mini-reasoning | Microsoft | mit | 3.8B | 36.5K | 252 | Apr 29 2025 |
| Qwen3-14B-NVFP4 | NVIDIA | apache-2.0 | 8.2B | 36.2K | 20 | Sep 09 2025 |
| Llama-3_3-Nemotron-Super-49B-v1 | NVIDIA | nvidia-open-model-license | 49.9B | 35.8K | 332 | Mar 16 2025 |
| granite-4.0-1b | IBM | apache-2.0 | 1.6B | 35.1K | 57 | Oct 07 2025 |
| Phi-3-vision-128k-instruct | Microsoft | mit | 4.1B | 34.5K | 979 | May 19 2024 |
| Molmo2-O-7B | Ai2 | apache-2.0 | 7.8B | 33.7K | 26 | Dec 14 2025 |
| Nemotron-Labs-Diffusion-3B | NVIDIA | nvidia-nemotron-open-model-license | 3.8B | 33.7K | 51 | Mar 02 2026 |
| DeepSeek-V4.1-Flash-NVFP4 | NVIDIA | mit | 763.2B | 33.3K | 128 | Sep 16 2026 |
| deepseek-coder-1.3b-instruct | DeepSeek | deepseek | 1.3B | 33.3K | 211 | Oct 29 2023 |
| falcon-7b-instruct | TII | apache-2.0 | 7.2B | 32.3K | 1,042 | Apr 25 2023 |
| OLMoE-1B-7B-0125-Instruct | Ai2 | apache-2.0 | 6.9B | 31.7K | 68 | Jan 27 2025 |
| MiMo-7B-RL | Xiaomi | mit | 7.8B | 30.9K | 281 | Apr 29 2025 |
| Qianfan-OCR | Baidu | apache-2.0 | 4.7B | 30K | 1,217 | Mar 18 2026 |
| gemma-3n-E2B-it | Google | gemma | 5.4B | 29.8K | 334 | Jun 12 2025 |
| d1-3B | Liquid AI | lfm1.0 | 3.1B | 29.3K | 500 | Oct 05 2026 |
| MiniMax-M1-40k | MiniMax | apache-2.0 | 456.1B | 29.3K | 185 | Jun 05 2025 |
| paligemma-3b-pt-224-jax | Google | gemma | n/a | 29.1K | 5 | May 05 2024 |
| OLMo-1B-hf | Ai2 | apache-2.0 | 1.2B | 28.4K | 29 | Apr 12 2024 |
| medgemma-27b-it | Google | health-ai-developer-foundations | 27.4B | 28.4K | 487 | Jul 09 2025 |
| Kimi-K3.1-Pro | Moonshot AI | kimi-k3.1-pro | 2.8T | 27.9K | 1,787 | Nov 02 2026 |
| DeepSeek-R1-Distill-Llama-70B | DeepSeek | mit | 70.6B | 27.9K | 824 | Jan 20 2025 |
| deepseek-llm-7b-base | DeepSeek | deepseek | n/a | 27.8K | 172 | Nov 29 2023 |
| LFM2.5-8B-A1B | Liquid AI | lfm1.0 | 8.5B | 27.5K | 807 | May 28 2026 |
| LFM2.5-230M | Liquid AI | lfm1.0 | 229.7M | 27.4K | 322 | Jun 24 2026 |
| GLM-4.6 | Z.ai | mit | 356.8B | 27.4K | 1,247 | Sep 29 2025 |
| LFM2.5-VL-3B | Liquid AI | lfm1.0 | 3.1B | 27.2K | 236 | Aug 11 2026 |
| granite-4.0-1b-base | IBM | apache-2.0 | 1.6B | 27.1K | 34 | Oct 07 2025 |
| granite-3.1-8b-instruct | IBM | apache-2.0 | 8.2B | 27.1K | 169 | Dec 06 2024 |
| Phi-tiny-MoE-instruct | Microsoft | mit | 3.8B | 26.4K | 52 | Jun 23 2025 |
| Florence-2-large-ft | Microsoft | mit | 770.4M | 26.1K | 399 | Jun 15 2024 |
| Molmo2-4B | Ai2 | apache-2.0 | 4.9B | 26K | 55 | Dec 14 2025 |
| gemma-7b-it | Google | gemma | 8.5B | 25.5K | 1,263 | Feb 13 2024 |
| LLaDA2.1-mini | inclusionAI | apache-2.0 | 16.3B | 25.3K | 128 | Feb 09 2026 |
| t5gemma-2-4b-4b | Google | gemma | 7.5B | 24.8K | 162 | Oct 25 2025 |
| granite-3.1-2b-instruct | IBM | apache-2.0 | 2.5B | 24.7K | 57 | Dec 06 2024 |
| DeepSeek-V2-Chat | DeepSeek | deepseek | 235.7B | 24.6K | 486 | Apr 28 2024 |
| MiMo-V2.6-Pro-RL | Xiaomi | mit | 1T | 24.3K | 813 | Sep 21 2026 |
| granite-4.0-micro | IBM | apache-2.0 | 3.4B | 24.1K | 274 | Sep 16 2025 |
| SmolLM-1.7B | Hugging Face | apache-2.0 | 1.7B | 24K | 187 | Jul 14 2024 |
| Llama-3.1-70B | Meta | llama3.1 | 70.6B | 24K | 453 | Jul 14 2024 |
| deepseek-coder-6.7b-instruct | DeepSeek | deepseek | 6.7B | 24K | 537 | Oct 29 2023 |
| granite-4.0-h-small | IBM | apache-2.0 | 32.2B | 23.9K | 313 | Sep 16 2025 |
| Phi-3.5-MoE-instruct | Microsoft | mit | 41.9B | 23.3K | 590 | Aug 17 2024 |
| granite-4.0-h-tiny | IBM | apache-2.0 | 6.9B | 23.2K | 217 | Sep 16 2025 |
| SmolLM-135M-Instruct | Hugging Face | apache-2.0 | 134.5M | 23.1K | 147 | Jul 15 2024 |
| gemma-2-27b-it | Google | gemma | 27.2B | 22.8K | 577 | Jun 24 2024 |
| aya-expanse-8b | Cohere | cc-by-nc-4.0 | 8B | 22.7K | 463 | Oct 23 2024 |
| gpt-oss-safeguard-20b | OpenAI | apache-2.0 | 21.5B | 22.5K | 291 | Sep 18 2025 |
| Llama-Guard-3-8B | Meta | llama3.1 | 8B | 22.3K | 353 | Jul 22 2024 |
| DAM-3B | NVIDIA | nvidia-non-commercial-license | n/a | 21.7K | 134 | Apr 21 2025 |
| tiny-aya-global | Cohere | cc-by-nc-4.0 | 3.3B | 21.3K | 200 | Feb 13 2026 |
| Olmo-3-7B-Instruct-DPO | Ai2 | apache-2.0 | 7.3B | 21.2K | 3 | Nov 19 2025 |
| Olmo-3.1-32B-Instruct | Ai2 | apache-2.0 | 32.2B | 21.1K | 88 | Dec 10 2025 |
| gemma-2-9b | Google | gemma | 9.2B | 21K | 747 | Jun 24 2024 |
| GLM-4-9B-0414 | Z.ai | mit | 9.4B | 20.9K | 111 | Apr 07 2025 |
| Nemotron-3.5-Parse | NVIDIA | openmdw-1.1 | 630.8M | 20.3K | 155 | Sep 03 2026 |
| MiMo-V2-Flash | Xiaomi | mit | 309.8B | 20.2K | 756 | Dec 16 2025 |
| LFM2-VL-450M | Liquid AI | lfm1.0 | 450.8M | 18.7K | 152 | Aug 12 2025 |
| Phi-3-medium-4k-instruct | Microsoft | mit | 14B | 18.5K | 231 | May 07 2024 |
| t5gemma-b-b-ul2 | Google | gemma | 591.5M | 18.1K | 15 | Jun 19 2025 |
| MiMo-V2.6-Distill-Qwen-9B | Xiaomi | mit | 9.4B | 17.8K | 698 | Sep 21 2026 |
| Cosmos-Reason1-7B | NVIDIA | nvidia-open-model-license | 8.3B | 17.8K | 249 | Apr 18 2025 |
| North-Micro-Vision-Instruct | Cohere | apache-2.0 | 2.5B | 17.6K | 154 | Aug 10 2026 |
| medgemma-27b-text-it | Google | health-ai-developer-foundations | 27B | 17.6K | 508 | May 19 2025 |
| granite-4.0-350m | IBM | apache-2.0 | 352.4M | 17.4K | 74 | Oct 07 2025 |
| deepseek-coder-1.3b-base | DeepSeek | deepseek-license | n/a | 17.3K | 136 | Oct 28 2023 |
| Muse-Spark-Open-816B-A42B-NVFP4 | Meta | muse-model-license | 822B | 17K | 251 | Sep 15 2026 |
| deepseek-llm-7b-chat | DeepSeek | deepseek | n/a | 16.7K | 247 | Nov 29 2023 |
| LFM3-VL-2B | Liquid AI | lfm1.0 | 1.9B | 16.7K | 270 | Oct 19 2026 |
| NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB | NVIDIA | openmdw-1.1 | 24.6B | 16.7K | 177 | Oct 07 2026 |
| LFM2-350M | Liquid AI | lfm1.0 | 354.5M | 16.5K | 261 | Jul 10 2025 |
| Moonlight-16B-A3B | Moonshot AI | mit | 16B | 16.4K | 118 | Feb 22 2025 |
| deepseek-moe-16b-chat | DeepSeek | deepseek | 16.4B | 16.4K | 177 | Jan 09 2024 |
| paligemma2-3b-ft-docci-448 | Google | gemma | 3B | 16.3K | 15 | Nov 21 2024 |
| Kimi-K2-Base | Moonshot AI | modified-mit | 1T | 16.2K | 306 | Jul 03 2025 |
| NVIDIA-Nemotron-Parse-2.0 | NVIDIA | openmdw-1.1 | 903M | 16K | 135 | Jun 30 2026 |
| LLaDA2.0-mini | inclusionAI | apache-2.0 | 16.3B | 15.9K | 71 | Nov 25 2025 |
| MiMo-V2.6-Pro-MOPD | Xiaomi | mit | 1T | 15.7K | 48 | Sep 27 2026 |
| command-a-plus-05-2026-bf16 | Cohere | apache-2.0 | 218.8B | 15.6K | 147 | May 11 2026 |
| gpt-oss-safeguard-120b | OpenAI | apache-2.0 | 116.8B | 15.5K | 125 | Sep 18 2025 |
| MiMo-V2.6-Flash-MOPD | Xiaomi | mit | 310.8B | 15.5K | 71 | Sep 27 2026 |
| GLM-4.5V | Z.ai | mit | 107.7B | 15.5K | 726 | Aug 10 2025 |
| ERNIE-4.5-21B-A3B-PT | Baidu | apache-2.0 | 21.9B | 15.4K | 178 | Jun 28 2025 |
| WeVisOne-VL-4B | Tencent | apache-2.0 | 4.8B | 15.4K | 113 | Oct 22 2026 |
| Llama-Guard-3-1B | Meta | llama3.2 | 1.5B | 15.4K | 140 | Sep 20 2024 |
| functiongemma-270m-it | Google | gemma | 268.1M | 15.3K | 1,127 | Oct 08 2025 |
| LFM2-1.2B | Liquid AI | lfm1.0 | 1.2B | 15.1K | 370 | Jul 10 2025 |
| SmolVLM-Instruct | Hugging Face | apache-2.0 | 2.2B | 15.1K | 602 | Nov 18 2024 |
| NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 | NVIDIA | nvidia-open-model-license | 13.2B | 14.7K | 95 | Oct 21 2025 |
| Olmo-3-1125-32B | Ai2 | apache-2.0 | 32.2B | 14.7K | 129 | Nov 04 2025 |
| Ling-3.0-tiny | inclusionAI | mit | 7.9B | 14.2K | 575 | Aug 10 2026 |
| MiMo-7B-Base | Xiaomi | mit | 7.8B | 14.2K | 139 | Apr 29 2025 |
| granite-4.1-8b-base | IBM | apache-2.0 | 8.4B | 14.2K | 28 | Apr 06 2026 |
| LFM2.5-VL-1.6B | Liquid AI | lfm1.0 | 1.6B | 13.8K | 333 | Jan 05 2026 |
| Eagle2-2B | NVIDIA | cc-by-nc-4.0 | 2.2B | 13.7K | 37 | Jan 10 2025 |
| WeVisOne-VL-2B | Tencent | apache-2.0 | 2.4B | 13.7K | 82 | Oct 22 2026 |
| gemma-2b-it | Google | gemma | 2.5B | 13.6K | 976 | Feb 08 2024 |
| OLMo-2-1124-13B-Instruct | Ai2 | apache-2.0 | 13.7B | 13.6K | 48 | Dec 18 2024 |
| Nemotron-3-Content-Safety | NVIDIA | nvidia-nemotron-open-model-license | 4.3B | 13.5K | 22 | Mar 06 2026 |
| biogpt | Microsoft | mit | n/a | 13.5K | 316 | Nov 20 2022 |
| MiniMax-M2.1 | MiniMax | modified-mit | 228.7B | 13.2K | 1,374 | Dec 20 2025 |
| granite-3.0-8b-instruct | IBM | apache-2.0 | 8.2B | 13K | 209 | Oct 02 2024 |
| Muse-Spark-Open-816B-A42B | Meta | muse-model-license | 822B | 13K | 747 | Sep 15 2026 |
| Olmo-3.1-32B-Instruct-DPO | Ai2 | apache-2.0 | 32.2B | 12.8K | 6 | Nov 18 2025 |
| Olmo-Hybrid-7B | Ai2 | apache-2.0 | 7.4B | 12.7K | 66 | Jan 28 2026 |
| LFM2-8B-A1B | Liquid AI | lfm1.0 | 8.3B | 12.6K | 375 | Oct 07 2025 |
| granite-4.1-30b | IBM | apache-2.0 | 28.9B | 12.5K | 148 | Apr 06 2026 |
| gemma-3n-E4B-it | Google | gemma | 7.8B | 12.3K | 938 | Jun 03 2025 |
| d1-omni-600M | Liquid AI | lfm1.0 | 587.2M | 12.2K | 159 | Oct 05 2026 |
| Mistral-Large-4-1T-A52B | Mistral AI | other | n/a | 12.1K | 1,644 | Sep 30 2026 |
| Olmo-3-32B-Think | Ai2 | apache-2.0 | 32.2B | 12K | 177 | Nov 19 2025 |
| Llama-4-Maverick-17B-128E-Instruct | Meta | llama4 | 401.6B | 11.9K | 520 | Apr 01 2025 |
| Hy4-preview | Tencent | apache-2.0 | 780B | 11.5K | 550 | Aug 27 2026 |
| falcon-40b-instruct | TII | apache-2.0 | n/a | 11.4K | 1,186 | May 25 2023 |
| NVIDIA-Nemotron-Parse-v1.2 | NVIDIA | nvidia-open-model-license | 936.3M | 11.4K | 81 | Feb 18 2026 |
| Kimi-VL-A3B-Thinking-2506 | Moonshot AI | mit | 16.4B | 11.2K | 387 | Jun 21 2025 |
| granite-4.1-3b-base | IBM | apache-2.0 | 3.4B | 11.2K | 25 | Apr 06 2026 |
| deepseek-llm-67b-base | DeepSeek | deepseek | n/a | 11K | 153 | Nov 29 2023 |
| deepseek-coder-33b-instruct | DeepSeek | deepseek | 33.3B | 11K | 606 | Nov 01 2023 |
| Eagle2.5-8B | NVIDIA | nsclv1 | 8.1B | 10.9K | 51 | Apr 12 2025 |
| Phi-3-medium-128k-instruct | Microsoft | mit | 14B | 10.9K | 395 | May 07 2024 |
| ERNIE-4.5-VL-28B-A3B-PT | Baidu | apache-2.0 | 29.4B | 10.7K | 106 | Jun 28 2025 |
| Falcon3-7B-Instruct | TII | falcon-llm-license | 7.5B | 10.6K | 80 | Nov 29 2024 |
| command-a-vision-07-2025 | Cohere | cc-by-nc-4.0 | 111.9B | 10.6K | 88 | Jul 28 2025 |
| bitnet-b1.58-2B-4T | Microsoft | mit | 849.8M | 10.6K | 1,516 | Apr 15 2025 |
| granite-3.1-1b-a400m-instruct | IBM | apache-2.0 | 1.3B | 10.5K | 24 | Dec 06 2024 |
| translategemma-4b-it | Google | gemma | 4.3B | 10.3K | 901 | Jan 12 2026 |
| Step-3.7-Flash | StepFun | apache-2.0 | 201.4B | 10.3K | 463 | May 23 2026 |
| OLMo-7B | Ai2 | apache-2.0 | 6.9B | 10.2K | 657 | Jan 09 2024 |
| academic-ds-9B | ByteDance | apache-2.0 | 9.4B | 10.2K | 16 | Apr 09 2025 |
| Llama-2-70b-hf | Meta | llama2 | 69B | 10.1K | 858 | Jul 11 2023 |
| d1-2B | Liquid AI | lfm1.0 | 1.9B | 10.1K | 250 | Oct 24 2026 |
| falcon-40b | TII | apache-2.0 | 41.8B | 10.1K | 2,450 | May 24 2023 |
| Molmo2-ER | Ai2 | apache-2.0 | 4.9B | 10.1K | 17 | May 04 2026 |
| Kimi-VL-A3B-Thinking | Moonshot AI | mit | 16.4B | 9.9K | 457 | Apr 09 2025 |
| Ling-3.0-flash-VL | inclusionAI | mit | 124.8B | 9.8K | 127 | Sep 04 2026 |
| shieldgemma-2-4b-it | Google | gemma | 4.3B | 9.7K | 190 | Mar 04 2025 |
| MiMo-V2.5-Pro | Xiaomi | mit | 1T | 9.7K | 772 | Apr 27 2026 |
| phi-1 | Microsoft | mit | 1.4B | 9.6K | 227 | Sep 10 2023 |
| gemma-3-27b-pt | Google | gemma | 27.4B | 9.6K | 126 | Mar 01 2025 |
| Ling-3.0-flash | inclusionAI | mit | 127.5B | 9.5K | 437 | Aug 02 2026 |
| Kimi-K2-Instruct-0905 | Moonshot AI | modified-mit | 1T | 9.4K | 790 | Sep 03 2025 |
| LFM3-1.7B-Base | Liquid AI | lfm1.0 | 1.7B | 9.3K | 154 | Oct 26 2026 |
| DeepSeek-V3.1-Terminus | DeepSeek | mit | 684.5B | 9.3K | 386 | Sep 22 2025 |
| Molmo-7B-D-0924 | Ai2 | apache-2.0 | 8B | 9.3K | 568 | Sep 25 2024 |
| OLMo-2-0325-32B-Instruct | Ai2 | apache-2.0 | 32.2B | 9.2K | 148 | Mar 12 2025 |
| granite-guardian-4.1-8b | IBM | apache-2.0 | 8.4B | 9K | 52 | Apr 16 2026 |
| ERNIE-4.5-0.3B-PT | Baidu | apache-2.0 | 360.7M | 9K | 112 | Jun 28 2025 |
| NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 | NVIDIA | openmdw-1.1 | 74.9B | 9K | 52 | Oct 09 2026 |
| OLMo-2-0425-1B-SFT | Ai2 | apache-2.0 | n/a | 8.9K | 5 | Apr 24 2025 |
| Llama-2-13b-hf | Meta | llama2 | 13B | 8.9K | 658 | Jul 13 2023 |
| Llama-3.1-405B-Instruct | Meta | llama3.1 | 405.9B | 8.9K | 608 | Jul 16 2024 |
| DeepSeek-V3.1-Base | DeepSeek | mit | 684.5B | 8.8K | 1,030 | Aug 19 2025 |
| DeepSeek-V2 | DeepSeek | deepseek | 235.7B | 8.8K | 360 | Apr 22 2024 |
| reformer-enwik8 | Google | n/a | n/a | 8.7K | 26 | Mar 02 2022 |
| DeepSeek-V4.1-Flash-1017-NVFP4 | NVIDIA | mit | 664.9B | 8.5K | 187 | Oct 30 2026 |
| LFM2.5-1.2B-Thinking | Liquid AI | lfm1.0 | 1.2B | 8.5K | 407 | Jan 20 2026 |
| SmolLM-360M | Hugging Face | apache-2.0 | 361.8M | 8.5K | 74 | Jul 14 2024 |
| Llama-3.1-Tulu-3-8B-SFT | Ai2 | llama3.1 | 8B | 8.4K | 37 | Nov 18 2024 |
| deepseek-moe-16b-base | DeepSeek | deepseek | 16.4B | 8.4K | 170 | Jan 08 2024 |
| North-Mini-Code-1.0 | Cohere | apache-2.0 | 30.5B | 8.3K | 588 | Jun 05 2026 |
| Llama-3.2-11B-Vision | Meta | llama3.2 | 10.6B | 8.3K | 619 | Sep 18 2024 |
| LFM2-VL-1.6B | Liquid AI | lfm1.0 | 1.6B | 8.1K | 234 | Aug 12 2025 |
| UI-Venus-2-9B | inclusionAI | n/a | 1.5M | 8.1K | 48 | Aug 26 2026 |
| Falcon3-1B-Instruct | TII | falcon-llm-license | 1.7B | 8K | 46 | Dec 14 2024 |
| Falcon3-10B-Instruct | TII | falcon-llm-license | 10.3B | 8K | 117 | Dec 14 2024 |
| LFM2.5-1.2B-Base | Liquid AI | other | 1.2B | 7.9K | 153 | Jan 05 2026 |
| granite-4.0-tiny-preview | IBM | apache-2.0 | 6.7B | 7.9K | 184 | Apr 30 2025 |
| GLM-4-32B-0414 | Z.ai | mit | 32.6B | 7.8K | 490 | Apr 07 2025 |
| SmolLM-360M-Instruct | Hugging Face | apache-2.0 | 361.8M | 7.8K | 89 | Jul 15 2024 |
| Ling-3.1-flash | inclusionAI | mit | 572.1B | 7.7K | 165 | Oct 13 2026 |
| SmolLM-1.7B-Instruct | Hugging Face | apache-2.0 | 1.7B | 7.7K | 124 | Jul 15 2024 |
| LFM2.5-2.6B-Base | Liquid AI | lfm1.0 | 2.7B | 7.7K | 52 | Aug 01 2026 |
| OLMoE-1B-7B-0125 | Ai2 | apache-2.0 | 6.9B | 7.6K | 41 | Jan 21 2025 |
| DeepSeek-Coder-V2-Lite-Base | DeepSeek | deepseek-license | 15.7B | 7.4K | 149 | Jun 14 2024 |
| Olmo-3-7B-Think-DPO | Ai2 | apache-2.0 | 7.3B | 7.3K | 7 | Nov 18 2025 |
| Olmo-3.1-32B-Think | Ai2 | apache-2.0 | 32.2B | 7.2K | 120 | Dec 10 2025 |
| Falcon3-3B-Instruct | TII | falcon-llm-license | 3.2B | 7.2K | 28 | Dec 14 2024 |
| deepseek-vl-1.3b-chat | DeepSeek | deepseek | 2B | 7.1K | 83 | Mar 07 2024 |
| Hy-Memory-30B-A3B | Tencent | apache-2.0 | 30.1B | 7.1K | 325 | Oct 15 2026 |
| Olmo-3-7B-Think-SFT | Ai2 | apache-2.0 | 7.3B | 6.9K | 14 | Oct 14 2025 |
| OLMoE-1B-7B-0924-Instruct | Ai2 | apache-2.0 | 6.9B | 6.9K | 99 | Aug 13 2024 |
| Phi-3-small-8k-instruct | Microsoft | mit | 7.4B | 6.9K | 182 | May 07 2024 |
| Phi-4-reasoning-vision-15B | Microsoft | mit | 15.1B | 6.8K | 185 | Jan 23 2026 |
| reformer-crime-and-punishment | Google | n/a | n/a | 6.7K | 21 | Mar 02 2022 |
| NVLM-D-72B | NVIDIA | cc-by-nc-4.0 | 79.4B | 6.7K | 780 | Sep 30 2024 |
| glm-4-9b | Z.ai | glm-4 | 9.4B | 6.5K | 145 | Jun 04 2024 |
| granite-4.0-h-micro | IBM | apache-2.0 | 3.2B | 6.4K | 148 | Sep 16 2025 |
| LFM2-2.6B | Liquid AI | lfm1.0 | 2.6B | 6.3K | 197 | Sep 22 2025 |
| translategemma-12b-it | Google | gemma | 12.2B | 6.3K | 367 | Jan 12 2026 |
| granitelib-core-r1.0 | IBM | apache-2.0 | n/a | 6.3K | 45 | Dec 10 2025 |
| aya-23-8B | Cohere | cc-by-nc-4.0 | 8B | 6.2K | 440 | May 19 2024 |
| MiniMax-VL-01 | MiniMax | n/a | 456.4B | 6.2K | 287 | Jan 12 2025 |
| Mage-VL | Microsoft | apache-2.0 | 4.7B | 6.2K | 451 | Jul 25 2026 |
| deepseek-vl-7b-chat | DeepSeek | deepseek | 7.3B | 6.2K | 289 | Mar 07 2024 |
| falcon-mamba-tiny-dev | TII | n/a | 8.8M | 6.2K | 2 | Oct 13 2024 |
| NV-Reason-CT | NVIDIA | openmdw-1.1 | 5.3B | 6K | 90 | Sep 08 2026 |
| glm-4-9b-chat-hf | Z.ai | glm-4 | 9.4B | 6K | 25 | Oct 23 2024 |
| Falcon-H1-0.5B-Base | TII | falcon-llm-license | 521.4M | 5.9K | 17 | May 01 2025 |
| Bolmo-1B-Stage1 | Ai2 | apache-2.0 | 1.5B | 5.9K | 0 | Aug 26 2026 |
| Llama-3.1-Tulu-3-8B-DPO | Ai2 | llama3.1 | 8B | 5.8K | 30 | Nov 20 2024 |
| Falcon-H1-3B-Instruct | TII | falcon-llm-license | 3.1B | 5.8K | 18 | May 01 2025 |
| tiny-aya-earth | Cohere | cc-by-nc-4.0 | 3.3B | 5.8K | 32 | Feb 13 2026 |
| txgemma-2b-predict | Google | health-ai-developer-foundations | 2.6B | 5.8K | 60 | Mar 21 2025 |
| granite-switch-4.1-8b | IBM | apache-2.0 | 8.8B | 5.7K | 56 | Oct 14 2026 |
| paligemma2-10b-ft-docci-448 | Google | gemma | 9.7B | 5.7K | 18 | Nov 21 2024 |
| Step3-VL-10B | StepFun | apache-2.0 | 10.2B | 5.6K | 418 | Jan 13 2026 |
| falcon-11B | TII | unknown | 11.1B | 5.5K | 219 | May 09 2024 |
| gemma-1.1-2b-it | Google | gemma | 2.5B | 5.5K | 175 | Mar 26 2024 |
| deepseek-vl2 | DeepSeek | deepseek | 27.5B | 5.5K | 410 | Dec 13 2024 |
| paligemma2-3b-pt-224 | Google | gemma | 3B | 5.5K | 186 | Nov 21 2024 |
| deepseek-vl2-small | DeepSeek | deepseek | 16.1B | 5.4K | 195 | Dec 13 2024 |
| GLM-4.6V | Z.ai | mit | 107.7B | 5.3K | 398 | Dec 07 2025 |
| granite-3.0-1b-a400m-instruct | IBM | apache-2.0 | 1.3B | 5.2K | 21 | Oct 03 2024 |
| granite-3.2-2b-instruct | IBM | apache-2.0 | 2.5B | 5.2K | 54 | Feb 17 2025 |
| Hunyuan-7B-Instruct | Tencent | n/a | 7.5B | 5.2K | 90 | Jul 30 2025 |
| falcon-rw-1b | TII | apache-2.0 | n/a | 5.1K | 120 | Apr 26 2023 |
| Llama-3.1-Tulu-3-8B | Ai2 | llama3.1 | 8B | 5.1K | 179 | Nov 20 2024 |
| translategemma-27b-it | Google | gemma | 27.4B | 5.1K | 408 | Jan 12 2026 |
| OLMo-7B-hf | Ai2 | apache-2.0 | 6.9B | 5K | 17 | Apr 12 2024 |
| llava-med-v1.5-mistral-7b | Microsoft | apache-2.0 | 7.6B | 5K | 140 | May 14 2024 |
| OLMo-2-0325-32B | Ai2 | apache-2.0 | 32.2B | 5K | 66 | Feb 23 2025 |
| Falcon3-7B-Base | TII | falcon-llm-license | 7.5B | 4.9K | 41 | Nov 21 2024 |
| Phi-4-reasoning-plus | Microsoft | mit | 14.7B | 4.9K | 354 | Apr 17 2025 |
| Falcon-H1-7B-Instruct | TII | falcon-llm-license | 7.6B | 4.9K | 36 | May 01 2025 |
| falcon-mamba-7b | TII | falcon-mamba-7b-license | 7.3B | 4.8K | 247 | Jul 17 2024 |
| command-a-plus-05-2026-w4a4 | Cohere | apache-2.0 | 218.8B | 4.7K | 249 | May 18 2026 |
| Seed-Coder-8B-Instruct | ByteDance | mit | 8.3B | 4.7K | 119 | Apr 27 2025 |
| granite-switch-4.1-30b | IBM | apache-2.0 | 30B | 4.6K | 46 | Oct 14 2026 |
| Llama-2-70b-chat-hf | Meta | llama2 | 69B | 4.6K | 2,211 | Jul 14 2023 |
| MolmoPoint-8B | Ai2 | apache-2.0 | 8.7B | 4.6K | 32 | Mar 16 2026 |
| OLMo-2-0425-1B-DPO | Ai2 | apache-2.0 | n/a | 4.6K | 4 | Apr 28 2025 |
| Ling-flash-2.0 | inclusionAI | mit | 102.9B | 4.5K | 222 | Sep 17 2025 |
| Ling-mini-2.0 | inclusionAI | mit | 16.3B | 4.5K | 209 | Sep 08 2025 |
| Falcon-H1-1.5B-Deep-Instruct | TII | falcon-llm-license | 1.6B | 4.4K | 40 | May 01 2025 |
| granite-3.1-1b-a400m-base | IBM | apache-2.0 | 1.3B | 4.4K | 14 | Dec 06 2024 |
| MiMo-VL-7B-RL-2508 | Xiaomi | mit | 8.3B | 4.4K | 97 | Aug 07 2025 |
| OLMo-1B-0724-hf | Ai2 | apache-2.0 | 1.3B | 4.4K | 24 | Jun 15 2024 |
| AutoGLM-Phone-9B | Z.ai | mit | 934.4K | 4.3K | 447 | Dec 08 2025 |
| granite-4.0-h-350m | IBM | apache-2.0 | 340.3M | 4.3K | 114 | Oct 07 2025 |
| DeepSeek-V2.5 | DeepSeek | deepseek | 235.7B | 4.3K | 750 | Sep 05 2024 |
| Falcon3-1B-Base | TII | falcon-llm-license | 1.7B | 4.2K | 31 | Dec 13 2024 |
| udop-large | Microsoft | mit | 741.7M | 4.2K | 130 | Feb 26 2024 |
| Youtu-LLM-2B | Tencent | youtu-llm | 2B | 4.2K | 242 | Dec 31 2025 |
| gemma-1.1-7b-it | Google | gemma | 8.5B | 4.2K | 276 | Mar 26 2024 |
| DeepSeek-Coder-V2-Base | DeepSeek | deepseek-license | 235.7B | 4.2K | 97 | Jun 14 2024 |
| Flora-Captioner | NVIDIA | nvidia-license | 27.4B | 4.1K | 40 | Oct 01 2026 |
| LLaDA2.2-mini | inclusionAI | apache-2.0 | 16.3B | 4.1K | 39 | Sep 05 2026 |
| Falcon-H1-0.5B-Instruct | TII | falcon-llm-license | 521.4M | 4.1K | 35 | May 01 2025 |
| codegemma-7b-it | Google | gemma | 8.5B | 4.1K | 267 | Mar 21 2024 |
| paligemma2-3b-pt-448 | Google | gemma | 3B | 4K | 58 | Nov 21 2024 |
| Olmo-3.1-7B-RL-Zero-Math | Ai2 | apache-2.0 | 7.3B | 3.9K | 13 | Dec 12 2025 |
| MiniMax-Text-01-hf | MiniMax | minimax | 456.1B | 3.9K | 11 | Jun 03 2025 |
| gemma-2-27b | Google | gemma | 27.2B | 3.8K | 211 | Jun 24 2024 |
| Ling-3.1-flash-1019 | inclusionAI | mit | 572.1B | 3.8K | 98 | Oct 27 2026 |
| Olmo-3-7B-RL-Zero-Math | Ai2 | apache-2.0 | 7.3B | 3.8K | 13 | Nov 17 2025 |
| Muse-Glimmer-30B-ExecuTorch-PTE | Meta | apache-2.0 | n/a | 3.7K | 43 | Aug 10 2026 |
| Mage-VL-Cowork | Microsoft | apache-2.0 | 4.7B | 3.7K | 216 | Oct 30 2026 |
| Phi-mini-MoE-instruct | Microsoft | mit | 7.6B | 3.7K | 52 | Jun 23 2025 |
| LFM2-VL-3B | Liquid AI | lfm1.0 | 3B | 3.7K | 142 | Oct 22 2025 |
| GLM-4.5-Air-Base | Z.ai | mit | 110.5B | 3.6K | 47 | Jul 20 2025 |
| Ling-1T | inclusionAI | mit | 999.7B | 3.6K | 546 | Oct 02 2025 |
| Muse-Glimmer-30B-NVFP4 | NVIDIA | apache-2.0 | 19.8B | 3.5K | 29 | Aug 27 2026 |
| Fara1.5-27B | Microsoft | mit | 27.4B | 3.5K | 299 | Jul 17 2026 |
| CodeLlama-34b-hf | Meta | llama2 | 33.7B | 3.4K | 20 | Mar 14 2024 |
| kosmos-2.5 | Microsoft | mit | 1.4B | 3.4K | 277 | May 13 2024 |
| ERNIE-4.5-300B-A47B-PT | Baidu | apache-2.0 | 300.5B | 3.4K | 78 | Jun 28 2025 |
| gemma-2-2b-jpn-it | Google | gemma | 2.6B | 3.3K | 218 | Sep 25 2024 |
| falcon-mamba-7b-instruct | TII | falcon-mamba-7b-license | 7.3B | 3.3K | 74 | Jul 30 2024 |
| Falcon-H1-34B-Instruct | TII | falcon-llm-license | 33.6B | 3.3K | 53 | May 01 2025 |
| shieldgemma-2b | Google | gemma | 2.6B | 3.2K | 142 | Jul 16 2024 |
| MiniMax-Text-01 | MiniMax | n/a | 456.1B | 3.2K | 657 | Jan 12 2025 |
| Falcon-H1-1.5B-Instruct | TII | falcon-llm-license | 1.6B | 3.2K | 17 | May 01 2025 |
| bitnet-b1.58-2B-4T-bf16 | Microsoft | mit | 2.4B | 3.1K | 53 | Apr 15 2025 |
| Youtu-Parsing-Omni | Tencent | youtu-parsing | 5.3B | 3.1K | 132 | Oct 09 2026 |
| Olmo-3-32B-Think-SFT | Ai2 | apache-2.0 | 32.2B | 3.1K | 4 | Nov 14 2025 |
| FlexOlmo-7x7B-1T | Ai2 | apache-2.0 | 33.3B | 3.1K | 43 | Jun 11 2025 |
| aya-expanse-32b | Cohere | cc-by-nc-4.0 | 32.3B | 3.1K | 303 | Oct 23 2024 |
| Falcon-H1-Tiny-90M-Instruct | TII | falcon-llm-license | 91.1M | 3.1K | 50 | Jan 12 2026 |
| Ling-2.6-flash | inclusionAI | mit | 107.5B | 3.1K | 506 | Apr 28 2026 |
| Fara-7B | Microsoft | mit | 8.3B | 3K | 625 | Oct 30 2025 |
| UI-Venus-1.5-2B | inclusionAI | apache-2.0 | 2.4B | 3K | 43 | Feb 09 2026 |
| CodeGPT-small-py | Microsoft | n/a | n/a | 3K | 29 | Mar 02 2022 |
| NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16 | NVIDIA | openmdw-1.1 | 124.5B | 3K | 61 | Oct 12 2026 |
| recurrentgemma-2b-it | Google | gemma | 2.7B | 3K | 118 | Apr 08 2024 |
| OLMo-2-1124-7B-SFT | Ai2 | apache-2.0 | n/a | 2.9K | 1 | Dec 18 2024 |
| Falcon3-10B-Base | TII | falcon-llm-license | 10.3B | 2.9K | 42 | Dec 03 2024 |
| UserRepGen-4B | Microsoft | mit | 4.5B | 2.9K | 28 | Oct 14 2026 |
| tiny-aya-base | Cohere | cc-by-nc-4.0 | 3.3B | 2.9K | 72 | Feb 13 2026 |
| Fara1.5-9B | Microsoft | mit | 9.4B | 2.8K | 64 | May 12 2026 |
| ERNIE-4.5-VL-28B-A3B-Thinking | Baidu | apache-2.0 | 29.7B | 2.8K | 548 | Nov 07 2025 |
| Olmo-Hybrid-Instruct-SFT-7B | Ai2 | apache-2.0 | 7.4B | 2.8K | 17 | Feb 19 2026 |
| Phi-3-small-128k-instruct | Microsoft | mit | 7.4B | 2.8K | 186 | May 07 2024 |
| LlamaGuard-7b | Meta | llama2 | 6.7B | 2.8K | 249 | Dec 05 2023 |
| codegemma-7b | Google | gemma | 8.5B | 2.7K | 226 | Mar 21 2024 |
| Falcon3-3B-Base | TII | falcon-llm-license | 3.2B | 2.7K | 17 | Dec 13 2024 |
| Sentiment-Analysis-7B | Microsoft | mit | 7.2B | 2.7K | 33 | Oct 17 2026 |
| DAM-3B-Video | NVIDIA | nvidia-non-commercial-license | n/a | 2.7K | 64 | Apr 21 2025 |
| granite-vision-3.2-2b | IBM | apache-2.0 | 3B | 2.7K | 126 | Feb 17 2025 |
| Kimi-Dev-72B | Moonshot AI | mit | 72.7B | 2.7K | 393 | Jun 16 2025 |
| AesCode-8B | Microsoft | apache-2.0 | 8.8B | 2.6K | 81 | Sep 29 2026 |
| Phi-4-mini-flash-reasoning | Microsoft | mit | 3.9B | 2.6K | 294 | Jun 19 2025 |
| Fara1.5-4B | Microsoft | mit | 4.5B | 2.6K | 60 | Jul 17 2026 |
| granite-4.0-3b-vision | IBM | apache-2.0 | 4B | 2.6K | 113 | Mar 03 2026 |
| Falcon-H1R-7B | TII | falcon-llm-license | 7.6B | 2.6K | 222 | Oct 29 2025 |
| Muse-Spark-Open-816B-A42B-DFlash2 | NVIDIA | nvidia-open-model-agreement-license | 4.4B | 2.6K | 9 | Oct 16 2026 |
| Molmo-7B-O-0924 | Ai2 | apache-2.0 | 7.7B | 2.6K | 165 | Sep 25 2024 |
| CodeLlama-7b-Instruct-hf | Meta | llama2 | 6.7B | 2.5K | 65 | Mar 13 2024 |
| Hunyuan-0.5B-Instruct | Tencent | n/a | 539M | 2.5K | 61 | Jul 30 2025 |
| VisionReward-Video | Z.ai | cogvlm2 | 12.5B | 2.5K | 10 | Dec 10 2024 |
| Falcon-H1-1.5B-Base | TII | falcon-llm-license | 1.6B | 2.5K | 3 | May 01 2025 |
| Ising-Calibration-1.5-31B-BF16 | NVIDIA | openmdw-1.1 | 31.3B | 2.5K | 14 | Jul 13 2026 |
| c4ai-command-r7b-12-2024 | Cohere | cc-by-nc-4.0 | 8B | 2.5K | 436 | Dec 11 2024 |
| Ling-3.0-flash-Sante | inclusionAI | mit | 127.5B | 2.5K | 85 | Oct 19 2026 |
| Laguna-M.1-NVFP4 | Poolside | apache-2.0 | 225.8B | 2.5K | 13 | May 21 2026 |
| UI-Venus-1.5-8B | inclusionAI | apache-2.0 | 8.8B | 2.4K | 29 | Feb 09 2026 |
| glm-4-9b-hf | Z.ai | glm-4 | 9.4B | 2.4K | 10 | Jan 16 2025 |
| Falcon-H1-3B-Base | TII | falcon-llm-license | 3.1B | 2.4K | 7 | May 01 2025 |
| Step-3.7-Flash-NVFP4 | StepFun | apache-2.0 | 103.8B | 2.4K | 67 | May 27 2026 |
| UI-TARS-2B-SFT | ByteDance | apache-2.0 | 2.4B | 2.3K | 41 | Jan 20 2025 |
| Ling-lite | inclusionAI | mit | 16.8B | 2.3K | 81 | Feb 28 2025 |
| UI-TARS-7B-DPO | ByteDance | apache-2.0 | 8.3B | 2.3K | 235 | Jan 22 2025 |
| UserLM-8b | Microsoft | mit | 8B | 2.3K | 401 | Sep 30 2025 |
| codegeex4-all-9b | Z.ai | codegeex4 | 9.4B | 2.3K | 277 | Jul 05 2024 |
| Ling-2.6-1T | inclusionAI | mit | 1T | 2.3K | 508 | Apr 29 2026 |
| glm-edge-1.5b-chat | Z.ai | glm-4 | 1.6B | 2.2K | 22 | Nov 20 2024 |
| CodeGPT-small-java-adaptedGPT2 | Microsoft | n/a | n/a | 2.2K | 19 | Mar 02 2022 |
| Ring-2.6-1T | inclusionAI | mit | 1T | 2.2K | 107 | May 14 2026 |
| Meta-Llama-Guard-2-8B | Meta | llama3 | 8B | 2.2K | 318 | Apr 17 2024 |
| AesCode-32B | Microsoft | apache-2.0 | 33.4B | 2.2K | 59 | Sep 29 2026 |
| GLM-Z1-9B-0414 | Z.ai | mit | 9.4B | 2.2K | 92 | Apr 08 2025 |
| GLM-4.1V-9B-Base | Z.ai | mit | 10.3B | 2.1K | 68 | Jun 28 2025 |
| ERNIE-4.5-0.3B-Base-PT | Baidu | apache-2.0 | 360.7M | 2.1K | 27 | Jun 28 2025 |
| MediPhi-Instruct | Microsoft | mit | 3.8B | 2.1K | 77 | Jul 11 2025 |
| granite-vision-3.3-2b-chart2csv-preview | IBM | apache-2.0 | 3B | 2.1K | 17 | Jan 29 2026 |
| glm-edge-4b-chat | Z.ai | glm-4 | 4.3B | 2.1K | 13 | Nov 20 2024 |
| Kimi-Linear-48B-A3B-Base | Moonshot AI | mit | 49.1B | 2.1K | 85 | Oct 30 2025 |
| vaultgemma-1b | Google | gemma | 1B | 2K | 412 | Sep 05 2025 |
| Ling-3.0-flash-Fin | inclusionAI | mit | 127.5B | 2K | 133 | Sep 03 2026 |
| Hunyuan-1.8B-Instruct | Tencent | n/a | 1.8B | 2K | 604 | Jul 30 2025 |
| NV-Reason-CXR-3B | NVIDIA | nsclv1 | 3.8B | 2K | 35 | Oct 16 2025 |
| Seed-OSS-36B-Base | ByteDance | apache-2.0 | 36.2B | 2K | 69 | Aug 20 2025 |
| Falcon-H1-Tiny-Multilingual-100M-Instruct | TII | falcon-llm-license | 107.9M | 2K | 14 | Jan 12 2026 |
| Laguna-XS-2.1-NVFP4 | Poolside | openmdw-1.1 | 33.4B | 2K | 17 | Jun 20 2026 |
| UI-TARS-72B-DPO | ByteDance | apache-2.0 | 73.4B | 2K | 159 | Jan 22 2025 |
| MagenticBrain | Microsoft | mit | 14.8B | 2K | 42 | May 12 2026 |
| MiniMax-M1-80k | MiniMax | apache-2.0 | 456.1B | 2K | 692 | Jun 13 2025 |
| Falcon3-Mamba-7B-Instruct | TII | falcon-llm-license | 7.3B | 1.9K | 34 | Dec 13 2024 |
| Molmo-72B-0924 | Ai2 | apache-2.0 | 73.3B | 1.9K | 300 | Sep 25 2024 |
| Llama-4-Scout-17B-16E | Meta | llama4 | 108.6B | 1.9K | 279 | Apr 02 2025 |
| NextCoder-7B | Microsoft | mit | 7.6B | 1.9K | 42 | May 03 2025 |
| MediPhi-Clinical | Microsoft | mit | 3.8B | 1.9K | 24 | May 29 2025 |
| Tencent-Hunyuan-Large | Tencent | tencent-license | n/a | 1.9K | 615 | Oct 22 2024 |
| Falcon-H1-Tiny-R-0.6B | TII | falcon-llm-license | 622.3M | 1.8K | 16 | Jan 12 2026 |
| granite-vision-3.1-2b-preview | IBM | apache-2.0 | 3B | 1.8K | 115 | Jan 27 2025 |
| MAI-DS-R1 | Microsoft | mit | 671B | 1.8K | 316 | Apr 16 2025 |
| codegemma-2b | Google | gemma | 2.5B | 1.8K | 102 | Mar 21 2024 |
| GLM-Z1-Rumination-32B-0414 | Z.ai | mit | 33.1B | 1.8K | 119 | Apr 13 2025 |
| LLaDA2.1-flash | inclusionAI | apache-2.0 | 102.9B | 1.8K | 95 | Feb 09 2026 |
| Falcon-H1-7B-Base | TII | falcon-llm-license | 7.6B | 1.8K | 17 | May 01 2025 |
| circuit-sparsity | OpenAI | apache-2.0 | 419.1M | 1.8K | 221 | Dec 11 2025 |
| ContextPilot-14B | Tencent | other | 14.8B | 1.8K | 23 | Aug 27 2026 |
| MiMo-VL-7B-SFT-2508 | Xiaomi | mit | 8.3B | 1.8K | 37 | Aug 07 2025 |
| HARC-Qwen2.5-7B-Instruct | Microsoft | apache-2.0 | 7.6B | 1.8K | 9 | Jul 02 2026 |
| MiMo-Embodied-7B | Xiaomi | mit | 8.3B | 1.8K | 77 | Nov 19 2025 |
| Youtu-LLM-A7B-Base | Tencent | youtu-llm | 118.4B | 1.8K | 42 | Jan 05 2026 |
| UI-Mate-9B | Tencent | apache-2.0 | 9.4B | 1.8K | 30 | Aug 14 2026 |
| AutoGLM-Phone-9B-Multilingual | Z.ai | mit | 934.4K | 1.8K | 244 | Dec 09 2025 |
| Seed-Coder-8B-Reasoning | ByteDance | mit | 8.3B | 1.7K | 154 | Apr 27 2025 |
| SmolVLM-256M-Base | Hugging Face | n/a | 256.5M | 1.7K | 24 | Jan 10 2025 |
| Glyph | Z.ai | mit | 10.3B | 1.7K | 75 | Oct 25 2025 |
| MiMo-7B-SFT | Xiaomi | mit | 7.8B | 1.7K | 31 | Apr 29 2025 |
| BFS-Prover-V2-7B | ByteDance | apache-2.0 | 7.6B | 1.7K | 7 | Oct 06 2025 |
| ERNIE-5.5-Flash-Base | Baidu | ernie-5.5-flash-base-model-license-v1.0 | 286.2B | 1.7K | 237 | Oct 21 2026 |
| c4ai-command-a-03-2025 | Cohere | cc-by-nc-4.0 | 111.1B | 1.7K | 397 | Mar 11 2025 |
| Ring-mini-2.0 | inclusionAI | mit | 16.3B | 1.7K | 188 | Sep 08 2025 |
| Ling-3.0-tiny-UBTree | inclusionAI | mit | 183.1M | 1.7K | 14 | Oct 21 2026 |
| MiMo-7B-RL-0530 | Xiaomi | mit | 7.8B | 1.7K | 50 | May 30 2025 |
| MolmoWeb-4B | Ai2 | apache-2.0 | 4.9B | 1.7K | 36 | Mar 20 2026 |
| recurrentgemma-2b | Google | gemma | 2.7B | 1.7K | 100 | Apr 06 2024 |
| X-Reasoner-7B | Microsoft | apache-2.0 | 8.3B | 1.6K | 16 | Feb 03 2026 |
| Falcon-H1-Tiny-R-90M | TII | falcon-llm-license | 91.1M | 1.6K | 29 | Jan 12 2026 |
| MiMo-V2.5-Pro-Base | Xiaomi | mit | 1T | 1.6K | 48 | Apr 27 2026 |
| SynLogic-7B | MiniMax | mit | 7.6B | 1.6K | 29 | Jun 03 2025 |
| Qwen2.5-VL-7B-Surg-CholecT50 | NVIDIA | nsclv1 | 8.3B | 1.6K | 14 | Oct 28 2025 |
| MolmoMotion-4B-H3-F30 | Ai2 | apache-2.0 | 4.9B | 1.6K | 13 | Jun 15 2026 |
| c4ai-command-r7b-arabic-02-2025 | Cohere | cc-by-nc-4.0 | 8B | 1.5K | 132 | Feb 27 2025 |
| PaCoRe-8B | StepFun | mit | 8.2B | 1.5K | 49 | Dec 09 2025 |
| SmolLM2-1.7B-Instruct-16k | Hugging Face | apache-2.0 | 1.7B | 1.5K | 10 | Feb 21 2025 |
| ContextPilot-E4B | Tencent | other | 7.9B | 1.5K | 9 | Aug 27 2026 |
| OmniParser | Microsoft | mit | n/a | 1.5K | 1,719 | Oct 07 2024 |
| cogvlm2-llama3-chat-19B | Z.ai | cogvlm2 | 19.5B | 1.5K | 220 | May 16 2024 |
| glm-4-9b-chat-1m-hf | Z.ai | glm-4 | 9.5B | 1.5K | 14 | Oct 24 2024 |
| Qwen3.8-27B-DSpine-b16 | Tencent | apache-2.0 | 2B | 1.5K | 144 | Nov 07 2026 |
| SmolVLM2-2.2B-Base | Hugging Face | apache-2.0 | 2.2B | 1.4K | 12 | Apr 14 2025 |
| nanowhale-100m-base | Hugging Face | apache-2.0 | 110.4M | 1.4K | 25 | Apr 24 2026 |
| StepFun-Formalizer-7B | StepFun | apache-2.0 | 7.6B | 1.4K | 7 | Aug 19 2025 |
| MiMo-VL-7B-RL | Xiaomi | mit | 8.3B | 1.4K | 171 | May 30 2025 |
| Youtu-VL-4B-Instruct | Tencent | youtu-vl | 5.3B | 1.4K | 164 | Jan 23 2026 |
| Ring-flash-2.0 | inclusionAI | mit | 102.9B | 1.4K | 104 | Sep 19 2025 |
| Ising-Calibration-1-35B-A3B | NVIDIA | nvidia-open-model-license | 664.9K | 1.4K | 62 | Mar 30 2026 |
| Ling-plus | inclusionAI | mit | 292.5B | 1.4K | 48 | Feb 28 2025 |
| Falcon-H1-Tiny-Tool-Calling-90M | TII | falcon-llm-license | 91.1M | 1.4K | 17 | Jan 12 2026 |
| Ling-lite-1.5-2507 | inclusionAI | mit | 16.8B | 1.4K | 77 | Aug 08 2025 |
| LFM2.5-VL-1.6B-Extract | Liquid AI | lfm1.0 | 1.6B | 1.4K | 72 | May 26 2026 |
| Ling-Coder-lite | inclusionAI | mit | 16.8B | 1.4K | 55 | Mar 08 2025 |
| d1-3B-w8a8 | Liquid AI | lfm1.0 | 3.1B | 1.4K | 12 | Oct 07 2026 |
| BFS-Prover-V1-7B | ByteDance | apache-2.0 | 7.6B | 1.4K | 26 | Feb 23 2025 |
| Ling-lite-1.5 | inclusionAI | mit | 16.8B | 1.4K | 57 | May 11 2025 |
| LBS-27B | inclusionAI | apache-2.0 | 3.1M | 1.4K | 16 | Oct 26 2026 |
| SWE-Dev-32B | Z.ai | mit | 32.8B | 1.4K | 31 | Apr 06 2025 |
| Step-3.5-Flash-Base-Midtrain | StepFun | apache-2.0 | 197.8B | 1.4K | 43 | Mar 02 2026 |
| Ring-1T | inclusionAI | mit | 999.7B | 1.4K | 233 | Oct 10 2025 |
| Hy3-preview-Base | Tencent | other | 298.8B | 1.4K | 24 | Apr 13 2026 |
| AReaL-boba-2-14B | inclusionAI | apache-2.0 | n/a | 1.3K | 21 | Jun 03 2025 |
| SmolVLM-500M-Base | Hugging Face | n/a | 507.5M | 1.3K | 12 | Jan 13 2025 |
| GELab-Zero-4B-preview | StepFun | apache-2.0 | 4.4B | 1.3K | 153 | Nov 28 2025 |
| AReaL-boba-2-32B | inclusionAI | apache-2.0 | n/a | 1.3K | 19 | Jun 03 2025 |
| Ling-3.0-flash-UBTree | inclusionAI | mit | 482M | 1.3K | 12 | Oct 14 2026 |
| LFM2.5-VL-450M-Extract | Liquid AI | lfm1.0 | 448.7M | 1.3K | 57 | May 26 2026 |
| Seed-Coder-8B-Base | ByteDance | mit | 8.3B | 1.3K | 74 | Apr 27 2025 |
| AReaL-boba-2-8B | inclusionAI | apache-2.0 | n/a | 1.3K | 28 | Jun 03 2025 |
| UniRG-CXR | Microsoft | apache-2.0 | 8.8B | 1.3K | 6 | Mar 19 2026 |
| Rubicon-Preview | inclusionAI | apache-2.0 | 30.5B | 1.3K | 26 | Aug 18 2025 |
| Nemotron-Labs-Diffusion-VLM-8B | NVIDIA | nscl-v1 | 8.9B | 1.3K | 38 | May 08 2026 |
| StepFun-Prover-Preview-7B | StepFun | apache-2.0 | 7.6B | 1.3K | 3 | Aug 13 2025 |
| Seed-Coder-8B-Reasoning-bf16 | ByteDance | mit | 8.3B | 1.2K | 23 | May 13 2025 |
| UI-Venus-Ground-7B | inclusionAI | apache-2.0 | 8.3B | 1.2K | 25 | Aug 15 2025 |
| StepFun-Prover-Preview-32B | StepFun | apache-2.0 | 32.8B | 1.2K | 11 | Aug 13 2025 |
| GUI-Actor-2B-Qwen2-VL | Microsoft | mit | 2.2B | 1.2K | 28 | Jun 01 2025 |
| LLaDA2.0-mini-preview | inclusionAI | apache-2.0 | 16.3B | 1.2K | 88 | Oct 17 2025 |
| TCAndon-Router | Tencent | apache-2.0 | 4B | 1.2K | 13 | Jan 07 2026 |
| Ring-2.5-1T | inclusionAI | mit | 1T | 1.2K | 243 | Feb 10 2026 |
| cosmo-1b | Hugging Face | apache-2.0 | 1.7B | 1.2K | 136 | Feb 19 2024 |
| AReaL-SEA-235B-A22B | inclusionAI | apache-2.0 | 235.1B | 1.2K | 5 | Mar 01 2026 |
| SmolVLM-Base | Hugging Face | apache-2.0 | 2.2B | 1.2K | 92 | Nov 22 2024 |
| CodeLlama-13b-Instruct-hf | Meta | llama2 | 13B | 1.2K | 28 | Mar 13 2024 |
| SmolLM3-3B-GSM8K-SFT | Hugging Face | apache-2.0 | 3.1B | 1.2K | 2 | Apr 03 2026 |
| SingGuard-2b | inclusionAI | apache-2.0 | 2.1B | 1.2K | 15 | May 25 2026 |
| qwen3-1.7b-gsm8k-sft | Hugging Face | apache-2.0 | 1.7B | 1.2K | 4 | Mar 25 2026 |
| ZwZ-7B | inclusionAI | apache-2.0 | 8.3B | 1.2K | 10 | Feb 12 2026 |
| Ling-2.6-1T-base | inclusionAI | mit | 1T | 1.2K | 13 | Jun 02 2026 |
| VISTA-9B | inclusionAI | apache-2.0 | 9.4B | 1.2K | 17 | Jun 12 2026 |
| Ling-plus-base | inclusionAI | mit | 292.5B | 1.1K | 31 | Feb 28 2025 |
| Youtu-LLM-2B-Base | Tencent | youtu-llm | 2B | 1.1K | 43 | Dec 31 2025 |
| nanowhale-100m | Hugging Face | apache-2.0 | 110.4M | 1.1K | 71 | Apr 24 2026 |
| POINTS-Reader | Tencent | apache-2.0 | 4.1B | 1.1K | 103 | Aug 15 2025 |
| SingGuard-NSFA-0.8B | inclusionAI | apache-2.0 | 1.1B | 1.1K | 10 | Jul 10 2026 |
| ZwZ-4B | inclusionAI | apache-2.0 | 4.8B | 1.1K | 34 | Feb 12 2026 |
| UI-Venus-Ground-72B | inclusionAI | apache-2.0 | 73.4B | 1.1K | 15 | Aug 16 2025 |
| Ling-lite-base-1.5 | inclusionAI | mit | 16.8B | 1.1K | 33 | May 11 2025 |
| ViLaSR | inclusionAI | n/a | 8.3B | 1.1K | 18 | Jun 01 2025 |
| Qwen3-32B-AWorld | inclusionAI | apache-2.0 | 32.8B | 1.1K | 15 | Aug 29 2025 |
| MolmoE-1B-0924 | Ai2 | apache-2.0 | n/a | 1.1K | 159 | Sep 24 2024 |
| UI-Venus-Navi-72B | inclusionAI | apache-2.0 | 73.4B | 1.1K | 8 | Aug 16 2025 |
| SingGuard-NSFA-4B | inclusionAI | apache-2.0 | 5.2B | 1K | 5 | Jul 10 2026 |
| t5gemma-s-s-ul2 | Google | gemma | 312.5M | 1K | 6 | Jun 19 2025 |
| ZwZ2-9B | inclusionAI | n/a | 9.4B | 1K | 17 | Oct 16 2026 |
| ERNIE-4.5-VL-424B-A47B-PT | Baidu | apache-2.0 | 423.5B | 1K | 107 | Jun 28 2025 |
| MiMo-VL-7B-SFT | Xiaomi | mit | 8.3B | 998 | 57 | May 30 2025 |
| ZwZ2-27B | inclusionAI | n/a | 27.4B | 989 | 15 | Oct 16 2026 |
| Hunyuan-4B-Instruct | Tencent | n/a | 4.2B | 982 | 30 | Jul 30 2025 |
| aya-vision-8b | Cohere | cc-by-nc-4.0 | 8.6B | 965 | 325 | Mar 02 2025 |
| GLM-4-32B-Base-0414 | Z.ai | mit | 32.6B | 958 | 38 | Apr 07 2025 |
| Hunyuan-7B-Pretrain | Tencent | n/a | 7.5B | 952 | 15 | Jul 28 2025 |
| gemma-7b-aps-it | Google | gemma | 8.5B | 905 | 47 | Sep 06 2024 |
| Llama-Guard-3-11B-Vision | Meta | llama3.2 | 10.7B | 899 | 77 | Sep 20 2024 |
| ContextPilot-8B | Tencent | other | 8.2B | 861 | 12 | Aug 27 2026 |
| ZwZ2-4B | inclusionAI | n/a | 4.5B | 853 | 12 | Oct 15 2026 |
| olmOCR-7B-0725 | Ai2 | apache-2.0 | 8.3B | 847 | 64 | Jul 22 2025 |
| UI-TARS-7B-SFT | ByteDance | apache-2.0 | 8.3B | 845 | 181 | Jan 20 2025 |
| North-Mini-Code-1.0-w4a16 | Cohere | apache-2.0 | 30.5B | 822 | 61 | Jun 16 2026 |
| glm-edge-v-2b | Z.ai | glm-4 | 2.1B | 818 | 18 | Nov 24 2024 |
| txgemma-27b-chat | Google | health-ai-developer-foundations | 27.2B | 814 | 62 | Mar 21 2025 |
| SynLogic-32B | MiniMax | mit | 32.8B | 788 | 17 | May 30 2025 |
| GUI-Actor-3B-Qwen2.5-VL | Microsoft | mit | 3.8B | 786 | 17 | Jun 01 2025 |
| tiny-aya-fire | Cohere | cc-by-nc-4.0 | 3.3B | 778 | 30 | Feb 13 2026 |
| GUI-Actor-Verifier-2B | Microsoft | mit | 2.2B | 769 | 17 | Jun 03 2025 |
| Step-3.5-Flash-Base | StepFun | apache-2.0 | 197.8B | 767 | 88 | Mar 02 2026 |
| MolmoWeb-8B | Ai2 | apache-2.0 | 8.7B | 766 | 69 | Mar 20 2026 |
| MiMo-V2.5-Base | Xiaomi | mit | 310.8B | 765 | 37 | Apr 27 2026 |
| cogvlm-chat-hf | Z.ai | apache-2.0 | 17.6B | 760 | 200 | Nov 16 2023 |
| ZwZ-2B | inclusionAI | apache-2.0 | 2.4B | 735 | 10 | Mar 04 2026 |
| Penguin-VL-2B | Tencent | apache-2.0 | 2.2B | 734 | 44 | Mar 05 2026 |
| Stable-DiffCoder-8B-Base | ByteDance | mit | 8.3B | 729 | 22 | Jan 15 2026 |
| LongCite-llama3.1-8b | Z.ai | n/a | 8B | 728 | 31 | Sep 02 2024 |
| Stable-DiffCoder-8B-Instruct | ByteDance | mit | 8.3B | 721 | 143 | Jan 15 2026 |
| glm-edge-v-5b | Z.ai | glm-4 | 4.9B | 720 | 17 | Nov 24 2024 |
| MiMo-V2-Flash-Base | Xiaomi | mit | 309.8B | 706 | 55 | Dec 16 2025 |
| Hunyuan-7B-Pretrain-0124 | Tencent | tencent-license | n/a | 685 | 11 | Jan 24 2025 |
| HY-Embodied-0.5 | Tencent | other | 3.8B | 685 | 923 | Apr 02 2026 |
| GLM-4.5-Base | Z.ai | mit | 358.3B | 666 | 61 | Jul 20 2025 |
| UI-Mate-27B | Tencent | apache-2.0 | 27.4B | 665 | 98 | Aug 14 2026 |
| Hy-Embodied-VLM-1.0 | Tencent | apache-2.0 | 30.5B | 661 | 26 | Jul 14 2026 |
| ERNIE-4.5-21B-A3B-Base-PT | Baidu | apache-2.0 | 21.8B | 652 | 35 | Jun 28 2025 |
| WeDLM-8B-Instruct | Tencent | apache-2.0 | 8.2B | 646 | 314 | Dec 25 2025 |
| olmOCR-7B-0825 | Ai2 | apache-2.0 | 8.3B | 645 | 58 | Aug 13 2025 |
| UI-Venus-1.5-30B-A3B | inclusionAI | apache-2.0 | 31.1B | 643 | 37 | Feb 09 2026 |
| MolmoMotion-4B-H1-F32 | Ai2 | apache-2.0 | 4.9B | 632 | 5 | Jun 15 2026 |
| ERNIE-4.5-VL-28B-A3B-Base-PT | Baidu | apache-2.0 | 29.4B | 624 | 42 | Jun 28 2025 |
| MiniMax-M1-80k-hf | MiniMax | apache-2.0 | 456.1B | 623 | 8 | Jul 01 2025 |
| ERNIE-5.5-Flash-Base-BF16 | Baidu | ernie-5.5-flash-base-model-license-v1.0 | 286.2B | 618 | 383 | Oct 21 2026 |
| Cola2 | ByteDance | n/a | 8.8B | 604 | 12 | Nov 06 2026 |
| Hunyuan-4B-Pretrain | Tencent | n/a | 4.2B | 602 | 8 | Jul 28 2025 |
| ERNIE-4.5-21B-A3B-Base-Paddle | Baidu | apache-2.0 | 22.1B | 599 | 16 | Jun 28 2025 |
| MiniMax-M1-40k-hf | MiniMax | apache-2.0 | 456.1B | 594 | 12 | Jul 01 2025 |
| ERNIE-4.5-0.3B-Paddle | Baidu | apache-2.0 | 360.7M | 583 | 30 | Jun 29 2025 |
| Seed-OSS-36B-Base-woSyn | ByteDance | apache-2.0 | 36.2B | 583 | 54 | Aug 20 2025 |
| HY-Embodied-0.5-X | Tencent | other | 3.8B | 561 | 17 | Apr 23 2026 |
| ERNIE-4.5-300B-A47B-Paddle | Baidu | apache-2.0 | 300.5B | 561 | 16 | Jun 28 2025 |
| GUI-Actor-7B-Qwen2.5-VL | Microsoft | mit | 8.4B | 558 | 32 | Jun 01 2025 |
| command-a-reasoning-08-2025 | Cohere | cc-by-nc-4.0 | 111.1B | 556 | 145 | Aug 12 2025 |
| DeepSeek-V3.1-Terminus-W4AFP8 | Tencent | mit | 349.2B | 538 | 15 | Oct 28 2025 |
| GUI-Actor-7B-Qwen2-VL | Microsoft | mit | 8.4B | 510 | 44 | Jun 01 2025 |
| UI2Code_N | Z.ai | mit | 10.3B | 503 | 27 | Nov 11 2025 |
| ERNIE-4.5-0.3B-Base-Paddle | Baidu | apache-2.0 | 360.7M | 502 | 17 | Jun 29 2025 |
| Hunyuan-0.5B-Pretrain | Tencent | n/a | 539M | 496 | 10 | Jul 28 2025 |
| CodeLlama-34b-Instruct-hf | Meta | llama2 | 33.7B | 495 | 19 | Mar 14 2024 |
| ERNIE-4.5-VL-28B-A3B-Base-Paddle | Baidu | apache-2.0 | 29.4B | 495 | 23 | Jun 28 2025 |
| Hunyuan-A13B-Pretrain | Tencent | tencent-hunyuan-a13b | 80.4B | 481 | 23 | Jun 27 2025 |
| Laguna-XS.2-NVFP4 | Poolside | apache-2.0 | 33.4B | 475 | 27 | Apr 23 2026 |
| Hunyuan-1.8B-Pretrain | Tencent | n/a | 1.8B | 473 | 11 | Jul 28 2025 |
| POINTS-GUI-G | Tencent | other | 8.9B | 473 | 22 | Feb 03 2026 |
| ERNIE-4.5-300B-A47B-Base-PT | Baidu | apache-2.0 | 299.5B | 469 | 42 | Jun 28 2025 |
| MiMo-7B-RL-Zero | Xiaomi | mit | 7.8B | 462 | 21 | Apr 29 2025 |
| HiLS-Attention-7B | Tencent | apache-2.0 | 7.3B | 459 | 24 | Jul 09 2026 |
| kosmos-2.5-chat | Microsoft | mit | 1.4B | 452 | 31 | Aug 22 2024 |
| Youtu-Parsing | Tencent | youtu-parsing | 2.5B | 449 | 44 | Jan 23 2026 |
| MolmoPoint-GUI-8B | Ai2 | apache-2.0 | 8.7B | 444 | 20 | Mar 17 2026 |
| WebVIA-Agent | Z.ai | mit | 10.3B | 443 | 21 | Nov 12 2025 |
| git-base-vatex | Microsoft | mit | 176.6M | 437 | 10 | Jan 02 2023 |
| udop-large-512-300k | Microsoft | mit | 741.7M | 436 | 40 | Feb 26 2024 |
| cogagent-9b-20241220 | Z.ai | other | 13.9B | 436 | 55 | Dec 20 2024 |
| LLaDA-UI | inclusionAI | n/a | 16.9B | 430 | 24 | Sep 06 2026 |
| ERNIE-4.5-VL-424B-A47B-Base-PT | Baidu | apache-2.0 | 423.5B | 404 | 91 | Jun 28 2025 |
| SynLogic-Mix-3-32B | MiniMax | mit | 32.8B | 402 | 21 | May 30 2025 |
| Penguin-VL-8B | Tencent | apache-2.0 | 8.7B | 400 | 80 | Mar 05 2026 |
| Step3-VL-10B-Base | StepFun | apache-2.0 | 10.2B | 395 | 52 | Jan 13 2026 |
| Qianfan-VL-8B | Baidu | other | 8.8B | 390 | 42 | Sep 16 2025 |
| UI-Mate-democua-27B | Tencent | apache-2.0 | 27.4B | 383 | 18 | Aug 14 2026 |
| ZwZ-8B | inclusionAI | apache-2.0 | 8.8B | 382 | 50 | Feb 12 2026 |
| RLVR-8B-0926 | StepFun | mit | 8.2B | 372 | 9 | Dec 09 2025 |
| udop-large-512 | Microsoft | mit | 741.7M | 369 | 12 | Feb 26 2024 |
| Llama-3.2-90B-Vision-Instruct | Meta | llama3.2 | 88.6B | 358 | 369 | Sep 19 2024 |
| tiny-aya-water | Cohere | cc-by-nc-4.0 | 3.3B | 358 | 49 | Feb 13 2026 |
| git-large-vatex | Microsoft | mit | n/a | 350 | 6 | Jan 02 2023 |
| WeDLM-7B-Instruct | Tencent | apache-2.0 | 7.6B | 347 | 34 | Dec 25 2025 |
| falcon-11B-vlm | TII | unknown | 11.4B | 339 | 49 | May 21 2024 |
| tiny-aya-l2-thinker | Cohere | cc-by-nc-4.0 | 3.4B | 333 | 16 | Sep 02 2026 |
| aya-23-35B | Cohere | cc-by-nc-4.0 | 35B | 319 | 291 | May 19 2024 |
| Sequential-Hidden-Decoding-8B-n8-Instruct | Tencent | sequential-hidden-decoding | 12.5B | 318 | 9 | Mar 31 2026 |
| Hunyuan-7B-Instruct-0124 | Tencent | tencent-license | n/a | 316 | 51 | Jan 24 2025 |
| StepFun-Formalizer-32B | StepFun | apache-2.0 | 32.8B | 313 | 12 | Aug 19 2025 |
| ERNIE-4.5-21B-A3B-Paddle | Baidu | apache-2.0 | 21.9B | 308 | 19 | Jun 28 2025 |
| Qianfan-VL-70B | Baidu | mit | 71.8B | 300 | 40 | Sep 16 2025 |
| ERNIE-4.5-300B-A47B-2Bits-Paddle | Baidu | apache-2.0 | 89.6B | 285 | 24 | Jun 28 2025 |
| WeDLM-8B-Base | Tencent | apache-2.0 | 8.2B | 268 | 18 | Dec 25 2025 |
| Qianfan-VL-3B | Baidu | other | 3.7B | 268 | 30 | Sep 16 2025 |
| SmolVLM-Synthetic | Hugging Face | apache-2.0 | 2.2B | 266 | 12 | Nov 22 2024 |
| Laguna-M.1-base | Poolside | apache-2.0 | 225.8B | 259 | 14 | Jun 05 2026 |
| UI-TARS-72B-SFT | ByteDance | apache-2.0 | 73.4B | 258 | 25 | Jan 20 2025 |
| SAIL-7B | ByteDance | apache-2.0 | n/a | 258 | 15 | May 07 2025 |
| ERNIE-4.5-VL-28B-A3B-Paddle | Baidu | apache-2.0 | 29.4B | 255 | 42 | Jun 28 2025 |
| viscon-contextual-captioner | TII | n/a | 8.4B | 254 | 2 | Jun 15 2024 |
| ArmorOCR | inclusionAI | apache-2.0 | 8.8B | 254 | 12 | Aug 20 2026 |
| WeDLM-7B-Base | Tencent | apache-2.0 | 7.6B | 250 | 12 | Dec 25 2025 |
| POINTS-Seeker | Tencent | license-term-of-points-seeker | 8.9B | 244 | 8 | Mar 24 2026 |
| ERNIE-4.5-VL-424B-A47B-Paddle | Baidu | apache-2.0 | 423.5B | 241 | 26 | Jun 28 2025 |
| c4ai-command-r-plus | Cohere | cc-by-nc-4.0 | 103.8B | 236 | 1,829 | Apr 03 2024 |
| ERNIE-4.5-300B-A47B-Base-Paddle | Baidu | apache-2.0 | 299.5B | 232 | 22 | Jun 28 2025 |
| ERNIE-4.5-VL-424B-A47B-Base-Paddle | Baidu | apache-2.0 | 423.5B | 231 | 68 | Jun 28 2025 |
| VISTA-4B | inclusionAI | apache-2.0 | 4.5B | 231 | 9 | Jun 12 2026 |
| SingGuard-NSFA-2B | inclusionAI | apache-2.0 | 2.7B | 225 | 3 | Jul 10 2026 |
| SingGuard-NSFA-9B | inclusionAI | apache-2.0 | 9.4B | 223 | 6 | Jul 10 2026 |
| c4ai-command-r-08-2024 | Cohere | cc-by-nc-4.0 | 32.3B | 203 | 174 | Aug 19 2024 |
| BFS-Prover-V2-32B | ByteDance | apache-2.0 | 32.8B | 196 | 12 | Sep 30 2025 |
| Llama-4-Maverick-17B-128E | Meta | llama4 | 401.6B | 189 | 105 | Apr 02 2025 |
| ERNIE-4.5-300B-A47B-W4A8C8-TP4-Paddle | Baidu | apache-2.0 | n/a | 181 | 16 | Jun 28 2025 |
| cudaLLM-8B | ByteDance | apache-2.0 | 8.2B | 181 | 29 | Aug 03 2025 |
| AHN-Mamba2-for-Qwen-2.5-Instruct-7B | ByteDance | apache-2.0 | 18.6M | 177 | 6 | Oct 08 2025 |
| c4ai-command-r-plus-08-2024 | Cohere | cc-by-nc-4.0 | 103.8B | 176 | 305 | Aug 21 2024 |
| AHN-Mamba2-for-Qwen-2.5-Instruct-14B | ByteDance | apache-2.0 | 51.4M | 174 | 16 | Oct 08 2025 |
| AHN-GDN-for-Qwen-2.5-Instruct-3B | ByteDance | qwen-research | 13M | 173 | 1 | Oct 08 2025 |
| smollm-360M-instruct-add-basics | Hugging Face | apache-2.0 | 361.8M | 170 | 5 | Aug 13 2024 |
| ERNIE-4.5-300B-A47B-2Bits-TP2-Paddle | Baidu | apache-2.0 | n/a | 169 | 11 | Jul 08 2025 |
| AHN-DN-for-Qwen-2.5-Instruct-7B | ByteDance | apache-2.0 | 18.5M | 169 | 1 | Oct 08 2025 |
| AHN-Mamba2-for-Qwen-2.5-Instruct-3B | ByteDance | qwen-research | 11.9M | 167 | 9 | Oct 08 2025 |
| AHN-GDN-for-Qwen-2.5-Instruct-7B | ByteDance | apache-2.0 | 21.3M | 165 | 1 | Oct 08 2025 |
| AHN-GDN-for-Qwen-2.5-Instruct-14B | ByteDance | apache-2.0 | 61M | 157 | 6 | Oct 08 2025 |
| AHN-DN-for-Qwen-2.5-Instruct-14B | ByteDance | apache-2.0 | 51.1M | 154 | 3 | Oct 08 2025 |
| AHN-DN-for-Qwen-2.5-Instruct-3B | ByteDance | qwen-research | 11.8M | 153 | 1 | Oct 08 2025 |
| Llama-2-7b | Meta | llama2 | n/a | 150 | 4,585 | Jul 09 2023 |
| aya-vision-32b | Cohere | cc-by-nc-4.0 | 33.1B | 138 | 227 | Mar 02 2025 |
| CodeLlama-7b-hf | Meta | llama2 | 6.7B | 123 | 129 | Mar 13 2024 |
| ERNIE-4.5-300B-A47B-2Bits-TP4-Paddle | Baidu | apache-2.0 | n/a | 122 | 5 | Jul 08 2025 |
| UI-Venus-Navi-7B | inclusionAI | apache-2.0 | 8.3B | 120 | 12 | Aug 16 2025 |
| Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8 | Meta | llama3.2 | n/a | 99 | 43 | Oct 23 2024 |
| tiny-aya-en-thinker | Cohere | cc-by-nc-4.0 | 3.4B | 88 | 12 | Sep 02 2026 |
| Cola-DLM | ByteDance | apache-2.0 | n/a | 86 | 51 | May 15 2026 |
| Llama-3.2-3B-Instruct-SpinQuant_INT4_EO8 | Meta | llama3.2 | n/a | 71 | 40 | Oct 23 2024 |
| command-a-translate-08-2025 | Cohere | cc-by-nc-4.0 | 111.1B | 60 | 79 | Aug 27 2025 |
| CodeLlama-70b-Instruct-hf | Meta | llama2 | 69B | 50 | 24 | Mar 13 2024 |
| SmolVLM-Instruct-DPO | Hugging Face | apache-2.0 | n/a | 46 | 22 | Nov 26 2024 |
| CodeLlama-13b-hf | Meta | llama2 | 13B | 41 | 22 | Mar 13 2024 |
| Llama-3.2-3B-Instruct-QLORA_INT4_EO8 | Meta | llama3.2 | n/a | 41 | 77 | Oct 23 2024 |
| CodeLlama-7b-Python-hf | Meta | llama2 | 6.7B | 37 | 28 | Mar 13 2024 |
| Llama-3.2-90B-Vision | Meta | llama3.2 | 88.6B | 35 | 135 | Sep 19 2024 |
| MolmoWeb-8B-Native | Ai2 | apache-2.0 | n/a | 27 | 9 | Mar 24 2026 |
| MolmoWeb-Pretrained-8B | Ai2 | apache-2.0 | n/a | 15 | 4 | Apr 09 2026 |
| MolmoWeb-4B-Native | Ai2 | apache-2.0 | n/a | 14 | 9 | Mar 23 2026 |
| MolmoWeb-Pretrained-4B | Ai2 | apache-2.0 | n/a | 14 | 2 | Apr 08 2026 |
| tiny-aya-base-32K | Cohere | cc-by-nc-4.0 | 3.4B | 13 | 5 | Sep 08 2026 |
| GELab-Zero-4B-preview-Sico-Evolution | Microsoft | apache-2.0 | 4.4B | 0 | 57 | Jun 30 2026 |
| Laguna-S-2.1-base | Poolside | openmdw-1.1 | n/a | 0 | 8 | Jul 20 2026 |
| Ming-Lite-Uni | inclusionAI | mit | 1.6B | 0 | 28 | May 02 2025 |

## Method

- Scope. Up to 40 of the most downloaded text-generation and image-text-to-text models from each of 24 labs on the Hugging Face Hub. The lab list is curated, so this is not a full census of open models.
- Original releases only. Repositories that are quantized copies or format conversions, such as GGUF and MLX builds, are filtered out by name. A fine-tune published by a listed lab is included.
- Downloads. Hugging Face reports downloads over the last 30 days. It counts automated pulls, such as CI runs, as well as people, so small models often rank high.
- License. The value is the one declared on the model card, and a custom license appears under its own name. Read the license itself before you ship: several restrict commercial use or large deployments.
- Parameter count comes from the model's safetensors metadata and is blank when the repository does not publish it.
- Generation. A script reads the Hugging Face API and stops if it finds fewer models than expected.

## Questions

### What is an open-weight model?

A model whose trained weights can be downloaded and run on your own hardware. Open weight does not always mean open source, because many licenses restrict how the model can be used and the training data is rarely published.

### Can I use these models commercially?

It depends on the license, which this dataset lists for each model. Apache 2.0 and MIT allow commercial use. Other licenses set conditions, so read the terms for the specific model.

### Why do some small models have the most downloads?

Download counts include automated use, such as test suites and library defaults. Treat them as a measure of how widely a model is pulled, not of quality.

## Sources

1. [Hugging Face Hub API](https://huggingface.co/api/models)

## Cite this dataset

AI for Devs. "Open-weight LLMs." Published November 10, 2026. https://aifordevs.org/data/open-weight-models

- Page: https://aifordevs.org/data/open-weight-models
- Markdown: https://aifordevs.org/data/open-weight-models.md
- Reuse: Free to reuse under CC BY 4.0 with a link to this page. License: https://creativecommons.org/licenses/by/4.0/
- Publisher: AI for Devs
- Published: 2026-11-10

---
title: "GPU rental prices"
type: "dataset"
url: "https://aifordevs.org/data/gpu-prices"
markdown: "https://aifordevs.org/data/gpu-prices.md"
publisher: "AI for Devs"
license: "https://creativecommons.org/licenses/by/4.0/"
attribution: "Credit AI for Devs and link to https://aifordevs.org/data/gpu-prices"
description: "On-demand and spot prices per GPU-hour for 19 GPU types across Runpod, CoreWeave, Lambda, Crusoe, Nebius, Jarvislabs, and Replicate, read from each cloud's own pricing page. Checked November 10, 2026."
checked: "2026-11-10"
csv: "https://aifordevs.org/data/gpu-prices.csv"
json: "https://aifordevs.org/data/gpu-prices.json"
rows: 62
---

# GPU rental prices

On-demand and spot prices per GPU-hour for 19 GPU types across Runpod, CoreWeave, Lambda, Crusoe, Nebius, Jarvislabs, and Replicate, read from each cloud's own pricing page. Checked November 10, 2026.

Download: [CSV](https://aifordevs.org/data/gpu-prices.csv) | [JSON](https://aifordevs.org/data/gpu-prices.json)

## Key findings

- As of November 10, 2026, H100 on-demand rates run from $2.89 (Runpod) to $6.155 (CoreWeave) per GPU-hour.
- H200 on-demand rates run from $4.59 (Jarvislabs) to $6.305 (CoreWeave) per GPU-hour, and B200 on-demand rates run from $7.99 (Runpod and Lambda) to $8.60 (CoreWeave) per GPU-hour.
- Spot rates are a median of 48.8 percent below on-demand across the 14 listings with both prices.
- Crusoe has published increases for 4 GPU types, effective December 1, 2026.
- Lambda lists new prices that take effect on November 16, 2026.

## The data

US dollars per GPU-hour, highest first, checked November 10, 2026. Filter by GPU or cloud, and sort any column.

| GPU | Details | On-demand | Spot | VRAM GB |
| --- | --- | --- | --- | --- |
| NVIDIA GB200 NVL72 | CoreWeave; Sold in 4-GPU instances | $10.50 | n/a | 186 |
| NVIDIA B200 | Crusoe; Dedicated endpoint; $10.2 from December 1, 2026 | $9.65 | n/a | 180 |
| NVIDIA HGX B300 | Nebius | $9.50 | n/a | n/a |
| B300 | Runpod | $8.99 | n/a | 288 |
| NVIDIA HGX B200 | CoreWeave; Sold in 8-GPU instances | $8.60 | $4.2637 | 180 |
| NVIDIA HGX B200 | Nebius | $8.50 | n/a | n/a |
| B200 | Runpod | $7.99 | n/a | 180 |
| NVIDIA B200 SXM6 | Lambda; Takes effect November 16, 2026 | $7.99 | n/a | 180 |
| AMD MI355X | Crusoe; Dedicated endpoint; $8.2 from December 1, 2026 | $7.70 | n/a | 288 |
| NVIDIA GH200 | CoreWeave | $6.50 | n/a | 96 |
| NVIDIA HGX H200 | CoreWeave; Sold in 8-GPU instances | $6.305 | $2.6162 | 141 |
| NVIDIA HGX H100 | CoreWeave; Sold in 8-GPU instances | $6.155 | $2.4638 | 80 |
| NVIDIA H200 | Crusoe; Dedicated endpoint; $7.50 from December 1, 2026 | $6 | n/a | 141 |
| NVIDIA H100 | Crusoe; Dedicated endpoint; $6.30 from December 1, 2026 | $5.50 | n/a | 80 |
| NVIDIA H100 | Replicate | $5.49 | n/a | 80 |
| NVIDIA H200 | Replicate | $5.49 | n/a | n/a |
| NVIDIA HGX H200 | Nebius | $5.40 | n/a | n/a |
| H200 | Runpod | $5.29 | n/a | 141 |
| NVIDIA A100 (80GB) | Replicate | $5.04 | n/a | 80 |
| NVIDIA H100 SXM | Lambda; Takes effect November 16, 2026 | $4.89 | n/a | 80 |
| NVIDIA H200 SXM | Jarvislabs | $4.59 | $2.29 | 141 |
| NVIDIA HGX H100 | Nebius | $4.50 | n/a | n/a |
| H100 SXM | Runpod | $3.99 | n/a | 80 |
| NVIDIA H100 PCIe | Lambda; Takes effect November 16, 2026 | $3.79 | n/a | 80 |
| NVIDIA L40S | Replicate | $3.51 | n/a | 48 |
| NVIDIA H100 SXM | Jarvislabs | $3.49 | $1.54 | 80 |
| H100 NVL | Runpod | $3.19 | n/a | 94 |
| H100 PCIe | Runpod | $2.89 | n/a | 80 |
| NVIDIA A100 | CoreWeave; Sold in 8-GPU instances | $2.70 | $1.2063 | 80 |
| NVIDIA GH200 | Lambda; Takes effect November 16, 2026 | $2.59 | n/a | 96 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition (High Memory) | CoreWeave; Sold in 8-GPU instances | $2.50 | $1.3862 | 96 |
| RTX Pro 6000 | Runpod | $2.49 | n/a | 96 |
| NVIDIA L40S | CoreWeave; Sold in 8-GPU instances | $2.25 | $0.985 | 48 |
| NVIDIA RTX Pro 6000 Blackwell | Jarvislabs | $2.19 | $1.15 | 96 |
| NVIDIA L40S with AMD CPU | Nebius | $1.82 | n/a | n/a |
| NVIDIA RTX PRO 6000 | Nebius | $1.80 | n/a | n/a |
| A100 PCIe | Runpod | $1.79 | n/a | 80 |
| A100 SXM | Runpod | $1.79 | n/a | 80 |
| NVIDIA L40S with Intel CPU | Nebius | $1.55 | n/a | n/a |
| NVIDIA A100 80GB | Jarvislabs | $1.49 | $0.89 | 80 |
| Pro 6000 MIG 48GB | Runpod | $1.29 | n/a | 48 |
| NVIDIA A10 | Lambda; Takes effect November 16, 2026 | $1.29 | n/a | 24 |
| NVIDIA L40 | CoreWeave; Sold in 8-GPU instances | $1.25 | $0.7837 | 48 |
| RTX 5090 | Runpod | $1.19 | n/a | 32 |
| L40S | Runpod | $1.09 | n/a | 48 |
| RTX 6000 Ada | Runpod | $0.99 | n/a | 48 |
| RTX 4090 | Runpod | $0.89 | n/a | 24 |
| NVIDIA A100 40GB | Jarvislabs | $0.89 | $0.79 | 40 |
| L40 | Runpod | $0.82 | n/a | 48 |
| NVIDIA T4 | Replicate | $0.81 | n/a | 16 |
| Pro 6000 MIG 24GB | Runpod | $0.69 | n/a | 24 |
| A40 | Runpod | $0.59 | n/a | 48 |
| RTX A6000 | Runpod | $0.59 | n/a | 48 |
| L4 | Runpod | $0.59 | n/a | 24 |
| RTX 3090 | Runpod | $0.50 | n/a | 24 |
| NVIDIA L4 | Jarvislabs | $0.44 | $0.29 | 24 |
| NVIDIA A30 | Jarvislabs | $0.41 | $0.29 | 24 |
| RTX A5000 | Runpod | $0.27 | n/a | 24 |
| NVIDIA GB300 NVL72 | CoreWeave; Sold in 4-GPU instances | n/a | n/a | 279 |
| NVIDIA HGX B300 | CoreWeave; Sold in 8-GPU instances | n/a | $4.48 | 270 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition (Standard Memory) | CoreWeave; Sold in 8-GPU instances | n/a | $1.195 | 96 |
| NVIDIA B300 | Crusoe; Dedicated endpoint; $13.80 from December 1, 2026 | n/a | n/a | 288 |

## Method

- Scope. Published, self-serve GPU prices from the clouds listed under Sources. Many clouds sell only through sales contact and are not included, so this is a sample of the market and not a full survey.
- Per GPU-hour. CoreWeave prices whole instances, so its rates are divided by the GPU count of the instance. Other sellers publish a per-GPU rate.
- Offering. Most rows are on-demand instances. Crusoe's rows are per-GPU rates for dedicated model endpoints, which include more than bare hardware, and are excluded from the findings and the chart.
- Spot. Spot prices can change at any time and instances can be reclaimed. Nebius publishes a from-price for preemptible instances, which is not included.
- Scheduled changes. Crusoe has published prices that apply from a later date, shown beside the current rate. Lambda's page lists prices that take effect on a future date, and those are the rates shown.
- Generation. A script reads each page, respects its robots.txt, and stops if a table changes shape. Each refresh is reviewed before it ships.

## Questions

### How much does an H100 cost per hour?

As of November 10, 2026, H100 on-demand rates run from $2.89 (Runpod) to $6.155 (CoreWeave) per GPU-hour among the clouds listed here. Prices depend on the interconnect, the region, and the contract length.

### Is spot pricing worth it?

Spot instances cost less but can be reclaimed at short notice. They suit batch jobs and training that checkpoints often, and they are a poor fit for serving live traffic.

### Why do prices for the same GPU differ so much?

Sellers differ in VRAM, interconnect, CPU and memory allocation, included storage, and whether capacity is shared or dedicated, so compare the full configuration before you choose.

## Sources

1. [Runpod pricing](https://www.runpod.io/pricing)
2. [CoreWeave pricing](https://www.coreweave.com/pricing)
3. [Lambda pricing](https://lambda.com/pricing)
4. [Crusoe pricing](https://crusoe.ai/cloud/pricing)
5. [Nebius pricing](https://nebius.com/prices)
6. [Jarvislabs pricing](https://jarvislabs.ai/pricing)
7. [Replicate pricing](https://replicate.com/pricing)

## Cite this dataset

AI for Devs. "GPU rental prices." Published November 10, 2026. https://aifordevs.org/data/gpu-prices

- Page: https://aifordevs.org/data/gpu-prices
- Markdown: https://aifordevs.org/data/gpu-prices.md
- Reuse: Free to reuse under CC BY 4.0 with a link to this page. License: https://creativecommons.org/licenses/by/4.0/
- Publisher: AI for Devs
- Published: 2026-11-10

---
title: "Best LLM for coding: November 2026 benchmarks, compared"
type: "deep dive"
url: "https://aifordevs.org/articles/best-llm-for-coding"
markdown: "https://aifordevs.org/articles/best-llm-for-coding.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/best-llm-for-coding"
description: "The best LLM for coding as of November 2026, by situation: agents, value, large repos, local hardware. Dated scores and who ran each benchmark."
author: "AI for Devs"
published: "2026-11-08"
level: "Intermediate"
topics: ["coding-agents", "models"]
---

# Best LLM for coding: November 2026 benchmarks, compared

The best LLM for coding as of November 2026, by situation: agents, value, large repos, local hardware. Dated scores and who ran each benchmark.

> **TL;DR**
>
> - As of November 8, 2026, the best LLM for coding agents is Claude Opus 5.5 in Claude Code, at 64.85% on the third-party Terminal-Bench 4.0 board. Its lead over Sonnet 5.5 (61.82%) falls inside the 95% intervals.
> - GPT-6.1 Sol is the best value: 58.18% on the same board at $1.92 per trial, against $14.30 per trial for Opus 5.5.
> - For prompts near 1M tokens, Claude Sonnet 5.5 is the default. An uncached 800K-token turn costs $1.65 at Anthropic's flat rate, about half of GPT-6.1 Sol's $3.28.
> - The best local LLM for coding on 24 to 64 GB is Qwen3.8-27B, at about 16 GiB of 4-bit weights plus 64 KiB per token of context. Every coding score for it is Qwen's own.
> - This article ran no benchmark. Every figure is dated, labeled third-party or vendor-reported, and linked to the page that published it.

Picking the best LLM for coding means reading boards that disagree: Gemini 3.8 Flash resolves 94.86% of public [SWE-Bench Pro V2](https://labs.scale.com/leaderboard/swe_bench_pro_public_v2) tasks and 19.09% of [Terminal-Bench 4.0](https://www.tbench.ai/) tasks. The two benchmarks use different tasks, and SWE-Bench Pro's public split comes from repositories a model could have trained on. Each pick below names the board behind it, who runs that board, and the evidence that would overturn it.

## Best LLM for coding by situation

The best LLM for coding depends on four things you can state in advance: unattended agent runs, cost per solved task, repository size per prompt, and your hardware's memory. The table gives one pick for each case, dated November 8, 2026.

| Situation | Pick | Evidence and class | What would change the pick |
| --- | --- | --- | --- |
| Agents that run unattended | Claude Opus 5.5 in Claude Code | Third-party: Terminal-Bench 4.0, 64.85% ±3.11 (row dated Sep 22, 2026); Coding Agent Index 62 | A rival within 2 points at a lower cost per solved task |
| Best value for daily agent work | GPT-6.1 Sol in Codex | Third-party: Terminal-Bench 4.0, 58.18% ±3.11, $1.92 per trial (Sep 29, 2026) | A price rise, or a DeepSeek V4.1 Flash row on Terminal-Bench 4.0 |
| Prompts near 1M tokens | Claude Sonnet 5.5 | Third-party: SWE Atlas QnA 71.8 ±8.0 (Nov 2026). Vendor price: flat across the window | OpenAI dropping its 272K surcharge, or tighter intervals that separate DeepSeek V4.1 Flash |
| Local, 24 to 64 GB | Qwen3.8-27B | Vendor only: 61.7 on SWE-bench Pro, reported by Qwen (Aug 2026) | Any third-party board row for it |
| Local, 128 GB or more | Qwen3.8-27B, or Laguna S 2.1 for a 1M window | Vendor only: 59.4 on SWE-Bench Pro public, reported by Poolside (Jul 21, 2026) | A third-party comparison of the two |

The agentic pick is narrow. Opus 5.5 at 64.85% and Sonnet 5.5 at 61.82% have overlapping intervals on Terminal-Bench 4.0, so a rerun could reverse their order. Sonnet 5.5 at max effort also cost more per trial on that board, $22.22 against $14.30, which is why Opus 5.5 keeps the pick despite a higher list price.

Start Opus 5.5 at high effort instead of max. On Cursor's own [CursorBench 4.0](https://cursor.com/en-US/cursorbench), Opus 5.5 scored 56.0% at high effort for $3.97 per task and 57.8% at max for $13.43. Cursor notes that small differences may not be statistically meaningful, and its board is vendor-run, so test the same trade-off on your repository.

## How this comparison was built

This article ran no benchmark of its own. Every score below was read on November 8, 2026, from the page that publishes it, and each is labeled third-party or vendor-reported. The table lists what each benchmark measures, who runs it, and the commercial interests attached to that operator.

| Benchmark | What it measures | Run by | Last update | Independence check |
| --- | --- | --- | --- | --- |
| [Terminal-Bench 4.0](https://www.tbench.ai/) | 66 command-line tasks, 5 trials each | Hosted by Stanford, Harbor, and Laude Institute | Oct 23, 2026 | Modal, Anthropic, OpenAI, and Google [sponsor compute](https://raw.githubusercontent.com/harbor-framework/terminal-bench/main/README.md). The board does not say who ran each row |
| [Coding Agent Index v2.0](https://artificialanalysis.ai/methodology/coding-agents-benchmarking) | Equal-weight mean of FrontierCode v1.1, Terminal-Bench 4.0, and regraded SWE-Atlas QnA | Artificial Analysis | Version since October 2026 | FrontierCode is built by [Cognition](https://cognition.com/), which sells the Devin agent |
| [SWE-Bench Pro V2](https://labs.scale.com/leaderboard/swe_bench_pro_public_v2) | 642 public and 272 private repository tasks | Scale AI | V2 released Sep 22, 2026; newer rows added since | Meta holds a [minority stake in Scale](https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolution), and its Muse models appear on the board. V2 was co-developed with [Reflection](https://reflection.ai/), which trains models |
| [SWE Atlas](https://labs.scale.com/leaderboard/swe-atlas?tab=qna) | Codebase questions, test writing, and refactoring on 15 repositories | Scale AI | November 2026 | Same ownership note as SWE-Bench Pro V2 |
| [SWE-bench Verified](https://www.vals.ai/benchmarks/swebench) | 500 human-validated issue fixes | Vals AI | Sep 1, 2026, archived | Vals [calls itself independent](https://www.vals.ai/about); its about page does not state its funding |
| [DeepSWE v1.1](https://deepswe.datacurve.ai/) | 113 tasks written from scratch | Datacurve | Sep 22, 2026 | Datacurve built the benchmark and [compares it favorably](https://arxiv.org/abs/2607.07946) with SWE-Bench Pro |

Vendor numbers come from launch posts and model cards, and each names the company that reported it. OpenAI's site refused automated reading on November 8 with an HTTP 403, so OpenAI-reported scores here are quoted from Anthropic's footnotes and labeled that way.

Third-party means the result appears on a board its vendor does not operate. It does not prove the vendor had no hand in the run, since Terminal-Bench's repository shows anyone running evaluations with its CLI.

Treat any gap smaller than a board's interval as a tie. Terminal-Bench 4.0 publishes 95% intervals, Scale publishes a ± value on most rows, and Artificial Analysis publishes whole numbers. Scores from different benchmarks are never comparable, even when the percentages look close.

## SWE-bench, Terminal-Bench, and agent index results

On third-party boards, Claude Opus 5.5 in Claude Code leads Terminal-Bench 4.0 and the Coding Agent Index, while GPT-6.1 Sol in Codex matches the pricier GPT-6 Astra on Terminal-Bench. The leading gaps are small, and the public SWE-Bench Pro V2 set has stopped separating top models at all.

All values were read on November 8, 2026. "Not listed" means no row on the pages read.

Terminal-Bench rows are max effort unless noted, and ± there is the board's 95% interval, with the row's own date. SWE Atlas does not state its interval level.

| Third-party results | Terminal-Bench 4.0 | Coding Agent Index v2.0 | Index cost per task | SWE Atlas QnA (Nov 2026) |
| --- | --- | --- | --- | --- |
| Claude Opus 5.5, Claude Code | 64.85 ±3.11 (Sep 22) | 62 | $11.42 | 77.4 ±7.5 |
| Claude Sonnet 5.5, Claude Code | 61.82 ±2.94 (Sep 28) | 59 | $13.30 | 71.8 ±8.0 |
| Claude Fable 5.1, Claude Code | 57.88 ±3.76 (Sep 1) | 58 | $11.53 | 73.4 ±7.9 |
| GPT-6 Astra, Codex | 58.18 ±2.79 (Sep 3) | 55 | $7.04 | 70.2 ±8.2 |
| GPT-6.1 Sol, Codex | 58.18 ±3.11 (Sep 29) | 54 | $1.50 | 71.0 ±8.1 |
| GLM-5.3 | 41.82 ±3.23, Claude Code (Aug 14) | 49, Opencode | $5.23 | Not listed |
| DeepSeek V4.1 Flash, Codex | Not listed | 47 | $0.39 | 71.0 ±8.1 |
| Gemini 3.8 Flash, mini-SWE-agent | 19.09 ±3.36, high effort (Sep 2) | Not listed | Not listed | 45.2 ±8.9 |

The index and cost columns come from the Artificial Analysis [Claude Code and Codex](https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-codex) and [Claude Code and Opencode](https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-opencode) tables. Artificial Analysis does not date individual rows.

SWE-Bench Pro V2's full set no longer ranks the frontier. Scale's [leaderboard](https://labs.scale.com/leaderboard/swe_bench_pro_public_v2) shows 14 of 18 rows at 94.86% or higher, with Opus 5 on top at 99.40 ±0.40. The separate 51-task HARD tab spreads those models from 98.00 down to 25.50, so read that tab or the private split instead.

## Vendor-reported numbers against the boards

Treat vendor launch numbers as claims to verify, because the settings differ even when the benchmark matches. Anthropic reports 70.6% for Sonnet 5.5 on Terminal-Bench 4.0 without naming an effort level, while the third-party board's only Sonnet 5.5 row, at max effort, reads 61.82%. OpenAI's 57.9% for GPT-6 Astra matches its board row.

| Vendor-reported figure | Reported by | Setting disclosed | Board value for comparison |
| --- | --- | --- | --- |
| Opus 5.5, Terminal-Bench 4.0: 66.4 (standard error 2.6) | [Anthropic](https://www.anthropic.com/claude-opus-5-5), Sep 22, 2026 | xhigh effort | 64.85, max effort |
| Sonnet 5.5, Terminal-Bench 4.0: 70.6 | [Anthropic](https://www.anthropic.com/claude-sonnet-5-5), Sep 28, 2026 | Not stated | 61.82, max effort |
| GPT-6 Astra, Terminal-Bench 4.0: 57.9 | OpenAI, quoted by Anthropic | high effort | 57.88, high effort |
| Qwen3.8-27B, SWE-bench Pro: 61.7 | [Qwen](https://huggingface.co/Qwen/Qwen3.8-27B), Aug 2026 | Claude Code runner, 256K window | None listed |
| Qwen3.8-27B, Terminal-Bench 2.1: 73.0 | Qwen, Aug 2026 | Terminus runner | None listed |
| Laguna S 2.1, SWE-Bench Pro public: 59.4 | [Poolside](https://huggingface.co/poolside/Laguna-S-2.1), Jul 21, 2026 | Not stated | None listed |
| DeepSeek V4.1 Flash, DeepSWE v1.1: 74.2 | [DeepSeek](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash), read Nov 8, 2026 | mini-SWE runner, max effort | None listed |

Anthropic's launch post says its safeguards handed some blocked tasks to Claude Opus 4.8 or Claude Opus 5, and that this likely lowered the Opus 5.5 scores. A safety fallback that changes which model completed a task is a product behavior your own eval should capture.

## What benchmarks cannot tell you

Benchmarks cannot tell you how a model behaves on your repository. They measure a bundle of model, agent runner, effort setting, and test suite, on tasks the model may have seen during training. Contamination, flawed verifiers, runner choice, and effort level each move scores by more than the gaps between leading models.

### Contamination

Scale reported no successful retrieval of solutions during its locked V2 runs, yet [private-split scores](https://labs.scale.com/blog/swe-bench-pro-v2) fell 17 to 22 points below public ones. Scale attributes that gap to exposure during training, which a locked test cannot undo, and treats only the private set as a clean measurement.

**Figure 1.** Every model Scale reported on both SWE-Bench Pro V2 splits loses 17 to 22 points on private code, so the public split overstates what a model can do on code it has not seen.

The private split only exists for five models in Scale's post, and the post does not cover Claude Opus 5.5 or GPT-6.1 Sol. Treat the figure as a measure of contamination, not as a current ranking.

### Verifiers and saturation

[SWE-bench](https://aifordevs.org/glossary#swe-bench) Verified has saturated. [Vals AI](https://www.vals.ai/benchmarks/swebench) archived it with Claude Opus 5 at 97.00% and no longer runs new models on it.

Test quality is a separate problem. [Aleithan et al.](https://arxiv.org/abs/2410.06992) found in 2024 that 32.67% of passing SWE-bench patches had the fix in the issue text, and 31.08% passed tests too weak to verify them.

Datacurve's [DeepSWE paper](https://arxiv.org/abs/2607.07946) reports an LLM judge disagreeing with SWE-Bench Pro's verifiers on 32.4% of 789 audited runs, against 1.4% of 735 on DeepSWE. Datacurve built DeepSWE, the judge was GPT-5.5, and the authors say the figure measures disagreement rather than ground truth. The same problem applies to any judge you build, as [LLM-as-a-judge](https://aifordevs.org/articles/llm-as-a-judge) explains.

### Agent runner and effort setting

The agent runner moves scores by several points. On Vals AI's [SWE-bench Verified run](https://www.vals.ai/benchmarks/swebench), GPT 5.5 scored 82.6% in mini-SWE-agent, 76.4% in Codex, and 76.2% in Factory. Claude Opus 4.8 scored 88.6% in mini-SWE-agent and 85.8% in Claude Code.

Effort moves scores further. Terminal-Bench 4.0 shows Opus 5 from 34.85% at low effort to 53.94% at xhigh.

**Figure 2.** One model can span 19 points by effort setting and 6 points by agent runner, more than the gaps between the top models, so any comparison must fix both.

Scale's [V2 study](https://labs.scale.com/blog/swe-bench-pro-v2) found the opposite for its strongest open-weight models. Mini-swe-agent, Pi, and OpenCode did not change the outcome on an equal budget, and Scale recommends picking a runner on token cost. Inkling did worse under Pi, so that finding does not cover every model.

### Model versus coding agent product

The best AI for coding is usually a product, and no board above measures a model alone. Artificial Analysis states it directly: "Each public row is an agent variant, not a model." The [agent loop](https://aifordevs.org/glossary#agent-loop), tool set, and defaults are all part of the score you see.

Anthropic's Sonnet 5.5 post gives a concrete case. It says Sonnet 5.5 scored lower on FrontierCode at max effort than at xhigh. At max effort it more often ran Claude Code's code-review skill, which in two cases Cognition examined caused a timeout or out-of-scope edits.

To learn how a model behaves on your code, [build your own eval](https://aifordevs.org/articles/llm-evaluation) instead of trusting anyone's average.

## The best AI model for coding on a budget

GPT-6.1 Sol is the best AI model for coding when cost per solved task matters most. On the Terminal-Bench 4.0 board it solved 192 of 330 trials for $634 in total, or $3.30 per solved trial, against $22.05 for Claude Opus 5.5 at max effort.

List prices below are per million tokens at standard tier, read November 8, 2026, from each provider's own page: [Anthropic](https://platform.claude.com/docs/en/about-claude/pricing), [OpenAI](https://developers.openai.com/api/docs/pricing), [Google](https://aistudio.google.com/docs/pricing), and [DeepSeek](https://api-docs.deepseek.com/quick_start/pricing/). The [Claude API pricing guide](https://aifordevs.org/articles/claude-api-pricing) and [Gemini API pricing guide](https://aifordevs.org/articles/gemini-api-pricing) go deeper on each.

| Model (vendor's list price) | Input | Cached input | Output | Long-prompt rule |
| --- | --- | --- | --- | --- |
| Claude Opus 5.5 | $4.00 | $0.20 | $20.00 | Same rate to 1M tokens |
| Claude Sonnet 5.5 | $2.00 | $0.10 | $10.00 | Same rate to 1M tokens |
| GPT-6.1 Sol | $2.00 | $0.10 | $10.00 | Over 272K input: 2x input and cache, 1.5x output |
| GPT-6.1 Terra | $1.00 | $0.05 | $5.00 | Same 272K rule |
| GPT-6 Astra | $10.00 | $1.00 | $50.00 | Same 272K rule |
| Gemini 3.8 Flash | $0.75 to Dec 31, 2026, then $1.50 | $0.075, then $0.15 | $3.75, then $7.50 | One listed rate |
| Gemini 4 Argon | $2.00 to Nov 30, 2026, then $4.00 | $0.10, then $0.20 | $10.00, then $20.00 | One listed rate |
| DeepSeek V4.1 Flash | $0.15 off-peak, $0.30 peak | $0.003, $0.006 | $0.60, $1.20 | One listed rate |

Sol and Sonnet 5.5 share a list price, so the cost gap between them in the board runs comes from how many tokens each run used. The Coding Agent Index shows the same order for Sonnet 5.5 and Opus 5.5, $13.30 against $11.42 per task.

**Figure 3.** Up to $1.50 per task, 23 index points cost $1.34; the next 8 points to Opus 5.5 cost $9.92, so stop at GPT-6.1 Sol until you measure the difference on your own work.

On Artificial Analysis's [Coding Agent Index](https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-codex), each point above GPT-6.1 Sol costs about $1.24 per task, against about six cents per point below it.

A third-party Terminal-Bench 4.0 row for DeepSeek V4.1 Flash, which costs $0.39 per index task, or a Sol price increase would change this pick.

## Best coding LLM for large repositories

Claude Sonnet 5.5 is the best coding LLM for prompts near 1M tokens. It scored 71.8 ±8.0 on [SWE Atlas QnA](https://labs.scale.com/leaderboard/swe-atlas?tab=qna), and Anthropic bills 1M-token requests at the standard rate, so an uncached 800K-token turn costs $1.65, about half of GPT-6.1 Sol's $3.28.

| Model | Context window (vendor) | SWE Atlas QnA (third-party) | 800K in, 5K out, cold | Same turn, 95% cache hits |
| --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 | 1M | 71.8 ±8.0 | $1.65 | $0.21 |
| Claude Opus 5.5 | 1M | 77.4 ±7.5 | $3.30 | $0.41 |
| GPT-6.1 Sol | 1.05M, 922K max input | 71.0 ±8.1 | $3.28 | $0.39 |
| GPT-6 Astra | 1.05M | 70.2 ±8.2 | $16.38 | $2.70 |
| Gemini 4 Argon, to Nov 30 | 1M | 65.3 ±8.5 | $1.65 | $0.21 |
| Gemini 3.8 Flash, to Dec 31 | 1,048,576 | 45.2 ±8.9 | $0.62 | $0.11 |
| DeepSeek V4.1 Flash | 1M | 71.0 ±8.1 | $0.12 to $0.25 | $0.01 to $0.02 |

Context windows come from each vendor's model pages: [Anthropic](https://platform.claude.com/docs/en/models/overview), [OpenAI](https://developers.openai.com/api/docs/models/gpt-6.1-sol), and [Google](https://aistudio.google.com/docs/models/gemini-3.8-flash).

Costs are derived from the list prices above, recomputed in code, and exclude cache-write charges. Opus 5.5 scores highest on this test, but its ±7.5 interval overlaps Sonnet 5.5's, and Sonnet 5.5 costs half as much per turn. OpenAI's [272K rule](https://developers.openai.com/api/docs/models/gpt-6.1-sol) applies to the full request, so a large prompt crosses it all at once.

DeepSeek V4.1 Flash matches Sonnet 5.5 on this test at about one-seventh to one-thirteenth the cold cost. Its [privacy policy](https://cdn.deepseek.com/policies/en-US/deepseek-privacy-policy.html) says DeepSeek stores personal data in China, so read its API terms before sending proprietary code. Its [MIT-licensed weights](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) total about 510 GB, so self-hosting needs a server.

Filling the [context window](https://aifordevs.org/glossary#context-window) is rarely the goal. [Context engineering](https://aifordevs.org/articles/context-engineering) covers keeping prompts narrow, and [prompt caching](https://aifordevs.org/glossary#prompt-caching) is what makes the last column possible.

## Best local LLM for coding, by hardware tier

The best local LLM for coding on 24 to 64 GB of memory is Qwen3.8-27B, under Apache 2.0. Plan for the GGUF file plus 64 KiB per token of context. Every coding score on [its model card](https://huggingface.co/Qwen/Qwen3.8-27B) comes from Qwen, and none of the third-party boards above lists it yet.

Estimates below add [GGUF](https://aifordevs.org/glossary#gguf) file sizes from Hugging Face to computed [KV cache](https://aifordevs.org/glossary#kv-cache) memory for the chosen context. They are estimates, not measurements: this article did not run these models. Add headroom for the operating system and your tools.

| Memory | Build ([quantization](https://aifordevs.org/glossary#quantization)) | Weights | Context | Estimated total | Use it for |
| --- | --- | --- | --- | --- | --- |
| 16 GB | Qwen3.8-27B, [UD-Q2_K_XL](https://huggingface.co/api/models/unsloth/Qwen3.8-27B-GGUF/tree/main) | 9.15 GiB | 32K | 11.15 GiB | Chat and small edits; Qwen publishes no 2-bit scores |
| 24 GB GPU | Qwen3.8-27B, UD-Q4_K_XL | 16.35 GiB | 64K | 20.35 GiB | Short agent runs |
| 32 GB unified | Qwen3.8-27B, UD-Q4_K_XL | 16.35 GiB | 128K | 24.35 GiB | Daily agent work |
| 64 GB | Qwen3.8-27B, [Q8_0](https://huggingface.co/api/models/ggml-org/Qwen3.8-27B-GGUF/tree/main) | 26.63 GiB | 256K | 42.63 GiB | Matches Qwen's 256K test window |
| 128 GB | Laguna S 2.1, [Q4_K_M](https://huggingface.co/api/models/poolside/Laguna-S-2.1-GGUF/tree/main) | 89.44 GiB | 256K | 101.51 GiB | Only if you need its 1M window |
| 256 GB and up | Laguna S 2.1, Q8_0 | 119.91 GiB | 1M | 167.98 GiB | Full 1M context |

Qwen's card gives no memory figure, so the KV cache is computed from each [configuration file](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json). Qwen3.8-27B stores keys and values for 16 full-attention layers out of 64, and its other 48 layers are linear-attention layers, assumed here to keep a fixed-size state.

[Laguna S 2.1](https://huggingface.co/poolside/Laguna-S-2.1/raw/main/config.json) keeps every token for 12 full-attention layers and only 512 tokens for its 36 sliding-window layers. The script below ran on Python 3.15.1 with no dependencies, and no model was loaded.

```python title="kv_budget.py"
GIB = 1024**3


def kv_bytes_per_token(full_layers, kv_heads, head_dim, elem_bytes=2):
    # One K tensor and one V tensor per layer that stores every token.
    return full_layers * 2 * kv_heads * head_dim * elem_bytes


def budget_gib(file_bytes, ctx_tokens, per_token_bytes, fixed_bytes=0):
    return (file_bytes + ctx_tokens * per_token_bytes + fixed_bytes) / GIB


qwen = kv_bytes_per_token(full_layers=16, kv_heads=4, head_dim=256)
laguna = kv_bytes_per_token(full_layers=12, kv_heads=8, head_dim=128)
laguna_sliding = 36 * 2 * 8 * 128 * 2 * 512  # 36 layers, 512-token window

print(f"Qwen3.8-27B: {qwen / 1024:.0f} KiB per token")
print(f"Laguna S 2.1: {laguna / 1024:.0f} KiB per token")
print(f"Qwen UD-Q4_K_XL at 128K context: {budget_gib(17_559_178_144, 131_072, qwen):.2f} GiB")
print(f"Laguna Q4_K_M at 256K context: {budget_gib(96_031_829_760, 262_144, laguna, laguna_sliding):.2f} GiB")
```

```text
Qwen3.8-27B: 64 KiB per token
Laguna S 2.1: 48 KiB per token
Qwen UD-Q4_K_XL at 128K context: 24.35 GiB
Laguna Q4_K_M at 256K context: 101.51 GiB
```

The 2 in `elem_bytes` is llama.cpp's [default 16-bit KV cache type](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp). The [KV cache explainer](https://aifordevs.org/articles/kv-cache-explained) covers why the cache grows with context, and the [inference engine comparison](https://aifordevs.org/articles/vllm-sglang-llama-cpp) covers how to serve these files.

More memory does not buy a better local model here. Poolside reports 59.4 for Laguna S 2.1 on SWE-Bench Pro public, and Qwen reports 61.7 for Qwen3.8-27B on SWE-bench Pro, each on its own test setup. Spend extra memory on context length first. 

Laguna S 2.1 activates about 8B of its 118B parameters per token, but as with any [mixture-of-experts model](https://aifordevs.org/articles/mixture-of-experts), all 89.44 GiB of Q4 weights must stay loaded.

Open weights do not always mean local. DeepSeek V4.1 Flash is about 510 GB of weights, and the [Kimi K3 repository](https://huggingface.co/moonshotai/Kimi-K3/tree/main) is 1.56 TB, so both are data-center deployments.

## Pick your best LLM for coding in one afternoon

You can settle the best LLM for coding on your own repository in one afternoon. Take 20 recent merged pull requests, strip the code changes, and run your two finalists through the same agent runner at the same effort setting. Count the patches you would merge, then divide total spend by that count.

1. Pick 20 merged pull requests that represent your work, and save each one's prompt or issue as your [golden dataset](https://aifordevs.org/glossary#golden-dataset).
2. Reset the repository to each pull request's parent commit and give both finalists the same description.
3. Run each model on each task at least three times. Terminal-Bench 4.0 runs five trials per task.
4. Review the diffs yourself, or use a calibrated LLM judge once you trust it, and record whether you would merge each one.
5. Choose the cheaper model as your default if it produces at least 85% as many mergeable patches at less than half the cost per mergeable patch, and route its failures to the stronger model.

That 85% threshold is a starting point, so adjust it to the cost of a bad merge on your team. Rerun the same set when the [evals hub](https://aifordevs.org/topics/evals) or the [news feed](https://aifordevs.org/news) reports a new model, because the boards in this article will move within months.

## Questions

### What is the best LLM for coding right now?

As of November 8, 2026, the best LLM for unattended coding agents is Claude Opus 5.5 in Claude Code. It scored 64.85% on the third-party Terminal-Bench 4.0 board and 62 on the Coding Agent Index. Its lead over Sonnet 5.5 falls inside the reported intervals, and a rerun could reverse the order.

### What is the best local LLM for coding?

Qwen3.8-27B is the best local LLM for coding on machines with 24 to 64 GB of memory. Its Apache-2.0 weights fit in about 16 GiB at 4-bit, and context adds about 64 KiB per token. Every coding score published for it comes from Qwen, and none of the boards in this article lists it yet.

### Is Claude or GPT better for coding?

Claude Opus 5.5 scores higher on the third-party Terminal-Bench 4.0 board, 64.85% against 58.18% for GPT-6.1 Sol and GPT-6 Astra. GPT-6.1 Sol cost about one seventh as much per trial on the same board, so it wins on cost per solved task. Measure both on your own repository before you commit.

### Are SWE-bench scores reliable?

SWE-bench scores on public datasets overstate what a model can do on unseen code. Vals AI archived SWE-bench Verified on September 1, 2026, after performance saturated. Scale's SWE-Bench Pro V2 shows every model it reported on both splits losing 17 to 22 points on private repositories. Use private splits and your own tasks.

### How much memory do I need to run a coding LLM locally?

Budget the GGUF file plus the KV cache for your context length. For Qwen3.8-27B at 4-bit, that is about 16 GiB of weights and 64 KiB per token, so a 128K-token window needs about 24 GiB before the operating system and your tools. A 32 GB machine is a practical floor for agent work.

### What is the cheapest good LLM for coding agents?

GPT-6.1 Sol is the cheapest model near the top of the third-party boards, at $1.92 per Terminal-Bench 4.0 trial and $1.50 per Coding Agent Index task. DeepSeek V4.1 Flash costs less, $0.39 per index task, but scores 47 on the index against 54 for Sol, and it has no Terminal-Bench 4.0 row.

## Sources

1. [Terminal-Bench 4.0 official leaderboard (Stanford, Harbor, Laude Institute)](https://www.tbench.ai/)
2. [Terminal-Bench repository README (Harbor Framework)](https://raw.githubusercontent.com/harbor-framework/terminal-bench/main/README.md)
3. [Artificial Analysis, Coding Agent Index methodology (2026)](https://artificialanalysis.ai/methodology/coding-agents-benchmarking)
4. [Artificial Analysis, Claude Code vs. Codex agent variants (2026)](https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-codex)
5. [Artificial Analysis, Claude Code vs. Opencode agent variants (2026)](https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-opencode)
6. [Scale Labs, SWE-Bench Pro V2 leaderboard (2026)](https://labs.scale.com/leaderboard/swe_bench_pro_public_v2)
7. [Scale Labs, SWE-Bench Pro V2: A Cleaner, Harder-to-Game Leaderboard (2026)](https://labs.scale.com/blog/swe-bench-pro-v2)
8. [Scale Labs, SWE Atlas leaderboard (2026)](https://labs.scale.com/leaderboard/swe-atlas?tab=qna)
9. [Scale AI, Scale AI announces next phase of company evolution (2025)](https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolution)
10. [Reflection, company site (2026)](https://reflection.ai/)
11. [Vals AI, SWE-bench Verified benchmark (updated September 1, 2026)](https://www.vals.ai/benchmarks/swebench)
12. [Vals AI, About (2026)](https://www.vals.ai/about)
13. [Datacurve, DeepSWE leaderboard v1.1 (2026)](https://deepswe.datacurve.ai/)
14. [Huang et al., DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks (2026)](https://arxiv.org/abs/2607.07946)
15. [Aleithan et al., SWE-Bench+: Enhanced Coding Benchmark for LLMs (2024)](https://arxiv.org/abs/2410.06992)
16. [Cognition, Devin and FrontierCode (2026)](https://cognition.com/)
17. [Cursor, CursorBench 4.0 (2026)](https://cursor.com/en-US/cursorbench)
18. [Anthropic, Introducing Claude Opus 5.5 (2026)](https://www.anthropic.com/claude-opus-5-5)
19. [Anthropic, Introducing Claude Sonnet 5.5 (2026)](https://www.anthropic.com/claude-sonnet-5-5)
20. [Anthropic, Pricing (read November 8, 2026)](https://platform.claude.com/docs/en/about-claude/pricing)
21. [Anthropic, Models overview (read November 8, 2026)](https://platform.claude.com/docs/en/models/overview)
22. [OpenAI, API pricing (read November 8, 2026)](https://developers.openai.com/api/docs/pricing)
23. [OpenAI, GPT-6.1 Sol model page (read November 8, 2026)](https://developers.openai.com/api/docs/models/gpt-6.1-sol)
24. [Google, Gemini API pricing (read November 8, 2026)](https://aistudio.google.com/docs/pricing)
25. [Google, Gemini 3.8 Flash model page (read November 8, 2026)](https://aistudio.google.com/docs/models/gemini-3.8-flash)
26. [DeepSeek, Models and pricing (read November 8, 2026)](https://api-docs.deepseek.com/quick_start/pricing/)
27. [DeepSeek, Privacy Policy (last updated February 10, 2026)](https://cdn.deepseek.com/policies/en-US/deepseek-privacy-policy.html)
28. [DeepSeek, DeepSeek-V4.1-Flash model card (2026)](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)
29. [Qwen Team, Qwen3.8-27B model card (2026)](https://huggingface.co/Qwen/Qwen3.8-27B)
30. [Qwen Team, Qwen3.8-27B configuration file (2026)](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json)
31. [ggml-org, Qwen3.8-27B GGUF files on Hugging Face (2026)](https://huggingface.co/api/models/ggml-org/Qwen3.8-27B-GGUF/tree/main)
32. [Unsloth, Qwen3.8-27B GGUF files on Hugging Face (2026)](https://huggingface.co/api/models/unsloth/Qwen3.8-27B-GGUF/tree/main)
33. [Poolside, Laguna S 2.1 model card (2026)](https://huggingface.co/poolside/Laguna-S-2.1)
34. [Poolside, Laguna S 2.1 configuration file (2026)](https://huggingface.co/poolside/Laguna-S-2.1/raw/main/config.json)
35. [Poolside, Laguna S 2.1 GGUF files on Hugging Face (2026)](https://huggingface.co/api/models/poolside/Laguna-S-2.1-GGUF/tree/main)
36. [Moonshot AI, Kimi K3 repository files (2026)](https://huggingface.co/moonshotai/Kimi-K3/tree/main)
37. [llama.cpp, default KV cache types in llama-context.cpp](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp)

## Cite this deep dive

AI for Devs. "Best LLM for coding: November 2026 benchmarks, compared." Published November 8, 2026. https://aifordevs.org/articles/best-llm-for-coding

- Page: https://aifordevs.org/articles/best-llm-for-coding
- Markdown: https://aifordevs.org/articles/best-llm-for-coding.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "Claude API pricing: caching, batch, and real bills"
type: "guide"
url: "https://aifordevs.org/articles/claude-api-pricing"
markdown: "https://aifordevs.org/articles/claude-api-pricing.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/claude-api-pricing"
description: "Claude API pricing for every current model as of November 8, 2026, with cache break-even math, batch stacking, tool fees, and a tested cost calculator."
author: "AI for Devs"
published: "2026-11-08"
level: "Intermediate"
topics: ["apis", "models"]
---

# Claude API pricing: caching, batch, and real bills

Claude API pricing for every current model as of November 8, 2026, with cache break-even math, batch stacking, tool fees, and a tested cost calculator.

> **TL;DR**
>
> - Claude API pricing as of November 8, 2026: Opus 5.5 costs $4 input and $20 output per million tokens, Sonnet 5.5 costs $2 and $10, and Haiku 5.5 costs $0.10 and $0.50 for prompts up to 100,000 tokens.
> - A 5-minute cache write pays for itself after 1 cache read, and a 1-hour write after 2 reads, on every current model.
> - Batch halves token prices and stacks with caching, so a batched cache read on Opus 5.5 costs $0.10 per million tokens.
> - Anthropic says models from Claude 4.7 onward count about 30% more tokens for the same text, which turns Opus 4.6 to Opus 5.5 from a 20% price cut into a rise of about 4%.
> - The TypeScript calculator in this guide ran on Node 26.12.1 with no API calls. It prices a 30-call Opus 5.5 agent session at $8.97 uncached and $1.49 with the 5-minute cache.

Claude API pricing is metered per million tokens, with separate rates for input, output, cache writes, and cache reads. Anthropic publishes those rates on its [pricing page](https://platform.claude.com/docs/en/about-claude/pricing), and every figure here was read from that page on November 8, 2026. A price sheet shows each rate, and it stops short of what a 30-call [agent loop](https://aifordevs.org/glossary#agent-loop) or a [RAG](https://aifordevs.org/glossary#rag) request costs once caching, batch, and tool fees combine. This guide works out those bills with a calculator you can run on your own `usage` objects.

## Claude API pricing by model

Claude API pricing as of November 8, 2026 runs from $0.10 input per million tokens on Haiku 5.5 to $10 on the Fable and Mythos models. Output costs five times input on every model in the table. These are first-party rates in US dollars, taken from [Anthropic's pricing page](https://platform.claude.com/docs/en/about-claude/pricing).

> **Prices last checked**
>
> Prices last checked November 8, 2026. This table is stale if Anthropic's page adds a model row, changes a cache-read footnote, or moves the 100,000-token Haiku 5.5 threshold.
>
> It also goes stale when a model retires. [Sonnet 4.5 retires November 30, 2026](https://platform.claude.com/docs/en/about-claude/model-deprecations). Opus 4.5 and Haiku 4.5 list earliest retirement dates of November 24 and October 15, 2026.

| Model | Input | 5-min write | 1-hour write | Cache read | Output | Batch input | Batch output |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Fable 5.1 | $10 | $12.50 | $20 | $0.25 | $50 | $5 | $25 |
| Mythos 5.1 (limited access) | $10 | $12.50 | $20 | $0.25 | $50 | $5 | $25 |
| Fable 5 | $10 | $12.50 | $20 | $1 | $50 | $5 | $25 |
| Mythos 5 (limited access) | $10 | $12.50 | $20 | $1 | $50 | $5 | $25 |
| Opus 5.5 | $4 | $5 | $8 | $0.20 | $20 | $2 | $10 |
| Opus 5 | $5 | $6.25 | $10 | $0.50 | $25 | $2.50 | $12.50 |
| Opus 4.8 | $5 | $6.25 | $10 | $0.50 | $25 | $2.50 | $12.50 |
| Opus 4.7 | $5 | $6.25 | $10 | $0.50 | $25 | $2.50 | $12.50 |
| Opus 4.6 | $5 | $6.25 | $10 | $0.50 | $25 | $2.50 | $12.50 |
| Opus 4.5 | $5 | $6.25 | $10 | $0.50 | $25 | $2.50 | $12.50 |
| Sonnet 5.5 | $2 | $2.50 | $4 | $0.10 | $10 | $1 | $5 |
| Sonnet 5 | $2 | $2.50 | $4 | $0.20 | $10 | $1 | $5 |
| Sonnet 4.6 | $3 | $3.75 | $6 | $0.30 | $15 | $1.50 | $7.50 |
| Sonnet 4.5 (deprecated) | $3 | $3.75 | $6 | $0.30 | $15 | $1.50 | $7.50 |
| Haiku 5.5, prompts up to 100,000 tokens | $0.10 | $0.125 | $0.20 | $0.01 | $0.50 | $0.05 | $0.25 |
| Haiku 5.5, prompts over 100,000 tokens | $0.50 | $0.625 | $1 | $0.05 | $2.50 | $0.25 | $1.25 |
| Haiku 4.5 | $1 | $1.25 | $2 | $0.10 | $5 | $0.50 | $2.50 |

All prices are USD per million tokens, the unit Anthropic calls MTok, and Anthropic API pricing and Claude API pricing refer to this same rate card. The page also lists three retired models: Opus 4.1 and Opus 4 at $15 and $75, and Sonnet 4 at $3 and $15.

Amazon Bedrock and Google-managed Claude on Google Cloud set their own prices, and their regional endpoints carry a 10% premium over global ones. Current model IDs are in [Anthropic's models overview](https://platform.claude.com/docs/en/models/overview), and the [platforms directory](https://aifordevs.org/platforms) lists other providers.

### Claude Opus pricing

Claude Opus pricing dropped with Opus 5.5, which costs $4 input and $20 output against $5 and $25 for Opus 5 through 4.5. Its cache reads cost $0.20 against $0.50, because Anthropic prices them at 0.05x of input instead of 0.1x. [Fast mode](https://platform.claude.com/docs/en/about-claude/pricing#fast-mode-pricing) is a separate premium rate of $8 and $40 on Opus 5.5, and $10 and $50 on Opus 5 and 4.8.

### Claude Sonnet pricing

Claude Sonnet pricing is $2 input and $10 output on both Sonnet 5.5 and Sonnet 5, but Sonnet 5.5 reads cache at $0.10 against $0.20. Sonnet 5's rate began as an introductory price and is now standard, per a footnote on Anthropic's page. The increase to $3 and $15 once scheduled for September 1, 2026 will not happen.

## Cache write premiums and read discounts

A cache write costs more than normal input and a cache read costs far less, so [prompt caching](https://aifordevs.org/glossary#prompt-caching) pays only when reads follow. [Anthropic's multipliers](https://platform.claude.com/docs/en/about-claude/pricing#prompt-caching) are 1.25x input for a 5-minute write and 2x for a 1-hour write. Reads cost 0.1x on most models, 0.05x on Opus 5.5 and Sonnet 5.5, and 0.025x on Fable 5.1 and Mythos 5.1.

### The break-even number of cache reads

You need 1 read to recover a 5-minute write and 2 reads to recover a 1-hour write, at every read multiplier Anthropic sells today. Compare one write plus N reads against sending the prefix uncached N + 1 times:

```text
with cache:     W + N * R
without cache:  1 + N
cache wins when N > (W - 1) / (1 - R)
```

| Read multiplier | Applies to | 5-minute write (W = 1.25) | 1-hour write (W = 2) |
| --- | --- | --- | --- |
| 0.1x | Most models | 0.28, so 1 read | 1.11, so 2 reads |
| 0.05x | Opus 5.5, Sonnet 5.5 | 0.26, so 1 read | 1.05, so 2 reads |
| 0.025x | Fable 5.1, Mythos 5.1 | 0.26, so 1 read | 1.03, so 2 reads |

Anthropic's page states the 1 and 2 for the 0.1x case. The table above shows the same integers hold at the cheaper read rates, though the 1-hour write only just needs its second read.

**Figure 1.** On Opus 5.5 rates, the 5-minute cache is cheaper than no cache from the second request, and the 1-hour cache from the third.

### Five-minute or one-hour cache

Pick the cache duration by the gap between requests to the same prefix. A hit [refreshes the entry at no extra cost](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), so traffic that arrives less than 5 minutes apart never needs the 1-hour write. Gaps of 5 to 60 minutes are where the 1-hour write earns its premium, because each 5-minute miss costs 1.25x, more than sending the prefix uncached at 1x.

The lifetime runs from the start of the request. A response that streams for 4 minutes leaves about 1 minute for the follow-up, which catches long agent turns.

**Figure 2.** Gaps of 13 and 21 minutes force the 5-minute cache to write three times for 3.90x. The 1-hour cache writes once for 2.25x, and no cache costs 6x. Read multiplier is 0.05x.

A third option is keeping a 5-minute entry warm with [`max_tokens: 0` requests](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#pre-warming-the-cache). Twelve pings across an idle hour cost 0.6x at 0.05x reads, below the 0.75x extra that a 1-hour write costs over a 5-minute write. At 0.1x reads the same pings cost 1.2x, so use the 1-hour cache on those models.

### Cache misses that return no error

Prompts below the model's minimum are [processed without caching and return no error](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#cache-limitations). The minimum is 512 tokens on Fable 5 and 5.1, Mythos 5 and 5.1, Opus 5.5 and 5, Sonnet 5.5, and Haiku 5.5. It is 1,024 on Opus 4.8, Sonnet 5, Sonnet 4.6, and Sonnet 4.5, then 2,048 on Opus 4.7, and 4,096 on Opus 4.6, Opus 4.5, and Haiku 4.5.

Any byte change in the prefix also forces a new write, and editing tool definitions invalidates the whole cache. Check that `usage.cache_read_input_tokens` is above zero, and follow the prefix ordering in this guide to [context engineering](https://aifordevs.org/articles/context-engineering). Anthropic says each entry holds [KV cache](https://aifordevs.org/articles/kv-cache-explained) representations and cryptographic hashes in memory, with no raw prompt text at rest.

## Batch discounts and how they stack

The Message Batches API takes 50% off input and output tokens, and that discount multiplies with the cache multipliers and the 1.1x US-only multiplier. A batched cache read on Opus 5.5 costs $0.10 per million tokens. [Anthropic's batch docs](https://platform.claude.com/docs/en/build-with-claude/batch-processing) say most batches finish within an hour, and requests still unsent at 24 hours expire unbilled.

| Opus 5.5 line item, per MTok | Standard | Batch | Batch and US-only |
| --- | --- | --- | --- |
| Input | $4 | $2 | $2.20 |
| 5-minute write | $5 | $2.50 | $2.75 |
| 1-hour write | $8 | $4 | $4.40 |
| Cache read | $0.20 | $0.10 | $0.11 |
| Output | $20 | $10 | $11 |

A batch holds up to 100,000 requests or 256 MB, and results stay available for 29 days. The [batch API](https://aifordevs.org/glossary#batch-api) cannot use fast mode, and the discount does not apply to Managed Agents sessions.

Cache hits are best effort because requests run concurrently, and Anthropic reports hit rates from 30% to 98%. It suggests the 1-hour cache for batches with shared context, since a batch often outlasts 5 minutes.

## Long-context pricing and the Haiku 5.5 cliff

Haiku 5.5 is the only current model with a long-context tier: a prompt over 100,000 tokens pays five times the normal rate on every token in that request, output included. [Claude 4.6 and later models](https://platform.claude.com/docs/en/about-claude/pricing#long-context-pricing) other than Haiku 5.5 bill the full 1M-token [context window](https://aifordevs.org/glossary#context-window) at standard rates.

The boundary is a cliff. With 1,000 output tokens, a 100,000-token prompt costs $0.0105 and a 100,001-token prompt costs $0.0525.

Cache reads and writes count toward the length, and each request is priced on its own, so one long turn does not reprice earlier ones. Models outside the 1M group, including Sonnet 4.5, have [200K-token windows](https://platform.claude.com/docs/en/build-with-claude/context-windows), and the pricing page lists no higher tier for them.

## Server-side tool fees and tool tokens

Web search is the only Claude tool with a per-use fee, at $10 per 1,000 searches plus tokens for the results. Web fetch is token-only, and code execution is free alongside current web tools. Every tool also adds input tokens through its definition, so tool calling has a cost on each request.

| Tool | Fee beyond tokens | Token overhead |
| --- | --- | --- |
| Web search | $10 per 1,000 searches; failed searches are free | Results count as input this turn and later |
| Web fetch | None | Fetched content counts as input |
| Code execution | Free with `web_search_20260209` or `web_fetch_20260209`; otherwise $0.05 per container hour after 1,550 free hours a month, 5-minute minimum | Standard tokens |
| Bash | None | 325 tokens on 4.7 and later, 244 earlier |
| Text editor | None | 974 tokens on 4.7 and later, 745 earlier |
| Computer use toolset | None | About 4,500 tokens with default members |
| Browser use toolset | None | About 6,600 tokens with default members |
| Any tool at all | None | 286 tokens on Opus 5.5, Sonnet 5.5, and Haiku 5.5 with `tool_choice` auto or none |
| Managed Agents session | $0.08 per session-hour while running | Standard tokens |

All figures come from [Anthropic's tool pricing section](https://platform.claude.com/docs/en/about-claude/pricing#tool-use-pricing) as of November 8, 2026. One $0.01 search equals 2,500 input tokens on Opus 5.5 and 100,000 on Haiku 5.5, so search fees matter most on cheap models. The 1,550 free hours cover 18,600 minimum-length runs a month, and a request with files bills execution time even if the tool is never called.

Tools you load from an [MCP server](https://aifordevs.org/articles/build-an-mcp-server) land in the same `tools` block, so their names, descriptions, and schemas bill on every call.

## Tokenizer differences change a Claude bill

Anthropic says Claude 4.7 and later models produce about 30% more tokens for the same text than Sonnet 4.6 and earlier, so per-token prices across that line are not comparable. Multiply the new price by 1.3 before comparing. The increase varies with content, and the [pricing page](https://platform.claude.com/docs/en/about-claude/pricing) gives no range for it.

| Migration | List price change | Effective change at 1.3x tokens |
| --- | --- | --- |
| Opus 4.6 to Opus 4.8 | 0% | +30% |
| Opus 4.6 to Opus 5.5 | -20% | +4% |
| Opus 4.8 to Opus 5.5 (same tokenizer) | -20% | -20% |
| Sonnet 4.6 to Sonnet 5.5 | -33% | -13% |
| Haiku 4.5 to Haiku 5.5, under 100,000 tokens | -90% | -87% |

The 30% is Anthropic's figure. This guide did not measure it, because no API key was available.

The newer [tokenizer](https://aifordevs.org/glossary#tokenizer) also moves limits that count tokens. [Anthropic says](https://platform.claude.com/docs/en/models/overview) 1M tokens holds about 555K words on the current tokenizer and about 750K on earlier ones.

A prompt near 100,000 tokens on Haiku 4.5 can land over the Haiku 5.5 threshold. Recount with the [token counting endpoint](https://platform.claude.com/docs/en/build-with-claude/token-counting), which is free, rate limited, and returns an estimate.

Thinking follows the same arithmetic. [Thinking tokens bill as output](https://platform.claude.com/docs/en/build-with-claude/thinking), even when the text is omitted from the response.

## A tested Claude API cost calculator

The calculator below takes token counts from the API's `usage` object and returns an itemized bill. It applies the published multipliers for cache writes, cache reads, batch, US-only inference, and the Haiku 5.5 long-context tier. It ran on Node 26.12.1 with no dependencies, and the file has no network code.

```ts title="claude-bill.ts"
// claude-bill.ts: turn token counts into a Claude API bill.
// Prices are USD per 1M tokens, read from Anthropic's pricing page on November 8, 2026.
type Rate = {
  input: number;
  output: number;
  read: number; // cache read multiplier on the input price
  longAt?: number; // prompt length above which a higher rate card applies
  longMult?: number; // multiplier on every token price above longAt
  noUsOnly?: boolean; // model rejects inference_geo
};

export const MODELS: Record<string, Rate> = {
  "claude-fable-5-1": { input: 10, output: 50, read: 0.025 },
  "claude-mythos-5-1": { input: 10, output: 50, read: 0.025 },
  "claude-fable-5": { input: 10, output: 50, read: 0.1 },
  "claude-mythos-5": { input: 10, output: 50, read: 0.1 },
  "claude-opus-5-5": { input: 4, output: 20, read: 0.05 },
  "claude-opus-5": { input: 5, output: 25, read: 0.1 },
  "claude-opus-4-8": { input: 5, output: 25, read: 0.1 },
  "claude-opus-4-7": { input: 5, output: 25, read: 0.1 },
  "claude-opus-4-6": { input: 5, output: 25, read: 0.1 },
  "claude-opus-4-5-20251101": { input: 5, output: 25, read: 0.1, noUsOnly: true },
  "claude-sonnet-5-5": { input: 2, output: 10, read: 0.05 },
  "claude-sonnet-5": { input: 2, output: 10, read: 0.1 },
  "claude-sonnet-4-6": { input: 3, output: 15, read: 0.1 },
  "claude-sonnet-4-5-20250929": { input: 3, output: 15, read: 0.1, noUsOnly: true },
  "claude-haiku-5-5": { input: 0.1, output: 0.5, read: 0.1, longAt: 100_000, longMult: 5 },
  "claude-haiku-4-5-20251001": { input: 1, output: 5, read: 0.1, noUsOnly: true },
};

export type Usage = {
  input: number; // usage.input_tokens
  output: number; // usage.output_tokens, thinking included
  cacheRead?: number; // usage.cache_read_input_tokens
  cacheWrite5m?: number; // usage.cache_creation.ephemeral_5m_input_tokens
  cacheWrite1h?: number; // usage.cache_creation.ephemeral_1h_input_tokens
  webSearches?: number; // usage.server_tool_use.web_search_requests
};

export type Options = { batch?: boolean; usOnly?: boolean };
export type Lines = Record<"input" | "write5m" | "write1h" | "read" | "output" | "search", number>;

export function bill(model: string, u: Usage, o: Options = {}) {
  const m = MODELS[model];
  if (!m) throw new Error(`Unknown model: ${model}`);
  if (o.usOnly && m.noUsOnly) throw new Error(`${model} rejects inference_geo`);
  const { cacheRead = 0, cacheWrite5m = 0, cacheWrite1h = 0, webSearches = 0 } = u;

  // Long-context pricing looks at the whole prompt, cached tokens included.
  const prompt = u.input + cacheRead + cacheWrite5m + cacheWrite1h;
  const tier = m.longAt && prompt > m.longAt ? m.longMult! : 1;
  const mod = tier * (o.batch ? 0.5 : 1) * (o.usOnly ? 1.1 : 1);
  const usd = (tokens: number, perMTok: number) => (tokens * perMTok * mod) / 1e6;

  const lines: Lines = {
    input: usd(u.input, m.input),
    write5m: usd(cacheWrite5m, m.input * 1.25),
    write1h: usd(cacheWrite1h, m.input * 2),
    read: usd(cacheRead, m.input * m.read),
    output: usd(u.output, m.output),
    search: (webSearches * 10) / 1000, // no token multipliers on the search fee
  };
  return { lines, total: Object.values(lines).reduce((a, b) => a + b, 0) };
}

// Cache reads needed before caching beats sending the prefix uncached.
export const breakEvenReads = (writeMult: number, readMult: number) =>
  Math.floor((writeMult - 1) / (1 - readMult)) + 1;

// An agent loop: each call resends the history, which grows by `delta` tokens.
export function agentSession(
  model: string,
  s: { calls: number; firstPrompt: number; delta: number; output: number; cache: "none" | "5m" | "1h" },
) {
  const sum: Lines = { input: 0, write5m: 0, write1h: 0, read: 0, output: 0, search: 0 };
  let prompt = s.firstPrompt;
  let cached = 0;
  for (let call = 1; call <= s.calls; call++) {
    const usage: Usage =
      s.cache === "none"
        ? { input: prompt, output: s.output }
        : {
            input: 0,
            output: s.output,
            cacheRead: cached,
            [s.cache === "5m" ? "cacheWrite5m" : "cacheWrite1h"]: prompt - cached,
          };
    const { lines } = bill(model, usage);
    for (const key of Object.keys(sum) as (keyof Lines)[]) sum[key] += lines[key];
    cached = prompt;
    prompt += s.delta;
  }
  return { lines: sum, total: Object.values(sum).reduce((a, b) => a + b, 0), finalPrompt: cached };
}

const $ = (n: number, digits = 4) => `$${n.toFixed(digits)}`;

if (import.meta.main) {
  console.log("Break-even cache reads");
  for (const read of [0.1, 0.05, 0.025]) {
    console.log(`  read at ${read}x: 5m write ${breakEvenReads(1.25, read)}, 1h write ${breakEvenReads(2, read)}`);
  }

  const session = { calls: 30, firstPrompt: 20_000, delta: 3_500, output: 800 };
  console.log("\nAgent session: 30 calls, 20K first prompt, +3.5K per call, 800 output per call");
  for (const [model, cache] of [
    ["claude-opus-5-5", "none"],
    ["claude-opus-5-5", "5m"],
    ["claude-opus-5-5", "1h"],
    ["claude-sonnet-5-5", "5m"],
    ["claude-haiku-5-5", "5m"],
  ] as const) {
    const r = agentSession(model, { ...session, cache });
    console.log(
      `  ${model.padEnd(18)} ${cache.padEnd(5)} total ${$(r.total)}  input ${$(r.lines.input)}  writes ${$(r.lines.write5m + r.lines.write1h)}  reads ${$(r.lines.read)}  output ${$(r.lines.output)}`,
    );
  }

  const rag = { prefix: 6_000, retrieved: 9_150, output: 500 };
  const uncached = { input: rag.prefix + rag.retrieved, output: rag.output };
  const cached = { input: rag.retrieved, cacheRead: rag.prefix, output: rag.output };
  console.log("\nRAG request: 6K shared prefix, 9.15K retrieved plus question, 500 output");
  const row = (label: string, model: string, u: Usage, o?: Options) =>
    console.log(`  ${label.padEnd(26)} ${$(bill(model, u, o).total, 6)}`);
  row("Sonnet 5.5, no cache", "claude-sonnet-5-5", uncached);
  row("Sonnet 5.5, cache hit", "claude-sonnet-5-5", cached);
  row("Sonnet 5.5, hit, 1 search", "claude-sonnet-5-5", { ...cached, webSearches: 1 });
  row("Sonnet 5.5, hit, batch", "claude-sonnet-5-5", cached, { batch: true });
  row("Haiku 5.5, cache hit", "claude-haiku-5-5", cached);
  row("Opus 5.5, cache hit", "claude-opus-5-5", cached);

  console.log("\nHaiku 5.5 prompt-length cliff, 1,000 output tokens");
  for (const input of [100_000, 100_001]) {
    console.log(`  ${input.toLocaleString("en-US")} prompt tokens: ${$(bill("claude-haiku-5-5", { input, output: 1_000 }).total)}`);
  }
}
```

Run it with `node claude-bill.ts`. This is the real output:

```text
Break-even cache reads
  read at 0.1x: 5m write 1, 1h write 2
  read at 0.05x: 5m write 1, 1h write 2
  read at 0.025x: 5m write 1, 1h write 2

Agent session: 30 calls, 20K first prompt, +3.5K per call, 800 output per call
  claude-opus-5-5    none  total $8.9700  input $8.4900  writes $0.0000  reads $0.0000  output $0.4800
  claude-opus-5-5    5m    total $1.4877  input $0.0000  writes $0.6075  reads $0.4002  output $0.4800
  claude-opus-5-5    1h    total $1.8522  input $0.0000  writes $0.9720  reads $0.4002  output $0.4800
  claude-sonnet-5-5  5m    total $0.7439  input $0.0000  writes $0.3038  reads $0.2001  output $0.2400
  claude-haiku-5-5   5m    total $0.1007  input $0.0000  writes $0.0274  reads $0.0501  output $0.0232

RAG request: 6K shared prefix, 9.15K retrieved plus question, 500 output
  Sonnet 5.5, no cache       $0.035300
  Sonnet 5.5, cache hit      $0.023900
  Sonnet 5.5, hit, 1 search  $0.033900
  Sonnet 5.5, hit, batch     $0.011950
  Haiku 5.5, cache hit       $0.001225
  Opus 5.5, cache hit        $0.047800

Haiku 5.5 prompt-length cliff, 1,000 output tokens
  100,000 prompt tokens: $0.0105
  100,001 prompt tokens: $0.0525
```

A separate script compared all 17 model rows and 17 batch rows above against the text of Anthropic's page, and every cell matched. The function also reproduces the token portion of Anthropic's own worked example, $0.50 and $0.348 for an Opus 5.5 session. No Claude request was sent, so the token counts below are assumptions and only the arithmetic is tested.

One figure is unclear on the source page. Anthropic applies the batch and US-only multipliers to token prices and does not say whether they touch the search fee, so the calculator leaves that fee alone.

## Worked example: a 30-call agent session

A 30-call Opus 5.5 agent session costs $8.97 without caching and $1.49 with the 5-minute cache, an 83% saving. The session starts at 20,000 prompt tokens and grows by 3,500 per call. The output line stays at $0.48 in both cases, because caching only changes what you pay for input.

Each call adds 800 output tokens, thinking included, and 2,700 tokens of tool results. Every call starts inside the cache lifetime, and each one reads the previous prompt and writes only the new tokens. An [AI agent](https://aifordevs.org/articles/what-is-an-ai-agent) built on the Messages API resends its history on each turn, which is why input dominates the uncached bill.

| Model and cache | Input | Cache writes | Cache reads | Output | Total |
| --- | --- | --- | --- | --- | --- |
| Opus 5.5, no cache | $8.49 | $0 | $0 | $0.48 | $8.97 |
| Opus 5.5, 5-minute | $0 | $0.61 | $0.40 | $0.48 | $1.49 |
| Opus 5.5, 1-hour | $0 | $0.97 | $0.40 | $0.48 | $1.85 |
| Sonnet 5.5, 5-minute | $0 | $0.30 | $0.20 | $0.24 | $0.74 |
| Haiku 5.5, 5-minute | $0 | $0.03 | $0.05 | $0.02 | $0.10 |

The 1-hour cache costs $0.36 more here because no gap outlasts 5 minutes, so its premium buys nothing. Haiku 5.5 crosses 100,000 prompt tokens at call 24, and the last 7 calls produce 66% of its bill. Without that tier the same session would cost $0.05, so keeping every prompt under the line is worth about half the bill.

## Worked example: a RAG request

A RAG request with a 6,000-token shared prefix costs $0.0353 on Sonnet 5.5 uncached and $0.0239 on a cache hit, a 32% saving. The other 9,150 prompt tokens are ten 900-token chunks and a 150-token question. RAG saves less than an agent session because those chunks change on each query and stay at the full input price.

| Configuration | Per request | Per 100,000 requests |
| --- | --- | --- |
| Sonnet 5.5, no cache | $0.035300 | $3,530 |
| Sonnet 5.5, cache hit | $0.023900 | $2,390 |
| Sonnet 5.5, cache hit, 1 web search | $0.033900 | $3,390 |
| Sonnet 5.5, cache hit, batch | $0.011950 | $1,195 |
| Haiku 5.5, cache hit | $0.001225 | $122.50 |
| Opus 5.5, cache hit | $0.047800 | $4,780 |

The request returns 500 output tokens, and the web search row excludes tokens for the search results. At $0.01, one search raises this request by 42% and costs twice the output line.

Retrieval size is the next lever, since each 900-token chunk costs $0.0018 on Sonnet 5.5. Dropping from 10 chunks to 5 with [reranking](https://aifordevs.org/glossary#reranking) saves $0.0090 per request. The [vector database guide](https://aifordevs.org/articles/what-is-a-vector-database) covers retrieving fewer, better chunks, and [RAG versus fine-tuning](https://aifordevs.org/articles/rag-vs-fine-tuning) covers whether to retrieve at all.

## Check your Claude API pricing against real usage

Take one day of production `usage` objects and run them through `bill()` before changing anything, then group the result by line item. The largest line tells you what to change first, because each lever acts on one line and leaves the rest alone. Apply these rules in order:

1. If `input` is the largest line and a prefix repeats, add `cache_control` and confirm `cache_read_input_tokens` rises above zero.
2. If `write5m` exceeds `read`, your gaps outlast 5 minutes, so move that prefix to the 1-hour cache.
3. If `output` is the largest line, caching will not help. Try a cheaper model or a lower thinking effort, and check quality with [LLM evaluation](https://aifordevs.org/articles/llm-evaluation) before you ship it.
4. If the work can wait up to 24 hours, move it to the Batch API and halve every token line.
5. If you run Haiku 5.5, alert when any prompt reaches 90,000 tokens, ahead of the 100,000-token cliff.

Rerun the numbers whenever Claude API pricing changes. Follow the [APIs and pricing topic](https://aifordevs.org/topics/apis) for new guides, [model release news](https://aifordevs.org/news) for price changes, and the [Gemini API pricing guide](https://aifordevs.org/articles/gemini-api-pricing) for how a storage-based cache changes the same math. Model choice is covered in [the best LLM for coding](https://aifordevs.org/articles/best-llm-for-coding) and the [models topic](https://aifordevs.org/topics/models).

## Questions

### How much does the Claude API cost?

As of November 8, 2026, the Claude API costs $4 input and $20 output per million tokens on Opus 5.5, $2 and $10 on Sonnet 5.5, $0.10 and $0.50 on Haiku 5.5 for prompts up to 100,000 tokens, and $10 and $50 on Fable 5.1. Cache writes, cache reads, batch, and tool fees change the final bill.

### Is the Claude API free?

No. The Claude API is pay-as-you-go, although Anthropic says new users receive a small amount of free credits to test it. The token counting endpoint is free to call, subject to rate limits, and web fetch adds no fee beyond tokens. Everything else bills per token, plus usage-based fees on some server-side tools.

### How much does Claude Opus cost per million tokens?

Claude Opus 5.5 costs $4 per million input tokens and $20 per million output tokens as of November 8, 2026. Its 5-minute cache write costs $5, its 1-hour write costs $8, and a cache read costs $0.20. Opus 5, 4.8, 4.7, 4.6, and 4.5 each cost $5 and $25, with reads at $0.50.

### Do Claude thinking tokens cost extra?

Thinking tokens bill at the model's normal output rate, so there is no separate thinking price. Anthropic charges for the full thinking generated, even when the API returns only a summary or omits the text. On Opus 5.5, that means thinking bills at $20 per million tokens as of November 8, 2026.

### Can I combine prompt caching with the Batch API?

Yes. Anthropic says the batch discount and cache multipliers stack, so a batched cache read on Opus 5.5 costs $0.10 per million tokens as of November 8, 2026. Cache hits inside a batch are best effort, and Anthropic reports hit rates from 30 to 98 percent. It suggests the 1-hour cache for batches that share context.

### How can I estimate a Claude API bill before sending a request?

Call the token counting endpoint with the exact model ID, then multiply by the rates in the pricing table. Anthropic makes that endpoint free to use, subject to rate limits, and says the count is an estimate. It rejects requests that include most server tools, such as web search, so measure those from the usage object of a real response.

### Why did my Claude API bill go up after switching models?

Check the tokenizer first. Anthropic says Claude 4.7 and later models produce about 30 percent more tokens for the same text than Sonnet 4.6 and earlier. A newer model with the same list price can therefore bill about 30 percent more, and token counts or budgets measured on the older model no longer apply.

## Sources

1. [Anthropic docs, Pricing (read November 8, 2026)](https://platform.claude.com/docs/en/about-claude/pricing)
2. [Anthropic docs, Prompt caching (read November 8, 2026)](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
3. [Anthropic docs, Batch processing (read November 8, 2026)](https://platform.claude.com/docs/en/build-with-claude/batch-processing)
4. [Anthropic docs, Token counting (read November 8, 2026)](https://platform.claude.com/docs/en/build-with-claude/token-counting)
5. [Anthropic docs, Thinking (read November 8, 2026)](https://platform.claude.com/docs/en/build-with-claude/thinking)
6. [Anthropic docs, Context windows (read November 8, 2026)](https://platform.claude.com/docs/en/build-with-claude/context-windows)
7. [Anthropic docs, Models overview (read November 8, 2026)](https://platform.claude.com/docs/en/models/overview)
8. [Anthropic docs, Model deprecations (read November 8, 2026)](https://platform.claude.com/docs/en/about-claude/model-deprecations)

## Cite this guide

AI for Devs. "Claude API pricing: caching, batch, and real bills." Published November 8, 2026. https://aifordevs.org/articles/claude-api-pricing

- Page: https://aifordevs.org/articles/claude-api-pricing
- Markdown: https://aifordevs.org/articles/claude-api-pricing.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "Gemini API pricing: free tier, caching, and real bills"
type: "guide"
url: "https://aifordevs.org/articles/gemini-api-pricing"
markdown: "https://aifordevs.org/articles/gemini-api-pricing.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/gemini-api-pricing"
description: "Gemini API pricing for every current model as of November 8, 2026: free versus paid, cache storage math, batch, grounding, thinking, and a tested calculator."
author: "AI for Devs"
published: "2026-11-08"
level: "Intermediate"
topics: ["apis", "models"]
---

# Gemini API pricing: free tier, caching, and real bills

Gemini API pricing for every current model as of November 8, 2026: free versus paid, cache storage math, batch, grounding, thinking, and a tested calculator.

> **TL;DR**
>
> - Gemini API pricing as of November 8, 2026: Gemini 3.8 Flash costs $0.75 input and $3.75 output per million tokens, 3.1 Pro Preview costs $2 and $12 for prompts up to 200,000 tokens, and 4 Argon costs $2 and $10.
> - Scheduled increases double Gemini 4 Argon on December 1, 2026, and Gemini 3.8 and 3.6 Flash on January 1, 2027, according to Google's pricing page.
> - Explicit caching bills storage by the hour, so its break-even is a request rate that does not depend on prefix size: 2.5 per hour on 3.1 Pro Preview and 0.74 on 3.8 Flash.
> - Thinking tokens bill at the output rate, and Google charges for the full thoughts while returning only summaries.
> - The TypeScript calculator in this guide ran on Node 26.12.1 with no API calls, and a script checked 101 of its price cells against Google's page.

Gemini API pricing is metered per million tokens, and output prices on text models include thinking tokens. Google publishes the rates on its [pricing page](https://aistudio.google.com/docs/pricing), and every figure here was read from that page on November 8, 2026. That page spans 34 model sections, most with several service tiers and footnotes. This guide reduces them to two tables, then works out what caching, thinking, and grounding do to a bill.

## Gemini API pricing by model

Gemini API pricing for text models as of November 8, 2026 runs from $0.10 input per million tokens on 2.5 Flash-Lite to $4 on 3.1 Pro Preview prompts over 200,000 tokens. Output prices include thinking tokens. These are paid Standard-tier rates in US dollars from [Google's pricing page](https://aistudio.google.com/docs/pricing).

> **Prices last checked**
>
> Prices last checked November 8, 2026. This table goes stale on December 1, 2026, when Gemini 4 Argon doubles and 2.5 Pro and 2.5 Flash requests reroute. It goes stale again on January 1, 2027, when 3.8 Flash and 3.6 Flash double.
>
> Google's page says discounts and model availability are subject to change. Recheck it if a model loses its Preview label, a pricing tab appears on a model, or a scheduled date moves.

| Model | Free tier | Input | Output and thinking | Cached input | Storage per MTok per hour | Batch input and output |
| --- | --- | --- | --- | --- | --- | --- |
| Gemini 4 Argon | No | $2.00 ($4.00 from Dec 1) | $10.00 ($20.00 from Dec 1) | $0.10 ($0.20 from Dec 1), implicit only | None | $1.00 and $5.00 ($2.00 and $10.00 from Dec 1) |
| Gemini 3.8 Flash | Yes | $0.75 ($1.50 from Jan 1) | $3.75 ($7.50 from Jan 1) | $0.075 ($0.15 from Jan 1) | $0.50 ($1.00 from Jan 1) | $0.375 and $1.875 ($0.75 and $3.75 from Jan 1) |
| Gemini 3.6 Flash | Yes | $0.75 ($1.50 from Jan 1) | $3.75 ($7.50 from Jan 1) | $0.075 ($0.15 from Jan 1) | $0.50 ($1.00 from Jan 1) | $0.375 and $1.875 ($0.75 and $3.75 from Jan 1) |
| Gemini 3.5 Flash-Lite | Yes | $0.30 | $2.50 | $0.03 | $1.00 | $0.15 and $1.25 |
| Gemini 3.1 Flash-Lite | Yes | $0.25 | $1.50 | $0.025 | $1.00 | $0.125 and $0.75 |
| Gemini 3.1 Pro Preview | No | $2.00 ($4.00 over 200K) | $12.00 ($18.00 over 200K) | $0.20 ($0.40 over 200K) | $4.50 | $1.00 and $6.00 ($2.00 and $9.00 over 200K) |
| Gemini 3 Flash Preview | Yes | $0.50 | $3.00 | $0.05 | $1.00 | $0.25 and $1.50 |
| Gemini 2.5 Pro (deprecated) | Yes | $1.25 ($2.50 over 200K) | $10.00 ($15.00 over 200K) | $0.125 ($0.25 over 200K) | $4.50 | $0.625 and $5.00 ($1.25 and $7.50 over 200K) |
| Gemini 2.5 Flash (deprecated) | Yes | $0.30 | $2.50 | $0.03 | $1.00 | $0.15 and $1.25 |
| Gemini 2.5 Flash-Lite | Yes | $0.10 | $0.40 | $0.01 | $1.00 | $0.05 and $0.20 |

Prices are per million tokens, written MTok above. Audio input costs more on 3.1 Flash-Lite ($0.50), 3 Flash Preview ($1.00), 2.5 Flash ($1.00), and 2.5 Flash-Lite ($0.30). Google [limits the 2.5 models](https://aistudio.google.com/docs/models) to accounts that already used them.

On [December 1, 2026](https://aistudio.google.com/docs/deprecations), `gemini-2.5-pro` requests reroute to 3.8 Flash and `gemini-2.5-flash` requests reroute to 3.5 Flash-Lite, each billed at the new model's rate. For 2.5 Pro users the input rate falls 40% in December and then rises when 3.8 Flash doubles in January.

### Audio, image, video, and embedding models

Specialized models bill audio, images, and video at their own rates, usually in tokens with a per-minute or per-image equivalent. The table lists paid Standard rates per million tokens unless a row says otherwise.

| Model | Free tier | Input | Output |
| --- | --- | --- | --- |
| 3.5 Transcribe Live | Yes | $3.50 or $0.005 per min (audio) | $21.00 or $0.004 per min (text) |
| 3.5 Transcribe | Yes | $2.00 or $0.003 per min (audio) | $12.00 or $0.002 per min (text) |
| 3.8 Live, 3.8 Live Extended Thinking, 3.1 Flash Live Preview | Yes | $0.75 text; $3.00 or $0.005 per min audio; $1.00 or $0.002 per min image and video | $4.50 text; $12.00 or $0.018 per min audio |
| 3.5 Live Translate | Yes | $3.50 or $0.0053 per min | $21.00 or $0.0315 per min |
| 2.5 Flash Native Audio | Yes | $0.50 text; $3.00 audio and video | $2.00 text; $12.00 audio |
| 3.8 Flash TTS | Yes | $0.50 ($1.00 from Jan 1) | $9.00 ($18.00 from Jan 1) |
| 3.8 Flash-Lite TTS | Yes | $0.50 ($1.00 from Jan 1) | $6.00 ($12.00 from Jan 1) |
| 3.1 Flash TTS Preview | Yes | $1.00 | $20.00 |
| 2.5 Flash Preview TTS | Yes | $0.50 | $10.00 |
| 2.5 Pro Preview TTS | No | $1.00 | $20.00 |
| Nano Banana 2.1 | No | $1.50 | $7.50 text and thinking; $30.00 images |
| 3.1 Flash Image | No | $0.50 | $3.00 text and thinking; $60.00 images |
| 3.1 Flash Lite Image | No | $0.25 | $1.50 text and thinking; $30.00 images |
| 3 Pro Image | No | $2.00 | $12.00 text and thinking; $120.00 images |
| 2.5 Flash Image (deprecated, shuts down Mar 15, 2027) | No | $0.30 | $0.039 per image |
| Omni 1.2 Flash | No | $1.50 | $9.00 text; $12.00 video |
| Omni 1.1 Flash (reroutes Dec 3) | No | $1.50 | $9.00 text; $17.50 video |
| Embedding 2 | Yes | $0.20 text; $0.45 image; $6.50 audio; $12.00 video | None |
| Embedding | Yes | $0.15 | None |
| Robotics ER 2 Preview | Yes | $1.00 ($2.00 from Jan 1) | $5.00 ($10.00 from Jan 1) |
| Robotics ER 2 Streaming Preview | Yes | $1.00 ($2.00 from Jan 1) | $5.00 ($10.00 from Jan 1) |
| Lyria 3.5 | No | None | $0.08 per full song |
| Lyria 3 | No | None | $0.04 per 30-second clip; $0.08 per full song |
| Gemma 4 | Yes | Free tier only | Free tier only |

## Free tier versus paid tier

The Gemini API free tier charges nothing for input and output on most Flash models, but Google can use that content to improve its products. The paid tier removes that use, adds [context caching](https://aifordevs.org/glossary#prompt-caching) on more models, and opens 4 Argon and 3.1 Pro Preview. Free rate limits appear in AI Studio, not on the pricing page.

Moving to paid requires [a linked billing account](https://aistudio.google.com/docs/billing), and Google may require at least $5 of prepaid credit, depending on the billing plan it assigns. Spending tiers cap at $250, $2,000, and then $20,000 to $100,000 or more. When a prepaid balance reaches $0, every API key on that billing account stops working, and requests fail with HTTP 402.

Among the Gemini text models, free-tier caching is listed only on 3.8 Flash, 3.6 Flash, and 3 Flash Preview. Google Search grounding on the free tier is limited to 2.5 Flash and 2.5 Flash-Lite, at 500 requests a day. Failed requests with a 400 or 500 status are not billed, and Google's billing FAQ says requests to the `GetTokens` API are not billed.

Two parts of the source page are unclear. Several Batch, Flex, and Priority tabs say "Free of charge" in the Free Tier column, and Google's tier docs do not explain free access to those tiers. The tools table also lists 500 free search requests a day without separating Gemini generations, while each Gemini 3 model row says "Not available."

## Context caching: implicit and explicit

Implicit caching is on by default for 2.5 and newer models, adds no storage fee, and gives no guarantee of a hit. Explicit caching, in beta through `generateContent` on Gemini 3 and 2.5 models, gives a guaranteed reduced rate and bills storage by the hour. Gemini 4 Argon and later support implicit caching only.

### Implicit caching

[Implicit caching](https://aistudio.google.com/docs/caching) needs a prompt of at least 4,096 tokens on 4 Argon, 3.8 Flash, 3.6 Flash, and 3.1 Pro Preview, and 2,048 on 2.5 Flash and 2.5 Pro. Google's table does not list a minimum for the Flash-Lite models or 3 Flash Preview. Put shared content first and send similar requests close together, then read `usage.total_cached_tokens` to confirm hits.

Agent loops on Gemini 4 have a separate rule. A request gets the full previous prompt cached only if it adds at least 2,050 tokens, and smaller additions reprocess the last 2,050 tokens at the input rate.

Google says most agent turns exceed that threshold. An [agent loop](https://aifordevs.org/glossary#agent-loop) still needs a stable prefix to hit the cache at all, so [order context for reuse](https://aifordevs.org/articles/context-engineering).

### Explicit caching and its storage charge

[Explicit caching](https://aistudio.google.com/docs/caching?generate-content=true) bills the cached read rate on each use, plus storage for the cache's whole lifetime. The default TTL is 1 hour, with no minimum or maximum, and you can delete a cache early. Implicit caching does not apply to a request that uses an explicit cache, and any extra input tokens in that request bill at the standard rate.

An explicit cache saves input price minus cached price on each request and costs storage every hour, so the prefix size cancels. The break-even is a rate, computed as `storage / (input - cached)`.

| Model | Input | Cached | Storage | Break-even requests per hour | Longest average gap |
| --- | --- | --- | --- | --- | --- |
| 3.8 Flash and 3.6 Flash | $0.75 | $0.075 | $0.50 | 0.74 | 81 min |
| 3 Flash Preview | $0.50 | $0.05 | $1.00 | 2.22 | 27 min |
| 3.1 Pro Preview, up to 200K | $2.00 | $0.20 | $4.50 | 2.50 | 24 min |
| 3.1 Pro Preview, over 200K | $4.00 | $0.40 | $4.50 | 1.25 | 48 min |
| 2.5 Pro, up to 200K | $1.25 | $0.125 | $4.50 | 4.00 | 15 min |
| 2.5 Pro, over 200K | $2.50 | $0.25 | $4.50 | 2.00 | 30 min |
| 3.5 Flash-Lite and 2.5 Flash | $0.30 | $0.03 | $1.00 | 3.70 | 16 min |
| 3.1 Flash-Lite | $0.25 | $0.025 | $1.00 | 4.44 | 14 min |
| 2.5 Flash-Lite | $0.10 | $0.01 | $1.00 | 11.11 | 5 min |

The January 2027 doubling leaves 3.8 Flash at 0.74, because input, cached, and storage prices all double together.

**Figure 1.** A 100K-token explicit cache on Gemini 3.1 Pro Preview costs $0.45 an hour before any request arrives and beats no cache above 2.5 requests per hour.

Two billing details are unclear on Google's Gemini API pages: whether creating a cache bills the input tokens, and whether storage is prorated within an hour. [Google Cloud's documentation](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/context-cache/context-cache-overview) for its separate Gemini Enterprise Agent Platform says creation bills at the standard input price and implicit caching has no storage cost. The calculator bills creation at the input rate and treats storage as continuous.

## Storage fees versus a write premium

A write premium charges once when an entry is created, and each hit extends the entry at no extra charge. Storage charges for every hour the entry exists, whether or not requests arrive. Anthropic's Claude API uses the first model and Gemini explicit caching uses the second, so request timing changes which one costs less.

The table compares a 100,000-token prefix for 8 hours, prefix cost only, on two models with the same $2 base input price as of November 8, 2026. Gemini is 3.1 Pro Preview. Claude is Sonnet 5.5 at $2.50 for a 5-minute write, $4 for a 1-hour write, and $0.10 for a read, per [Anthropic's pricing page](https://platform.claude.com/docs/en/about-claude/pricing).

| Traffic over 8 hours | No cache | Gemini explicit, 8-hour TTL | Claude 5-minute | Claude 1-hour |
| --- | --- | --- | --- | --- |
| 60 requests per hour | $96.00 | $13.40 | $5.04 | $5.19 |
| 2 requests per hour | $3.20 | $4.12 | $4.00 | $0.55 |
| Two bursts of 5 requests, 6 hours apart | $2.00 | $4.00 | $0.58 | $0.88 |

Gemini's figure combines $0.20 to create the cache, $3.60 of storage, and $0.02 per hit. At 2 requests per hour the storage alone exceeds sending the prefix uncached, because 2 is below the 2.5 break-even. The [Claude API pricing guide](https://aifordevs.org/articles/claude-api-pricing) derives the Claude columns.

**Figure 2.** Two bursts of five requests over 8 hours: the explicit cache keeps billing through the quiet hours and ends at $4.00, while a 5-minute write premium ends at $0.58 and no cache at $2.00.

Cheap storage changes the result on low-traffic prefixes. At 1 request per hour for 8 hours, an explicit cache on 3.8 Flash costs $0.535 against $0.60 with no cache. Implicit caching is the other Gemini advantage, since a miss costs exactly the input price while a Claude write costs 1.25x or 2x even if no read follows.

These numbers compare token for token, and token counts do not transfer between providers. [Google says](https://aistudio.google.com/docs/tokens) a Gemini token is about 4 characters, and [Anthropic says](https://platform.claude.com/docs/en/models/overview) 1M Claude tokens is about 2.5M characters on its current tokenizer. By those rules of thumb the same text produces about 1.6x as many Claude tokens, so count your own prompts on both before comparing bills.

## Batch, Flex, and Priority tiers

Batch and Flex cost half of Standard for input and output, and Priority costs 1.8x. The tiers differ on latency and on what happens to cached reads, which do not follow the same discount across models. Choose by how long a job can wait, then check the cached rate for your exact model on the [pricing page](https://aistudio.google.com/docs/pricing).

| Tier | Input and output | Cached reads | How it runs |
| --- | --- | --- | --- |
| Standard | Base rate | Base rate | Synchronous |
| Batch | 50% of Standard | 50% on 4 Argon, 3.8 and 3.6 Flash, 3.5 and 3.1 Flash-Lite; base rate on 3.1 Pro Preview, 3 Flash Preview, and 2.5 models | Asynchronous, 24-hour target |
| Flex (preview) | 50% of Standard | Like Batch, except $0.02 on 3.5 Flash-Lite | Synchronous, best effort |
| Priority (preview) | 180% of Standard | About 180%, with $0.05 on 3.5 Flash-Lite | Synchronous, overflow drops to Standard |

[Batch](https://aistudio.google.com/docs/batch-api) targets a 24-hour turnaround, and the [batch API](https://aifordevs.org/glossary#batch-api) discount suits evals and backfills. [Flex](https://aistudio.google.com/docs/flex-inference) can return 503 or 429 errors with no automatic upgrade to Standard, so you need retries and timeouts of 10 minutes or more.

[Priority](https://aistudio.google.com/docs/priority-inference) overflow downgrades to Standard instead of failing. An invite-only Contributor tier lists half-price rates on 4 Argon, 3.8 Flash, 3.5 Flash-Lite, and 3.1 Pro Preview in exchange for using your content to improve Google's products.

## Long-prompt tiers at 200,000 tokens

Only Gemini 3.1 Pro Preview and 2.5 Pro have long-prompt tiers, and both switch at 200,000 prompt tokens. Above that line input, output, and cached reads all reprice, so 3.1 Pro Preview moves from $2 and $12 to $4 and $18. Flash, Flash-Lite, and 4 Argon list one rate at every prompt length as of November 8, 2026.

The boundary is a cliff if the higher rate covers the whole request. With 1,000 output tokens on 3.1 Pro Preview, a 200,000-token prompt costs $0.4120 and a 200,001-token prompt costs $0.8180.

Google's page writes these rates per prompt length and does not say whether the higher rate covers every token or only those past 200,000. It also does not say whether cached tokens count toward the limit. The calculator applies the higher rate to the whole request and counts cached tokens, which is the conservative reading.

## Thinking tokens bill at the output rate

Thinking tokens bill at the output rate, and [Google charges for the full thoughts](https://aistudio.google.com/docs/thinking) even though the API returns only summaries. Output prices on the pricing page already include them. A thinking-heavy answer can cost several times its visible output, so log `total_thought_tokens` on every response and report it separately from output.

Defaults vary by model. 3.1 Pro Preview and 3 Flash Preview default to high, while 4 Argon, 3.8 Flash, and 3.6 Flash default to medium. 3.5 Flash-Lite defaults to minimal, and 2.5 Flash-Lite ships with thinking off.

Google recommends lowering `thinking_level` instead of cutting `max_output_tokens`, because that limit covers thinking and output combined and a low value truncates responses.

On 3.1 Pro Preview, a 400-token answer costs $0.0048 in output without thinking. Add 3,000 thinking tokens and the same answer costs $0.0408, which is 8.5 times as much.

## Grounding and other tool fees

Grounding with Google Search on Gemini 3 and newer costs $14 per 1,000 search queries after 5,000 free queries a month, shared across those models. One prompt can run several queries, and [each query bills](https://aistudio.google.com/docs/google-search). Gemini 2.5 bills per grounded prompt, at $35 per 1,000 after 1,500 free requests a day.

| Tool | Free allowance on paid tier | Then |
| --- | --- | --- |
| Google Search, Gemini 3 and newer | 5,000 queries a month, shared | $14 per 1,000 queries; default cap of 1,500 a day per project |
| Google Search, Gemini 2.5 | 1,500 requests a day; Flash and Flash-Lite share one limit | $35 per 1,000 grounded prompts |
| Google Maps, Gemini 3 and newer | 5,000 requests a month, shared | $14 per 1,000 queries |
| Google Maps, Gemini 2.5 | 1,500 requests a day on Flash and Flash-Lite, 10,000 on Pro | $25 per 1,000 grounded prompts |
| Code execution | No separate fee | Generated code and results bill as output tokens, then as input tokens when reused |
| URL context | No separate fee | Retrieved content bills as input tokens |
| Computer use | No separate fee | Regular token rates |
| File search | No separate fee | Embeddings at the embedding model's rate; retrieved tokens at the model's rate |

One query costs $0.014, the same as 18,667 input tokens on 3.8 Flash. At the default cap of 1,500 queries a day, a 30-day month holds 45,000 queries, and the 40,000 past the free allowance cost $560.

File search charges for embeddings and for the tokens it retrieves, so the retrieval size in a [RAG](https://aifordevs.org/glossary#rag) pipeline shows up directly on the bill. The [vector database guide](https://aifordevs.org/articles/what-is-a-vector-database) covers how to retrieve less.

## A tested Gemini API cost calculator

The calculator below takes the fields from a Gemini `usage` object and returns an itemized bill for paid Standard and Batch requests. It handles the scheduled price dates, the 200,000-token tier, thinking tokens, explicit cache storage, and billable search queries. It ran on Node 26.12.1 with no dependencies and no network code.

```ts title="gemini-bill.ts"
// gemini-bill.ts: turn Gemini API usage into a bill.
// Paid Standard-tier prices, USD per 1M tokens, read from Google's pricing page on November 8, 2026.
type Rate = { input: number; output: number; cached: number; storage?: number };
type Card = Rate & {
  until?: string; // last day this price applies (ISO date)
  longAt?: number; // prompt length above which `long` applies
  long?: Rate;
  batchCached?: number; // multiplier on cached reads under Batch (default 0.5)
  batchStorage?: number; // multiplier on storage under Batch (default 1)
};

const proLong = (input: number, output: number, cached: number): Rate => ({ input, output, cached, storage: 4.5 });

export const MODELS: Record<string, Card[]> = {
  "gemini-4-argon": [
    { until: "2026-11-30", input: 2, output: 10, cached: 0.1 },
    { input: 4, output: 20, cached: 0.2 },
  ],
  "gemini-3.8-flash": [
    { until: "2026-12-31", input: 0.75, output: 3.75, cached: 0.075, storage: 0.5 },
    { input: 1.5, output: 7.5, cached: 0.15, storage: 1 },
  ],
  "gemini-3.6-flash": [
    { until: "2026-12-31", input: 0.75, output: 3.75, cached: 0.075, storage: 0.5 },
    { input: 1.5, output: 7.5, cached: 0.15, storage: 1 },
  ],
  "gemini-3.5-flash-lite": [{ input: 0.3, output: 2.5, cached: 0.03, storage: 1 }],
  "gemini-3.1-flash-lite": [{ input: 0.25, output: 1.5, cached: 0.025, storage: 1, batchStorage: 0.5 }],
  "gemini-3.1-pro-preview": [
    { input: 2, output: 12, cached: 0.2, storage: 4.5, batchCached: 1, longAt: 200_000, long: proLong(4, 18, 0.4) },
  ],
  "gemini-3-flash-preview": [{ input: 0.5, output: 3, cached: 0.05, storage: 1, batchCached: 1 }],
  "gemini-2.5-pro": [
    { input: 1.25, output: 10, cached: 0.125, storage: 4.5, batchCached: 1, longAt: 200_000, long: proLong(2.5, 15, 0.25) },
  ],
  "gemini-2.5-flash": [{ input: 0.3, output: 2.5, cached: 0.03, storage: 1, batchCached: 1 }],
  "gemini-2.5-flash-lite": [{ input: 0.1, output: 0.4, cached: 0.01, storage: 1, batchCached: 1 }],
};

export type Usage = {
  prompt: number; // usage.total_input_tokens, cached tokens included
  cached?: number; // usage.total_cached_tokens
  output: number; // usage.total_output_tokens
  thinking?: number; // usage.total_thought_tokens
  cacheCreate?: number; // tokens sent to create an explicit cache
  storedTokens?: number; // explicit cache size
  storedHours?: number; // how long that cache lives
  billableSearches?: number; // queries past the monthly free allowance
};

export type Options = { date?: string; batch?: boolean };
export type Lines = Record<"input" | "cached" | "output" | "create" | "storage" | "search", number>;

export function card(model: string, date = "2026-11-08"): Card {
  const cards = MODELS[model];
  if (!cards) throw new Error(`Unknown model: ${model}`);
  return cards.find((c) => !c.until || date <= c.until)!;
}

export function bill(model: string, u: Usage, o: Options = {}) {
  const c = card(model, o.date);
  const { cached = 0, thinking = 0, cacheCreate = 0, storedTokens = 0, storedHours = 0, billableSearches = 0 } = u;
  // The long-prompt card applies to the whole request, counting cached tokens as prompt tokens.
  const r: Rate = c.longAt && u.prompt > c.longAt ? c.long! : c;
  const io = o.batch ? 0.5 : 1;
  const cacheMult = o.batch ? (c.batchCached ?? 0.5) : 1;
  const storageMult = o.batch ? (c.batchStorage ?? 1) : 1;
  const usd = (tokens: number, perMTok: number) => (tokens * perMTok) / 1e6;
  const searchRate = model.startsWith("gemini-2.5") ? 35 : 14;

  const lines: Lines = {
    input: usd(u.prompt - cached, r.input * io),
    cached: usd(cached, r.cached * cacheMult),
    output: usd(u.output + thinking, r.output * io), // thinking bills at the output rate
    create: usd(cacheCreate, r.input), // assumption: creation bills at the standard input rate
    storage: usd(storedTokens * storedHours, (r.storage ?? 0) * storageMult),
    search: (billableSearches * searchRate) / 1000,
  };
  return { lines, total: Object.values(lines).reduce((a, b) => a + b, 0) };
}

// Requests per hour above which an explicit cache beats no cache.
// Prefix size cancels out, so this is one number per model.
export function breakEvenHitsPerHour(model: string, date = "2026-11-08") {
  const c = card(model, date);
  return c.storage! / (c.input - c.cached);
}

const $ = (n: number) => `$${n.toFixed(4)}`;

if (import.meta.main) {
  console.log("Explicit cache break-even, requests per hour");
  for (const m of ["gemini-3.8-flash", "gemini-3-flash-preview", "gemini-3.5-flash-lite", "gemini-3.1-pro-preview"]) {
    console.log(`  ${m.padEnd(26)} ${breakEvenHitsPerHour(m).toFixed(2)}`);
  }

  // 100K-token prefix on 3.1 Pro Preview for an 8-hour day. Prefix cost only.
  const prefix = 100_000;
  console.log("\n100K prefix, gemini-3.1-pro-preview, 8 hours, prefix cost only");
  for (const perHour of [2, 60]) {
    const n = perHour * 8;
    const none = n * bill("gemini-3.1-pro-preview", { prompt: prefix, output: 0 }).total;
    const hit = bill("gemini-3.1-pro-preview", { prompt: prefix, cached: prefix, output: 0 }).total;
    const setup = bill("gemini-3.1-pro-preview", {
      prompt: 0, output: 0, cacheCreate: prefix, storedTokens: prefix, storedHours: 8,
    }).total;
    console.log(`  ${String(perHour).padStart(2)}/hour: none ${$(none)}  explicit ${$(setup + n * hit)}  (setup and storage ${$(setup)})`);
  }

  console.log("\nThinking on gemini-3.1-pro-preview: 400 visible output tokens");
  console.log(`  no thinking            ${$(bill("gemini-3.1-pro-preview", { prompt: 2_000, output: 400 }).lines.output)}`);
  console.log(`  3,000 thinking tokens  ${$(bill("gemini-3.1-pro-preview", { prompt: 2_000, output: 400, thinking: 3_000 }).lines.output)}`);

  console.log("\nLong-prompt tier, gemini-3.1-pro-preview, 1,000 output tokens");
  for (const prompt of [200_000, 200_001]) {
    console.log(`  ${prompt.toLocaleString("en-US")} prompt tokens: ${$(bill("gemini-3.1-pro-preview", { prompt, output: 1_000 }).total)}`);
  }

  const rag = { prompt: 15_150, cached: 6_000, output: 500, thinking: 1_500 };
  console.log("\nRAG request: 15,150 prompt tokens, 6,000 cached, 500 output, 1,500 thinking");
  console.log(`  3.8 Flash, November 2026   ${$(bill("gemini-3.8-flash", rag).total)}`);
  console.log(`  3.8 Flash, January 2027    ${$(bill("gemini-3.8-flash", rag, { date: "2027-01-01" }).total)}`);
  console.log(`  3.8 Flash, Batch, November ${$(bill("gemini-3.8-flash", rag, { batch: true }).total)}`);
  console.log(`  3.5 Flash-Lite             ${$(bill("gemini-3.5-flash-lite", rag).total)}`);
  console.log(`  4 Argon, November 2026     ${$(bill("gemini-4-argon", rag).total)}`);
  console.log(`  4 Argon, December 2026     ${$(bill("gemini-4-argon", rag, { date: "2026-12-01" }).total)}`);
  console.log(`  4 Argon, December, 3 billable searches ${$(bill("gemini-4-argon", { ...rag, billableSearches: 3 }, { date: "2026-12-01" }).total)}`);
}
```

Run it with `node gemini-bill.ts`. This is the real output:

```text
Explicit cache break-even, requests per hour
  gemini-3.8-flash           0.74
  gemini-3-flash-preview     2.22
  gemini-3.5-flash-lite      3.70
  gemini-3.1-pro-preview     2.50

100K prefix, gemini-3.1-pro-preview, 8 hours, prefix cost only
   2/hour: none $3.2000  explicit $4.1200  (setup and storage $3.8000)
  60/hour: none $96.0000  explicit $13.4000  (setup and storage $3.8000)

Thinking on gemini-3.1-pro-preview: 400 visible output tokens
  no thinking            $0.0048
  3,000 thinking tokens  $0.0408

Long-prompt tier, gemini-3.1-pro-preview, 1,000 output tokens
  200,000 prompt tokens: $0.4120
  200,001 prompt tokens: $0.8180

RAG request: 15,150 prompt tokens, 6,000 cached, 500 output, 1,500 thinking
  3.8 Flash, November 2026   $0.0148
  3.8 Flash, January 2027    $0.0296
  3.8 Flash, Batch, November $0.0074
  3.5 Flash-Lite             $0.0079
  4 Argon, November 2026     $0.0389
  4 Argon, December 2026     $0.0778
  4 Argon, December, 3 billable searches $0.1198
```

A separate script compared 101 Standard and Batch price cells for the 10 text models against the text of Google's page, and every one matched. No Gemini request was sent, so the token counts are assumptions and only the arithmetic is tested. The script does not price Flex, Priority, audio input, or image output, so read those rates from the tables above.

The RAG example shows where the money goes. On 3.8 Flash in November, the 1,500 thinking tokens and 500 output tokens cost $0.0075, about half of the request. Removing the thinking would save $0.0056, more than the $0.0041 that caching saves on the input line, so a [reasoning model](https://aifordevs.org/glossary#reasoning-model) setting is the first thing to tune.

## Apply Gemini API pricing to your own traffic

Log `usage` for one day, group requests by shared prefix, and compute each prefix's requests per hour. Compare that rate with the break-even table before you create an explicit cache, because the rate decides whether storage pays and prefix size only sets the amount at stake. Then work through these steps:

1. Rerun `bill()` with `date: "2026-12-01"` and `date: "2027-01-01"`, and budget for doubled rates on any Argon or 3.8 Flash route.
2. Below the break-even rate, skip explicit caching and rely on implicit caching. Move shared content to the start of the prompt, as the [KV cache explainer](https://aifordevs.org/articles/kv-cache-explained) describes.
3. Above the break-even rate, create an explicit cache with the shortest TTL that covers the traffic window, and delete it when the window ends.
4. Log `total_thought_tokens` per route, lower `thinking_level` where quality holds, and confirm with [LLM evaluation](https://aifordevs.org/articles/llm-evaluation).
5. Move work that can wait to Batch after you check the cached rate for that model.
6. Keep confidential data off the free tier, where Google can use your content to improve its products.

Gemini API pricing changes on fixed dates this winter, so set reminders for both. Check [model release news](https://aifordevs.org/news) for updates and browse the [APIs and pricing topic](https://aifordevs.org/topics/apis) for related guides.

Read about [picking a coding model](https://aifordevs.org/articles/best-llm-for-coding) before you commit a route to one tier. The [AI agent guide](https://aifordevs.org/articles/what-is-an-ai-agent) covers how loops multiply these per-request numbers, and the [platforms directory](https://aifordevs.org/platforms) lists other providers.

## Questions

### How much does the Gemini API cost?

As of November 8, 2026, the Gemini API costs $0.75 input and $3.75 output per million tokens on Gemini 3.8 Flash, $2 and $12 on Gemini 3.1 Pro Preview for prompts up to 200,000 tokens, and $2 and $10 on Gemini 4 Argon. Argon doubles on December 1, 2026, and 3.8 Flash doubles on January 1, 2027.

### Is the Gemini API free?

The Gemini API has a free tier with no token charges on most Flash models, including 3.8 Flash, but Gemini 4 Argon and 3.1 Pro Preview are paid only. On the free tier, Google may use your content to improve its products. Rate limits apply, and paid tiers need a linked billing account, often with at least $5 of prepaid credit.

### Does Gemini charge for thinking tokens?

Yes. Thinking tokens bill at the output rate, and Google's pricing page labels output prices as including them. Google charges for the full thoughts generated even though the API returns only summaries. On Gemini 3.1 Pro Preview, 3,000 thinking tokens cost $0.036 at the $12 output rate as of November 8, 2026.

### How does Gemini context caching pricing work?

Implicit caching runs by default and bills cached tokens at a reduced rate, with no storage fee and no guarantee of a hit. Explicit caching, available on Gemini 3 and 2.5 models, bills the same reduced read rate plus storage per million tokens per hour. On 3.1 Pro Preview that storage costs $4.50 as of November 8, 2026.

### Is an explicit Gemini context cache worth it?

An explicit cache pays off when requests per hour exceed a model-specific break-even, which is the storage price divided by the difference between the input and cached prices. As of November 8, 2026, that is 2.5 requests per hour on Gemini 3.1 Pro Preview and 0.74 per hour on Gemini 3.8 Flash. Prefix size does not change it.

### Does the Gemini API charge for Google Search grounding?

Yes. On Gemini 3 and newer models, the first 5,000 search queries each month are free across those models, then 1,000 queries cost $14 as of November 8, 2026. One prompt can trigger several queries, and each one bills. Gemini 2.5 models bill per grounded prompt at $35 per 1,000 after 1,500 free requests a day.

### What is the difference between Batch and Flex on the Gemini API?

Both cost 50% of Standard for input and output as of November 8, 2026. Batch is asynchronous with a 24-hour target turnaround. Flex is a synchronous preview tier with best-effort availability, so requests can fail with 429 or 503 errors and need your own retries. Cached rates vary by model, so check the row for yours.

## Sources

1. [Google, Gemini Developer API pricing (read November 8, 2026)](https://aistudio.google.com/docs/pricing)
2. [Google, Context caching for the Gemini API (read November 8, 2026)](https://aistudio.google.com/docs/caching)
3. [Google, Explicit context caching with generateContent (read November 8, 2026)](https://aistudio.google.com/docs/caching?generate-content=true)
4. [Google, Thinking in the Gemini API (read November 8, 2026)](https://aistudio.google.com/docs/thinking)
5. [Google, Tokens in the Gemini API (read November 8, 2026)](https://aistudio.google.com/docs/tokens)
6. [Google, Batch API (read November 8, 2026)](https://aistudio.google.com/docs/batch-api)
7. [Google, Flex inference (read November 8, 2026)](https://aistudio.google.com/docs/flex-inference)
8. [Google, Priority inference (read November 8, 2026)](https://aistudio.google.com/docs/priority-inference)
9. [Google, Billing for the Gemini API (read November 8, 2026)](https://aistudio.google.com/docs/billing)
10. [Google, Grounding with Google Search (read November 8, 2026)](https://aistudio.google.com/docs/google-search)
11. [Google, Gemini models (read November 8, 2026)](https://aistudio.google.com/docs/models)
12. [Google, Gemini deprecations (read November 8, 2026)](https://aistudio.google.com/docs/deprecations)
13. [Google Cloud, Context cache overview for Gemini Enterprise Agent Platform (read November 8, 2026)](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/context-cache/context-cache-overview)
14. [Anthropic docs, Pricing (read November 8, 2026)](https://platform.claude.com/docs/en/about-claude/pricing)
15. [Anthropic docs, Models overview (read November 8, 2026)](https://platform.claude.com/docs/en/models/overview)

## Cite this guide

AI for Devs. "Gemini API pricing: free tier, caching, and real bills." Published November 8, 2026. https://aifordevs.org/articles/gemini-api-pricing

- Page: https://aifordevs.org/articles/gemini-api-pricing
- Markdown: https://aifordevs.org/articles/gemini-api-pricing.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "LLM as a judge: biases, validation, and when to distrust it"
type: "deep dive"
url: "https://aifordevs.org/articles/llm-as-a-judge"
markdown: "https://aifordevs.org/articles/llm-as-a-judge.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/llm-as-a-judge"
description: "LLM as a judge means grading model output with a model. See what Zheng et al. measured, where bias enters, and how to validate a judge against human labels."
author: "AI for Devs"
published: "2026-11-08"
level: "Advanced"
topics: ["evals"]
---

# LLM as a judge: biases, validation, and when to distrust it

LLM as a judge means grading model output with a model. See what Zheng et al. measured, where bias enters, and how to validate a judge against human labels.

> **TL;DR**
>
> - LLM as a judge means using one model to grade another model's output against a written criterion or by comparing two answers.
> - Zheng et al. (2023) reported 85 percent agreement between GPT-4 and humans on MT-Bench with ties excluded, and 66 percent with ties and inconsistent votes counted. Read both numbers.
> - Position, verbosity, and self-preference biases all have primary measurements, and each has a test you can run on your own judge.
> - Validate against human labels with kappa and a bootstrap interval. In this article's illustrative dataset, 88 percent agreement hides a kappa of 0.61 (95 percent interval 0.46 to 0.74) and a 35 percent false-pass rate. Python 3.15.1 ran this with scripted clients and no hosted model.
> - Do not trust a judge alone on factual claims without a reference, on text an attacker can write, or when the lower end of its kappa interval is under 0.60.

LLM as a judge is the practice of using a language model to grade another model's output against a written criterion or by comparing answers. It lets you score open-ended outputs at a volume that human reviewers cannot reach. A judge can also agree with humans 88 percent of the time and still pass a third of the outputs a person would reject. This article starts from the 2023 MT-Bench paper, reads the later bias work, and shows the calculation to run before you trust the score.

## What LLM as a judge means in practice

LLM as a judge turns a quality question into a model call that returns a verdict. Three forms are common: grading one answer, comparing two answers, and grading against a reference answer. Each form has different failure modes, so the choice of form is part of the measurement.

[Zheng et al. (2023)](https://arxiv.org/abs/2306.05685) tested pairwise comparison and single-answer grading. [OpenAI's evaluation guide](https://developers.openai.com/api/docs/guides/evaluation-best-practices) lists pairwise comparison, single-answer grading, and reference-guided grading, and recommends pairwise or pass/fail formats for reliability.

| Form | Input | Output | Best fit | Main weakness |
| --- | --- | --- | --- | --- |
| Single answer | Question, answer, criterion | Pass or fail, or a score | Regression tracking on one model | Absolute scoring is easiest to attack |
| Pairwise | Question, two answers | Winner or tie | Choosing between two versions | Position bias |
| Reference-guided | Question, answer, known-correct reference | Pass or fail | Math, facts, extraction | Needs a trustworthy reference |

A judge fits inside a larger [eval](https://aifordevs.org/glossary#eval) suite, usually as the metric for open-ended answers. The [LLM evaluation guide](https://aifordevs.org/articles/llm-evaluation) covers where it sits, and the [glossary entry for LLM as a judge](https://aifordevs.org/glossary#llm-as-a-judge) gives the short definition.

## What Zheng et al. measured

The 2023 MT-Bench paper reported that GPT-4 agreed with human preferences more often than humans agreed with each other, in one specific setup. That setup excludes ties. The paper reports both ways of counting, and the gap between them is the first thing to read about judge accuracy.

MT-Bench has 80 multi-turn questions in 8 categories, and the paper compares six models. About 3,000 votes came from 58 expert labelers, mostly graduate students.

Setup S1 counts ties and position-inconsistent votes as ties, and its chance baseline is 33 percent. Setup S2 counts only non-tie votes, and its chance baseline is 50 percent.

| Comparison, first turn | S1 (ties and inconsistent counted) | S2 (non-tie only) |
| --- | --- | --- |
| GPT-4 pairwise vs humans | 66% (1,343 votes) | 85% (859 votes) |
| GPT-4 single answer vs humans | 60% (1,280 votes) | 85% (739 votes) |
| Human vs human | 63% (721 votes) | 81% (479 votes) |

The 85 percent headline holds only after dropping every tie and every case where the judge changed its verdict when the answers swapped. The same table shows S2 removing more than a third of the GPT-4 pairwise votes, from 1,343 to 859. Your own validation has to make that choice about what counts in the open.

Treat this as a 2023 result. The authors used `gpt-4-0314`, and they say the examples may not be fully reproducible on later GPT-4 versions. They also say the work largely sets aside safety, including honesty and harmlessness.

## Position bias

Position bias means a judge's verdict depends on which slot an answer occupies, so swapping the order changes the winner. On MT-Bench, GPT-4 stayed consistent in 65.0 percent of swapped cases with the default prompt, against 46.2 percent for GPT-3.5 and 23.8 percent for Claude-v1. Run each pair in both orders.

**Figure 1.** A consistent judge picks the same answer in both orders. A position-biased judge picks the same slot, so the winner changes when the order does, and the pair should count as a tie.

Few-shot examples help and cost you. The MT-Bench authors raised GPT-4 consistency from 65.0 to 77.5 percent with few-shot examples, and longer prompts made API calls about 4 times as expensive. Check current rates in [Claude API pricing](https://aifordevs.org/articles/claude-api-pricing) or [Gemini API pricing](https://aifordevs.org/articles/gemini-api-pricing) before you run a few-shot judge on every trace.

[Wang et al. (2023)](https://arxiv.org/abs/2305.17926) showed how far order alone can move a ranking. Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the evaluator, and their calibration methods aim to reduce exactly that.

[Shi et al. (2024)](https://arxiv.org/abs/2406.07791) tested 15 judges across more than 150,000 evaluation instances. Position bias varied by judge and task, and it was strongly affected by the quality gap between solutions.

Close versions of your prompt are the hard case, because the authors found answers of similar quality are harder to judge and more prone to position bias. Count order-inconsistent pairs as ties, and report the tie rate next to the win rate.

## Verbosity bias

Verbosity bias means a judge prefers the longer answer when the content is otherwise equal or worse. An MT-Bench test padded 23 answers with a rephrased list. Claude-v1 and GPT-3.5 preferred the padded version 91.3 percent of the time, and GPT-4 did so 8.7 percent of the time.

[Saito et al. (2023)](https://arxiv.org/abs/2310.10076) defined a verbosity metric where positive values favor wordier answers, and reported 0.328 for GPT-4 and 0.428 for GPT-3.5. Once the word-count difference grows large enough, GPT-4 almost always preferred the longer answer in their study.

Length also distorts leaderboards like the ones behind [best LLM for coding](https://aifordevs.org/articles/best-llm-for-coding) rankings. [Dubois et al. (2024)](https://arxiv.org/abs/2404.04475) estimated AlpacaEval preferences at equal output length and raised its Spearman correlation with Chatbot Arena from 0.94 to 0.98.

A prompt change that adds a closing paragraph can look like a win for that reason alone.

Test yours directly. Take a passing answer, add a paragraph restating it, and re-judge. If the verdict flips, or the pairwise judge prefers the padded version, you have measured the bias on your own rubric.

## Self-preference bias

Self-preference bias means a judge scores text from its own model family higher than a neutral rater would. The MT-Bench authors said their data could not settle the question. Later studies found the bias in GPT-4 and did not find it uniformly across models, so use a judge from a different family.

[Panickssery et al. (2024)](https://arxiv.org/abs/2404.13076) measured summary preferences on XSUM and CNN. GPT-4's normalized preference for its own summaries was 0.705 and 0.912, against 0.511 and 0.505 for Llama-2-7b, where 0.5 would be indifference.

GPT-3.5 scored 0.582 and 0.431, so the direction is not universal. After fine-tuning, self-recognition and self-preference moved together in a linear relationship.

[Wataoka et al. (2024)](https://arxiv.org/abs/2410.21819) reported significant self-preference in GPT-4 and traced it to perplexity, since judges gave higher scores to lower-perplexity text than humans did. That points to a broader risk: a judge may reward text that resembles what its training made likely.

Anthropic's [eval docs](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) note in their judge examples that it is generally best to evaluate with a different model from the one that generated the output. When you compare model families, run the judge from a third family, or run two judges and report where they disagree.

## How to build a judge you can check

Build a judge as a small function with one criterion, a binary verdict, and an explicit error state. One criterion per judge makes disagreement diagnosable, and a binary verdict is easier to validate than a 1 to 5 score. Require reasoning before the verdict, which OpenAI's guide says improves eval performance.

The MT-Bench paper shows why the format matters. On 10 math questions with 20 test cases, the default prompt failed 14 of 20, chain-of-thought failed 6 of 20, and a reference-guided prompt failed 3 of 20. Give the judge the correct answer whenever you have one.

Keep these rules for the first version:

1. Write one judge per criterion, in the language of your rubric, and keep it in its own [system prompt](https://aifordevs.org/glossary#system-prompt).
2. Return [JSON](https://aifordevs.org/glossary#structured-outputs) with reasoning first and the verdict last.
3. Treat any unparseable or out-of-range output as an error, never as a pass.
4. Add few-shot examples only from your development set.
5. Pin the judge model and version, and log the raw output.

The code below implements those rules plus the swap test, against a scripted client so it runs offline. Replace `scripted_client` with a function that calls your provider.

```python title="judge.py"
#!/usr/bin/env python3
"""A pass/fail judge and a position-swap check, run against a scripted client."""
import json

CRITERION = "The answer states that refunds over $500 need a second approver."

SINGLE_PROMPT = """Grade one answer against one criterion.
Question: {question}
Answer: {answer}
Criterion: {criterion}
Return JSON only: {{"reasoning": "<one sentence>", "verdict": "pass" or "fail"}}"""

PAIR_PROMPT = """Which response better meets the criterion?
Question: {question}
Response A: {a}
Response B: {b}
Criterion: {criterion}
Return JSON only: {{"reasoning": "<one sentence>", "winner": "A" or "B" or "tie"}}"""


def judge_single(client, question, answer, criterion=CRITERION):
    raw = client(SINGLE_PROMPT.format(question=question, answer=answer, criterion=criterion))
    data = json.loads(raw)
    if data.get("verdict") not in {"pass", "fail"}:
        return {"verdict": "error", "reasoning": raw}
    return data


def judge_pair_swapped(client, question, x, y, criterion=CRITERION):
    """Ask twice with the order swapped. Disagreement becomes a tie."""
    first = json.loads(client(PAIR_PROMPT.format(question=question, a=x, b=y, criterion=criterion)))["winner"]
    second = json.loads(client(PAIR_PROMPT.format(question=question, a=y, b=x, criterion=criterion)))["winner"]
    first_pick = {"A": "x", "B": "y"}.get(first, "tie")
    second_pick = {"A": "y", "B": "x"}.get(second, "tie")
    if first_pick == second_pick:
        return first_pick, first, second
    return "tie (inconsistent)", first, second


def scripted_client(replies):
    queue = iter(replies)
    return lambda _prompt: next(queue)


question = "Who approves a $600 refund?"
good = "Refunds over $500 need a second approver in finance."
bad = "Refunds over $500 are approved by the finance team."

client = scripted_client([
    '{"reasoning": "It names a second approver.", "verdict": "pass"}',
    '{"reasoning": "No second approver is named.", "verdict": "fail"}',
    '{"reasoning": "Unparseable label.", "verdict": "maybe"}',
])
print("single:", judge_single(client, question, good)["verdict"])
print("single:", judge_single(client, question, bad)["verdict"])
print("single:", judge_single(client, question, good)["verdict"])

# Position-biased client: always prefers the first response.
biased = scripted_client(['{"reasoning": "First.", "winner": "A"}'] * 2)
print("biased pair:", judge_pair_swapped(biased, question, good, bad))

# Consistent client: prefers the same response in both orders.
stable = scripted_client(['{"reasoning": "x wins.", "winner": "A"}', '{"reasoning": "x wins.", "winner": "B"}'])
print("stable pair:", judge_pair_swapped(stable, question, good, bad))
```

Tested with Python 3.15.1 on November 8, 2026:

```text
single: pass
single: fail
single: error
biased pair: ('tie (inconsistent)', 'A', 'A')
stable pair: ('x', 'A', 'B')
```

The third line shows an out-of-range verdict becoming an error. The biased pair shows a client that always prefers slot A being counted as a tie, which is the logic of Figure 1.

This run exercised prompt formatting, JSON parsing, the error state, and swap logic. It did not exercise any model's actual judgment.

## Validate the judge against human labels

Validate a judge by scoring a sealed set of human-labeled outputs once, then reporting agreement, a chance-corrected statistic, and a false-pass rate with confidence intervals. Raw agreement alone overstates the judge when one label dominates, so a tool that always says pass can look accurate on a mostly passing set.

**Figure 2.** Human labels split into a development set that can change the rubric and a holdout that cannot. The holdout's kappa interval decides how far the judge is trusted.

Sample real outputs into a [golden dataset](https://aifordevs.org/glossary#golden-dataset) and oversample failures on purpose, so the false-pass rate has enough failed cases to estimate. Have two people label an overlapping subset first, because the agreement between humans is the ceiling for any judge on that rubric.

If two people cannot agree, fix the rubric before you touch a model.

Split the labels and seal the holdout. Figure 2 shows a 30 and 70 percent split, which is a starting point. Tune the rubric and examples only on the development set, and score the holdout once at the end, as OpenAI's guide says to [calibrate automated scoring against human feedback](https://developers.openai.com/api/docs/guides/evaluation-best-practices).

The script below computes raw agreement, Cohen's kappa, and a percentile bootstrap interval. [Cohen's kappa](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html) is `(p_o - p_e) / (1 - p_e)`, where `p_o` is observed agreement and `p_e` is the agreement expected by chance from each rater's label rates.

The label counts below are illustrative, written by hand for this article. They are not a measurement of any model.

```python title="agreement.py"
#!/usr/bin/env python3
"""Judge versus human labels: agreement, kappa, and a bootstrap interval.
The label counts below are ILLUSTRATIVE, written by hand for this article.
They are not a measurement of any model or any judge."""
import math
import random

SEED, RESAMPLES = 42, 10_000
Z = 1.959964  # two-sided 95 percent


def expand(tp, fn, fp, tn):
    """(human, judge) pairs, 1 for pass and 0 for fail."""
    return [(1, 1)] * tp + [(1, 0)] * fn + [(0, 1)] * fp + [(0, 0)] * tn


def cohen_kappa(pairs):
    n = len(pairs)
    observed = sum(h == j for h, j in pairs) / n
    h_pass = sum(h for h, _ in pairs) / n
    j_pass = sum(j for _, j in pairs) / n
    expected = h_pass * j_pass + (1 - h_pass) * (1 - j_pass)
    if expected == 1:
        return 1.0
    return (observed - expected) / (1 - expected)


def agreement(pairs):
    return sum(h == j for h, j in pairs) / len(pairs)


def false_pass_rate(pairs):
    fails = [j for h, j in pairs if h == 0]
    return sum(fails) / len(fails) if fails else float("nan")


def percentile(values, q):
    ordered = sorted(values)
    return ordered[int(q * (len(ordered) - 1))]


def bootstrap(pairs, stat, rng):
    n = len(pairs)
    samples = [stat(rng.choices(pairs, k=n)) for _ in range(RESAMPLES)]
    low, high = percentile(samples, 0.025), percentile(samples, 0.975)
    return f"[{low:.3f}, {high:.3f}]"


def report(name, pairs):
    rng = random.Random(SEED)
    n = len(pairs)
    print(f"{name} (n={n})")
    print(f"  raw agreement    {agreement(pairs):.3f}  95% CI {bootstrap(pairs, agreement, rng)}")
    print(f"  Cohen's kappa    {cohen_kappa(pairs):.3f}  95% CI {bootstrap(pairs, cohen_kappa, rng)}")
    print(f"  false-pass rate  {false_pass_rate(pairs):.3f}  95% CI {bootstrap(pairs, false_pass_rate, rng)}")


# Lenient judge on a set where humans fail 40 of 200 outputs.
lenient = expand(tp=150, fn=10, fp=14, tn=26)
report("lenient judge, 20% human-fail rate", lenient)

# Same raw agreement, rarer failures: kappa collapses.
skewed = expand(tp=180, fn=10, fp=10, tn=0)
report("skewed set, 5% human-fail rate", skewed)

# Correct the judge's pass rate on unlabeled production data.
sens = 150 / 160  # P(judge pass | human pass)
spec = 26 / 40  # P(judge fail | human fail)
for observed in (0.70, 0.82, 0.90):
    corrected = (observed + spec - 1) / (spec + sens - 1)
    print(f"judge pass rate {observed:.2f} -> corrected true pass rate {corrected:.3f}")

# How wide is a false-pass estimate of 0.35 at different failure counts?
def wilson(p, n, z=Z):
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return centre - half, centre + half


for fails in (20, 40, 100, 200, 400):
    low, high = wilson(0.35, fails)
    print(f"human-failed cases {fails:>3}: false-pass 0.35, 95% CI [{low:.3f}, {high:.3f}], width {high - low:.3f}")
```

Run with Python 3.15.1, standard library only. The kappa values match scikit-learn 1.9.1's `cohen_kappa_score` (0.6104 and minus 0.0526).

```text
lenient judge, 20% human-fail rate (n=200)
  raw agreement    0.880  95% CI [0.835, 0.925]
  Cohen's kappa    0.610  95% CI [0.458, 0.743]
  false-pass rate  0.350  95% CI [0.206, 0.500]
skewed set, 5% human-fail rate (n=200)
  raw agreement    0.900  95% CI [0.855, 0.940]
  Cohen's kappa    -0.053  95% CI [-0.075, -0.026]
  false-pass rate  1.000  95% CI [1.000, 1.000]
judge pass rate 0.70 -> corrected true pass rate 0.596
judge pass rate 0.82 -> corrected true pass rate 0.800
judge pass rate 0.90 -> corrected true pass rate 0.936
human-failed cases  20: false-pass 0.35, 95% CI [0.181, 0.567], width 0.386
human-failed cases  40: false-pass 0.35, 95% CI [0.221, 0.505], width 0.284
human-failed cases 100: false-pass 0.35, 95% CI [0.264, 0.447], width 0.184
human-failed cases 200: false-pass 0.35, 95% CI [0.287, 0.418], width 0.131
human-failed cases 400: false-pass 0.35, 95% CI [0.305, 0.398], width 0.093
```

**Figure 3.** Similar raw agreement, opposite conclusions. On the illustrative counts, 88 percent agreement carries a kappa of 0.61, and 90 percent agreement on rare failures carries a kappa of minus 0.05.

Start with the left panel. An 88 percent agreement rate sounds fine until you see a kappa of 0.61 and a judge that passes 14 of the 40 outputs a person failed. The bootstrap interval for that kappa runs from 0.458 to 0.743, so this data cannot rule out a weak judge.

The bootstrap resamples the label pairs with replacement, recomputes the statistic each time, and reads the interval off the distribution. [SciPy's docs](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html) describe the percentile method and note that it is rarely used in practice. It appears here for clarity, and bias-corrected methods exist for small or skewed data.

Label enough failures, because the false-pass rate is only as precise as your failed-case count. At a 0.35 false-pass rate, 40 failed cases give a 95 percent interval from 0.221 to 0.505, and 400 give 0.305 to 0.398. A 200-row sample with 20 percent failures is enough to find a weak judge and too small to certify a strong one.

## Correct the pass rate with the judge's error rates

A judge's raw pass rate on unlabeled traffic is a biased estimate of the true pass rate unless the judge is perfect. [Lee et al. (2025)](https://arxiv.org/abs/2511.21140) give a correction that uses the judge's sensitivity and specificity from your human-labeled calibration set. On the illustrative labels, a measured pass rate of 0.90 corresponds to 0.936.

The estimator is `theta = (p + q0 - 1) / (q0 + q1 - 1)`, where `p` is the judge's observed pass rate. `q1` is sensitivity, the share of human-passed outputs the judge passes. `q0` is specificity, the share of human-failed outputs the judge fails.

The illustrative judge has a sensitivity of 150/160 and a specificity of 26/40. Its output above shows an imperfect judge compressing the range. A measured pass rate of 0.70 maps to a true rate of about 0.596, and 0.90 maps to about 0.936.

Bad weeks look better than they were and good weeks look worse, so trend lines on raw judge scores understate real change.

Lee et al. also build confidence intervals that account for uncertainty in both the test set and the calibration set, which the point estimate above does not do. If you report a corrected number, report its interval too, using their method.

## When a judge should not be trusted

Do not let a judge alone gate a change if code can check the output or an attacker can shape the text. Its validation must also clear a kappa lower bound of 0.60. A judge is a measurement instrument, and an unvalidated one produces numbers that look more precise than they are.

| Situation | Evidence | What to do instead |
| --- | --- | --- |
| Facts or math with no reference | Zheng et al.: default prompt failed 14 of 20 math cases | Supply a reference, compute the answer in code, or use retrieval to limit [hallucination](https://aifordevs.org/glossary#hallucination) |
| Close calls between similar versions | Shi et al.: position bias strongly tied to the quality gap | Swap orders, count ties, add human review |
| Text an attacker can influence | Raina et al., Shi et al. | Code checks and human review for adversarial input |
| Judge from the candidate's family | Panickssery et al., Wataoka et al. | Judge from another family |
| Lower bound of kappa under 0.60 | McHugh: below 0.60 is inadequate agreement | Triage only, humans decide |
| After a model, prompt, or domain change | Zheng et al.: examples may not reproduce on later GPT-4 | Re-validate on the holdout |
| Humans disagree on the rubric | Human agreement sets the ceiling | Fix the rubric first |

Attacks are a real category of failure here. [Raina et al. (2024)](https://arxiv.org/abs/2402.14016) showed short adversarial phrases can inflate judge scores and transfer to unseen models, and found absolute scoring more exposed than comparison.

[Shi et al. (2024)](https://arxiv.org/abs/2403.17710) showed optimization-based prompt injection against judges, and reported that known-answer and perplexity detection were not enough.

The threshold is an editorial choice. [McHugh (2012)](https://www.biochemia-medica.com/en/journal/22/3/10.11613/BM.2012.031) treats a kappa below 0.60 as inadequate for health research, and no comparable standard exists for LLM judges.

Use the lower end of the interval, because a point estimate near the threshold is exactly when you cannot tell. Also set a ceiling on the false-pass rate, because that error is the one that ships.

## Start with one judge and one holdout

Start your first LLM as a judge check with one criterion that a person already reviews by hand. Have that person label 200 real outputs with failures oversampled, run the validation above, and only then let the judge run on new traffic. That one cycle tells you whether a judge belongs in your pipeline.

Apply this decision rule to the holdout result:

- If the lower bound of kappa clears 0.60 and the false-pass ceiling is acceptable, let the judge gate that criterion.
- If the point estimate clears 0.60 and the lower bound does not, use the judge to rank items for human review.
- If the point estimate is under 0.60, change the rubric or keep the check with people.

Run the position swap and the length padding test before either of the first two outcomes counts. For how this slots into a release process, see the [LLM evaluation guide](https://aifordevs.org/articles/llm-evaluation) and the [evals hub](https://aifordevs.org/topics/evals).

The [context engineering guide](https://aifordevs.org/articles/context-engineering) covers the layout changes you will most often be judging. [What is an AI agent](https://aifordevs.org/articles/what-is-an-ai-agent) covers where judge checks fit in an [agent loop](https://aifordevs.org/glossary#agent-loop), and [RAG vs fine-tuning](https://aifordevs.org/articles/rag-vs-fine-tuning) covers the retrieval systems whose faithfulness you may need to grade.

## Questions

### What is LLM as a judge?

LLM as a judge means using one language model to grade another model's output against a written criterion or by comparing two answers. It replaces or supplements human review for open-ended qualities such as whether an answer is complete or faithful to its sources. Treat the result as a measurement that must be calibrated against human labels.

### Is GPT-4 as accurate as a human judge?

Zheng et al. (2023) reported 85 percent agreement between GPT-4 and humans on MT-Bench with ties excluded, above the 81 percent between humans on the same subset. With ties and inconsistent votes counted, GPT-4 agreed 66 percent of the time. Those numbers describe 2023 models on one benchmark, so measure your own judge on your own task.

### What biases affect LLM judges?

Primary studies document position bias, where the slot an answer occupies changes the verdict; verbosity bias, where longer answers win more often; and self-preference, where a judge favors text from its own model family. Zheng et al. could not determine self-enhancement bias, while later work reports it for GPT-4. Swap answer order, control for length, and use a different family.

### How do I validate an LLM judge?

Label a sample of real outputs by hand, split it into a development set and a sealed holdout, and tune the rubric only on the development set. Then score the holdout once and report raw agreement, Cohen's kappa with a bootstrap interval, and the false-pass rate. Re-validate whenever the judge model, prompt, or task changes.

### What is a good Cohen's kappa for an LLM judge?

No standard exists for LLM judges. McHugh's 2012 review of the statistic treats any kappa below 0.60 as inadequate agreement for health research, and this article borrows that floor. Use the lower end of the confidence interval and check the false-pass rate, because a kappa near 0.6 can still let a third of bad outputs through.

### Can an LLM judge be attacked?

Yes. Raina et al. (2024) showed that short universal adversarial phrases can inflate a judge's scores and transfer to other models, with absolute scoring more exposed than comparison. Shi et al. (2024) showed optimization-based prompt injection against judges and reported that tested defenses were not enough. Do not rely on a judge alone for text an attacker can write.

## Sources

1. [Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)](https://arxiv.org/abs/2306.05685)
2. [Wang et al., Large Language Models are not Fair Evaluators (2023)](https://arxiv.org/abs/2305.17926)
3. [Shi et al., Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (2024)](https://arxiv.org/abs/2406.07791)
4. [Saito et al., Verbosity Bias in Preference Labeling by Large Language Models (2023)](https://arxiv.org/abs/2310.10076)
5. [Dubois et al., Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024)](https://arxiv.org/abs/2404.04475)
6. [Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations (2024)](https://arxiv.org/abs/2404.13076)
7. [Wataoka et al., Self-Preference Bias in LLM-as-a-Judge (2024)](https://arxiv.org/abs/2410.21819)
8. [Raina et al., Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment (2024)](https://arxiv.org/abs/2402.14016)
9. [Shi et al., Optimization-based Prompt Injection Attack to LLM-as-a-Judge (2024)](https://arxiv.org/abs/2403.17710)
10. [Lee et al., How to Correctly Report LLM-as-a-Judge Evaluations (2025)](https://arxiv.org/abs/2511.21140)
11. [McHugh, Interrater reliability: the kappa statistic (2012)](https://www.biochemia-medica.com/en/journal/22/3/10.11613/BM.2012.031)
12. [scikit-learn docs: cohen_kappa_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html)
13. [SciPy docs: scipy.stats.bootstrap](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html)
14. [OpenAI docs: Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)
15. [Anthropic docs: Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)

## Cite this deep dive

AI for Devs. "LLM as a judge: biases, validation, and when to distrust it." Published November 8, 2026. https://aifordevs.org/articles/llm-as-a-judge

- Page: https://aifordevs.org/articles/llm-as-a-judge
- Markdown: https://aifordevs.org/articles/llm-as-a-judge.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "LLM evaluation: build an eval suite and gate deploys in CI"
type: "guide"
url: "https://aifordevs.org/articles/llm-evaluation"
markdown: "https://aifordevs.org/articles/llm-evaluation.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/llm-evaluation"
description: "LLM evaluation, step by step: define failure modes, build a golden dataset, choose metrics, run an offline suite, compare to a baseline, and gate CI."
author: "AI for Devs"
published: "2026-11-08"
level: "Intermediate"
topics: ["evals", "agents"]
---

# LLM evaluation: build an eval suite and gate deploys in CI

LLM evaluation, step by step: define failure modes, build a golden dataset, choose metrics, run an offline suite, compare to a baseline, and gate CI.

> **TL;DR**
>
> - LLM evaluation is a repeatable test that runs your feature on fixed inputs, scores each output, and compares the result with the last version you shipped.
> - Score each task with its own metric: exact match for classification, schema validity for extraction, recall at k for retrieval, and a validated judge for open-ended answers.
> - An overall score can hide a regression. In this guide's offline run, the pass rate rose from 14 of 17 to 15 of 17 while a must-pass case broke, and the gate blocked the deploy.
> - Detecting a 5-point drop from a 90 percent pass rate at 80 percent power takes about 686 cases per group unpaired, or about 280 paired under the assumptions in this guide.
> - All code ran on Python 3.15.1 with a scripted fake model and judge on November 8, 2026. No hosted model, real judge, or CI run was exercised.

LLM evaluation is the practice of scoring a model-backed feature against fixed test cases so you can tell whether a change helped or hurt. A prompt edit, a model upgrade, or a retrieval tweak can change outputs without failing a single unit test. This guide builds the suite in the order you need it: failure modes, golden data, metrics, a runner, a baseline comparison, and a CI gate. It also covers agent evaluation and RAG evaluation, which need their own scoring logic.

## What LLM evaluation measures

LLM evaluation measures whether your feature meets a written standard on realistic inputs, and whether a change moved that result. Public benchmarks rank models on someone else's task, so they cannot tell you that your prompt edit broke refund handling. An eval suite answers that question for your code.

**Figure 1.** Evaluation is a loop. Production traffic feeds the golden set, each change is scored against the last shipped run, and new failures become new cases.

An [eval](https://aifordevs.org/glossary#eval) here means a test you can rerun, with stored inputs, a grader, and a score. [Rankings of general model quality](https://aifordevs.org/articles/best-llm-for-coding) help you shortlist models, and your own suite decides whether one of them works for you.

Keep the suite in your own repository, as data and code. As of November 8, 2026, OpenAI's [evaluation guide](https://developers.openai.com/api/docs/guides/evaluation-best-practices) says its hosted Evals platform would become read-only for existing users on October 31, 2026. The same notice schedules shutdown for November 30, 2026, so your golden data should live somewhere you control.

## Define the task and its failure modes

Write the task as one sentence, then read 30 to 50 real outputs and group what went wrong. Failure modes come from reading outputs. [Shankar et al. (2024)](https://arxiv.org/abs/2404.12272) call this criteria drift, because grading outputs is what teaches you the criteria.

This guide uses an illustrative billing support assistant. It routes tickets, extracts refund details, retrieves runbooks, and answers questions.

For each failure mode, record how bad it is. A wrong routing label costs a support agent a minute, and an invented refund policy costs money and trust. Severity decides which cases can never be allowed to fail.

| Failure mode | Example | Grader | Severity |
| --- | --- | --- | --- |
| Wrong queue | Refund ticket routed to how-to | Exact match | Medium |
| Malformed extraction | `amount` returned as a string | Schema check | High |
| Retrieval miss | Approval runbook not in the top 3 | Recall at k | High |
| Missing requirement | Answer omits the second-approver rule | Judged, must-pass | Blocks deploy |
| Invented policy | Answer cites a refund window that does not exist | Judged against sources | Blocks deploy |

## Assemble a golden dataset from real traffic

A [golden dataset](https://aifordevs.org/glossary#golden-dataset) is a versioned set of real inputs paired with the expected behavior for each one. Sample it from production logs, because OpenAI's guide warns against eval data that does not reproduce production traffic, and hand-written cases skew toward failures you already imagined.

Build the first version in this order:

1. Log inputs, outputs, retrieved context, and tool calls for every request.
2. Sample a random slice to capture the real mix of requests.
3. Add a failure slice from escalations, negative feedback, and bug reports.
4. Remove secrets and personal data before anything enters the repository.
5. Have someone who knows the domain write the expected behavior.
6. Store each case with an ID, task type, input, expected result, tags, and source.
7. Hold back a slice you never look at while tuning prompts.

Start smaller than you think. Anthropic says [20 to 50 tasks drawn from real failures](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) is a good start for an agent suite, because early changes tend to have large effects.

Grow it by rule. When a bug reaches production, add a failing case before you fix the prompt. [Anthropic's eval docs](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) favor volume, saying more cases with slightly lower-signal automated grading beat fewer hand-graded ones.

## Choose LLM evaluation metrics per task type

Pick one primary metric per task type and prefer code over models. Exact match and schema checks are free and deterministic, so they can run on every commit. Use a judged metric only where the answer is open-ended, and validate that judge before it can block a release.

| Task type | Primary metric | How it is computed | Watch for |
| --- | --- | --- | --- |
| Classification, routing | Exact match | Compare the normalized output with the expected label | Rare labels vanish in the average, so report per-class results |
| Extraction | Schema validity | Parse JSON, then check types and allowed values | Valid JSON can still hold wrong values, so add field accuracy |
| Retrieval | Recall at k | Share of relevant documents found in the top k | Only as good as your relevance labels |
| Open-ended answers | Judged pass or fail, per criterion | A validated judge returns a verdict for one criterion | Position, verbosity, and self-preference bias |
| Agents | Outcome, plus rule checks on the path | Compare the final state with a goal state | Step-order checks break on harmless changes |

Retrieval recall is the [fraction of relevant documents retrieved among all relevant documents](https://lightning.ai/docs/torchmetrics/stable/retrieval/recall), counted over the top k results. [Structured outputs](https://aifordevs.org/glossary#structured-outputs) reduce schema failures and leave field accuracy untouched, so test both.

Prefer binary verdicts to 1 to 5 scores when you can. [OpenAI's guide](https://developers.openai.com/api/docs/guides/evaluation-best-practices) recommends pairwise or pass/fail formats for reliability. Track latency and cost next to these metrics as separate limits, so that a slower or pricier change cannot hide inside an accuracy gain.

## Run LLM evals offline

The runner below scores 17 cases across four task types, compares a candidate against a baseline, and exits non-zero when the gate blocks. It uses only the Python standard library and a scripted fake model, so you can run it without an API key and replace the fake with your own client later.

```python title="run_evals.py"
#!/usr/bin/env python3
"""Offline LLM eval runner. Standard library only.

Usage: python3 run_evals.py [candidate|fixed]
Compares a scripted candidate against a scripted baseline, prints the
per-task report, and exits 1 if the deploy gate blocks.
"""
import json
import math
import sys
from collections import defaultdict

# Golden dataset. In your repo, keep this as a versioned golden.jsonl.
# These cases are illustrative; yours should come from sampled traffic.
GOLDEN = [
    {"id": "C1", "task": "classify", "input": "Charged twice in October", "expected": "billing"},
    {"id": "C2", "task": "classify", "input": "App crashes on login", "expected": "bug"},
    {"id": "C3", "task": "classify", "input": "How do I export invoices?", "expected": "how_to"},
    {"id": "C4", "task": "classify", "input": "Refund my annual plan", "expected": "billing"},
    {"id": "C5", "task": "classify", "input": "SSO setup fails with 403", "expected": "bug"},
    {"id": "E1", "task": "extract", "input": "Refund $40 on order A-17"},
    {"id": "E2", "task": "extract", "input": "Please return 25 EUR for B-09"},
    {"id": "E3", "task": "extract", "input": "Refund 18.50 GBP, order C-33"},
    {"id": "E4", "task": "extract", "input": "Refund $5 on order D-21"},
    {"id": "R1", "task": "retrieve", "input": "refund approval limits", "relevant": ["approvals", "refund-policy"]},
    {"id": "R2", "task": "retrieve", "input": "SSO error codes", "relevant": ["sso-errors"]},
    {"id": "R3", "task": "retrieve", "input": "invoice export formats", "relevant": ["export", "invoice-api"]},
    {"id": "R4", "task": "retrieve", "input": "dunning retry schedule", "relevant": ["dunning", "retries"]},
    {"id": "A1", "task": "answer", "input": "Can I get a refund after 30 days?", "rubric": ["within 30 days"]},
    {"id": "A2", "task": "answer", "input": "Why was my card charged twice?", "rubric": ["authorization"]},
    {"id": "A3", "task": "answer", "input": "Who approves a $600 refund?", "rubric": ["second approver"], "must_pass": True},
    {"id": "A4", "task": "answer", "input": "How long do dunning retries last?", "rubric": ["14 days"]},
]

REFUND_SCHEMA = {"order_id": str, "amount": (int, float), "currency": str}
CURRENCIES = {"USD", "EUR", "GBP"}
K = 3

# Scripted fake model outputs. Replace complete() with a real API call.
BASELINE = {
    "C1": "billing", "C2": "bug", "C3": "how_to", "C4": "how_to", "C5": "bug",
    "E1": '{"order_id": "A-17", "amount": 40, "currency": "USD"}',
    "E2": '{"order_id": "B-09", "amount": 25, "currency": "EUR"}',
    "E3": '{"order_id": "C-33", "amount": "18.50 GBP"}',
    "E4": '{"order_id": "D-21", "amount": 5, "currency": "USD"}',
    "R1": ["approvals", "refund-policy", "faq"],
    "R2": ["sso-errors", "login", "faq"],
    "R3": ["export", "faq", "billing-api"],
    "R4": ["dunning", "retries", "faq"],
    "A1": "You can request a refund within 30 days of purchase.",
    "A2": "Two authorization holds posted; one drops off in 3 days.",
    "A3": "Refunds over $500 need a second approver in finance.",
    "A4": "Dunning retries run for 14 days, then the plan pauses.",
}
CANDIDATE = BASELINE | {
    "C4": "billing",
    "E3": '{"order_id": "C-33", "amount": 18.5, "currency": "GBP"}',
    "R3": ["export", "invoice-api", "faq"],
    "R4": ["dunning", "faq", "billing-api"],
    "A3": "Refunds over $500 are approved by the finance team.",
}
FIXED = CANDIDATE | {
    "R4": ["dunning", "retries", "faq"],
    "A3": "Refunds over $500 need a second approver in finance.",
}


class ScriptedModel:
    """Stands in for a real client."""

    def __init__(self, name, outputs):
        self.name, self.outputs = name, outputs

    def complete(self, case):
        return self.outputs[case["id"]]


class ScriptedJudge:
    """Stands in for an LLM judge: passes if every rubric phrase appears."""

    def grade(self, case, output):
        missing = [p for p in case["rubric"] if p not in output.lower()]
        return not missing, f"missing {missing}" if missing else "ok"


def exact_match(case, output, _judge):
    ok = output.strip().lower() == case["expected"].lower()
    return float(ok), f"got {output!r}"


def schema_valid(case, output, _judge):
    try:
        data = json.loads(output)
    except json.JSONDecodeError as err:
        return 0.0, f"invalid JSON: {err}"
    for field, kind in REFUND_SCHEMA.items():
        if not isinstance(data.get(field), kind):
            return 0.0, f"bad field {field}: {data.get(field)!r}"
    if data["currency"] not in CURRENCIES:
        return 0.0, f"unknown currency {data['currency']}"
    return 1.0, "ok"


def recall_at_k(case, output, _judge):
    relevant = set(case["relevant"])
    score = len(relevant & set(output[:K])) / len(relevant)
    return score, f"recall@{K}={score:.2f}"


def judged(case, output, judge):
    passed, why = judge.grade(case, output)
    return float(passed), why


METRICS = {
    "classify": ("exact_match", exact_match, 0.90),
    "extract": ("schema_valid", schema_valid, 0.95),
    "retrieve": (f"recall@{K}", recall_at_k, 0.85),
    "answer": ("judged_pass", judged, 0.75),
}


def run(model, judge):
    results = {}
    for case in GOLDEN:
        _name, grader, _floor = METRICS[case["task"]]
        score, note = grader(case, model.complete(case), judge)
        results[case["id"]] = {"score": score, "passed": score == 1.0, "note": note}
    return results


def mcnemar_exact_p(new_fail, new_pass):
    """Two-sided exact McNemar: the discordant pairs are Binomial(n, 0.5)."""
    n = new_fail + new_pass
    if n == 0:
        return 1.0
    tail = sum(math.comb(n, i) for i in range(min(new_fail, new_pass) + 1)) / 2**n
    return min(1.0, 2 * tail)


def compare(base, cand):
    by_task = defaultdict(list)
    for case in GOLDEN:
        by_task[case["task"]].append(case)
    blockers, rows = [], []
    for task, cases in by_task.items():
        metric, _grader, floor = METRICS[task]
        ids = [c["id"] for c in cases]
        b_mean = sum(base[i]["score"] for i in ids) / len(ids)
        c_mean = sum(cand[i]["score"] for i in ids) / len(ids)
        new_fail = [i for i in ids if base[i]["passed"] and not cand[i]["passed"]]
        new_pass = [i for i in ids if not base[i]["passed"] and cand[i]["passed"]]
        rows.append((task, metric, b_mean, c_mean, new_fail, new_pass))
        if c_mean < floor:
            blockers.append(f"{task}: {metric} {c_mean:.3f} is below the floor {floor:.2f}")
        if len(new_fail) > len(new_pass):
            blockers.append(f"{task}: net regression, new failures {new_fail}")
    for case in GOLDEN:
        if case.get("must_pass") and not cand[case["id"]]["passed"]:
            blockers.append(f"must-pass case {case['id']} failed: {cand[case['id']]['note']}")
    return rows, blockers


def main():
    name = sys.argv[1] if len(sys.argv) > 1 else "candidate"
    outputs = {"candidate": CANDIDATE, "fixed": FIXED}[name]
    judge = ScriptedJudge()
    base = run(ScriptedModel("baseline", BASELINE), judge)
    cand = run(ScriptedModel(name, outputs), judge)
    rows, blockers = compare(base, cand)

    print(f"baseline vs {name}, {len(GOLDEN)} cases")
    print(f"{'task':<9}{'metric':<14}{'baseline':>9}{'cand':>7}  new_fail  new_pass")
    for task, metric, b_mean, c_mean, new_fail, new_pass in rows:
        print(f"{task:<9}{metric:<14}{b_mean:>9.3f}{c_mean:>7.3f}  "
              f"{','.join(new_fail) or '-':<9} {','.join(new_pass) or '-'}")

    b_pass = sum(r["passed"] for r in base.values())
    c_pass = sum(r["passed"] for r in cand.values())
    new_fail = sum(base[i]["passed"] and not cand[i]["passed"] for i in base)
    new_pass = sum(cand[i]["passed"] and not base[i]["passed"] for i in base)
    print(f"overall pass rate: {b_pass}/{len(base)} -> {c_pass}/{len(cand)}")
    print(f"exact McNemar p = {mcnemar_exact_p(new_fail, new_pass):.3f} "
          f"({new_fail} new failures, {new_pass} new passes)")

    print("gate: BLOCK" if blockers else "gate: PASS")
    for blocker in blockers:
        print(f"  - {blocker}")
    sys.exit(1 if blockers else 0)


if __name__ == "__main__":
    main()
```

Run it twice, once for a candidate with a regression and once for a fixed version. Tested with Python 3.15.1 on November 8, 2026:

```text
$ python3 run_evals.py candidate
baseline vs candidate, 17 cases
task     metric         baseline   cand  new_fail  new_pass
classify exact_match       0.800  1.000  -         C4
extract  schema_valid      0.750  1.000  -         E3
retrieve recall@3          0.875  0.875  R4        R3
answer   judged_pass       1.000  0.750  A3        -
overall pass rate: 14/17 -> 15/17
exact McNemar p = 1.000 (2 new failures, 3 new passes)
gate: BLOCK
  - answer: net regression, new failures ['A3']
  - must-pass case A3 failed: missing ['second approver']
$ echo $?
1
$ python3 run_evals.py fixed
baseline vs fixed, 17 cases
task     metric         baseline   cand  new_fail  new_pass
classify exact_match       0.800  1.000  -         C4
extract  schema_valid      0.750  1.000  -         E3
retrieve recall@3          0.875  1.000  -         R3
answer   judged_pass       1.000  1.000  -         -
overall pass rate: 14/17 -> 17/17
exact McNemar p = 0.250 (0 new failures, 3 new passes)
gate: PASS
$ echo $?
0
```

This run exercised the dataset loop, all four graders, the paired comparison, the exact McNemar calculation, the gate rules, and both exit codes. It did not exercise a hosted model, a real LLM judge, retries, rate limits, concurrency, or real run-to-run variation.

The `ScriptedJudge` is a phrase matcher. Replace it with a judge you have validated, using the method in [LLM as a judge](https://aifordevs.org/articles/llm-as-a-judge), and replace `ScriptedModel.complete` with your client.

## Compare against a baseline

Compare the candidate with the last shipped run case by case, because the same overall score can hide a swap. In the first run, recall at 3 stayed at 0.875 while R3 recovered and R4 broke. Only the paired list of new failures and new passes shows that.

The overall pass rate rose from 14 of 17 to 15 of 17. A gate on the average would have shipped this candidate. The per-task view shows the answer task losing A3, the must-pass case about refund approval, and that is why the gate blocked.

In CI, produce the baseline by running the suite on your main branch and on the change, against the same golden file. If that costs too much, cache baseline scores keyed by a hash of the golden file, the model ID, and the prompt. Re-score whenever any of those change.

The printed p-value comes from an [exact McNemar test](https://www.statsmodels.org/stable/generated/statsmodels.stats.contingency_tables.mcnemar.html), which uses only the cases that changed, under a binomial distribution. At p equal to 1.000 on 2 new failures and 3 new passes, a 17-case suite cannot separate this change from noise. Treat the p-value as a reminder to read the case list.

If you sample at a temperature above zero, the same case can pass and fail on different runs. Run each case several times and count a flaky case as a failure until you have fixed it.

## Sample size for LLM evals

You need far more cases to measure a small drop than to catch a large one. At a 90 percent pass rate, a paired design needs about 280 cases to detect a 5-point drop with 80 percent power. A 17-case suite can only flag specific must-pass cases.

This script computes [Wilson intervals](https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm) and sample sizes, then checks the paired formula by simulating an exact McNemar test.

```python title="sample_size.py"
#!/usr/bin/env python3
"""Sample-size math for LLM evals. Standard library only."""
import math
import random
from statistics import NormalDist

Z = NormalDist().inv_cdf
ALPHA, POWER = 0.05, 0.80
Z_A, Z_B = Z(1 - ALPHA / 2), Z(POWER)


def wilson(p, n, z=Z_A):
    centre = (p + z**2 / (2 * n)) / (1 + z**2 / n)
    half = z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / (1 + z**2 / n)
    return centre - half, centre + half


def unpaired_n(p1, p2):
    bar = (p1 + p2) / 2
    num = Z_A * math.sqrt(2 * bar * (1 - bar)) + Z_B * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
    return math.ceil(num**2 / (p1 - p2) ** 2)


def paired_n(delta, flip_up):
    """McNemar sample size. b = delta + flip_up flips down, flip_up flips up."""
    psi = delta + 2 * flip_up
    num = Z_A * math.sqrt(psi) + Z_B * math.sqrt(psi - delta**2)
    return math.ceil(num**2 / delta**2)


def exact_mcnemar_p(b, c):
    n = b + c
    if n == 0:
        return 1.0
    tail = sum(math.comb(n, i) for i in range(min(b, c) + 1)) / 2**n
    return min(1.0, 2 * tail)


def simulate_power(n, delta, flip_up, trials=4000, seed=7):
    rng = random.Random(seed)
    down, up = delta + flip_up, flip_up
    hits = 0
    for _ in range(trials):
        b = c = 0
        for _ in range(n):
            u = rng.random()
            if u < down:
                b += 1
            elif u < down + up:
                c += 1
        if b > c and exact_mcnemar_p(b, c) < ALPHA:
            hits += 1
    return hits / trials


print("Wilson 95% interval when the observed pass rate is 90%")
for n in (50, 100, 200, 400, 800, 1600):
    low, high = wilson(0.90, n)
    print(f"  n={n:>5}: {low:.3f} to {high:.3f} (width {high - low:.3f})")

print("\nCases to detect a drop from a 90% pass rate (alpha 0.05, power 0.80)")
print("  drop  unpaired per arm  paired (1.5% flip up)")
for drop in (0.10, 0.05, 0.03, 0.02):
    print(f"  {drop:.2f}  {unpaired_n(0.90, 0.90 - drop):>16}  {paired_n(drop, 0.015):>21}")

n = paired_n(0.05, 0.015)
print(f"\nSimulated power of exact McNemar, drop 0.05, n={n}: {simulate_power(n, 0.05, 0.015):.3f}")
for try_n in (260, 280, 300):
    print(f"Simulated power, n={try_n}: {simulate_power(try_n, 0.05, 0.015):.3f}")

print("\nOne-sided 95% upper bound on the failure rate after zero failures in n cases")
for n in (30, 100, 300, 1000):
    print(f"  n={n:>5}: {1 - 0.05 ** (1 / n):.4f}")
```

Tested with Python 3.15.1. The Wilson intervals and exact p-values match SciPy 1.18.1 on the cases checked.

```text
Wilson 95% interval when the observed pass rate is 90%
  n=   50: 0.786 to 0.957 (width 0.170)
  n=  100: 0.826 to 0.945 (width 0.119)
  n=  200: 0.851 to 0.934 (width 0.084)
  n=  400: 0.867 to 0.926 (width 0.059)
  n=  800: 0.877 to 0.919 (width 0.042)
  n= 1600: 0.884 to 0.914 (width 0.029)

Cases to detect a drop from a 90% pass rate (alpha 0.05, power 0.80)
  drop  unpaired per arm  paired (1.5% flip up)
  0.10               199                    100
  0.05               686                    249
  0.03              1774                    521
  0.02              3841                    979

Simulated power of exact McNemar, drop 0.05, n=249: 0.769
Simulated power, n=260: 0.779
Simulated power, n=280: 0.817
Simulated power, n=300: 0.855

One-sided 95% upper bound on the failure rate after zero failures in n cases
  n=   30: 0.0950
  n=  100: 0.0295
  n=  300: 0.0099
  n= 1000: 0.0030
```

| Drop to detect | Unpaired, cases per group | Paired, 1.5 percent flip up |
| --- | --- | --- |
| 10 points | 199 | 100 |
| 5 points | 686 | 249 |
| 3 points | 1,774 | 521 |
| 2 points | 3,841 | 979 |

Read this as a planning table. The paired columns assume 1.5 percent of cases flip from fail to pass, which is an assumption made for this article.

Measure your own rate by running the same baseline twice and counting flips. Pairing helps because only the cases that changed carry information.

The formula is an approximation. At the 5-point drop, the formula's 249 cases simulated to 0.769 power, and it took about 280 to pass 0.80. Zero failures are weaker evidence than they look, since 300 clean passes bound the failure rate only below 1 percent at one-sided 95 percent confidence.

Use these sizes to set expectations. Up to about 50 cases, trust must-pass cases and individual regressions, and read averages as direction only. Around 300 paired cases per task, a 5-point drop becomes detectable.

## Gate a deploy in CI

A CI gate is the eval runner's exit code wired to a required check. GitHub Actions fails a step when the last command exits non-zero, so the gate needs no plugin. The runner above exited 1 for the regressed candidate and 0 for the fixed one.

```yaml title=".github/workflows/evals.yml"
name: evals
on:
  pull_request:

jobs:
  eval-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.15"
      - name: Run the eval gate
        run: python3 evals/run_evals.py candidate
```

This workflow was not run. The action versions came from the [actions/checkout](https://github.com/actions/checkout) and [actions/setup-python](https://github.com/actions/setup-python) repositories on November 8, 2026. Only the YAML was parsed.

GitHub documents [how a `run` step reports status](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax): the runner marks the step failed or succeeded from the exit code, and `continue-on-error` prevents a job from failing when a step fails. Leave that off the gate. Non-zero exits also [fail the action](https://docs.github.com/en/actions/how-tos/create-and-publish-actions/set-exit-codes) if you package this as one.

Choose gates in this order:

1. Must-pass cases, where a single failure blocks.
2. Per-task floors, for the deterministic metrics.
3. Net regressions per task, new failures minus new passes.
4. Judged metrics, report-only until the judge is validated.

Run deterministic graders on every pull request and slower judged metrics on a schedule or before release. GitHub warns that a workflow skipped by a path filter leaves its checks pending, which can block a pull request that requires them. If you make this check required, do not add a `paths` filter without a job that always reports a status.

## Agent evaluation: outcome first, trajectory second

Agent evaluation should grade the final state first and the path second. [Anthropic's agent eval guide](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) calls exact step-order checks brittle, and tau-bench compares the final database state with an annotated goal state. Add trajectory checks only for rules that must hold on every path.

An [agent loop](https://aifordevs.org/glossary#agent-loop) can reach the right result by many routes, so an exact sequence check fails on harmless variation. An outcome check alone misses a policy violation that happens to end well. This script grades four scripted runs with both, and compares them with the agent's own claim of success.

```python title="agent_check.py"
#!/usr/bin/env python3
"""Grade scripted agent runs by outcome and by trajectory rules."""
GOAL = {"R-1042": "refunded"}
EXPECTED_SEQUENCE = ["get_order", "verify_identity", "issue_refund"]

RUNS = {
    "T1": (["get_order", "verify_identity", "issue_refund"], {"R-1042": "refunded"}, "Refund issued."),
    "T2": (["verify_identity", "get_order", "issue_refund"], {"R-1042": "refunded"}, "Refund issued."),
    "T3": (["get_order", "issue_refund", "verify_identity"], {"R-1042": "refunded"}, "Refund issued."),
    "T4": (["get_order", "verify_identity"], {"R-1042": "open"}, "Refund issued."),
}


def outcome_passed(final_state):
    return final_state == GOAL


def exact_sequence_passed(calls):
    return calls == EXPECTED_SEQUENCE


def policy_passed(calls):
    """Identity check must happen before any refund."""
    if "issue_refund" not in calls:
        return True
    return "verify_identity" in calls[: calls.index("issue_refund")]


def text_claims_success(reply):
    return "refund issued" in reply.lower()


print(f"{'run':<4}{'reply says done':<17}{'outcome':<9}{'exact seq':<11}{'policy':<8}verdict")
for run, (calls, state, reply) in RUNS.items():
    outcome, policy = outcome_passed(state), policy_passed(calls)
    verdict = "pass" if outcome and policy else "fail"
    print(f"{run:<4}{str(text_claims_success(reply)):<17}{str(outcome):<9}"
          f"{str(exact_sequence_passed(calls)):<11}{str(policy):<8}{verdict}")
```

Tested with Python 3.15.1:

```text
run reply says done  outcome  exact seq  policy  verdict
T1  True             True     True       True    pass
T2  True             True     False      True    pass
T3  True             True     False      False   fail
T4  True             False    False      True    fail
```

T2 fails the exact-sequence check even though it did the right thing in a different order. T3 reaches the goal but issues the refund before verifying identity, which a final-state check cannot see. T4 replies that the refund happened while the database says it did not, which is why you grade state and ignore the reply.

Outcome checks are how the benchmarks work. [SWE-bench](https://arxiv.org/abs/2310.06770) applies a generated patch and runs the repository's tests, and [tau-bench](https://arxiv.org/abs/2406.12045) checks the database. Tau-bench also reported in 2024 that gpt-4o solved fewer than 50 percent of tasks, with pass^8 below 25 percent in its retail domain.

Report both rates for an agent. Anthropic's guide defines pass@k as at least one success in k attempts, and pass^k as success on all k attempts. Use pass^k for customer-facing agents, where one bad run in eight is a real incident.

See [what is an AI agent](https://aifordevs.org/articles/what-is-an-ai-agent) and the [agents hub](https://aifordevs.org/topics/agents) for the loop itself.

## RAG evaluation: score retrieval and generation separately

RAG evaluation splits one score into two, because a bad answer can come from the retriever or the generator. Measure recall at k on labeled documents first. Then measure whether the answer stays faithful to the retrieved text, using a judge you have validated against human labels.

| Recall at k | Faithfulness | Likely cause | Fix first |
| --- | --- | --- | --- |
| Low | High | Retriever misses the answer | Chunking, hybrid search, reranking |
| High | Low | Generator ignores or misreads the context | Prompt, context order, model |
| Low | Low | Both stages are failing | Retriever, then generator |
| High | High, answers still wrong | Stale labels or stale documents | Re-check labels and corpus dates |

[Ragas (Es et al., 2023)](https://arxiv.org/abs/2309.15217) defines faithfulness as the share of an answer's statements that can be inferred from the retrieved context. It also scores answer relevance and context relevance. Those metrics call a language model internally, so they inherit judge bias and need the same validation.

Label a few hundred question-to-document pairs by hand, and treat them as part of the golden dataset. [Hybrid search](https://aifordevs.org/glossary#hybrid-search), [reranking](https://aifordevs.org/glossary#reranking), and new [chunking](https://aifordevs.org/glossary#chunking) choices should each move recall at k before you expect answers to improve.

Retrieval order matters too. The [context engineering guide](https://aifordevs.org/articles/context-engineering) covers what position research found, and the [vector database guide](https://aifordevs.org/articles/what-is-a-vector-database) and [RAG vs fine-tuning](https://aifordevs.org/articles/rag-vs-fine-tuning) cover the retrieval stack.

## Start your first LLM evaluation this week

Start your first LLM evaluation with the last ten bugs your feature shipped. Write each as a golden case, mark the worst as must-pass, and wire the exit code into CI before you add a single judged metric. That suite is small, and it already blocks the regressions you have paid for once.

Then grow the gate as the dataset grows:

- Under 50 cases: block on must-pass cases only, and report everything else.
- 50 to 200 cases: add per-task floors and net-regression checks.
- Over 200 paired cases per task: add a significance check on aggregate changes.
- Any judged metric: keep it report-only until it passes the human-label validation in [LLM as a judge](https://aifordevs.org/articles/llm-as-a-judge).

The [evals hub](https://aifordevs.org/topics/evals) collects the rest of the reading on this topic.

## Questions

### What is LLM evaluation?

LLM evaluation is a repeatable test of a feature that uses a language model. It runs the feature on fixed inputs, scores each output with a metric chosen for that task, and compares the scores with the last version you shipped. A good suite tells you which cases got worse and blocks a release when a critical case fails.

### What are the main LLM evaluation metrics?

Use exact match for classification, schema validity for extraction, recall at k and precision at k for retrieval, and a validated judge for open-ended answers. Agents add outcome checks against the final state. Pick one primary metric per task type and treat latency and cost as separate limits, so one averaged score cannot hide a regression.

### How many test cases do I need for LLM evals?

Start with 20 to 50 cases drawn from real failures, which Anthropic says is enough for an early agent suite. To measure a 5-point drop from a 90 percent pass rate with 80 percent power, this guide calculated about 280 paired cases, or 686 per group if the runs are unpaired, under the stated assumptions.

### What is a golden dataset in LLM evaluation?

A golden dataset is a versioned set of real inputs paired with the expected behavior for each one. Build it from sampled production traffic plus known failures, label it with people who know the domain, and keep part of it out of prompt tuning. Treat each change to the file as a change to your test suite.

### Should I use an LLM judge or code-based checks?

Use code-based checks wherever the correct behavior can be written as a rule, because they are free, fast, and deterministic. Use an LLM judge for open-ended qualities such as whether an answer stays faithful to its sources. Validate the judge against human labels before it can block a deploy, and keep it report-only until it passes.

### How do I run LLM evals in CI?

Write a script that runs the suite and exits non-zero when the gate blocks, then call it from a required check. GitHub Actions marks a bash step as failed when the last command exits non-zero. Run deterministic graders on every pull request and run slower judged metrics on a schedule or before release, so the gate stays fast.

## Sources

1. [Anthropic docs: Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)
2. [Anthropic, Demystifying evals for AI agents (2026)](https://www.anthropic.com/research/demystifying-evals-for-ai-agents)
3. [OpenAI docs: Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)
4. [Shankar et al., Who Validates the Validators? (2024)](https://arxiv.org/abs/2404.12272)
5. [Yao et al., tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024)](https://arxiv.org/abs/2406.12045)
6. [Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023)](https://arxiv.org/abs/2310.06770)
7. [Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation (2023)](https://arxiv.org/abs/2309.15217)
8. [TorchMetrics docs: Retrieval recall](https://lightning.ai/docs/torchmetrics/stable/retrieval/recall)
9. [NIST/SEMATECH e-Handbook: Confidence intervals for a proportion](https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm)
10. [statsmodels docs: McNemar test](https://www.statsmodels.org/stable/generated/statsmodels.stats.contingency_tables.mcnemar.html)
11. [GitHub docs: Workflow syntax for GitHub Actions](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax)
12. [GitHub docs: Setting exit codes for actions](https://docs.github.com/en/actions/how-tos/create-and-publish-actions/set-exit-codes)
13. [actions/checkout on GitHub](https://github.com/actions/checkout)
14. [actions/setup-python on GitHub](https://github.com/actions/setup-python)

## Cite this guide

AI for Devs. "LLM evaluation: build an eval suite and gate deploys in CI." Published November 8, 2026. https://aifordevs.org/articles/llm-evaluation

- Page: https://aifordevs.org/articles/llm-evaluation
- Markdown: https://aifordevs.org/articles/llm-evaluation.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script"
type: "guide"
url: "https://aifordevs.org/articles/lora-fine-tuning"
markdown: "https://aifordevs.org/articles/lora-fine-tuning.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/lora-fine-tuning"
description: "LoRA fine tuning from the paper to serving: r, alpha, QLoRA, dataset format, and a PEFT script run on CPU with real output, adapter size, and evals."
author: "AI for Devs"
published: "2026-11-08"
level: "Intermediate"
topics: ["fine-tuning", "models"]
---

# LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script

LoRA fine tuning from the paper to serving: r, alpha, QLoRA, dataset format, and a PEFT script run on CPU with real output, adapter size, and evals.

> **TL;DR**
>
> - LoRA fine tuning freezes a model's weights and trains a small update, `(alpha / r) * B @ A`, on selected linear layers. In this guide's run, one 576 by 576 weight matrix got 9,216 trainable numbers instead of 331,776.
> - The PEFT script on this page ran on an Apple M2 CPU on November 8, 2026, with Python 3.13.18, PyTorch 2.15.0, Transformers 5.22.2, PEFT 0.21.2, and Accelerate 1.15.0. It trained 460,800 parameters of SmolLM2-135M and wrote a 1.86 MB adapter.
> - That run proves the API and the mechanics. It does not demonstrate model quality, because a 135M model on a synthetic task is only a smoke test.
> - QLoRA (Dettmers et al., 2023) keeps the frozen weights in 4-bit NF4 and attaches adapters to all linear layers. The paper reports fine tuning a 65B model on one 48 GB GPU.
> - Merge the adapter when one task owns the deployment, and load adapters at runtime when several tasks share a base model. vLLM's default `--max-lora-rank` is 16, so raise it for `r=64` adapters.

LoRA fine tuning trains a small low-rank update on top of frozen model weights, so a task-specific model costs megabytes instead of a full copy of the weights. Hu et al. introduced [LoRA](https://aifordevs.org/glossary#lora) in 2021 and reported 10,000 times fewer trainable parameters than full [fine tuning](https://aifordevs.org/glossary#fine-tuning) on GPT-3 175B. This guide starts from that paper's equation, covers QLoRA, and walks through a Hugging Face PEFT training script that was executed for this page. Every command that was not run is labeled.

## How LoRA fine tuning changes a weight matrix

LoRA fine tuning leaves the original weight matrix `W0` frozen and trains two small matrices whose product is added to it. For a matrix of shape `d x k`, `B` has shape `d x r`, `A` has shape `r x k`, and only those two receive gradients. The update is `(alpha / r) * B @ A`.

[Hu et al.](https://arxiv.org/abs/2106.09685) initialize `A` with Gaussian noise and `B` with zeros, so the update is exactly zero before training starts. PEFT reaches the same starting point, because its [`LoraConfig` reference](https://huggingface.co/docs/peft/main/en/package_reference/lora) says the default initialization sets `B` to 0 and makes an untrained adapter a no-op.

**Figure 1.** The frozen matrix and the trainable product run side by side and their outputs are summed. The whole correction fits through an r-sized bottleneck, which caps the rank of the update at r.

The trainable count for one layer is `r * (d + k)`. For the 576 by 576 layer in Figure 1, that is 9,216 numbers against 331,776 in `W0`, or 2.8 percent.

Hu et al. report a checkpoint falling from about 350 GB to 35 MB on GPT-3 175B, with `r = 4` on the query and value matrices. Training VRAM fell from 1.2 TB to 350 GB.

A small rank is often enough. In Table 6 of the paper, adapting the query and value matrices scored between 73.3 and 73.8 percent on WikiSQL across ranks 1, 2, 4, 8, and 64. The authors add in a footnote that they do not expect a small `r` to work for every task.

## What r and alpha do in a LoRA config

`r` sets how much the update can express, and `alpha` scales it. PEFT multiplies the update by `lora_alpha / r`, or by `lora_alpha / sqrt(r)` with `use_rslora=True`. Raising `r` adds parameters and file size, and Hu et al. note that tuning `alpha` is roughly the same as tuning the learning rate.

Published settings vary, so treat any single default as a starting point.

| Source | `r` | `alpha` | Scale | Adapters on |
| --- | --- | --- | --- | --- |
| Hu et al., GPT-3 175B | 4 | Set to the first `r` tried, then not tuned | 1 at that first `r` | Query and value |
| Dettmers et al., QLoRA | 64 | 16 | 0.25 | All linear layers |
| PEFT 0.21.2 default | 8 | 8 | 1 | Chosen by architecture |
| Run in this guide | 8 | 16 | 2 | `q_proj`, `v_proj` |

[Shuttleworth et al.](https://arxiv.org/abs/2410.21228) compared `alpha = 2r` with a fixed `alpha = 8` and report better generalization at `alpha = 2r`, particularly at higher ranks. That result is why the run below uses `r=8` and `alpha=16` instead of PEFT's default of 8 and 8.

Set `r=8` and `alpha=16` for a first run, and change `r` last. Dettmers et al. report that `r` had no relationship with final performance once adapters covered all layers, so adding modules comes before adding rank.

## Which modules get adapters

Adapters attach to linear layers, and on large models the choice of layers matters more than `r`. Hu et al. adapted only the attention query and value matrices in most experiments. [Dettmers et al.](https://arxiv.org/abs/2305.14314) report that this placement did not replicate full fine tuning on large base models and that adapters on all linear layers were required.

PEFT exposes the choice through `target_modules`. If you leave it empty, PEFT picks by architecture, and in PEFT 0.21.2 that selected `q_proj` and `v_proj` for the Llama-architecture model used below. The string `"all-linear"` selects every linear layer except the output layer.

**Figure 2.** Moving from query and value matrices to all linear layers multiplies this adapter's trainable count by 5.3. The feed-forward layers add the most because they are the widest.

These counts come from calling `get_peft_model` on SmolLM2-135M, which has 134,515,008 parameters. Sizes are `trainable * 4 bytes` for 32-bit floats, before the file header.

| Target modules | `r` | Trainable | Share of base | Weight bytes |
| --- | --- | --- | --- | --- |
| `q_proj`, `v_proj` | 4 | 230,400 | 0.17% | 0.92 MB |
| `q_proj`, `v_proj` | 8 | 460,800 | 0.34% | 1.84 MB |
| `q_proj`, `v_proj` | 64 | 3,686,400 | 2.74% | 14.75 MB |
| All attention | 8 | 921,600 | 0.69% | 3.69 MB |
| `all-linear` | 8 | 2,442,240 | 1.82% | 9.77 MB |
| `all-linear` | 64 | 19,537,920 | 14.52% | 78.15 MB |

Use `q_proj` and `v_proj` for a smoke test or when adapter size is your constraint. Use `all-linear` for a real model, because that is the placement with published evidence from 7B upward. The last row shows where the small-adapter story ends on a tiny model, with an adapter at 14.52 percent of the base.

## QLoRA: fine tuning on 4-bit weights

QLoRA stores the frozen base model in 4-bit NF4 and decompresses weights to 16-bit only for the matrix multiplication. The adapters stay in 16-bit and are the only weights that receive gradients. Dettmers et al. report that this cut fine tuning of a 65B LLaMA from more than 780 GB of GPU memory to under 48 GB.

NF4, double quantization, and paged optimizers supply that saving.

1. NF4 is a 4-bit data type built for zero-centered, normally distributed weights, so each [quantization](https://aifordevs.org/glossary#quantization) bin holds an equal share of values.
2. Double quantization compresses the quantization constants, reducing their cost from 0.5 to 0.127 bits per parameter at a block size of 64.
3. Paged optimizers use NVIDIA unified memory to move optimizer state to CPU RAM when the GPU runs out.

The base model is where the memory goes. For a 7B LLaMA at batch size 1, the paper measures 26 MB of LoRA parameters and 18 MB of input gradients with gradient checkpointing. The 4-bit base takes 5,048 MB, so shrinking `r` saves little, which is why [QLoRA](https://aifordevs.org/glossary#qlora) can afford `r = 64` on all linear layers.

Treat the results as the authors' own numbers. They report that 4-bit QLoRA matched 16-bit fine tuning on academic benchmarks. They also report Guanaco 65B at 99.3 percent of ChatGPT on the Vicuna benchmark, a score assigned by GPT-4.

The same paper states that the authors did not establish QLoRA matching full 16-bit fine tuning at 33B and 65B. This guide found no independent replication of the 99.3 percent figure.

PEFT's [quantization guide](https://huggingface.co/docs/peft/main/en/developer_guides/quantization) documents the `bitsandbytes` path and says QLoRA-style training means `target_modules="all-linear"`. The GPU section below covers that path and labels it as not run.

## Format the dataset for the chat template

Keep each LoRA fine tuning example as one JSON object with a `messages` list, render it with the tokenizer's chat template, and mask every prompt token with `-100`. The loss ignores masked tokens, so only the answer is trained. Dettmers et al. found that training on the target alone raised MMLU scores across four instruction datasets.

The run below uses synthetic incident lines that convert to a fixed JSON object. The task is deliberately small and checkable.

```python title="make_data.py"
import json
import random

SYSTEM = "Return one JSON object with keys service, region, status, minutes. No prose."
TEMPLATES = [
    "{service} returned {status} for {minutes} minutes in {region}",
    "In {region}, {service} served {status} errors for {minutes} minutes",
    "{minutes} minutes of {status} responses from {service} ({region})",
    "Alert: {service}/{region} status {status}, duration {minutes}m",
]
TRAIN_NAMES = ["checkout-api", "billing-worker", "auth-gateway", "search-indexer",
               "email-sender", "image-resizer", "ledger-sync", "cart-cache"]
TEST_NAMES = ["payments-edge", "profile-store", "invoice-renderer"]
TRAIN_REGIONS = ["us-east-1", "us-west-2", "eu-west-1", "eu-central-1"]
TEST_REGIONS = ["ap-south-1", "sa-east-1"]


def example(rng, names, regions):
    fields = {"service": rng.choice(names), "region": rng.choice(regions),
              "status": rng.choice([429, 500, 502, 503, 504]),
              "minutes": rng.randint(2, 90)}
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": rng.choice(TEMPLATES).format(**fields)},
        {"role": "assistant", "content": json.dumps(fields, separators=(",", ":"))},
    ]}


def write(path, rows):
    with open(path, "w") as f:
        f.writelines(json.dumps(r) + "\n" for r in rows)


rng = random.Random(7)
write("train.jsonl", [example(rng, TRAIN_NAMES, TRAIN_REGIONS) for _ in range(96)])
write("test.jsonl", [example(rng, TEST_NAMES, TEST_REGIONS) for _ in range(24)])
```

One line of `train.jsonl` looks like this:

```json title="train.jsonl"
{"messages": [{"role": "system", "content": "Return one JSON object with keys service, region, status, minutes. No prose."}, {"role": "user", "content": "image-resizer returned 503 for 85 minutes in us-west-2"}, {"role": "assistant", "content": "{\"service\":\"image-resizer\",\"region\":\"us-west-2\",\"status\":503,\"minutes\":85}"}]}
```

These choices in that file carry over to real data.

- The held-out split uses service names and regions that never appear in training, so copying memorized strings cannot score.
- The system prompt states the schema in both training and evaluation, so the base model gets a fair baseline.
- The training labels include the end-of-turn token, which trains the adapter to stop generating.

## Run a PEFT LoRA script on CPU

The script loads SmolLM2-135M on CPU, wraps it with `get_peft_model`, trains 96 steps with `Trainer`, and saves only the adapter. It needs no GPU and finishes in minutes. It ran on November 8, 2026 on an Apple M2 with Python 3.13.18, PyTorch 2.15.0, Transformers 5.22.2, PEFT 0.21.2, and Accelerate 1.15.0.

PyYAML had no Python 3.15 wheel for this machine on that date, so the run used a 3.13 environment.

```bash
uv venv --python 3.13 .venv
uv pip install --python .venv/bin/python torch==2.15.0 transformers==5.22.2 peft==0.21.2 accelerate==1.15.0
```

The base model is [SmolLM2-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct), an Apache 2.0 model that Transformers loads as `LlamaForCausalLM`.

```python title="train_lora.py"
import hashlib
import json

import torch
from peft import LoraConfig, TaskType, get_peft_model
from transformers import (AutoModelForCausalLM, AutoTokenizer, DataCollatorForSeq2Seq,
                          Trainer, TrainingArguments, set_seed)

BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
set_seed(7)

tokenizer = AutoTokenizer.from_pretrained(BASE)


def encode(row):
    prompt = tokenizer.apply_chat_template(
        row["messages"][:-1], tokenize=False, add_generation_prompt=True)
    full = tokenizer.apply_chat_template(row["messages"], tokenize=False)
    prompt_ids = tokenizer(prompt, add_special_tokens=False)["input_ids"]
    full_ids = tokenizer(full, add_special_tokens=False)["input_ids"]
    return {
        "input_ids": full_ids,
        "attention_mask": [1] * len(full_ids),
        # -100 tells the loss to ignore the prompt; only the answer is scored.
        "labels": [-100] * len(prompt_ids) + full_ids[len(prompt_ids):],
    }


train = [encode(json.loads(line)) for line in open("train.jsonl")]

base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)
config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    bias="none",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()


def frozen_digest(m):
    h = hashlib.sha256()
    for _, p in sorted(m.named_parameters(), key=lambda kv: kv[0]):
        if not p.requires_grad:
            h.update(p.detach().numpy().tobytes())
    return h.hexdigest()[:16]


before = frozen_digest(model)

args = TrainingArguments(
    output_dir="checkpoints",
    use_cpu=True,
    per_device_train_batch_size=8,
    num_train_epochs=8,
    learning_rate=2e-3,
    logging_steps=12,
    save_strategy="no",
    report_to="none",
    seed=7,
)
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train,
    data_collator=DataCollatorForSeq2Seq(tokenizer, label_pad_token_id=-100),
)
trainer.train()

print("frozen weights unchanged:", before == frozen_digest(model))
model.save_pretrained("adapter")
```

```text title="Real output, progress bars trimmed"
trainable params: 460,800 || all params: 134,975,808 || trainable%: 0.3414
{'loss': '0.2007', 'grad_norm': '0.05098', 'learning_rate': '0.001771', 'epoch': '1'}
{'loss': '0.003873', 'grad_norm': '0.03281', 'learning_rate': '0.001521', 'epoch': '2'}
{'loss': '0.0008523', 'grad_norm': '0.005672', 'learning_rate': '0.001271', 'epoch': '3'}
{'loss': '0.0004551', 'grad_norm': '0.001646', 'learning_rate': '0.001021', 'epoch': '4'}
{'loss': '0.0003418', 'grad_norm': '0.001403', 'learning_rate': '0.0007708', 'epoch': '5'}
{'loss': '0.0002982', 'grad_norm': '0.001455', 'learning_rate': '0.0005208', 'epoch': '6'}
{'loss': '0.0002769', 'grad_norm': '0.0009515', 'learning_rate': '0.0002708', 'epoch': '7'}
{'loss': '0.0002669', 'grad_norm': '0.001162', 'learning_rate': '2.083e-05', 'epoch': '8'}
{'train_runtime': '237.8', 'train_samples_per_second': '3.229', 'train_steps_per_second': '0.404', 'train_loss': '0.02588', 'epoch': '8'}
frozen weights unchanged: True
```

```text title="wc -c adapter/*"
    5224 adapter/README.md
    1113 adapter/adapter_config.json
 1858776 adapter/adapter_model.safetensors
 1865113 total
```

The 460,800 figure is `30 layers * (9,216 + 6,144)`. `q_proj` is 576 by 576, and `v_proj` is 576 by 192 because this model uses fewer key and value heads than query heads, a design called [grouped-query attention](https://aifordevs.org/glossary#grouped-query-attention). The safetensors file is 460,800 weights at 4 bytes each, plus a small header.

Training took 128 seconds in one run and 238 seconds in a repeat with other work on the machine, with identical losses. The learning rate of `2e-3` is far above the `1e-4` to `2e-4` in QLoRA's Table 9, and it was picked only so that 96 CPU steps converge.

> **What this run proves**
>
> The run proves the API and the mechanics. The config is accepted, the base weights stay frozen, the update has rank 8, the adapter saves and reloads, and merging preserves outputs. It does not demonstrate model quality, because SmolLM2-135M is a tiny model, the task is synthetic, and the held-out set has 24 rows.

## Inspect the adapter: rank, size, and merge

This inspection confirms that the update has rank `r`, that it equals `(alpha / r) * B @ A`, and that merging preserves the outputs. Run a check like this once per training setup, because a wrong `target_modules` value or a quantized merge shows up here before it shows up in an eval.

```python title="inspect_adapter.py"
import time
from pathlib import Path

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)

start = time.perf_counter()
model = PeftModel.from_pretrained(base, "adapter").eval()
print(f"attach adapter at runtime: {time.perf_counter() - start:.2f} s")

# One wrapped layer: the query projection of the first transformer block.
layer = model.base_model.model.model.layers[0].self_attn.q_proj
A = layer.lora_A["default"].weight.detach()
B = layer.lora_B["default"].weight.detach()
W0 = layer.base_layer.weight.detach().clone()
delta = layer.get_delta_weight("default").detach()

print("W0 shape:", tuple(W0.shape), "A shape:", tuple(A.shape), "B shape:", tuple(B.shape))
print("scaling alpha/r:", layer.scaling["default"])
print("delta equals (alpha/r) * B @ A:",
      torch.allclose(delta, layer.scaling["default"] * B @ A))
print("rank of W0:   ", torch.linalg.matrix_rank(W0).item())
print("rank of delta:", torch.linalg.matrix_rank(delta).item())
print("top singular values of delta:",
      [round(s, 4) for s in torch.linalg.svdvals(delta)[:10].tolist()])
print(f"Frobenius norm of delta / W0: {delta.norm() / W0.norm():.4f}")

inputs = tokenizer("checkout-api returned 503 for 12 minutes in us-east-1",
                   return_tensors="pt")
with torch.no_grad():
    runtime_logits = model(**inputs).logits
    with model.disable_adapter():
        base_logits = model(**inputs).logits

start = time.perf_counter()
merged = model.merge_and_unload()
print(f"merge_and_unload: {time.perf_counter() - start:.2f} s")
with torch.no_grad():
    merged_logits = merged(**inputs).logits

print("max |adapter - base| logit:  ", (runtime_logits - base_logits).abs().max().item())
print("max |adapter - merged| logit:", (runtime_logits - merged_logits).abs().max().item())

merged.save_pretrained("merged")
size = lambda p: sum(f.stat().st_size for f in Path(p).glob("*.safetensors")) / 1e6
print(f"adapter weights: {size('adapter'):.2f} MB, merged weights: {size('merged'):.1f} MB")
```

```text title="Real output, progress bars trimmed"
attach adapter at runtime: 0.16 s
W0 shape: (576, 576) A shape: (8, 576) B shape: (576, 8)
scaling alpha/r: 2.0
delta equals (alpha/r) * B @ A: True
rank of W0:    571
rank of delta: 8
top singular values of delta: [1.6702, 0.6284, 0.3488, 0.3016, 0.2203, 0.1518, 0.1275, 0.1032, 0.0, 0.0]
Frobenius norm of delta / W0: 0.0115
merge_and_unload: 0.03 s
max |adapter - base| logit:   29.580005645751953
max |adapter - merged| logit: 0.00015211105346679688
adapter weights: 1.86 MB, merged weights: 538.1 MB
```

The first `q_proj` has a nearly full-rank `W0` and a `delta` with exactly eight non-zero singular values. Its update has 1.15 percent of the norm of `W0`, and across all wrapped layers the adapter moved the largest logit by 29.58. The largest gap between runtime and merged logits is 0.00015, which is floating-point rounding.

The merged checkpoint is 538.1 MB against 1.86 MB for the adapter, a ratio of 289 to 1 on this model. An adapter keeps each layer's input and output dimensions, so [KV cache sizing](https://aifordevs.org/articles/kv-cache-explained) does not change.

## Evaluate the adapter against the base model

Score the same held-out prompts twice, once inside `model.disable_adapter()` and once with the adapter active. Use a metric tied to the task, on data the adapter never saw. The script below compares the two on valid JSON, exact match, and per-field accuracy across 24 rows.

```python title="evaluate_adapter.py"
import json

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(BASE)
tokenizer.padding_side = "left"

base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)
model = PeftModel.from_pretrained(base, "adapter").eval()

rows = [json.loads(line)["messages"] for line in open("test.jsonl")]
prompts = [tokenizer.apply_chat_template(m[:-1], tokenize=False, add_generation_prompt=True)
           for m in rows]
gold = [json.loads(m[-1]["content"]) for m in rows]


@torch.no_grad()
def generate():
    out = []
    for start in range(0, len(prompts), 8):
        batch = tokenizer(prompts[start:start + 8], return_tensors="pt", padding=True,
                          add_special_tokens=False)
        ids = model.generate(**batch, max_new_tokens=48, do_sample=False)
        out += tokenizer.batch_decode(ids[:, batch["input_ids"].shape[1]:],
                                      skip_special_tokens=True)
    return out


def score(texts):
    parsed, exact, fields = 0, 0, 0
    for text, want in zip(texts, gold):
        try:
            got = json.loads(text.strip())
        except json.JSONDecodeError:
            continue
        if not isinstance(got, dict):
            continue
        parsed += 1
        exact += got == want
        fields += sum(got.get(k) == v for k, v in want.items())
    n = len(gold)
    return f"valid JSON {parsed}/{n}, exact match {exact}/{n}, fields right {fields}/{4 * n}"


with model.disable_adapter():
    base_out = generate()
adapter_out = generate()

print("adapter off:", score(base_out))
print("adapter on: ", score(adapter_out))
print("sample input:", rows[0][1]["content"])
print("adapter off :", base_out[0].strip()[:120])
print("adapter on  :", adapter_out[0].strip()[:120])
print("gold        :", rows[0][2]["content"])
```

```text title="Real output, progress bars trimmed"
adapter off: valid JSON 0/24, exact match 0/24, fields right 0/96
adapter on:  valid JSON 24/24, exact match 19/24, fields right 87/96
sample input: In sa-east-1, profile-store served 504 errors for 43 minutes
adapter off : The error message indicates that the service failed to respond to the request for the specified service. The service is 
adapter on  : {"service":"profile-store","region":"sa-east-1","status":504,"minutes":43}
gold        : {"service":"profile-store","region":"sa-east-1","status":504,"minutes":43}
```

The adapter moved the output from zero valid JSON objects to 24 of 24, with 19 exact matches. All five misses involve `payments-edge`, and in each one the adapter emitted a leading space inside a string value, as in `" payments-edge"`. An aggregate score hides that pattern, so read the misses.

Do not read 19 of 24 as a quality result. A larger base model might pass this task with the prompt alone. The comparison that matters on real work is the tuned model against the base model with your best prompt.

For a real model, build a [golden dataset](https://aifordevs.org/glossary#golden-dataset), choose metrics with the [LLM evaluation guide](https://aifordevs.org/articles/llm-evaluation), and use [LLM as a judge](https://aifordevs.org/articles/llm-as-a-judge) for open-ended outputs. Also test tasks the adapter should leave alone. Shuttleworth et al. report that LoRA tends to forget less pre-training behavior than full fine tuning, with exceptions at large learning rates.

## Serve the adapter: merge or load at runtime

Merge the adapter when one task owns the deployment, because merged weights need no extra step per layer and load like any other checkpoint. Load adapters at runtime when several tasks share one base model, because each adapter costs megabytes and the base stays in GPU memory once.

| Option | How | Memory | Added latency | Use when |
| --- | --- | --- | --- | --- |
| Merge | `merge_and_unload()` | One full model per adapter | None by construction, per Hu et al. | One task owns the deployment |
| Runtime in PEFT | `load_adapter()`, `set_adapter()` | Base once, plus adapters | PEFT warns of added latency | Experiments and low-traffic tools |
| Multi-LoRA in vLLM | `--enable-lora`, `--lora-modules` | Base once, adapters in GPU or CPU memory | vLLM says minimal overhead, no published number | Several tasks on shared GPUs |

`merge_and_unload()` returns a new model, so assign the result. Hu et al. note that merging makes it hard to batch inputs for different tasks in one forward pass.

PEFT's [LoRA guide](https://huggingface.co/docs/peft/main/package_reference/lora) warns that on [mixture-of-experts](https://aifordevs.org/articles/mixture-of-experts) models it computes each expert's LoRA contribution even when few experts are active, which can slow inference substantially. It recommends merging on those models.

vLLM serves adapters per request through its OpenAI-compatible server. The commands below are adapted from [vLLM's LoRA docs](https://docs.vllm.ai/en/stable/features/lora/) as of November 8, 2026, and were not run for this guide. They need a GPU server.

```bash title="Not run here: adapted from the vLLM docs"
vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --enable-lora \
  --max-lora-rank 64 \
  --max-loras 2 \
  --max-cpu-loras 8 \
  --lora-modules \
    '{"name": "incident-json", "path": "/models/lora/incident-json", "base_model_name": "meta-llama/Llama-3.2-3B-Instruct"}' \
    '{"name": "support-tone", "path": "/models/lora/support-tone", "base_model_name": "meta-llama/Llama-3.2-3B-Instruct"}'

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "incident-json", "prompt": "checkout-api returned 503", "max_tokens": 48, "temperature": 0}'
```

The [CLI reference](https://docs.vllm.ai/en/stable/cli/serve/) sets `--max-lora-rank` to 16 and `--max-loras` to 1 by default, so a QLoRA-style `r=64` adapter needs the first flag raised. `--max-cpu-loras` must be at least `--max-loras`. The `/v1/models` endpoint lists each adapter with a `parent` field pointing to the base model.

vLLM can also load adapters while running, but only with `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`. Its docs warn of security risks and advise against that mode in production unless the environment is isolated and fully trusted. For engine choice, see [vLLM, SGLang, or llama.cpp](https://aifordevs.org/articles/vllm-sglang-llama-cpp).

## Changes for a real model and GPU

LoRA fine tuning on a real model uses the same steps, and none of the code below was run. It comes from PEFT's quantization guide and the QLoRA paper, and it replaces the model loading and training arguments in the script above. Dataset formatting, the evaluation script, and the inspection checks carry over unchanged.

```python title="Not run here: 4-bit setup from the PEFT docs"
import torch
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=bnb)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
))
```

The `r` and `alpha` values above are this guide's starting suggestion, following the `alpha = 2r` result. Make these changes to the training script as well.

1. Remove `use_cpu=True` from `TrainingArguments`.
2. Lower `learning_rate` to the `1e-4` to `2e-4` range in QLoRA's Table 9.
3. Keep gradient checkpointing, which `prepare_model_for_kbit_training` enables by default in PEFT 0.21.2. The paper measured input gradients falling from 567 MB to 18 MB with it.
4. Plan for far more steps. QLoRA's runs at 7B used 1,875 to 10,000.

If you want a merged model, merge against the unquantized base. PEFT's docs say merging into quantized weights dequantizes, adds the adapter, and quantizes again, and repeated cycles can accumulate rounding error.

## Start your first LoRA fine tuning run

Run your first LoRA fine tuning job only after a prompt and a schema have failed on an eval you trust. Spend the first day on a small model and small data. LLM fine tuning bugs in formatting and labels show up in minutes on CPU and cost GPU hours later.

1. Check the failure against the table in [RAG vs fine tuning](https://aifordevs.org/articles/rag-vs-fine-tuning). An adapter does nothing for stale facts, and [context engineering](https://aifordevs.org/articles/context-engineering) or [structured outputs](https://aifordevs.org/glossary#structured-outputs) may fix the rest.
2. Write 50 examples in the `messages` format, the [starting point OpenAI recommends](https://developers.openai.com/api/docs/guides/supervised-fine-tuning), and hold out a fifth, split by entity. Google's [tuning guide](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/supervised-tuning/use) recommends at least 100 to 500 for its service.
3. Swap your files in for `train.jsonl` and `test.jsonl`, and run the three scripts above without other changes.
4. Move to a 7B-class base with the QLoRA setup above, which was not run here, and rerun the same evaluation.
5. Ship the adapter only if it beats the base model with your best prompt on the held-out set.

More guides in this area are collected under [fine tuning topics](https://aifordevs.org/topics/fine-tuning) and [inference](https://aifordevs.org/topics/inference), and new model releases that change these defaults appear in the [news feed](https://aifordevs.org/news).

## Questions

### What is the difference between LoRA and QLoRA?

LoRA trains a low-rank update on frozen weights. QLoRA does the same while storing those frozen weights in 4-bit NF4, which cuts training memory. Dettmers et al. also attach adapters to all linear layers and report that this placement was required to match full fine tuning on large base models. The adapter weights stay in 16-bit precision during training.

### What are good values for r and alpha in LoRA?

Start with an r of 8 and an alpha of 16, which gives an update scale of 2 under PEFT's default formula. Dettmers et al. used an r of 64 and an alpha of 16 on all linear layers and report that r had no relationship with final performance in that setup. Increase r only when your held-out eval shows the adapter underfits.

### How many training examples does LoRA fine tuning need?

No fixed number exists, because the answer depends on the task. OpenAI's fine-tuning guide, written for hosted models, recommends starting with 50 well-crafted demonstrations and reconsidering the task if those have no effect. Google's Gemini tuning page recommends at least 100 to 500 examples. Start small, hold out data, and add examples only while the held-out score keeps improving.

### Can you run LoRA fine tuning on a CPU?

Yes for a small model, and the run on this page did that: SmolLM2-135M trained for 96 steps on an Apple M2 CPU in 128 to 238 seconds. CPU runs are useful for testing code and data formatting. Real models need a GPU, and the 4-bit bitsandbytes path that QLoRA uses is a separate code path this guide did not run.

### Do LoRA adapters add inference latency?

A merged adapter adds none by construction, because Hu et al. show the update can be added to the weights once and inference proceeds as usual. An unmerged adapter runs extra matrix multiplications per wrapped layer. PEFT's guide warns that you may see latency from loading the base and adapter separately, and vLLM describes per-request adapters as minimal overhead without publishing a number.

### Is LoRA as good as full fine tuning?

On the tasks Hu et al. tested, LoRA matched or beat full fine tuning. Dettmers et al. report that 4-bit LoRA matched 16-bit full fine tuning on academic benchmarks, though they did not establish this at 33B and 65B. Shuttleworth et al. found that LoRA produces structurally different weight updates. Evaluate the adapter on your own held-out data before you rely on any of this.

## Sources

1. [Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)
2. [Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)
3. [Shuttleworth et al., LoRA vs Full Fine-tuning: An Illusion of Equivalence (2024)](https://arxiv.org/abs/2410.21228)
4. [Hugging Face, PEFT LoRA configuration reference](https://huggingface.co/docs/peft/main/en/package_reference/lora)
5. [Hugging Face, PEFT LoRA developer guide](https://huggingface.co/docs/peft/main/package_reference/lora)
6. [Hugging Face, PEFT quantization guide](https://huggingface.co/docs/peft/main/en/developer_guides/quantization)
7. [vLLM, LoRA Adapters](https://docs.vllm.ai/en/stable/features/lora/)
8. [vLLM, vllm serve CLI reference](https://docs.vllm.ai/en/stable/cli/serve/)
9. [Hugging Face, SmolLM2-135M-Instruct model card](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct)
10. [OpenAI, Supervised fine-tuning guide](https://developers.openai.com/api/docs/guides/supervised-fine-tuning)
11. [Google Cloud, Use supervised tuning for Gemini models](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/supervised-tuning/use)

## Cite this guide

AI for Devs. "LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script." Published November 8, 2026. https://aifordevs.org/articles/lora-fine-tuning

- Page: https://aifordevs.org/articles/lora-fine-tuning
- Markdown: https://aifordevs.org/articles/lora-fine-tuning.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "Mixture of experts: how MoE works and what it costs"
type: "deep dive"
url: "https://aifordevs.org/articles/mixture-of-experts"
markdown: "https://aifordevs.org/articles/mixture-of-experts.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/mixture-of-experts"
description: "Mixture of experts LLMs load every expert but run only a few per token. Compare total and active parameters, and see what memory, batching, and routing cost."
author: "AI for Devs"
published: "2026-11-08"
level: "Advanced"
topics: ["models", "inference"]
---

# Mixture of experts: how MoE works and what it costs

Mixture of experts LLMs load every expert but run only a few per token. Compare total and active parameters, and see what memory, batching, and routing cost.

> **TL;DR**
>
> - Mixture of experts replaces a transformer's dense feed-forward layers with many parallel expert networks and a router that runs only a few of them per token.
> - Serving cost splits in two. Memory follows total parameters and compute follows active parameters, so DeepSeek-V3 loads 671B and computes 37B per token.
> - Batching removes the memory-bandwidth discount but keeps the compute discount. Under uniform routing, a Mixtral-shaped model touches 99 percent of its experts by batch 16.
> - As of November 8, 2026, model cards put most large open-weight MoE models at 3 to 6 percent active, compared with 27.7 percent for Mixtral 8x7B.
> - The occupancy script ran on Python 3.15.1 with a Monte Carlo cross-check. No MoE model or GPU ran for this post, so batching figures are arithmetic.

Mixture of experts (MoE) has two meanings, and search results for the term now treat the LLM meaning as the default. The classic meaning is the 1991 method of Jacobs, Jordan, Nowlan, and Hinton, in which a gating network assigns each training case to a few expert networks. A mixture-of-experts LLM is a [transformer](https://aifordevs.org/glossary#transformer) whose feed-forward layers are replaced by many parallel expert networks and a router that sends each token to only a few of them. The rest of this post covers that LLM design and what it costs an engineer who has to serve it.

## What mixture of experts means in an LLM

A [mixture of experts](https://aifordevs.org/glossary#mixture-of-experts) LLM keeps attention dense but replaces each feed-forward layer with several parallel feed-forward networks and a small router. The router scores every expert for each token and runs only the top few, so one token's compute touches a small fraction of the weights.

**Figure 1.** Attention runs fully for every token. The router scores all experts but only the top two compute, so this layer holds eight experts in memory and spends compute on two.

Routing happens per token and per layer. In [Mixtral 8x7B](https://arxiv.org/abs/2401.04088), each of 32 layers has 8 experts, a linear router picks the top two, and a softmax over those two scores weights their outputs. Only the selected experts run.

Some models add shared experts that every token uses. [DeepSeek-V3](https://arxiv.org/abs/2412.19437) has 1 shared expert and 256 routed experts per MoE layer, with 8 routed experts active per token, so each token goes through 9 experts per layer.

Experts replace only the feed-forward block, and [attention](https://aifordevs.org/glossary#attention) stays dense. [Fedus et al.](https://arxiv.org/abs/2101.03961) tested attention-layer experts and left them out because they were unstable in bfloat16. Dense attention also means an MoE model has the same [KV cache](https://aifordevs.org/articles/kv-cache-explained) cost as a dense model with the same attention layout.

## How MoE moved from 1991 to today's LLMs

Mixture of experts started as a 1991 way to split training cases across networks, became a sparse layer in 2017, and moved into large transformers in 2020 and 2021. Mixtral 8x7B then shipped as an open-weight model built on that design, and later large open-weight models followed.

| Year | Work | What changed |
| --- | --- | --- |
| 1991 | [Jacobs et al.](https://www.cs.toronto.edu/~fritz/absps/jjnh91.pdf) | A gating network assigns training cases to experts |
| 2017 | [Shazeer et al.](https://arxiv.org/abs/1701.06538) | Noisy top-k gating and balancing losses; up to 137B parameters and 131,072 experts in an LSTM |
| 2020 | [GShard](https://arxiv.org/abs/2006.16668) | Experts replace feed-forward layers in a transformer; 600B parameters with 2,048 experts per layer; top-2 gating |
| 2021 | [Switch Transformers](https://arxiv.org/abs/2101.03961) | Top-1 routing, 1.6T parameters, and author-reported pre-training speedups of up to 7x over T5 at equal compute |
| 2024 | [Mixtral 8x7B](https://arxiv.org/abs/2401.04088) | Open weights: 47B total and 13B active |
| 2024 | [DeepSeek-V3](https://arxiv.org/abs/2412.19437) | 671B total and 37B active, balanced by per-expert bias terms with only a small sequence-wise loss |

Scale is the through line. Shazeer et al. got up to 137B parameters into a 2017 LSTM model and reported best results at 68B. Switch reached 1.6T with one expert per token.

Later designs activate more experts per token, two in Mixtral and eight routed plus one shared in DeepSeek-V3.

## Why load balancing is the hard part of MoE

A router without balancing can concentrate tokens on a few experts, which wastes the others in training and overloads them in serving. Shazeer et al., Switch Transformers, and DeepSeek-V3 each added a mechanism to spread the load, from auxiliary losses and capacity limits to per-expert bias terms.

[Shazeer et al.](https://arxiv.org/abs/1701.06538) added an importance loss and a load loss that penalize uneven gate values across a batch. [Switch Transformers](https://arxiv.org/abs/2101.03961) used one auxiliary loss with weight 0.01, and set an expert capacity so overflow tokens pass through the residual connection. Dropped tokens were typically under 1 percent in those experiments.

[DeepSeek-V3](https://arxiv.org/abs/2412.19437) replaced the per-step auxiliary loss with a bias term per expert. After each step, overloaded experts' biases go down and underloaded ones go up, which shifts routing without distorting the gate values used for output. A small sequence-wise balance loss remains as a guard against extreme imbalance within one sequence.

Routing is not a topic switch. [Mixtral's authors](https://arxiv.org/abs/2401.04088) report no obvious topic-based pattern in expert assignment across their layers, with assignment following syntax more than domain.

Consecutive tokens often reused the same expert at higher layers. That locality matters for serving, because it can oversubscribe a device.

## Loaded versus active parameters in open-weight MoE models

Most large open-weight MoE models load 18 to 33 times more parameters than they run per token. Model cards put DeepSeek-V4-Pro at 1.6T total and 49B active, and Kimi K3 at 2.8T total and 104B active, as of November 8, 2026. Mixtral 8x7B's ratio was 3.6 times.

| Model | Total | Active per token | Active share | Total / active | Experts per layer |
| --- | --- | --- | --- | --- | --- |
| [Mixtral 8x7B](https://arxiv.org/abs/2401.04088) | 47B | 13B | 27.7% | 3.6x | 2 of 8 |
| [Llama 4 Scout](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) | 109B | 17B | 15.6% | 6.4x | 16 experts |
| [DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3) | 671B | 37B | 5.5% | 18.1x | 8 of 256 + 1 shared |
| [GLM-5](https://huggingface.co/zai-org/GLM-5) | 744B | 40B | 5.4% | 18.6x | 8 of 256 + 1 shared |
| [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) | 428B | 23B | 5.4% | 18.6x | 4 of 128 + 1 shared |
| [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) | 117B | 5.1B | 4.4% | 22.9x | 4 of 128 |
| [Llama 4 Maverick](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) | 400B | 17B | 4.3% | 23.5x | 128 experts |
| [Kimi K3](https://huggingface.co/moonshotai/Kimi-K3) | 2.8T | 104B | 3.7% | 26.9x | 16 of 896 |
| [Trinity-Large-Preview](https://huggingface.co/arcee-ai/Trinity-Large-Preview) | 398B | 13B | 3.3% | 30.6x | 4 of 256 (1 shared) |
| [Kimi K2](https://huggingface.co/moonshotai/Kimi-K2-Instruct) | 1T | 32B | 3.2% | 31.3x | 8 of 384 + 1 shared |
| [DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) | 1.6T | 49B | 3.1% | 32.7x | 6 of 384 + 1 shared |

Totals and active counts are as stated on each vendor's page as of November 8, 2026, sorted by active share, with ratios recomputed by code. Expert counts come from each model's Hugging Face `config.json` or card. Meta's post gives expert counts for Llama 4 but not how many activate per token.

A count from [DeepSeek-V3's config](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json) explains the split. Each expert holds about 44M parameters per layer, and 58 MoE layers with 257 experts come to 656.5B, or 97.8 percent of 671B. One token runs 9 experts per layer, which adds 23.0B to the remaining 14.5B and lands near the published 37B.

Hugging Face files can exceed the headline. [DeepSeek's README](https://github.com/deepseek-ai/DeepSeek-V3) notes 685B on disk, because 14B of multi-token prediction weights sit on top of the 671B main model. Those weights are what [speculative decoding](https://aifordevs.org/articles/speculative-decoding) can reuse as a draft.

## What mixture of experts costs in memory

MoE memory follows total parameters, because any token can route to any expert on the next step, so every expert stays in GPU memory. Mixtral 8x7B needs about 93 GB of 16-bit weights to run 13B active parameters, about 5.8 times the weights of Llama 3.1 8B.

[Mixtral's authors](https://arxiv.org/abs/2401.04088) state that memory cost for serving tracks the sparse parameter count, 47B. The weights do not shrink because routing is sparse, and routing cannot be predicted before the token exists.

Run the budget on a [141 GB H200](https://www.nvidia.com/en-us/data-center/h200/), using 92 percent as usable memory. Llama 3.1 8B leaves 105.9 GiB for the cache, which holds 105 sequences at 8K tokens. Mixtral 8x7B leaves 33.8 GiB after 93.4 GB of weights, which holds 33 sequences, because both models need 128 KiB of cache per token.

The largest models do not fit on one GPU. [DeepSeek-V3's published weights](https://huggingface.co/deepseek-ai/DeepSeek-V3) total about 688 GB, mostly 8-bit, or 5.3 H200s at 92 percent. [GLM-5](https://huggingface.co/zai-org/GLM-5) at 16 bits needs about 1.5 TB, or 11.6 H200s, before any cache.

Quantizing experts is a common escape. [gpt-oss-120b's card](https://huggingface.co/openai/gpt-oss-120b) says MXFP4 quantization of its MoE weights lets it fit a single 80 GB GPU, and [DeepSeek-V4-Pro's card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) lists FP4 precision for MoE expert parameters. Test [quantization](https://aifordevs.org/glossary#quantization) on your own task before you serve it.

## What mixture of experts costs in compute

MoE compute follows active parameters, so a Mixtral 8x7B token runs about 13B parameters and a DeepSeek-V3 token about 37B. That saving holds at every batch size. The price is routing overhead and extra memory loads when more than one expert runs per device, which the [Mixtral paper](https://arxiv.org/abs/2401.04088) names as a hardware efficiency cost.

A sparse layer does work a dense layer does not: scoring experts, then grouping tokens by expert before each feed-forward multiply. Mixtral's authors report that overhead, along with MegaBlocks kernels in vLLM that recast the expert multiplies as sparse matrix work. They say the design suits batched workloads, where each expert still sees enough tokens to use the hardware well.

Per-expert batches are small during decode and large during prefill. Mean tokens per expert per layer is `batch x k / experts`, so a DeepSeek-V3-shaped layer at decode batch 256 gives each expert about 8 tokens. A 10,000-token prefill gives each expert about 313 tokens, which is why [prefill and decode](https://aifordevs.org/glossary#prefill-and-decode) load the experts so differently.

DeepSeek reports the same regime at production scale. In its [V3 decoding deployment](https://arxiv.org/abs/2412.19437), the report says per-expert batches usually stay within 256 tokens and that the bottleneck is memory access rather than computation.

## How batching changes the mixture of experts math

At batch 1, a decode step reads only the active experts' weights. As batch size grows, tokens spread across experts, so a step soon reads most of them. Under uniform routing, a Mixtral-shaped model touches 99 percent of its experts by batch 16, which removes the memory-bandwidth discount and keeps the compute discount.

For Mixtral that is 13B of 47B at batch 1 and all 47B at batch 32, shared across 32 tokens. Expected occupancy has a closed form when routing is uniform.

```python title="moe_touch.py"
"""Share of routed experts a batch touches, assuming uniform routing."""


def share_touched(k: int, experts: int, batch: int) -> float:
    """Expected share of routed experts used by at least one token."""
    return 1 - (1 - k / experts) ** batch


models = [("Mixtral 8x7B", 2, 8), ("DeepSeek-V3", 8, 256), ("Kimi K3", 16, 896)]
batches = (1, 8, 16, 32, 64, 128, 256)
print(f"{'batch':<14}", *(f"{b:>6}" for b in batches))
for name, k, n in models:
    row = (f"{share_touched(k, n, b):6.1%}" for b in batches)
    print(f"{name:<14}", *row)
```

```text
batch               1      8     16     32     64    128    256
Mixtral 8x7B    25.0%  90.0%  99.0% 100.0% 100.0% 100.0% 100.0%
DeepSeek-V3      3.1%  22.4%  39.8%  63.8%  86.9%  98.3% 100.0%
Kimi K3          1.8%  13.4%  25.0%  43.8%  68.4%  90.0%  99.0%
```

A Monte Carlo run with random top-k routing matched three sampled entries to within 0.05 percentage points. The script ran on Python 3.15.1 and no model was involved. Real routers can be skewed or show temporal locality, and both reduce occupancy, so read these as upper bounds on how many experts a batch touches.

**Figure 2.** Under uniform routing, batch size decides how much of an MoE model each decode step reads. Mixtral's experts are 99 percent touched by batch 16, and DeepSeek-V3's by batch 128.

Mixtral's numbers show the trade. The table counts weights read in one step, using the same uniform routing assumption, against a dense model with 13B or 47B parameters.

| Batch | Experts touched | Mixtral weights read | Mixtral per token | Dense 13B per token | Dense 47B per token |
| --- | --- | --- | --- | --- | --- |
| 1 | 25.0% | 12.9B | 12.88B | 13.00B | 46.70B |
| 4 | 68.4% | 32.4B | 8.11B | 3.25B | 11.68B |
| 8 | 90.0% | 42.2B | 5.27B | 1.62B | 5.84B |
| 16 | 99.0% | 46.3B | 2.89B | 0.81B | 2.92B |
| 32 | 100.0% | 46.7B | 1.46B | 0.41B | 1.46B |

At batch 1, Mixtral reads no more than a dense 13B model. By batch 16, it reads as much per token as a dense 47B, and at batch 32 it spends 3.6 times the memory traffic per token of a dense 13B.

The compute discount is still there at batch 32, and the bandwidth discount is gone. If your deployment runs at [large batch sizes](https://aifordevs.org/glossary#continuous-batching), judge an MoE model on its compute and capacity numbers.

## How expert parallelism spreads experts across GPUs

Expert parallelism puts different experts on different GPUs and ships each token across the network to the experts it needs. vLLM documents this for MoE models, and DeepSeek's V3 decoding fleet ran 320-way expert parallelism. The cost is all-to-all traffic, and the slowest GPU sets each step's time.

**Figure 3.** Expert parallelism spreads experts across GPUs, so tokens cross the network to reach them. One busy expert, E5 here, sets the step time for its whole group, which is why load balancers duplicate hot experts.

Without [expert parallelism](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/), vLLM runs MoE layers with tensor parallelism across the group, like a dense model. With `--enable-expert-parallel`, vLLM shards experts across EP ranks, and the expert count per rank is the total experts divided by the rank count.

[DeepSeek's V3 report](https://arxiv.org/abs/2412.19437) gives a production layout. Its minimum decoding unit is 40 nodes with 320 GPUs, each GPU hosts one expert, and 64 GPUs host redundant and shared experts. The prefilling unit is 4 nodes with 32 GPUs using 32-way expert parallelism.

The communication has its own libraries. [DeepEP](https://github.com/deepseek-ai/DeepEP) provides the all-to-all dispatch and combine kernels that move tokens to and from experts.

Imbalance sets the ceiling, because a step waits for the busiest expert. [EPLB](https://github.com/deepseek-ai/EPLB), DeepSeek's balancer, duplicates hot experts and packs them across GPUs. vLLM's `--enable-eplb` moves experts based on load statistics and warns that redundant experts cost extra memory, so on a tight deployment the fix can break your memory budget.

## What to check before you serve an MoE model

Before you serve a mixture of experts model, price memory first, then decide whether expert parallelism is worth the network. If the weights and your KV cache fit on one GPU, run it as a single model and skip expert parallelism. Run these checks in order.

1. Multiply total parameters by bytes per parameter at your serving precision, using the model card and the Hugging Face file sizes.
2. Add the [KV cache](https://aifordevs.org/articles/kv-cache-explained) for your target concurrency and context length. Attention stays dense, so MoE does not change it.
3. Estimate decode batch size, then run `moe_touch.py` to see how much of the model each step reads.
4. If the weights exceed one GPU, plan the interconnect and load balancing before the GPU count, because the slowest expert sets the step time.

Skip a large MoE model for an interactive endpoint with a tight memory budget, because the weights will crowd out the cache. Choose it when you can batch traffic and want total-parameter capacity at active-parameter compute.

Check the [engine comparison](https://aifordevs.org/articles/vllm-sglang-llama-cpp) for your server choice and the [best LLM for coding](https://aifordevs.org/articles/best-llm-for-coding) comparison for which models are strong on code today. New open-weight MoE releases land often, so watch the [news page](https://aifordevs.org/news) and the [models topic page](https://aifordevs.org/topics/models), and read each model card before you size hardware. The [inference topic page](https://aifordevs.org/topics/inference) collects more serving guides.

## Questions

### What is mixture of experts in an LLM?

Mixture of experts is a transformer design where each feed-forward layer is replaced by several parallel networks, called experts, plus a router. For every token at every layer, the router scores the experts and runs only the top few. Attention stays dense. Models such as Mixtral 8x7B, DeepSeek-V3, and Kimi K3 use this design.

### What does active parameters mean for an MoE model?

Active parameters are the weights one token actually runs through. They equal the non-expert weights, mostly attention and normalization, plus the chosen experts per layer. Total parameters are every weight in the checkpoint. DeepSeek-V3 has 671B total and 37B active, so only about 5.5 percent of its weights do work for any given token.

### Why can't you load only the active experts?

The router picks experts from each token's own hidden state at each layer, so the required experts are only known at run time. Different tokens, and later layers for the same token, pick different experts. Paging an expert in from storage mid-step would cost far more than holding it in GPU memory.

### How much memory does an MoE model need?

MoE memory follows total parameters, and quantization lowers the bytes per parameter. Mixtral 8x7B needs about 93 GB of 16-bit weights. DeepSeek-V3 shipped as mostly 8-bit checkpoints at about 688 GB, and GLM-5 needs about 1.5 TB at 16 bits. Attention is unchanged by MoE, so KV cache needs stay separate and add on top.

### Is a mixture of experts model faster than a dense model?

An MoE model computes only its active parameters per token, so compute per token is close to a dense model of the active size. Per-token memory traffic depends on batch size. At batch 1 it matches the active size, and at large batches it approaches the total weight count, because almost every expert gets touched in each step.

### Do mixture of experts models specialize experts by topic?

Mostly no. The Mixtral paper reports no obvious topic-based assignment of experts across layers, and notes that routing tends to follow syntax more than domain. Consecutive tokens often reuse the same expert at higher layers, which the authors call temporal locality. Treat experts as a compute-saving structure, not as named specialists you can target.

## Sources

1. [Jacobs, Jordan, Nowlan, and Hinton, Adaptive Mixtures of Local Experts (1991)](https://www.cs.toronto.edu/~fritz/absps/jjnh91.pdf)
2. [Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017)](https://arxiv.org/abs/1701.06538)
3. [Lepikhin et al., GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020)](https://arxiv.org/abs/2006.16668)
4. [Fedus, Zoph, and Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021)](https://arxiv.org/abs/2101.03961)
5. [Jiang et al., Mixtral of Experts (2024)](https://arxiv.org/abs/2401.04088)
6. [DeepSeek-AI, DeepSeek-V3 Technical Report (2024), Sections 2.1, 3.4, and 4.2](https://arxiv.org/abs/2412.19437)
7. [DeepSeek-V3 README on GitHub, total parameter note](https://github.com/deepseek-ai/DeepSeek-V3)
8. [DeepSeek-V3 config.json on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json)
9. [Meta, The Llama 4 herd (2025)](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)
10. [OpenAI, gpt-oss-120b model card](https://huggingface.co/openai/gpt-oss-120b)
11. [Moonshot AI, Kimi K2 model card](https://huggingface.co/moonshotai/Kimi-K2-Instruct)
12. [Moonshot AI, Kimi K3 model card](https://huggingface.co/moonshotai/Kimi-K3)
13. [Z.ai, GLM-5 model card](https://huggingface.co/zai-org/GLM-5)
14. [MiniMax, MiniMax-M3 model card](https://huggingface.co/MiniMaxAI/MiniMax-M3)
15. [Arcee AI, Trinity-Large-Preview model card](https://huggingface.co/arcee-ai/Trinity-Large-Preview)
16. [DeepSeek-AI, DeepSeek-V4-Pro model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)
17. [Mixtral-8x7B model page on Hugging Face](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)
18. [NVIDIA H200 specifications](https://www.nvidia.com/en-us/data-center/h200/)
19. [vLLM docs, Expert Parallel Deployment](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/)
20. [DeepSeek, DeepEP](https://github.com/deepseek-ai/DeepEP)
21. [DeepSeek, EPLB](https://github.com/deepseek-ai/EPLB)

## Cite this deep dive

AI for Devs. "Mixture of experts: how MoE works and what it costs." Published November 8, 2026. https://aifordevs.org/articles/mixture-of-experts

- Page: https://aifordevs.org/articles/mixture-of-experts
- Markdown: https://aifordevs.org/articles/mixture-of-experts.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "RAG vs fine tuning: which failures each one can fix"
type: "deep dive"
url: "https://aifordevs.org/articles/rag-vs-fine-tuning"
markdown: "https://aifordevs.org/articles/rag-vs-fine-tuning.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/rag-vs-fine-tuning"
description: "RAG vs fine tuning compared by failure mode: what each changes, who still sells fine tuning, dated costs worked in code, and when to combine them."
author: "AI for Devs"
published: "2026-11-08"
level: "Intermediate"
topics: ["fine-tuning", "rag"]
---

# RAG vs fine tuning: which failures each one can fix

RAG vs fine tuning compared by failure mode: what each changes, who still sells fine tuning, dated costs worked in code, and when to combine them.

> **TL;DR**
>
> - RAG vs fine tuning comes down to which part of the system is wrong. RAG changes the input on each request, and fine tuning changes the weights before deployment.
> - Try prompting and context first. Add RAG when the failure is missing or stale facts. Fine-tune only when an eval shows a behavior failure that survives both.
> - Format errors usually need neither. Schema enforcement, such as OpenAI's Structured Outputs or vLLM's grammar backends, targets syntax with no training job.
> - As of November 8, 2026, OpenAI has closed fine tuning to new organizations and ends new jobs for existing customers on January 6, 2027. Google, Amazon Bedrock, Together AI, and Fireworks AI still sell it.
> - In this article's worked cost model, a cached prompt was cheapest at $1,375 a month on Gemini 3.5 Flash-Lite. A tuned model cost 5 percent more there and 76 percent more on OpenAI's `gpt-4.1-mini`.

RAG vs fine tuning is a choice between two places to change an LLM system: the context a request carries and the weights that process it. Retrieval-augmented generation, introduced by Lewis et al. in 2020, adds retrieved passages to the input at request time. Fine tuning trains new weights from labeled examples before any request arrives. This article organizes the decision by failure mode, because the failure you observe tells you which part to change, and it prices each option with dated numbers.

## What RAG and fine tuning each change

[RAG](https://aifordevs.org/glossary#rag) changes the context window on each request, and [fine tuning](https://aifordevs.org/glossary#fine-tuning) changes the weights before deployment. That difference sets how fast a correction reaches users, what it costs per request, and what it can repair. Prompting sits in the same place as RAG, and constrained decoding acts after both.

**Figure 1.** RAG writes into the context on every request and leaves the weights alone. Fine tuning changes the weights once, so any later change needs another training job. A schema mask works at decode time and depends on neither.

[Lewis et al.](https://arxiv.org/abs/2005.11401) describe RAG as a generator paired with a retriever over a dense vector index, which they frame as parametric and non-parametric memory. Weights are the parametric part, and the index is the part you can edit without training. Fine tuning writes into the parametric part, so a change takes a job, an eval, and a deployment.

The [LoRA fine tuning guide](https://aifordevs.org/articles/lora-fine-tuning) shows what that training job does to the weights on an open model. This article stays at the level of which failures justify running one.

## Try prompting and context first

Prompting and context come first in any RAG vs fine tuning decision. They cost no training, take effect on the next request, and set the baseline every other option must beat. Most teams should stay here until an eval proves otherwise.

[Google's tuning guide](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/supervised-tuning/use) says to consider tuning only when prompts do not produce the expected results consistently enough. [OpenAI's fine-tuning guide](https://developers.openai.com/api/docs/guides/supervised-fine-tuning) says to set up evals first and to rethink the task or prompt if 50 examples have no impact. Do the prompt work with a test set beside you.

Work through these before you spend on anything else.

1. Add the failing case as a few-shot example, and move stable instructions to the top of the prompt.
2. Cut retrieved or pasted text that does not change the answer. The [context engineering guide](https://aifordevs.org/articles/context-engineering) covers budgeting tokens across instructions, tools, and history.
3. Put a schema on the output if the failure is structural.
4. Turn on [prompt caching](https://aifordevs.org/glossary#prompt-caching) so a long stable prefix stops being an argument for shortening it.

Stop when the cases in your eval still fail after you added an instruction and an example that cover them. Whatever fails at that point is your real target.

## Decision table by failure mode

To use this RAG vs fine tuning table, find the failure you observe in the left column and apply the first fix before you consider retrieval or training. The verdict names the option that addresses the cause, and rows marked "Cannot" are structural, so more budget will not change them.

| Failure you observe | Try first | RAG | Fine tuning | Verdict |
| --- | --- | --- | --- | --- |
| Stale facts, such as last week's price | Paste the current value into context | Fixes it, swap the index | Cannot fix it without retraining | RAG |
| Facts the model never saw, such as internal docs | Add the document if it is short | Fixes it | Weak on rare facts | RAG |
| No citation or audit trail | None | Fixes it, retrieved chunks have IDs | Cannot, weights carry no source pointer | RAG |
| Per-user data access | Filter before the prompt is built | Fixes it, filter at retrieval | Cannot, weights are shared by all callers | RAG |
| Invalid JSON or schema violations | Structured outputs | No effect | Reduces the rate, no guarantee | Schema enforcement |
| Right schema, wrong field values | Few-shot examples | Helps when the value is in a document | Helps with consistent mappings | Prompt, then fine tuning |
| Tone or house style | System prompt and examples | No effect | Fixes consistent style | Fine tuning if prompts fail |
| Wrong tool selected | Fewer tools, better descriptions | Tool search retrieves from the catalog | Provider-suggested for large tool sets | Context, then tool search, then fine tuning |
| Malformed tool arguments | Strict schema mode | No effect | Helps with habits | Strict mode |
| Long prompt driving cost | Prompt caching | Adds tokens | Shortens the prompt, may raise the price | Caching, then recheck |
| Answer passage not retrieved | Improve chunking and search | The failing component | Cannot fix it | Fix retrieval |
| Passage retrieved, output ignores it | Reorder context, ask for quotes | Already present | Train with retrieved context | Combine |

## What RAG can and cannot fix

RAG fixes failures where the needed fact exists in a document you control and fails to reach the model. It cannot fix a retriever that misses the passage, an output that ignores a correct passage, or a style problem. Retrieved text changes what the model reads and leaves the weights alone.

The clearest evidence is Lewis et al.'s index hot-swap in Section 4.5. They built one index from a December 2016 Wikipedia dump, kept another from December 2018, and asked about 82 world leaders who had changed. RAG answered 70 percent correctly with the 2016 index on 2016 leaders and 68 percent with the 2018 index on 2018 leaders.

Mismatched indices scored 12 percent and 4 percent. The paper notes that parametric-only models need further training to pick up the same change.

Rare facts tell the same story. [Soudani et al.](https://arxiv.org/abs/2403.01432) tested twelve language models. They report that RAG beat fine tuning by a large margin on the least popular knowledge, while fine tuning still helped across popularity levels.

[Berglund et al.](https://arxiv.org/abs/2309.12288) report a related limit. GPT-3 and Llama-1 models fine-tuned on "A is B" statements could not answer the reversed question. The authors note that the reverse works when the statement sits in the context window.

Citations and permissions follow directly from where the data lives. A retrieved chunk has a document ID you can show and check, and a retrieval filter can apply per-user access rules. Weights have neither, and every caller shares them.

When the right passage never arrives, the fix is in retrieval. Check [chunking](https://aifordevs.org/glossary#chunking), then [hybrid search](https://aifordevs.org/glossary#hybrid-search) and [reranking](https://aifordevs.org/glossary#reranking), and see the [vector database guide](https://aifordevs.org/articles/what-is-a-vector-database) for index choices. Measure with an eval that separates retrieval hits from answer quality, as the [LLM evaluation guide](https://aifordevs.org/articles/llm-evaluation) describes.

## What fine tuning can and cannot fix

Fine tuning fixes behavior that stays consistent across requests, such as output structure, tone, classification boundaries, and tool-calling habits. It cannot deliver facts that change after training, cite a source, or enforce per-user access. Google's tuning page lists classification, summarization, extractive question answering, and chat among its use cases.

OpenAI's [optimization guide](https://developers.openai.com/api/docs/guides/model-optimization) adds nuanced translation and correcting instruction-following failures. Its [preference-optimization guide](https://developers.openai.com/api/docs/guides/direct-preference-optimization) targets chat messages with the right tone and style. Those are behaviors, and every one of them can be demonstrated with examples.

Format errors have a cheaper fix. OpenAI says [Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs) make responses follow your JSON Schema, with refusals as the exception. On open models, [vLLM's structured outputs](https://docs.vllm.ai/en/stable/features/structured_outputs/) use xgrammar or guidance as [constrained decoding](https://aifordevs.org/glossary#constrained-decoding) backends.

Do not stack both by default. [Google warns](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/supervised-tuning) that applying controlled generation to tuned Gemini models can decrease quality because of a mismatch between tuning and inference.

Tool-call accuracy has more than one layer. OpenAI's [function-calling guide](https://developers.openai.com/api/docs/guides/function-calling) advises fewer than 20 functions at the start of a turn and recommends strict mode. It suggests fine tuning for large function sets or difficult tasks.

Anthropic says Claude's [tool selection](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) degrades past 30 to 50 tools, a vendor figure with no independent measurement cited, and offers tool search as a fix.

Tool search is RAG over your tool catalog. Try it before fine tuning on any agent with a large [tool-calling](https://aifordevs.org/glossary#tool-calling) surface. The guides on [AI agents](https://aifordevs.org/articles/what-is-an-ai-agent) and [MCP servers](https://aifordevs.org/articles/build-an-mcp-server) cover the tool design around it.

A tuned model also inherits its base model's lifespan. OpenAI's [deprecations page](https://developers.openai.com/api/docs/deprecations) says fine-tuned versions follow the same shutdown schedule as their base, and `gpt-4.1-mini` is scheduled to shut down on January 14, 2027.

Amazon Bedrock's [lifecycle policy](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html) blocks new fine-tuning jobs once a model enters Legacy, and Google excludes supervised fine tuning from its SLOs. Weights you host on an open base do not expire on a vendor's schedule.

## Who still offers fine tuning

Google's Agent Platform, Amazon Bedrock, Together AI, and Fireworks AI still sell fine tuning as of November 8, 2026. OpenAI is winding its platform down, Mistral has deprecated its fine-tuning feature, and Google's Gemini API has no tunable model. Check availability before you plan a RAG vs fine tuning decision around it.

| Provider | Status on November 8, 2026 | What you can tune | Training price | Tuned inference |
| --- | --- | --- | --- | --- |
| OpenAI | Closed to new organizations. Existing customers until January 6, 2027 | `gpt-4.1` and `gpt-4.1-mini` snapshots | $5.00 per 1M tokens on `gpt-4.1-mini` | $0.80 input, $3.20 output per 1M tokens, 2 times base |
| Google Agent Platform | Available | Gemini 3.5 Flash-Lite, 3.5 Flash, 3.1 Flash-Lite, 2.5 Pro, 2.5 Flash-Lite, 2.5 Flash | $3.00 per 1M tokens on 3.5 Flash-Lite | Base price in global region, 1.5 times in non-global regions for Gemini 3 and later |
| Gemini API and AI Studio | No tunable model | None | None | None |
| Amazon Bedrock | Available | Nova, Titan, Llama, and Claude 3 Haiku per its table | Per token, plus monthly storage | Not compared here |
| Mistral | Deprecated, no longer actively supported | Not stated | $4 minimum per job, $2 monthly storage per model | Not stated |
| Anthropic | No fine-tuning rate on its pricing page | Bedrock's table lists Claude 3 Haiku | None listed | None listed |
| Together AI | Available | Open models, LoRA by default | From $0.34 per 1M tokens, job minimums from $4 | Dedicated endpoints billed per minute |
| Fireworks AI | Available | Open models | $0.50 per 1M tokens LoRA SFT, up to 16B | Deployment billed per GPU-second |

Sources for the table are [OpenAI](https://developers.openai.com/api/docs/deprecations) and [its pricing page](https://developers.openai.com/api/docs/pricing), [Google's pricing page](https://cloud.google.com/model-garden/pricing), [AI Studio](https://aistudio.google.com/docs/model-tuning), [Bedrock customization](https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html) and [its model table](https://docs.aws.amazon.com/bedrock/latest/userguide/custom-model-fine-tuning.html), [Mistral](https://docs.mistral.ai/resources/deprecated/finetuning), [Anthropic](https://platform.claude.com/docs/en/about-claude/pricing), [Together's pricing](https://www.together.ai/pricing) and [fine-tuning docs](https://docs.together.ai/docs/fine-tuning/overview), and [Fireworks](https://fireworks.ai/pricing). The [platforms directory](https://aifordevs.org/platforms) tracks hosts beyond these.

## RAG vs fine tuning costs, worked in code

In this model, fine tuning does not win on token cost once prompt caching is in the comparison, and training is the smallest line. The script below prices one workload five ways at rates fetched on November 8, 2026. Token counts are assumptions for illustration, so change them to your traffic before you trust the result.

Prices come from [Google's Agent Platform pricing page](https://cloud.google.com/model-garden/pricing), whose documentation still sits under Vertex AI URLs, and [OpenAI's pricing page](https://developers.openai.com/api/docs/pricing). OpenAI's column is useful only if you already have access, and Gemini Embedding 2 is priced as a preview.

```python title="costs.py"
REQUESTS = 1_000_000  # per month
PREFIX = 5_000        # instructions, tool definitions, few-shot examples
TUNED_PREFIX = 1_200  # the examples moved into the weights
USER = 300
RETRIEVED = 3_000     # six 500-token chunks
OUTPUT = 400
HIT = 0.9             # assumed share of a long stable prefix served from cache

# USD per 1M tokens, as of November 8, 2026.
VERTEX = dict(input=0.30, cached=0.03, output=2.50, embed=0.20, train=3.00)
OPENAI = dict(input=0.40, cached=0.10, output=1.60, embed=0.02, train=5.00)
VERTEX_TUNED_GLOBAL = dict(input=0.30, cached=0.03, output=2.50)
VERTEX_TUNED_REGIONAL = dict(input=0.33 * 1.5, cached=0.033 * 1.5, output=2.75 * 1.5)
OPENAI_TUNED = dict(input=0.80, cached=0.20, output=3.20)


def monthly(price, prefix, hit=0.0, retrieved=0, embed=None):
    cached = prefix * hit
    fresh = prefix - cached + USER + retrieved
    embedding = USER * embed if embed else 0
    parts = {
        "cached_in": cached * price["cached"],
        "fresh_in": fresh * price["input"] + embedding,
        "output": OUTPUT * price["output"],
    }
    return {k: v * REQUESTS / 1e6 for k, v in parts.items()}


def report(name, base, tuned):
    embed = base["embed"]
    rows = [
        ("prompt, no cache", monthly(base, PREFIX)),
        ("prompt + cache", monthly(base, PREFIX, HIT)),
        ("RAG + cache", monthly(base, PREFIX, HIT, RETRIEVED, embed)),
        ("tuned", monthly(tuned, TUNED_PREFIX)),
        ("tuned + RAG", monthly(tuned, TUNED_PREFIX, 0.0, RETRIEVED, embed)),
    ]
    print(name)
    for label, parts in rows:
        total = sum(parts.values())
        detail = "  ".join(f"{k} {v:8,.0f}" for k, v in parts.items())
        print(f"  {label:18s} ${total:>8,.0f}   {detail}")
    # Prefix length at which tuning matches prompt + cache.
    tuned_cost = TUNED_PREFIX * tuned["input"] + USER * tuned["input"] + OUTPUT * tuned["output"]
    per_prefix_token = HIT * base["cached"] + (1 - HIT) * base["input"]
    fixed = USER * base["input"] + OUTPUT * base["output"]
    print(f"  break-even prefix: {(tuned_cost - fixed) / per_prefix_token:,.0f} tokens")


report("Vertex global, Gemini 3.5 Flash-Lite", VERTEX, VERTEX_TUNED_GLOBAL)
report("Vertex non-global tuned endpoint (1.5x)", VERTEX | dict(input=0.33, cached=0.033, output=2.75), VERTEX_TUNED_REGIONAL)
report("OpenAI gpt-4.1-mini", OPENAI, OPENAI_TUNED)

# One-off costs.
examples, tokens_each, epochs = 2_000, 1_900, 3
train_tokens = examples * tokens_each * epochs
print(f"training tokens: {train_tokens:,}")
for name, rate in [("Vertex 3.5 Flash-Lite", 3.00), ("OpenAI gpt-4.1-mini", 5.00), ("Fireworks LoRA SFT, up to 16B", 0.50)]:
    print(f"  {name:32s} ${train_tokens * rate / 1e6:,.2f}")
corpus = 200_000 * 500
print(f"corpus embedding, {corpus:,} tokens: OpenAI ${corpus * 0.02 / 1e6:,.2f}, Gemini Embedding 2 ${corpus * 0.20 / 1e6:,.2f}")
print(f"dedicated H100 for a month: ${8.00 * 730:,.0f}")
# If OpenAI's tuned prefix did hit the cache.
alt = monthly(OPENAI_TUNED, TUNED_PREFIX, HIT)
print(f"OpenAI tuned with cached prefix: ${sum(alt.values()):,.0f}")
```

```text title="Real output of python3 costs.py"
Vertex global, Gemini 3.5 Flash-Lite
  prompt, no cache   $   2,590   cached_in        0  fresh_in    1,590  output    1,000
  prompt + cache     $   1,375   cached_in      135  fresh_in      240  output    1,000
  RAG + cache        $   2,335   cached_in      135  fresh_in    1,200  output    1,000
  tuned              $   1,450   cached_in        0  fresh_in      450  output    1,000
  tuned + RAG        $   2,410   cached_in        0  fresh_in    1,410  output    1,000
  break-even prefix: 6,316 tokens
Vertex non-global tuned endpoint (1.5x)
  prompt, no cache   $   2,849   cached_in        0  fresh_in    1,749  output    1,100
  prompt + cache     $   1,512   cached_in      148  fresh_in      264  output    1,100
  RAG + cache        $   2,562   cached_in      148  fresh_in    1,314  output    1,100
  tuned              $   2,392   cached_in        0  fresh_in      742  output    1,650
  tuned + RAG        $   3,938   cached_in        0  fresh_in    2,288  output    1,650
  break-even prefix: 19,035 tokens
OpenAI gpt-4.1-mini
  prompt, no cache   $   2,760   cached_in        0  fresh_in    2,120  output      640
  prompt + cache     $   1,410   cached_in      450  fresh_in      320  output      640
  RAG + cache        $   2,616   cached_in      450  fresh_in    1,526  output      640
  tuned              $   2,480   cached_in        0  fresh_in    1,200  output    1,280
  tuned + RAG        $   4,886   cached_in        0  fresh_in    3,606  output    1,280
  break-even prefix: 13,231 tokens
training tokens: 11,400,000
  Vertex 3.5 Flash-Lite            $34.20
  OpenAI gpt-4.1-mini              $57.00
  Fireworks LoRA SFT, up to 16B    $5.70
corpus embedding, 100,000,000 tokens: OpenAI $2.00, Gemini Embedding 2 $20.00
dedicated H100 for a month: $5,840
OpenAI tuned with cached prefix: $1,832
```

| Option, USD per month | Gemini 3.5 Flash-Lite, global | Gemini, non-global | OpenAI `gpt-4.1-mini` |
| --- | --- | --- | --- |
| Prompt, no cache | $2,590 | $2,849 | $2,760 |
| Prompt + cache | $1,375 | $1,512 | $1,410 |
| RAG + cache | $2,335 | $2,562 | $2,616 |
| Tuned | $1,450 | $2,392 | $2,480 |
| Tuned + RAG | $2,410 | $3,938 | $4,886 |
| Break-even stable prefix, tokens | 6,316 | 19,035 | 13,231 |

The break-even row is the prefix length at which a tuned model matches a cached prompt. Below 6,316 tokens on Gemini's global endpoint, caching the long prompt costs less than tuning it away. That number climbs to 13,231 tokens on OpenAI, where tuned tokens cost twice as much.

**Figure 2.** A cached prompt is the cheapest row on both providers. Retrieved tokens dominate the RAG rows, and a two times premium on every tuned token makes tuned plus RAG the most expensive option on OpenAI.

The tuned prefix stays uncached in this model. Google documents a [4,096-token caching minimum](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/context-cache/context-cache-overview) for most Gemini 3 models, and [OpenAI's minimum](https://developers.openai.com/api/docs/guides/prompt-caching) varies by request settings before GPT-5.6. If OpenAI's tuned prefix did hit the cache, the tuned row would fall to $1,832, still above the cached prompt.

Training is a rounding error at this scale. 2,000 examples at 1,900 tokens for three epochs is 11.4M training tokens. That costs $34.20 on Gemini 3.5 Flash-Lite, $57.00 on OpenAI, and $5.70 for LoRA on Fireworks under 16B.

Embedding a 100M-token corpus once costs $2 on OpenAI or $20 on Gemini Embedding 2. A dedicated GPU changes the shape of the bill. One on-demand H100 on Fireworks at $8.00 per GPU-hour is $5,840 for a 730-hour month, before redundancy.

The script leaves out vector store hosting, reranking, ingestion, labeling, and eval time. Add those from your own quotes, and compare token prices with the [Claude API pricing](https://aifordevs.org/articles/claude-api-pricing) and [Gemini API pricing](https://aifordevs.org/articles/gemini-api-pricing) guides.

## When to combine RAG and fine tuning

Combine them when facts change and a behavior failure also survives a good prompt, or when retrieval finds the right passage and the output still ignores it. Train on the retrieved context you serve in production, because a model trained on clean question-answer pairs never saw the distractors your retriever returns.

[Zhang et al.'s RAFT](https://arxiv.org/abs/2403.10131) makes that explicit. Training examples mix relevant documents with distractors that do not contain the answer, and the target output is a reasoning chain that cites the source passage.

The authors report that RAFT on LLaMA2-7B scored 74.00 on HuggingFace API questions, against 26.43 for the same model with plain RAG and 29.08 for GPT-3.5 with RAG. On HotPot QA, GPT-3.5 with RAG scored 41.5 and RAFT scored 35.28, so the gain is not universal.

Use these criteria before committing to both.

- A fact-shaped bucket and a behavior-shaped bucket both remain in your eval after prompt work.
- The retriever hits the right passage on most failing cases, so the output is the failing part.
- One task carries enough volume that its stable prefix is past the break-even length above.

Remember the premium. On OpenAI's rates, tuned plus RAG cost $4,886 against $2,616 for RAG alone, because every retrieved token is billed at the tuned rate. On Gemini's global endpoint the same pair cost $2,410 against $2,335.

## Settle RAG vs fine tuning with a failure audit

Settle RAG vs fine tuning by counting real failures, because the largest failure bucket names the cheapest fix. Spend a day on this before you build a pipeline or a training set. The steps below need only your logs and a spreadsheet.

1. Pull 100 recent failures from logs or a [golden dataset](https://aifordevs.org/glossary#golden-dataset).
2. Label each one with a row from the decision table. If it fits none, write a new row.
3. Fix the largest bucket with the first-try column, and rerun the eval.
4. Repeat until the remaining buckets resist prompting and context.
5. Route fact-shaped survivors to RAG. See [RAG topics](https://aifordevs.org/topics/rag) for the retrieval side.
6. Route behavior-shaped survivors to fine tuning, starting with 50 examples, and compare against your best cached prompt on accuracy and monthly cost. The [LLM as a judge guide](https://aifordevs.org/articles/llm-as-a-judge) covers open-ended scoring.

If half your failures are stale or missing facts, stop and build retrieval. If what survives is a small bucket of format or style failures, start with a LoRA adapter on an open model. Choose a host that will not retire your base before the adapter pays off.

## Questions

### Is RAG better than fine tuning?

RAG is better for facts and fine tuning is better for consistent behavior, so neither wins in general. Use RAG when answers depend on documents that change or that you must cite. Use fine tuning when evals show a format, style, or classification failure that survives a good prompt. Most teams should try prompting and context before either one.

### Can you combine RAG and fine tuning?

Yes, and the useful version trains on retrieved context. Training on clean question-answer pairs and adding retrieval only at serving time mismatches what the model sees in production. Zhang et al.'s RAFT trains on documents that include distractors and on answers that cite the relevant passage. Expect to pay any tuned-model price premium on every retrieved token.

### Does fine tuning reduce hallucinations?

Fine tuning can correct instruction-following failures, but it does not give a model access to facts it lacks. Soudani et al. found that RAG beat fine tuning by a large margin on the least popular facts. If a hallucination comes from missing knowledge, retrieval is the direct fix, and you still need to evaluate whether answers stay faithful to the retrieved text.

### Is fine tuning cheaper than RAG?

Training is cheap, and tuned inference is where the cost sits. In this article's example at 1M requests a month, training cost $34 to $57 once, a tuned model cost $1,450 to $2,480 a month, and RAG with a cached prefix cost $2,335 to $2,616. The two options solve different problems, so compare them only when either one would fix the failure.

### Can you still fine-tune GPT models with OpenAI?

Only if your organization already used the platform. As of November 8, 2026, OpenAI has closed fine tuning to new organizations, and eligible existing customers can create jobs until January 6, 2027, unless the base model shuts down earlier. Inference on fine-tuned models continues until the base model is retired, and gpt-4.1-mini is scheduled to shut down on January 14, 2027.

### When should you fine-tune an LLM instead of changing the prompt?

Fine-tune when an eval shows a behavior failure that persists after you have added instructions, examples, and a schema, and you have labeled data. Google's tuning guide says to consider tuning when prompts do not produce the expected results consistently enough. OpenAI's guide says to set up evals first and to rethink the task if 50 examples have no impact.

### How do you fine-tune an LLM?

Collect labeled examples in your provider's JSONL format, hold out a test set, train, and compare the tuned model with the base model and your best prompt. On open models, LoRA through Hugging Face PEFT is a common low-cost path, and the LoRA guide on this site includes a script that was executed with real output. On hosted models, check availability first.

## Sources

1. [Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)](https://arxiv.org/abs/2005.11401)
2. [Soudani et al., Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge (2024)](https://arxiv.org/abs/2403.01432)
3. [Berglund et al., The Reversal Curse (2023)](https://arxiv.org/abs/2309.12288)
4. [Zhang et al., RAFT: Adapting Language Model to Domain Specific RAG (2024)](https://arxiv.org/abs/2403.10131)
5. [OpenAI, Model optimization guide](https://developers.openai.com/api/docs/guides/model-optimization)
6. [OpenAI, Supervised fine-tuning guide](https://developers.openai.com/api/docs/guides/supervised-fine-tuning)
7. [OpenAI, Deprecations](https://developers.openai.com/api/docs/deprecations)
8. [OpenAI, API pricing](https://developers.openai.com/api/docs/pricing)
9. [OpenAI, Structured Outputs guide](https://developers.openai.com/api/docs/guides/structured-outputs)
10. [OpenAI, Function calling guide](https://developers.openai.com/api/docs/guides/function-calling)
11. [OpenAI, Prompt caching guide](https://developers.openai.com/api/docs/guides/prompt-caching)
12. [OpenAI, Direct preference optimization guide](https://developers.openai.com/api/docs/guides/direct-preference-optimization)
13. [Google Cloud, Agent Platform pricing](https://cloud.google.com/model-garden/pricing)
14. [Google Cloud, About supervised fine-tuning for Gemini models](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/supervised-tuning)
15. [Google Cloud, Use supervised tuning for Gemini models](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/supervised-tuning/use)
16. [Google Cloud, Context caching overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/context-cache/context-cache-overview)
17. [Google AI Studio, Model tuning](https://aistudio.google.com/docs/model-tuning)
18. [Amazon Bedrock, Customize your model](https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html)
19. [Amazon Bedrock, Fine-tuning supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/custom-model-fine-tuning.html)
20. [Amazon Bedrock, Model lifecycle](https://docs.aws.amazon.com/bedrock/latest/userguide/model-lifecycle.html)
21. [Mistral AI, Fine-tuning API (legacy, marked deprecated)](https://docs.mistral.ai/resources/deprecated/finetuning)
22. [Anthropic, Pricing](https://platform.claude.com/docs/en/about-claude/pricing)
23. [Anthropic, Tool search tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool)
24. [Together AI, Pricing](https://www.together.ai/pricing)
25. [Together AI, Fine-tuning pricing](https://docs.together.ai/docs/fine-tuning/pricing)
26. [Together AI, Fine-tuning overview](https://docs.together.ai/docs/fine-tuning/overview)
27. [Fireworks AI, Pricing](https://fireworks.ai/pricing)
28. [vLLM, Structured Outputs](https://docs.vllm.ai/en/stable/features/structured_outputs/)

## Cite this deep dive

AI for Devs. "RAG vs fine tuning: which failures each one can fix." Published November 8, 2026. https://aifordevs.org/articles/rag-vs-fine-tuning

- Page: https://aifordevs.org/articles/rag-vs-fine-tuning
- Markdown: https://aifordevs.org/articles/rag-vs-fine-tuning.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "What is a vector database? HNSW, IVF, and when to skip one"
type: "deep dive"
url: "https://aifordevs.org/articles/what-is-a-vector-database"
markdown: "https://aifordevs.org/articles/what-is-a-vector-database.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/what-is-a-vector-database"
description: "What is a vector database, how HNSW and IVF search work, recall measured on generated vectors, and when pgvector or no database is the right choice."
author: "AI for Devs"
published: "2026-11-08"
level: "Intermediate"
topics: ["rag"]
---

# What is a vector database? HNSW, IVF, and when to skip one

What is a vector database, how HNSW and IVF search work, recall measured on generated vectors, and when pgvector or no database is the right choice.

> **TL;DR**
>
> - What is a vector database: a data store that indexes embeddings and returns the items nearest a query vector, usually through an approximate index such as HNSW or IVF.
> - Exact search cost grows linearly with vector count. In this article's test, on 400,000 generated 768-dimensional vectors, exact search used 41.8 ms of CPU per query, and HNSW used 0.17 ms at 0.877 recall (Faiss 1.15.1, NumPy 2.5.4, Python 3.15.1, Apple M2).
> - Recall depends on your data, so test on your own embeddings. The same HNSW settings scored 0.936 on tightly clustered vectors, 0.479 on loosely clustered ones, and 0.019 on structureless Gaussian vectors.
> - A separate vector database is unnecessary when each query searches fewer than about 100,000 vectors, or when your vectors fit on a Postgres instance you already run with pgvector.
> - Pick a dedicated store by workload and team, as of November 8, 2026: filters, tenants, scale, and who runs it. There is no best vector database in general.

A vector database is a data store that indexes embeddings and returns the items closest to a query vector, usually through an approximate nearest-neighbor index. To answer what is a vector database in engineering terms, this guide starts with embeddings and nearest-neighbor search, then opens up HNSW and IVF. A runnable, offline test compares exact and approximate search on generated vectors. The last sections give decision rules for choosing a store, and a position on when you should not add one.

## What is a vector database

A vector database stores embeddings with IDs and metadata, builds an index over them, and answers nearest-neighbor queries through an API. An index library such as Faiss does only the search step. The database adds durable storage, filtering, updates, replication, and access control, and those additions cause most of the operational differences between products.

Faiss's [guidelines](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index) say all of its indexes are stored in RAM, and a library has no notion of tenants or backups. Those are the jobs a database takes on.

[RAG](https://aifordevs.org/glossary#rag) is a common reason developers meet one. A RAG system [chunks](https://aifordevs.org/glossary#chunking) documents, embeds each chunk, and retrieves the closest chunks for each question. If you are deciding whether retrieval is the right approach at all, [RAG vs fine-tuning](https://aifordevs.org/articles/rag-vs-fine-tuning) covers that choice first.

## How embeddings turn similarity into distance

An [embedding](https://aifordevs.org/glossary#embedding) model maps text, images, or code to a fixed-length list of numbers, and nearby vectors mean related content. OpenAI's [embeddings guide](https://developers.openai.com/api/docs/guides/embeddings) says small distances suggest high relatedness. Search then means finding the stored vectors closest to the query's vector.

As of November 8, 2026, OpenAI's `text-embedding-3-small` returns 1,536 dimensions by default and `text-embedding-3-large` returns 3,072. Those vectors are normalized to length 1, so cosine similarity reduces to a dot product and ranks results the same way as Euclidean distance.

The example later in this article uses inner product on normalized vectors for that reason. If your model does not normalize, check which distance metric your store expects, because a mismatch silently changes rankings.

## Why exact nearest-neighbor search stops scaling

Exact search compares the query with every stored vector, so its cost grows linearly with both vector count and dimension. It gives perfect recall, and it is fine for tens of thousands of vectors. Doubling the corpus doubles the CPU per query, and a single core soon cannot keep up with real traffic.

Elastic's [kNN docs](https://www.elastic.co/docs/solutions/search/vector/knn) say exact search "does not scale well for large datasets." The measurements below show the shape on 768-dimensional generated vectors. They use one CPU core and report CPU time per query, so other jobs on the machine do not distort the result:

| Vectors | Exact, CPU ms per query | Exact, queries per second on one core | HNSW `efSearch` 40, CPU ms per query | HNSW recall@10 |
| --- | --- | --- | --- | --- |
| 50,000 | 5.26 | 190 | 0.309 | 0.921 |
| 100,000 | 10.22 | 98 | 0.226 | 0.935 |
| 200,000 | 24.90 | 40 | 0.237 | 0.936 |
| 400,000 | 41.79 | 24 | 0.169 | 0.877 |

Exact CPU time grows about linearly with N. HNSW at `efSearch` 40 barely moves, because a graph walk visits a small fraction of the vectors. Its recall did drift, to 0.877 at 400,000 vectors, so a larger corpus needs a larger `efSearch`. Memory is the other limit. One million 1,536-dimensional vectors need 5.7 GiB as raw `float32`, before any index.

## How HNSW searches a layered graph

[HNSW](https://aifordevs.org/glossary#hnsw) stacks proximity graphs so that each layer holds a smaller random subset of the vectors. A query starts at the top layer, walks greedily toward the target, drops down at each local minimum, and searches widely only at the bottom. Malkov and Yashunin [introduced it](https://arxiv.org/abs/1603.09320) in 2016, and the name stands for Hierarchical Navigable Small World.

**Figure 1.** HNSW moves greedily through sparse upper layers with a candidate list of one, then explores efSearch candidates at layer 0. Horizontal position stands in for position in vector space.

The paper assigns each vector a maximum layer of `floor(-ln(unif(0,1)) * mL)` and suggests `mL = 1/ln(M)`, which matches a skip list where each layer keeps 1 in `M` nodes. With `M = 16` and one million vectors, expect about 62,500 nodes in layer 1, 3,906 in layer 2, 244 in layer 3, 15 in layer 4, and 1 in layer 5.

Search uses `ef = 1` in the upper layers, which is why the walk in Figure 1 follows a single path. At layer 0 it keeps `efSearch` candidates and returns the best k. The paper reports a reasonable `M` range of 5 to 48, and link memory of about 60 to 450 bytes per object, excluding the vectors.

Defaults differ across stores, so "HNSW with default settings" means different things:

| Store | Graph degree | Build list | Query list |
| --- | --- | --- | --- |
| [pgvector](https://github.com/pgvector/pgvector) | `m` = 16 | `ef_construction` = 64 | `hnsw.ef_search` = 40 |
| [Qdrant](https://qdrant.tech/documentation/manage-data/indexing/) | `m` = 16 | `ef_construct` = 100 | Equals `ef_construct` by default |
| [Weaviate](https://docs.weaviate.io/weaviate/config-refs/indexing/vector-index) | `maxConnections` = 32 | `efConstruction` = 128 | `ef` = -1, dynamic |

HNSW's costs are memory and build time. Faiss documents `d * 4 + M * 2 * 4` bytes per vector, and pgvector's README says HNSW has slower builds and uses more memory than IVFFlat. Faiss HNSW also [cannot remove vectors](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index), so deletes need a store that handles them for you.

## How IVF partitions vectors into cells

IVF clusters the vectors with k-means, stores each vector in the list of its nearest centroid, and at query time scans only the `nprobe` closest lists. Jégou, Douze, and Schmid's [IVFADC paper](https://inria.hal.science/file/index/docid/825085/filename/jegou_pq_postprint.pdf) pairs that inverted file with compressed codes. Raising `nprobe` improves recall and scans more vectors.

**Figure 2.** IVF compares the query only with vectors in the nprobe closest cells. A true neighbor just across a cell boundary is missed until nprobe grows large enough to include its cell.

Faiss's [guidelines](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index) suggest between `4*sqrt(N)` and `16*sqrt(N)` cells below one million vectors, with 30 to 256 training vectors per cell. IVF needs that training step before it accepts data, which HNSW does not.

A flat IVF list costs `4*d + 8` bytes per vector, according to [Faiss's index table](https://github.com/facebookresearch/faiss/wiki/Faiss-indexes). Product quantization replaces the raw vector with `ceil(M * nbits / 8)` bytes, which is how Jégou's team [validated the approach on two billion vectors](https://inria.hal.science/file/index/docid/825085/filename/jegou_pq_postprint.pdf). The Faiss authors [built a nearest-neighbor graph over one billion vectors](https://arxiv.org/abs/1702.08734) in under 12 hours on four GPUs. pgvector's README describes IVFFlat as faster to build and lighter on memory than HNSW, with a worse speed-recall trade-off.

## Exact vs HNSW vs IVF on generated vectors

On 200,000 generated vectors, HNSW at `efSearch` 40 returned 93.6% of the true top 10 and cost about 1% of exact search's CPU time per query. IVF did better on this dataset because its k-means cells line up with the topics the generator created. Treat these numbers as a method, because recall changes with the shape of your data.

```python title="ann_compare.py"
"""Compare exact, HNSW, and IVF search on generated vectors. Runs offline."""
import sys
import time

import faiss
import numpy as np

N = int(sys.argv[1]) if len(sys.argv) > 1 else 200_000
SPREAD = sys.argv[2] if len(sys.argv) > 2 else "0.5"  # or "gaussian"
D, K, QUERIES = 768, 10, 500


def make_vectors(n, seed):
    rng = np.random.default_rng(seed)
    if SPREAD == "gaussian":  # no structure at all
        x = rng.standard_normal((n, D), dtype=np.float32)
    else:  # 2,000 topics, each varying along 16 hidden directions
        topics = np.random.default_rng(1).standard_normal((2_000, D), dtype=np.float32)
        basis = np.random.default_rng(2).standard_normal((16, D), dtype=np.float32) / 4
        x = topics[rng.integers(0, 2_000, n)]
        x += float(SPREAD) * (rng.standard_normal((n, 16), dtype=np.float32) @ basis)
        x += 0.05 * rng.standard_normal((n, D), dtype=np.float32)
    return x / np.linalg.norm(x, axis=1, keepdims=True)  # cosine = inner product


def measure(index, queries, truth):
    """Single-thread CPU time per query, one query at a time, plus recall@K."""
    faiss.omp_set_num_threads(1)
    found, start = [], time.process_time()  # CPU time resists noisy neighbors
    for q in queries:
        found.append(index.search(q[None, :], K)[1][0])
    ms = (time.process_time() - start) / len(queries) * 1000
    hits = sum(len(set(f) & set(t)) for f, t in zip(found, truth))
    return hits / (K * len(queries)), ms


def report(name, recall, ms, build_s, index):
    mb = faiss.serialize_index(index).size / 1e6
    print(f"{name:22} recall@{K}={recall:6.3f} {ms:8.3f} cpu_ms/query  "
          f"build={build_s:6.1f}s size={mb:6.0f} MB", flush=True)


xb = make_vectors(N, seed=3)
xq = make_vectors(QUERIES, seed=4)
threads = faiss.omp_get_max_threads()
print(f"faiss {faiss.__version__}, numpy {np.__version__}, N={N:,}, d={D}, spread={SPREAD}")

flat = faiss.IndexFlatIP(D)
flat.add(xb)
truth = flat.search(xq, K)[1]  # ground truth, computed with all threads
recall, ms = measure(flat, xq, truth)
report("exact (Flat)", recall, ms, 0.0, flat)

faiss.omp_set_num_threads(threads)
start = time.perf_counter()
hnsw = faiss.IndexHNSWFlat(D, 16, faiss.METRIC_INNER_PRODUCT)  # M=16
hnsw.hnsw.efConstruction = 64
hnsw.add(xb)
build = time.perf_counter() - start
for ef in (10, 40, 160):
    hnsw.hnsw.efSearch = ef
    recall, ms = measure(hnsw, xq, truth)
    report(f"HNSW efSearch={ef}", recall, ms, build, hnsw)

faiss.omp_set_num_threads(threads)
nlist = int(4 * np.sqrt(N))
start = time.perf_counter()
ivf = faiss.IndexIVFFlat(faiss.IndexFlatIP(D), D, nlist, faiss.METRIC_INNER_PRODUCT)
ivf.train(xb)
ivf.add(xb)
build = time.perf_counter() - start
for nprobe in (1, 10, 40):
    ivf.nprobe = nprobe
    recall, ms = measure(ivf, xq, truth)
    report(f"IVF nprobe={nprobe}/{nlist}", recall, ms, build, ivf)
```

Run with `python ann_compare.py 200000 0.5`. Tested on Apple M2, 16 GB, Python 3.15.1, NumPy 2.5.4, and `faiss-cpu` 1.15.1. Index builds used all 8 cores, and query timings used one. Real output:

```text
faiss 1.15.1, numpy 2.5.4, N=200,000, d=768, spread=0.5
exact (Flat)           recall@10= 1.000   24.896 cpu_ms/query  build=   0.0s size=   614 MB
HNSW efSearch=10       recall@10= 0.671    0.128 cpu_ms/query  build=  66.0s size=   643 MB
HNSW efSearch=40       recall@10= 0.936    0.237 cpu_ms/query  build=  66.0s size=   643 MB
HNSW efSearch=160      recall@10= 0.991    0.657 cpu_ms/query  build=  66.0s size=   643 MB
IVF nprobe=1/1788      recall@10= 0.983    0.199 cpu_ms/query  build=  17.3s size=   622 MB
IVF nprobe=10/1788     recall@10= 1.000    0.297 cpu_ms/query  build=  17.3s size=   622 MB
IVF nprobe=40/1788     recall@10= 1.000    0.662 cpu_ms/query  build=  17.3s size=   622 MB
```

This test exercised Faiss's in-process indexes only. It did not exercise pgvector, Pinecone, Qdrant, Weaviate, Milvus, filters, updates, concurrent queries, or real embeddings. The machine was shared with other jobs, which is why the timings are CPU time.

Changing only the data shape shows why you should not trust someone else's recall numbers:

| Generated data, N = 200,000 | HNSW `efSearch` 40 recall@10 | HNSW `efSearch` 160 recall@10 | IVF `nprobe` 10 recall@10 | IVF `nprobe` 40 recall@10 |
| --- | --- | --- | --- | --- |
| Tight topics (`spread` 0.5) | 0.936 | 0.991 | 1.000 | 1.000 |
| Loose topics (`spread` 0.8) | 0.479 | 0.724 | 0.667 | 0.801 |
| No structure (`gaussian`) | 0.019 | 0.062 | 0.034 | 0.092 |

Structureless Gaussian vectors give an index nothing to exploit, and the bottom row shows it. Tight, cleanly separated clusters flatter IVF in the opposite direction. Your embeddings will land somewhere on this range, and only your own data can say where. Measure retrieval quality with the methods in [LLM evaluation](https://aifordevs.org/articles/llm-evaluation) once you have a candidate.

## When you do not need a separate vector database

You do not need a separate vector database when each query searches a small scope, or when your vectors fit on a Postgres instance you already run. A second datastore adds a second backup plan, a sync job, and a new way to serve stale data. Make it earn that cost with a measured limit.

Here is the position this article takes, which is judgment built on the numbers above and not a benchmark result:

1. Under about 100,000 vectors per search scope, use exact search. At this size the test above spent about 10 ms of CPU per query on one core. Multi-tenant apps often qualify, because each tenant is its own scope.
2. If you already run Postgres and your vectors number in the low millions, use pgvector with HNSW. You keep [ACID compliance, point-in-time recovery, and JOINs](https://github.com/pgvector/pgvector), and retrieval joins the same transaction as your rows.
3. Add a dedicated store only after you see a concrete signal, such as those below.

Stay in Postgres until one of these happens:

- The `hnsw graph no longer fits into maintenance_work_mem` notice appears and you cannot grow memory.
- Filtered queries miss your recall target even with iterative scans.
- Index builds or vacuum start to hurt transactional latency.

Do the storage arithmetic first. pgvector documents `4 * dimensions + 8` bytes per vector, so five million 1,536-dimensional vectors take 28.65 GiB of vector storage before any index. That fits on a large single instance, but plan for the index on top.

If you are adding [agents](https://aifordevs.org/articles/what-is-an-ai-agent) to an existing product, check the database it already runs before you add another.

## Best vector database by workload and team

No product is the best vector database in general, so pick by your workload and the team that will operate it. The table gives a starting point for each common situation, with facts from each project's own docs as of November 8, 2026. It is not a ranking, and the order of rows implies nothing.

| Your situation | Start with | From its docs | Watch for |
| --- | --- | --- | --- |
| Already on Postgres, vectors under 2,000 dimensions | [pgvector](https://github.com/pgvector/pgvector) | Vectors live in Postgres rows, with exact, HNSW, and IVFFlat search | `vector` indexes stop at 2,000 dimensions and `halfvec` at 4,000 |
| No one to run a database, and hosted is acceptable | [Pinecone](https://docs.pinecone.io/guides/get-started/architecture) | Managed on AWS, GCP, and Azure, with [bring-your-own-cloud](https://docs.pinecone.io/guides/production/bring-your-own-cloud) on those clouds | No self-hosted option in the docs; [Starter limits](https://docs.pinecone.io/reference/api/database-limits/object-limits) cap storage at 2 GB and namespaces at 100 |
| Filter-heavy search, self-hosted | [Qdrant](https://qdrant.tech/documentation/manage-data/indexing/) | Filterable HNSW with payload indexes, and [scalar, binary, and product quantization](https://qdrant.tech/documentation/manage-data/quantization/) | Create payload indexes before ingest, or rebuild; [resharding is Cloud-only](https://qdrant.tech/documentation/scaling/distributed_deployment/) |
| Many tenants of uneven size, hybrid search built in | [Weaviate](https://docs.weaviate.io/weaviate/manage-collections/multi-tenancy) | One shard per tenant, with active, inactive, and offloaded states; [`dynamic` index](https://docs.weaviate.io/weaviate/concepts/vector-index) converts flat to HNSW at 10,000 objects | One-way upgrade; deletes do not convert it back |
| Billions of vectors, with a platform team | [Milvus](https://milvus.io/docs/architecture_overview.md) | Storage and compute are disaggregated; [Distributed mode targets tens of billions of vectors](https://milvus.io/docs/install-overview.md); [DiskANN and GPU indexes](https://milvus.io/docs/index-explained.md) | Operates etcd, object storage, and several node types |
| In-process, no server | [LanceDB](https://docs.lancedb.com/) or [Milvus Lite](https://milvus.io/docs/milvus_lite.md) | LanceDB runs inside your application process; Milvus Lite suits local and small workloads | Milvus Lite allows one process per data directory |
| Local first, then hosted | [Chroma](https://docs.trychroma.com/docs/overview/introduction) | Run locally, self-host, or use Chroma Cloud | Its intro page names no index type, so measure recall yourself |
| Already running Elasticsearch for keyword search | [Elasticsearch](https://www.elastic.co/docs/solutions/search/vector/knn) | `dense_vector` fields with approximate and exact kNN | Add vectors there before adding a new cluster |

Versions current on November 8, 2026 were pgvector 0.8.7, Qdrant 1.20.0, Weaviate 1.40.4, and Milvus 3.0.5.

Team shape decides the rest. If nobody owns database on-call, start with a managed service. If a platform team already runs Kubernetes and object storage, Milvus's moving parts will be familiar. Compare current managed offerings in the [platforms directory](https://aifordevs.org/platforms).

## Filtering and hybrid search change the answer

Filters are where stores differ most, because an approximate index can return too few results when the filter runs after the graph walk. pgvector's README gives a concrete case: with HNSW and the default `hnsw.ef_search` of 40, a filter matching 10% of rows returns about 4 rows on average.

pgvector's fix, since version 0.8.0, is [iterative index scans](https://github.com/pgvector/pgvector), which keep scanning until enough rows match or a limit such as `hnsw.max_scan_tuples` is reached. Qdrant takes another route: its [filterable HNSW](https://qdrant.tech/documentation/manage-data/indexing/) adds filter-aware edges, which only help if payload indexes exist before ingest.

[Hybrid search](https://aifordevs.org/glossary#hybrid-search) combines keyword and vector results, and exact identifiers such as error codes often need it. [Weaviate](https://docs.weaviate.io/weaviate/search/hybrid) fuses BM25F and vector scores, with Relative Score Fusion as the default since v1.24. pgvector's README points to Postgres full-text search combined with reciprocal rank fusion or a cross-encoder. Pinecone's [search docs](https://docs.pinecone.io/guides/search/hybrid-search) describe a single index that handles full-text and semantic search together.

Add a [reranking](https://aifordevs.org/glossary#reranking) step after retrieval if the top results are close but misordered. Retrieval quality is also a context problem, covered in [context engineering](https://aifordevs.org/articles/context-engineering).

## Test a vector database on your own embeddings

Replace `make_vectors` in the script above with a NumPy array of your real embeddings, then measure recall@10 against exact search at your actual filter selectivity. If pgvector or a flat index meets your recall and latency targets on a copy of your data, postpone adding a vector database and keep one fewer system to run.

1. Export 100,000 real embeddings and 500 real queries, normalized the same way your model produces them.
2. Compute exact top-10 results as ground truth with `IndexFlatIP`.
3. Tune HNSW `efSearch` until recall@10 reaches your target, and record CPU time per query.
4. Repeat with a filter that matches the same fraction of rows your production query matches.
5. Load the same vectors into a candidate store, and compare only at matched recall.

Set a recall target before you start, such as 0.95 at your p95 latency budget. Choose the store that reaches it with the least new operational work. New index types and store releases land in the [news feed](https://aifordevs.org/news), and the [RAG topic page](https://aifordevs.org/topics/rag) collects the rest of this retrieval series. If you also self-host the embedding model, the [vLLM vs SGLang comparison](https://aifordevs.org/articles/vllm-sglang-llama-cpp) covers the serving side.

## Questions

### What is a vector database used for?

A vector database is used to find items similar to a query by comparing embeddings. Common uses are retrieval for RAG, semantic search, recommendations, and deduplication. It stores vectors with IDs and metadata, indexes them for fast nearest-neighbor lookup, and applies filters such as tenant or date. A plain index library covers only the search step.

### Do I need a vector database for RAG?

Not always. If each query searches fewer than about 100,000 chunks, exact search is often fast enough. If you already run Postgres, pgvector adds HNSW and IVFFlat indexes to the same database. Add a dedicated store when you hit a measured limit, such as memory, filtered recall, or tenant isolation, and test it on your own embeddings first.

### Is PostgreSQL a vector database?

PostgreSQL becomes one with the pgvector extension, which adds vector columns plus exact and approximate nearest-neighbor search through HNSW and IVFFlat. Vectors sit next to your relational rows, so you keep ACID transactions, JOINs, and point-in-time recovery. Its README lists limits, such as indexing up to 2,000 dimensions for the vector type and 4,000 for halfvec.

### How is a vector database different from a relational database?

A relational database answers exact predicates, such as rows where status equals shipped, using B-tree and similar indexes. A vector database answers similarity queries, returning the rows closest to a query embedding, usually through an approximate index. Many systems need both in one query, which is why filtered vector search and extensions such as pgvector exist.

### What is HNSW?

HNSW, or Hierarchical Navigable Small World, is a graph index for approximate nearest-neighbor search, introduced by Malkov and Yashunin. It stacks proximity graphs in layers of decreasing size. A query moves greedily from the top layer to the bottom, where a list of efSearch candidates produces the results. Raising efSearch improves recall and costs more CPU per query.

### What is the best vector database?

No single vector database is best, because the right choice depends on your existing infrastructure, filter patterns, tenant count, and who runs it. Use pgvector if you already run Postgres. Consider Qdrant for heavy filtering, Weaviate for many tenants and built-in hybrid search, Milvus for billions of vectors, and Pinecone if you want a managed service.

## Sources

1. [Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (2016, revised 2018)](https://arxiv.org/abs/1603.09320)
2. [Jégou, Douze, and Schmid, Product Quantization for Nearest Neighbor Search (2011), author postprint](https://inria.hal.science/file/index/docid/825085/filename/jegou_pq_postprint.pdf)
3. [Johnson, Douze, and Jégou, Billion-scale similarity search with GPUs (2017)](https://arxiv.org/abs/1702.08734)
4. [Faiss wiki, Guidelines to choose an index](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index)
5. [Faiss wiki, Faiss indexes](https://github.com/facebookresearch/faiss/wiki/Faiss-indexes)
6. [OpenAI docs, Vector embeddings](https://developers.openai.com/api/docs/guides/embeddings)
7. [pgvector README](https://github.com/pgvector/pgvector)
8. [Pinecone docs, Database architecture](https://docs.pinecone.io/guides/get-started/architecture)
9. [Pinecone docs, Object limits](https://docs.pinecone.io/reference/api/database-limits/object-limits)
10. [Pinecone docs, Bring your own cloud](https://docs.pinecone.io/guides/production/bring-your-own-cloud)
11. [Pinecone docs, Hybrid search](https://docs.pinecone.io/guides/search/hybrid-search)
12. [Qdrant docs, Indexing](https://qdrant.tech/documentation/manage-data/indexing/)
13. [Qdrant docs, Distributed deployment](https://qdrant.tech/documentation/scaling/distributed_deployment/)
14. [Qdrant docs, Quantization](https://qdrant.tech/documentation/manage-data/quantization/)
15. [Weaviate docs, Vector indexing concepts](https://docs.weaviate.io/weaviate/concepts/vector-index)
16. [Weaviate docs, Vector index configuration](https://docs.weaviate.io/weaviate/config-refs/indexing/vector-index)
17. [Weaviate docs, Multi-tenancy](https://docs.weaviate.io/weaviate/manage-collections/multi-tenancy)
18. [Weaviate docs, Hybrid search](https://docs.weaviate.io/weaviate/search/hybrid)
19. [Milvus docs, Index explained](https://milvus.io/docs/index-explained.md)
20. [Milvus docs, Architecture overview](https://milvus.io/docs/architecture_overview.md)
21. [Milvus docs, Deployment options](https://milvus.io/docs/install-overview.md)
22. [Milvus docs, Milvus Lite](https://milvus.io/docs/milvus_lite.md)
23. [Chroma docs, Introduction](https://docs.trychroma.com/docs/overview/introduction)
24. [LanceDB docs](https://docs.lancedb.com/)
25. [Elastic docs, kNN search](https://www.elastic.co/docs/solutions/search/vector/knn)

## Cite this deep dive

AI for Devs. "What is a vector database? HNSW, IVF, and when to skip one." Published November 8, 2026. https://aifordevs.org/articles/what-is-a-vector-database

- Page: https://aifordevs.org/articles/what-is-a-vector-database
- Markdown: https://aifordevs.org/articles/what-is-a-vector-database.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "What is an AI agent? The loop in code, and where it fails"
type: "deep dive"
url: "https://aifordevs.org/articles/what-is-an-ai-agent"
markdown: "https://aifordevs.org/articles/what-is-an-ai-agent.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/what-is-an-ai-agent"
description: "What is an AI agent? A tested agent loop in TypeScript, run offline against a scripted model, with traces of runaway loops, tool errors, and token growth."
author: "AI for Devs"
published: "2026-11-08"
level: "Intro"
topics: ["agents", "coding-agents", "mcp"]
---

# What is an AI agent? The loop in code, and where it fails

What is an AI agent? A tested agent loop in TypeScript, run offline against a scripted model, with traces of runaway loops, tool errors, and token growth.

> **TL;DR**
>
> - What is an AI agent? It is a program that calls a language model in a loop, executes the tool calls it returns, and stops on a defined condition.
> - The loop on this page is 55 lines of TypeScript with no dependencies, tested on Node.js 26.12.1. It ran offline against a scripted fake model, so no hosted model was called.
> - A working loop needs four stops: no tool calls, an input-token budget, repeated-call detection, and a turn limit. The traces in this article show what each one catches.
> - Input cost grows faster than turn count because every call resends the history. Eight scripted turns used 15,382 estimated input tokens, and capping tool output at 200 characters cut that to 2,726.
> - Use a workflow when you can write the steps down, and an agent only when the next step depends on what a tool returned.

An AI agent is a program that calls a language model in a loop, executes the tool calls it returns, and stops on a defined condition. Most answers to what is an AI agent end there, with a list of agent types. This page runs the loop as code and breaks it on purpose, using a scripted fake model and no API key.

## What is an AI agent

An AI agent is software in which a language model's output sets the next step. Your code sends a task and tools to the model, runs the tools named in the reply, and repeats until a reply has no tool calls or a limit fires. A chatbot, by contrast, makes one model call per message.

An [agent](https://aifordevs.org/glossary#agent) has three parts: a model, tools, and a loop. You call the model through an API, and the tools and the loop are code you own. [Anthropic's tool-use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works) state the boundary directly: the model runs nothing itself, and it emits a structured request that your application executes.

Vendors draw the same line. [Anthropic](https://www.anthropic.com/research/building-effective-agents) describes agents as models using tools in a loop based on feedback from the environment, and [LangChain](https://docs.langchain.com/oss/python/langgraph/workflows-agents) contrasts them with workflows that follow predetermined code paths. The phrase [agentic AI](https://aifordevs.org/glossary#agentic-ai) is used more loosely, so apply a mechanical test: does code execute tool calls and feed the results back?

Explainers such as [IBM's overview](https://www.ibm.com/think/topics/ai-agents) also sort agents into types like simple reflex, goal-based, and learning. That taxonomy does not change the code you write, so this page skips it.

The [ReAct paper](https://arxiv.org/abs/2210.03629) from 2022 interleaved reasoning traces with actions. Its authors reported absolute success-rate gains of 34 and 10 percentage points on ALFWorld and WebShop over imitation and reinforcement-learning baselines. Current APIs return the action as a structured block, so your code no longer parses it from prose.

## How AI agents work: the agent loop

An [agent loop](https://aifordevs.org/glossary#agent-loop) sends the full message list to the model, checks the reply for tool calls, runs them, appends the results, and calls the model again. It ends when a reply contains no tool calls or a limit your code set fires. Every other feature in an agent framework sits around that cycle.

**Figure 1.** The loop has one normal exit, a reply with no tool calls, and several forced exits. The model API supplies replies, and your code supplies everything that stops the run.

The wire formats differ, and the loop stays the same. The names below come from the [Anthropic docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls) and the [OpenAI function-calling guide](https://developers.openai.com/api/docs/guides/function-calling), as of November 8, 2026:

| Step | Anthropic Messages API | OpenAI Responses API |
| --- | --- | --- |
| Tool call in the reply | `tool_use` block with `stop_reason: "tool_use"` | `function_call` item |
| Your code answers | `tool_result` block | `function_call_output` item |
| Marking a failure | `is_error: true` | No dedicated field, so the text carries it |
| Loop continues while | `stop_reason` is `"tool_use"` | The response contains tool calls |

Frameworks wrap the same cycle. The [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/running_agents/) treats a reply with no tool calls as final output, runs any tool calls it finds, and raises `MaxTurnsExceeded` when a run passes `max_turns`.

## AI agent architecture in 55 lines

The core of an AI agent architecture is one loop, a map of tools, and a set of limits. The TypeScript below has no dependencies and takes the model as a parameter, so the same loop can run against a hosted API or a script. It ran on Node.js 26.12.1, which executes TypeScript files directly.

```ts title="agent.ts"
export type ToolCall = { id: string; name: string; input: Record<string, unknown> };
export type Reply = { text: string; toolCalls: ToolCall[]; inputTokens: number };
export type Message =
  | { role: "user"; content: string }
  | { role: "assistant"; text: string; toolCalls: ToolCall[] }
  | { role: "tool"; toolCallId: string; content: string; isError: boolean };
export type Model = { complete(messages: Message[]): Promise<Reply> };
export type Tool = (input: Record<string, unknown>) => string | Promise<string>;
export type Limits = { maxTurns: number; maxTotalInputTokens: number; maxRepeats: number; maxToolChars: number };
export type Result = { status: "done" | "max_turns" | "budget" | "repeated_call"; answer: string; turns: number; totalInputTokens: number };

async function runTool(tools: Record<string, Tool>, call: ToolCall) {
  const tool = tools[call.name];
  if (!tool) return { content: `Unknown tool ${call.name}. Available: ${Object.keys(tools).join(", ")}.`, isError: true };
  try {
    return { content: await tool(call.input), isError: false };
  } catch (err) {
    return { content: err instanceof Error ? err.message : String(err), isError: true };
  }
}

export async function runAgent(task: string, model: Model, tools: Record<string, Tool>, limits: Limits): Promise<Result> {
  const messages: Message[] = [{ role: "user", content: task }];
  const repeats = new Map<string, number>();
  let totalInputTokens = 0;

  for (let turn = 1; turn <= limits.maxTurns; turn++) {
    const reply = await model.complete(messages);
    totalInputTokens += reply.inputTokens;
    messages.push({ role: "assistant", text: reply.text, toolCalls: reply.toolCalls });
    console.log(`turn ${turn}: input ${reply.inputTokens}, total ${totalInputTokens}`);

    // Stop 1: the model returned no tool calls, so this reply is the answer.
    if (reply.toolCalls.length === 0) return { status: "done", answer: reply.text, turns: turn, totalInputTokens };

    // Stop 2: the run has spent its input-token budget.
    if (totalInputTokens >= limits.maxTotalInputTokens) return { status: "budget", answer: "", turns: turn, totalInputTokens };

    for (const call of reply.toolCalls) {
      const key = `${call.name}:${JSON.stringify(call.input)}`;
      const seen = (repeats.get(key) ?? 0) + 1;
      repeats.set(key, seen);

      // Stop 3: the same call with the same arguments came back too many times.
      if (seen > limits.maxRepeats) return { status: "repeated_call", answer: "", turns: turn, totalInputTokens };

      const result = await runTool(tools, call);
      const content = result.content.slice(0, limits.maxToolChars);
      console.log(`  ${call.name} ${JSON.stringify(call.input)} -> ${result.isError ? "ERROR: " : ""}${content.slice(0, 40).replaceAll("\n", " ")}`);
      messages.push({ role: "tool", toolCallId: call.id, content, isError: result.isError });
    }
  }
  // Stop 4: the turn limit, for everything the other checks missed.
  return { status: "max_turns", answer: "", turns: limits.maxTurns, totalInputTokens };
}
```

The fake model is a stand-in for a hosted API:

```ts title="fake-model.ts"
import type { Message, Model, ToolCall } from "./agent.ts";

type PlannedCall = Omit<ToolCall, "id">;
type Step = string | PlannedCall[];
export const call = (name: string, input: Record<string, unknown>): PlannedCall => ({ name, input });

// Replays a fixed script. It reads the messages only to estimate their size.
// A string is a final answer. An array is a set of tool calls.
export function scriptedModel(script: Step[] | ((turn: number) => Step)): Model {
  let turn = 0;
  return {
    async complete(messages: Message[]) {
      turn += 1;
      const step = typeof script === "function" ? script(turn) : script[turn - 1];
      if (step === undefined) throw new Error(`Script ran out at turn ${turn}`);
      // Rough estimate of 4 characters per token. A real API returns exact counts.
      const inputTokens = Math.ceil(JSON.stringify(messages).length / 4);
      if (typeof step === "string") return { text: step, toolCalls: [], inputTokens };
      return { text: "", toolCalls: step.map((c, i) => ({ id: `call_${turn}_${i}`, ...c })), inputTokens };
    },
  };
}
```

`scriptedModel` replays a fixed list of replies, one per call. It does no inference and reads the messages only to estimate their size at four characters per token, so real token counts would differ. No hosted model was called and no network request was made, so these traces test the loop and its stops and say nothing about any model's quality.

The scenarios live in `run.ts`:

```ts title="run.ts"
import { runAgent, type Limits, type Tool } from "./agent.ts";
import { call, scriptedModel } from "./fake-model.ts";

const files: Record<string, string> = {
  "infra/billing.env": "PORT=8443\nLOG_LEVEL=info\n",
  "infra/auth.env": "PORT=9000\n",
};
const tools: Record<string, Tool> = {
  search_files: ({ query }) => Object.keys(files).filter((p) => p.includes(String(query))).join("\n") || "No matches.",
  read_file: ({ path }) => {
    const text = files[String(path)];
    if (text === undefined) throw new Error(`No file named ${path}. Call search_files to find the path.`);
    return text;
  },
  read_log: ({ page }) => `page ${page}: ` + "x".repeat(2000),
};
const limits: Limits = { maxTurns: 8, maxTotalInputTokens: 20_000, maxRepeats: 2, maxToolChars: 10_000 };

const scenario = process.argv[2];
const task = "Which port does the billing service use?";

if (scenario === "recover") {
  const model = scriptedModel([
    [call("read_file", { path: "billing.env" })],
    [call("search_files", { query: "billing" })],
    [call("read_file", { path: "infra/billing.env" })],
    "The billing service uses port 8443.",
  ]);
  console.log(await runAgent(task, model, tools, limits));
}

if (scenario === "repeat") {
  const model = scriptedModel(() => [call("read_file", { path: "billing.env" })]);
  console.log(await runAgent(task, model, tools, limits));
}

if (scenario === "drift") {
  const model = scriptedModel((turn) => [call("read_log", { page: turn })]);
  console.log(await runAgent(task, model, tools, limits));
}

if (scenario === "budget") {
  const model = scriptedModel((turn) => [call("read_log", { page: turn })]);
  console.log(await runAgent(task, model, tools, { ...limits, maxTurns: 50, maxTotalInputTokens: 8_000 }));
}

if (scenario === "trim") {
  const model = scriptedModel((turn) => [call("read_log", { page: turn })]);
  console.log(await runAgent(task, model, tools, { ...limits, maxToolChars: 200 }));
}
```

Here is the normal path from `node run.ts recover`. The script's first tool call fails on purpose:

```text
turn 1: input 18, total 18
  read_file {"path":"billing.env"} -> ERROR: No file named billing.env. Call search_f
turn 2: input 79, total 97
  search_files {"query":"billing"} -> infra/billing.env
turn 3: input 128, total 225
  read_file {"path":"infra/billing.env"} -> PORT=8443 LOG_LEVEL=info 
turn 4: input 182, total 407
{
  status: 'done',
  answer: 'The billing service uses port 8443.',
  turns: 4,
  totalInputTokens: 407
}
```

## Workflows, agents, and multi-agent systems

A workflow runs steps your code fixed in advance, an agent lets model output set each next step, and a multi-agent system has one agent hand work to others. Choose by how much of the path you can write down, because each move along that list adds cost and new ways to fail.

| | Workflow | Agent | Multi-agent system |
| --- | --- | --- | --- |
| Next step set by | Your code | Model output, inside your limits | A lead agent's output, then each worker's |
| Number of steps | Known in advance | Unknown until the run ends | Unknown, often in parallel |
| Main control | Branches and retries | Stop conditions and tool design | Delegation rules and shared state |
| Typical failure | A branch you did not write | Loops and context growth | Over-delegation and duplicated work |
| Tokens versus a chat, per Anthropic | Not reported | About 4 times | About 15 times |
| Choose it when | You can list the steps | The next step depends on a tool result | Subtasks are independent and can run at once |

Anthropic's [guidance on building agents](https://www.anthropic.com/research/building-effective-agents) recommends the simplest solution that works, and it says agentic systems often trade latency and cost for task performance. Take that as the default. If your feature has a known path, a workflow is cheaper to run and easier to [test](https://aifordevs.org/articles/llm-evaluation).

The token multiples come from [Anthropic's June 2025 account of its research system](https://www.anthropic.com/research/multi-agent-research-system), which uses a lead agent that delegates to subagents working in parallel. They are one vendor's measurements of its own product, and no independent measurement exists. The same post reports early versions spawning 50 subagents for simple queries.

## Unbounded loops need two kinds of stop

An unbounded loop happens when the model keeps returning tool calls and nothing in your code ends the run. A repeat detector catches the same call sent again with the same arguments, and a turn limit catches everything else. You need both, because a run that changes one argument each turn never trips the repeat check.

The `repeat` scenario returns the same failing call forever. The loop stops on turn 3, before it runs the tool a third time:

```text
turn 1: input 18, total 18
  read_file {"path":"billing.env"} -> ERROR: No file named billing.env. Call search_f
turn 2: input 79, total 97
  read_file {"path":"billing.env"} -> ERROR: No file named billing.env. Call search_f
turn 3: input 139, total 236
{
  status: 'repeated_call',
  answer: '',
  turns: 3,
  totalInputTokens: 236
}
```

The `drift` scenario asks for page 1, page 2, page 3, and so on. Every call is new, so the repeat check stays silent, and only the turn limit ends the run. This is `node run.ts drift | grep -v read_log`:

```text
turn 1: input 18, total 18
turn 2: input 562, total 580
turn 3: input 1106, total 1686
turn 4: input 1651, total 3337
turn 5: input 2195, total 5532
turn 6: input 2739, total 8271
turn 7: input 3283, total 11554
turn 8: input 3828, total 15382
{ status: 'max_turns', answer: '', turns: 8, totalInputTokens: 15382 }
```

Anthropic [reports the same failure mode](https://www.anthropic.com/research/multi-agent-research-system) in its own early agents, which searched endlessly for sources that did not exist. Set the turn limit from observed runs of the task, and treat `max_turns` as an error to log. A run that hits it has no answer, and the caller needs to know that.

## Tool errors are input for the next turn

Send a tool failure back to the model as a tool result. An uncaught exception ends the run, and a readable message gives the next model call something to act on. In the `recover` trace above, the first read fails, the error names `search_files`, and the run finishes on turn 4.

`runTool` catches every exception and returns it with `isError: true`.

Anthropic's docs [advise instructive error text](https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls) that says what went wrong and what to try next. They also say a call with invalid parameters typically gets 2 to 3 corrected attempts. The [MCP tools spec](https://modelcontextprotocol.io/specification/2026-07-28/server/tools) has a similar split: `isError` results are meant to reach the model, and clients may or may not forward protocol errors.

Three tool-level problems need code that the sample leaves out:

- Timeouts: wrap each call with `AbortSignal.timeout`, since one hung tool blocks the whole loop.
- Repeated side effects: a retried write can run twice, so use idempotency keys and put [a confirmation step](https://aifordevs.org/articles/build-an-mcp-server) on anything that changes data.
- Injected instructions: tool results can carry indirect prompt injection, as [Anthropic's docs warn](https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls), so treat them as untrusted data and enforce [guardrails](https://aifordevs.org/glossary#guardrails) in code.

## Context growth makes every turn larger

Context grows because each model call resends the whole message list, including every earlier tool result. In the scripted run, each 2,000-character tool result added about 544 estimated tokens, and input climbed from 18 tokens on turn 1 to 3,828 on turn 8. The 200-character cap kept turn 8 at 664.

**Figure 2.** Each model call resends everything before it, so one tool result keeps costing tokens on every later turn. Capping tool output at 200 characters flattens the slope.

The `budget` scenario caps total input at 8,000 tokens. The check runs after each model call, so the run overshoots by one turn and stops at 8,271. This is `node run.ts budget | grep -v read_log`, trimmed to its last lines:

```text
turn 5: input 2195, total 5532
turn 6: input 2739, total 8271
{ status: 'budget', answer: '', turns: 6, totalInputTokens: 8271 }
```

A hosted model has a hard ceiling too. Anthropic's API returns [`model_context_window_exceeded`](https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons) when a response fills the [context window](https://aifordevs.org/glossary#context-window). Anthropic's [context engineering guide](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents) names compaction and tool result clearing as the standard responses, and [context engineering](https://aifordevs.org/articles/context-engineering) covers the budget in detail.

Truncation has a price. A 200-character cap keeps the loop cheap and can cut the one line the next step needed. Pair a cap with pagination, or with a tool that fetches more on request.

## What the loop costs

Agent cost follows total input tokens, and total input grows roughly with the square of the turn count. Doubling the turns in the table below roughly quadruples the input tokens, so a turn limit is also a spending limit. Output tokens and any per-call tool fees come on top of these numbers.

| Turns | Total input tokens | Last call's input | Uncached | With prompt caching |
| --- | --- | --- | --- | --- |
| 5 | 5,532 | 2,195 | $0.0111 | $0.0058 |
| 10 | 24,670 | 4,916 | $0.0493 | $0.0143 |
| 25 | 163,842 | 13,095 | $0.3277 | $0.0478 |
| 50 | 668,420 | 26,726 | $1.3368 | $0.1310 |
| 100 | 2,699,920 | 53,989 | $5.3998 | $0.3996 |

The token columns come from the same loop and fake model, with every tool returning about 2,000 characters. The dollars use [Claude Sonnet 5.5 rates](https://platform.claude.com/docs/en/about-claude/pricing) as of November 8, 2026: $2 per million input tokens, $2.50 for 5-minute cache writes, and $0.10 for cache hits. They are arithmetic on estimated tokens, and no API was billed.

The caching column assumes every call lands inside the cache window and hits the full prefix from the call before it. Under that assumption, [prompt caching](https://aifordevs.org/glossary#prompt-caching) makes the 100-turn run 13.5 times cheaper, and the cost still rises faster than the turn count. See [Claude API pricing](https://aifordevs.org/articles/claude-api-pricing) and [Gemini API pricing](https://aifordevs.org/articles/gemini-api-pricing) for current rates, and rerun the arithmetic with your own tool output sizes.

## Start with a workflow, then earn the loop

Build an agent only after you can name the step a fixed workflow cannot handle. If you can list the steps, write a workflow. If the next step depends on what a tool returned, that test is the useful answer to what is an AI agent in your own codebase.

Take these steps in order:

1. Copy the loop above and write all four stops before you tune a single prompt.
2. Log turns and input tokens per run, then set limits from the distribution of real tasks.
3. Add tools through a [tested MCP server](https://aifordevs.org/articles/build-an-mcp-server) when more than one host needs them, and browse [MCP guides](https://aifordevs.org/topics/mcp) for the rest of the protocol.
4. Score finished runs with [LLM evaluation](https://aifordevs.org/articles/llm-evaluation) and an [LLM judge](https://aifordevs.org/articles/llm-as-a-judge), since a completed loop says nothing about a correct answer.
5. Compare models for your task with [the best LLM for coding](https://aifordevs.org/articles/best-llm-for-coding) and the [platforms directory](https://aifordevs.org/platforms), then read more in the [coding agents](https://aifordevs.org/topics/coding-agents) and [agents](https://aifordevs.org/topics/agents) hubs.

## Questions

### What is an AI agent in simple terms?

An AI agent is software that repeatedly sends a task to a language model, runs the tools named in the reply, and feeds the results back until the job ends or a limit fires. A chatbot makes one model call per message. An agent can make many model calls for one request.

### What is the difference between an AI agent and a chatbot?

A chatbot generates a reply to each message and stops. An AI agent runs a loop: model output names a tool, your code runs it, and the result goes back for another model call. The difference is that code executes actions and returns their results before anything is shown to the user as finished.

### Is ChatGPT an AI agent?

Apply a mechanical test instead of relying on the product name. A session that only generates text makes one model call per message, so it does not meet the definition used here. A session that runs tools in a loop and feeds the results back to the model does meet it. The same test works for any product.

### What are the main components of an AI agent?

An AI agent has a model that returns text or tool calls, a set of tools with schemas, and a loop that runs those tools and resends the results. Production agents add stop conditions, a token budget, logging, and permission checks. The loop and its limits are code you write, and the model API does not supply them.

### What is the difference between an AI agent and a workflow?

In a workflow, your code fixes the order of steps in advance, and a model may fill in some of them. In an AI agent, the model's output sets each next step at run time, inside limits your code enforces. Choose a workflow when you can write the steps down, because it is cheaper to run and easier to test.

### How do you stop an AI agent from looping forever?

Give the loop several independent stops. End when a reply has no tool calls, cap the turn count, cap total input tokens, and stop when the same tool call repeats with the same arguments. Repeat detection alone is not enough, because a run that changes one argument each turn never trips it.

### How does MCP relate to AI agents?

MCP is a standard way to connect an agent's host application to tools and data. The agent loop still runs in the host: model output names a tool, and the host sends that call to an MCP server. You can build an agent with no MCP at all.

## Sources

1. [Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models (2022)](https://arxiv.org/abs/2210.03629)
2. [Anthropic, Building effective agents (December 2024)](https://www.anthropic.com/research/building-effective-agents)
3. [IBM, What are AI agents?](https://www.ibm.com/think/topics/ai-agents)
4. [Anthropic, How we built our multi-agent research system (June 2025)](https://www.anthropic.com/research/multi-agent-research-system)
5. [Anthropic, Effective context engineering for AI agents](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents)
6. [Anthropic docs, how tool use works](https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works)
7. [Anthropic docs, handle tool calls](https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls)
8. [Anthropic docs, handling stop reasons](https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons)
9. [Anthropic docs, pricing](https://platform.claude.com/docs/en/about-claude/pricing)
10. [OpenAI docs, function calling](https://developers.openai.com/api/docs/guides/function-calling)
11. [OpenAI Agents SDK, running agents](https://openai.github.io/openai-agents-python/running_agents/)
12. [LangChain docs, workflows and agents](https://docs.langchain.com/oss/python/langgraph/workflows-agents)
13. [Model Context Protocol, tools, revision 2026-07-28](https://modelcontextprotocol.io/specification/2026-07-28/server/tools)

## Cite this deep dive

AI for Devs. "What is an AI agent? The loop in code, and where it fails." Published November 8, 2026. https://aifordevs.org/articles/what-is-an-ai-agent

- Page: https://aifordevs.org/articles/what-is-an-ai-agent
- Markdown: https://aifordevs.org/articles/what-is-an-ai-agent.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08

---
title: "Context engineering: budget and order the context window"
type: "guide"
url: "https://aifordevs.org/articles/context-engineering"
markdown: "https://aifordevs.org/articles/context-engineering.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/context-engineering"
description: "Context engineering is choosing what a model reads on each call. Size each section, order it for prompt caching, trim inputs, and test every layout change."
author: "AI for Devs"
published: "2026-11-06"
updated: "2026-11-08"
level: "Intermediate"
topics: ["context", "rag", "agents"]
---

# Context engineering: budget and order the context window

Context engineering is choosing what a model reads on each call. Size each section, order it for prompt caching, trim inputs, and test every layout change.

> **TL;DR**
>
> - Context engineering is the practice of choosing which tokens reach a model on each call: instructions, tool definitions, retrieved documents, tool results, and conversation history.
> - Size the context window by section. Every current Claude model listed a 1M-token window as of November 8, 2026, and Anthropic's docs still warn that accuracy and recall degrade as token count grows.
> - Put stable content first so prompt caching can reuse it. Minimum cacheable sizes ranged from 512 to 4,096 tokens on Claude models as of November 8, 2026, and Anthropic and OpenAI require an exact prefix match.
> - Trim retrieved documents, history, and tool results before they reach the window, and rerun your evals after any layout change, because reordering can change answers as well as cost.
> - The TypeScript budget helper in this guide ran on Node.js 26.12.1 with `js-tiktoken` 1.0.21 on November 8, 2026. It used synthetic text and called no provider API.

Context engineering is the practice of deciding which instructions, documents, tool results, and history reach a model on each call, and how much room each gets. A coding agent reading a repository or a support bot pulling runbooks rebuilds that block of tokens for every request. Failures show up as truncated answers, a cache that never hits, or an agent that loses a decision made 30 turns ago. This guide turns the window into something you can size, order, log, and test.

## What is context engineering

Context engineering is the work of assembling everything a model reads for one request, then keeping that block small enough to pay for and clear enough to use. Anthropic describes it as strategies for "curating and maintaining the optimal set of tokens" during inference. In practice it is code that runs before every API call.

[Anthropic's context window docs](https://platform.claude.com/docs/en/build-with-claude/context-windows) count everything in the request toward the limit: the system prompt, every message, tool results, images, documents, and tool definitions. The output counts too. That is the full list of parts you are engineering, and each one changes at a different rate.

| Section | How often it changes | What goes wrong without a cap |
| --- | --- | --- |
| Tool definitions | Per release, or per user if you filter tools | Thousands of tokens spent before the first turn |
| System prompt and rules | Per release | Rules pile up and contradict each other |
| Shared documents | Per day or week | Stale facts get cached and reused |
| Conversation history | Per turn | The window fills and early decisions disappear |
| Retrieved documents | Per request | Noise crowds out the passage that answers the question |
| Tool results | Per step | One raw API response takes the whole budget |

The term spread in mid-2025. [Simon Willison's June 27, 2025 note](https://simonwillison.net/2025/Jun/27/context-engineering/) credits Shopify's Tobi Lutke with an early endorsement and Andrej Karpathy with amplifying it. If you want the working definitions behind the label, see the glossary entries for [context engineering](https://aifordevs.org/glossary#context-engineering) and the [context window](https://aifordevs.org/glossary#context-window).

## Context engineering vs prompt engineering

Prompt engineering is the wording of instructions. Context engineering is the selection and layout of everything around those instructions, including the instructions themselves. Prompt engineering stays part of the job, and much of this work was practiced as retrieval, summarization, and tool design before anyone used the term.

The new name earned its place because of agent loops. A prompt is written once and reviewed by a person. In an [agent loop](https://aifordevs.org/glossary#agent-loop), code rebuilds the input on every step from tool results and history that no person ever reads.

| | Prompt engineering | Context engineering |
| --- | --- | --- |
| Unit of work | One [system prompt](https://aifordevs.org/glossary#system-prompt) or template | Every token in the request |
| Main question | How should this instruction read? | What should be in the window on this call? |
| Lives in | A prompt file | Assembly code, retrieval, tools, memory |
| Typical failure | Instruction misread or ignored | Right answer absent, buried, or evicted |
| How you test it | Eval on wording changes | Eval on layout, retrieval, and history changes |

For a single-turn feature with a fixed input, prompt engineering is the whole job and the rest of this guide is overhead. Once you add [retrieval](https://aifordevs.org/topics/rag), [tool calling](https://aifordevs.org/glossary#tool-calling), or multi-turn state, you are already making these decisions, whether you name them or not.

Nothing about the term gives a model new abilities. It names choices that used to stay implicit.

## Budget the context window by section

Give every section of the context window a token cap before you write the code that fills it. Retrieval that can return 80K tokens and history with no limit will eventually collide. The first symptom is usually a failed request or an answer that stops early.

**Figure 1.** An example allocation for one call. The numbers are a starting point for a 128K-token window, and each section has a size someone chose on purpose.

On Claude, overflow has two forms, per [Anthropic's context window docs](https://platform.claude.com/docs/en/build-with-claude/context-windows) as of November 8, 2026. If the input alone exceeds the window, the API returns a 400 error.

On Claude 4.5 models and newer, input plus `max_tokens` can exceed the window, and generation stops with `stop_reason: "model_context_window_exceeded"` if it reaches the limit. Treat both as bugs in your budget.

Large windows do not remove the budget. Every current Claude model listed a [1M-token window](https://platform.claude.com/docs/en/models/overview) on November 8, 2026. The same docs say accuracy and recall degrade as token count grows, a problem Anthropic calls context rot, and cached prefixes still count toward the window.

Recount when you change models. [Anthropic's token counting docs](https://platform.claude.com/docs/en/build-with-claude/token-counting) say Claude Opus 4.7 and later models use a tokenizer that produces about 30 percent more tokens for the same text than earlier models. A budget measured on the old [tokenizer](https://aifordevs.org/glossary#tokenizer) will overflow on the new one.

## Order the window for prompt caching

Prompt caching reuses a prefix of your input that matches an earlier request exactly, so the order of your sections decides whether you get the discount. Put the parts that never change first and the parts that change per request last. One changed token near the start forfeits everything after it.

**Figure 2.** With stable sections first, only the tail is reprocessed. A timestamp at the start changes the prefix, so nothing after it can match.

Use this order:

1. Tool definitions
2. System prompt and rules
3. Long shared documents, such as a policy manual
4. Conversation history, oldest first
5. Retrieved documents for this turn
6. The current request

An earlier version of this article put the system prompt before tools. Anthropic's API renders `tools`, then `system`, then `messages`, so tools come first there. Provider behavior also moved since the article first ran, and this table reflects each provider's docs on November 8, 2026.

| Provider | How it turns on | Minimum size | Lifetime | Where hits are reported |
| --- | --- | --- | --- | --- |
| [Anthropic](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) | Top-level `cache_control` for automatic breakpoints, or up to 4 explicit breakpoints | 512 to 4,096 tokens, by model | 5 minutes by default, 1 hour optional | `cache_read_input_tokens` |
| [OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching) | On by default; GPT-5.6 and later also accept explicit breakpoints | 1,024 visible tokens on GPT-5.6 and later | At least 30 minutes on GPT-5.6 and later | `usage.input_tokens_details.cached_tokens` |
| [Google Gemini](https://aistudio.google.com/docs/caching) | Implicit caching on by default for Gemini 2.5 and newer | 2,048 on Gemini 2.5, 4,096 on Gemini 3.x and Gemini 4 Argon | Not stated on the caching page | `usage.total_cached_tokens` |

Several ordinary changes break a cache without any error. Anthropic documents that editing tool definitions invalidates the tools, system, and messages caches. OpenAI lists `tools`, `reasoning.effort`, and `text.format` among settings that can change the prefix.

A cache entry also becomes available only after the first response starts, per Anthropic's docs, so parallel requests sent together miss it.

Long conversations have a second trap on Claude. Anthropic's lookup checks at most 20 positions back from each breakpoint, so a turn that adds many blocks can walk past the last cached entry. Add a breakpoint inside long turns.

Caching pays back fast. Claude Opus 5.5 listed $4 per million input tokens on November 8, 2026, with cache writes at 1.25 times that and reads at 0.05 times. A 20K-token prefix sent 10 times inside the 5-minute window costs $0.80 uncached and $0.136 cached, 83 percent less.

The cache pays for itself on the second call, or on the third with the 1-hour write at 2 times the input rate. See [Claude API pricing](https://aifordevs.org/articles/claude-api-pricing) and [Gemini API pricing](https://aifordevs.org/articles/gemini-api-pricing) for current rates, and the [KV cache article](https://aifordevs.org/articles/kv-cache-explained) for what the server stores.

## Retrieve less, and label it

Retrieve a small, reranked set of chunks and label each one with its source, because more retrieved text adds noise faster than it adds answers. [Liu et al.](https://arxiv.org/abs/2307.03172) found that moving from 20 to 50 retrieved documents added about 1.5 points of accuracy for GPT-3.5-Turbo and about 1 point for Claude-1.3.

The same paper is the source of the "lost in the middle" result, and it deserves a careful reading. On multi-document QA with 20 documents, GPT-3.5-Turbo scored 75.8 percent with the answer first, 53.8 percent in the middle, and 63.2 percent last. Its closed-book score was 56.1 percent, so a middle placement did worse than giving the model no documents at all.

The effect has limits. Claude-1.3 scored nearly perfectly on the paper's key-value retrieval task at every length tested, and every model tested dates from 2023. Later work narrows the claim:

- [Veseli et al. (2025)](https://arxiv.org/abs/2508.07479) report the middle penalty is strongest when inputs fill up to 50 percent of the window, and shifts toward favoring text near the end as inputs get longer.
- [Gabín et al. (2026)](https://arxiv.org/abs/2605.27105) report that order and context size interact strongly with retrieval quality and model choice, so results from idealized setups do not always transfer.

The practical rule is to rerank, then put the strongest evidence near the start or end, and measure the effect on your own model. Keep only chunks above a score threshold, and wrap each with an ID so the model can cite it and you can check the citation:

```xml
<document id="doc-3" source="runbooks/payments.md" updated="2026-10-14">
Refunds over $500 need a second approver in the finance channel.
</document>
```

[Reranking](https://aifordevs.org/glossary#reranking) and [chunking](https://aifordevs.org/glossary#chunking) decide what makes this list. The [vector database guide](https://aifordevs.org/articles/what-is-a-vector-database) covers retrieval, and [RAG vs fine-tuning](https://aifordevs.org/articles/rag-vs-fine-tuning) covers when retrieval is the wrong fix.

## Compress history instead of dropping it

Keep the last few turns verbatim and replace older turns with a summary that preserves decisions, identifiers, file paths, and open questions. Anthropic and OpenAI both offered server-side compaction as of November 8, 2026, so you may not need your own summarizer, but you still need to test what each summary loses.

[Anthropic's compaction](https://platform.claude.com/docs/en/build-with-claude/compaction) was in beta on that date, in two forms: on demand, or triggered by a token threshold you set. [OpenAI's compaction](https://developers.openai.com/api/docs/guides/compaction) uses `context_management` with a `compact_threshold`, and the returned item is opaque.

If you write your own summarizer, follow [Anthropic's advice](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents) to tune for recall first, then cut what you do not need. A summary saying only "the user asked about refunds" keeps none of the parts a later turn needs.

For agents, clear old tool results once their conclusions are recorded. [Anthropic's context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing), also in beta on November 8, 2026, clears old tool results past a trigger and defaults to keeping the last 3 tool uses. It invalidates cached prefixes when it fires, so set `clear_at_least` high enough that each clearing is worth the cache write.

## Trim tool results at the source

A tool that returns a full API response fills the window with fields the next step never reads. Return only the fields the next step needs, cap list lengths, and paginate. Anthropic reported in September 2025 that Claude Code restricts tool responses to 25,000 tokens by default.

[Anthropic's tool-design post](https://www.anthropic.com/research/writing-tools-for-agents) recommends pagination, filtering, range selection, and truncation with sensible defaults. In its Slack example, a concise response format used about a third of the tokens of a detailed one. Error results should state what to change, because the next call depends on what the message says.

You control this most directly when you write the tool yourself. The [MCP server guide](https://aifordevs.org/articles/build-an-mcp-server) shows a small server with narrow results, and [what is an AI agent](https://aifordevs.org/articles/what-is-an-ai-agent) covers the loop that consumes them.

## Measure every section

Log the token count of each section on every call, next to latency and cost. Provider usage fields report totals and cache splits, so the per-section numbers have to come from your own code. The helper below checks caps, computes the cacheable prefix, and fails loudly when a section is over budget.

```ts title="budget.ts"
export type Section = { name: string; text: string; max: number; stable?: boolean };

type Options = {
  windowTokens: number;
  reserveForOutput: number;
  count?: (text: string) => number;
};

// Rough default: about 4 characters per token for English prose.
export const estimate = (text: string) => Math.ceil(text.length / 4);

export function assemble(sections: Section[], options: Options) {
  const { windowTokens, reserveForOutput, count = estimate } = options;
  const report = sections.map((s) => ({
    name: s.name,
    tokens: count(s.text),
    max: s.max,
    stable: Boolean(s.stable),
  }));

  const over = report.filter((r) => r.tokens > r.max).map((r) => r.name);
  const total = report.reduce((sum, r) => sum + r.tokens, 0);
  const headroom = windowTokens - reserveForOutput - total;
  if (over.length || headroom < 0) {
    throw new Error(`Context over budget: ${JSON.stringify({ total, headroom, over })}`);
  }

  // Cacheable prefix: tokens before the first section that changes per request.
  const firstVolatile = report.findIndex((r) => !r.stable);
  const cut = firstVolatile === -1 ? report.length : firstVolatile;
  const stablePrefix = report.slice(0, cut).reduce((sum, r) => sum + r.tokens, 0);

  return { text: sections.map((s) => s.text).join("\n\n"), report, total, stablePrefix, headroom };
}
```

A driver script fills the sections with synthetic support-assistant text, prints the report, cross-checks the estimate against OpenAI's `o200k_base` encoding, and then tries a section over its cap.

```ts title="demo.ts"
import { assemble, estimate, type Section } from "./budget.ts";
import { getEncoding } from "js-tiktoken";

// Synthetic content, shaped like a support assistant's context.
const rules = "You answer billing questions for the support team. Cite a document id for every claim. ".repeat(40);
const tools = JSON.stringify(
  Array.from({ length: 12 }, (_, i) => ({
    name: `lookup_${i}`,
    description: "Look up a billing record by id and return its status.",
    input_schema: { type: "object", properties: { id: { type: "string" } }, required: ["id"] },
  })),
);
const docs = Array.from(
  { length: 8 },
  (_, i) => `<document id="doc-${i}" source="runbooks/payments.md">\nRefunds over $500 need a second approver in the finance channel.\n</document>`,
).join("\n").repeat(12);
const history = "user: Why was my card charged twice?\nassistant: Two authorizations posted; one will drop off in 3 days.\n".repeat(90);
const task = "Which approval does refund R-1042 need, and who can give it?";

const sections: Section[] = [
  { name: "tools", text: tools, max: 7_000, stable: true },
  { name: "rules", text: rules, max: 3_000, stable: true },
  { name: "history", text: history, max: 28_000 },
  { name: "documents", text: docs, max: 36_000 },
  { name: "task", text: task, max: 6_000 },
];

const budget = { windowTokens: 128_000, reserveForOutput: 16_000 };
const out = assemble(sections, budget);
console.table(out.report);
console.log({ total: out.total, stablePrefix: out.stablePrefix, headroom: out.headroom });

// Cross-check the 4-characters rule against a real tokenizer (OpenAI's o200k_base).
const enc = getEncoding("o200k_base");
console.table(
  sections.map((s) => {
    const real = enc.encode(s.text).length;
    const est = estimate(s.text);
    return { name: s.name, chars: s.text.length, estimate: est, o200k: real, "est/real": Number((est / real).toFixed(2)) };
  }),
);

// A section over its cap fails the build instead of truncating in production.
try {
  assemble([...sections, { name: "ticket_dump", text: "x".repeat(40_000), max: 1_000 }], budget);
} catch (e) {
  console.log(String(e));
}
```

Set `"type": "module"` in `package.json`, install `js-tiktoken`, and run `node demo.ts`. Node.js 26.12.1 strips the types natively, and printed:

```text
┌─────────┬─────────────┬────────┬───────┬────────┐
│ (index) │ name        │ tokens │ max   │ stable │
├─────────┼─────────────┼────────┼───────┼────────┤
│ 0       │ 'tools'     │ 538    │ 7000  │ true   │
│ 1       │ 'rules'     │ 870    │ 3000  │ true   │
│ 2       │ 'history'   │ 2340   │ 28000 │ false  │
│ 3       │ 'documents' │ 3093   │ 36000 │ false  │
│ 4       │ 'task'      │ 15     │ 6000  │ false  │
└─────────┴─────────────┴────────┴───────┴────────┘
{ total: 6856, stablePrefix: 1408, headroom: 105144 }
┌─────────┬─────────────┬───────┬──────────┬───────┬──────────┐
│ (index) │ name        │ chars │ estimate │ o200k │ est/real │
├─────────┼─────────────┼───────┼──────────┼───────┼──────────┤
│ 0       │ 'tools'     │ 2151  │ 538      │ 506   │ 1.06     │
│ 1       │ 'rules'     │ 3480  │ 870      │ 681   │ 1.28     │
│ 2       │ 'history'   │ 9360  │ 2340     │ 2250  │ 1.04     │
│ 3       │ 'documents' │ 12372 │ 3093     │ 3253  │ 0.95     │
│ 4       │ 'task'      │ 60    │ 15       │ 16    │ 0.94     │
└─────────┴─────────────┴───────┴──────────┴───────┴──────────┘
Error: Context over budget: {"total":16856,"headroom":95144,"over":["ticket_dump"]}
```

The 4-character rule was 28 percent high on the repetitive rules text and 5 percent low on the documents. Use it for rough budgets and use a real count for anything you enforce. [Anthropic's token counting endpoint](https://platform.claude.com/docs/en/build-with-claude/token-counting) was free and rate limited on November 8, 2026, and its docs call the result an estimate.

This run exercised the helper, its over-budget error, and the `js-tiktoken` comparison. It did not exercise any provider API, the token counting endpoint, or a real prompt cache. The `o200k_base` encoding is OpenAI's, so its counts say nothing exact about Claude.

The 1,408-token stable prefix would clear the 512-token minimum on Opus 5.5 and miss the 4,096-token minimum on Claude Haiku 4.5.

## Test context engineering changes like code

Start context engineering work with one logged request. Dump each section's token count, mark it stable or volatile, and move the first volatile section after the last stable one. Then rerun your evals, because a reordering can change answers as well as cost.

Use these rules to pick the next change:

- If cached reads are under half of input tokens on repeat traffic, fix section order before you touch anything else.
- If one section is more than half of the window on most calls, cap that section first.
- If answers get worse after a retrieval or history change, compare against the previous layout on the same test cases.

Moving instructions, adding a tool, and raising top-k are code changes, and they regress the same way. The [LLM evaluation guide](https://aifordevs.org/articles/llm-evaluation) shows how to compare two layouts on the same test cases and block the worse one in CI.

If your scoring depends on a model grading open-ended answers, read [LLM as a judge](https://aifordevs.org/articles/llm-as-a-judge) before you trust the number. More on this topic is in the [prompting and context hub](https://aifordevs.org/topics/context).

## Questions

### What is context engineering?

Context engineering is the practice of deciding which tokens reach a model on each call and how much room each kind of content gets. It covers system prompts, tool definitions, retrieved documents, tool results, and conversation history. Most of the work is code that assembles those parts before each API request, and it applies most to agents and long conversations.

### Is context engineering different from prompt engineering?

Prompt engineering covers how instructions are worded. Context engineering includes those instructions and adds what else the model reads on a call: retrieved text, tool results, tool definitions, and history. For a single-turn prompt with a fixed input, prompt engineering is the whole job. For an agent that rebuilds its input on every step, context engineering is where most failures start.

### If a model has a 1M-token window, do I still need to trim?

Yes. Every token costs money and adds latency, and Anthropic's context window docs warn that accuracy and recall degrade as token count grows, as of November 8, 2026. A large window raises the ceiling on what fits. It does not change the price of filling it or the need to choose what goes in.

### How does prompt caching change the way you order context?

Prompt caching reuses a matching prefix of your input, so the order of context sections decides whether you get the discount. Anthropic and OpenAI both require an exact prefix match, as of November 8, 2026. Put tool definitions, system rules, and shared documents first, and put retrieved documents and the current request last.

### How do I count tokens for each context section?

Run a tokenizer or your provider's counting endpoint on each section separately and log the results next to latency and cost. Anthropic's token counting endpoint was free and rate limited as of November 8, 2026, and its docs describe the count as an estimate. A four-characters-per-token rule works for rough budgets and can miss by more than a quarter on some content.

### What are the main context engineering techniques?

The main techniques are sizing each section with a token cap, ordering stable content first for prompt caching, retrieving a small reranked set of documents, summarizing or compacting old turns, and trimming tool results at the source. Anthropic also describes keeping notes outside the window and delegating work to sub-agents that return short summaries for long-running tasks.

## Sources

1. [Anthropic, Effective context engineering for AI agents (2025)](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents)
2. [Anthropic docs: Context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows)
3. [Anthropic docs: Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
4. [Anthropic docs: Token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting)
5. [Anthropic docs: Models overview and pricing](https://platform.claude.com/docs/en/models/overview)
6. [Anthropic docs: Compaction overview](https://platform.claude.com/docs/en/build-with-claude/compaction)
7. [Anthropic docs: Context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing)
8. [Anthropic, Writing tools for agents (2025)](https://www.anthropic.com/research/writing-tools-for-agents)
9. [OpenAI docs: Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)
10. [OpenAI docs: Compaction](https://developers.openai.com/api/docs/guides/compaction)
11. [Google docs: Context caching](https://aistudio.google.com/docs/caching)
12. [Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)](https://arxiv.org/abs/2307.03172)
13. [Veseli et al., Positional Biases Shift as Inputs Approach Context Window Limits (2025)](https://arxiv.org/abs/2508.07479)
14. [Gabín et al., Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG (2026)](https://arxiv.org/abs/2605.27105)
15. [Simon Willison, Context engineering (June 27, 2025)](https://simonwillison.net/2025/Jun/27/context-engineering/)

## Cite this guide

AI for Devs. "Context engineering: budget and order the context window." Published November 6, 2026, updated November 8, 2026. https://aifordevs.org/articles/context-engineering

- Page: https://aifordevs.org/articles/context-engineering
- Markdown: https://aifordevs.org/articles/context-engineering.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-06
- Updated: 2026-11-08

---
title: "KV cache explained: what it stores and what it costs"
type: "deep dive"
url: "https://aifordevs.org/articles/kv-cache-explained"
markdown: "https://aifordevs.org/articles/kv-cache-explained.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/kv-cache-explained"
description: "A KV cache holds attention keys and values during LLM decode. Size one from config.json, see what concurrency costs, and how prompt caching reuses it."
author: "AI for Devs"
published: "2026-11-06"
updated: "2026-11-08"
level: "Intermediate"
topics: ["inference", "apis"]
---

# KV cache explained: what it stores and what it costs

A KV cache holds attention keys and values during LLM decode. Size one from config.json, see what concurrency costs, and how prompt caching reuses it.

> **TL;DR**
>
> - A KV cache holds the attention keys and values of every token an LLM has already processed, so each new token computes only its own and outputs stay the same.
> - KV cache size follows one formula, which gives 320 KiB per token for Llama 3.1 70B at 16 bits, recomputed here from Table 3 of the Llama 3 paper.
> - After the weights load, one 141 GB H200 holds about 105 Llama 3.1 8B sequences at 8K tokens and about 6 at 128K tokens.
> - Provider prompt caching retains this same data across requests. OpenAI's guide says its prompt cache stores KV tensors, and reads cost 0.1x the input rate on GPT-5.6 and later, as of November 8, 2026.
> - The sizing script ran on Python 3.15.1 against three public config files. No GPU workload ran, so every concurrency figure in this article is arithmetic.

A KV cache is the block of GPU memory that holds the attention keys and values a transformer computed for every earlier token in a sequence. An LLM reads it on every generated token, because each new token attends to all of those earlier positions. Without it, token 1,001 would repeat the key and value work for the previous 1,000 tokens in every layer. The cache removes that repeated math and charges you in memory, and that memory is what caps how many users fit on a GPU.

## What is a KV cache

A KV cache stores a key tensor and a value tensor per KV head, per layer, for every token, so later tokens read them instead of recomputing them. Keys and values for earlier positions do not change when a new token arrives, so caching them changes speed and memory without changing the math.

[Attention](https://aifordevs.org/glossary#attention) projects each token into a query, a key, and a value, the scheme [Vaswani et al.](https://arxiv.org/abs/1706.03762) introduced in 2017. To produce the next token, each layer compares the newest query against the keys of all earlier tokens and mixes their values.

Only the newest token needs a new query. Everything else a [transformer](https://aifordevs.org/glossary#transformer) layer needs from earlier positions is already sitting in the cache.

Scope is where people mix up the terms. The cache lives inside one generation. [Prompt caching](https://aifordevs.org/glossary#prompt-caching) keeps part of it alive across separate requests, and [the section below](#how-the-kv-cache-relates-to-prompt-caching) ties the two together.

## How the KV cache works in prefill and decode

The cache is filled during prefill and read during decode. Prefill computes keys and values for the whole prompt in one parallel pass and writes them to the cache. Decode then generates one token per step, reading every cached entry and appending one new entry per layer until the response ends.

**Figure 1.** Prefill writes keys and values for the whole prompt in one parallel pass. Each decode step computes one token, reads every cached entry, and appends one new entry.

[Prefill and decode](https://aifordevs.org/glossary#prefill-and-decode) load the hardware differently. Prefill is a large batch of matrix multiplications, and the first output token cannot appear until it finishes, which makes prefill time a floor under [time to first token](https://aifordevs.org/glossary#ttft).

Decode does little arithmetic per token but moves a lot of data. [Shazeer](https://arxiv.org/abs/1911.02150) named this problem in 2019: repeatedly loading large key and value tensors eats memory bandwidth. The cost grows with every token in the sequence, so decode slows as a conversation gets longer.

## How big a KV cache gets

KV cache size is one multiplication from a model's config: two tensors, times layers, times KV heads, times head dimension, times bytes per value. Llama 3.1 70B comes to 320 KiB per token at 16 bits, which becomes 2.5 GiB for an 8K-token sequence and 40 GiB at 128K tokens, per sequence.

```text
bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_value
```

The 2 counts keys and values, and at 16 bits `bytes_per_value` is 2. `head_dim` is the model dimension divided by the attention heads. Architecture numbers below come from [Table 3 of the Llama 3 paper](https://arxiv.org/abs/2407.21783), recomputed with code on November 8, 2026.

| Model | Layers | KV heads | Head dim | Per token | 8K tokens | 32K tokens | 128K tokens |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama 3.1 8B | 32 | 8 | 128 | 128 KiB | 1.00 GiB | 4.00 GiB | 16.0 GiB |
| Llama 3.1 70B | 80 | 8 | 128 | 320 KiB | 2.50 GiB | 10.00 GiB | 40.0 GiB |
| Llama 3.1 405B | 126 | 8 | 128 | 504 KiB | 3.94 GiB | 15.75 GiB | 63.0 GiB |

Here 8K means 8,192 tokens and 128K means 131,072 tokens. Sixteen Llama 3.1 70B sequences at 8K tokens need 40 GiB of cache on top of 141 GB of 16-bit weights.

The same arithmetic runs against any Hugging Face `config.json`. This script ran on Python 3.15.1 and handles the compressed cache format DeepSeek models use.

```python title="kv_size.py"
"""Size a KV cache from a Hugging Face config.json. Tested with Python 3.15."""
import json
import sys


def kv_bytes_per_token(config_path: str, bytes_per_value: int = 2) -> int:
    cfg = json.load(open(config_path))
    layers = cfg["num_hidden_layers"]
    # DeepSeek-style MLA caches one compressed latent, not per-head K and V.
    if cfg.get("kv_lora_rank"):
        latent = cfg["kv_lora_rank"] + cfg["qk_rope_head_dim"]
        return layers * latent * bytes_per_value
    kv_heads = cfg.get("num_key_value_heads") or cfg["num_attention_heads"]
    head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
    return 2 * layers * kv_heads * head_dim * bytes_per_value


if __name__ == "__main__":
    for path in sys.argv[1:]:
        per_token = kv_bytes_per_token(path)
        name = path.rsplit("/", 1)[-1]
        print(f"{name}: {per_token / 1024:.1f} KiB per token, "
              f"{per_token * 8192 / 2**30:.2f} GiB per 8K-token sequence")
```

Meta's own Llama 3.1 repository is gated, so the run below reads [NousResearch's public copy](https://huggingface.co/NousResearch/Meta-Llama-3.1-8B/blob/main/config.json), whose layer and head counts match Table 3. [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B/blob/main/config.json) and [DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json) publish theirs openly.

```bash
curl -sL -o qwen3-8b.json https://huggingface.co/Qwen/Qwen3-8B/resolve/main/config.json
curl -sL -o llama-3.1-8b.json https://huggingface.co/NousResearch/Meta-Llama-3.1-8B/resolve/main/config.json
curl -sL -o deepseek-v3.json https://huggingface.co/deepseek-ai/DeepSeek-V3/resolve/main/config.json
python3 kv_size.py qwen3-8b.json llama-3.1-8b.json deepseek-v3.json
```

```text
qwen3-8b.json: 144.0 KiB per token, 1.12 GiB per 8K-token sequence
llama-3.1-8b.json: 128.0 KiB per token, 1.00 GiB per 8K-token sequence
deepseek-v3.json: 68.6 KiB per token, 0.54 GiB per 8K-token sequence
```

Qwen3-8B has 36 layers to Llama 3.1 8B's 32, which accounts for its extra 16 KiB per token.

## When the standard KV cache formula is wrong

The standard formula overstates any model that does not store a key and value per head in every layer. DeepSeek-V3 caches one compressed latent per layer, and gpt-oss-120b alternates sliding-window and full-attention layers, so a sizing script has to read those config fields before it trusts the textbook math.

DeepSeek introduced multi-head latent attention (MLA) in [its V2 report](https://arxiv.org/abs/2405.04434). MLA caches one compressed vector plus a small RoPE key, so per-token cache grows with `kv_lora_rank + qk_rope_head_dim` per layer instead of `2 x kv_heads x head_dim`.

Feed DeepSeek-V3's config through the naive formula and you get 1.67 MiB per token. The MLA branch in the script returns 68.6 KiB, about 25 times less. vLLM's [cache spec](https://github.com/vllm-project/vllm/blob/main/vllm/v1/kv_cache_interface.py) treats MLA the same way, with a single latent per state and no separate value tensor.

Sliding-window layers cap the cache at the window size. The [gpt-oss-120b config](https://huggingface.co/openai/gpt-oss-120b/blob/main/config.json) alternates 18 sliding-window layers with 18 full-attention layers and a 128-token window.

Counting all 36 layers as full attention gives 72 KiB per token. Counting only the 18 full layers gives 36 KiB, so the formula overstates a long sequence by about two times.

Vendors keep adding variants. MiniMax says its M3 sparse attention [reduces attention compute and memory compared with GQA](https://huggingface.co/MiniMaxAI/MiniMax-M3), which is the vendor's own claim. Read the model card before you trust any single formula.

## What concurrency costs on a real GPU

Concurrency cost equals cache per sequence times active sequences, paid from whatever memory the weights leave. A 141 GB H200 running Llama 3.1 8B at 16 bits leaves about 106 GiB for the cache. That holds about 105 sequences at 8K tokens and about 6 at 128K tokens.

| Deployment | Weights | KV pool | Sequences at 8K | At 32K | At 128K |
| --- | --- | --- | --- | --- | --- |
| Llama 3.1 8B, 16-bit, 1x H100 (80 GB) | 16.1 GB | 53.6 GiB | 53 | 13 | 3 |
| Llama 3.1 8B, 16-bit, 1x H200 (141 GB) | 16.1 GB | 105.9 GiB | 105 | 26 | 6 |
| Llama 3.1 70B, 16-bit, 2x H200 | 141.1 GB | 110.2 GiB | 44 | 11 | 2 |
| Llama 3.1 70B, FP8 weights and cache, 1x H200 | 70.6 GB | 55.1 GiB | 44 | 11 | 2 |

Counts are full-length sequences held at once, recomputed with code on November 8, 2026. The KV pool is GPU memory times vLLM's [default `gpu_memory_utilization` of 0.92](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) for a fixed-size cache, minus weights. GPU sizes come from NVIDIA's [H100](https://www.nvidia.com/en-us/data-center/h100/) and [H200](https://www.nvidia.com/en-us/data-center/h200/) pages, and weight sizes from each model's Hugging Face parameter count.

Going from 8K to 128K tokens cuts concurrency by 16 times, because the cache grows linearly with sequence length. Real traffic runs shorter than the maximum, which raises these counts, and activation memory lowers them.

Quantization moves the budget more than any other row.

FP8 weights and an FP8 cache fit Llama 3.1 70B on one H200 with the same 44-sequence capacity as two H200s at 16 bits. That only helps if [quantization](https://aifordevs.org/glossary#quantization) holds quality on your task. A [mixture-of-experts model](https://aifordevs.org/articles/mixture-of-experts) goes the other way, with far more memory spent on loaded weights and less left for the cache.

## Why decode is memory-bound

Decode is memory-bound because each step reads the weights and every active sequence's cache from GPU memory while doing little arithmetic per token. Batching shares the weight read across sequences but not the cache read, so cache traffic can pass weight traffic as contexts grow.

A rough floor shows the split. An [H100 moves 3.35 TB/s](https://www.nvidia.com/en-us/data-center/h100/) of GPU memory, so reading Llama 3.1 8B's 16.1 GB of 16-bit weights once takes 4.8 ms. One 8K-token sequence's 1 GiB cache adds 0.32 ms, and a batch of 32 such sequences adds 10.3 ms, twice the weight read.

At 128K tokens, the same batch would spend 164 ms per step on cache reads. Those figures are lower bounds from capacity and bandwidth alone, and they ignore DRAM efficiency and on-package caches. No benchmark sits behind them.

Published measurements point the same way. [Recasens et al.](https://arxiv.org/abs/2503.08311) report that large-batch GPU inference stays limited by DRAM bandwidth saturation, leaving compute capacity unused. DeepSeek's [V3 report](https://arxiv.org/abs/2412.19437) says decoding in its own fleet is bottlenecked by memory access rather than computation.

[Speculative decoding](https://aifordevs.org/articles/speculative-decoding) exists to spend that idle compute. A long-context request batched with short ones slows all of them, because each step waits on every cache read.

## How serving engines shrink a KV cache

Serving systems shrink or share the cache in four ways: fewer KV heads, paged allocation, lower-precision storage, and offloading cold entries. Fewer KV heads is decided at training time, so you choose it by choosing a model. The other three are server settings you can test without changing the weights.

### Fewer KV heads

[Multi-query attention](https://arxiv.org/abs/1911.02150) shares one key-value head across all query heads. [Grouped-query attention](https://aifordevs.org/glossary#grouped-query-attention), from [Ainslie et al.](https://arxiv.org/abs/2305.13245), uses more than one but fewer than the query heads.

Llama 3.1 70B has 64 query heads and 8 KV heads. Its 320 KiB per token is eight times smaller than the 2,560 KiB full multi-head attention would need.

### Paged allocation

[PagedAttention](https://aifordevs.org/glossary#paged-attention) stores each sequence's cache in fixed-size blocks that need not be contiguous, and maps them through a table the way an operating system pages memory. [Kwon et al.](https://arxiv.org/abs/2309.06180) measured that existing systems used only 20.4 to 38.2 percent of their KV cache memory for actual token state.

**Figure 2.** A contiguous reservation wastes the space a request never reaches. Paged allocation hands out fixed-size blocks on demand from one shared pool, so more requests fit on the same GPU.

### Quantized cache

Storing keys and values in 8 bits halves the cache. vLLM documents this as [`--kv-cache-dtype fp8`](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/) and warns that some attention types, such as sliding-window, are more sensitive to it. [SGLang's `--kv-cache-dtype`](https://docs.sglang.io/docs/advanced_features/server_arguments) accepts `fp8_e5m2`, `fp8_e4m3`, and 4-bit options.

### Offloading

Offloading moves inactive entries out of GPU memory. vLLM's [cache config](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) offers `native` and `lmcache` backends, and SGLang has [`--enable-hierarchical-cache`](https://docs.sglang.io/docs/advanced_features/server_arguments). Offloading pays off when a prefix is reused more often than it is evicted, because each hit still costs a read from the slower tier.

The [vLLM, SGLang, or llama.cpp comparison](https://aifordevs.org/articles/vllm-sglang-llama-cpp) covers how these engines differ more broadly.

## How the KV cache relates to prompt caching

Prompt caching is a retained KV cache sold as a discount. The provider keeps the KV tensors for a prompt prefix, reuses them for later requests that match exactly, skips prefill for those tokens, and charges a reduced input rate. OpenAI's guide says this directly, and the other providers below document the same exact-prefix contract.

Inside one server, the same idea is [prefix caching](https://aifordevs.org/glossary#prefix-caching). [vLLM](https://docs.vllm.ai/en/latest/design/prefix_caching/) hashes each cache block with its tokens and its parent's hash, and caches only full blocks. [SGLang's RadixAttention](https://arxiv.org/abs/2312.07104) keeps token prefixes in a radix tree and evicts least-recently-used leaves first.

Shared prefixes change the memory math. Take 32 users who each share a 16K-token system prompt and add 4K private tokens.

Without sharing, Llama 3.1 70B needs 655,360 cached tokens, or 200 GiB. With one shared prefix it needs 147,456 tokens, or 45 GiB, which is 4.4 times less.

| Provider | Stated storage | Read price | Write price | Lifetime |
| --- | --- | --- | --- | --- |
| [OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching) (GPT-5.6 and later) | KV tensors, may sit in GPU-local storage | 0.1x input, 0.05x on some models | 1.25x for 30 minutes, 2.5x for 12 hours | 30 minutes by default, 12-hour option |
| [Anthropic](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) | Not stated | 0.1x base input; 0.05x or 0.025x on some models | 1.25x for 5 minutes, 2x for 1 hour | 5 minutes by default, 1-hour option |
| [DeepSeek](https://api-docs.deepseek.com/guides/kv_cache) | Disk | Not on the caching page | Automatic | Cleared hours to days after last use, best-effort |

Prices are from each provider's docs as of November 8, 2026. A 1.25x write and 0.1x read cost 67.5 percent of the uncached rate by the second request and 21.5 percent by the tenth. A 5-minute cache pays for itself on the first reuse.

A 2x write costs 105 percent on the second request and 73.3 percent on the third, so a 1-hour cache needs at least two reads.

Retention has a real memory cost.

A 50,000-token prefix on a Llama 3.1 70B-sized model occupies 15.3 GiB while it stays resident. That is a reasonable explanation for write premiums and lifetime limits, although no provider publishes the breakdown. Check [Claude API pricing](https://aifordevs.org/articles/claude-api-pricing) and [Gemini API pricing](https://aifordevs.org/articles/gemini-api-pricing) for current per-token rates.

## How to budget a KV cache before you buy a GPU

Budget the cache by working backward from your p95 context length and target concurrency. The weights decide whether a model fits, and the cache decides how many users fit. Run these steps before you size a deployment, and repeat them whenever context limits or traffic change.

1. Run `kv_size.py` on your model's `config.json` and multiply by your p95 sequence length.
2. Multiply that by peak concurrent sequences, then add the weights in bytes at your serving precision.
3. Compare the total against 92 percent of your GPU memory.
4. If it does not fit, try a shared prefix first, then an FP8 cache, then a model with fewer KV heads or MLA, then more GPUs.

If the cache exceeds the weights at your target concurrency, spend on GPU memory before you spend on compute.

If you only call a hosted API, put stable content such as system prompts and tool definitions first, as the [context engineering guide](https://aifordevs.org/articles/context-engineering) describes. Then confirm hits in `usage.input_tokens_details.cached_tokens` on OpenAI, `cache_read_input_tokens` on Anthropic, or `prompt_cache_hit_tokens` on DeepSeek.

More serving guides are on the [inference topic page](https://aifordevs.org/topics/inference).

## Questions

### What is a KV cache in an LLM?

A KV cache is GPU memory that holds the attention keys and values a transformer computed for every earlier token in a sequence. Each new token reads those tensors instead of recomputing them, so generation stays fast. The tradeoff is memory that grows with layers, KV heads, head dimension, context length, and the number of active sequences.

### Does the KV cache change what the model outputs?

No. It stores values the model would otherwise recompute, so the math and the output distribution stay the same. Quantizing the cache to 8 bits or fewer is a separate choice. It halves memory or better, and it can change outputs slightly, so measure quality on your own task before you enable it.

### How much memory does a KV cache need?

Multiply two tensors by layers, KV heads, head dimension, and bytes per value. Llama 3.1 70B at 16 bits comes to 320 KiB per token, 2.5 GiB for an 8K-token sequence, and 40 GiB at 128K tokens. Models with latent attention or sliding windows need a different formula, so check the config fields first.

### Is a KV cache the same as prompt caching?

They are the same data at different scopes. A KV cache exists inside every generation. Prompt caching keeps part of that cache alive across requests that share an exact prefix, so later requests skip prefill for those tokens. OpenAI's prompt-caching guide states that its prompt cache stores KV tensors rather than the tokens themselves.

### Why does the KV cache limit concurrency?

Every active sequence needs its own keys and values for its whole context, on GPU memory, for its whole generation. After the weights load, the remaining memory divided by the per-sequence cache gives your concurrency ceiling. Doubling context length halves the number of sequences you can hold, because the cache grows linearly with tokens.

### Can a KV cache be stored on SSD or CPU memory?

Yes, for entries that are not being read on the current decode step. vLLM has a KV offloading backend setting with native and LMCache options, and SGLang has a hierarchical cache flag. Offloading trades GPU memory for slower access on a hit, so it pays off when prefixes get reused across requests.

## Sources

1. [Vaswani et al., Attention Is All You Need (2017)](https://arxiv.org/abs/1706.03762)
2. [Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019)](https://arxiv.org/abs/1911.02150)
3. [Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023)](https://arxiv.org/abs/2305.13245)
4. [Llama Team, The Llama 3 Herd of Models (2024), Table 3](https://arxiv.org/abs/2407.21783)
5. [DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024), Table 1](https://arxiv.org/abs/2405.04434)
6. [DeepSeek-AI, DeepSeek-V3 Technical Report (2024), Section 3.4](https://arxiv.org/abs/2412.19437)
7. [Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (2023)](https://arxiv.org/abs/2309.06180)
8. [Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (2023)](https://arxiv.org/abs/2312.07104)
9. [Recasens et al., Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference (2025)](https://arxiv.org/abs/2503.08311)
10. [vLLM, kv_cache_interface.py (main, November 2026)](https://github.com/vllm-project/vllm/blob/main/vllm/v1/kv_cache_interface.py)
11. [vLLM, cache configuration (main, November 2026)](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py)
12. [vLLM docs, Automatic Prefix Caching design](https://docs.vllm.ai/en/latest/design/prefix_caching/)
13. [vLLM docs, Quantized KV Cache](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/)
14. [SGLang docs, Server Arguments](https://docs.sglang.io/docs/advanced_features/server_arguments)
15. [OpenAI API docs, Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)
16. [Anthropic docs, Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
17. [DeepSeek API docs, Context Caching](https://api-docs.deepseek.com/guides/kv_cache)
18. [NVIDIA H100 specifications](https://www.nvidia.com/en-us/data-center/h100/)
19. [NVIDIA H200 specifications](https://www.nvidia.com/en-us/data-center/h200/)
20. [Qwen3-8B config.json on Hugging Face](https://huggingface.co/Qwen/Qwen3-8B/blob/main/config.json)
21. [NousResearch, public copy of the Llama 3.1 8B config.json](https://huggingface.co/NousResearch/Meta-Llama-3.1-8B/blob/main/config.json)
22. [DeepSeek-V3 config.json on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json)
23. [gpt-oss-120b config.json on Hugging Face](https://huggingface.co/openai/gpt-oss-120b/blob/main/config.json)
24. [MiniMax-M3 model card on Hugging Face](https://huggingface.co/MiniMaxAI/MiniMax-M3)

## Cite this deep dive

AI for Devs. "KV cache explained: what it stores and what it costs." Published November 6, 2026, updated November 8, 2026. https://aifordevs.org/articles/kv-cache-explained

- Page: https://aifordevs.org/articles/kv-cache-explained
- Markdown: https://aifordevs.org/articles/kv-cache-explained.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-06
- Updated: 2026-11-08

---
title: "Speculative decoding: how drafts speed up LLM inference"
type: "deep dive"
url: "https://aifordevs.org/articles/speculative-decoding"
markdown: "https://aifordevs.org/articles/speculative-decoding.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/speculative-decoding"
description: "Speculative decoding has a cheap draft propose tokens the target model verifies in one pass. See the math, the costs, and what vLLM and SGLang support."
author: "AI for Devs"
published: "2026-11-06"
updated: "2026-11-08"
level: "Advanced"
topics: ["inference"]
---

# Speculative decoding: how drafts speed up LLM inference

Speculative decoding has a cheap draft propose tokens the target model verifies in one pass. See the math, the costs, and what vLLM and SGLang support.

> **TL;DR**
>
> - Speculative decoding has a cheap draft propose tokens that the target model verifies in one pass, and the standard acceptance rule keeps the output distribution identical to the target alone.
> - Expected tokens per target pass equal `(1 - alpha^(k + 1)) / (1 - alpha)`, Equation 1 in Leviathan et al. At alpha 0.8 and k 4 that is 3.36, and a draft costing 10 percent of a target step cuts the walltime gain to 2.40 times.
> - vLLM removed Medusa and the MLP speculator in v0.33.0, published November 2, 2026, and no newer release has reversed that as of November 8, 2026.
> - vLLM and SGLang both document EAGLE, multi-token prediction, block-diffusion, and n-gram drafts today, under different flag names.
> - The simulation in this article ran on Python 3.15.1 with hand-built tables. No LLM, GPU, vLLM, or SGLang process ran for this post.

Speculative decoding speeds up LLM [inference](https://aifordevs.org/glossary#inference) by having a cheap draft propose several tokens that a large target model checks in a single forward pass. With the standard acceptance rule, the output distribution matches sampling from the target alone. The gain depends on acceptance rate, draft cost, and spare GPU compute, and you can measure all three. Engine support also changes fast, and vLLM v0.33.0 removed two draft methods in November 2026.

## What is speculative decoding

Speculative decoding is a way to generate tokens from a large model faster without changing its outputs. A cheap draft proposes several tokens, the large target model verifies them in one forward pass, and every accepted token saves a full target step. It targets [decode](https://aifordevs.org/glossary#prefill-and-decode), where the GPU mostly waits on memory.

During decode, a target model produces one [token](https://aifordevs.org/glossary#token) per forward pass, and each pass reads its weights and its [KV cache](https://aifordevs.org/articles/kv-cache-explained) from GPU memory. The arithmetic units mostly wait for that data, so spare compute is available to verify extra positions in the same pass.

[Leviathan et al.](https://arxiv.org/abs/2211.17192) posted the method in November 2022, and [Chen et al.](https://arxiv.org/abs/2302.01318) posted it independently in February 2023 as speculative sampling.

The first paper reports 2x to 3x on T5-XXL, and the second reports 2x to 2.5x on the 70B Chinchilla model. Both figures come from the authors.

## How one round of speculative decoding works

One round drafts k tokens, runs one target pass over the prefix plus all k drafts, and accepts them left to right until one fails. The first rejected position gets a target-sampled token, the rest are discarded, and if all k pass, the target adds one bonus token.

**Figure 1.** The draft proposes four tokens. One target pass scores all four positions, accepts the first two, rejects the third, and samples its own token in that position. The fourth draft token is never checked.

The target model is the large model whose outputs you want. The draft is any cheap source of guesses. Here is one round in order:

1. The draft generates k tokens one at a time.
2. The target runs one forward pass over the prefix plus all k drafts, which gives its probability for every drafted position.
3. Starting at the first draft token, each token is accepted with probability `min(1, p(x) / q(x))`, where `p` is the target and `q` is the draft.
4. At the first rejection, the target samples a replacement from the adjusted distribution, `max(0, p - q)` renormalized. Later drafts are discarded.
5. If all k tokens pass, the same forward pass already scored position k + 1, so the target adds that token at no extra cost.

The paper's [Algorithm 1](https://arxiv.org/abs/2211.17192) returns between 1 and k + 1 tokens per round. The cost is one target pass plus k draft steps.

## Why speculative decoding preserves outputs

The method preserves the target's output distribution because its acceptance rule is rejection sampling. A draft token `x` is kept with probability `min(1, p(x) / q(x))`, and each rejection draws from the leftover probability mass, which reproduces `p` exactly. Chen et al. describe the same guarantee, within hardware numerics.

A small simulation makes the guarantee and Equation 1 concrete. It uses hand-built next-token tables as the target and draft, samples 400,000 six-token sequences, and compares them with plain sampling from the target. No LLM or GPU ran, so this tests the rule and the formula and says nothing about any model's real acceptance rate.

```python title="spec_sim.py"
"""Speculative sampling on a toy Markov source. Python stdlib only.

No model runs here: TARGET and DRAFT are hand-built next-token tables.
It tests the acceptance rule and Equation 1, not any real model's alpha.
"""
import random
from collections import Counter

TOKENS = "abcd"


def table(seed):
    rng = random.Random(seed)
    tbl = {}
    for ctx in TOKENS:
        w = [rng.random() + 0.05 for _ in TOKENS]
        tbl[ctx] = {t: v / sum(w) for t, v in zip(TOKENS, w)}
    return tbl


TARGET = table(7)
NOISE = table(11)
DRAFT = {c: {t: 0.3 * TARGET[c][t] + 0.7 * NOISE[c][t] for t in TOKENS} for c in TOKENS}


def draw(dist, rng):
    x, total = rng.random(), 0.0
    for token, p in dist.items():
        total += p
        if x < total:
            return token
    return token


def residual(p, q):
    left = {t: max(0.0, p[t] - q[t]) for t in p}
    norm = sum(left.values())
    return {t: v / norm for t, v in left.items()}


def spec_round(last, k, rng, stats):
    """One target pass over k drafted tokens."""
    drafts, ctx = [], last
    for _ in range(k):
        token = draw(DRAFT[ctx], rng)
        drafts.append((ctx, token))
        ctx = token
    stats["passes"] += 1
    out = []
    for ctx, token in drafts:
        p, q = TARGET[ctx], DRAFT[ctx]
        accept = min(1.0, p[token] / q[token])
        stats["accept_sum"] += accept
        stats["checked"] += 1
        if rng.random() < accept:
            out.append(token)
        else:
            out.append(draw(residual(p, q), rng))  # the corrected token
            return out
    out.append(draw(TARGET[drafts[-1][1]], rng))  # the bonus token
    return out


def speculative(n, k, seed, length=6):
    rng, stats, seqs, made = random.Random(seed), Counter(), Counter(), 0
    for _ in range(n):
        seq, last = "", "a"
        while len(seq) < length:
            new = spec_round(last, k, rng, stats)
            seq += "".join(new)
            made += len(new)
            last = seq[-1]
        seqs[seq[:length]] += 1
    alpha = stats["accept_sum"] / stats["checked"]
    per_pass = made / stats["passes"]
    return seqs, alpha, per_pass


def plain(n, seed, length=6):
    rng, seqs = random.Random(seed), Counter()
    for _ in range(n):
        seq, last = "", "a"
        while len(seq) < length:
            last = draw(TARGET[last], rng)
            seq += last
        seqs[seq] += 1
    return seqs


def tv(a, b, n):
    return 0.5 * sum(abs(a[s] - b[s]) / n for s in set(a) | set(b))


N = 400_000
ref = plain(N, seed=1)
noise_floor = tv(ref, plain(N, seed=2), N)
print(f"total variation, plain vs plain:       {noise_floor:.4f}")
for k in (1, 2, 4, 8):
    seqs, alpha, per_pass = speculative(N, k, seed=3 + k)
    eq1 = (1 - alpha ** (k + 1)) / (1 - alpha)
    print(f"k={k}: TV vs plain {tv(ref, seqs, N):.4f}, "
          f"measured alpha {alpha:.3f}, tokens per pass {per_pass:.3f}, Eq. 1 {eq1:.3f}")
```

```text
total variation, plain vs plain:       0.0421
k=1: TV vs plain 0.0418, measured alpha 0.858, tokens per pass 1.858, Eq. 1 1.858
k=2: TV vs plain 0.0430, measured alpha 0.856, tokens per pass 2.589, Eq. 1 2.588
k=4: TV vs plain 0.0426, measured alpha 0.857, tokens per pass 3.755, Eq. 1 3.760
k=8: TV vs plain 0.0428, measured alpha 0.857, tokens per pass 5.248, Eq. 1 5.257
```

Total variation distance stays at the 0.042 noise floor of two plain sampling runs, and tokens per pass match Equation 1 to within 0.01.

## How to estimate speculative decoding speedup

Tokens per target pass come from Equation 1 in Leviathan et al., `(1 - alpha^(k + 1)) / (1 - alpha)`. Divide by `k * c + 1`, where `c` is the draft's cost relative to one target step, to get the walltime speedup in Theorem 3.8. Both assume acceptance is independent across positions.

```python
def tokens_per_pass(alpha: float, k: int) -> float:
    return (1 - alpha ** (k + 1)) / (1 - alpha)

def speedup(alpha: float, k: int, c: float) -> float:
    return tokens_per_pass(alpha, k) / (k * c + 1)
```

For k = 4, recomputed with code on November 8, 2026:

| Acceptance rate | Tokens per pass | Speedup at c = 0.02 | c = 0.05 | c = 0.10 | c = 0.33 |
| --- | --- | --- | --- | --- | --- |
| 0.5 | 1.94 | 1.79x | 1.61x | 1.38x | 0.84x |
| 0.6 | 2.31 | 2.13x | 1.92x | 1.65x | 0.99x |
| 0.7 | 2.77 | 2.57x | 2.31x | 1.98x | 1.20x |
| 0.8 | 3.36 | 3.11x | 2.80x | 2.40x | 1.45x |
| 0.9 | 4.10 | 3.79x | 3.41x | 2.93x | 1.77x |

Treat these as ceilings. The paper calls the independence assumption an approximation, and its walltime analysis assumes the target can verify k + 1 positions in parallel without slowing down. A slow draft eats most of the gain: a draft at one third of a target step, with 0.5 acceptance, runs slower than no speculation at all.

The best draft length depends on acceptance and cost, and the paper finds it numerically.

**Figure 2.** At 0.7 acceptance, tokens per pass keep climbing with draft length, but once you pay for the draft the speedup peaks and then falls. The best k shrinks as the draft gets more expensive.

At alpha 0.8, a drafter with c = 0.05 is fastest at k = 8 (3.09x), and one with c = 0.33 is fastest at k = 3 (1.48x). Going from k = 4 to k = 8 at alpha 0.7 adds only 0.43 expected tokens per pass while doubling draft work.

## Where draft tokens come from

Draft tokens come from a separate small model, a small network on the target's hidden features, multi-token heads trained with the target, a block-diffusion drafter, or lookup in text already seen. They trade draft quality against draft cost, the `c` in the speedup formula.

### A separate small model

The original setup pairs a small model with a much larger sibling. vLLM's [draft model docs](https://docs.vllm.ai/en/latest/features/speculative_decoding/) use Qwen3-0.6B as a draft for Qwen3-8B. The draft needs its own weights and its own KV cache, which raises `c` and memory use.

### Hidden features (EAGLE)

[EAGLE](https://arxiv.org/abs/2401.15077) drafts from the target's hidden features instead of tokens, using a small layer trained on the target. Its authors report about 0.8 acceptance against about 0.6 for [Medusa](https://arxiv.org/abs/2401.10774), which adds extra prediction heads to the target. Those are author-reported figures on the paper's benchmarks.

### Multi-token prediction (MTP)

MTP heads are trained with the target itself, so the checkpoint already carries a draft. [DeepSeek-V3](https://arxiv.org/abs/2412.19437) trained them and notes they can be reused for speculative decoding. vLLM says [MTP only works for model families it supports](https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/).

### Block diffusion

[DFlash](https://arxiv.org/abs/2602.06036) proposes a whole block of tokens in one pass with a small diffusion model conditioned on the target's features. Its authors report over 6x lossless acceleration and up to 2.5x the speedup of EAGLE-3, on their own benchmarks. vLLM's [v0.33.0 notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3) describe DSpark as a similar block-diffusion drafter.

### N-gram and suffix lookup

Lookup methods copy from text already seen, so they need no model. They work best when output repeats input, such as code edits and summarization.

vLLM measured [up to 2.8x with prompt lookup on CNN/DailyMail](https://vllm.ai/blog/2024-10-17-spec-decode) at QPS 1, in its own 2024 benchmark. [Best LLM for coding](https://aifordevs.org/articles/best-llm-for-coding) covers the model choice for that workload.

## What vLLM and SGLang support today

As of November 8, 2026, vLLM's `speculative_config` accepts `ngram`, `ngram_gpu`, `draft_model`, `suffix`, `eagle`, `eagle3`, `mtp`, `dflash`, `dspark`, and `extract_hidden_states`. SGLang documents EAGLE, EAGLE3, STANDALONE, NGRAM, DFLASH, and UNO. Medusa is in neither list, because vLLM removed it in v0.33.0 and SGLang's docs do not mention it.

| Draft source | vLLM `method` | SGLang `--speculative-algorithm` |
| --- | --- | --- |
| Separate small model | `draft_model` | `STANDALONE` |
| EAGLE family | `eagle`, `eagle3` | `EAGLE` (EAGLE-2), `EAGLE3`, `NEXTN` as an alias of `EAGLE` |
| Multi-token heads | `mtp` | `EAGLE` with an MTP checkpoint |
| Block diffusion | `dflash`, `dspark` | `DFLASH` |
| Prompt lookup | `ngram`, `ngram_gpu` | `NGRAM` |
| Earlier responses | `suffix` | Not documented |
| LoRA on the target | Not documented | `UNO` |
| Medusa, MLP speculator | Removed in v0.33.0 | Not documented |

The method names come from [vLLM's config source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py), [its docs](https://docs.vllm.ai/en/latest/features/speculative_decoding/), and [SGLang's docs](https://docs.sglang.io/docs/advanced_features/speculative_decoding). None of these flags ran for this post.

vLLM merged the [Medusa removal](https://github.com/vllm-project/vllm/pull/61701) on October 15, 2026. The pull request says Medusa was not supported by the newer model runner, the MLP speculator was already disabled, and EAGLE, MTP, DFlash, and DSpark cover the use case. The [v0.33.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3) list it under breaking changes.

Those notes sit on the tag `v0.33.0rc3`, published November 2, 2026, while the bare `v0.33.0` tag page has no notes. A config that still sets `method="medusa"` now fails validation.

The newest vLLM tag on November 8, 2026 is still `v0.33.0`, and `main` does not bring either method back. The [vLLM, SGLang, or llama.cpp comparison](https://aifordevs.org/articles/vllm-sglang-llama-cpp) covers the engines more broadly.

## When speculative decoding hurts

Speculative decoding hurts when the GPU has no compute to spare, when acceptance is low, or when the draft costs too much. vLLM's own 2024 benchmarks showed 1.4x and 1.8x slowdowns on Llama 3 70B at high QPS, and its docs now aim the feature at medium-to-low QPS, memory-bound workloads.

### Load

vLLM's [2024 post](https://vllm.ai/blog/2024-10-17-spec-decode) reported up to 1.5x on ShareGPT at QPS 1 and the slowdowns above at high QPS, on four H100s, with no independent measurement. Verification multiplies the effective batch: vLLM's [dynamic speculative decoding docs](https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/) say that once batch size times K passes a critical point, decode speed drops. Their fix is a per-concurrency K, with `[[1, 64, 3], [65, 128, 1], [129, 512, 0]]` as the example.

### Low acceptance

High [temperature](https://aifordevs.org/glossary#temperature) and open-ended prose are harder to predict. At an alpha of 0.3 and k = 4, Equation 1 gives 1.43 tokens per pass, and a draft at c = 0.10 brings the speedup to 1.02x. Most draft work is thrown away at that rate.

### Memory for the draft

A separate draft adds its own weights and its own KV cache, which vLLM's memory manager [now tracks alongside the target's](https://vllm.ai/blog/2024-10-17-spec-decode). MTP and EAGLE heads avoid a whole second model, and lookup methods add no weights.

### Lossy verification

Some speedups give up the distribution guarantee on purpose. vLLM's [FLy verification](https://docs.vllm.ai/en/latest/features/speculative_decoding/fly/) defers some rejections and is documented as intentionally lossy, with [1.04x to 1.31x decode throughput reported on MI355X and B300](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3) in vLLM's own release notes. Use it only if you test output quality on your task.

## How to measure speculative decoding on your traffic

Measure acceptance, per-token latency, and the batch sizes you actually run. vLLM exposes `vllm:spec_decode_num_draft_tokens` and `vllm:spec_decode_num_accepted_tokens` at `/metrics`, and their ratio is your acceptance rate. Compare time per output token with speculation on and off at your real concurrency.

The metric names come from [vLLM's source](https://github.com/vllm-project/vllm/blob/main/vllm/v1/spec_decode/metrics.py). The per-request flag [`--per-request-spec-decode-metrics summary`](https://docs.vllm.ai/en/latest/features/speculative_decoding/acceptance_metrics/) reports a `mean_acceptance_length` under `metrics.speculative_decoding`, which is experimental and may change between releases.

Run the [speedup function](#how-to-estimate-speculative-decoding-speedup) with your measured `alpha` and `c` before you pay for a new draft model. Watch [throughput](https://aifordevs.org/glossary#throughput) as well as [time to first token](https://aifordevs.org/glossary#ttft), because a high-QPS endpoint can lose throughput while a low-load test still looks fine.

## How to decide on speculative decoding

Turn it on for latency-sensitive, low-concurrency endpoints first, such as interactive chat and code completion. Leave high-throughput batch jobs alone until a load test shows a gain. Use this order when you pick a method, and retest after each engine upgrade.

1. If your model ships MTP heads or an EAGLE checkpoint, start there, because the draft is trained on the target and adds little memory.
2. If your prompts repeat in the output, enable `ngram` or `suffix`, because they cost almost no compute.
3. If neither applies, try a small draft from the same model family and measure `alpha`, `c`, and memory.
4. If you expect high concurrency, set a per-concurrency K of zero at the top of the range.

The [KV cache](https://aifordevs.org/glossary#kv-cache) budget and [batching](https://aifordevs.org/glossary#continuous-batching) strategy sit underneath this choice, and a sparse [mixture-of-experts model](https://aifordevs.org/articles/mixture-of-experts) changes that math again. The [inference topic page](https://aifordevs.org/topics/inference) collects the related guides.

## Questions

### What is speculative decoding in LLMs?

Speculative decoding is a method for generating tokens from a large model faster. A cheap draft proposes several tokens, the large target model checks them all in one forward pass, and every accepted token saves one target step. With the standard acceptance rule, the output distribution is the same as sampling from the target model alone.

### Does speculative decoding change the model's answers?

With the standard acceptance rule from Leviathan et al. and Chen et al., no. The output follows the same distribution as the target alone, apart from hardware numerics. Lossy methods such as FLy, which vLLM documents as intentionally lossy, relax that guarantee for speed and may change outputs. Check which verification policy your config uses.

### What is a good acceptance rate for speculative decoding?

Measure it on your own traffic, because acceptance varies with the draft and the task. The EAGLE paper reports about 0.8 for its method and about 0.6 for Medusa on its benchmark. At an alpha of 0.8 and four draft tokens, Equation 1 gives 3.36 tokens per target pass, and low acceptance near 0.3 gives only about 1.4.

### Why does speculative decoding hurt at high load?

Verification multiplies the effective batch size, because each sequence checks K tokens per step. vLLM's docs say that once batch size times K passes a critical point, decode speed drops. vLLM measured slowdowns of 1.4x and 1.8x on Llama 3 70B at high QPS, so throughput-focused deployments often need a smaller K or none.

### Is Medusa still supported in vLLM?

No. vLLM removed Medusa and the MLP speculator in pull request 61701, merged October 15, 2026, and listed the removal under breaking changes in the v0.33.0 notes published November 2, 2026. A config that still sets method to medusa now fails validation. As of November 8, 2026, no newer vLLM release has reversed that.

### What is the difference between speculative decoding and speculative sampling?

They describe the same family of methods. Leviathan et al. used the term speculative decoding, and Chen et al. used speculative sampling for the same idea, posted independently a few months apart. Both pair a cheap draft with a target model and use modified rejection sampling to keep the target's output distribution.

## Sources

1. [Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2022)](https://arxiv.org/abs/2211.17192)
2. [Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling (2023)](https://arxiv.org/abs/2302.01318)
3. [Cai et al., Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (2024)](https://arxiv.org/abs/2401.10774)
4. [Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024)](https://arxiv.org/abs/2401.15077)
5. [Chen, Liang, and Liu, DFlash: Block Diffusion for Flash Speculative Decoding (2026)](https://arxiv.org/abs/2602.06036)
6. [DeepSeek-AI, DeepSeek-V3 Technical Report (2024), Section 2.2 on multi-token prediction](https://arxiv.org/abs/2412.19437)
7. [vLLM v0.33.0 release notes (attached to tag v0.33.0rc3)](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3)
8. [vLLM pull request 61701, Remove Medusa and MLP speculator support](https://github.com/vllm-project/vllm/pull/61701)
9. [vLLM speculative configuration on main (November 2026)](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py)
10. [vLLM docs, Speculative Decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/)
11. [vLLM docs, Multi-Token Prediction](https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/)
12. [vLLM docs, Dynamic Speculative Decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/)
13. [vLLM docs, FLy Verification](https://docs.vllm.ai/en/latest/features/speculative_decoding/fly/)
14. [vLLM docs, Per-Request Acceptance Metrics](https://docs.vllm.ai/en/latest/features/speculative_decoding/acceptance_metrics/)
15. [vLLM metrics source (main, November 2026)](https://github.com/vllm-project/vllm/blob/main/vllm/v1/spec_decode/metrics.py)
16. [vLLM blog, How Speculative Decoding Boosts vLLM Performance (2024)](https://vllm.ai/blog/2024-10-17-spec-decode)
17. [SGLang docs, Speculative Decoding](https://docs.sglang.io/docs/advanced_features/speculative_decoding)

## Cite this deep dive

AI for Devs. "Speculative decoding: how drafts speed up LLM inference." Published November 6, 2026, updated November 8, 2026. https://aifordevs.org/articles/speculative-decoding

- Page: https://aifordevs.org/articles/speculative-decoding
- Markdown: https://aifordevs.org/articles/speculative-decoding.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-06
- Updated: 2026-11-08

---
title: "vLLM vs SGLang, plus Ollama and llama.cpp compared"
type: "deep dive"
url: "https://aifordevs.org/articles/vllm-sglang-llama-cpp"
markdown: "https://aifordevs.org/articles/vllm-sglang-llama-cpp.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/vllm-sglang-llama-cpp"
description: "vLLM vs SGLang compared on prefix caching, hardware, and constrained decoding, plus Ollama and llama.cpp. Every default checked November 8, 2026."
author: "AI for Devs"
published: "2026-11-06"
updated: "2026-11-08"
level: "Intermediate"
topics: ["inference", "models"]
---

# vLLM vs SGLang, plus Ollama and llama.cpp compared

vLLM vs SGLang compared on prefix caching, hardware, and constrained decoding, plus Ollama and llama.cpp. Every default checked November 8, 2026.

> **TL;DR**
>
> - For vLLM vs SGLang, start with vLLM when you need its documented 200+ model architectures, and with SGLang when most requests share long prefixes. Both run on NVIDIA and AMD GPUs with prefix caching on by default as of November 8, 2026.
> - vLLM v0.33.0 reuses prefixes in full 16-token blocks, while SGLang v0.5.23 matches at one-token pages. On a 4,000-token shared prompt, that gap is at most 15 tokens.
> - Ollama runs one request per model by default. Going from one to four llama-server slots raised throughput 3.2 to 3.8 times on four concurrent requests in this article's test.
> - llama-server now listens on port 9931 by default, so older guides that assume 8080 will send your client to the wrong port.
> - Tested here: llama.cpp build b12202, compiled from source on an Apple M2. vLLM v0.33.0, SGLang v0.5.23, and Ollama v0.40.15 were not run, and their commands come from current docs.

vLLM and SGLang are open-source [inference](https://aifordevs.org/glossary#inference) engines that run open-weight models on GPU servers behind OpenAI-compatible APIs. The vLLM vs SGLang decision usually turns on hardware, cache behavior, and which engine supports your model on the day you need it. Ollama and llama.cpp solve a different problem: one machine, few users, and quantized weights. This comparison checks each engine's defaults against its own docs and release notes as of November 8, 2026.

## vLLM vs SGLang at a glance

Start with vLLM if you want the safest default for multi-user GPU serving, because its README documents 200+ model architectures and a long list of hardware plugins. Start with SGLang if agent loops and multi-turn chat dominate your traffic, because its radix cache matches prefixes at token granularity. Both are Apache-2.0.

| | vLLM v0.33.0 | SGLang v0.5.23 | llama-server b12202 | Ollama v0.40.15 |
| --- | --- | --- | --- | --- |
| Built for | Multi-user GPU serving | Multi-user GPU serving | One binary on most hardware | A local daemon with a model library |
| Default port | [8000](https://docs.vllm.ai/en/latest/getting_started/quickstart/) | [30000](https://docs.sglang.io/docs/advanced_features/server_arguments) | [9931](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) | [11434](https://docs.ollama.com/faq) |
| Weights | Hugging Face; GGUF through a plugin | Hugging Face; a `gguf` load format | GGUF | GGUF and Safetensors import |
| Prefix cache | On, 16-token blocks | On, one-token pages | On by default (`cache_prompt`) | Not described in the FAQ |
| Concurrent requests | Continuous batching | [Batch scheduler](https://www.lmsys.org/blog/2024-12-04-sglang-v0-4/), FCFS by default | Slots, auto (4 here) | 1 per model, 512 queued |
| License | [Apache-2.0](https://github.com/vllm-project/vllm) | [Apache-2.0](https://github.com/sgl-project/sglang) | [MIT](https://github.com/ggml-org/llama.cpp) | [MIT](https://github.com/ollama/ollama) |

All four expose [OpenAI-compatible routes](https://docs.ollama.com/api/openai-compatibility), so moving between them is usually a base URL change. Extras such as sampling options and structured output fields differ, and the [inference platforms directory](https://aifordevs.org/platforms) lists hosted options if you would rather not run any of them.

## SGLang vs vLLM on prefix caching

Both engines keep [KV cache](https://aifordevs.org/articles/kv-cache-explained) entries from earlier requests and skip [prefill](https://aifordevs.org/glossary#prefill-and-decode) for any prefix they have already seen. vLLM finds those entries with a chain of block hashes, and SGLang finds them by walking a radix tree of token IDs. The structures differ in match granularity, eviction choices, and what you can tune.

**Figure 1.** vLLM only reuses complete blocks, so two requests sharing 28 tokens reuse 16 of them. SGLang's tree stores one token per page by default and reuses all 28.

### vLLM: hashed blocks

vLLM's [design doc](https://docs.vllm.ai/en/latest/design/prefix_caching/) hashes each full block together with its parent block's hash, so a block's identity depends on everything before it. It caches full blocks only. Eviction takes the least recently used block from a free queue.

The block size is 16 tokens in [v0.33.0's CacheConfig](https://github.com/vllm-project/vllm/blob/v0.33.0rc3/vllm/config/cache.py), where `enable_prefix_caching` defaults to true. [Pull request 63231](https://github.com/vllm-project/vllm/pull/63231), merged on November 5, 2026, raises the default on main to 64 and lists coarser prefix hits as a trade-off. Check the block size again when you upgrade.

Per-request `cache_salt` values feed the first block's hash, which isolates tenants that should never share cached prefixes.

### SGLang: a radix tree

SGLang's RadixAttention [keeps KV cache for prompts and generations in a radix tree](https://www.lmsys.org/blog/2024-01-17-sglang/), mapping token sequences to cache tensors on the GPU while the tree lives on the CPU. The current [server arguments](https://docs.sglang.io/docs/advanced_features/server_arguments) set `--page-size` to 1 and leave `--disable-radix-cache` off, so token-level reuse is on by default.

The scheduler is less cache-driven by default than older write-ups suggest. `--schedule-policy` defaults to `fcfs`, and the longest-prefix-match policy `lpm` is something you opt into. [Eviction](https://docs.sglang.io/docs/advanced_features/radix_eviction_policy) defaults to LRU over leaf nodes, with `lfu`, `slru`, `priority`, and `tlru` policies for skewed traffic.

### Granularity matters most for short prefixes

The gap between the two designs is smaller than a diagram suggests. A vLLM block of 16 tokens forfeits at most 15 tokens per request, which is 0.4% of a 4,000-token shared prompt. At the new 64-token default, the worst case is 63 tokens, or 1.6%.

Granularity matters when shared prefixes are short, or when many branches diverge just past a block boundary. Measure it instead of guessing. vLLM exposes `vllm:prefix_cache_queries` and `vllm:prefix_cache_hits` in its [metrics](https://docs.vllm.ai/en/latest/design/metrics/), and SGLang's `--enable-cache-report` adds cached token counts to each response. To raise hit rates on either engine, [put stable content first](https://aifordevs.org/articles/context-engineering).

## Concurrent requests: batches, slots, and a queue

The engines differ most in what happens when four requests arrive at once. vLLM and SGLang batch them into shared GPU steps. llama-server assigns each one to a slot in a single process. Ollama runs one request per model and queues the rest, unless you change its default parallelism setting.

**Figure 2.** The four engines at their defaults. Only Ollama holds the second, third, and fourth requests back, and one environment variable changes that.

vLLM's [PagedAttention paper](https://arxiv.org/abs/2309.06180) reports 2 to 4 times the throughput of FasterTransformer and Orca at the same latency, a figure from the vLLM authors on 2023 models. [Continuous batching](https://aifordevs.org/glossary#continuous-batching) is why short requests do not wait for long ones to finish.

llama-server has [continuous batching on by default](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) too. Posts that describe llama.cpp as strictly sequential are out of date; the limit is slot count and memory. On the test machine, the automatic `-np -1` setting created 4 slots sharing one unified KV buffer.

## Hardware and model format support

vLLM and SGLang now cover far more than NVIDIA and AMD, and llama.cpp lists more consumer backends than the others. Check the platform you own against this table first, because a missing backend ends the comparison. Entries come from each project's own README and install docs on November 8, 2026.

| Hardware | vLLM | SGLang | llama.cpp | Ollama |
| --- | --- | --- | --- | --- |
| NVIDIA | [CUDA on Linux](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/), compute capability 7.5+ | [B200, H100, A100, RTX 30 to 50](https://github.com/sgl-project/sglang) | [CUDA backend](https://github.com/ggml-org/llama.cpp) | [Capability 5.0+](https://docs.ollama.com/gpu), driver 550+ |
| AMD | ROCm 6.3+ | Instinct MI300X to MI355X | HIP, Vulkan | ROCm v7, Vulkan |
| Intel | Intel XPU | Arc B-Series GPUs, Xeon CPUs | SYCL | Not named; Vulkan adds other GPUs |
| Apple silicon | [vLLM-Metal plugin](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/); [CPU build experimental](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) | [Metal and MLX, from source](https://docs.sglang.io/docs/hardware-platforms/apple_metal) | Metal, on by default | Metal, plus [MLX engine](https://ollama.com/blog/mlx) |
| CPU | x86, Arm, and PowerPC | Xeon | Core target | Supported |
| Other | Plugins for TPU, Gaudi, Ascend, and more | TPU v6e and v7, Ascend, Moore Threads | CANN, Hexagon, MUSA, WebGPU, and more | None listed |

The docs and release notes disagree on CUDA. The [v0.33.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3) say release wheels and images are CUDA 13.0 only, while the [GPU install page](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/) still shows CUDA 12.9 on November 8, 2026. Trust the release notes, and match your drivers to them.

### Model formats

vLLM and SGLang load Hugging Face weights. vLLM's [GGUF support](https://docs.vllm.ai/en/latest/features/quantization/gguf/) is "highly experimental" by its own docs and requires the separate `vllm-gguf-plugin`, and SGLang lists `gguf` as a `--load-format` option.

llama.cpp reads [GGUF](https://aifordevs.org/glossary#gguf) files and pulls them from Hugging Face with `-hf`. Its README lists 1.5-bit through 8-bit integer [quantization](https://aifordevs.org/glossary#quantization) for lower memory use. Ollama [imports GGUF files and Safetensors directories](https://docs.ollama.com/import) through a Modelfile, and it runs [GGUF through llama.cpp](https://ollama.com/blog/improved-performance-and-model-support-with-gguf) alongside its MLX engine on Apple silicon.

## Constrained decoding across the four engines

All four engines can force output to match a JSON schema, but only vLLM, SGLang, and llama.cpp document grammars beyond JSON. If you need regex or EBNF, Ollama is out, and on vLLM and SGLang the backend you choose sets the accepted syntax. This matters for [structured outputs](https://aifordevs.org/glossary#structured-outputs) at scale.

| Engine | Default backend | Constraints documented |
| --- | --- | --- |
| [vLLM](https://docs.vllm.ai/en/latest/features/structured_outputs/) | `auto`, choosing among XGrammar, Guidance, and Outlines | JSON schema, regex, choice, EBNF grammar, structural tags |
| [SGLang](https://docs.sglang.io/docs/advanced_features/structured_outputs) | XGrammar, with Outlines and llguidance selectable | JSON schema, regex, EBNF, structural tags |
| [llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) | Built-in [GBNF grammars](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) | JSON schema, GBNF |
| [Ollama](https://docs.ollama.com/capabilities/structured-outputs) | `format` field | JSON or a JSON schema |

Two details cause migrations to fail. vLLM deprecated `lm-format-enforcer` in v0.33.0 and plans to remove it in v0.35.0, and SGLang accepts only one of `json_schema`, `regex`, or `ebnf` per request.

[Constrained decoding](https://aifordevs.org/glossary#constrained-decoding) also works on small local models. The llama-server run in the Ollama vs llama.cpp section below enforces a schema through the OpenAI-style `response_format` field on a 0.8B model.

## vLLM vs Ollama: a shared GPU server or a local daemon

Use Ollama when one developer runs models on one machine, and use vLLM when several people or services share a GPU. Ollama optimizes for pulling and running a model with one command. vLLM optimizes for batching concurrent traffic on Linux GPUs, and it asks for more setup in return.

Ollama's [FAQ](https://docs.ollama.com/faq) spells out its defaults: `OLLAMA_NUM_PARALLEL` is 1 per model, `OLLAMA_MAX_QUEUE` is 512, models stay loaded for 5 minutes, and the context window is 4,096 tokens. Required RAM scales with parallelism times context length, so check `OLLAMA_CONTEXT_LENGTH` before you send a [RAG](https://aifordevs.org/glossary#rag) prompt longer than 4,096 tokens.

vLLM's [route list](https://docs.vllm.ai/en/latest/serving/online_serving/) covers OpenAI chat, completions, responses, and embeddings, plus Anthropic `/v1/messages`, rerank, and score endpoints. Those routes help once [coding agents](https://aifordevs.org/articles/best-llm-for-coding) and internal tools hit the same server. vLLM also needs Python 3.11 to 3.14 and Linux for GPU serving.

Move to vLLM when concurrency stops fitting within `OLLAMA_NUM_PARALLEL`, or when the model you want has no GGUF build. Ollama can stay on developer laptops while production points at vLLM through a different base URL.

## Ollama vs llama.cpp: convenience or control

Ollama is a daemon and model library that [uses llama.cpp for GGUF models](https://ollama.com/blog/improved-performance-and-model-support-with-gguf), while llama.cpp is the inference library plus a server you configure with flags. Pick Ollama when you want pull-and-run. Pick llama-server when you need to set slots, grammars, cache behavior, or a specific build.

The most visible default difference is parallelism. The test below measures it on llama-server by comparing one slot against the automatic setting. Ollama itself was not run for this article, so `-np 1` shows the effect of serving one request at a time, and it says nothing about Ollama's speed.

Tested on an Apple M2 with 16 GB of memory, llama.cpp build b12202 compiled from source with Metal, and `ggml-org/gemma-3-270m-it-GGUF` at Q8_0 (292 MB). The machine was running other jobs, so read the ratios, not the absolute numbers.

```bash title="build and run"
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF \
  -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --target llama-server -j 7

LLAMA_CACHE=./models ./build/bin/llama-server -hf ggml-org/gemma-3-270m-it-GGUF
```

```python title="load_test.py"
"""Send the same chat request to any OpenAI-compatible server at two concurrency levels."""
import json, sys, time, urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE = sys.argv[1]  # for example http://localhost:9931
MODEL = sys.argv[2]
REQUESTS, MAX_TOKENS = 8, 128


def chat(i):
    body = {
        "model": MODEL,
        "messages": [{"role": "user", "content": f"Request {i}: list facts about caches."}],
        "max_tokens": MAX_TOKENS,
        "temperature": 0,
        "ignore_eos": True,  # llama-server extra: every request decodes MAX_TOKENS
    }
    req = urllib.request.Request(
        f"{BASE}/v1/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=600) as r:
        usage = json.load(r)["usage"]
    return usage["completion_tokens"], time.perf_counter() - start


for workers in (1, 4):
    start = time.perf_counter()
    with ThreadPoolExecutor(workers) as pool:
        results = list(pool.map(chat, range(REQUESTS)))
    wall = time.perf_counter() - start
    tokens = sum(t for t, _ in results)
    mean = sum(s for _, s in results) / len(results)
    print(f"concurrency={workers} requests={REQUESTS} tokens={tokens} "
          f"wall={wall:.2f}s throughput={tokens / wall:.1f} tok/s mean_latency={mean:.2f}s")
```

Real output, default slots and then `-np 1`:

```text
# default: total_slots 4
concurrency=1 requests=8 tokens=1024 wall=23.06s throughput=44.4 tok/s mean_latency=2.88s
concurrency=4 requests=8 tokens=1024 wall=6.05s throughput=169.3 tok/s mean_latency=3.02s
# -np 1: total_slots 1
concurrency=1 requests=8 tokens=1024 wall=22.63s throughput=45.2 tok/s mean_latency=2.83s
concurrency=4 requests=8 tokens=1024 wall=22.72s throughput=45.1 tok/s mean_latency=9.21s
```

Four slots delivered 3.8 times the [throughput](https://aifordevs.org/glossary#throughput) of one slot here, and 3.2 times in an earlier round on the same machine (145.4 against 45.2 tokens per second). With one slot, mean latency for four concurrent requests was 9.2 seconds instead of 3.0.

Prefix reuse and JSON-schema decoding both work on llama-server, and reuse depends on the model. This script ran against `ggml-org/Qwen3.5-0.8B-GGUF` at Q8_0, started with `--reasoning off`:

```python title="prefix_and_schema.py"
"""Show llama-server's prefix reuse and JSON-schema constrained decoding."""
import json, urllib.request

URL = "http://localhost:9931/v1/chat/completions"
SYSTEM = "You answer questions about inference servers. " * 60


def post(**body):
    req = urllib.request.Request(URL, data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        return json.load(r)


for question in ("What is a slot?", "What is a grammar?"):
    out = post(model="gemma", max_tokens=8, temperature=0, messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": question},
    ])
    t = out["timings"]
    print(f"{question:20} cached={t['cache_n']:4} evaluated={t['prompt_n']:4} "
          f"prefill={t['prompt_ms']:.0f} ms")

schema = {
    "type": "object",
    "properties": {
        "engine": {"type": "string", "enum": ["vllm", "sglang", "llama.cpp", "ollama"]},
        "default_port": {"type": "integer"},
    },
    "required": ["engine", "default_port"],
    "additionalProperties": False,
}
out = post(model="gemma", max_tokens=64, temperature=0,
           messages=[{"role": "user", "content": "Name one inference engine and a port."}],
           response_format={"type": "json_schema",
                            "json_schema": {"name": "engine", "schema": schema}})
print(out["choices"][0]["message"]["content"])
print("parsed:", json.loads(out["choices"][0]["message"]["content"]))
```

```text
What is a slot?      cached=   0 evaluated= 442 prefill=1170 ms
What is a grammar?   cached= 425 evaluated=  17 prefill=151 ms
{
  "engine": "vllm",
  "default_port": 8000
}
parsed: {'engine': 'vllm', 'default_port': 8000}
```

The second request reused 425 cached prompt tokens, and prefill fell from 1,170 ms to 151 ms. The JSON output parsed against the schema on the first try. The same script on Gemma 3 270M reused 1 token, and the verbose server log shows context checkpoints being invalidated on that model (`n_swa = 512`). Test prefix reuse on the model you plan to ship.

## vLLM vs llama.cpp: batching or portability

Choose vLLM when many requests share a GPU and throughput per dollar decides the bill. Choose llama.cpp when you need one binary on a laptop, an edge device, or a machine with no CUDA. The overlap is small, because vLLM treats GGUF as experimental and llama.cpp loads GGUF files.

A useful test is where your weights already live. If your team ships Hugging Face checkpoints at FP8 or NVFP4, stay with vLLM or SGLang. If you distribute a single quantized GGUF file to users' machines, llama.cpp's `-hf` flag and Metal, Vulkan, and CPU backends do the job with no Python environment.

llama.cpp's [install docs](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) list Homebrew, conda-forge, Winget, MacPorts, and Nix, and the README adds an install script and a Docker image. vLLM's route is a Python 3.11 to 3.14 environment and `uv pip install vllm --torch-backend=auto`, then `vllm download-kernels`.

## Start an OpenAI-compatible server with each engine

Each of these commands starts a server, and the same chat request then works against all four once you change the port and model name. Only the llama-server command ran for this article. The other three come from the [vLLM](https://docs.vllm.ai/en/latest/getting_started/quickstart/) and [SGLang](https://docs.sglang.io/docs/get-started/quickstart) quickstarts and Ollama's [quickstart](https://docs.ollama.com/quickstart) and [Linux install](https://docs.ollama.com/linux) pages on November 8, 2026, and they need the hardware they name.

```bash title="commands, not run except llama-server"
# NOT RUN HERE. vLLM quickstart, Linux GPU host, port 8000.
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
vllm serve Qwen/Qwen2.5-1.5B-Instruct

# NOT RUN HERE. SGLang quickstart, GPU host, port 30000.
uv venv --python 3.12 && source .venv/bin/activate
uv pip install --prerelease=allow sglang
sglang serve Qwen/Qwen3-0.6B --host 0.0.0.0 --port 30000

# NOT RUN HERE. Ollama Linux install and quickstart, port 11434.
curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma4:e2b

# RUN HERE on an Apple M2. llama-server, port 9931.
llama-server -hf ggml-org/gemma-3-270m-it-GGUF
```

```bash title="same request, change port and model"
curl http://localhost:9931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma", "messages": [{"role": "user", "content": "Explain the KV cache in one sentence."}], "max_tokens": 60, "temperature": 0}'
```

Ollama's OpenAI routes sit under `http://localhost:11434/v1/`. vLLM and SGLang use `/v1` on their own ports. Running them on a shared host means binding to `0.0.0.0` on purpose, because SGLang, llama-server, and Ollama all default to `127.0.0.1`.

## Published benchmarks are project-run and dated

Most public vLLM vs SGLang numbers were published by one of the two projects, on specific models, at versions that are now old. Treat them as evidence that a design works on one workload, and do not use them to predict your own latency. Newer per-feature numbers appear in release notes.

| Published by | Claim | Setup and date |
| --- | --- | --- |
| [SGLang team](https://www.lmsys.org/blog/2024-07-25-sglang-llama3/) | Up to 3.1x vLLM's throughput on Llama 70B | vLLM 0.5.2, A100 and H100, July 25, 2024 |
| [SGLang authors](https://arxiv.org/abs/2312.07104) | Up to 6.4x higher throughput than other systems | Paper, multiple structured tasks, 2024 |
| [vLLM team](https://vllm.ai/blog/2024-09-05-perf-update) | v0.6.0 at 2.7x the throughput of v0.5.3 on Llama 3 8B | One H100, September 5, 2024 |
| [vLLM v0.33.0 notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3) | 1,373 to 3,390 output tok/s for one drafter change | DeepSeek-V4.1-Flash plus DSpark on GB300, the PR's benchmark |
| [SGLang v0.5.23 notes](https://github.com/sgl-project/sglang/releases/tag/v0.5.23) | Up to 44% faster single-user decode | DeepSeek-V4.1-Flash on GB300, opt-in kernels |

SemiAnalysis's [InferenceX](https://github.com/SemiAnalysisAI/InferenceX) is the outside check to read next. It publishes continuous benchmark runs covering both engines, with code and data in the open. This article did not extract a head-to-head number from it, so filter by your GPU and model yourself.

Both projects keep adding [speculative decoding](https://aifordevs.org/articles/speculative-decoding) methods and hardware-specific kernels, as the release notes above show, so rerun any comparison when you upgrade.

## Choose your inference engine with one replay test

Run one replay test before you commit to any inference engine, and let your own cache hit rate and concurrency pick the winner. A sample of real prompts answers the vLLM vs SGLang question in an afternoon, and it tells you whether a local daemon was ever going to be enough.

1. Capture 1,000 real requests, with prompts and output lengths, from production or a staging copy.
2. Compute the share of prompt tokens that already appeared earlier in the sample. That approximates your prefix hit rate.
3. On one GPU, serve the same model at the same dtype on vLLM v0.33.0 and SGLang v0.5.23, and replay at your target concurrency.
4. Record [time to first token](https://aifordevs.org/glossary#ttft), time per output token, and tokens per second at your p95 target, and read each engine's cache metrics.

As a starting rule, not a measured threshold: when more than half of your prompt tokens repeat, try SGLang first. Otherwise start with vLLM. For one machine and one user, use llama-server if you need slots and grammars, and Ollama if you do not.

Vector stores make the same trade between convenience and control, which the [vector database guide](https://aifordevs.org/articles/what-is-a-vector-database) works through. New engine releases and benchmark changes land in the [news feed](https://aifordevs.org/news) and on the [inference topic page](https://aifordevs.org/topics/inference).

## Questions

### Is SGLang faster than vLLM?

SGLang's own 2024 benchmark reports up to 3.1 times vLLM's throughput on Llama 70B, against vLLM 0.5.2. Those versions are more than two years old, so the number says little about v0.33.0 and v0.5.23. Replay your own traffic on both, and check SemiAnalysis's InferenceX for current runs by an outside party.

### Is vLLM better than Ollama?

vLLM is better for a shared GPU server, and Ollama is better for one developer on one machine. vLLM batches concurrent requests on Linux GPUs. Ollama pulls and runs models with one command, handles one request per model by default, and queues up to 512 more, so it degrades under concurrent load unless you raise OLLAMA_NUM_PARALLEL.

### Does Ollama use llama.cpp?

Yes, for GGUF models. Ollama's June 5, 2026 post says version 0.30 improved GGUF support through llama.cpp, and that this adds to its MLX engine on Apple silicon. Ollama adds a model library, a daemon, and automatic loading and unloading on top. If you need slot counts or grammars, run llama-server directly.

### Can vLLM run GGUF models?

vLLM can load GGUF, but its docs call that support highly experimental and under-optimized, and it now needs the separate vllm-gguf-plugin package. Use GGUF on vLLM to reduce memory in a pinch, and pass the base model's tokenizer as the docs recommend. For normal vLLM serving, use Hugging Face weights instead.

### Does vLLM run on a Mac?

Only in limited ways. vLLM's GPU install page says Linux is the only supported OS and points Apple silicon users to the vLLM-Metal plugin, which uses MLX and Metal. Its CPU page calls native macOS support experimental and source-only, with no prebuilt Apple silicon wheels. For local work on a Mac, llama.cpp and Ollama are simpler.

### What port does llama-server use by default?

llama-server listens on 127.0.0.1 port 9931 by default, per its server README and the b12202 build tested for this article. Older guides and Docker examples that pass --port 8080 are setting that value explicitly. If your client gets connection refused, check the port first, or start the server with your own --port value.

## Sources

1. [vLLM on GitHub](https://github.com/vllm-project/vllm)
2. [vLLM v0.33.0 release notes (attached to tag v0.33.0rc3)](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3)
3. [vLLM docs, Quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart/)
4. [vLLM docs, GPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/)
5. [vLLM docs, CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
6. [vLLM docs, Automatic Prefix Caching](https://docs.vllm.ai/en/latest/design/prefix_caching/)
7. [vLLM docs, Metrics](https://docs.vllm.ai/en/latest/design/metrics/)
8. [vLLM docs, Structured Outputs](https://docs.vllm.ai/en/latest/features/structured_outputs/)
9. [vLLM docs, GGUF](https://docs.vllm.ai/en/latest/features/quantization/gguf/)
10. [vLLM docs, online serving routes](https://docs.vllm.ai/en/latest/serving/online_serving/)
11. [vLLM CacheConfig source at tag v0.33.0rc3](https://github.com/vllm-project/vllm/blob/v0.33.0rc3/vllm/config/cache.py)
12. [vLLM pull request 63231, change default block size from 16 to 64](https://github.com/vllm-project/vllm/pull/63231)
13. [vLLM team, performance update for v0.6.0 (2024)](https://vllm.ai/blog/2024-09-05-perf-update)
14. [Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)](https://arxiv.org/abs/2309.06180)
15. [SGLang on GitHub](https://github.com/sgl-project/sglang)
16. [SGLang v0.5.23 release notes](https://github.com/sgl-project/sglang/releases/tag/v0.5.23)
17. [SGLang docs, Quickstart](https://docs.sglang.io/docs/get-started/quickstart)
18. [SGLang docs, Server Arguments](https://docs.sglang.io/docs/advanced_features/server_arguments)
19. [SGLang docs, Structured Outputs](https://docs.sglang.io/docs/advanced_features/structured_outputs)
20. [SGLang docs, Radix Cache Eviction Policies](https://docs.sglang.io/docs/advanced_features/radix_eviction_policy)
21. [SGLang docs, Apple Silicon with Metal](https://docs.sglang.io/docs/hardware-platforms/apple_metal)
22. [Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (2024)](https://arxiv.org/abs/2312.07104)
23. [SGLang team, fast and expressive LLM inference with RadixAttention (2024)](https://www.lmsys.org/blog/2024-01-17-sglang/)
24. [SGLang team, faster open-source Llama 3 serving (2024)](https://www.lmsys.org/blog/2024-07-25-sglang-llama3/)
25. [SGLang team, SGLang v0.4 (2024)](https://www.lmsys.org/blog/2024-12-04-sglang-v0-4/)
26. [llama.cpp on GitHub](https://github.com/ggml-org/llama.cpp)
27. [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
28. [llama.cpp install docs](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md)
29. [llama.cpp GBNF grammar docs](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md)
30. [Ollama FAQ](https://docs.ollama.com/faq)
31. [Ollama docs, OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility)
32. [Ollama docs, Hardware support](https://docs.ollama.com/gpu)
33. [Ollama docs, Structured outputs](https://docs.ollama.com/capabilities/structured-outputs)
34. [Ollama docs, Importing a model](https://docs.ollama.com/import)
35. [Ollama docs, Linux install](https://docs.ollama.com/linux)
36. [Ollama docs, Quickstart](https://docs.ollama.com/quickstart)
37. [Ollama, MLX on Apple silicon in preview (March 30, 2026)](https://ollama.com/blog/mlx)
38. [Ollama, improved performance and model support with GGUF (June 5, 2026)](https://ollama.com/blog/improved-performance-and-model-support-with-gguf)
39. [Ollama on GitHub](https://github.com/ollama/ollama)
40. [SemiAnalysis, InferenceX benchmark repository](https://github.com/SemiAnalysisAI/InferenceX)

## Cite this deep dive

AI for Devs. "vLLM vs SGLang, plus Ollama and llama.cpp compared." Published November 6, 2026, updated November 8, 2026. https://aifordevs.org/articles/vllm-sglang-llama-cpp

- Page: https://aifordevs.org/articles/vllm-sglang-llama-cpp
- Markdown: https://aifordevs.org/articles/vllm-sglang-llama-cpp.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-06
- Updated: 2026-11-08

---
title: "What is an MCP server? Build and test one in TypeScript"
type: "guide"
url: "https://aifordevs.org/articles/build-an-mcp-server"
markdown: "https://aifordevs.org/articles/build-an-mcp-server.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/build-an-mcp-server"
description: "What is an MCP server? Build a tested TypeScript one with a confirmed write tool, debug it with MCP Inspector, and apply the spec's security requirements."
author: "AI for Devs"
published: "2026-11-06"
updated: "2026-11-08"
level: "Intermediate"
topics: ["mcp", "agents", "coding-agents"]
---

# What is an MCP server? Build and test one in TypeScript

What is an MCP server? Build a tested TypeScript one with a confirmed write tool, debug it with MCP Inspector, and apply the spec's security requirements.

> **TL;DR**
>
> - What is an MCP server? It is a program that exposes tools, resources, and prompts to AI applications over the Model Context Protocol, which uses JSON-RPC 2.0 messages.
> - The current protocol revision is 2026-07-28, as of November 8, 2026. It removed the `initialize` handshake and protocol sessions, so the code here uses the v2 TypeScript SDK packages.
> - This guide builds a notes server with a search tool, a resource, and a write tool that waits for a user's approval through the host.
> - Every code block ran on November 8, 2026 with Node.js 26.12.1, `@modelcontextprotocol/server` 2.4.0, `@modelcontextprotocol/client` 2.4.0, `zod` 4.6.7, `tsx` 4.23.15, and MCP Inspector 2.14.0.
> - `claude mcp add` and its connection check ran against a throwaway config on Claude Code 2.1.318. A live Claude Code session with a model did not run.

An MCP server gives AI applications a standard way to call your code and read your data. If you searched for what is an MCP server, the short answer is a program that speaks the [Model Context Protocol](https://aifordevs.org/glossary#mcp) so that any compatible host can use it. This guide answers that question, builds one server from read-only to a confirmed write, and finishes with an Inspector workflow for CI and a security checklist taken from the spec.

## What is an MCP server

An [MCP server](https://aifordevs.org/glossary#mcp-server) is a program that exposes tools, resources, and prompts to AI applications over the Model Context Protocol. It runs on your machine or a remote host and answers JSON-RPC 2.0 requests. The model sits behind the host, and the server is one of the things that host can reach.

Anthropic [introduced MCP in November 2024](https://www.anthropic.com/news/model-context-protocol) as an open standard for connecting AI applications to data and tools. To answer what is an MCP server precisely, the [architecture overview](https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture) names three participants. A host is the AI application, a client is the connector the host creates for one server, and a server is the program that provides context.

**Figure 1.** One host runs one client per server. Each server decides what it exposes, and stdio or Streamable HTTP carries the same JSON-RPC messages.

A server can offer three primitives. The [server overview](https://modelcontextprotocol.io/specification/2026-07-28/server) defines who controls each one, and that column decides how a host treats them.

| Primitive | Controlled by | What it is | In this guide |
| --- | --- | --- | --- |
| Tool | The model | A function exposed to the model, with typed arguments | `search_notes`, `append_note` |
| Resource | The application | Data the client reads and attaches as context | `notes://{name}` |
| Prompt | The user | A template a person picks, often as a slash command | Not used |

The model side of this picture is the [agent loop](https://aifordevs.org/articles/what-is-an-ai-agent), which turns a [tool call](https://aifordevs.org/glossary#tool-calling) from the model into a request to your server. Tool definitions and results also take up room in the [context window](https://aifordevs.org/glossary#context-window), which is why [context engineering](https://aifordevs.org/articles/context-engineering) applies to server design.

## How an MCP server differs from an API

An MCP server is a layer for AI hosts, and it usually calls an API you already run. A REST API serves developers who read documentation and write client code. An MCP server describes each operation in a schema that a host fetches at runtime, then returns results as content for the model.

| Question | REST API | MCP server |
| --- | --- | --- |
| Who consumes it | Code a developer wrote | A host application and its model |
| How features are found | Docs or an OpenAPI file, at build time | `tools/list`, at runtime |
| How a caller reads a failure | HTTP status and body | `isError: true` with text for the model |
| Who approves a side effect | Your application | The host, with a person who can deny the call |

Skip the server if you have one application, one model, and one integration. A plain [function tool](https://aifordevs.org/glossary#tool-calling) in your own code is smaller and easier to debug. Build an MCP server when two or more hosts need the same capability, or when people outside your team will connect their own agents.

## What changed in revision 2026-07-28

The current MCP revision is 2026-07-28, and it made the protocol stateless. Each request now carries its protocol version and client capabilities in `_meta`, the `initialize` handshake is gone, and every server must answer `server/discover`. A server that follows the older revisions needs code changes.

The [changelog](https://modelcontextprotocol.io/specification/2026-07-28/changelog) lists the details. These changes affect how you write a server:

- Protocol sessions and `Mcp-Session-Id` are gone, so state that spans calls travels in explicit handles passed as tool arguments.
- Servers no longer send requests to clients. A handler that needs input returns an `input_required` result, and the client retries the call with the answer.
- HTTP+SSE, Roots, Sampling, and Logging are marked deprecated, which means new servers should avoid them.

The TypeScript SDK split to match. The older `@modelcontextprotocol/sdk` package, at 1.32.2 on November 8, 2026, tops out at revision 2025-11-25 in its own constants.

The [v2 packages](https://ts.sdk.modelcontextprotocol.io/v2/) implement 2026-07-28. The [SDK README](https://raw.githubusercontent.com/modelcontextprotocol/typescript-sdk/main/README.md) says v1 gets fixes for at least six months after the July 27, 2026 release of v2. The first version of this guide used v1, so its imports changed in this update.

Older clients keep working. `serveStdio` [serves both protocol eras](https://ts.sdk.modelcontextprotocol.io/v2/protocol-versions.html) by default, and the handler below returned the same results to clients on 2025-11-25 and 2026-07-28.

## How to build an MCP server in TypeScript

To build an MCP server, install the SDK's server package, register each tool with a zod schema, and hand a factory function to a transport. The steps below produce a stdio server that searches a folder of Markdown notes. The SDK needs Node.js 20 or later, and this run used 26.12.1.

Create the project:

```bash
mkdir notes-mcp && cd notes-mcp
npm init -y
npm pkg set type=module
npm install @modelcontextprotocol/server @modelcontextprotocol/client zod
npm install -D tsx
mkdir notes
printf '# KV cache\n\nThe KV cache stores attention keys and values.\n' > notes/kv-cache.md
```

Save the server as `server.ts`:

```ts title="server.ts"
import { constants } from "node:fs";
import { readdir, readFile } from "node:fs/promises";
import path from "node:path";
import { McpServer, ResourceNotFoundError, ResourceTemplate } from "@modelcontextprotocol/server";
import { serveStdio } from "@modelcontextprotocol/server/stdio";
import * as z from "zod/v4";

const NOTES_DIR = path.resolve(process.env.NOTES_DIR ?? "./notes");
const READ_FLAGS = constants.O_RDONLY | constants.O_NOFOLLOW;

// One folder, Markdown files only, no subpaths.
function notePath(name: string): string {
  const file = path.resolve(NOTES_DIR, name);
  if (path.dirname(file) !== NOTES_DIR || path.extname(file) !== ".md") {
    throw new Error(`Invalid note name "${name}"`);
  }
  return file;
}

function createServer(): McpServer {
  const server = new McpServer({ name: "notes", version: "1.0.0" });

  server.registerTool(
    "search_notes",
    {
      title: "Search notes",
      description: "Find Markdown notes that contain a phrase. Returns matching file names, one per line.",
      inputSchema: z.object({ query: z.string().min(2).describe("Phrase to find, case-insensitive") }),
      annotations: { readOnlyHint: true, openWorldHint: false },
    },
    async ({ query }) => {
      const entries = await readdir(NOTES_DIR, { withFileTypes: true });
      const files = entries.filter((e) => e.isFile() && e.name.endsWith(".md")).map((e) => e.name).sort();
      const hits: string[] = [];
      for (const file of files) {
        const text = await readFile(notePath(file), { encoding: "utf8", flag: READ_FLAGS });
        if (text.toLowerCase().includes(query.toLowerCase())) hits.push(file);
      }
      return { content: [{ type: "text", text: hits.length ? hits.join("\n") : "No matches." }] };
    },
  );

  server.registerResource(
    "note",
    new ResourceTemplate("notes://{name}", { list: undefined }),
    { title: "Note", description: "One note as Markdown", mimeType: "text/markdown" },
    async (uri, { name }) => {
      try {
        const text = await readFile(notePath(String(name)), { encoding: "utf8", flag: READ_FLAGS });
        return { contents: [{ uri: uri.href, mimeType: "text/markdown", text }] };
      } catch {
        throw new ResourceNotFoundError(uri.href);
      }
    },
  );

  return server;
}

void serveStdio(createServer);
console.error("notes MCP server running on stdio");
```

From the one zod schema, the SDK derives the JSON Schema that hosts see, validates arguments before your handler runs, and types the handler. The `describe()` text is the only documentation the model gets for that argument, so write it for a reader who has nothing else.

`notePath` confines every read to one folder, and `O_NOFOLLOW` makes the open fail on a symbolic link. The catch block turns any read failure into a standard not-found error, so absolute paths never reach the model.

## Test it with a client script

Test the server with the SDK's client before you involve a host, because a script can assert exact results and needs no API key. The client below connects over stdio, negotiates the newest protocol era, and calls the tool with good and bad arguments. Save it as `client.ts`:

```ts title="client.ts"
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";

const client = new Client({ name: "smoke-test", version: "1.0.0" }, { versionNegotiation: { mode: "auto" } });
await client.connect(new StdioClientTransport({ command: "npx", args: ["tsx", "server.ts"] }));
console.log("era:", client.getProtocolEra());

const { tools } = await client.listTools();
console.log("tools:", tools.map((t) => t.name));

const call = async (name: string, args: Record<string, unknown>) => {
  const { content, isError } = await client.callTool({ name, arguments: args });
  console.log(name, JSON.stringify(args), "=>", JSON.stringify({ content, isError }));
};

await call("search_notes", { query: "kv" });
await call("search_notes", { query: "k" });

const note = await client.readResource({ uri: "notes://kv-cache.md" });
console.log("resource:", JSON.stringify(note.contents[0].text));
await client.readResource({ uri: "notes://..%2F..%2Fetc%2Fpasswd" }).catch((e) => console.log("rejected:", e.code, e.message));

await client.close();
```

Run it with `npx tsx client.ts`. The server's startup line on stderr is omitted here:

```text
era: modern
tools: [ 'search_notes' ]
search_notes {"query":"kv"} => {"content":[{"type":"text","text":"kv-cache.md"}]}
search_notes {"query":"k"} => {"content":[{"type":"text","text":"Input validation error: Invalid arguments for tool search_notes: query: Too small: expected string to have >=2 characters"}],"isError":true}
resource: "# KV cache\n\nThe KV cache stores attention keys and values.\n"
rejected: -32602 Resource not found: notes://..%2F..%2Fetc%2Fpasswd
```

The second call breaks the `min(2)` rule, and the SDK answers with a tool result marked `isError: true`. The [tools page of the spec](https://modelcontextprotocol.io/specification/2026-07-28/server/tools) classes input validation failures as tool execution errors, which clients should pass to the model so the next call can correct the arguments. An unknown tool name is a protocol error instead, and it came back as a rejected call with code `-32602`.

MCP over stdio is newline-delimited JSON, so you can talk to the server with no SDK at all:

```bash
echo '{"jsonrpc":"2.0","id":1,"method":"server/discover","params":{"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientCapabilities":{}}}}' | npx tsx server.ts 2>/dev/null
```

```json
{"result":{"supportedVersions":["2026-07-28"],"capabilities":{"tools":{"listChanged":true},"resources":{"listChanged":true}},"resultType":"complete","ttlMs":0,"cacheScope":"private","_meta":{"io.modelcontextprotocol/serverInfo":{"name":"notes","version":"1.0.0"}}},"jsonrpc":"2.0","id":1}
```

## Add a write tool with confirmation

A write tool should ask a person through the host before it changes anything. In revision 2026-07-28, the server returns an `input_required` result that names the question, the host collects the answer, and the client retries the same call with the answer attached. The file stays untouched until that retry arrives.

A `confirm` argument on the tool would not do this job. Model output supplies the arguments, so the flag would be model output approving itself. [Elicitation](https://modelcontextprotocol.io/specification/2026-07-28/client/elicitation) routes the question to the user through the client, outside the model's output.

Replace the two matching imports at the top of `server.ts` with these:

```ts title="server.ts"
import { appendFile, readdir, readFile } from "node:fs/promises";
import {
  acceptedContent,
  inputRequired,
  inputResponse,
  McpServer,
  ResourceNotFoundError,
  ResourceTemplate,
  type CallToolResult,
  type InputRequiredResult,
} from "@modelcontextprotocol/server";
```

Add this block above `createServer`, then add `registerAppendNote(server);` just before `return server;`:

```ts title="server.ts"
const APPEND_FLAGS = constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | constants.O_NOFOLLOW;
const noteName = z.string().regex(/^[\w-]+\.md$/).describe("Note file name, such as kv-cache.md");
const approval = z.object({ confirm: z.boolean().meta({ title: "Write this text to the note" }) });

// At most 10 writes per minute for this process.
const recentWrites: number[] = [];
function underWriteLimit(now = Date.now()): boolean {
  while (recentWrites.length && recentWrites[0] < now - 60_000) recentWrites.shift();
  if (recentWrites.length >= 10) return false;
  recentWrites.push(now);
  return true;
}

function registerAppendNote(server: McpServer) {
  server.registerTool(
    "append_note",
    {
      title: "Append to a note",
      description: "Append text to one note after the user approves the exact change. Creates the note if it is missing.",
      inputSchema: z.object({ name: noteName, text: z.string().min(1).max(2000) }),
      annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: false },
    },
    async ({ name, text }, ctx): Promise<CallToolResult | InputRequiredResult> => {
      const answer = inputResponse(ctx.mcpReq.inputResponses, "confirm");
      if (answer.kind === "elicit" && answer.action !== "accept") {
        return { content: [{ type: "text", text: `Nothing written: the user chose ${answer.action}.` }] };
      }
      const approved = acceptedContent(ctx.mcpReq.inputResponses, "confirm", approval);
      if (approved === undefined) {
        return inputRequired({
          inputRequests: {
            confirm: inputRequired.elicit({ message: `Append to ${name}?\n\n${text}`, requestedSchema: approval }),
          },
        });
      }
      if (!approved.confirm) {
        return { content: [{ type: "text", text: "Nothing written: the box was left unchecked." }] };
      }
      if (!underWriteLimit()) {
        return { content: [{ type: "text", text: "Write limit reached. Try again in a minute." }], isError: true };
      }
      try {
        await appendFile(notePath(name), `\n${text}\n`, { flag: APPEND_FLAGS });
      } catch (err) {
        console.error(err);
        return { content: [{ type: "text", text: `Could not write ${name}.` }], isError: true };
      }
      return { content: [{ type: "text", text: `Appended ${text.length} characters to ${name}.` }] };
    },
  );
}
```

The handler runs once per round. On the first entry there is no answer, so it returns the question.

On the retry, `acceptedContent` validates the answer against the same schema, and only then does the handler write. The [SDK's input-required guide](https://ts.sdk.modelcontextprotocol.io/v2/servers/input-required.html) calls these responses untrusted for that reason.

**Figure 2.** On a 2026-07-28 connection, approval is a round trip between the host and the server. The conversation with the model holds only the tool call and the final result.

Test both answers with a script that stands in for the host's dialog. Save it as `write-test.ts`:

```ts title="write-test.ts"
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";

const reply = process.argv[2]; // "accept" or "decline"
const client = new Client(
  { name: "write-test", version: "1.0.0" },
  { versionNegotiation: { mode: "auto" }, capabilities: { elicitation: { form: {} } } },
);

// Stands in for the host's approval dialog.
client.setRequestHandler("elicitation/create", async (request) => {
  console.log("dialog:", JSON.stringify(request.params.message));
  return reply === "accept" ? { action: "accept", content: { confirm: true } } : { action: "decline" };
});

await client.connect(new StdioClientTransport({ command: "npx", args: ["tsx", "server.ts"] }));
const { content, isError } = await client.callTool({
  name: "append_note",
  arguments: { name: "kv-cache.md", text: "Reviewed on November 8." },
});
console.log(JSON.stringify({ content, isError }));
await client.close();
```

Output of `npx tsx write-test.ts accept`, then `npx tsx write-test.ts decline`, with the server's stderr line omitted:

```text
dialog: "Append to kv-cache.md?\n\nReviewed on November 8."
{"content":[{"type":"text","text":"Appended 23 characters to kv-cache.md."}]}
dialog: "Append to kv-cache.md?\n\nReviewed on November 8."
{"content":[{"type":"text","text":"Nothing written: the user chose decline."}]}
```

The other paths behaved like this in the same run:

| Case | Result |
| --- | --- |
| Box left unchecked | `Nothing written: the box was left unchecked.` |
| Client without elicitation, 2026-07-28 | Rejected call, code `-32021`, nothing written |
| Client without elicitation, 2025-11-25 | `isError: true`, nothing written |
| `name` set to `../secrets.md` | Schema rejects it before the handler runs |
| Note is a symbolic link | `Could not write link.md.` |
| Over 10 write attempts in a minute | `Write limit reached. Try again in a minute.` |

This check keeps the approval out of model output, and it cannot prove a person answered. A raw `tools/call` that arrived with `inputResponses` already attached wrote the file on the first try, because the server keeps no state between rounds.

[Claude Code](https://code.claude.com/docs/en/mcp) also lets users auto-answer these requests with an `Elicitation` hook. Keep your real limits in the server: the folder boundary, the schema, and the write cap.

## Debug with MCP Inspector

MCP Inspector is the project's [reference tool](https://modelcontextprotocol.io/docs/2026-07-28/tools/inspector) for testing MCP servers without a model or an API key. It connects to your server, lets you run each tool by hand, and shows the JSON-RPC traffic. It has a browser UI, a terminal UI, and a CLI.

Version 2.14.0 requires Node.js 22.23.4, 24.22.1, or 26.12.1 and later on those lines.

Start the browser UI from the project folder:

```bash
npx @modelcontextprotocol/inspector npx tsx server.ts
```

The launch banner prints a local URL with a per-launch API token. Leave that token on, since `DANGEROUSLY_OMIT_AUTH` disables it on a tool that can start processes.

In the page, flip the toggle on the server card until it reads Connected. The card also shows the negotiated revision, which was `MCP 2025-11-25` by default in 2.14.0.

Open the Tools tab, pick `append_note`, fill in the form, and press Execute Tool. An Elicitation Request dialog then offers Cancel, Decline, and Submit, and the side panel lists `tools/call` and `elicitation/create` with their timings.

The CLI suits CI because a failed tool sets the exit code. To test the 2026-07-28 path, add `--protocol-era modern`:

```bash
npx @modelcontextprotocol/inspector --cli npx tsx server.ts --method tools/call --tool-name search_notes --tool-arg query=k
echo "exit: $?"
```

```text
{
  "content": [
    {
      "type": "text",
      "text": "Input validation error: Invalid arguments for tool search_notes: query: Too small: expected string to have >=2 characters"
    }
  ],
  "isError": true
}
exit: 5
```

[Exit code 5](https://modelcontextprotocol.io/docs/2026-07-28/tools/inspector/cli) means the tool returned `isError: true` or was not found, and the CLI also writes a one-line error envelope to stderr, omitted above. Pin an exact Inspector version in CI so a new release cannot change a build.

If a server misbehaves, check stdout first. The [stdio transport](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/stdio) forbids anything on stdout except valid MCP messages.

In this test, a plain `console.log` line was skipped by the 2.4.0 client and `claude mcp list` still reported Connected. A JSON-shaped log line reached the client's `onerror` as a schema failure. Other clients can be stricter, so send logs to stderr with `console.error`.

## Connect it to Claude Code

Register the server with `claude mcp add` and use absolute paths, because the host starts the process itself. The [Claude Code MCP docs](https://code.claude.com/docs/en/mcp) moved to code.claude.com, and the old docs.anthropic.com address now redirects there. Everything after `--` is the launch command, and `--env` sets variables for the server process.

```bash
claude mcp add --env NOTES_DIR="$PWD/notes" --transport stdio notes -- "$PWD/node_modules/.bin/tsx" "$PWD/server.ts"
claude mcp list
```

Output, with the temporary project path shortened:

```text
Added stdio MCP server notes with command: /path/to/notes-mcp/node_modules/.bin/tsx /path/to/notes-mcp/server.ts to local config
Checking MCP server health…

notes: /path/to/notes-mcp/node_modules/.bin/tsx /path/to/notes-mcp/server.ts - ✔ Connected
```

This ran on Claude Code 2.1.318 with temporary `HOME` and `CLAUDE_CONFIG_DIR` folders, so no real configuration changed. The health check started the server and connected. The `/mcp` panel, the approval dialog in the terminal, and a model calling these tools were not run, because no API key was available.

The default scope is local, stored in `~/.claude.json` for that one project. Use `--scope project` to share a server through `.mcp.json`, or `--scope user` for every project. Claude Code warns when tool output exceeds 10,000 tokens and limits it to 25,000 by default, as of November 8, 2026.

Other hosts take the same command, arguments, and environment in their own settings. The [MCP introduction](https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro) names several, and the [platforms directory](https://aifordevs.org/platforms) lists more [coding agents](https://aifordevs.org/topics/coding-agents).

## MCP server security: what the spec requires

MCP gives a model a path to run code, so the [spec](https://modelcontextprotocol.io/specification/2026-07-28) puts consent, input checks, and access limits on implementers. Its tools page says servers must validate inputs, apply access controls, rate limit calls, and sanitize outputs. Hosts should show inputs and ask a person to confirm sensitive operations.

| Spec requirement for servers | In the notes server | Still your job |
| --- | --- | --- |
| Validate all tool inputs | zod schemas and `notePath` | Rules a schema cannot express |
| Implement access controls | One folder, no symbolic links | The process keeps your user's permissions |
| Rate limit tool invocations | 10 writes a minute | Limit reads that cost money |
| Sanitize tool outputs | File names only, no absolute paths in errors | Note text can still carry instructions |

Treat everything a tool returns as untrusted data. Anthropic's [tool-use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls) warn that tool results can carry indirect prompt injection, and a note that says "ignore your instructions" reaches the model as text. The spec also says clients must treat tool annotations as untrusted unless they come from a trusted server, so `readOnlyHint` is a label for the host and enforces nothing.

### Local servers

A local server runs with the permissions of whoever starts it. The [security best practices](https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices) describe startup commands that exfiltrate SSH keys, and they require one-click installers to show the full command for approval. Read the command before you add any server, and prefer stdio over a local HTTP port so other processes cannot reach it.

### Remote servers

Authorization is optional in the protocol, but a remote server holding user data needs it. The [authorization page](https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization) builds on OAuth 2.1 for HTTP transports, and it tells stdio servers to read credentials from the environment instead. Check these before you expose a server:

- Publish OAuth 2.0 Protected Resource Metadata (RFC 9728), and answer unauthenticated requests with `401` and a `WWW-Authenticate` header.
- Validate that each access token was issued for your server. Clients must send the RFC 8707 `resource` parameter to make that possible.
- Never forward a client's token to another API. The spec forbids this token passthrough.
- Prefer Client ID Metadata Documents for client registration, since Dynamic Client Registration is deprecated in this revision.
- Validate the `Origin` header and return `403` if it is invalid, as the [Streamable HTTP page](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/streamable-http) requires, and bind local servers to 127.0.0.1.
- Never treat a state handle as proof of identity, and require per-client consent if your server proxies a third-party API.

To serve the same tools over HTTP, pass the factory to [`createMcpHandler`](https://ts.sdk.modelcontextprotocol.io/v2/serving/http.html). This guide did not run an HTTP deployment or an OAuth flow, so treat the list above as the spec's requirements and test yours against them.

## Ship your first MCP server

Start with one read-only tool over data you already own, and add writes only after a person has used it for a week. Once you can explain what is an MCP server doing in your stack, you can decide what else belongs in it. A narrow first server is easier to secure and cheaper in context.

Use this order for the first release:

1. Write one read-only tool with a schema and a description aimed at the model.
2. Add an Inspector CLI call with a pinned version to CI, and fail the build on a nonzero exit code.
3. Register the server in one host and log the arguments in each tool call.
4. Add one write tool with elicitation, a rate limit, and a hard boundary on what it can touch.

Before you connect a server to an agent that runs unattended, read how an [AI agent loop fails](https://aifordevs.org/articles/what-is-an-ai-agent) and set its limits first. Then measure the result with [LLM evaluation](https://aifordevs.org/articles/llm-evaluation), because tool descriptions change model behavior as much as prompts do. The [MCP topic hub](https://aifordevs.org/topics/mcp) and the [agents hub](https://aifordevs.org/topics/agents) collect related guides, and [`/news`](https://aifordevs.org/news) tracks protocol and SDK releases.

## Questions

### What is an MCP server in simple terms?

An MCP server is a small program that makes your tools and data available to AI applications through one standard protocol. A coding agent or chat app connects to it, asks what it offers, and calls those functions when needed. You write the server once, and any compatible host can use it.

### What is the difference between an API and an MCP server?

An API is written for developers who read its documentation and write client code. An MCP server is written for AI hosts, which fetch its tool list and JSON schemas at runtime. Most MCP servers call an existing API internally, so the two work together and the server adds a discoverable layer.

### Should my MCP server use stdio or HTTP?

Use stdio when the host launches the server on the same machine, such as a tool for local files or a command-line program. Use Streamable HTTP when the server runs remotely or serves many users. A remote server needs authentication, Origin header validation, and token checks that a local stdio server does not.

### What is the difference between an MCP tool and a resource?

A tool is a function exposed to the model with typed arguments, such as searching notes or creating a ticket. A resource is data addressed by a URI that the client application reads and attaches as context. The MCP specification calls tools model-controlled and resources application-controlled, so the host treats them differently.

### Do MCP servers need authentication?

Authorization is optional in the MCP specification, and a local stdio server should read credentials from its environment. A remote HTTP server that handles user data should implement the OAuth-based flow, publish Protected Resource Metadata, validate that each token was issued for that server, and never pass a client's token to another API.

### Are MCP servers safe to install?

A local MCP server runs with the permissions of the user who starts it, so treat it like any program you install. The specification warns that a startup command can run arbitrary code and that tool descriptions are untrusted unless the server is trusted. Read the exact command before you approve it.

### What is MCP Inspector used for?

MCP Inspector is the project's reference tool for testing MCP servers without a model or an API key. It connects to a server and lets you list tools, run them, read resources, and watch the JSON-RPC messages. It has a browser UI, a terminal UI, and a CLI with exit codes that suit CI jobs.

## Sources

1. [Model Context Protocol, specification overview, revision 2026-07-28](https://modelcontextprotocol.io/specification/2026-07-28)
2. [Model Context Protocol, key changes in revision 2026-07-28](https://modelcontextprotocol.io/specification/2026-07-28/changelog)
3. [Model Context Protocol, versioning](https://modelcontextprotocol.io/docs/2026-07-28/learn/versioning)
4. [Model Context Protocol, architecture overview](https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture)
5. [Model Context Protocol, server features overview](https://modelcontextprotocol.io/specification/2026-07-28/server)
6. [Model Context Protocol, tools](https://modelcontextprotocol.io/specification/2026-07-28/server/tools)
7. [Model Context Protocol, elicitation](https://modelcontextprotocol.io/specification/2026-07-28/client/elicitation)
8. [Model Context Protocol, stdio transport](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/stdio)
9. [Model Context Protocol, Streamable HTTP transport](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/streamable-http)
10. [Model Context Protocol, authorization](https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization)
11. [Model Context Protocol, security best practices](https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices)
12. [MCP Inspector documentation](https://modelcontextprotocol.io/docs/2026-07-28/tools/inspector)
13. [MCP Inspector CLI documentation, including exit codes](https://modelcontextprotocol.io/docs/2026-07-28/tools/inspector/cli)
14. [MCP TypeScript SDK README, main branch](https://raw.githubusercontent.com/modelcontextprotocol/typescript-sdk/main/README.md)
15. [MCP TypeScript SDK v2 documentation](https://ts.sdk.modelcontextprotocol.io/v2/)
16. [MCP TypeScript SDK v2, input required](https://ts.sdk.modelcontextprotocol.io/v2/servers/input-required.html)
17. [MCP TypeScript SDK v2, protocol versions](https://ts.sdk.modelcontextprotocol.io/v2/protocol-versions.html)
18. [MCP TypeScript SDK v2, serve over HTTP](https://ts.sdk.modelcontextprotocol.io/v2/serving/http.html)
19. [Claude Code docs, connect to tools via MCP](https://code.claude.com/docs/en/mcp)
20. [Anthropic, Introducing the Model Context Protocol (November 2024)](https://www.anthropic.com/news/model-context-protocol)
21. [Anthropic docs, handle tool calls](https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls)

## Cite this guide

AI for Devs. "What is an MCP server? Build and test one in TypeScript." Published November 6, 2026, updated November 8, 2026. https://aifordevs.org/articles/build-an-mcp-server

- Page: https://aifordevs.org/articles/build-an-mcp-server
- Markdown: https://aifordevs.org/articles/build-an-mcp-server.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-06
- Updated: 2026-11-08

---
title: "AI for Devs glossary"
type: "glossary"
url: "https://aifordevs.org/glossary"
markdown: "https://aifordevs.org/glossary.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary"
terms: 54
---

# AI for Devs glossary

Definitions of LLM and AI engineering terms. Each entry gives a short definition, how it works, a worked example, and a primary source.

## A2A (Agent2Agent protocol)

Page: https://aifordevs.org/glossary#a2a | Markdown: https://aifordevs.org/glossary/a2a.md

A2A (Agent2Agent) is an open protocol, introduced by Google in April 2025, that lets agents built by different vendors discover each other, exchange tasks, and stream results. MCP connects an agent to tools and data, while A2A connects agents to other agents.

A remote agent publishes a JSON Agent Card at /.well-known/agent-card.json listing its skills, endpoint, and auth requirements. Work runs as a task with a defined lifecycle, so a long job can pause for input and report status. As of November 8, 2026, the spec is at version 1.0.0 with JSON-RPC, gRPC, and HTTP+JSON bindings, and A2A is a Growth Stage project at the Agentic AI Foundation.

### Example

A client fetches https://example.com/.well-known/agent-card.json, finds the skill it needs, and sends a message. The task that message starts moves from SUBMITTED to WORKING to COMPLETED.

### Source

[A2A Protocol specification](https://a2a-protocol.org/latest/specification/)

### Related terms

- [MCP](https://aifordevs.org/glossary#mcp)
- [Agent](https://aifordevs.org/glossary#agent)

## Agent

Page: https://aifordevs.org/glossary#agent | Markdown: https://aifordevs.org/glossary/agent.md

An agent is a program that calls a model in a loop, runs the tools the model requests, and feeds the results back until a stop condition ends the loop. The tools and the stop condition set what the agent can do.

Each pass sends the accumulated history, so cost and latency grow with every turn. Anthropic's guide describes agents as LLMs using tools based on environmental feedback in a loop, and it recommends a stopping condition such as a maximum iteration count. The same guide advises finding the simplest solution possible and adding complexity only when needed.

### Example

Asked to fix failing tests, an agent runs npm test, reads two files, edits one, reruns the tests, and then returns text with no tool call. That run takes four turns.

### Source

[Anthropic, Building effective agents (2024)](https://www.anthropic.com/research/building-effective-agents)

### Related terms

- [Agent loop](https://aifordevs.org/glossary#agent-loop)
- [Agentic AI](https://aifordevs.org/glossary#agentic-ai)
- [Tool calling](https://aifordevs.org/glossary#tool-calling)
- [MCP](https://aifordevs.org/glossary#mcp)
- [A2A](https://aifordevs.org/glossary#a2a)

### Read the article

[What is an AI agent? The loop in code, and where it fails](https://aifordevs.org/articles/what-is-an-ai-agent)

## Agent loop

Page: https://aifordevs.org/glossary#agent-loop | Markdown: https://aifordevs.org/glossary/agent-loop.md

An agent loop is the code that sends the conversation to a model, runs any tool calls in the response, appends the results, and calls the model again. The loop ends when a response contains no tool calls or a limit is reached.

A turn is one pass: model output, then tool execution if the output asked for it. Without a cap, the loop runs until the model returns text with no tool calls, so set a turn limit and a spend limit in production. In the Claude Agent SDK those limits are max_turns and max_budget_usd, and as of November 8, 2026, both default to no limit.

### Example

A bug fix takes four turns: run the tests, read two files, edit and rerun, then a text-only reply. With max_turns=2 the run stops before the edit and returns error_max_turns.

### Source

[Claude Agent SDK docs: How the agent loop works](https://code.claude.com/docs/en/agent-sdk/agent-loop)

### Related terms

- [Agent](https://aifordevs.org/glossary#agent)
- [Agentic AI](https://aifordevs.org/glossary#agentic-ai)
- [Tool calling](https://aifordevs.org/glossary#tool-calling)

### Read the article

[What is an AI agent? The loop in code, and where it fails](https://aifordevs.org/articles/what-is-an-ai-agent)

## Agentic AI

Page: https://aifordevs.org/glossary#agentic-ai | Markdown: https://aifordevs.org/glossary/agentic-ai.md

Agentic AI is software in which a model's output selects the next step, usually a tool call, instead of a fixed script setting the order of steps. The term covers single coding agents as well as multi-agent systems.

Anthropic's guide draws the line at control flow. A workflow runs LLMs and tools through predefined code paths, and an agent lets model output direct the process and its tool use. The guide favors workflows for well-defined tasks because they are more predictable.

### Example

A support bot that always classifies, retrieves, then replies is a workflow. A bot whose model output can request a refund tool, a lookup tool, or an escalation tool at each step is agentic.

### Source

[Anthropic, Building effective agents (2024)](https://www.anthropic.com/research/building-effective-agents)

### Related terms

- [Agent](https://aifordevs.org/glossary#agent)
- [Agent loop](https://aifordevs.org/glossary#agent-loop)

### Read the article

[What is an AI agent? The loop in code, and where it fails](https://aifordevs.org/articles/what-is-an-ai-agent)

## AI gateway

Page: https://aifordevs.org/glossary#ai-gateway | Markdown: https://aifordevs.org/glossary/ai-gateway.md

An AI gateway is a proxy between your application and one or more model providers that gives you one API for many models. It also adds routing, fallbacks, caching, rate limits, cost tracking, and logging in one place.

A gateway sits in the request path, so it adds a network hop and needs its own uptime plan. It pays off once you call more than one provider, need per-team budgets, or want failover without code changes. A gateway's response cache is separate from provider prompt caching, which discounts repeated prefixes on the provider's side.

### Example

When provider A returns a 429, the gateway retries the same request on provider B, and your application code receives one successful response.

### Related terms

- [Prompt caching](https://aifordevs.org/glossary#prompt-caching)
- [Guardrails](https://aifordevs.org/glossary#guardrails)

## Attention

Page: https://aifordevs.org/glossary#attention | Markdown: https://aifordevs.org/glossary/attention.md

Attention is the transformer operation that lets each token weigh the tokens before it when computing its own representation. Its cost grows with sequence length, which is why long contexts are expensive and why the KV cache exists.

Each token gets a query, a key, and a value vector. Scaled dot-product attention computes softmax(QK^T / sqrt(d_k)) V, so queries score keys and the scores weight the values. A model runs this across several heads in every layer.

### Example

A 1,000-token sequence with a causal mask has 500,500 query-key pairs per head, per layer.

### Source

[Vaswani et al., Attention Is All You Need (2017)](https://arxiv.org/abs/1706.03762)

### Related terms

- [Transformer](https://aifordevs.org/glossary#transformer)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [Grouped-query attention](https://aifordevs.org/glossary#grouped-query-attention)

### Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## Batch API

Page: https://aifordevs.org/glossary#batch-api | Markdown: https://aifordevs.org/glossary/batch-api.md

A batch API is an asynchronous endpoint that takes many model requests at once, processes them within a time window, and bills them at a discount. You give up immediate responses in exchange for a lower price.

You submit a list or file of requests, poll for completion, then download results keyed by your own request IDs. As of November 8, 2026, OpenAI, Anthropic, and Google all bill batch work at 50% of standard rates, with a 24-hour window or target. Use it for evals, backfills, and classification jobs where nobody is waiting.

### Example

As of November 8, 2026, Claude Sonnet 5.5 costs $2 input and $10 output per million tokens, or $1 and $5 in a batch. A job with 10 million input and 2 million output tokens drops from $40 to $20.

### Source

[Anthropic docs: Pricing](https://platform.claude.com/docs/en/about-claude/pricing)

### Related terms

- [Prompt caching](https://aifordevs.org/glossary#prompt-caching)
- [Throughput](https://aifordevs.org/glossary#throughput)

### Read the article

[Claude API pricing: caching, batch, and real bills](https://aifordevs.org/articles/claude-api-pricing)

## Chunking

Page: https://aifordevs.org/glossary#chunking | Markdown: https://aifordevs.org/glossary/chunking.md

Chunking is splitting documents into smaller passages before you embed and index them for retrieval. Chunk size sets how much text a single retrieved passage can bring into the prompt.

Small chunks match queries closely but lose surrounding context, and large chunks keep context but blur the embedding. Overlapping neighbors keeps a sentence from being cut at a boundary. Anthropic's Contextual Retrieval post says RAG systems usually chunk to no more than a few hundred tokens, and that size, boundaries, and overlap all affect retrieval.

### Example

A 10,000-token document split into 500-token chunks with 50 tokens of overlap produces 23 chunks, because each new chunk starts 450 tokens after the last.

### Source

[Anthropic, Introducing Contextual Retrieval (2024)](https://www.anthropic.com/research/contextual-retrieval)

### Related terms

- [RAG](https://aifordevs.org/glossary#rag)
- [Embedding](https://aifordevs.org/glossary#embedding)
- [Vector database](https://aifordevs.org/glossary#vector-database)
- [Reranking](https://aifordevs.org/glossary#reranking)
- [Hybrid search](https://aifordevs.org/glossary#hybrid-search)

### Read the article

[What is a vector database? HNSW, IVF, and when to skip one](https://aifordevs.org/articles/what-is-a-vector-database)

## Constrained decoding

Page: https://aifordevs.org/glossary#constrained-decoding | Markdown: https://aifordevs.org/glossary/constrained-decoding.md

Constrained decoding is generation in which tokens that would break a grammar or schema are masked at each step before sampling. The output follows the grammar by construction, unless generation stops early.

Willard and Louf (2023) frame it as walking a finite-state machine: precompute which vocabulary tokens are legal in each state, then mask the rest. The schema has to be compiled first, so the first request with a new schema is slower. Anthropic's docs describe compiling JSON Schemas into grammars and caching them for 24 hours after last use.

### Example

With a field limited to "yes" or "no", every token that cannot extend one of those two strings gets probability zero at that position.

### Source

[Willard and Louf, Efficient Guided Generation for Large Language Models (2023)](https://arxiv.org/abs/2307.09702)

### Related terms

- [Structured outputs](https://aifordevs.org/glossary#structured-outputs)
- [Tool calling](https://aifordevs.org/glossary#tool-calling)

## Context engineering

Page: https://aifordevs.org/glossary#context-engineering | Markdown: https://aifordevs.org/glossary/context-engineering.md

Context engineering is the practice of choosing what goes into a model's context window for each call, and in what order: instructions, examples, retrieved documents, tool results, and memory. You treat the window as a limited budget and spend it on what the next step needs.

Anthropic defines it as curating and maintaining the best set of tokens during inference, which extends prompt engineering to tools, external data, and message history. Its working rule is the smallest set of high-signal tokens that makes the desired outcome likely. Long agent runs need compaction or tool-result clearing to stay inside the window.

### Example

An agent that loads a 20,000-token log and needs three error lines should keep those lines and clear the raw result. Every later call then carries a few hundred tokens instead of 20,000.

### Source

[Anthropic, Effective context engineering for AI agents (2025)](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents)

### Related terms

- [Context window](https://aifordevs.org/glossary#context-window)
- [System prompt](https://aifordevs.org/glossary#system-prompt)
- [RAG](https://aifordevs.org/glossary#rag)
- [Prompt caching](https://aifordevs.org/glossary#prompt-caching)

### Read the article

[Context engineering: budget and order the context window](https://aifordevs.org/articles/context-engineering)

## Context window

Page: https://aifordevs.org/glossary#context-window | Markdown: https://aifordevs.org/glossary/context-window.md

A context window is the maximum number of tokens a model can handle in one request, usually counting the prompt and the output together. Everything the model has to work with for that request has to fit inside it.

On the Claude API, the system prompt, messages, tool definitions, tool results, generated output, and thinking all count, and cached tokens still occupy the window. As of November 8, 2026, Google's docs list separate input and output budgets for Gemini 3 and 2.5, and one shared 1M window for Gemini 4. Anthropic's docs warn that accuracy and recall degrade as token count grows, so a larger window is no reason to fill it.

### Example

As of November 8, 2026, Claude Opus 5.5 has a 1M-token window and can generate up to 128K tokens per request. A 950,000-token prompt leaves 50,000 tokens for output, and generation stops there with model_context_window_exceeded.

### Source

[Anthropic docs: Context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows)

### Related terms

- [Token](https://aifordevs.org/glossary#token)
- [Context engineering](https://aifordevs.org/glossary#context-engineering)
- [Reasoning model](https://aifordevs.org/glossary#reasoning-model)

### Read the article

[Context engineering: budget and order the context window](https://aifordevs.org/articles/context-engineering)

## Continuous batching

Page: https://aifordevs.org/glossary#continuous-batching | Markdown: https://aifordevs.org/glossary/continuous-batching.md

Continuous batching is a serving technique that schedules work one model iteration at a time, so new requests join a running batch and finished ones leave it immediately. Requests stop waiting for the slowest sequence in the batch.

The Orca paper (Yu et al., OSDI 2022) calls this iteration-level scheduling and pairs it with selective batching, which applies batching only to a chosen subset of operations. Static batching holds every slot until the longest sequence finishes, so short requests leave GPU capacity idle.

### Example

Orca reports a 36.9x throughput gain over NVIDIA FasterTransformer at the same latency on GPT-3 175B. That figure comes from the authors' own evaluation.

### Source

[Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)](https://www.usenix.org/conference/osdi22/presentation/yu)

### Related terms

- [Throughput](https://aifordevs.org/glossary#throughput)
- [Inference](https://aifordevs.org/glossary#inference)
- [PagedAttention](https://aifordevs.org/glossary#paged-attention)

### Read the article

[vLLM vs SGLang, plus Ollama and llama.cpp compared](https://aifordevs.org/articles/vllm-sglang-llama-cpp)

## Distillation

Page: https://aifordevs.org/glossary#distillation | Markdown: https://aifordevs.org/glossary/distillation.md

Distillation is training a smaller student model to reproduce the outputs of a larger teacher model. It is a common way to get a cheaper model that keeps much of the teacher's quality on a narrow task.

Hinton et al. (2015) train the student on soft targets: both models divide their logits by a temperature above 1, so the teacher's near-miss answers carry signal, and the student returns to a temperature of 1 after training. DeepSeek's R1 paper reports that reasoning patterns from large models can also improve smaller ones.

### Example

DistilBERT is 40% smaller and 60% faster than BERT, and its authors report that it keeps 97% of BERT's language understanding performance.

### Source

[Hinton, Vinyals, and Dean, Distilling the Knowledge in a Neural Network (2015)](https://arxiv.org/abs/1503.02531)

### Related terms

- [Fine-tuning](https://aifordevs.org/glossary#fine-tuning)
- [Small language model](https://aifordevs.org/glossary#small-language-model)

## Embedding

Page: https://aifordevs.org/glossary#embedding | Markdown: https://aifordevs.org/glossary/embedding.md

An embedding is a fixed-length vector of numbers that represents a piece of text, an image, or code, produced by an embedding model. Items with similar meaning end up close together, which is what vector search relies on.

Embed queries and documents with the same model, because vectors from different models are not comparable. As of November 8, 2026, OpenAI's text-embedding-3-small returns 1,536 dimensions by default and accepts a dimensions parameter to shorten the vector. OpenAI recommends cosine similarity, which on its unit-length vectors reduces to a dot product.

### Example

In pgvector a vector takes 4 bytes per dimension plus 8 bytes. At 1,536 dimensions that is 6,152 bytes each, or about 6.15 GB for one million vectors before any index.

### Source

[OpenAI docs: Vector embeddings](https://developers.openai.com/api/docs/guides/embeddings)

### Related terms

- [Vector database](https://aifordevs.org/glossary#vector-database)
- [RAG](https://aifordevs.org/glossary#rag)
- [Chunking](https://aifordevs.org/glossary#chunking)
- [HNSW](https://aifordevs.org/glossary#hnsw)

### Read the article

[What is a vector database? HNSW, IVF, and when to skip one](https://aifordevs.org/articles/what-is-a-vector-database)

## Eval

Page: https://aifordevs.org/glossary#eval | Markdown: https://aifordevs.org/glossary/eval.md

An eval is a repeatable test of a model or an agent on examples from your task, scored by code, by people, or by another model. Evals show whether a prompt change, a model swap, or a new tool made results better or worse.

Anthropic's vocabulary is a useful starting point: a task has defined inputs and success criteria, a grader scores one aspect of the result, and a trial is one attempt. Outputs vary between runs, so run each task several times and compare pass rates.

### Example

On a 40-task suite, a prompt change that moves passing tasks from 31 to 36 raises the pass rate from 77.5% to 90%.

### Source

[Anthropic, Demystifying evals for AI agents (2026)](https://www.anthropic.com/research/demystifying-evals-for-ai-agents)

### Related terms

- [Golden dataset](https://aifordevs.org/glossary#golden-dataset)
- [LLM-as-a-judge](https://aifordevs.org/glossary#llm-as-a-judge)
- [SWE-bench](https://aifordevs.org/glossary#swe-bench)
- [Hallucination](https://aifordevs.org/glossary#hallucination)
- [Guardrails](https://aifordevs.org/glossary#guardrails)

### Read the article

[LLM evaluation: build an eval suite and gate deploys in CI](https://aifordevs.org/articles/llm-evaluation)

## Fine-tuning

Page: https://aifordevs.org/glossary#fine-tuning | Markdown: https://aifordevs.org/glossary/fine-tuning.md

Fine-tuning is continuing to train a pretrained model on your own examples so it adopts a format, a style, or a skill. Full fine-tuning updates every weight, while parameter-efficient methods such as LoRA update a small set of added weights.

Fine-tuning changes behavior, so it suits output format and style. Facts that change belong in retrieval, because updating a fine-tuned model means training it again. As of November 8, 2026, Anthropic's glossary says the Claude API does not offer fine-tuning.

### Example

Hu et al. report that full fine-tuning of GPT-3 175B needs about 1.2 TB of training memory, against about 350 GB with LoRA.

### Source

[Anthropic docs: Glossary](https://platform.claude.com/docs/en/about-claude/glossary)

### Related terms

- [LoRA](https://aifordevs.org/glossary#lora)
- [QLoRA](https://aifordevs.org/glossary#qlora)
- [RLHF](https://aifordevs.org/glossary#rlhf)
- [Distillation](https://aifordevs.org/glossary#distillation)
- [RAG](https://aifordevs.org/glossary#rag)

### Read the article

[LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script](https://aifordevs.org/articles/lora-fine-tuning)

## GGUF

Page: https://aifordevs.org/glossary#gguf | Markdown: https://aifordevs.org/glossary/gguf.md

GGUF is the single-file model format used by llama.cpp and by tools that read it, such as Ollama. A GGUF file holds the weights, often quantized to 4 to 8 bits, along with the tokenizer and metadata needed to run the model.

A GGUF file has a header, key-value metadata, tensor info, and tensor data, so a runtime needs no external files to load the model. The spec describes version 3 of the format. Its tokenizer.ggml keys hold the vocabulary, and the spec warns that those keys are not guaranteed to be standardized across models.

### Example

To run a local file in Ollama, write FROM /path/to/file.gguf in a Modelfile and run ollama create my-model. Ollama does not quantize the file on import, so quantize it first with llama-quantize.

### Source

[ggml docs: GGUF specification](https://raw.githubusercontent.com/ggml-org/ggml/master/docs/gguf.md)

### Related terms

- [Quantization](https://aifordevs.org/glossary#quantization)
- [Inference](https://aifordevs.org/glossary#inference)
- [Small language model](https://aifordevs.org/glossary#small-language-model)

### Read the article

[vLLM vs SGLang, plus Ollama and llama.cpp compared](https://aifordevs.org/articles/vllm-sglang-llama-cpp)

## Golden dataset

Page: https://aifordevs.org/glossary#golden-dataset | Markdown: https://aifordevs.org/glossary/golden-dataset.md

A golden dataset is a fixed, versioned set of inputs with known-good outputs or success criteria that you use to score a model or an agent. Scores stay comparable across runs because the set does not change between them.

Build it from real usage and real failures, then add a case each time a bug reaches users. Anthropic's eval guidance says 20 to 50 simple tasks drawn from real failures is a great start, and that teams often wait too long for hundreds. Keep it separate from the examples you tune prompts against, or the score stops predicting production.

### Example

A support agent's set might hold 40 past tickets, each with the correct policy, the expected tool call, and a grader that checks both.

### Source

[Anthropic, Demystifying evals for AI agents (2026)](https://www.anthropic.com/research/demystifying-evals-for-ai-agents)

### Related terms

- [Eval](https://aifordevs.org/glossary#eval)
- [LLM-as-a-judge](https://aifordevs.org/glossary#llm-as-a-judge)
- [SWE-bench](https://aifordevs.org/glossary#swe-bench)

### Read the article

[LLM evaluation: build an eval suite and gate deploys in CI](https://aifordevs.org/articles/llm-evaluation)

## Grouped-query attention (GQA)

Page: https://aifordevs.org/glossary#grouped-query-attention | Markdown: https://aifordevs.org/glossary/grouped-query-attention.md

Grouped-query attention (GQA) is an attention variant in which several query heads share one key and value head. It uses more key-value heads than multi-query attention and fewer than multi-head attention.

The KV cache stores only key and value heads, so cutting them shrinks cache memory and the memory read on every decode step. Ainslie et al. (2023) report that GQA reaches quality close to multi-head attention at a speed comparable to multi-query attention.

### Example

Llama 3.1 70B has 64 query heads and 8 key-value heads, so its KV cache is 8 times smaller than it would be with full multi-head attention.

### Source

[Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023)](https://arxiv.org/abs/2305.13245)

### Related terms

- [Attention](https://aifordevs.org/glossary#attention)
- [KV cache](https://aifordevs.org/glossary#kv-cache)

### Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## Guardrails

Page: https://aifordevs.org/glossary#guardrails | Markdown: https://aifordevs.org/glossary/guardrails.md

Guardrails are checks that run on a model's input or output to block, rewrite, or flag content such as personal data, prompt injection, policy violations, or malformed responses. They run outside the model, so they still apply when the model's output is wrong.

Each check adds latency, and running input checks in parallel with the model call limits that cost. OWASP lists prompt injection as LLM01 and says it is unclear whether fool-proof prevention exists. Treat guardrails as one layer alongside least-privilege tools and human approval for risky actions.

### Example

An output guardrail can scan a reply for a regex matching card numbers and replace any match before the text reaches the user.

### Source

[OWASP, LLM01:2025 Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/)

### Related terms

- [Eval](https://aifordevs.org/glossary#eval)
- [Hallucination](https://aifordevs.org/glossary#hallucination)
- [AI gateway](https://aifordevs.org/glossary#ai-gateway)

## Hallucination

Page: https://aifordevs.org/glossary#hallucination | Markdown: https://aifordevs.org/glossary/hallucination.md

A hallucination is model output that reads as confident and fluent but is false or unsupported by the input, such as an invented API, citation, or number. Retrieval and tool use reduce it, and evals measure it, but none of them remove it.

Kalai et al. (2025) trace it to incentives: pretraining produces errors when a model cannot separate true statements from false ones, and most benchmarks grade like exams where guessing can beat saying "I don't know." In your own evals, score a wrong answer worse than an abstention.

### Example

A coding assistant that suggests a function missing from the library's current version has hallucinated. A type check catches it before the code runs.

### Source

[Kalai et al., Why Language Models Hallucinate (2025)](https://arxiv.org/abs/2509.04664)

### Related terms

- [RAG](https://aifordevs.org/glossary#rag)
- [Eval](https://aifordevs.org/glossary#eval)
- [Guardrails](https://aifordevs.org/glossary#guardrails)

## HNSW (Hierarchical Navigable Small World)

Page: https://aifordevs.org/glossary#hnsw | Markdown: https://aifordevs.org/glossary/hnsw.md

HNSW (Hierarchical Navigable Small World) is a graph index for approximate nearest neighbor search that stacks proximity graphs, with fewer points in each higher layer. Queries start at the top layer and descend, which gives logarithmic scaling with dataset size.

Each vector gets a random top layer with an exponentially decaying probability, so upper layers stay sparse. M limits links per node, ef_construction sets the candidate list at build time, and ef_search sets it at query time, which trades recall for latency. pgvector's docs list IVFFlat as faster to build and lighter on memory than HNSW, with lower query performance at the same recall.

### Example

pgvector defaults to m = 16, ef_construction = 64, and hnsw.ef_search = 40. Raise ef_search for a query when recall matters more than latency.

### Source

[Malkov and Yashunin, Hierarchical Navigable Small World graphs (2016)](https://arxiv.org/abs/1603.09320)

### Related terms

- [Vector database](https://aifordevs.org/glossary#vector-database)
- [Embedding](https://aifordevs.org/glossary#embedding)
- [Hybrid search](https://aifordevs.org/glossary#hybrid-search)

### Read the article

[What is a vector database? HNSW, IVF, and when to skip one](https://aifordevs.org/articles/what-is-a-vector-database)

## Hybrid search

Page: https://aifordevs.org/glossary#hybrid-search | Markdown: https://aifordevs.org/glossary/hybrid-search.md

Hybrid search combines keyword search, usually BM25, with vector search and merges the two ranked lists into one result. Keyword matching catches exact identifiers that embeddings miss, and vectors catch paraphrases that keywords miss.

Reciprocal rank fusion is a common merge step because it uses ranks and needs no score normalization. Each document scores the sum of 1 / (k + rank) across lists, and Elasticsearch defaults k to 60. In Anthropic's own test, adding contextual BM25 to contextual embeddings raised the cut in top-20 retrieval failures from 35% to 49%.

### Example

A document ranked 1st by BM25 and 3rd by vectors scores 1/61 + 1/63, or about 0.0323, with k = 60.

### Source

[Elastic docs: Reciprocal rank fusion](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion)

### Related terms

- [Vector database](https://aifordevs.org/glossary#vector-database)
- [HNSW](https://aifordevs.org/glossary#hnsw)
- [Reranking](https://aifordevs.org/glossary#reranking)
- [RAG](https://aifordevs.org/glossary#rag)
- [Chunking](https://aifordevs.org/glossary#chunking)

### Read the article

[What is a vector database? HNSW, IVF, and when to skip one](https://aifordevs.org/articles/what-is-a-vector-database)

## Inference

Page: https://aifordevs.org/glossary#inference | Markdown: https://aifordevs.org/glossary/inference.md

Inference is running a trained model to produce output. For an LLM that means processing the prompt, then generating tokens one at a time until a stop token or a length limit.

LLM inference has two phases with different hardware limits, prefill and decode. Serving engines such as vLLM, SGLang, and llama.cpp schedule these phases and manage the KV cache, so the engine you pick changes latency and cost on the same hardware.

### Example

Generating a 500-token answer at 25 ms per token takes about 12.5 seconds of decode time.

### Related terms

- [Prefill and decode](https://aifordevs.org/glossary#prefill-and-decode)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [Throughput](https://aifordevs.org/glossary#throughput)
- [TTFT](https://aifordevs.org/glossary#ttft)
- [Continuous batching](https://aifordevs.org/glossary#continuous-batching)
- [GGUF](https://aifordevs.org/glossary#gguf)
- [Reasoning model](https://aifordevs.org/glossary#reasoning-model)
- [Speculative decoding](https://aifordevs.org/glossary#speculative-decoding)

### Read the article

[vLLM vs SGLang, plus Ollama and llama.cpp compared](https://aifordevs.org/articles/vllm-sglang-llama-cpp)

## KV cache (Key-value cache)

Page: https://aifordevs.org/glossary#kv-cache | Markdown: https://aifordevs.org/glossary/kv-cache.md

A KV cache (key-value cache) is memory that stores the attention keys and values of every token already processed, so each new token computes only its own. It trades compute for memory, cutting per-step attention cost from quadratic to linear in sequence length while the cache grows with sequence length and batch size.

Per token, each layer stores one key and one value vector for each KV head, so bytes per token = 2 × layers × KV heads × head dim × bytes per value. The cache belongs to one sequence, so concurrent requests multiply the total. Servers ration that memory with managers such as PagedAttention.

### Example

Llama 3.1 70B has 80 layers, 8 KV heads, and a head dim of 128. At 16-bit values that is 327,680 bytes (320 KiB) per token, or 2.5 GiB for an 8,192-token sequence.

### Source

[Hugging Face Transformers docs: How caching works](https://huggingface.co/docs/transformers/en/cache_explanation)

### Related terms

- [Attention](https://aifordevs.org/glossary#attention)
- [PagedAttention](https://aifordevs.org/glossary#paged-attention)
- [Prefill and decode](https://aifordevs.org/glossary#prefill-and-decode)
- [Inference](https://aifordevs.org/glossary#inference)
- [Grouped-query attention](https://aifordevs.org/glossary#grouped-query-attention)
- [Quantization](https://aifordevs.org/glossary#quantization)
- [Prefix caching](https://aifordevs.org/glossary#prefix-caching)
- [Prompt caching](https://aifordevs.org/glossary#prompt-caching)

### Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## LLM-as-a-judge

Page: https://aifordevs.org/glossary#llm-as-a-judge | Markdown: https://aifordevs.org/glossary/llm-as-a-judge.md

LLM-as-a-judge is an evaluation method that uses one model to score the outputs of another against a rubric or a comparison prompt. It gives you a scorer for open-ended text that exact-match checks cannot grade.

Zheng et al. (2023) found that strong judges such as GPT-4 agreed with human preferences more than 80% of the time, about the level of agreement among humans. The same paper documents position, verbosity, and self-enhancement biases, so validate a judge against human labels on your own task before you trust it.

### Example

Run the judge twice with the answers swapped. Zheng et al. count a win only when the same answer is preferred in both orders, and call it a tie otherwise.

### Source

[Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)](https://arxiv.org/abs/2306.05685)

### Related terms

- [Eval](https://aifordevs.org/glossary#eval)
- [Golden dataset](https://aifordevs.org/glossary#golden-dataset)

### Read the article

[LLM as a judge: biases, validation, and when to distrust it](https://aifordevs.org/articles/llm-as-a-judge)

## LoRA (Low-rank adaptation)

Page: https://aifordevs.org/glossary#lora | Markdown: https://aifordevs.org/glossary/lora.md

LoRA (low-rank adaptation) is a fine-tuning method from Hu et al. (2021) that freezes the base model and trains small low-rank matrices added to selected layers. An adapter is far smaller than the model it modifies, so one base model can serve many adapters.

Each adapted weight becomes W + (alpha / r) BA, where B and A have rank r and only they are trained. Rank r sets adapter size and capacity, and alpha scales the strength of the update. Hu et al. adapted only attention projections on GPT-3 175B and report training about 0.01% of the parameters.

### Example

For GPT-3 175B at r = 4, Hu et al. report a checkpoint of about 35 MB against about 350 GB for the full model, roughly 10,000 times smaller.

### Source

[Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)

### Related terms

- [Fine-tuning](https://aifordevs.org/glossary#fine-tuning)
- [QLoRA](https://aifordevs.org/glossary#qlora)

### Read the article

[LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script](https://aifordevs.org/articles/lora-fine-tuning)

## MCP (Model Context Protocol)

Page: https://aifordevs.org/glossary#mcp | Markdown: https://aifordevs.org/glossary/mcp.md

MCP (Model Context Protocol) is an open protocol, introduced by Anthropic in November 2024, that standardizes how AI applications connect to tools and data. A host application runs MCP clients that talk to MCP servers over JSON-RPC, using stdio for local servers or Streamable HTTP for remote ones.

Servers offer resources, prompts, and tools. The 2026-07-28 revision makes requests stateless and self-contained and negotiates capabilities per request, replacing the earlier initialize handshake. Anthropic donated MCP to the Agentic AI Foundation, a Linux Foundation fund, in December 2025.

### Example

To run a tool, a client sends a JSON-RPC tools/call request with the tool name and arguments, then passes the result back to the model.

### Source

[Model Context Protocol specification, revision 2026-07-28](https://modelcontextprotocol.io/specification/2026-07-28)

### Related terms

- [MCP server](https://aifordevs.org/glossary#mcp-server)
- [Tool calling](https://aifordevs.org/glossary#tool-calling)
- [A2A](https://aifordevs.org/glossary#a2a)
- [Agent](https://aifordevs.org/glossary#agent)

### Read the article

[What is an MCP server? Build and test one in TypeScript](https://aifordevs.org/articles/build-an-mcp-server)

## MCP server

Page: https://aifordevs.org/glossary#mcp-server | Markdown: https://aifordevs.org/glossary/mcp-server.md

An MCP server is a program that exposes tools, resources, and prompts to any MCP-compatible client. Write one server for your API or database, and every MCP host, from coding agents to chat apps, can use it.

Tools are model-controlled functions, resources hold context data, and prompts are templates for users. A server declares each capability, then answers requests such as tools/list and tools/call. The spec says a human should be able to deny tool invocations, and that tool annotations are untrusted unless they come from a trusted server.

### Example

A weather server lists a get_weather tool whose inputSchema requires location. A client calls it with tools/call and {"location": "New York"}, then receives text content back.

### Source

[Model Context Protocol specification: Tools (2026-07-28)](https://modelcontextprotocol.io/specification/2026-07-28/server/tools)

### Related terms

- [MCP](https://aifordevs.org/glossary#mcp)
- [Tool calling](https://aifordevs.org/glossary#tool-calling)

### Read the article

[What is an MCP server? Build and test one in TypeScript](https://aifordevs.org/articles/build-an-mcp-server)

## Mixture of experts (MoE)

Page: https://aifordevs.org/glossary#mixture-of-experts | Markdown: https://aifordevs.org/glossary/mixture-of-experts.md

Mixture of experts (MoE) is a model architecture that replaces selected feed-forward layers with many expert sub-networks and routes each token to only a few of them. Total parameters can be very large while the compute per token stays close to that of a much smaller dense model.

Shazeer et al. (2017) introduced the sparsely gated MoE layer, where a trainable gating network selects experts for each input. Memory scales with total parameters because every expert must be loaded, while compute per token scales with the active ones.

### Example

DeepSeek-V3 has 671B total parameters and 37B active per token, about 5.5%. Mixtral 8x7B routes each token to 2 of 8 experts per layer, with about 13B of its 47B parameters active.

### Source

[Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017)](https://arxiv.org/abs/1701.06538)

### Related terms

- [Transformer](https://aifordevs.org/glossary#transformer)

### Read the article

[Mixture of experts: how MoE works and what it costs](https://aifordevs.org/articles/mixture-of-experts)

## PagedAttention

Page: https://aifordevs.org/glossary#paged-attention | Markdown: https://aifordevs.org/glossary/paged-attention.md

PagedAttention is the KV cache memory manager introduced with vLLM (Kwon et al., 2023). It stores the cache in fixed-size blocks, like pages in virtual memory, so requests do not need one contiguous allocation and fragmentation stays low.

Equal-size blocks remove external fragmentation, and small blocks limit internal fragmentation. Kwon et al. measured that earlier systems used only 20.4% to 38.2% of KV cache memory for actual token state, and they report 2 to 4 times the throughput at the same latency.

### Example

vLLM defaults to 16-token blocks. A 100-token sequence takes 7 blocks, or 112 slots, so 12 slots sit idle.

### Source

[Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)](https://arxiv.org/abs/2309.06180)

### Related terms

- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [Throughput](https://aifordevs.org/glossary#throughput)
- [Continuous batching](https://aifordevs.org/glossary#continuous-batching)
- [Prefix caching](https://aifordevs.org/glossary#prefix-caching)

### Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## Prefill and decode

Page: https://aifordevs.org/glossary#prefill-and-decode | Markdown: https://aifordevs.org/glossary/prefill-and-decode.md

Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in parallel and is limited by compute, and decode generates one token per step and is limited by memory bandwidth.

Patel et al. (Splitwise, 2023) call these prompt computation and token generation, and show that token generation underuses compute. Time to first token mostly reflects prefill and time between tokens reflects decode, so the two respond to different fixes.

### Example

Splitwise reports that a 1,500-token prompt on BLOOM-176B takes as long as generating only 6 output tokens.

### Source

[Patel et al., Splitwise: Efficient generative LLM inference using phase splitting (2023)](https://arxiv.org/abs/2311.18677)

### Related terms

- [Inference](https://aifordevs.org/glossary#inference)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [TTFT](https://aifordevs.org/glossary#ttft)

### Read the article

[KV cache explained: what it stores and what it costs](https://aifordevs.org/articles/kv-cache-explained)

## Prefix caching

Page: https://aifordevs.org/glossary#prefix-caching | Markdown: https://aifordevs.org/glossary/prefix-caching.md

Prefix caching is a serving-engine technique that reuses KV cache blocks from earlier requests when a new prompt starts with the same tokens. Prefill skips the matched tokens, which lowers time to first token and frees GPU compute.

vLLM hashes each block from its parent's hash, its own tokens, and extras such as LoRA IDs or a cache salt, so a block matches only when its whole prefix matches, and only full blocks are cached. SGLang's RadixAttention keeps prefixes in a radix tree with LRU eviction. Prompt caching is the hosted API feature built on the same idea, with its own pricing.

### Example

With 16-token blocks, a shared 100-token prefix gives 6 full cached blocks, or 96 tokens. The remaining 4 tokens get recomputed.

### Source

[vLLM docs: Automatic Prefix Caching](https://docs.vllm.ai/en/stable/design/prefix_caching/)

### Related terms

- [Prompt caching](https://aifordevs.org/glossary#prompt-caching)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [TTFT](https://aifordevs.org/glossary#ttft)
- [PagedAttention](https://aifordevs.org/glossary#paged-attention)

### Read the article

[vLLM vs SGLang, plus Ollama and llama.cpp compared](https://aifordevs.org/articles/vllm-sglang-llama-cpp)

## Prompt caching

Page: https://aifordevs.org/glossary#prompt-caching | Markdown: https://aifordevs.org/glossary/prompt-caching.md

Prompt caching is a hosted API feature that reuses the processed state of a repeated prompt prefix, such as a long system prompt or a document, across calls. Providers that offer it bill cached input tokens at a discount and return the first token sooner.

Hits need an exact match of everything before the cache point, so put stable content first and volatile content last. Anthropic hashes tools, then system, then messages, so changing a tool definition invalidates the whole cache. As of November 8, 2026, Claude charges 1.25 times the input price for a 5-minute write and 0.025 to 0.1 times for reads, while OpenAI caches automatically from 1,024 tokens on GPT-5.6 and later.

### Example

As of November 8, 2026, Claude Opus 5.5 lists $4 input, $5 cache write, and $0.20 cache read per million tokens. Ten calls within five minutes on a 100,000-token prefix cost $0.68 for that prefix with caching and $4.00 without.

### Source

[Anthropic docs: Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)

### Related terms

- [Prefix caching](https://aifordevs.org/glossary#prefix-caching)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [Context engineering](https://aifordevs.org/glossary#context-engineering)
- [System prompt](https://aifordevs.org/glossary#system-prompt)
- [AI gateway](https://aifordevs.org/glossary#ai-gateway)
- [Batch API](https://aifordevs.org/glossary#batch-api)
- [TTFT](https://aifordevs.org/glossary#ttft)

### Read the article

[Context engineering: budget and order the context window](https://aifordevs.org/articles/context-engineering)

## QLoRA

Page: https://aifordevs.org/glossary#qlora | Markdown: https://aifordevs.org/glossary/qlora.md

QLoRA is a fine-tuning method from Dettmers et al. (2023) that trains LoRA adapters on a base model quantized to 4 bits. It lowers the memory needed to fine-tune a large model.

The base weights use a 4-bit NormalFloat (NF4) format, double quantization compresses the quantization constants, and paged optimizers manage memory spikes during training. Reach for it when the model will not fit on your GPU at full precision.

### Example

Dettmers et al. report fine-tuning a 65B-parameter model on a single 48 GB GPU while keeping the task performance of full 16-bit fine-tuning.

### Source

[Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)

### Related terms

- [LoRA](https://aifordevs.org/glossary#lora)
- [Fine-tuning](https://aifordevs.org/glossary#fine-tuning)
- [Quantization](https://aifordevs.org/glossary#quantization)

### Read the article

[LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script](https://aifordevs.org/articles/lora-fine-tuning)

## Quantization

Page: https://aifordevs.org/glossary#quantization | Markdown: https://aifordevs.org/glossary/quantization.md

Quantization is storing model weights, and sometimes activations or the KV cache, in fewer bits, for example 8-bit or 4-bit instead of 16-bit. It cuts memory and often speeds up decoding, at some cost in quality that you should measure on your task.

GPTQ (Frantar et al., 2022) brought weights down to 3 or 4 bits and ran a 175B-parameter model on a single GPU. Decode is limited by memory bandwidth, so smaller weights can raise tokens per second. Quality loss varies by model and task, so run your own evals before and after.

### Example

A 70B-parameter model takes about 140 GB at 16 bits, 70 GB at 8 bits, and 35 GB at 4 bits, before the KV cache.

### Source

[Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022)](https://arxiv.org/abs/2210.17323)

### Related terms

- [GGUF](https://aifordevs.org/glossary#gguf)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [QLoRA](https://aifordevs.org/glossary#qlora)
- [Small language model](https://aifordevs.org/glossary#small-language-model)

## RAG (Retrieval-augmented generation)

Page: https://aifordevs.org/glossary#rag | Markdown: https://aifordevs.org/glossary/rag.md

RAG (retrieval-augmented generation) is a technique that fetches relevant documents at request time and puts them in the prompt so the model answers from them. The term comes from Lewis et al. (2020).

A typical pipeline chunks documents, embeds them, retrieves the nearest passages for each query, and often reranks them before the model call. If retrieval misses the right passage, the model cannot answer from it, so measure retrieval and generation separately. Use RAG for facts that change or need citations, and fine-tuning for format and style.

### Example

Five retrieved chunks of 400 tokens each add 2,000 input tokens to every request, which is the number to budget against your context window and price.

### Source

[Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)](https://arxiv.org/abs/2005.11401)

### Related terms

- [Embedding](https://aifordevs.org/glossary#embedding)
- [Vector database](https://aifordevs.org/glossary#vector-database)
- [Chunking](https://aifordevs.org/glossary#chunking)
- [Reranking](https://aifordevs.org/glossary#reranking)
- [Hybrid search](https://aifordevs.org/glossary#hybrid-search)
- [Context engineering](https://aifordevs.org/glossary#context-engineering)
- [Hallucination](https://aifordevs.org/glossary#hallucination)
- [Fine-tuning](https://aifordevs.org/glossary#fine-tuning)

### Read the article

[RAG vs fine tuning: which failures each one can fix](https://aifordevs.org/articles/rag-vs-fine-tuning)

## Reasoning model

Page: https://aifordevs.org/glossary#reasoning-model | Markdown: https://aifordevs.org/glossary/reasoning-model.md

A reasoning model is a model trained to generate intermediate reasoning tokens before its final answer. It trades latency and output tokens for better results on math, code, and multi-step problems.

OpenAI bills reasoning tokens as output tokens and counts them against the context window, even though the API does not return them. DeepSeek-R1 reports that reinforcement learning alone can produce this behavior, without human-labeled reasoning examples. OpenAI and Anthropic both expose an effort setting that trades reasoning depth for latency and token use.

### Example

A request that returns 300 visible tokens after 3,000 reasoning tokens is billed for 3,300 output tokens.

### Source

[OpenAI docs: Reasoning models](https://developers.openai.com/api/docs/guides/reasoning)

### Related terms

- [Token](https://aifordevs.org/glossary#token)
- [Inference](https://aifordevs.org/glossary#inference)
- [Context window](https://aifordevs.org/glossary#context-window)

## Reranking

Page: https://aifordevs.org/glossary#reranking | Markdown: https://aifordevs.org/glossary/reranking.md

Reranking is a second retrieval pass that rescores a shortlist of candidates with a slower, more accurate model and keeps the best few. The first pass can favor recall because the reranker sets the final order.

A reranker scores the query and each passage together, so it catches relevance that separate embeddings miss, at the cost of added latency per candidate. Nogueira and Cho (2019) reported a 27% relative gain in MRR@10 on MS MARCO with BERT as the reranker.

### Example

Anthropic's Contextual Retrieval test took the top 150 chunks, reranked them with Cohere's model, and kept 20. Anthropic reports that the full stack cut top-20 retrieval failures by 67%, from 5.7% to 1.9%.

### Source

[Nogueira and Cho, Passage Re-ranking with BERT (2019)](https://arxiv.org/abs/1901.04085)

### Related terms

- [RAG](https://aifordevs.org/glossary#rag)
- [Hybrid search](https://aifordevs.org/glossary#hybrid-search)
- [Chunking](https://aifordevs.org/glossary#chunking)
- [Vector database](https://aifordevs.org/glossary#vector-database)

### Read the article

[What is a vector database? HNSW, IVF, and when to skip one](https://aifordevs.org/articles/what-is-a-vector-database)

## RLHF (Reinforcement learning from human feedback)

Page: https://aifordevs.org/glossary#rlhf | Markdown: https://aifordevs.org/glossary/rlhf.md

RLHF (reinforcement learning from human feedback) is training a model with a reward signal learned from human preference comparisons. It is one of the post-training steps that turns a pretrained model into an assistant that follows instructions.

People rank model outputs, a reward model learns those rankings, and reinforcement learning pushes the model toward outputs the reward model scores highly. Christiano et al. (2017) demonstrated learning from pairwise preferences on Atari and robot locomotion, and InstructGPT (Ouyang et al., 2022) applied it to language models.

### Example

In the InstructGPT paper, human evaluators preferred the output of a 1.3B-parameter model over that of the 175B-parameter GPT-3.

### Source

[Ouyang et al., Training language models to follow instructions with human feedback (2022)](https://arxiv.org/abs/2203.02155)

### Related terms

- [Fine-tuning](https://aifordevs.org/glossary#fine-tuning)

## Small language model

Page: https://aifordevs.org/glossary#small-language-model | Markdown: https://aifordevs.org/glossary/small-language-model.md

A small language model is a language model small enough to run on a single consumer GPU, a laptop, or a phone, usually under about 10 billion parameters. It suits narrow tasks, low latency, and keeping data on the device.

Belcak et al. (2025) define the term by device fit: a model that runs on a common consumer device at low enough latency to serve one user. They say they would count most models below 10 billion parameters, and they argue that many agent calls do not need a larger model.

### Example

An 8-billion-parameter model quantized to 4 bits takes about 4 GB, which fits in a laptop's memory with room for the KV cache.

### Source

[Belcak et al., Small Language Models are the Future of Agentic AI (2025)](https://arxiv.org/abs/2506.02153)

### Related terms

- [Quantization](https://aifordevs.org/glossary#quantization)
- [Distillation](https://aifordevs.org/glossary#distillation)
- [GGUF](https://aifordevs.org/glossary#gguf)

## Speculative decoding

Page: https://aifordevs.org/glossary#speculative-decoding | Markdown: https://aifordevs.org/glossary/speculative-decoding.md

Speculative decoding is a decoding method where a fast draft step proposes several tokens and the large model verifies them in one forward pass, keeping the longest accepted prefix. With the standard acceptance rule, the output distribution matches the large model's.

Leviathan et al. (2023) accept each draft token with probability min(1, p / q), where p is the target model's probability and q is the draft's. A rejection discards the later drafts and resamples from an adjusted distribution. The target model never needs more serial steps than standard decoding, and the speedup grows with the acceptance rate.

### Example

With 4 drafts per step and a 70% acceptance rate, the expected yield is (1 - 0.7^5) / (1 - 0.7), or about 2.8 tokens per target-model call. Leviathan et al. report 2x to 3x faster decoding on T5-XXL.

### Source

[Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023)](https://arxiv.org/abs/2211.17192)

### Related terms

- [Inference](https://aifordevs.org/glossary#inference)
- [Throughput](https://aifordevs.org/glossary#throughput)

### Read the article

[Speculative decoding: how drafts speed up LLM inference](https://aifordevs.org/articles/speculative-decoding)

## Structured outputs

Page: https://aifordevs.org/glossary#structured-outputs | Markdown: https://aifordevs.org/glossary/structured-outputs.md

Structured outputs is an API feature that constrains a model's output to match a JSON Schema you supply, so successful responses parse against your schema. It is usually implemented with constrained decoding, which masks tokens that would break the schema.

Handle the exceptions: a refusal can return text outside the schema, and hitting a token limit can cut the JSON short. Check the refusal field or stop reason before parsing. The first request with a new schema adds latency while the API processes it.

### Example

On the Claude API, set output_config.format to a json_schema and read stop_reason before calling JSON.parse. A stop_reason of max_tokens means the JSON may be truncated.

### Source

[Anthropic docs: Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)

### Related terms

- [Tool calling](https://aifordevs.org/glossary#tool-calling)
- [Constrained decoding](https://aifordevs.org/glossary#constrained-decoding)

## SWE-bench

Page: https://aifordevs.org/glossary#swe-bench | Markdown: https://aifordevs.org/glossary/swe-bench.md

SWE-bench is a benchmark that asks a model or agent to fix real GitHub issues by editing the repository, then runs the project's tests to check the patch. The original set has 2,294 problems from 12 Python repositories.

SWE-bench Verified is a 500-problem subset that engineers confirmed as solvable. Compare scores only within one dataset and one agent setup, because the code around the model changes the result.

### Example

A score of 70% on Verified means 350 of the 500 patches applied and passed the required tests.

### Source

[Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023)](https://arxiv.org/abs/2310.06770)

### Related terms

- [Eval](https://aifordevs.org/glossary#eval)
- [Golden dataset](https://aifordevs.org/glossary#golden-dataset)

### Read the article

[Best LLM for coding: November 2026 benchmarks, compared](https://aifordevs.org/articles/best-llm-for-coding)

## System prompt

Page: https://aifordevs.org/glossary#system-prompt | Markdown: https://aifordevs.org/glossary/system-prompt.md

A system prompt is a set of instructions placed before the conversation that sets the model's role, rules, and output format for every turn. Keeping it stable makes it a good candidate for prompt caching.

APIs name it differently: Anthropic uses a top-level system parameter, and OpenAI's Responses API uses instructions or developer messages, which take priority over user messages. Keep secrets and permission checks out of it, and enforce those in the code that runs tools.

### Example

Put a 5,000-token system prompt first and the user's question last. A timestamp at the top of the prompt changes the prefix on every request and prevents a cache hit.

### Source

[OpenAI docs: Text generation](https://developers.openai.com/api/docs/guides/text)

### Related terms

- [Context engineering](https://aifordevs.org/glossary#context-engineering)
- [Prompt caching](https://aifordevs.org/glossary#prompt-caching)

### Read the article

[Context engineering: budget and order the context window](https://aifordevs.org/articles/context-engineering)

## Temperature

Page: https://aifordevs.org/glossary#temperature | Markdown: https://aifordevs.org/glossary/temperature.md

Temperature is a sampling setting that divides the model's logits before they are converted to token probabilities. Lower values make output more deterministic, and higher values make it more varied.

A temperature of 0 does not guarantee identical outputs, as Anthropic's glossary says of its own API. As of November 8, 2026, Claude 4.7 and later models accept only the default sampling values, and API version 2026-10-21 rejects any request that sends temperature, top_p, or top_k. On those models, steer output with the prompt and effort settings.

### Example

Two tokens with logits 2.0 and 1.0 get probabilities 0.73 and 0.27 at temperature 1. At temperature 0.5 they get 0.88 and 0.12.

### Source

[Anthropic docs: Migrating to API version 2026-10-21](https://platform.claude.com/docs/en/api/migration-2026-10-21)

### Related terms

- [Top-p](https://aifordevs.org/glossary#top-p)
- [Token](https://aifordevs.org/glossary#token)

## Throughput

Page: https://aifordevs.org/glossary#throughput | Markdown: https://aifordevs.org/glossary/throughput.md

Throughput is how many tokens a system generates per second, either for one request or summed across all requests on a deployment. Batching raises total throughput while usually lowering speed per request.

Check which number a benchmark reports. Tokens per second per request is what a user feels, and total tokens per second is what sets your serving cost. Continuous batching and better memory management raise the total by keeping GPU slots busy.

### Example

Forty concurrent requests at 25 tokens per second each is 1,000 tokens per second in total, even though each user still sees 25.

### Related terms

- [TTFT](https://aifordevs.org/glossary#ttft)
- [Inference](https://aifordevs.org/glossary#inference)
- [Continuous batching](https://aifordevs.org/glossary#continuous-batching)
- [Batch API](https://aifordevs.org/glossary#batch-api)
- [PagedAttention](https://aifordevs.org/glossary#paged-attention)
- [Speculative decoding](https://aifordevs.org/glossary#speculative-decoding)

## Token

Page: https://aifordevs.org/glossary#token | Markdown: https://aifordevs.org/glossary/token.md

A token is the unit of text a model takes in and generates: a word, part of a word, a symbol, or a byte sequence, depending on the tokenizer. Context limits, speed, and API prices are all counted in tokens.

Token counts differ by model, so use the provider's counting tool for anything that affects cost. Google says a Gemini token is about 4 characters. Anthropic says about 3.5 English characters on earlier Claude models, and that Claude 4.7 and later produce about 30% more tokens for the same text.

### Example

Text that took 1,000 tokens on an earlier Claude model takes about 1,300 on Claude 4.7 or later, so budget for the difference when you migrate.

### Source

[Anthropic docs: Glossary](https://platform.claude.com/docs/en/about-claude/glossary)

### Related terms

- [Tokenizer](https://aifordevs.org/glossary#tokenizer)
- [Context window](https://aifordevs.org/glossary#context-window)
- [Temperature](https://aifordevs.org/glossary#temperature)
- [Reasoning model](https://aifordevs.org/glossary#reasoning-model)

### Read the article

[Claude API pricing: caching, batch, and real bills](https://aifordevs.org/articles/claude-api-pricing)

## Tokenizer

Page: https://aifordevs.org/glossary#tokenizer | Markdown: https://aifordevs.org/glossary/tokenizer.md

A tokenizer is the component that splits text into tokens and maps them to integer IDs before the model processes them. Different model families use different tokenizers, so the same text can produce a different number of tokens.

Byte pair encoding is a common method: it builds a vocabulary by repeatedly merging frequent adjacent symbol pairs, and Sennrich et al. (2015) applied it to rare words in neural translation. Count with the tokenizer that matches the model you call, because a mismatch gives wrong numbers.

### Example

OpenAI's o200k_base encoding splits "tiktoken is great!" into 6 tokens: t, ikt, oken, is, great, and !. That is 3 characters per token on this string.

### Source

[OpenAI Cookbook: How to count tokens with tiktoken](https://developers.openai.com/cookbook/examples/how_to_count_tokens_with_tiktoken)

### Related terms

- [Token](https://aifordevs.org/glossary#token)

## Tool calling (Function calling)

Page: https://aifordevs.org/glossary#tool-calling | Markdown: https://aifordevs.org/glossary/tool-calling.md

Tool calling is a model returning a structured request to run a function you defined, with arguments, instead of plain text. Your code runs the function and sends the result back, which is how an agent loop reaches systems outside the model.

The model only produces the request, and your code or the provider's servers do the execution. On the Claude API a client tool comes back as a tool_use block with stop_reason tool_use, and you reply with a tool_result block. Tool names, descriptions, and schemas all count as input tokens, so a large tool set shrinks your context budget.

### Example

Given a get_weather tool, the model returns {"location": "San Francisco, CA"} as arguments, your code calls the weather API, and the next request carries the result as a tool_result.

### Source

[Anthropic docs: Tool use with Claude](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview)

### Related terms

- [Agent](https://aifordevs.org/glossary#agent)
- [Agent loop](https://aifordevs.org/glossary#agent-loop)
- [MCP](https://aifordevs.org/glossary#mcp)
- [MCP server](https://aifordevs.org/glossary#mcp-server)
- [Structured outputs](https://aifordevs.org/glossary#structured-outputs)
- [Constrained decoding](https://aifordevs.org/glossary#constrained-decoding)

### Read the article

[What is an AI agent? The loop in code, and where it fails](https://aifordevs.org/articles/what-is-an-ai-agent)

## Top-p (Nucleus sampling)

Page: https://aifordevs.org/glossary#top-p | Markdown: https://aifordevs.org/glossary/top-p.md

Top-p, or nucleus sampling, is a sampling method that keeps only the smallest set of most likely tokens whose probabilities add up to at least p, then samples from that set. The candidate count adapts to how concentrated the distribution is at each step.

Holtzman et al. (2019) proposed it to trim the unreliable low-probability tail while keeping diversity. As of November 8, 2026, API version 2026-10-21 of the Claude API rejects requests that send top_p, so check your provider's docs before you depend on it.

### Example

With token probabilities 0.50, 0.30, 0.15, and 0.05 and p = 0.9, the first three tokens stay (cumulative 0.95). Renormalized, they become about 0.53, 0.32, and 0.16.

### Source

[Holtzman et al., The Curious Case of Neural Text Degeneration (2019)](https://arxiv.org/abs/1904.09751)

### Related terms

- [Temperature](https://aifordevs.org/glossary#temperature)

## Transformer

Page: https://aifordevs.org/glossary#transformer | Markdown: https://aifordevs.org/glossary/transformer.md

A transformer is the neural network architecture behind modern LLMs, introduced in the 2017 paper Attention Is All You Need (Vaswani et al.). It stacks attention and feed-forward layers and processes all tokens of the input in parallel.

A transformer has no recurrence or convolution, so it adds positional encodings to the token embeddings to carry word order. Prefill can process a whole prompt at once because of this design, while generation still proceeds one token at a time.

### Example

The 2017 model used 512-dimensional layers and 8 attention heads. Llama 3.1 70B uses 80 layers, 8,192 dimensions, and 64 heads.

### Source

[Vaswani et al., Attention Is All You Need (2017)](https://arxiv.org/abs/1706.03762)

### Related terms

- [Attention](https://aifordevs.org/glossary#attention)
- [Mixture of experts](https://aifordevs.org/glossary#mixture-of-experts)

## TTFT (Time to first token)

Page: https://aifordevs.org/glossary#ttft | Markdown: https://aifordevs.org/glossary/ttft.md

TTFT (time to first token) is the delay between sending a request and receiving the first output token. It covers queueing and prefill, so long prompts raise it and prompt caching lowers it.

TTFT sets how responsive a streaming interface feels, and it says nothing about how fast the rest of the reply arrives. Report it as a percentile under load, because queueing time only shows up when the server is busy.

### Example

Splitwise measured a median TTFT of 185 ms on an A100 and 95 ms on an H100 for Llama 70B on a coding workload with no batching.

### Source

[Patel et al., Splitwise: Efficient generative LLM inference using phase splitting (2023)](https://arxiv.org/abs/2311.18677)

### Related terms

- [Prefill and decode](https://aifordevs.org/glossary#prefill-and-decode)
- [Throughput](https://aifordevs.org/glossary#throughput)
- [Inference](https://aifordevs.org/glossary#inference)
- [Prefix caching](https://aifordevs.org/glossary#prefix-caching)
- [Prompt caching](https://aifordevs.org/glossary#prompt-caching)

## Vector database

Page: https://aifordevs.org/glossary#vector-database | Markdown: https://aifordevs.org/glossary/vector-database.md

A vector database is a database that indexes embeddings and returns the nearest neighbors to a query vector, usually with approximate search such as HNSW. It is a common retrieval layer in RAG systems.

Approximate indexes trade some recall for speed, and you tune that trade with parameters such as ef_search. pgvector runs exact search by default and switches to approximate search when you add an HNSW or IVFFlat index. Metadata filters interact with the index, so measure recall with your real filters on.

### Example

pgvector's HNSW index accepts vector columns of up to 2,000 dimensions, so a 3,072-dimension text-embedding-3-large vector must be shortened or stored as halfvec.

### Source

[pgvector on GitHub](https://github.com/pgvector/pgvector)

### Related terms

- [Embedding](https://aifordevs.org/glossary#embedding)
- [RAG](https://aifordevs.org/glossary#rag)
- [HNSW](https://aifordevs.org/glossary#hnsw)
- [Hybrid search](https://aifordevs.org/glossary#hybrid-search)
- [Reranking](https://aifordevs.org/glossary#reranking)
- [Chunking](https://aifordevs.org/glossary#chunking)

### Read the article

[What is a vector database? HNSW, IVF, and when to skip one](https://aifordevs.org/articles/what-is-a-vector-database)

## Cite the glossary

AI for Devs. "AI for Devs glossary." https://aifordevs.org/glossary

- Page: https://aifordevs.org/glossary
- Markdown: https://aifordevs.org/glossary.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs

---
title: "AI for Devs platform directory"
type: "directory"
url: "https://aifordevs.org/platforms"
markdown: "https://aifordevs.org/platforms.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/platforms"
platforms: 47
---

# AI for Devs platform directory

What each platform is and where it fits, from model APIs to vector databases. The directory lists facts that stay true for months. Prices and benchmarks live in dated news briefs.

## Model APIs

### Amazon Bedrock

Page: https://aifordevs.org/platforms#amazon-bedrock | Markdown: https://aifordevs.org/platforms/amazon-bedrock.md

Managed access to models from several labs inside your AWS account, with IAM, VPC, and guardrails.

- Maker: AWS
- Category: Model APIs
- Website: https://aws.amazon.com/bedrock/
- Tags: cloud, enterprise

### Claude API

Page: https://aifordevs.org/platforms#anthropic-api | Markdown: https://aifordevs.org/platforms/anthropic-api.md

Claude models through the Messages API, with tool use, prompt caching, and batch processing.

- Maker: Anthropic
- Category: Model APIs
- Website: https://docs.anthropic.com
- Tags: frontier, prompt caching, tool use

### DeepSeek API

Page: https://aifordevs.org/platforms#deepseek-api | Markdown: https://aifordevs.org/platforms/deepseek-api.md

OpenAI-compatible API for DeepSeek's chat and reasoning models, which are also published as open weights.

- Maker: DeepSeek
- Category: Model APIs
- Website: https://api-docs.deepseek.com
- Tags: open weights, openai compatible

### Gemini API

Page: https://aifordevs.org/platforms#gemini-api | Markdown: https://aifordevs.org/platforms/gemini-api.md

Gemini models for text, image, audio, and video input, available through Google AI Studio and Vertex AI.

- Maker: Google
- Category: Model APIs
- Website: https://ai.google.dev
- Tags: frontier, multimodal, long context

### Mistral AI

Page: https://aifordevs.org/platforms#mistral | Markdown: https://aifordevs.org/platforms/mistral.md

Hosted API for Mistral's models, several of which are also released as open weights.

- Maker: Mistral AI
- Category: Model APIs
- Website: https://docs.mistral.ai
- Tags: open weights, europe

### OpenAI API

Page: https://aifordevs.org/platforms#openai-api | Markdown: https://aifordevs.org/platforms/openai-api.md

GPT and reasoning models with the Responses API, tool calling, structured outputs, and realtime audio.

- Maker: OpenAI
- Category: Model APIs
- Website: https://platform.openai.com/docs
- Tags: frontier, tool calling, structured outputs

### Vertex AI

Page: https://aifordevs.org/platforms#vertex-ai | Markdown: https://aifordevs.org/platforms/vertex-ai.md

Google Cloud's platform for Gemini and third-party models, with tuning, evaluation, and agent tooling.

- Maker: Google Cloud
- Category: Model APIs
- Website: https://cloud.google.com/vertex-ai
- Tags: cloud, enterprise

### xAI API

Page: https://aifordevs.org/platforms#xai-api | Markdown: https://aifordevs.org/platforms/xai-api.md

Grok models through an API that is compatible with the OpenAI and Anthropic SDKs.

- Maker: xAI
- Category: Model APIs
- Website: https://docs.x.ai
- Tags: frontier

## Inference platforms

### Baseten

Page: https://aifordevs.org/platforms#baseten | Markdown: https://aifordevs.org/platforms/baseten.md

Dedicated and shared inference for open-weight and custom models, plus training and the Truss packaging tool.

- Maker: Baseten
- Category: Inference platforms
- Website: https://www.baseten.co
- Tags: dedicated inference, training

### Fireworks AI

Page: https://aifordevs.org/platforms#fireworks-ai | Markdown: https://aifordevs.org/platforms/fireworks-ai.md

Fast inference for open-weight models, with fine-tuning and on-demand deployments.

- Maker: Fireworks AI
- Category: Inference platforms
- Website: https://fireworks.ai
- Tags: open weights, fine-tuning

### Groq

Page: https://aifordevs.org/platforms#groq | Markdown: https://aifordevs.org/platforms/groq.md

Inference API running open-weight models on Groq's own LPU chips, built for low latency.

- Maker: Groq
- Category: Inference platforms
- Website: https://groq.com
- Tags: custom silicon, low latency

### Hugging Face

Page: https://aifordevs.org/platforms#hugging-face | Markdown: https://aifordevs.org/platforms/hugging-face.md

The main hub for open-weight models and datasets, plus Inference Providers and dedicated Inference Endpoints.

- Maker: Hugging Face
- Category: Inference platforms
- Website: https://huggingface.co
- Tags: open weights, model hub

### Modal

Page: https://aifordevs.org/platforms#modal | Markdown: https://aifordevs.org/platforms/modal.md

Serverless GPUs driven from Python code, used for inference, batch jobs, and fine-tuning.

- Maker: Modal Labs
- Category: Inference platforms
- Website: https://modal.com
- Tags: serverless, python

### OpenRouter

Page: https://aifordevs.org/platforms#openrouter | Markdown: https://aifordevs.org/platforms/openrouter.md

One OpenAI-compatible API and one bill for hundreds of models across many providers.

- Maker: OpenRouter
- Category: Inference platforms
- Website: https://openrouter.ai
- Tags: router, openai compatible

### Replicate

Page: https://aifordevs.org/platforms#replicate | Markdown: https://aifordevs.org/platforms/replicate.md

Run community and official models through an API, and package your own with the open-source Cog tool.

- Maker: Replicate
- Category: Inference platforms
- Website: https://replicate.com
- Tags: image, video, api

### Together AI

Page: https://aifordevs.org/platforms#together-ai | Markdown: https://aifordevs.org/platforms/together-ai.md

Serverless and dedicated inference for open-weight models, with fine-tuning and GPU clusters.

- Maker: Together AI
- Category: Inference platforms
- Website: https://www.together.ai
- Tags: open weights, fine-tuning

## Inference engines

### llama.cpp

Page: https://aifordevs.org/platforms#llama-cpp | Markdown: https://aifordevs.org/platforms/llama-cpp.md

C/C++ inference for GGUF models on CPUs, Apple silicon, and consumer GPUs, with heavy quantization support.

- Maker: ggml-org
- Category: Inference engines
- Website: https://github.com/ggml-org/llama.cpp
- Open source: yes
- Tags: local, gguf, quantization

### Ollama

Page: https://aifordevs.org/platforms#ollama | Markdown: https://aifordevs.org/platforms/ollama.md

Pull and run open-weight models locally with one command, behind a local REST API.

- Maker: Ollama
- Category: Inference engines
- Website: https://ollama.com
- Open source: yes
- Tags: local, developer tool

### SGLang

Page: https://aifordevs.org/platforms#sglang | Markdown: https://aifordevs.org/platforms/sglang.md

Serving engine with RadixAttention prefix caching and a fast runtime for structured and multi-call workloads.

- Maker: SGLang project
- Category: Inference engines
- Website: https://github.com/sgl-project/sglang
- Open source: yes
- Tags: prefix caching, gpu serving

### TensorRT-LLM

Page: https://aifordevs.org/platforms#tensorrt-llm | Markdown: https://aifordevs.org/platforms/tensorrt-llm.md

NVIDIA's library for optimized LLM inference on NVIDIA GPUs, with in-flight batching and quantization.

- Maker: NVIDIA
- Category: Inference engines
- Website: https://github.com/NVIDIA/TensorRT-LLM
- Open source: yes
- Tags: nvidia, gpu serving

### vLLM

Page: https://aifordevs.org/platforms#vllm | Markdown: https://aifordevs.org/platforms/vllm.md

High-throughput serving engine built around PagedAttention, continuous batching, and an OpenAI-compatible server.

- Maker: vLLM project
- Category: Inference engines
- Website: https://github.com/vllm-project/vllm
- Open source: yes
- Tags: paged attention, gpu serving

## Coding agents

### Aider

Page: https://aifordevs.org/platforms#aider | Markdown: https://aifordevs.org/platforms/aider.md

Open-source pair programming in the terminal that edits files and commits changes to git.

- Maker: Aider
- Category: Coding agents
- Website: https://aider.chat
- Open source: yes
- Tags: terminal, git

### Claude Code

Page: https://aifordevs.org/platforms#claude-code | Markdown: https://aifordevs.org/platforms/claude-code.md

Agentic coding tool that works in your terminal, IDE, desktop app, and browser, and can run commands and edit files.

- Maker: Anthropic
- Category: Coding agents
- Website: https://docs.anthropic.com/en/docs/claude-code/overview
- Tags: terminal, agent

### Cline

Page: https://aifordevs.org/platforms#cline | Markdown: https://aifordevs.org/platforms/cline.md

Open-source coding agent for VS Code that asks for approval before it edits files or runs commands.

- Maker: Cline
- Category: Coding agents
- Website: https://cline.bot
- Open source: yes
- Tags: editor, mcp

### Codex

Page: https://aifordevs.org/platforms#codex | Markdown: https://aifordevs.org/platforms/codex.md

OpenAI's coding agent, available as an open-source CLI, IDE extension, and cloud agent.

- Maker: OpenAI
- Category: Coding agents
- Website: https://developers.openai.com/codex
- Tags: terminal, cloud agent

### Cursor

Page: https://aifordevs.org/platforms#cursor | Markdown: https://aifordevs.org/platforms/cursor.md

AI code editor built on VS Code, with tab completion, an agent mode, and background agents.

- Maker: Anysphere
- Category: Coding agents
- Website: https://cursor.com
- Tags: editor, agent

### Gemini CLI

Page: https://aifordevs.org/platforms#gemini-cli | Markdown: https://aifordevs.org/platforms/gemini-cli.md

Open-source terminal agent that runs on Gemini models and supports MCP servers.

- Maker: Google
- Category: Coding agents
- Website: https://github.com/google-gemini/gemini-cli
- Open source: yes
- Tags: terminal, mcp

### GitHub Copilot

Page: https://aifordevs.org/platforms#github-copilot | Markdown: https://aifordevs.org/platforms/github-copilot.md

Completions, chat, and a coding agent that works from GitHub issues and opens pull requests.

- Maker: GitHub
- Category: Coding agents
- Website: https://github.com/features/copilot
- Tags: editor, github

## Agent frameworks

### AI SDK

Page: https://aifordevs.org/platforms#vercel-ai-sdk | Markdown: https://aifordevs.org/platforms/vercel-ai-sdk.md

TypeScript toolkit for streaming text, structured output, and tool calls across many providers.

- Maker: Vercel
- Category: Agent frameworks
- Website: https://github.com/vercel/ai
- Open source: yes
- Tags: typescript, streaming

### CrewAI

Page: https://aifordevs.org/platforms#crewai | Markdown: https://aifordevs.org/platforms/crewai.md

Python framework for teams of role-based agents and event-driven flows.

- Maker: CrewAI
- Category: Agent frameworks
- Website: https://github.com/crewAIInc/crewAI
- Open source: yes
- Tags: python, multi-agent

### LangChain

Page: https://aifordevs.org/platforms#langchain | Markdown: https://aifordevs.org/platforms/langchain.md

Integrations and building blocks for LLM apps: models, retrievers, tools, and agent loops.

- Maker: LangChain
- Category: Agent frameworks
- Website: https://github.com/langchain-ai/langchain
- Open source: yes
- Tags: python, typescript

### LangGraph

Page: https://aifordevs.org/platforms#langgraph | Markdown: https://aifordevs.org/platforms/langgraph.md

Build agents as stateful graphs with checkpoints, human-in-the-loop steps, and streaming.

- Maker: LangChain
- Category: Agent frameworks
- Website: https://github.com/langchain-ai/langgraph
- Open source: yes
- Tags: python, typescript, stateful

### LlamaIndex

Page: https://aifordevs.org/platforms#llamaindex | Markdown: https://aifordevs.org/platforms/llamaindex.md

Framework for agents over your data, with document parsing, indexing, and retrieval built in.

- Maker: LlamaIndex
- Category: Agent frameworks
- Website: https://github.com/run-llama/llama_index
- Open source: yes
- Tags: rag, python

### OpenAI Agents SDK

Page: https://aifordevs.org/platforms#openai-agents-sdk | Markdown: https://aifordevs.org/platforms/openai-agents-sdk.md

Lightweight SDK for agents with handoffs, guardrails, sessions, and built-in tracing.

- Maker: OpenAI
- Category: Agent frameworks
- Website: https://github.com/openai/openai-agents-python
- Open source: yes
- Tags: python, typescript

### Pydantic AI

Page: https://aifordevs.org/platforms#pydantic-ai | Markdown: https://aifordevs.org/platforms/pydantic-ai.md

Type-safe Python agent framework from the Pydantic team, with validated structured outputs.

- Maker: Pydantic
- Category: Agent frameworks
- Website: https://github.com/pydantic/pydantic-ai
- Open source: yes
- Tags: python, typed

## Gateways

### Bifrost

Page: https://aifordevs.org/platforms#bifrost | Markdown: https://aifordevs.org/platforms/bifrost.md

Open-source AI gateway written in Go, with one API for many providers plus failover, caching, and governance.

- Maker: Maxim AI
- Category: Gateways
- Website: https://github.com/maximhq/bifrost
- Open source: yes
- Tags: go, mcp gateway

### Cloudflare AI Gateway

Page: https://aifordevs.org/platforms#cloudflare-ai-gateway | Markdown: https://aifordevs.org/platforms/cloudflare-ai-gateway.md

Managed gateway on Cloudflare's network with caching, rate limiting, retries, and analytics.

- Maker: Cloudflare
- Category: Gateways
- Website: https://developers.cloudflare.com/ai-gateway/
- Tags: managed, edge

### LiteLLM

Page: https://aifordevs.org/platforms#litellm | Markdown: https://aifordevs.org/platforms/litellm.md

Python SDK and proxy server that call many providers through the OpenAI format.

- Maker: BerriAI
- Category: Gateways
- Website: https://github.com/BerriAI/litellm
- Open source: yes
- Tags: python, proxy

## Observability and evals

### Langfuse

Page: https://aifordevs.org/platforms#langfuse | Markdown: https://aifordevs.org/platforms/langfuse.md

Open-source LLM engineering platform for tracing, prompt management, and evaluations.

- Maker: Langfuse
- Category: Observability and evals
- Website: https://github.com/langfuse/langfuse
- Open source: yes
- Tags: tracing, self-hostable

### Maxim AI

Page: https://aifordevs.org/platforms#maxim-ai | Markdown: https://aifordevs.org/platforms/maxim-ai.md

Platform for simulating, evaluating, and observing AI agents before and after release.

- Maker: Maxim AI
- Category: Observability and evals
- Website: https://www.getmaxim.ai
- Tags: evals, simulation

### Phoenix

Page: https://aifordevs.org/platforms#arize-phoenix | Markdown: https://aifordevs.org/platforms/arize-phoenix.md

Source-available tracing and evaluation tool built on OpenTelemetry, which you can run locally.

- Maker: Arize AI
- Category: Observability and evals
- Website: https://github.com/Arize-ai/phoenix
- Tags: opentelemetry, tracing

## Vector databases

### Chroma

Page: https://aifordevs.org/platforms#chroma | Markdown: https://aifordevs.org/platforms/chroma.md

Embedding database that runs in-process for prototypes and as a server or cloud service in production.

- Maker: Chroma
- Category: Vector databases
- Website: https://github.com/chroma-core/chroma
- Open source: yes
- Tags: python, embedded

### Milvus

Page: https://aifordevs.org/platforms#milvus | Markdown: https://aifordevs.org/platforms/milvus.md

Distributed vector database built for billions of vectors, with GPU-accelerated indexes.

- Maker: Zilliz
- Category: Vector databases
- Website: https://github.com/milvus-io/milvus
- Open source: yes
- Tags: distributed, scale

### pgvector

Page: https://aifordevs.org/platforms#pgvector | Markdown: https://aifordevs.org/platforms/pgvector.md

Postgres extension for vector similarity search, so embeddings live next to your relational data.

- Maker: pgvector
- Category: Vector databases
- Website: https://github.com/pgvector/pgvector
- Open source: yes
- Tags: postgres

### Pinecone

Page: https://aifordevs.org/platforms#pinecone | Markdown: https://aifordevs.org/platforms/pinecone.md

Managed serverless vector database with namespaces, metadata filtering, and hybrid search.

- Maker: Pinecone
- Category: Vector databases
- Website: https://www.pinecone.io
- Tags: managed, serverless

### Qdrant

Page: https://aifordevs.org/platforms#qdrant | Markdown: https://aifordevs.org/platforms/qdrant.md

Vector database written in Rust, with payload filtering and a managed cloud option.

- Maker: Qdrant
- Category: Vector databases
- Website: https://github.com/qdrant/qdrant
- Open source: yes
- Tags: rust, filtering

### Weaviate

Page: https://aifordevs.org/platforms#weaviate | Markdown: https://aifordevs.org/platforms/weaviate.md

Vector database with built-in hybrid search and modules for generating embeddings.

- Maker: Weaviate
- Category: Vector databases
- Website: https://github.com/weaviate/weaviate
- Open source: yes
- Tags: hybrid search

## Cite the directory

AI for Devs. "AI for Devs platform directory." https://aifordevs.org/platforms

- Page: https://aifordevs.org/platforms
- Markdown: https://aifordevs.org/platforms.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs

---
title: "AI for Devs news"
type: "news"
url: "https://aifordevs.org/news"
markdown: "https://aifordevs.org/news.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/news"
updated: "2026-11-09"
---

# AI for Devs news

Dated briefs on model releases, API and pricing changes, tooling, infrastructure, and research, newest first. Each one links to its primary source.

## Anthropic releases Claude Fable 5.2 at lower per-token prices

Page: https://aifordevs.org/news#claude-fable-5-2 | Markdown: https://aifordevs.org/news/claude-fable-5-2.md

The claude-fable-5-2 model costs $8 input and $40 output per million tokens, down from $10 and $50 for Fable 5.1, with a 1M-token context and 128K max output. Before you switch, check any code that edits history. Changing the system prompt, tools, or earlier messages and then replaying a later thinking block returns a 400 on every account.

- Date: 2026-11-09
- Beat: Models
- Organizations: Anthropic
- Primary source: [Introducing Claude Fable 5.2 and Claude Mythos 5.2](https://www.anthropic.com/claude-fable-and-mythos-5-2)
- Also see: [Claude Platform release notes](https://platform.claude.com/docs/en/release-notes/overview#claude-fable-5-2)

## Meta adds Muse Ember, its smallest model, to the Meta Model API

Page: https://aifordevs.org/news#muse-ember | Markdown: https://aifordevs.org/news/muse-ember.md

It costs $0.10 per million input tokens and $0.50 per million output tokens, with a 1M-token context window and up to 512K output tokens. Meta says it matches Claude Haiku 5.5's best Artificial Analysis score at about a third of the cost, though Meta's chart labels Ember's per-task costs as estimates at proposed pricing.

- Date: 2026-11-09
- Beat: Models
- Organizations: Meta
- Primary source: [Introducing Muse Ember](https://research.meta.ai/blog/muse-ember)

## Anthropic adds fast mode for Claude Fable 5.2 on the Messages API

Page: https://aifordevs.org/news#claude-fable-5-2-fast-mode | Markdown: https://aifordevs.org/news/claude-fable-5-2-fast-mode.md

Fast mode for Claude Fable 5.2 costs $16 per million input tokens and $80 per million output tokens on the Claude API. Anthropic reports up to 2.5x higher output tokens per second. You turn it on by setting speed: "fast" on a request, with no beta header.

- Date: 2026-11-09
- Beat: APIs
- Organizations: Anthropic
- Primary source: [Claude Platform release notes, November 9, 2026](https://platform.claude.com/docs/en/release-notes/overview#fast-mode-claude-fable-5-2)

## GitHub secret scanning now checks GitHub Actions workflow logs

Page: https://aifordevs.org/news#github-secret-scanning-actions-logs | Markdown: https://aifordevs.org/news/github-secret-scanning-actions-logs.md

GitHub secret scanning now scans GitHub Actions workflow logs, in public preview for GitHub Enterprise Cloud with Secret Protection or Advanced Security. Enterprise owners turn on the policy, which is off by default. Only runs that finish after you enable it are scanned, and preview alerts send no notifications.

- Date: 2026-11-09
- Beat: Tooling
- Organizations: GitHub
- Primary source: [Secret scanning detects secrets in GitHub Actions workflow logs](https://github.blog/changelog/2026-11-09-secret-scanning-detects-secrets-in-github-actions-workflow-logs)

## GitHub opens Copilot code review settings to its REST and GraphQL APIs

Page: https://aifordevs.org/news#copilot-code-review-api | Markdown: https://aifordevs.org/news/copilot-code-review-api.md

You can now manage code review settings through the APIs from enterprise policy down to individual repositories, and see which skills, MCP servers, and custom instructions Copilot reported as sources. Enterprise and organization endpoints don't support GitHub Apps or fine-grained personal access tokens; repository endpoints support both.

- Date: 2026-11-06
- Beat: APIs
- Organizations: GitHub
- Primary source: [GitHub Changelog: Copilot code review API management](https://github.blog/changelog/2026-11-06-copilot-code-review-api-management-and-review-overview-updates)

## Google routes gemini-embedding-2-preview requests to gemini-embedding-2

Page: https://aifordevs.org/news#gemini-embedding-2-preview-retired | Markdown: https://aifordevs.org/news/gemini-embedding-2-preview-retired.md

As of November 6, 2026, Gemini API requests to gemini-embedding-2-preview route to gemini-embedding-2 and are billed at Gemini Embedding 2 rates. Google says the stable model uses the same API schema, pricing, and embedding space, so you only need to update the model string, with no vector re-indexing.

- Date: 2026-11-06
- Beat: APIs
- Organizations: Google
- Primary source: [Gemini API changelog, November 6, 2026](https://aistudio.google.com/docs/changelog#11-06-2026)

## Gemini Omni 1.2 Flash is generally available with 15-second video

Page: https://aifordevs.org/news#gemini-omni-1-2-flash-ga | Markdown: https://aifordevs.org/news/gemini-omni-1-2-flash-ga.md

It adds native clips up to 15 seconds and multi-turn editing on the Gemini API at $0.07 per second for 720p video, 30% below Omni 1.1, with doubled paid-tier rate limits. Requests to gemini-omni-1.1-flash start routing to gemini-omni-1.2-flash at its rates on December 3, 2026, so test any pinned prompts before then.

- Date: 2026-11-05
- Beat: Models
- Organizations: Google DeepMind
- Primary source: [Gemini Omni 1.2 Flash announcement](https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemini-omni-1-2-flash/)
- Also see: [Gemini API changelog](https://aistudio.google.com/docs/changelog#11-05-2026)

## OpenAI limits GPT-6 Cyber to approved Daybreak Red customers

Page: https://aifordevs.org/news#gpt-6-cyber | Markdown: https://aifordevs.org/news/gpt-6-cyber.md

The gpt-6-cyber model is available through the API and Codex to a select group, with wider Daybreak Red access planned. OpenAI reports 48.5% on ExploitGym, up from 32.1% for GPT-5.6 Cyber, and classifies the model as Critical for cybersecurity under its Preparedness Framework. Daybreak Red API access requires IP allowlisting by default.

- Date: 2026-11-05
- Beat: Models
- Organizations: OpenAI
- Primary source: [GPT-6 Cyber announcement](https://openai.com/index/introducing-gpt-6-cyber)
- Also see: [OpenAI API changelog](https://developers.openai.com/api/docs/changelog)

## Cursor cloud agents can now build and deploy internal apps

Page: https://aifordevs.org/news#cursor-live-apps | Markdown: https://aifordevs.org/news/cursor-live-apps.md

Ask a cloud agent for an app and Cursor scaffolds it, adds Postgres if needed, puts authentication behind Cursor SSO, and provisions hosting. Nothing goes live until security checks pass and you publish, and team admins can see every app and secret. It is available on all paid plans with cloud agents enabled.

- Date: 2026-11-05
- Beat: Tooling
- Organizations: Cursor
- Primary source: [Build Live Apps in Cursor](https://cursor.com/changelog/build-live-apps)

## Mistral Studio adds managed agents and a one-command MCP deploy

Page: https://aifordevs.org/news#mistral-studio-managed-agents | Markdown: https://aifordevs.org/news/mistral-studio-managed-agents.md

You can now build agents with skills and code sandboxes, then run them alongside workflows and apps in the cloud, your own VPC, or on premises. A new Mistral MCP server lets you build with Mistral Vibe or another coding agent, test locally, and deploy to a Studio environment with one command.

- Date: 2026-11-05
- Beat: Tooling
- Organizations: Mistral AI
- Primary source: [Le Chonk brings the brains. Studio puts it to work.](https://mistral.ai/news/build-with-studio/)

## Anthropic adds an organization IP allowlist to the Claude Console in beta

Page: https://aifordevs.org/news#claude-console-ip-allowlist | Markdown: https://aifordevs.org/news/claude-console-ip-allowlist.md

Claude Console organizations can now set an IP allowlist, in beta. Requests to covered services from addresses not on the list are refused, which limits where a leaked API key works. You can add IPv4 and IPv6 addresses or CIDR ranges, up to 1,000 per organization.

- Date: 2026-11-05
- Beat: APIs
- Organizations: Anthropic
- Primary source: [Claude Platform release notes, November 5, 2026](https://platform.claude.com/docs/en/release-notes/overview#ip-allowlisting)

## OpenAI's Images 2.6 brings two API models and SVG output

Page: https://aifordevs.org/news#openai-images-2-6 | Markdown: https://aifordevs.org/news/openai-images-2-6.md

The gpt-image-2.6-flare and gpt-image-2.6-sunburst models work on the generations and edits endpoints, with adjustable complexity, SVG generation, and animated GIFs. OpenAI says Images 2.6 generates 50% faster than before and that Flare can return an image in as little as six seconds at low quality.

- Date: 2026-11-04
- Beat: APIs
- Organizations: OpenAI
- Primary source: [Introducing Images 2.6](https://openai.com/index/introducing-images-2-6)
- Also see: [OpenAI API changelog](https://developers.openai.com/api/docs/changelog)

## GitHub Copilot joins Xcode's built-in coding assistant in preview

Page: https://aifordevs.org/news#copilot-xcode-coding-assistant | Markdown: https://aifordevs.org/news/copilot-xcode-coding-assistant.md

The public preview starts with Xcode 27.3 beta, needs macOS Tahoe 26.6 or later, and works on all Copilot plans. You set it up in Xcode's Intelligence settings. The existing GitHub Copilot for Xcode extension stays available as a separate option.

- Date: 2026-11-04
- Beat: Tooling
- Organizations: GitHub
- Primary source: [GitHub Changelog: Copilot in Xcode's coding assistant](https://github.blog/changelog/2026-11-04-github-copilot-is-now-available-in-xcodes-coding-assistant)

## Claude API adds message threads in beta to cut time to first token

Page: https://aifordevs.org/news#claude-message-threads-beta | Markdown: https://aifordevs.org/news/claude-message-threads-beta.md

With the message-threads-2026-08-12 header, later requests send only new messages plus the previous response ID, and the server keeps the context for 5 minutes after the last request. Pricing does not change; Anthropic reports about 35% lower median time to first token on 500K to 1M token requests in Claude Code, and the feature is not eligible for zero data retention.

- Date: 2026-11-02
- Beat: APIs
- Organizations: Anthropic
- Primary source: [Message threads, Claude Platform Docs](https://platform.claude.com/docs/en/build-with-claude/message-threads)
- Also see: [Anthropic: message threads](https://claude.com/resources/articles/message-threads)

## vLLM v0.33.0 makes CUDA 13.0 the default and removes Medusa

Page: https://aifordevs.org/news#vllm-0-33 | Markdown: https://aifordevs.org/news/vllm-0-33.md

The release stops publishing CUDA 12.9 wheels and images, moves to PyTorch 2.14, makes ROCm 10.0 the default AMD build, and removes Medusa, the MLP speculator, and the legacy Ray compiled-graph executor. Teams still on CUDA 12.x or Medusa-based speculative decoding need an upgrade plan before they bump the version.

- Date: 2026-11-02
- Beat: Infrastructure
- Organizations: vLLM project
- Primary source: [vLLM v0.33.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0)

## Google deprecates gemini-2.5-pro and gemini-2.5-flash

Page: https://aifordevs.org/news#gemini-2-5-deprecation | Markdown: https://aifordevs.org/news/gemini-2-5-deprecation.md

Starting December 1, 2026, calls to the two models auto-route to gemini-3.8-flash and gemini-3.5-flash-lite. If you pin model IDs for evals or cost reasons, rerun your tests on the new models before the switch happens for you.

- Date: 2026-11-02
- Beat: APIs
- Organizations: Google
- Primary source: [Gemini API changelog](https://aistudio.google.com/docs/changelog)

## MIT CSAIL report has an LLM edit another model's weights directly

Page: https://aifordevs.org/news#weight-space-programming | Markdown: https://aifordevs.org/news/weight-space-programming.md

In "Sparks of Weight-Space Programming", GPT-6 Astra edited 131,072 weights (0.016%) of Qwen3.5-0.8B from a natural-language goal, with no gradient descent, and MATH-500 rose from 7.9% to 48.1%. The authors call the results early and small-scale; the report is not peer reviewed, and an arXiv critique questions whether the weight reading reflects memorization.

- Date: 2026-11-02
- Beat: Research
- Organizations: MIT CSAIL
- Primary source: [Sparks of Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/)
- Also see: [Critique on arXiv](https://arxiv.org/html/2611.08070)

## SGLang 0.5.23 adds TPU7x support and drops legacy checkpoint loading

Page: https://aifordevs.org/news#sglang-0-5-23 | Markdown: https://aifordevs.org/news/sglang-0-5-23.md

The SGLang project reports up to 1.5x more prefill throughput and about 28% lower first-token latency for DeepSeek-V4.1 on GB300 with 32K to 64K prompts. Before upgrading, convert legacy tar-format .pt checkpoints to safetensors or zip-format .pt, and drop --speculative-skip-dp-mlp-sync from launch commands, which now fail with it.

- Date: 2026-11-02
- Beat: Infrastructure
- Organizations: SGLang
- Primary source: [SGLang v0.5.23 release notes](https://github.com/sgl-project/sglang/releases/tag/v0.5.23)

## Anthropic will remove fast mode for Claude Opus 4.8 on November 30

Page: https://aifordevs.org/news#claude-opus-4-8-fast-mode-removal | Markdown: https://aifordevs.org/news/claude-opus-4-8-fast-mode-removal.md

Fast mode for Claude Opus 4.8 is deprecated and will be removed on November 30, 2026. After that date, requests to claude-opus-4-8 with speed: "fast" return an error. If you rely on fast mode, migrate those requests to Claude Opus 5.5 before the removal date.

- Date: 2026-10-30
- Beat: APIs
- Organizations: Anthropic
- Primary source: [Claude Platform release notes, October 30, 2026](https://platform.claude.com/docs/en/release-notes/overview#october-30-2026)

## Mistral publishes open weights for Mistral Large 4

Page: https://aifordevs.org/news#mistral-large-4-open-weights | Markdown: https://aifordevs.org/news/mistral-large-4-open-weights.md

The multimodal mixture-of-experts model has about 1.05T total and 52B active parameters, a 1M-token context, and runs on vLLM, SGLang, llama.cpp, and Transformers, with an NVFP4 checkpoint for Blackwell GPUs. It ships under the Mistral Ecosystem Open-Weight License, which requires a separate commercial license for model-as-a-service businesses.

- Date: 2026-10-29
- Beat: Models
- Organizations: Mistral AI
- Primary source: [Mistral Large 4 is now open weight](https://mistral.ai/news/mistral-large-4-oss/)
- Also see: [Model card on Hugging Face](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B)

## GitHub Copilot Fusion multi-model routing is generally available

Page: https://aifordevs.org/news#copilot-fusion-ga | Markdown: https://aifordevs.org/news/copilot-fusion-ga.md

You pick an effort level (Standard, High, or Max) and Fusion routes each prompt through a Single, Cascade, or Critique pattern in VS Code, Copilot CLI, JetBrains IDEs, and the new Copilot app. Each underlying model is billed at its standard rate, and admins can exclude models or make Fusion the default.

- Date: 2026-10-28
- Beat: Tooling
- Organizations: GitHub
- Primary source: [GitHub Changelog: Copilot Fusion](https://github.blog/changelog/2026-10-28-route-and-orchestrate-models-with-github-copilot-fusion)

## Octoverse 2026: agents drafted over 40% of September pull requests

Page: https://aifordevs.org/news#octoverse-2026 | Markdown: https://aifordevs.org/news/octoverse-2026.md

GitHub found that a detectable coding agent wrote the first draft of more than 40% of pull requests in September 2026, up from 17% in March, and that 90% of agent-started PRs were merged within two weeks. GitHub says detection is a floor, and the report measures acceptance, not code quality.

- Date: 2026-10-28
- Beat: Research
- Organizations: GitHub
- Primary source: [Octoverse 2026](https://github.blog/news-insights/octoverse/octoverse-commits-pushes-reach-new-highs-in-2026-as-developers-build-with-agents/)

## OpenAI ships GPT-6.1 Terra at $1 and $5 per million tokens

Page: https://aifordevs.org/news#gpt-6-1-terra | Markdown: https://aifordevs.org/news/gpt-6-1-terra.md

The medium-size model for visual reasoning, computer use, and writing is available on the Responses, Chat Completions, and Batch APIs with a 1,050,000-token context and 128K max output. The model page lists $1 input, $0.05 cached input, and $5 output per million tokens.

- Date: 2026-10-26
- Beat: Models
- Organizations: OpenAI
- Primary source: [GPT-6.1 Terra model page](https://developers.openai.com/api/docs/models/gpt-6.1-terra)
- Also see: [OpenAI developer changelog](https://developers.openai.com/changelog/)

## Ollama adds local text-to-speech with an OpenAI-compatible endpoint

Page: https://aifordevs.org/news#ollama-speech-models | Markdown: https://aifordevs.org/news/ollama-speech-models.md

Ollama 0.40.11 and later can run Gradium's Phonon speech models behind POST /v1/audio/speech, so code written for OpenAI's speech API can point at localhost. Support is macOS on Apple silicon only for now.

- Date: 2026-10-24
- Beat: Infrastructure
- Organizations: Ollama, Gradium
- Primary source: [Speech models in Ollama](https://ollama.com/blog/speech-models)

## Anthropic donates the Agent Skills standard to the Agentic AI Foundation

Page: https://aifordevs.org/news#agent-skills-aaif | Markdown: https://aifordevs.org/news/agent-skills-aaif.md

The Linux Foundation project that already hosts MCP now owns the Agent Skills name, the agentskills GitHub org, and agentskills.io. A skill stays a folder with a SKILL.md file plus optional scripts, and the proposed roadmap adds versioned releases and security guidance for installing skills.

- Date: 2026-10-22
- Beat: APIs
- Organizations: Anthropic, Agentic AI Foundation
- Primary source: [Donating Agent Skills to the Agentic AI Foundation](https://www.anthropic.com/news/donating-agent-skills-to-the-agentic-ai-foundation)

## OpenAI's Decisions API for typed answers is generally available

Page: https://aifordevs.org/news#openai-decisions-api-ga | Markdown: https://aifordevs.org/news/openai-decisions-api-ga.md

POST /v1/decisions with gpt-6-luna returns typed probabilities, choices, or scores for text, image, and JSON input, now with up to 400 questions per request. It costs $0.10 per million uncached input tokens with no output charge, which fits classification, routing, and eval-judging work.

- Date: 2026-10-22
- Beat: APIs
- Organizations: OpenAI
- Primary source: [Decisions API guide](https://developers.openai.com/api/docs/guides/decisions)
- Also see: [OpenAI developer changelog](https://developers.openai.com/changelog/)

## Cursor's Agent SDK for custom agents is generally available

Page: https://aifordevs.org/news#cursor-agent-sdk-ga | Markdown: https://aifordevs.org/news/cursor-agent-sdk-ga.md

An agent is a folder with instructions.md, typed tools, skills, MCP connections, and channels; you test it with evals and pull-request replays, then deploy it with agent-sdk deploy. Agents can run on GitHub, GitLab, or Bitbucket events, Slack messages, webhooks, or a cron schedule.

- Date: 2026-10-20
- Beat: Tooling
- Organizations: Cursor
- Primary source: [Cursor changelog](https://cursor.com/changelog)

## OpenAI adds opt-in 12-hour prompt caching to the Responses API

Page: https://aifordevs.org/news#openai-12h-prompt-caching | Markdown: https://aifordevs.org/news/openai-12h-prompt-caching.md

Cache writes with the 12-hour lifetime cost 2.5 times the input token price. It pays off for long, stable prefixes such as agent system prompts or shared documents that are reused many times a day.

- Date: 2026-10-20
- Beat: APIs
- Organizations: OpenAI
- Primary source: [OpenAI developer changelog](https://developers.openai.com/changelog/)

## Meta releases Muse Spark Open, an 816B mixture-of-experts model

Page: https://aifordevs.org/news#muse-spark-open | Markdown: https://aifordevs.org/news/muse-spark-open.md

The model has 42B active parameters and a 1M-token context, with day-one support in SGLang, vLLM, TensorRT-LLM, Transformers, MLX, and llama.cpp. The Muse Model License limits commercial use, so read it before you build on the weights.

- Date: 2026-10-15
- Beat: Models
- Organizations: Meta
- Primary source: [Introducing Muse Spark Open](https://research.meta.ai/blog/muse-spark-open)

## Gemini 4 Argon is generally available in the Gemini API

Page: https://aifordevs.org/news#gemini-4-argon-ga | Markdown: https://aifordevs.org/news/gemini-4-argon-ga.md

It has a 1,048,576-token context and supports function calling, structured outputs, code execution, and computer use. Introductory pricing of $2 input and $10 output per million tokens runs through November 30, 2026, then doubles, so budget at the higher rate.

- Date: 2026-10-13
- Beat: Models
- Organizations: Google DeepMind
- Primary source: [Build with Gemini 4 Argon](https://blog.google/innovation-and-ai/technology/developers-tools/build-with-gemini-4-argon/)
- Also see: [Gemini API pricing](https://aistudio.google.com/docs/pricing)

## Anthropic releases Claude Haiku 5.5 with tiered pricing

Page: https://aifordevs.org/news#claude-haiku-5-5 | Markdown: https://aifordevs.org/news/claude-haiku-5-5.md

It costs $0.10 input and $0.50 output per million tokens for prompts up to 100K tokens, and $0.50 and $2.50 above that, with a 1M-token context and 128K max output. It is the first Haiku model with an effort setting; the same announcement cut Sonnet 5.5 cache reads by 50% to $0.10 per million tokens.

- Date: 2026-10-07
- Beat: Models
- Organizations: Anthropic
- Primary source: [Introducing Claude Haiku 5.5](https://www.anthropic.com/claude-haiku-5-5)
- Also see: [Models overview, Claude Platform Docs](https://platform.claude.com/docs/en/about-claude/models/overview)

## Cite the news page

AI for Devs. "AI for Devs news." https://aifordevs.org/news

- Page: https://aifordevs.org/news
- Markdown: https://aifordevs.org/news.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs