vLLM vs SGLang, plus Ollama and llama.cpp compared

vLLM and SGLang are open-source inference engines that run open-weight models on GPU servers behind OpenAI-compatible APIs. The vLLM vs SGLang decision usually turns on hardware, cache behavior, and which engine supports your model on the day you need it. Ollama and llama.cpp solve a different problem: one machine, few users, and quantized weights. This comparison checks each engine's defaults against its own docs and release notes as of November 8, 2026.

#vLLM vs SGLang at a glance

Start with vLLM if you want the safest default for multi-user GPU serving, because its README documents 200+ model architectures and a long list of hardware plugins. Start with SGLang if agent loops and multi-turn chat dominate your traffic, because its radix cache matches prefixes at token granularity. Both are Apache-2.0.

All four expose OpenAI-compatible routes, so moving between them is usually a base URL change. Extras such as sampling options and structured output fields differ, and the inference platforms directory lists hosted options if you would rather not run any of them.

#SGLang vs vLLM on prefix caching

Both engines keep KV cache entries from earlier requests and skip prefill for any prefix they have already seen. vLLM finds those entries with a chain of block hashes, and SGLang finds them by walking a radix tree of token IDs. The structures differ in match granularity, eviction choices, and what you can tune.

#vLLM: hashed blocks

vLLM's design doc hashes each full block together with its parent block's hash, so a block's identity depends on everything before it. It caches full blocks only. Eviction takes the least recently used block from a free queue.

The block size is 16 tokens in v0.33.0's CacheConfig, where enable_prefix_caching defaults to true. Pull request 63231, merged on November 5, 2026, raises the default on main to 64 and lists coarser prefix hits as a trade-off. Check the block size again when you upgrade.

Per-request cache_salt values feed the first block's hash, which isolates tenants that should never share cached prefixes.

#SGLang: a radix tree

SGLang's RadixAttention keeps KV cache for prompts and generations in a radix tree, mapping token sequences to cache tensors on the GPU while the tree lives on the CPU. The current server arguments set --page-size to 1 and leave --disable-radix-cache off, so token-level reuse is on by default.

The scheduler is less cache-driven by default than older write-ups suggest. --schedule-policy defaults to fcfs, and the longest-prefix-match policy lpm is something you opt into. Eviction defaults to LRU over leaf nodes, with lfu, slru, priority, and tlru policies for skewed traffic.

#Granularity matters most for short prefixes

The gap between the two designs is smaller than a diagram suggests. A vLLM block of 16 tokens forfeits at most 15 tokens per request, which is 0.4% of a 4,000-token shared prompt. At the new 64-token default, the worst case is 63 tokens, or 1.6%.

Granularity matters when shared prefixes are short, or when many branches diverge just past a block boundary. Measure it instead of guessing. vLLM exposes vllm:prefix_cache_queries and vllm:prefix_cache_hits in its metrics, and SGLang's --enable-cache-report adds cached token counts to each response. To raise hit rates on either engine, put stable content first.

#Concurrent requests: batches, slots, and a queue

The engines differ most in what happens when four requests arrive at once. vLLM and SGLang batch them into shared GPU steps. llama-server assigns each one to a slot in a single process. Ollama runs one request per model and queues the rest, unless you change its default parallelism setting.

vLLM's PagedAttention paper reports 2 to 4 times the throughput of FasterTransformer and Orca at the same latency, a figure from the vLLM authors on 2023 models. Continuous batching is why short requests do not wait for long ones to finish.

llama-server has continuous batching on by default too. Posts that describe llama.cpp as strictly sequential are out of date; the limit is slot count and memory. On the test machine, the automatic -np -1 setting created 4 slots sharing one unified KV buffer.

#Hardware and model format support

vLLM and SGLang now cover far more than NVIDIA and AMD, and llama.cpp lists more consumer backends than the others. Check the platform you own against this table first, because a missing backend ends the comparison. Entries come from each project's own README and install docs on November 8, 2026.

The docs and release notes disagree on CUDA. The v0.33.0 release notes say release wheels and images are CUDA 13.0 only, while the GPU install page still shows CUDA 12.9 on November 8, 2026. Trust the release notes, and match your drivers to them.

#Model formats

vLLM and SGLang load Hugging Face weights. vLLM's GGUF support is "highly experimental" by its own docs and requires the separate vllm-gguf-plugin, and SGLang lists gguf as a --load-format option.

llama.cpp reads GGUF files and pulls them from Hugging Face with -hf. Its README lists 1.5-bit through 8-bit integer quantization for lower memory use. Ollama imports GGUF files and Safetensors directories through a Modelfile, and it runs GGUF through llama.cpp alongside its MLX engine on Apple silicon.

#Constrained decoding across the four engines

All four engines can force output to match a JSON schema, but only vLLM, SGLang, and llama.cpp document grammars beyond JSON. If you need regex or EBNF, Ollama is out, and on vLLM and SGLang the backend you choose sets the accepted syntax. This matters for structured outputs at scale.

Two details cause migrations to fail. vLLM deprecated lm-format-enforcer in v0.33.0 and plans to remove it in v0.35.0, and SGLang accepts only one of json_schema, regex, or ebnf per request.

Constrained decoding also works on small local models. The llama-server run in the Ollama vs llama.cpp section below enforces a schema through the OpenAI-style response_format field on a 0.8B model.

#vLLM vs Ollama: a shared GPU server or a local daemon

Use Ollama when one developer runs models on one machine, and use vLLM when several people or services share a GPU. Ollama optimizes for pulling and running a model with one command. vLLM optimizes for batching concurrent traffic on Linux GPUs, and it asks for more setup in return.

Ollama's FAQ spells out its defaults: OLLAMA_NUM_PARALLEL is 1 per model, OLLAMA_MAX_QUEUE is 512, models stay loaded for 5 minutes, and the context window is 4,096 tokens. Required RAM scales with parallelism times context length, so check OLLAMA_CONTEXT_LENGTH before you send a RAG prompt longer than 4,096 tokens.

vLLM's route list covers OpenAI chat, completions, responses, and embeddings, plus Anthropic /v1/messages, rerank, and score endpoints. Those routes help once coding agents and internal tools hit the same server. vLLM also needs Python 3.11 to 3.14 and Linux for GPU serving.

Move to vLLM when concurrency stops fitting within OLLAMA_NUM_PARALLEL, or when the model you want has no GGUF build. Ollama can stay on developer laptops while production points at vLLM through a different base URL.

#Ollama vs llama.cpp: convenience or control

Ollama is a daemon and model library that uses llama.cpp for GGUF models, while llama.cpp is the inference library plus a server you configure with flags. Pick Ollama when you want pull-and-run. Pick llama-server when you need to set slots, grammars, cache behavior, or a specific build.

The most visible default difference is parallelism. The test below measures it on llama-server by comparing one slot against the automatic setting. Ollama itself was not run for this article, so -np 1 shows the effect of serving one request at a time, and it says nothing about Ollama's speed.

Tested on an Apple M2 with 16 GB of memory, llama.cpp build b12202 compiled from source with Metal, and ggml-org/gemma-3-270m-it-GGUF at Q8_0 (292 MB). The machine was running other jobs, so read the ratios, not the absolute numbers.

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF \
  -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --target llama-server -j 7
 
LLAMA_CACHE=./models ./build/bin/llama-server -hf ggml-org/gemma-3-270m-it-GGUF

"""Send the same chat request to any OpenAI-compatible server at two concurrency levels."""
import json, sys, time, urllib.request
from concurrent.futures import ThreadPoolExecutor
 
BASE = sys.argv[1]  # for example http://localhost:9931
MODEL = sys.argv[2]
REQUESTS, MAX_TOKENS = 8, 128
 
 
def chat(i):
    body = {
        "model": MODEL,
        "messages": [{"role": "user", "content": f"Request {i}: list facts about caches."}],
        "max_tokens": MAX_TOKENS,
        "temperature": 0,
        "ignore_eos": True,  # llama-server extra: every request decodes MAX_TOKENS
    }
    req = urllib.request.Request(
        f"{BASE}/v1/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=600) as r:
        usage = json.load(r)["usage"]
    return usage["completion_tokens"], time.perf_counter() - start
 
 
for workers in (1, 4):
    start = time.perf_counter()
    with ThreadPoolExecutor(workers) as pool:
        results = list(pool.map(chat, range(REQUESTS)))
    wall = time.perf_counter() - start
    tokens = sum(t for t, _ in results)
    mean = sum(s for _, s in results) / len(results)
    print(f"concurrency={workers} requests={REQUESTS} tokens={tokens} "
          f"wall={wall:.2f}s throughput={tokens / wall:.1f} tok/s mean_latency={mean:.2f}s")

Real output, default slots and then -np 1:

# default: total_slots 4
concurrency=1 requests=8 tokens=1024 wall=23.06s throughput=44.4 tok/s mean_latency=2.88s
concurrency=4 requests=8 tokens=1024 wall=6.05s throughput=169.3 tok/s mean_latency=3.02s
# -np 1: total_slots 1
concurrency=1 requests=8 tokens=1024 wall=22.63s throughput=45.2 tok/s mean_latency=2.83s
concurrency=4 requests=8 tokens=1024 wall=22.72s throughput=45.1 tok/s mean_latency=9.21s

Four slots delivered 3.8 times the throughput of one slot here, and 3.2 times in an earlier round on the same machine (145.4 against 45.2 tokens per second). With one slot, mean latency for four concurrent requests was 9.2 seconds instead of 3.0.

Prefix reuse and JSON-schema decoding both work on llama-server, and reuse depends on the model. This script ran against ggml-org/Qwen3.5-0.8B-GGUF at Q8_0, started with --reasoning off:

"""Show llama-server's prefix reuse and JSON-schema constrained decoding."""
import json, urllib.request
 
URL = "http://localhost:9931/v1/chat/completions"
SYSTEM = "You answer questions about inference servers. " * 60
 
 
def post(**body):
    req = urllib.request.Request(URL, data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        return json.load(r)
 
 
for question in ("What is a slot?", "What is a grammar?"):
    out = post(model="gemma", max_tokens=8, temperature=0, messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": question},
    ])
    t = out["timings"]
    print(f"{question:20} cached={t['cache_n']:4} evaluated={t['prompt_n']:4} "
          f"prefill={t['prompt_ms']:.0f} ms")
 
schema = {
    "type": "object",
    "properties": {
        "engine": {"type": "string", "enum": ["vllm", "sglang", "llama.cpp", "ollama"]},
        "default_port": {"type": "integer"},
    },
    "required": ["engine", "default_port"],
    "additionalProperties": False,
}
out = post(model="gemma", max_tokens=64, temperature=0,
           messages=[{"role": "user", "content": "Name one inference engine and a port."}],
           response_format={"type": "json_schema",
                            "json_schema": {"name": "engine", "schema": schema}})
print(out["choices"][0]["message"]["content"])
print("parsed:", json.loads(out["choices"][0]["message"]["content"]))

What is a slot?      cached=   0 evaluated= 442 prefill=1170 ms
What is a grammar?   cached= 425 evaluated=  17 prefill=151 ms
{
  "engine": "vllm",
  "default_port": 8000
}
parsed: {'engine': 'vllm', 'default_port': 8000}

The second request reused 425 cached prompt tokens, and prefill fell from 1,170 ms to 151 ms. The JSON output parsed against the schema on the first try. The same script on Gemma 3 270M reused 1 token, and the verbose server log shows context checkpoints being invalidated on that model (n_swa = 512). Test prefix reuse on the model you plan to ship.

#vLLM vs llama.cpp: batching or portability

Choose vLLM when many requests share a GPU and throughput per dollar decides the bill. Choose llama.cpp when you need one binary on a laptop, an edge device, or a machine with no CUDA. The overlap is small, because vLLM treats GGUF as experimental and llama.cpp loads GGUF files.

A useful test is where your weights already live. If your team ships Hugging Face checkpoints at FP8 or NVFP4, stay with vLLM or SGLang. If you distribute a single quantized GGUF file to users' machines, llama.cpp's -hf flag and Metal, Vulkan, and CPU backends do the job with no Python environment.

llama.cpp's install docs list Homebrew, conda-forge, Winget, MacPorts, and Nix, and the README adds an install script and a Docker image. vLLM's route is a Python 3.11 to 3.14 environment and uv pip install vllm --torch-backend=auto, then vllm download-kernels.

#Start an OpenAI-compatible server with each engine

Each of these commands starts a server, and the same chat request then works against all four once you change the port and model name. Only the llama-server command ran for this article. The other three come from the vLLM and SGLang quickstarts and Ollama's quickstart and Linux install pages on November 8, 2026, and they need the hardware they name.

# NOT RUN HERE. vLLM quickstart, Linux GPU host, port 8000.
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
vllm serve Qwen/Qwen2.5-1.5B-Instruct
 
# NOT RUN HERE. SGLang quickstart, GPU host, port 30000.
uv venv --python 3.12 && source .venv/bin/activate
uv pip install --prerelease=allow sglang
sglang serve Qwen/Qwen3-0.6B --host 0.0.0.0 --port 30000
 
# NOT RUN HERE. Ollama Linux install and quickstart, port 11434.
curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma4:e2b
 
# RUN HERE on an Apple M2. llama-server, port 9931.
llama-server -hf ggml-org/gemma-3-270m-it-GGUF

curl http://localhost:9931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma", "messages": [{"role": "user", "content": "Explain the KV cache in one sentence."}], "max_tokens": 60, "temperature": 0}'

Ollama's OpenAI routes sit under http://localhost:11434/v1/. vLLM and SGLang use /v1 on their own ports. Running them on a shared host means binding to 0.0.0.0 on purpose, because SGLang, llama-server, and Ollama all default to 127.0.0.1.

#Published benchmarks are project-run and dated

Most public vLLM vs SGLang numbers were published by one of the two projects, on specific models, at versions that are now old. Treat them as evidence that a design works on one workload, and do not use them to predict your own latency. Newer per-feature numbers appear in release notes.

SemiAnalysis's InferenceX is the outside check to read next. It publishes continuous benchmark runs covering both engines, with code and data in the open. This article did not extract a head-to-head number from it, so filter by your GPU and model yourself.

Both projects keep adding speculative decoding methods and hardware-specific kernels, as the release notes above show, so rerun any comparison when you upgrade.

#Choose your inference engine with one replay test

Run one replay test before you commit to any inference engine, and let your own cache hit rate and concurrency pick the winner. A sample of real prompts answers the vLLM vs SGLang question in an afternoon, and it tells you whether a local daemon was ever going to be enough.

1. Capture 1,000 real requests, with prompts and output lengths, from production or a staging copy.
2. Compute the share of prompt tokens that already appeared earlier in the sample. That approximates your prefix hit rate.
3. On one GPU, serve the same model at the same dtype on vLLM v0.33.0 and SGLang v0.5.23, and replay at your target concurrency.
4. Record time to first token, time per output token, and tokens per second at your p95 target, and read each engine's cache metrics.

As a starting rule, not a measured threshold: when more than half of your prompt tokens repeat, try SGLang first. Otherwise start with vLLM. For one machine and one user, use llama-server if you need slots and grammars, and Ollama if you do not.

Vector stores make the same trade between convenience and control, which the vector database guide works through. New engine releases and benchmark changes land in the news feed and on the inference topic page.

#Questions