---
title: "GGUF"
type: "glossary term"
url: "https://aifordevs.org/glossary#gguf"
markdown: "https://aifordevs.org/glossary/gguf.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#gguf"
---

# GGUF

GGUF is the single-file model format used by llama.cpp and by tools that read it, such as Ollama. A GGUF file holds the weights, often quantized to 4 to 8 bits, along with the tokenizer and metadata needed to run the model.

A GGUF file has a header, key-value metadata, tensor info, and tensor data, so a runtime needs no external files to load the model. The spec describes version 3 of the format. Its tokenizer.ggml keys hold the vocabulary, and the spec warns that those keys are not guaranteed to be standardized across models.

## Example

To run a local file in Ollama, write FROM /path/to/file.gguf in a Modelfile and run ollama create my-model. Ollama does not quantize the file on import, so quantize it first with llama-quantize.

## Source

[ggml docs: GGUF specification](https://raw.githubusercontent.com/ggml-org/ggml/master/docs/gguf.md)

## Related terms

- [Quantization](https://aifordevs.org/glossary#quantization)
- [Inference](https://aifordevs.org/glossary#inference)
- [Small language model](https://aifordevs.org/glossary#small-language-model)

## Read the article

[vLLM vs SGLang, plus Ollama and llama.cpp compared](https://aifordevs.org/articles/vllm-sglang-llama-cpp)

## Cite this term

AI for Devs. "GGUF." https://aifordevs.org/glossary#gguf

- Page: https://aifordevs.org/glossary#gguf
- Markdown: https://aifordevs.org/glossary/gguf.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs