---
title: "Tokenizer"
type: "glossary term"
url: "https://aifordevs.org/glossary#tokenizer"
markdown: "https://aifordevs.org/glossary/tokenizer.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#tokenizer"
---

# Tokenizer

A tokenizer is the component that splits text into tokens and maps them to integer IDs before the model processes them. Different model families use different tokenizers, so the same text can produce a different number of tokens.

Byte pair encoding is a common method: it builds a vocabulary by repeatedly merging frequent adjacent symbol pairs, and Sennrich et al. (2015) applied it to rare words in neural translation. Count with the tokenizer that matches the model you call, because a mismatch gives wrong numbers.

## Example

OpenAI's o200k_base encoding splits "tiktoken is great!" into 6 tokens: t, ikt, oken, is, great, and !. That is 3 characters per token on this string.

## Source

[OpenAI Cookbook: How to count tokens with tiktoken](https://developers.openai.com/cookbook/examples/how_to_count_tokens_with_tiktoken)

## Related terms

- [Token](https://aifordevs.org/glossary#token)

## Cite this term

AI for Devs. "Tokenizer." https://aifordevs.org/glossary#tokenizer

- Page: https://aifordevs.org/glossary#tokenizer
- Markdown: https://aifordevs.org/glossary/tokenizer.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs