# Tokenizer

Concept · [Building with AI](https://ecosystem.samcarlton.com/ai)

Also called: tokenizers, tokenization, tokenisation, BPE, byte pair encoding

The step that splits text into [tokens](https://ecosystem.samcarlton.com/token), the small pieces a model actually reads, and [turns](https://ecosystem.samcarlton.com/turn) each one into a number. Different models split text differently.

**For example:** A tokenizer might split 'unbelievable' into 'un', 'believ' and 'able', three [tokens](https://ecosystem.samcarlton.com/token).

## Connects to (4)

- [Token](https://ecosystem.samcarlton.com/token): In AI, a small chunk of text, roughly a word or a piece of a word, that an LLM reads and writes.
- [Transformer](https://ecosystem.samcarlton.com/transformer): The design that most modern language models are built on, introduced by Google researchers in 2017.
- [Embedding](https://ecosystem.samcarlton.com/embedding): A way of turning a piece of text into a list of numbers that captures its meaning, so software can tell which pieces of text are about…
- [Context window](https://ecosystem.samcarlton.com/context-window): How much text an LLM can take into account at one time.

Map view: https://ecosystem.samcarlton.com/tokenizer