Concept · Building with AI

Tokenizer

Also called: tokenizers, tokenization, tokenisation, BPE, byte pair encoding

The step that splits text into tokens, the small pieces a model actually reads, and turns each one into a number. Different models split text differently.

For example: A tokenizer might split 'unbelievable' into 'un', 'believ' and 'able', three tokens.

Connects to 4

• Token Concept
• Transformer Concept
• Embedding Concept
• Context window Concept