Concept · Building with AI

Attention

Also called: attention mechanism, self-attention, transformer attention, attention heads, attention layer

The part of a language model that decides, for each token, how much every other token in the input should shape its meaning. It is how a model connects a word like 'it' to the right noun several sentences back. It is also part of why very long inputs are harder for a model to handle well.

For example: In 'The investor read the deck and loved it', attention is what lets the model link 'it' to 'the deck'.

Connects to 8

• Transformer Concept
• LLM Concept
• Token Concept
• Attention relationship Concept
• Attention budget Concept
• Latent space Concept
• Embedding Concept
• Context window Concept