# Attention

Concept · [Building with AI](https://ecosystem.samcarlton.com/ai)

Also called: attention mechanism, self-attention, transformer attention, attention heads, attention layer

The part of a [language model](https://ecosystem.samcarlton.com/llm) that decides, for each [token](https://ecosystem.samcarlton.com/token), how much every other token in the input should shape its meaning. It is how a model connects a word like 'it' to the right noun several sentences back. It is also part of why very long inputs are harder for a model to handle well.

**For example:** In 'The investor read the [deck](https://ecosystem.samcarlton.com/pitch-deck) and loved it', attention is what lets the model link 'it' to 'the deck'.

## Connects to (8)

- [Transformer](https://ecosystem.samcarlton.com/transformer): The design that most modern language models are built on, introduced by Google researchers in 2017.
- [LLM](https://ecosystem.samcarlton.com/llm): Large language model.
- [Token](https://ecosystem.samcarlton.com/token): In AI, a small chunk of text, roughly a word or a piece of a word, that an LLM reads and writes.
- [Attention relationship](https://ecosystem.samcarlton.com/attention-relationship): The link between two tokens in the input, measured by how much one shapes the model's reading of the other.
- [Attention budget](https://ecosystem.samcarlton.com/attention-budget): A way of picturing attention as a fixed amount each token can spread across everything else in the input.
- [Latent space](https://ecosystem.samcarlton.com/latent-space): The internal map of numbers where a model places the things it reads.
- [Embedding](https://ecosystem.samcarlton.com/embedding): A way of turning a piece of text into a list of numbers that captures its meaning, so software can tell which pieces of text are about…
- [Context window](https://ecosystem.samcarlton.com/context-window): How much text an LLM can take into account at one time.

Map view: https://ecosystem.samcarlton.com/attention