# Transformer

Concept · [Building with AI](https://ecosystem.samcarlton.com/ai)

Also called: transformers, transformer model, transformer architecture, transformer-based, the T in GPT, GPT architecture

The design that most modern [language models](https://ecosystem.samcarlton.com/llm) are built on, introduced by [Google](https://ecosystem.samcarlton.com/google) researchers in 2017. Its central idea is [attention](https://ecosystem.samcarlton.com/attention), which lets the model weigh every part of the input against every other part. The 'T' in [GPT](https://ecosystem.samcarlton.com/chatgpt) stands for transformer.

**For example:** When someone says their model is 'transformer-based', they mean it uses the same basic design as the models behind [ChatGPT](https://ecosystem.samcarlton.com/chatgpt), [Claude](https://ecosystem.samcarlton.com/claude) and [Gemini](https://ecosystem.samcarlton.com/gemini).

## Connects to (9)

- [LLM](https://ecosystem.samcarlton.com/llm): Large language model.
- [Attention](https://ecosystem.samcarlton.com/attention): The part of a language model that decides, for each token, how much every other token in the input should shape its meaning.
- [Model](https://ecosystem.samcarlton.com/model): The trained system at the center of an AI product.
- [Training](https://ecosystem.samcarlton.com/training): Teaching an AI model by showing it a very large number of examples before anyone uses it.
- [Parameters](https://ecosystem.samcarlton.com/parameters): The values a model learns during training.
- [Next-token prediction](https://ecosystem.samcarlton.com/next-token-prediction): The one job a language model performs.
- [Neural network](https://ecosystem.samcarlton.com/neural-network): A kind of software model made of many layers of simple math units connected by weights, loosely inspired by the brain.
- [Tokenizer](https://ecosystem.samcarlton.com/tokenizer): The step that splits text into tokens, the small pieces a model actually reads, and turns each one into a number.
- [Startup ecosystem](https://ecosystem.samcarlton.com/startup-ecosystem): The people, money and organizations in one place that help new companies start and grow.

## Sources

- <https://arxiv.org/abs/1706.03762>


Map view: https://ecosystem.samcarlton.com/transformer