Concept · Building with AI

Transformer

Also called: transformers, transformer model, transformer architecture, transformer-based, the T in GPT, GPT architecture

The design that most modern language models are built on, introduced by Google researchers in 2017. Its central idea is attention, which lets the model weigh every part of the input against every other part. The 'T' in GPT stands for transformer.

For example: When someone says their model is 'transformer-based', they mean it uses the same basic design as the models behind ChatGPT, Claude and Gemini.

Connects to 9

• LLM Concept
• Attention Concept
• Model Concept
• Training Concept
• Parameters Concept
• Next-token prediction Concept
• Neural network Concept
• Tokenizer Concept
• Startup ecosystem Concept

Sources

• https://arxiv.org/abs/1706.03762