Understanding Transformers: The Architecture Behind Modern AI

Why Understanding This Matters

You do not need to understand how a car engine works to drive, but mechanics make better drivers. The same applies to AI: understanding how transformers work helps you write better prompts, anticipate limitations, and have smarter conversations about what AI can and cannot do.

This article explains transformers conceptually — no maths, no code, just clear mental models. If you have ever wondered why AI sometimes “hallucinates”, why context windows matter, or why bigger models tend to be smarter, this article answers those questions.

Tokens: How AI Reads Text

AI models do not read words — they read tokens. A token is a chunk of text, roughly equivalent to three-quarters of a word. The word “understanding” might be split into “under” + “standing”. Common words like “the” are single tokens.

Tokenisation is the first step in processing any text. Your prompt gets broken into tokens, the model processes those tokens, and it generates new tokens one at a time as its response.

This is why AI models have “context windows” measured in tokens (e.g., 200,000 tokens for Claude). The context window is how much text the model can consider at once — like the size of its working memory.

Attention: The Core Innovation

The breakthrough idea behind transformers is attention — the ability for every token in a sequence to look at every other token and decide which ones are relevant.

Imagine reading the sentence: “The cat sat on the mat because it was tired.” When processing the word “it”, the model needs to figure out what “it” refers to. Attention lets the model look back at all previous words and determine that “it” most likely refers to “cat”, not “mat.”

This happens simultaneously across all positions in the text — hence the name “self-attention.” Every word considers its relationship to every other word, all at once. This parallel processing is what makes transformers so powerful and so fast compared to earlier approaches.

Think of attention like a conversation in a room. Instead of speaking one at a time (sequential), everyone can hear everyone else simultaneously (parallel). Each person decides who is most worth listening to based on what they need to know.

Training: How Models Learn

A transformer model learns by reading enormous amounts of text and learning to predict the next token. Given “The capital of France is”, it learns that “Paris” is the most likely next token.

This simple objective — predict the next word — turns out to be extraordinarily powerful. To predict well, the model must learn grammar, facts, reasoning patterns, writing styles, and even common-sense knowledge. All of this emerges from the training process without being explicitly programmed.

Training happens in two phases:

1. Pre-training: The model reads trillions of tokens from the internet, books, and other text sources. This builds its general knowledge and language ability.
2. Fine-tuning: The model is further trained on curated examples of helpful, harmless, and honest behaviour. This is what turns a text predictor into a useful assistant.

Inference: How Responses Are Generated

When you send a prompt to an AI model, it processes your entire prompt at once (using attention), then generates the response one token at a time. Each new token is predicted based on the prompt plus all previously generated tokens.

This is why AI responses appear to stream in word by word — because that is literally what is happening. The model generates one token, adds it to the sequence, then uses the updated sequence to predict the next token.

An important implication: the model does not “plan” its response in advance. It does not know how its sentence will end when it starts writing. This explains some AI behaviours — like starting a list and then struggling to maintain a consistent number of items — that seem odd when you expect deliberate planning.

Why AI Hallucinates

Understanding token prediction explains hallucination perfectly. The model does not “know” things — it predicts likely next tokens based on patterns learned during training. When it generates a plausible-sounding but factually incorrect statement, it is because that statement was statistically likely given the context.

This is not a bug that will be fixed — it is a fundamental property of how these models work. Hallucination can be reduced through better training, larger models, and techniques like retrieval-augmented generation (RAG), but it cannot be eliminated entirely.

Knowing this helps you use AI more effectively: always verify factual claims, use AI for reasoning and creativity rather than as a database, and provide source material when accuracy matters.

Why Bigger Models Are Smarter

Transformer models are defined by their number of parameters — the adjustable values learned during training. GPT-4 has an estimated 1.8 trillion parameters. More parameters means more capacity to learn patterns, make distinctions, and handle nuance.

The relationship between model size and capability is not linear — it shows “emergent abilities.” Certain capabilities (like complex reasoning, code generation, and nuanced instruction-following) only appear once models reach a sufficient size. Below that threshold, the model simply cannot perform the task. Above it, performance improves rapidly.

This is why the jump from GPT-3 to GPT-4, or from Claude 2 to Claude 3, felt so dramatic. It was not just “a bit better” — entirely new capabilities emerged.