AI Technology Explained

Transformer Architecture Explained Simply

Last updated: June 2025 · AI Pentium Editorial Team

Why Transformers Changed Everything

Before 2017, sequence modeling was dominated by Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs). These process sequences step by step — each token depends on the hidden state of the previous token. This created two fundamental problems: sequential computation (impossible to parallelize during training) and information bottleneck (early-sequence information degrades as it passes through many steps).

The Transformer paper "Attention Is All You Need" (Vaswani et al., Google, 2017) replaced sequential processing with self-attention: every token attends to every other token simultaneously, in parallel, regardless of their distance. This enabled massive parallelization on GPUs and direct modeling of long-range dependencies — the two ingredients that made trillion-parameter models feasible.

The Building Blocks: Tokens and Embeddings

Input text is first tokenized into a sequence of token IDs. Each token ID is mapped to a dense vector (the token embedding) of dimension d_model (e.g. 768 for BERT-base, 12,288 for GPT-4). A positional encoding is added to each embedding to inject position information, since self-attention is otherwise permutation-invariant.

Original Transformers used fixed sinusoidal positional encodings. Modern LLMs typically use learned embeddings or relative position schemes like Rotary Position Embedding (RoPE), which extrapolate better to sequence lengths beyond those seen during training.

The Self-Attention Mechanism

Self-attention is the core operation of the Transformer. For each token in the sequence, it computes three vectors from the embedding: a Query (Q), a Key (K), and a Value (V) — each produced by a separate linear projection matrix.

The attention weight between token i and token j is computed as the scaled dot product of Q_i and K_j, divided by √d_k (to prevent vanishingly small gradients in high dimensions), then passed through softmax. The output for token i is the weighted sum of all Value vectors:

Attention(Q, K, V) = softmax(QK^T / √d_k) · V

This single equation allows each token to "look at" every other token in the sequence and gather information weighted by relevance — the mechanism behind all of ChatGPT's contextual understanding.

Multi-Head Attention (MHA)

A single attention head computes one set of Q/K/V projections. Multi-head attention runs h attention heads in parallel, each with its own Q, K, V weight matrices and operating in a lower-dimensional subspace (d_k = d_model / h). The outputs of all heads are concatenated and linearly projected back to d_model.

Why multiple heads? Each head can specialize in attending to different types of relationships: one head might track syntactic dependencies, another co-references, another long-range semantic relationships. GPT-3 has 96 attention heads per layer. Empirical analysis of attention patterns shows different heads learn distinctly interpretable behaviors.

The Feed-Forward Sublayer

After multi-head attention, each Transformer layer applies a position-wise feed-forward network (FFN): two linear transformations with a non-linear activation (ReLU or GELU) in between. The inner dimension is typically 4× d_model:

FFN(x) = max(0, xW₁ + b₁)W₂ + b₂

The FFN operates independently on each token position (no cross-token interaction). Research suggests FFN layers act as key-value memories, storing factual associations learned during pre-training. The FFN typically contains ~2/3 of a Transformer's total parameters.

Layer Normalization and Residual Connections

Each sublayer (attention and FFN) is wrapped with a residual connection (output = sublayer(x) + x) and layer normalization. The original paper applied LayerNorm after the residual (Post-LN). Modern LLMs (GPT-2+, LLaMA) apply it before (Pre-LN), which improves training stability for very deep networks.

Encoder vs Decoder: Two Architectural Variants

The original Transformer had both an encoder and decoder. Modern LLMs use one or the other:

Scaling Laws

A landmark 2020 paper from OpenAI ("Scaling Laws for Neural Language Models", Kaplan et al.) showed that LLM performance improves predictably as a power law with model size (parameters N), dataset size (tokens D), and compute budget (FLOPs C). The key insight: you must scale all three together — a 10× bigger model trained on the same data is not as efficient as a model of intermediate size trained on proportionally more data.

The Chinchilla scaling laws (Hoffmann et al., DeepMind, 2022) refined this: for a given compute budget, the optimal strategy is roughly 1:1 ratio of model parameters to training tokens. This led to models like LLaMA-3 (8B parameters on ~15T tokens) outperforming earlier much-larger models on many benchmarks.

Modern Transformer Innovations

• Grouped Query Attention (GQA): Multiple query heads share key/value heads, reducing KV-cache memory and inference cost. Used in LLaMA-3, Mistral.
• Sliding Window Attention: Tokens attend only to a local window, enabling efficient O(n) scaling for long contexts. Used in Mistral, Mamba hybrid models.
• Mixture of Experts (MoE): The FFN is replaced by multiple "expert" FFNs; a router selects which experts activate for each token. Dramatically increases model capacity with lower per-token compute. Used in GPT-4 (estimated), Mixtral-8x7B.
• Flash Attention: An IO-aware exact attention implementation that avoids materializing the large attention matrix in HBM, yielding 2–4× speedups and enabling longer contexts.

Further Reading

• How ChatGPT Works — pre-training, RLHF, and inference
• What is RAG? — how to augment Transformers with retrieval
• ChatGPT vs Claude — architecture and capability comparison
• LLM research papers