Purpose
This article explains the core architectural components of Transformer-based language models, with particular focus on self-attention, positional information, causal masking, and the major Transformer model families used in modern LLM systems.
Overview
Modern large language models are almost universally based on the Transformer architecture introduced in Attention Is All You Need. The defining computational primitive of the Transformer is attention, which replaces recurrence with direct token-to-token interaction.
In recurrent architectures such as RNNs and LSTMs, hidden state is propagated sequentially from left to right. This creates two major limitations:
- Long-range dependencies are hard to preserve, because information must traverse many recurrent steps.
- Training and inference are inherently sequential across time, which limits hardware utilization.
Transformers remove this sequential dependency inside each layer. For a sequence of length T, each token representation can attend directly to all relevant positions in the context window. This enables much more effective modeling of long-range dependencies and maps well onto highly parallel hardware such as GPUs.
A standard Transformer block contains:
- a multi-head attention sublayer
- a position-wise feed-forward network (FFN / MLP)
- residual connections
- normalization layers (LayerNorm or RMSNorm, depending on the architecture)
In decoder-only LLMs, these blocks are stacked many times and trained with a next-token prediction objective.
Scaled dot-product attention
The canonical attention mechanism operates on three learned projections of an input hidden state matrix X in R^(T x d_model):
Q = X W_QK = X W_KV = X W_V
where:
Tis the sequence lengthd_modelis the model hidden sizeW_Q, W_K, W_Vare learned projection matrices
The raw attention score matrix is:
S = Q K^T / sqrt(d_k)
where d_k is the key/query head dimension.
The scaling factor 1 / sqrt(d_k) is not cosmetic. Without it, the magnitude of dot products grows with dimensionality, causing the softmax to saturate and gradients to become poorly behaved.
The normalized attention weights are:
A = softmax(S + M)
where M is an additive mask, typically:
- zero on allowed positions
-infon disallowed positions
The output is then:
O = A V
Interpretation:
- query: what the current token is searching for
- key: how each candidate token can be matched
- value: what information is actually retrieved if attended to
The critical point is that attention is not “copying tokens.” It computes a learned weighted mixture of transformed value vectors, conditioned on contextual compatibility between queries and keys.
Multi-head attention
In practice, modern Transformers use multi-head attention (MHA). Instead of computing one attention map over the full hidden space, the model splits attention into multiple heads:
hheads- each with head dimension
d_head - typically
d_model = h * d_head
Each head learns separate Q/K/V projections and can specialize in different interaction patterns, such as:
- short-range syntax
- long-range coreference
- positional relations
- delimiter / structure tracking
- induction patterns
Formally, each head computes its own attention output, and the head outputs are concatenated and projected back to d_model.
This factorization improves expressivity by allowing multiple relational subspaces to be modeled in parallel.
Why attention needs positional information
Pure self-attention is permutation-equivariant with respect to token order. If the model receives the same set of token embeddings in a different order, and no positional signal is provided, the attention mechanism has no inherent way to distinguish those sequences.
This is why positional information must be injected.
Absolute sinusoidal positional encoding
The original Transformer used deterministic sinusoidal embeddings added to token embeddings:
- sine on even dimensions
- cosine on odd dimensions
- frequencies varying geometrically across dimensions
This gives every position a unique embedding and allows relative offsets to be represented through linear operations.
Learned positional embeddings
Many models replace fixed sinusoids with learned positional embeddings. This is simple and effective, but extrapolation beyond the trained context length is generally weaker.
Relative and rotary methods
Modern LLMs frequently use relative positional mechanisms, because attention fundamentally models pairwise token interactions.
Important variants include:
- relative position bias: adds position-dependent terms directly into attention logits
- RoPE (Rotary Positional Embedding): rotates queries and keys in a way that makes their dot product depend on relative position
- ALiBi: adds head-specific linear biases that favor nearer tokens without introducing learned embeddings per position
RoPE is especially common in modern decoder-only LLMs because it preserves the attention formulation while introducing relative phase information directly into Q and K. It is also compatible with many long-context extrapolation techniques, although extrapolation quality depends heavily on how the model was trained and how RoPE scaling is handled.
Causal masking
Decoder-only LLMs are trained autoregressively: at position t, the model predicts token x_t using only tokens x_<t.
To enforce this during training, the self-attention matrix is masked with a lower-triangular causal mask. For token position i, attention to positions j > i is forbidden.
Without causal masking:
- the model would access future tokens during training
- the training objective would no longer match inference-time conditions
- perplexity would appear artificially low because of information leakage
This mask is usually applied additively to attention logits before softmax, with disallowed positions set to -inf.
In implementations, causal masking can be combined with:
- padding masks
- sliding-window masks
- block-sparse masks
- prefix-LM masks
- KV-cache offsets for incremental decoding
Self-attention vs cross-attention
Self-attention
In self-attention, Q, K, and V all come from the same sequence. This is the dominant mechanism in decoder-only LLMs.
Use case:
- each token contextualizes itself against the preceding context
Cross-attention
In cross-attention, queries come from one sequence and keys/values come from another sequence.
This is most important in encoder-decoder models:
- encoder outputs provide
KandV - decoder hidden states provide
Q
This lets the decoder condition generation on an encoded source sequence. For tasks such as translation, summarization, structured transformation, and speech/text transduction, this is often a more natural architecture than a pure decoder-only setup.
Transformer block internals
A modern Transformer block is usually not just “attention + MLP.” There are architectural details that strongly affect optimization and performance.
Residual path
Each sublayer is wrapped in a residual connection:
x = x + Attention(Norm(x))x = x + MLP(Norm(x))
or, in post-norm variants:
x = Norm(x + Attention(x))x = Norm(x + MLP(x))
Most modern LLMs prefer pre-norm because it improves optimization stability in deep networks.
Feed-forward network
The MLP sublayer is typically much wider than d_model. A common pattern is:
- project from
d_modeltod_ff - apply nonlinearity
- project back to
d_model
Modern LLMs often use gated MLP variants such as SwiGLU or GEGLU rather than a simple ReLU MLP. These usually improve parameter efficiency and quality.
Normalization
Older Transformers used LayerNorm everywhere. Many recent LLMs use RMSNorm, which normalizes by root-mean-square magnitude without mean-centering. RMSNorm is cheaper and often works well in large decoder stacks.
Decoder-only, encoder-only, and encoder-decoder models
Decoder-only
Examples: GPT family, Llama family, Mistral-style architectures.
Properties:
- masked self-attention only
- trained with next-token prediction
- ideal for open-ended generation
- naturally supports incremental decoding with KV-cache reuse
This is the dominant design for modern general-purpose LLM serving.
Encoder-only
Examples: BERT, RoBERTa.
Properties:
- bidirectional self-attention
- no causal mask
- typically trained with masked-language-modeling or contrastive objectives
- strong for classification, retrieval, reranking, token labeling, and representation learning
These models are usually not used for left-to-right free-form generation.
Encoder-decoder
Examples: T5, BART.
Properties:
- bidirectional encoder
- causal decoder
- cross-attention from decoder to encoder outputs
- well suited for sequence-to-sequence tasks
These architectures remain very strong for structured conditional generation, especially when the source-target boundary is explicit.
Practical attention variants for real systems
Attention quality matters, but in production systems the memory and bandwidth profile matters just as much.
Multi-Head Attention (MHA)
Each head has separate Q/K/V. Highest flexibility, but also highest KV-cache cost during autoregressive decoding.
KV-cache cost grows with:
- number of layers
- context length
- batch / concurrency
- number of KV heads
- precision used for cached tensors
Multi-Query Attention (MQA)
MQA shares keys and values across all heads while retaining separate queries per head.
This drastically reduces KV-cache size and memory bandwidth at inference time. The tradeoff is that all query heads retrieve from the same K/V representation, which can reduce expressivity.
Grouped-Query Attention (GQA)
GQA is a compromise:
- multiple query heads
- fewer KV heads
- several query heads share one KV group
This reduces memory cost while preserving more modeling power than pure MQA. Many modern LLMs use GQA because it improves serving efficiency without the full quality drop sometimes observed with MQA.
FlashAttention
FlashAttention is not a new attention formula; it is an exact IO-aware implementation of attention that reduces HBM memory traffic by tiling the computation and avoiding materializing large intermediate attention matrices in memory.
Why it matters:
- standard attention is often bandwidth-bound
- FlashAttention improves throughput and memory efficiency
- especially important for long sequences and training workloads
PagedAttention
PagedAttention is a serving-oriented memory management strategy, especially associated with systems like vLLM. It manages KV-cache in virtualized pages rather than requiring large contiguous allocations.
Why it matters:
- improves memory utilization under variable sequence lengths
- reduces fragmentation
- enables more efficient batching and continuous serving
This is a systems-level optimization, not a new model-layer definition, but it has major practical impact on LLM inference infrastructure.
KV cache and autoregressive inference
For decoder-only generation, naive decoding would recompute keys and values for the entire prefix at every step. This is prohibitively expensive.
Instead, inference servers maintain a KV cache per layer:
- after processing a token, store its
KandV - for the next token, reuse all previous
K/V - compute only the new query and append new key/value states
This changes per-token decoding cost substantially and is one of the main reasons attention variants like MQA/GQA matter operationally.
A large fraction of serving constraints for LLMs are really KV-cache constraints, not only weight-memory constraints.
Why this matters in practice
These architectural choices surface directly in deployment behavior:
- context length scaling depends on positional encoding and attention implementation
- GPU memory pressure is often dominated by KV-cache design
- throughput under concurrency depends strongly on attention kernel efficiency and cache management
- latency per token depends on decoder architecture, cache reuse, and batching strategy
- model quality vs serving cost is often partly a choice between MHA, GQA, MQA, and long-context design tradeoffs
Understanding attention at this level makes later topics such as quantization, serving, speculative decoding, and long-context optimization much easier to reason about.
Key takeaways
- Attention computes contextual token mixing through learned query-key compatibility and value aggregation.
- The
1 / sqrt(d_k)scaling factor is necessary for stable optimization. - Positional information is required because self-attention alone does not encode order.
- Causal masking enforces the autoregressive constraint used by decoder-only LLMs.
- Decoder-only, encoder-only, and encoder-decoder architectures are optimized for different problem classes.
- MQA, GQA, FlashAttention, and PagedAttention are critical not only for model design but also for production inference efficiency.
Comments
0 comments
Article is closed for comments.