Attention Math: Query, Key, Value
Scaled dot-product attention with pen-and-paper numbers: build Q, K, V from I love AI, score, scale, softmax, mix Values — plus masks, multi-head, and gotchas.
Scaled dot-product attention — Query, Key, and Value — with every matrix multiply small enough to do by hand.
Attention compares what each token is looking for (Query) with what every token offers (Key), then mixes the actual payload (Value) by those scores.
Goal for this page: take the sentence I love AI, build Q / K / V
from tiny embeddings, compute scores, scale by √dₖ, softmax, and multiply by V —
matching
softmax(QKᵀ/√dₖ)V
cell by cell. Then note masks, multi-head, and how this ties to KV Cache.
Before matrices: attention is a soft lookup. Each token asks a question, scores every other token’s “label,” then blends their notes.
A student walks in with a question (Query). Classmates wear name tags for what they know (Keys). Matching tags decide how much of each classmate’s notes (Values) get copied into the student’s answer.
Softmax is the polite rule: attention shares add to 100%. Nobody gets a negative share; stronger matches get more of the budget.
If matching and mixing used the same numbers, you could not specialize. Separate projections let a token ask for one thing (Q), advertise another (K), and contribute yet another (V). Training learns three different maps from the same embedding.
One line is the whole mechanism. The rest of the page unpacks each symbol with numbers.
Query
What this token is looking for when it scans the sequence.
Key
What each token advertises — the tag Queries match against.
Value
The content that gets mixed in once the scores are known.
| Symbol | Meaning |
|---|---|
| Q × Kᵀ | Raw similarity scores (every token → every token) |
| √dₖ | Scale so softmax stays well-behaved as width grows |
| softmax | Turn each row of scores into probabilities |
| × V | Weighted sum of Value vectors |
Three words, embedding size 4. Real models use hundreds of dimensions — same layout.
Stack the embeddings as rows. Shape 3 × 4 — three tokens, four numbers each. In a full Transformer these rows already include position information; we skip that here to keep the multiply readable.
Multiply X by three learned weight matrices. We pick small integer weights so the arithmetic stays honest.
Each weight matrix is 4 × 3 (embedding in, head dim out). Row i of Q is “token i’s Query.” Same for K and V. In training these weights are learned; here they are fixed for the walkthrough.
[1, 0, 1, 0] × W_Q
→ [1·1+0·0+1·1+0·0, 1·0+0·1+1·0+0·1, 1·1+0·0+1·0+0·1] = [2, 0, 1]
[1, 0, 1, 0] × W_K → [0, 1, 1]
[1, 0, 1, 0] × W_V → [1, 0, 1]
| Tensor | Shape here | In general |
|---|---|---|
| X | 3 × 4 | n × d_model |
| W_Q, W_K, W_V | 4 × 3 | d_model × dₖ |
| Q, K, V | 3 × 3 | n × dₖ |
Each Query dots with every Key. High score = “this token looks relevant to me.”
Kᵀ swaps rows and columns so shapes line up: Q is 3×3, Kᵀ is 3×3, product is 3×3 — one score from every token to every token. Cell (i, j) is “how much token i wants token j.”
I → I: [2,0,1]·[0,1,1] = 1
I → love: [2,0,1]·[2,1,1] = 5
I → AI: [2,0,1]·[1,1,1] = 3
I scores love highest (5). love ties everyone at 3. AI prefers love (4). Green cells mark the row max. These are still raw logits — not yet probabilities.
Divide every score by √dₖ so softmax does not saturate when dimensions grow.
Dot products grow with dimension. Huge positive/negative scores push softmax toward 0 or 1 and flatten gradients. Dividing by √dₖ keeps typical scores in a friendlier range. Here dₖ = 3, so √3 ≈ 1.732.
Scaling does not change the argmax inside a row — only the sharpness before softmax.
[1, 5, 3] / 1.732 ≈ [0.577, 2.887, 1.732]
Per row: exp each score, divide by the row sum. That turns scores into attention weights.
Softmax never looks across rows. Token “I”’s weights are independent of how “AI” distributes its attention. Each row is its own probability distribution over the sequence.
e^0.577 ≈ 1.781 | e^2.887 ≈ 17.940 | e^1.732 ≈ 5.651
sum ≈ 25.372
weights ≈ [0.070, 0.707, 0.223]
All scaled scores equal → uniform weights 1/3 each. Softmax of a constant vector is always uniform.
e^1.155 ≈ 3.174 | e^2.309 ≈ 10.063 | e^1.732 ≈ 5.651
sum ≈ 18.888 → weights ≈ [0.168, 0.533, 0.299]
Mix the Value rows with those probabilities. Each token gets a new vector shaped by context.
0.070·[1, 0, 1] + 0.707·[1, 2, 0] + 0.223·[1, 1, 0]
dim0: 0.070·1 + 0.707·1 + 0.223·1 = 1.000
dim1: 0.070·0 + 0.707·2 + 0.223·1 = 1.637
dim2: 0.070·1 + 0.707·0 + 0.223·0 = 0.070
Dominated by love’s Value — matching the 0.707 weight. The second component (1.637) is mostly 0.707 × 2 from love.
(1/3)([1,0,1] + [1,2,0] + [1,1,0]) ≈ [1.000, 1.000, 0.333]
The output row for a token is no longer “just itself.” It is a weighted blend of every Value — that is how attention moves information across the sequence. Later layers and the feed-forward block consume this contextual vector.
Our toy matrix lets every token see every token. Real LLMs often forbid looking ahead.
Causal (decoder) mask
Position i may only attend to positions ≤ i. Future Keys are blocked (scores set to −∞ before softmax → weight 0). That is why generation can append tokens without rewriting the past — and why KV Cache works.
Padding / custom masks
Batched sequences of different lengths mask pad tokens so they never receive attention. Encoder-style models may use full (bidirectional) attention over the whole input.
Real Transformers run several attentions side by side, then stitch the results.
Each head has its own W_Q, W_K, W_V (or slices of larger matrices). Head 1 might track syntax; head 2 might track a subject–verb link. Outputs are concatenated and passed through a final linear layer W_O.
Our example is one head with dₖ = 3. A model with 32 heads and d_model = 4096 often uses dₖ = 128 per head — same algebra, more copies.
- 1 · Embed Words → matrix X (here 3 × 4).
- 2 · Project Q = X W_Q, K = X W_K, V = X W_V.
- 3 · Score Q × Kᵀ — relevance of every token to every token.
- 4 · Scale Divide by √dₖ (optionally apply a mask first).
- 5 · Softmax Rows become attention weights.
- 6 · Mix Weights × V → contextual output vectors.
- Attention weights ≠ “the model’s thoughts” They are useful diagnostics, not a full explanation of why a model said something. Treat them as one lens, not a mind-read.
- Q is not stored in the KV Cache Only Keys and Values of past tokens are reused across decode steps. Each new token brings a fresh Query. Details in KV Cache in LLMs.
- Self-attention vs cross-attention Here Q, K, and V all come from the same sequence (self-attention). In encoder–decoder setups, Queries can come from one side and Keys/Values from another (cross-attention) — same formula, different sources.
- Our numbers are pedagogical Embeddings and weights in real models are floats learned from data, not neat integers. The shapes and the six steps stay the same.
Takeaway
Attention is compare with Keys, mix Values. Queries ask, Keys answer the match, Values supply the content, softmax turns scores into a budget of attention, and the √dₖ factor keeps that softmax trainable as width grows.