AI::COSMOSNeural Networks • Transformers • RAG • Agents

Attention Is All You Need • Scaled Dot-Product Engine

Transformer Self-Attention Lab

Deconstruct the core math that powers modern LLMs (GPT-4, Gemini, Claude). Step through linear Query (Q), Key (K), Value (V) projections, scaled dot products, softmax probability normalizations, multi-head attention decomposition, and causal autoregressive masking.

Input Token Sequence

Parsed Tokens (7):[0]The[1]transformer[2]model[3]processes[4]human[5]language[6]efficiently

Multi-Head Attention (MHA) Projection Headsd_k = 64 • 8 Heads Parallel

Attention Weight Heatmap Softmax(Q · Kᵀ / √d_k)

Hover cell to inspect

K:The

K:transformer

K:model

K:processes

K:human

K:language

K:efficiently

Q:The

89%0.89

2%0.02

1%0.01

2%0.02

Q:transformer

6%0.06

9%0.09

11%0.11

6%0.06

21%0.21

38%0.38

9%0.09

Q:model

11%0.11

10%0.10

12%0.12

11%0.11

15%0.15

32%0.32

10%0.10

Q:processes

4%0.04

23%0.23

6%0.06

4%0.04

27%0.27

13%0.13

22%0.22

Q:human

92%0.92

2%0.02

1%0.01

2%0.02

Q:language

2%0.02

4%0.04

54%0.54

26%0.26

4%0.04

5%0.05

Q:efficiently

90%0.90

2%0.02

1%0.01

2%0.02

Low Attention (0%)

High Attention (100%)

Active Cell Math Inspector

Hover matrix cell

Move your cursor over any matrix cell on the left to see the exact dot-product, scaling, and softmax computations.

Canonical Transformer Equation

Attention(Q, K, V) = Softmax(Q · Kᵀ / √d_k) · V

Why Divide by √d_k?

For large dimensions, dot products grow large in magnitude, pushing Softmax into regions with extremely small gradients (≈ 0). Scaling by 1/√d_k preserves unit variance.

Causal Masking in Autoregressive LLMs

Decoder models like GPT prevent tokens from looking into the future by setting upper-triangular logits to −∞, ensuring next-token predictions depend only on past context.

Multi-Head Synergy

Rather than performing a single attention function, Multi-Head Attention linearly projects Q, K, V into h different subspaces, allowing the model to jointly attend to syntax, coreference, and semantics simultaneously.

Loading AI::COSMOS...