In self-attention, input tokens are projected into Query (Q), Key (K), and Value (V) matrices. The attention weight between token i and token j is computed as: Attention(Q, K, V) = softmax(Q · Kᵀ / √d_k) · V. During course playback, the animated board highlights the dot-product heat distribution dynamically as the AI voice agent narrates each calculation step.

[Input Sequence: "AI Agents Build Future"]
        │
        ▼ Tokenizer & Positional Embedding
 ┌───────────────┬───────────────┬───────────────┐
 │   Query (Q)   │    Key (K)    │   Value (V)   │
 └───────┬───────┴───────┬───────┴───────┬───────┘
         └───────────────┼───────────────┘
                         ▼ MatMul + Scale (1/√d) + Mask
                  [Softmax Attention Scores]
                         ▼ MatMul (V)
              [Multi-Head Aggregated Context]
                         ▼ Feed-Forward + LayerNorm
                  [Next-Token Probability Vector]

The schematic above illustrates token routing across the transformer self-attention head, demonstrating how context is synthesized prior to layer normalization.

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Computes Scaled Dot-Product Attention as defined in Attention Is All You Need.
    """
    d_k = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    attn_weights = F.softmax(scores, dim=-1)
    output = torch.matmul(attn_weights, V)
    return output, attn_weights

# Simulate a 4-token sequence with embedding dimension 64
Q = torch.randn(1, 4, 64)
K = torch.randn(1, 4, 64)
V = torch.randn(1, 4, 64)

context, weights = scaled_dot_product_attention(Q, K, V)
print("Attention output shape:", context.shape)  # [1, 4, 64]

Code Explanation: The function computes the matrix multiplication of Q and transpose K, scales by the square root of dimension d_k to stabilize gradients, applies softmax to convert logits to probabilities, and multiplies by V to extract context-enriched representations.