How a Transformer Plays Tic-Tac-Toe

What's Covered

An interactive guide to the architecture behind modern language models. Instead of predicting the next word, this Transformer predicts the next move in a game of fading Tic-Tac-Toe-making every step of the model easy to visualize and understand. Play the game, inspect every matrix multiplication, and watch tokens flow through the network in real time.

• Tokenization and embeddings
• Learned positional encoding
• Self-attention (Q, K, V)
• Multi-head attention
• Causal masking and softmax
• Residual connections and layer normalization
• MLP (feed-forward network)
• Unembedding and sampling
• Model ablations (no positional encoding, no causal mask, no MLP, no residual stream)

Includes interactive visualizations for every stage of the Transformer pipeline - from input tokens to the final prediction.

https://sbondaryev.dev/articles/transformer

Top comments (0)