Skip to main content

2026


Multihead attention. Operations are purple, given tensors are green. Each head is its own box.

Notes on Transformers

·3266 words·16 mins
Walking through the shapes of every operation in GPT-2 small, with some extras about inference

2019