Skip to content
ignitae
Academy
A tiny neural net
A linear layer, cross-entropy loss and a single attention head: the forward pass, from scratch.
Syllabus
Linear layer
A linear layer is a matrix multiply plus a bias
Linear layer (exercise)
Cross-entropy
Cross-entropy measures how wrong a prediction is
Cross-entropy loss (exercise)
Attention
Attention weighs values by how well they match a query
A single attention head (exercise)