Transformers: Big Picture

Transformers: Big Picture

Step 1 of 12

A transformer is a stack of identical blocks refining one sequence. Start with the text.

The Idea

A transformer turns tokens into vectors, adds position information, then pushes them through many identical blocks. Each block mixes across positions with attention, then transforms each position on its own. The last layer scores every vocabulary entry.

Real-World Example

A relay of copy-editors. One manuscript passes across thirty identical desks; each editor may read the whole document and nudge every sentence slightly. No single desk rewrites it.

The Tradeoff

Attention compares every position with every other, so cost grows with the square of sequence length — which is why long inputs are expensive and why approximate attention schemes exist at all. Older recurrent models were cheaper per token but had to read strictly in order; transformers buy parallel training with that quadratic bill.

Your turn

Put the steps in the right order.

1. 1Run the stack of identical blocks: attention, then a per-position network↑↓
2. 2Split the text into tokens and look up a vector for each↑↓
3. 3Score every vocabulary entry to pick what comes next↑↓
4. 4Add position information so word order is not lost↑↓

Check order

Mini quiz

1 / 3

Why must position information be added to the token vectors?

• AAttention on its own treats the input as an unordered set
• BTokens carry no meaning
• CIt compresses the sequence
• DIt slows training down deliberately