Attention, Intuitively

The Idea

Attention lets each position build its own mixture of the others. Relevance scores are normalised into weights that sum to one, and the output is the blended values. The weights are recomputed for every input.

Real-World Example

A sound engineer at a mixing desk pushes up the two microphones that carry this line and pulls the rest down, then rebalances the whole desk for the next line.

The Code

values = {"the": [0.0, 1.0], "river": [1.0, 0.0], "bank": [0.5, 0.5]}
weights = {"the": 0.1, "river": 0.7, "bank": 0.2}   # already sum to 1.0

out = [0.0, 0.0]
for word, w in weights.items():
    for i, v in enumerate(values[word]):
        out[i] += w * v                    # blend, do not choose

print([round(x, 2) for x in out])   # [0.8, 0.2] pulled toward "river"

values = {"the": [0.0, 1.0], "river": [1.0, 0.0], "bank": [0.5, 0.5]}
weights = {"the": 0.2, "river": 0.2, "bank": 0.6}

out = [0.0, 0.0]
for word, w in weights.items():
  for i, v in enumerate(values[word]):
      out[i] += w * v

print([round(x, 2) for x in out])