Explore modules

Attention Mechanism

Attention Mechanism

Free theory article

Attention Mechanism

Concept & interpretation

A query can put more weight on a key that is farther away on a scatter plot. Riano lets you test why: its scores use dot products, not Euclidean distance. Calculate the scores for four fixed keys, turn them into softmax weights, then vary the query and temperature without changing the keys.

Move the query and inspect a bar

Open Attention Mechanism. Query X and Query Y move the query; Temperature controls score concentration. Hovering over the weight bars reveals percentages.

  1. Set Query X to 0, Query Y to 0 and Temperature to 1. Check that the four bars have equal 25% weights.
  2. Set Query X to 1 while keeping Query Y at 0. Hover the K2 bar and compare its roughly 72.6% weight with the calculation.
  3. Raise Temperature to 5. The ordering stays the same because division by a positive number preserves score order, but the probability differences shrink.
  4. Return Temperature to 1 and set Query X to −1. Compare K1 and K2 and explain why reversing the query reverses which x coordinates score well.

The module's keys are fixed. Moving the cross changes scores against the same reference points; it is not retraining word embeddings. Circles use attention weight as a size cue, so apparent size is also an output, not an independent key attribute.

Read the full free theory article