Rotary Position Embeddings

In transformer architectures, self-attention has no inherent notion of token position. Rotary Position Embeddings (RoPE) address this by rotating the query and key representations at each position. The rotation angle depends on both the dimension index (the frequency) and the token's position in the sequence. Because the dot product of two 2D vectors only depends on their relative angle, the attention score inherently respects their relative distance.

Experiment with relative rotation

Open the Rotary Position Embeddings module.

1. Select "Orthogonal" vectors. Set Query position to 1, Key position to 0, and Frequency to π/6. Observe that the original dot product is 0, but the rotary dot product becomes 0.5.
2. Change the "Shared shift" from 0 to 1, then to 3. Watch how the final vectors rotate on the canvas, yet the "Shifted dot product" remains perfectly stable at 0.5, because their relative position difference (1 - 0 = 1) is unchanged.
3. Switch the Frequency to π/2 and Vectors to "Diagonal". A position difference of 1 rotates one vector by 90 degrees relative to the other, leading to opposing vectors and a dot product of -2.0.