Linear Attention Reassociation
Reassociate positive-feature attention into compact running sums and expose zero-denominator cases.
Free theory articleReassociate positive-feature attention into compact running sums and expose zero-denominator cases.
Concept & interpretation
Standard self-attention operations compute similarities between queries and keys utilizing a softmax function. This incurs a quadratic time complexity with respect to the sequence length. Linear Attention provides a way to reduce this to linear time by replacing the softmax with a non-negative feature map φ(·) applied directly to queries and keys.
Experimenting with the Module
The interactive Linear Attention Reassociation module allows you to explore this mechanism with 2D queries and keys, testing how feature maps change the running sums and the final normalized attention.
To investigate these behaviors:
- Adjust the Preset to explore regular or adversarial conditions (e.g., all negative values).
- Switch the Feature map to observe how ELU+1 vs. ReLU impacts denominator stability, particularly where the denominator becomes exactly zero.
- Change the Prefix to dynamically update the running sums and verify the observable attention output against the provided reference.