LoRA Factor Gradients and Initialization

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that freezes pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture. For a frozen weight matrix W0, the update is represented as ΔW = B · A. To ensure the model begins training exactly as the original pre-trained model, the matrix B is typically initialized to zero, forcing the initial update ΔW to be exactly zero.

Experiment: Zero Initialization Dynamics

The interactive visualization at https://riano.app/module/lora-factor-gradient-initialization illustrates why one factor (typically B) is initialized to zero. Follow these steps to explore the initialization dynamics:

1. Set the B vector to "Zero", A vector to "Difference", and Scale to 1. Notice the prediction matches the frozen base model behavior, and the total update matrix is exactly zero.
2. Observe the gradients: ∇A is exactly zero, but ∇B is non-zero. Even though factor B is entirely zero, the non-zero residual allows B to receive a gradient and update.
3. Now change the A vector to "Zero" while keeping the B vector at "Zero". You'll see that both ∇A and ∇B are frozen at zero. This happens because each gradient depends on the other factor; if both start at zero, neither factor ever updates.
4. Reset to A vector "Difference" and B vector "Zero", then change the Input to "Sum" and the Target to "Zero". Notice that ∇B becomes zero. The input `x = [1, 1]` is orthogonal to `a = [1, -1]`, making the scalar projection `a · x = 0`, freezing B's gradient for that specific data point.