PCGrad Conflicting Task Gradients
Remove conflicting gradient components and inspect each task’s local descent direction.
Free theory articleRemove conflicting gradient components and inspect each task’s local descent direction.
Concept & interpretation
In multi-task learning, different tasks can produce gradients that point in opposing directions. When this happens, a joint optimization step might degrade the performance of one or both tasks. The PCGrad (Projecting Conflicting Gradients) algorithm mitigates this issue by identifying when two task gradients conflict and projecting them so they no longer interfere.
Experiment
You can use the interactive module to observe how different scaling factors and initial gradient alignments affect the final projected descent direction.
- Open the PCGrad Conflicting Task Gradients module.
- Set Gradients to "Conflict". You will see the original gradients pointing in opposing directions.
- Set Second scale to 2. Notice how the magnitude of g2 increases, which changes the projection vectors and the final combined gradient.
- Change Aggregation to "Mean". The combined gradient will be halved, but the direction remains the same.