FedAvg Quadratic Client Drift
Several local learning steps can converge reliably to the wrong global point when client curvatures differ.
Free theory articleSeveral local learning steps can converge reliably to the wrong global point when client curvatures differ.
Concept & interpretation
Federated Averaging (FedAvg) is a standard algorithm in federated learning where participating clients perform multiple local training steps on their local data before communicating the updated weights to a central server. This allows for reduced communication frequency. However, when the client loss functions are heterogeneous—specifically when they have different curvatures (second derivatives)—FedAvg can suffer from client drift.
Experiment
In the interactive model, you can visualize how this drift occurs.
- Observe the local trajectories of the two clients starting from the server parameter.
- Notice how the server updates its parameter based on the averaged client endpoints.
- Increase the local steps `E` and observe the discrepancy between the FedAvg fixed point and the global minimizer when the second client's curvature `h2` is unequal to `h1=1`.
- Set local steps `E = 1` and verify that the fixed point perfectly aligns with the global optimum, recovering centralized gradient descent.