Generalized Advantage Trace
Decompose an advantage estimate into temporal residuals and distinguish termination from truncation.
Free theory articleDecompose an advantage estimate into temporal residuals and distinguish termination from truncation.
Concept & interpretation
Decompose an advantage estimate into temporal residuals and distinguish termination from truncation.
Experimenting with advantage traces
Explore the relationship between TD residuals and advantage estimation in the Generalized Advantage Trace interactive module.
- Vary Trace lambda: Set Trace lambda to 1. Notice how A0 now equals 3.078, which is exactly the Discounted bootstrap return (4.078) minus the baseline V0 (1). Then set it to 0 and observe that A0 simply equals the first TD residual (0.45).
- Change Horizon: Reduce the Horizon to 1. The trace truncates early, leaving only a single TD residual of 0.45, regardless of the Trace lambda setting.
- Test True Terminal: With Horizon 3 and Trace lambda 0.5, switch Ending to "True terminal". The bootstrap value VH is forced to 0. A0 changes to 0.680625 and the Lambda-return target to 1.680625 because the final reward is no longer supplemented by the subsequent value approximation.