Changdae Oh, Wendi Li, Seongheon Park, Samuel Yeh, Tanwi Mallick, Sharon Li
The 'progress advantage,' a signal automatically generated during RL post-training, enables evaluating and improving agent step-level performance without a dedicated reward model.
Building process reward models for agentic settings is prohibitively difficult. Long-horizon interactions, irreversible actions, and stochastic environment feedback make human annotation and Monte Carlo estimation infeasible at scale.
The paper derives an implicit advantage called 'progress advantage' under a general stochastic Markov decision process (MDP). It is the log-probability ratio between the RL-trained policy and its reference policy, which exactly recovers the optimal advantage function. This signal is annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline.
The effectiveness of progress advantage is validated across three applications (test-time scaling, uncertainty quantification, failure attribution) on five benchmarks and four model families. It consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models in all settings.