Chenchen Zhang
A survey paper systematically categorizing 47 methods for the credit assignment problem caused by sparse rewards in LLM reinforcement learning, divided into two regimes: reasoning RL and agentic RL.
When applying reinforcement learning to LLMs, long sequences (500–30K+ tokens) or multi-turn environments (100+ turns) only provide final rewards, making it difficult to know which tokens or actions contributed to success. This credit assignment problem manifests differently in reasoning RL (chain-of-thought generation) and agentic RL (environment interaction).
We collected 47 credit assignment methods (41 core, 6 adjacent enablers) published between 2024 and early 2026 and organized them into a two-dimensional taxonomy. The first axis is assignment granularity (token, segment, step, turn, multi-agent), and the second axis is methodology (Monte Carlo, temporal difference, model-based, game-theoretic, information-theoretic). Additionally, we provide a machine-readable paper inventory, a reporting checklist, and a benchmark protocol specification.
We find that in reasoning RL, process reward models and critic-free group comparison are maturing, while in agentic RL, new approaches such as hindsight counterfactual analysis, privileged asymmetric critics, and turn-level MDP reformulations are emerging. Our systematic categorization and reusable resources help fill methodological gaps in future research.