Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He et al.
We show that reinforcement learning with a single training example (1-shot RLVR) can significantly improve the math reasoning capabilities of LLMs.
Reinforcement learning with verifiable reward (RLVR) for improving math reasoning in LLMs typically requires thousands of training examples. We investigate whether extreme reduction in the number of examples can still be effective and explore the underlying mechanism.
We apply RLVR using only a single math problem example to the Qwen2.5-Math-1.5B model, using GRPO and PPO algorithms. We promote exploration by appropriately tuning the entropy loss coefficient. Experiments are conducted across various models and examples, and the causes of performance improvement are analyzed.
1-shot RLVR improves MATH500 accuracy from 36.0% to 73.6% and average performance across six math benchmarks from 17.6% to 35.7%, matching results obtained with 1,200 examples. We observe cross-category generalization, increased frequency of self-reflection, and post-saturation generalization (test performance continues to improve after training accuracy saturates). We verify that the policy gradient loss is the primary driver, distinguishing it from grokking, and highlight the importance of promoting exploration.