d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning

TL;DR

This is the first study to extend reasoning capabilities in diffusion-based LLMs via reinforcement learning, achieving significant performance gains through masked SFT and the diffu-GRPO algorithm.

Problem

Existing diffusion-based LLMs (dLLMs) achieve good language modeling performance, but it is unclear whether they can benefit from RL-based reasoning training like autoregressive models. This work investigates whether dLLMs can leverage RL for reasoning improvement.

Approach

We propose d1, a framework that fine-tunes pre-trained masked dLLMs using SFT and RL. Specifically: (a) we use masked SFT to distill knowledge and instill self-improvement behavior from existing datasets; (b) we introduce diffu-GRPO, a novel critic-free, policy-gradient-based RL algorithm, which is the first integration of policy gradient methods into masked dLLMs.

Results & Contribution

We achieve significant performance improvements over state-of-the-art dLLMs on multiple mathematical and planning benchmarks. This is the first successful application of RL-based reasoning training to dLLMs, opening up the possibility of extending reasoning capabilities in the non-autoregressive generation paradigm.