Rodrigo Casado Noguerales, Bernhard Schölkopf, Thomas Hofmann, Aran Raoufi
This research rigorously identifies the mathematical nature of what discrete diffusion models learn and their optimization objective, unifying existing models under a single framework.
It was unclear what neural networks in discrete diffusion models actually learn (e.g., denoiser, score ratio), and conflicting interpretations in different coordinate systems could distort model training and sampling.
The authors rigorously derive the ELBO for continuous-time Markov chains (CTMC) and prove the 'Oracle Distance' theorem, showing the negative ELBO exactly equals data entropy plus the path KL divergence from the oracle reverse process. For sequences with token-factorizing noise, they derive three exact coordinates for the optimizer (denoiser, cavity, score) and closed-form conversions among them.
The framework recovers existing models like MDM, UDM, SEDD, and GIDD as special cases and explains why denoiser and cavity coincide for masked diffusion but not for uniform diffusion. It proves that uniform ELBO diverges at initialization while the bridge plug-in stays finite, and all identities are verified numerically on an exactly solvable model without approximation.