Simplex Diffusion Models seek to curb information collapse in discrete diffusion

A 29 September preprint proposes Simplex Diffusion Models (SDMs) to reduce information loss in discrete diffusion. According to the preprint, current discrete diffusion models discard uncertainty at intermediate steps through categorical sampling, an effect it calls information collapse that degrades sample quality arXiv:2609.35553.

SDMs lift the process to a probability simplex, carrying uncertainty across denoising steps. Unlike Dirichlet Flow Matching, which requires integrating an ordinary differential equation, SDMs use a tunable stochastic sampler. The authors say this mitigates information collapse seen in prior work.

On OpenWebText, SDMs achieve a generative perplexity of 17.0 at 5.46 unigram entropy in 64 sampling steps. For code generation on TinyGSM at temperature 0.1, they reach 49.0% accuracy without self-conditioning, outperforming masked and uniform diffusion models that use self-conditioning (45.8%). Distilled to eight steps, SDMs solve 32.1% of GSM8K problems, exceeding the 21.4% achieved by distilled discrete diffusion models using 128 steps.

It has not been peer-reviewed; independent replication has not been reported.