Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

Status: preprint on arXiv, under review at TMLR · My role: investigated AI-for-Science applications (13th of 23 authors).

Paper (arXiv) Curated paper list

Summary

In continuous diffusion, Gaussian noise gives a natural geometry. In discrete domains — text, code, proteins, molecules, action sequences — there is no obvious notion of a “small perturbation”. The survey’s central thesis is that tokenization is the primary design axis: how the discrete state space is built determines the topology of corruption, how hard denoising is, and downstream controllability, validity, and cost.

• A tokenization-centric lens over three token families: semantic (subwords), quantized (VQ codebooks), and natural alphabets (amino acids, nucleotides, atom types).
• A four-component framework — corruption operator, denoiser parameterization, training objective, sampler — under which D3PM, masked diffusion, SEDD, and discrete flow matching emerge as special cases, often differing in a single component.
• A cross-domain map spanning text and code, multimodal generation, proteins, genomics, molecules and graphs, planning and agents, and tabular data.

Led by Ye Yuan in Prof. Steve Liu’s Cyber-Physical Intelligence Lab, with collaborators across nine institutions and guidance from Prof. Dawn Song and Prof. Philip Yu.

Recommended citation: Ye Yuan, Weien Li, Rui Song, et al. (including Yonghan Yang), Dawn Song, Philip S. Yu, and Xue Liu. (2026). "Discrete Diffusion Models: A Unified Framework from Tokenization to Generation." arXiv:2607.13431. Under review at TMLR.
Download Paper