Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Continuous Diffusion Model for Language Modeling
Authors: Jaehyeong Jo, Sung Ju Hwang
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Comprehensive experiments on language modeling benchmarks and other modalities show that our method outperforms existing discrete diffusion models and approaches the performance of autoregressive models. |
| Researcher Affiliation | Collaboration | Jaehyeong Jo1, Sung Ju Hwang1,2 KAIST1, Deep Auto.ai2 EMAIL |
| Pseudocode | Yes | Algorithm 1 Training Input: Initial point u, model pθ, vocabulary size d, token sequence length L, time distribution q(t), pre-computed {αi/K, ρi/K}K i=0 Algorithm 2 Sampling Input: Initial point u, trained model pθ, vocabulary size d, number of sampling steps M, token sequence length L, noise schedule σt Algorithm 3 Pre-computing parameters of Riemannian normal before training Input: Initial point u, vocabulary size d, number of simulations N, number of discretization steps K, noise schedule σt, time change coefficient γt |
| Open Source Code | Yes | The code is available at https://github.com/harryjo97/RDLM. |
| Open Datasets | Yes | We evaluate our Riemannian Diffusion Language Model (RDLM) for text generation tasks on two language benchmarks: Text8 [42] and One Billion Words Dataset [7]. As shown in Table 3, our method achieves the lowest Bits Per Dimension (BPD), outperforming the discrete diffusion models [2, 52] and autoregressive baselines [11, 13]. We evaluate our method on the promoter DNA sequence design task, which aims to generate valid promoter DNA sequences conditioned on transcription profiles. A detailed description of the task is provided in Appendix B.4. |
| Dataset Splits | Yes | Following the previous works [2, 39, 49], we split the dataset into 90M/5M/5M with a fixed sequence length of 256. For One Billion Words, we use the same tokenizer as in He et al. [25] with context size 128. Following the previous work [15, 54], we use the same data split of 88,470/3,933/7,497 and identical model architecture consisting of 20-layer 1-D CNN with 13.3M trainable parameters. |
| Hardware Specification | Yes | For all experiments, we use NVIDIA RTX A5000 and H100. |
| Software Dependencies | No | The paper mentions a "diffusion transformer architecture [44] with rotary positional embeddings [55]" and an "optimizer Adam W [38]". However, it does not explicitly list specific version numbers for software components such as programming languages, deep learning frameworks (e.g., PyTorch, TensorFlow), or CUDA, which are necessary for reproducible software dependencies. |
| Experiment Setup | Yes | We train our model for 1M iterations with batch size 512 as done in previous works, using the same learning rate, optimizer Adam W [38], and exponential moving average (EMA) with decay rate 0.9999. We train our model for 100k iterations with batch size 256 and Adam W [38] optimizer. |