[Paper] [ArXiv] [Code (Coming Soon)]
ADLM introduces an anchor network that predicts important (e.g., 349 (‘cat’) and 329 (‘dog’)) token mixture of a sequence. These anchored predictions guide a denoiser network to better estimate the likelihoods of masked (50257) tokens. Here, we illustrate the pathways for tokens: 1760 (‘playing’) and 64 (‘a’). ADLM anchors through important tokens that help narrow the performance gap with autoregressive models.
Diffusion Language Models (DLMs) promise parallel generation and bidirectional context, yet they underperform autoregressive (AR) models in both likelihood modeling and generated text quality. We identify that this performance gap arises when important tokens (e.g., key words or low-frequency words that anchor a sentence) are masked early in the forward process, limiting contextual information for accurate reconstruction. To address this, we introduce the Anchored Diffusion Language Model (ADLM), a novel two-stage framework that first predicts distributions over important tokens via an anchor network, and then predicts the likelihoods of missing tokens conditioned on the anchored predictions. ADLM significantly improves test perplexity on LM1B and OpenWebText, achieving up to 25.4% gains over prior DLMs, and narrows the gap with strong AR baselines. It also achieves state-of-the-art performance in zero-shot generalization across seven benchmarks and surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model. Theoretically, we derive an Anchored Negative Evidence Lower Bound (ANELBO) objective and show that anchoring improves sample complexity and likelihood modeling. Beyond diffusion, anchoring boosts performance in AR models and enhances reasoning in math and logic tasks, outperforming existing chain-of-thought approaches.
Test perplexities (PPL↓) on LM1B and OWT. †Reported in (Sahoo et al., 2024). Bold: Best diffusion method. We retrain AR and MDLM to match performance reported in original papers. Our method outperforms previous diffusion language models using the same number of training tokens.
GPT2-Large perplexities (PPL; ↓) on OWT (524B tokens). We evaluate ADLM with the locked-in and remasking samplers each with 1000 steps.
Zero-shot validation perplexities (↓) of models trained on 524B tokens from OWT. ADLM achieves a new state-of-the-art among diffusion language models and outperforms autoregressive (AR) models on three benchmarks: Lambada, PubMed, and ArXiv. All models use 1024 NFEs.
Generative perplexities evaluated over 1024 unconditional generations using GPT2Large (774M params) as the evaluation model. ADLM significantly outperforms prior masked and diffusion language models under comparable sampling configurations. Notably, it surpasses Plaid (Gulrajani & Hashimoto, 2023) while using only ∼20% of the parameters, and nearly matches the performance of GPT2-medium despite having ∼50M fewer parameters.
Evaluation of sample quality using the ADLM with remasking sampler (Wang et al., 2025a) on OWT. ADLM† outperforms state-of-the-art masked diffusion and flow-matching methods. For T = 2048 and T = 4096, ADLM surpasses AR in MAUVE score (measures human-like text).
Anchored diffusion becomes the first diffusion language model (DLM) to outperform autoregressive LLMs in terms of MAUVE score, while also significantly narrowing the gap in generative perplexity (measured by GPT2-Large) and achieving comparable natural language diversity (measured by entropy).
Training of standard autoregressive (AR) models. A neural network is trained to predict the next token using causal attention (leftto-right context). All tokens contribute equally to the training loss, and the model treats the sequence uniformly without structural guidance.
Training of anchored autoregressive (A2R) models. An anchor network first identifies important tokens (e.g., ‘cat’, ‘dog’ shown in blue), which are supervised via an auxiliary anchor loss. A lightweight LLM is then trained to predict the next token based on anchored predictions.
Anchoring improves autoregressive modeling on OWT. Test perplexities (PPL; ↓) for standard AR models and our anchored variant (A2R) at various training scales. † Results from (Sahoo et al., 2024). A2R consistently improves perplexity by introducing a two-stage prediction process: anchor tokens are first predicted, then used to guide next-token prediction.
Accuracy (%) on Math and Logical Reasoning. ACoT improves the performance of prior (continuous) latent reasoning methods despite using the same multi-stage training setup as COCONUT. † indicates the results reported in COCONUT.
A key insight from Question 1 is that standard CoT processes the question in a purely left-to-right manner. It computes 16 - 3 - 4 = 9, and then 9 * 2 = 18, following the order in which quantities appear in the question. In contrast, ACoT introduces [ANT] to capture important tokens, which allows it to reason more globally. Specifically, ACoT first computes 3 + 4 = 7 to aggregate all consumption before subtracting from the total (16 - 7 = 9), a pattern more aligned with human intuition. This demonstrates how anchoring enables a “look-ahead” planning behavior, in contrast to the left-to-right decoding bias of standard CoT.
Anchoring enables a “look-ahead” planning behavior, which is in contrast to the commonly observed left-to-right decoding bias of standard chain-of-thought (CoT) reasoning.
Examples of logical reasoning tasks with symbolic reasoning traces from ProntoQA (Saparov & He, 2023). Each row shows the input question, groundtruth reasoning trace (CoT), answer, the model’s generated output sequence, and the extracted final answer. CoT, COCONUT, and ACoT share nearly identical training setups, with the only distinction being the use of anchor tokens in ACoT and continuous thoughts in COCONUT. Our model implicitly reasons through anchoring tokens ([ANT]) to infer logical relationships.
While standard multi-stage training in COCONUT leads to a marginal improvement from 98.8% (CoT) to 99.8%, our anchoring mechanism further enhances performance. By incorporating explicit supervision through [ANT], ACoT achieves a perfect accuracy of 100% on this relatively easier benchmark, demonstrating the effectiveness of anchoring in logical reasoning.
Examples of logical reasoning tasks with symbolic reasoning traces from ProsQA (Hao et al., 2024). Each row shows the input question, groundtruth reasoning trace (CoT), answer, our model’s generated output sequence, and the extracted final answer. Our model implicitly reasons through anchoring tokens ([ANT]) to infer logical relationships. In this experiment, we employ the gradual CoT removal scheme used in prior works (Deng et al., 2024; Hao et al., 2024) to demonstrate the reasoning ability in the anchored latent space without producing word tokens.
While standard multi-stage training in COCONUT leads to a marginal improvement from 98.8% (CoT) to 99.8%, our anchoring mechanism further enhances performance. By incorporating explicit supervision through [ANT], ACoT achieves a perfect accuracy of 100% on this relatively easier benchmark, demonstrating the effectiveness of anchoring in logical reasoning.
Sampling Strategy: Remasking sampler with 4096 sampling steps
Note: The generated text demonstrates strong discourse-level coherence, well-structured paragraphs, and natural transitions between topics. For instance, the model begins with a detailed sports commentary and transitions smoothly into a socio-political news report. These shifts are logically segmented with paragraph-level breaks and consistent timelines (e.g., 18:00 PST, 30:00 PST), giving the impression of human-authored journals. This qualitative improvement aligns with the quantitative results reported in Table 3, where ADLM achieves the highest MAUVE score among existing DLMs.
@article{rout2025anchored,
title = {Anchored Diffusion Language Model},
author = {Rout, Litu and Caramanis, Constantine and Shakkottai, Sanjay},
booktitle = {arXiv preprint},
year = {2025},
url = {https://arxiv.org/pdf/2505.18456}
}
Acknowledgements: This research has been supported by NSF Grants 2019844 and 2112471, the UT Austin Machine Learning Lab, and computing support on the Vista GPU Cluster through the Center for Generative AI (CGAI) and the Texas Advanced Computing Center (TACC) at UT Austin.