Learning to Parallel

Autoregressive decoding in large language models (LLMs) requires \(\mathcal{O}(n)\) sequential steps for \(n\) tokens, fundamentally limiting inference throughput. Recent diffusion-based LLMs (dLLMs) enable parallel token generation through iterative denoising. However, current parallel decoding strategies rely on fixed, input-agnostic heuristics (e.g., confidence thresholds), which fail to adapt to input-specific characteristics, resulting in suboptimal speed-quality trade-offs across diverse NLP tasks. In this work, we explore a more flexible and dynamic approach to parallel decoding. We propose \(\textbf{Learning to Parallel Decode (Learn2PD)}\), a framework that trains a lightweight and adaptive filter model to predict, for each token position, whether the current prediction matches the final output. This learned filter approximates an oracle parallel decoding strategy that unmasks tokens only when correctly predicted. Importantly, the filter model is learned in a post-training manner, requiring only a small amount of computation to optimize it (minute-level GPU time). Additionally, we introduce \(\textbf{End-of-Text Prediction (EoTP)}\) to detect decoding completion at the end of sequence, avoiding redundant decoding of padding tokens. Experiments on the LLaDA benchmark demonstrate that our method achieves up to \(\textbf{22.58×}\) speedup without any performance drop, and up to \(\textbf{57.51×}\) when combined with KV-Cache.