💡 The paper introduces AURORA-LM, a continuous-latent diffusion language model that addresses the problem of text generation in continuous latent spaces. Unlike existing models that either inherit embedding spaces not designed for joint generation and decoding or compress autoencoded latent spaces to ease diffusion, sacrificing token-level fidelity, AURORA-LM preserves a high-capacity, decodable text latent and designs the diffusion model to learn its distribution directly.

The model consists of a query-based encoder-decoder that organizes text into a high-capacity, prefix-aligned latent sequence, and a block-causal diffusion transformer that learns its distribution through flow matching, generating blocks left to right while denoising positions within each block in parallel. To accommodate the harder-to-model latent, AURORA-LM restricts only the noisy-input pathway while retaining the full clean-latent prediction target, accommodating full-width latents without reducing decoder-facing capacity.

The authors further calibrate the noise-level distribution to the latent width and introduce self-trajectory consistency to bridge independently sampled training noise and iterative denoising at inference. The results show that AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. Scaling to 1 billion parameters with about 1500 EFLOPs of total compute yields further gains, surpassing a larger publicly released latent-diffusion language model under a matched evaluation protocol. All experiments are conducted on Ascend NPUs.

The key contributions of the paper are the introduction of a novel continuous-latent diffusion language model that preserves high-capacity, decodable text latents and designs the diffusion model to learn its distribution directly, and the demonstration of its effectiveness in achieving state-of-the-art results on text generation and summarization tasks.