💡 The paper introduces Orthrus, a dual architecture framework that combines the strengths of autoregressive large language models and diffusion models to achieve fast parallel token generation while maintaining exact inference fidelity. The problem with standard autoregressive decoding is that it is sequential, which represents a fundamental bottleneck for high throughput inference. Diffusion language models try to address this issue with parallel generation, but they suffer from performance degradation, high training costs, and lack of convergence guarantees.

The Orthrus framework resolves this issue by augmenting a frozen large language model with a lightweight trainable module to create a parallel diffusion view alongside the standard autoregressive view. Both views attend to the same high fidelity key value cache, where the autoregressive head executes context pre filling to construct accurate key value representations, and the diffusion head executes parallel generation. The framework employs an exact consensus mechanism between the two views to guarantee lossless inference.

The results show that Orthrus delivers a speedup of up to 7.8 times with only a constant memory cache overhead and minimal parameter additions. This is achieved by sharing key value caches and using a consensus mechanism, which allows the framework to maintain exact inference fidelity while generating tokens in parallel. Overall, the Orthrus framework provides a simple and efficient solution to the problem of slow sequential decoding in autoregressive large language models, and it has the potential to be seamlessly integrated into existing transformer architectures.

💡 The paper introduces KVarN, a new method for quantizing KV-cache in large language models to reduce error accumulation during autoregressive decoding. The problem addressed is that test-time scaling, which improves reasoning in large language models, becomes memory-bottlenecked during long-horizon decoding due to the growing KV-cache. Existing KV-cache quantization methods are not effective in this setting because they are evaluated under prefill-like settings, where errors behave differently than in autoregressive decoding. In autoregressive decoding, quantization errors accumulate across timesteps, primarily due to incorrect token scales.

The KVarN method addresses this issue by applying a Hadamard rotation followed by a dual-scaling variance normalization across both axes of the K and V matrices. This combination fixes outlying token-scale errors and substantially reduces error accumulation. The method is calibration-free, meaning it does not require any additional calibration steps.

The results show that KVarN establishes a new state-of-the-art for KV-cache quantization on generative benchmarks, including MATH500, AIME24, and HumanEval, at 2-bit precision. This means that KVarN is able to achieve better performance than existing methods while using less memory. The KVarN method is also available for implementation in large language models, providing a practical solution to the problem of error accumulation in autoregressive decoding. Overall, the paper contributes a new and effective method for quantizing KV-cache in large language models, which can improve the performance and efficiency of these models in reasoning tasks.

💡 The paper proposes a parallelized autoregressive framework for dense video captioning, which aims to generate temporally grounded descriptions of video events. The existing autoregressive models have limitations in terms of inference efficiency and scalability due to the token-by-token generation paradigm. To address this issue, the authors propose a framework that exploits weak local dependencies across temporally distinct events to enable parallel generation of tokens. The key insight is to restructure the causal dependency graph to allow for lossless parallel decoding, where tokens with weak cross-event dependencies can be decoded in parallel, while tightly coupled tokens within each event retain sequential decoding.

The authors introduce two key components to realize this insight: a latent global planning mechanism and an event-factorized parallel decoding mechanism. The latent global planning mechanism learns the event-level structure and produces compact tokens encoding global inter-event causality, while adaptively aggregating event-level audio-visual semantics. The event-factorized parallel decoding mechanism balances local focus with global inter-event awareness, enabling effective parallel decoding.

The proposed framework improves generation efficiency and enhances temporally grounded captioning performance. The experiments on various benchmarks demonstrate the clear advantage of the approach in both efficiency and performance in omni-modal event grounding and captioning. The results show that the parallelized autoregressive framework can generate dense captions more efficiently and accurately, making it a promising solution for large-scale video captioning tasks. Overall, the paper contributes to the development of more efficient and effective dense video captioning models, which can benefit both event-level video understanding and generation.