🔥 OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

💡 The paper proposes a new method called OSCAR for ultra-low-bit key-value cache quantization, which is crucial for efficient deployment of large language models. The problem addressed is that existing quantization methods, such as simple rotations like Hadamard transforms, degrade in accuracy when applied to very low-bit representations, like 2-bit integers. This degradation occurs because these methods do not account for the attention-aware covariance structures that the model actually uses.

To solve this problem, OSCAR estimates the attention-aware covariance structures offline and uses them to derive fixed rotations and clipping thresholds for quantization. This approach aligns the key-value cache quantization with the covariance structures that the model consumes, leading to higher accuracy and efficiency.

The authors provide theoretical justification for OSCAR and develop a fully deployable system that is compatible with modern large language model serving frameworks. They evaluate OSCAR on several reasoning models with long context lengths, up to 32,000 tokens, and achieve significant improvements in accuracy compared to naive rotation methods. Specifically, OSCAR reduces the accuracy gap to 3.78 and 1.42 points on two models, while naive rotation methods collapse to nearly zero.

The results also show that OSCAR scales well to larger models, remaining effectively on par with higher-precision representations. Additionally, OSCAR achieves significant system-wise improvements, including reducing key-value cache memory by approximately 8 times, improving throughput by up to 7 times, and accelerating batch-size-1 decoding by up to 3 times over higher-precision representations. Overall, the paper demonstrates that OSCAR is an effective and efficient method for ultra-low-bit key-value cache quantization, enabling the deployment of large language models with high accuracy and efficiency.

📅 Published on May 18

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.17757
• PDF: https://arxiv.org/pdf/2605.17757
• Project Page: https://oscar-quantize.github.io/

🤖 Models citing this paper:
• https://huggingface.co/Zhongzhu/OSCAR-RotationZoo

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#QuantizationMethods #LowBitRepresentations #KeyvalueCache #SpectralCovariance #EfficientDeployment