💡 The paper introduces ViQ, a visual quantization framework that aims to balance semantic richness and detail preservation in discrete representations of images. The goal is to create a unified representation for text and vision that enables simpler multimodal modeling and more efficient training. Existing methods struggle to balance low-level details and high-level semantics in discrete representations, often resulting in severe information loss.

The ViQ framework addresses this issue by structuring quantization learning into two stages: text-aligned pre-training and feature discretization. In the first stage, the visual encoder is pre-trained with semantic-rich supervision from a pre-trained language model, allowing it to process native-resolution visual inputs. In the second stage, a proximal representation learning strategy is used to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions.

The results show that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. Additionally, multimodal training with visual quantized representations using ViQ leads to significant efficiency improvements, with up to 20-70 percent acceleration in training time compared to different base language models and training recipes. Overall, the paper presents a novel approach to visual quantization that balances semantics and details in discrete representations, enabling more efficient and effective multimodal modeling.