💡 The paper introduces SmolDocling, a compact vision-language model designed for end-to-end document conversion. The model aims to process entire pages and generate a new universal markup format called DocTags, which captures all page elements in their full context with location. Unlike existing approaches that rely on large foundational models or ensemble solutions, SmolDocling offers a single end-to-end conversion model with 256M parameters. This approach allows for accurately capturing content, structure, and spatial location of document elements.

The model is trained to reproduce document features such as code listings, tables, equations, charts, lists, and more across a diverse range of document types, including business documents, academic papers, technical reports, patents, and forms. The authors also contribute novel publicly sourced datasets for charts, tables, equations, and code recognition.

Experimental results demonstrate that SmolDocling performs competitively with other vision language models that are up to 27 times larger in size, while reducing computational requirements substantially. The model's compact size and robust performance make it a significant contribution to the field of document conversion. The authors plan to make the model and datasets publicly available, which will facilitate further research and development in this area. Overall, SmolDocling offers a efficient and effective solution for end-to-end document conversion, with potential applications in various industries and domains.

💡 The paper introduces LeWorldModel, a stable end to end joint embedding predictive architecture that trains efficiently from raw pixels. Existing methods for learning world models in compact latent spaces are fragile and rely on complex loss terms, pre trained encoders, or auxiliary supervision to avoid representation collapse. LeWorldModel addresses this issue by using only two loss terms, a next embedding prediction loss and a regularizer, to train the model end to end from raw pixels. This approach reduces the number of tunable loss hyperparameters from six to one compared to existing methods. The model has approximately 15 million parameters and can be trained on a single GPU in a few hours, making it up to 48 times faster than foundation model based world models. The results show that LeWorldModel remains competitive across diverse 2D and 3D control tasks and encodes meaningful physical structures in its latent space. The model is also able to reliably detect physically implausible events, demonstrating its ability to learn a robust and generalizable representation of the world. Overall, LeWorldModel provides a stable and efficient framework for learning world models from raw pixels, making it a significant contribution to the field of artificial intelligence.

💡 The paper introduces dots.ocr, a unified Vision-Language Model that achieves state-of-the-art performance on document layout parsing by jointly learning layout detection, text recognition, and relational understanding. The current methods for document layout parsing rely on fragmented, multi-stage pipelines that suffer from error propagation and fail to leverage the synergies of joint training. The proposed model addresses this issue by using a single Vision-Language Model that jointly learns the three core tasks within a unified, end-to-end framework. This is made possible by a highly scalable data engine that synthesizes a vast multilingual corpus, enabling the model to deliver robust performance across a wide array of tasks, languages, layouts, and domains. The model is validated on the OmniDocBench and XDocParse benchmarks, with the latter being a new challenging benchmark introduced in the paper that spans 126 languages. The results show that dots.ocr establishes a powerful new baseline, outperforming the next-best competitor by a 7.4 point margin and proving its unparalleled multilingual capabilities. The paper's contributions include the introduction of a unified Vision-Language Model that achieves state-of-the-art performance on document layout parsing, the creation of a new benchmark for multilingual document intelligence, and the demonstration of the advantages of jointly learning layout detection, text recognition, and relational understanding within a single model.

💡 The paper presents SCAIL-2, a framework for controlled character animation that enables end-to-end motion transfer from driving videos to reference characters without using intermediate representations. Prior methods relied on intermediate representations such as pose skeletons or masked backgrounds, which led to information loss. SCAIL-2 addresses this issue by directly concatenating driving videos to the sequence, allowing the model to obtain all required visual information from the input video.

To overcome the lack of end-to-end data, the authors unify sub-tasks of character animation with decoupled conditions and create a pipeline to synthesize a large dataset called MotionPair-60K, which contains heterogeneous tasks of character animation. The framework utilizes in-context mask conditioning and mode-specific RoPE as soft guidance beyond textual instructions and raw visual information.

The authors also propose Bias-Aware DPO to mitigate errors caused by synthetic discrepancies in detailed regions. This approach constructs preference items to address the issue. Extensive experiments demonstrate that SCAIL-2 substantially outperforms existing state-of-the-art approaches in various character animation tasks.

The key contributions of the paper are the development of an end-to-end character animation framework that bypasses intermediate representations, the creation of a large synthetic dataset for motion transfer, and the proposal of a novel method to address synthetic discrepancies. The results show that SCAIL-2 achieves superior performance compared to existing methods, and the authors plan to release a large subset of synthetic data and model weights to facilitate further research.

💡 The paper introduces GEAR, a method for training a vector-quantized tokenizer and an autoregressive generator jointly and end-to-end for image synthesis. Typically, these models are trained in two stages, where the tokenizer is first trained and then frozen, and then the generator is trained on its output. However, this approach has a limitation, as the tokenizer is not aware of what the generator finds easy to model.

GEAR overcomes this limitation by training the tokenizer and generator jointly, guided by representation alignment. The key challenge is that the output of the tokenizer is non-differentiable, making it difficult to train the tokenizer and generator jointly. To address this, GEAR uses a dual read-out approach, where the tokenizer output is used in two different ways. A hard, one-hot branch is used to train the autoregressive generator, while a differentiable soft branch is used to carry a representation-alignment loss that guides the tokenizer.

This approach allows the autoregressive generator to steer the tokenizer towards an index distribution that it can predict more easily. As a result, the tokenizer's features become less complex, while the autoregressive generator's features become more complex and semantic. The paper demonstrates that GEAR speeds up convergence by up to 10 times relative to a strong baseline, and learns better patch-level and spatially-coherent features. Additionally, GEAR generalizes across different quantizers and can be applied to text-to-image generation. Overall, GEAR provides a new approach for training visual generative models, and achieves state-of-the-art results in image synthesis.