✨SO-Bench: A Structural Output Evaluation of Multimodal LLMs

📝 Summary:
SO-Bench is a new benchmark evaluating MLLMs ability to generate schema-compliant structured outputs from visual inputs. It reveals significant gaps in current models performance, highlighting the need for better multimodal structured reasoning.

🔹 Publication Date: Published on Nov 23

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.21750
• PDF: https://arxiv.org/pdf/2511.21750

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MultimodalLLMs #StructuredOutput #LLMEvaluation #AIResearch #ComputerVision

✨Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment

📝 Summary:
The SANTA framework addresses object and action hallucinations in multimodal LLM video captions. It uses self-augmented contrastive alignment to identify potential hallucinations and then aligns regional objects and actions with visual phrases, improving factual accuracy. Experiments show SANTA o...

🔹 Publication Date: Published on Dec 4

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.04356
• PDF: https://arxiv.org/pdf/2512.04356
• Project Page: https://kpc0810.github.io/santa/
• Github: https://kpc0810.github.io/santa/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MultimodalLLMs #AI #Hallucinations #VideoUnderstanding #ContrastiveLearning

✨OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs

📝 Summary:
OpenRT is an open-source framework that unifies and modularizes red-teaming for multimodal LLMs. It exposes significant safety gaps in frontier models, which fail to generalize across diverse attacks, showing attack success rates up to 49.14%.

🔹 Publication Date: Published on Jan 4

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.01592
• PDF: https://arxiv.org/pdf/2601.01592
• Project Page: https://ai45lab.github.io/OpenRT/
• Github: https://github.com/AI45Lab/OpenRT

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#RedTeaming #MultimodalLLMs #AISafety #LLMSecurity #AIResearch

✨FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

📝 Summary:
FutureOmni is the first benchmark evaluating multimodal models ability to forecast future events from audio-visual data. Current models struggle, particularly with speech-heavy scenarios. The paper proposes an improved training strategy, Omni-Modal Future Forecasting, which enhances performance a...

🔹 Publication Date: Published on Jan 20

🔹 Paper Links:
• arXiv Page: https://arxiv.org/pdf/2601.13836
• PDF: https://arxiv.org/pdf/2601.13836
• Project Page: https://openmoss.github.io/FutureOmni
• Github: https://openmoss.github.io/FutureOmni

✨ Datasets citing this paper:
• https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MultimodalLLMs #FutureForecasting #AIResearch #DeepLearning #Benchmarking

🔥 Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

💡 The paper explores the design space for multimodal large language models that use a mixture of vision encoders and resolutions to improve performance. The goal is to accurately interpret complex visual information, which is crucial for tasks such as optical character recognition and document analysis. Recent work has shown that using multiple vision encoders can enhance visual perception and reduce hallucinations, but there is a lack of systematic comparisons and detailed ablation studies on this topic.

To address this, the authors conducted an extensive exploration of the design space for multimodal large language models using a mixture of vision encoders and resolutions. They found that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. They also introduced a Pre-Alignment mechanism to bridge the gap between vision-focused encoders and language tokens, which enhances model coherence.

The resulting family of multimodal large language models, called Eagle, surpasses other leading open-source models on major benchmarks. The authors discovered that their streamlined yet effective design approach is based on several underlying principles common to various existing strategies. The Eagle models and code are available online, providing a valuable resource for the research community. Overall, the paper contributes to the development of more effective multimodal large language models by providing a systematic exploration of the design space and introducing a simple yet effective design approach.

📅 Published on Aug 28, 2024

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2408.15998
• PDF: https://arxiv.org/pdf/2408.15998
• Project Page: https://huggingface.co/papers/2407.02392

🤖 Models citing this paper:
• https://huggingface.co/NVEagle/Eagle-X5-13B-Chat
• https://huggingface.co/NVEagle/Eagle-X5-7B
• https://huggingface.co/NVEagle/Eagle-X5-13B

📊 Datasets citing this paper:
• https://huggingface.co/datasets/shi-labs/Eagle-1.8M

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/merve/vision_papers
• https://huggingface.co/spaces/NVEagle/Eagle-X5-13B-Chat
• https://huggingface.co/spaces/shaktibiplab/Eagle-X5-13B-Chat

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLLMs #MixtureOfEncoders #VisionEncoderDesign #OpticalCharacterRecognition #DocumentAnalysisModels