✨Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following

📝 Summary:
Multi-Crit evaluates multimodal models as judges on following diverse criteria using novel metrics. Findings reveal current models struggle with consistent adherence and flexibility to pluralistic criteria. This highlights gaps in capabilities and lays a foundation for building reliable AI evalua...

🔹 Publication Date: Published on Nov 26

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.21662
• PDF: https://arxiv.org/pdf/2511.21662
• Project Page: https://multi-crit.github.io/
• Github: https://multi-crit.github.io/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MultimodalAI #AIEvaluation #BenchmarkingAI #AIJudges #MachineLearning

✨ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models

📝 Summary:
This paper introduces ProactiveBench to measure if MLLMs can proactively ask for user help on challenging tasks. It finds MLLMs generally lack this proactiveness, and conversational history can even hinder it. However, reinforcement learning shows promise for teaching models this crucial collabor...

🔹 Publication Date: Published on Mar 19

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.19466
• PDF: https://arxiv.org/pdf/2603.19466
• Project Page: https://huggingface.co/datasets/tdemin16/ProactiveBench
• Github: https://github.com/tdemin16/proactivebench

✨ Datasets citing this paper:
• https://huggingface.co/datasets/tdemin16/ProactiveBench

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MLLMs #AIProactiveness #BenchmarkingAI #ReinforcementLearning #LargeLanguageModels

✨GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

📝 Summary:
GTA-2 is a new benchmark for General Tool Agents, covering both atomic and real-world, open-ended workflows. It shows frontier models struggle significantly, especially on workflows. The study emphasizes that execution frameworks are crucial for performance, more so than just model capacity.

🔹 Publication Date: Published on Apr 17

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.15715
• PDF: https://arxiv.org/pdf/2604.15715
• Github: https://github.com/open-compass/GTA

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AIAgents #BenchmarkingAI #LLMs #AIWorkflows #AIResearch

🔥 EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

💡 The paper introduces EnterpriseClawBench, a benchmark for evaluating enterprise agents based on real-world sessions. Enterprise agents are increasingly used in workspaces to read files, invoke tools, and deliver business artifacts. However, existing evaluation metrics are limited, focusing on single performance scores. To address this, the authors created EnterpriseClawBench, which consists of 852 reproducible tasks derived from a large archive of workplace sessions. Each task is paired with relevant information such as fixtures, prompts, and semantic rubrics.

The benchmark is not publicly released due to the proprietary nature of the data, but the construction and evaluation protocol is made available. The authors used this benchmark to evaluate the performance of various agent configurations and found that the best configuration achieved a score of 0.663, indicating that there is still significant room for improvement.

The key contribution of this paper is the introduction of a comprehensive evaluation protocol that goes beyond single performance scores. The authors argue that enterprise agent evaluation should consider multiple factors, including harness-model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior. This approach provides a more nuanced understanding of an agent's capabilities and limitations, allowing for more effective evaluation and development of enterprise agents.

Overall, the paper highlights the need for more comprehensive evaluation metrics for enterprise agents and provides a benchmark and evaluation protocol to support this goal. The results demonstrate the challenges of developing effective enterprise agents and the importance of considering multiple factors in their evaluation.

📅 Published on Jun 22

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.23654
• PDF: https://arxiv.org/pdf/2606.23654
• Project Page: https://frontisai.github.io/EnterpriseClawBench/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#EnterpriseAgents #WorkplaceAutomation #BenchmarkingAI #ArtificialIntelligenceInBusiness #EnterpriseArtificialIntelligence

🔥 Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

💡 The paper introduces Apple-PI, a benchmark for evaluating video models based on their ability to reason about physical laws. Current video models are often evaluated only on their output, without considering whether they arrive at that output through a faithful, law-grounded reasoning process. Apple-PI addresses this limitation by explicitly anchoring video model evaluation in physical laws.

The benchmark consists of three components: Orchard, a dataset of 400 videos covering ten canonical tasks in classical mechanics, which separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization; Benchmark Protocol, a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction, which uses chain-of-frames prompting on infographic-annotated first frames to treat the generated video as the model's visible reasoning trace; and Evaluation Suite, a hybrid evaluation suite that combines machine learning-based subjective scoring with physics-law-grounded objective measures.

The authors used Apple-PI to benchmark 11 video models and found that current models remain far from reliable law-grounded world simulators, with the best video models scoring only 0.473. The stage-, pillar-, and source-resolved analyses further exposed a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap.

The paper positions Apple-PI as a diagnostic foundation for guiding future video models towards world models with law-grounded physical intelligence. The benchmark provides a comprehensive evaluation framework for video models, enabling researchers to identify areas for improvement and develop more robust and physically informed models.

📅 Published on Jul 17

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.16401
• PDF: https://arxiv.org/pdf/2607.16401
• Project Page: https://21yrm.github.io/Apple-PI-homepage

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/yaorunmao/apple-pi-leaderboard

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#VideoModelEvaluation #PhysicalIntelligence #LawGroundedReasoning #ClassicalMechanics #BenchmarkingAI