✨Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following

📝 Summary:
Multi-Crit evaluates multimodal models as judges on following diverse criteria using novel metrics. Findings reveal current models struggle with consistent adherence and flexibility to pluralistic criteria. This highlights gaps in capabilities and lays a foundation for building reliable AI evalua...

🔹 Publication Date: Published on Nov 26

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.21662
• PDF: https://arxiv.org/pdf/2511.21662
• Project Page: https://multi-crit.github.io/
• Github: https://multi-crit.github.io/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MultimodalAI #AIEvaluation #BenchmarkingAI #AIJudges #MachineLearning

✨ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models

📝 Summary:
This paper introduces ProactiveBench to measure if MLLMs can proactively ask for user help on challenging tasks. It finds MLLMs generally lack this proactiveness, and conversational history can even hinder it. However, reinforcement learning shows promise for teaching models this crucial collabor...

🔹 Publication Date: Published on Mar 19

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.19466
• PDF: https://arxiv.org/pdf/2603.19466
• Project Page: https://huggingface.co/datasets/tdemin16/ProactiveBench
• Github: https://github.com/tdemin16/proactivebench

✨ Datasets citing this paper:
• https://huggingface.co/datasets/tdemin16/ProactiveBench

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MLLMs #AIProactiveness #BenchmarkingAI #ReinforcementLearning #LargeLanguageModels

✨GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

📝 Summary:
GTA-2 is a new benchmark for General Tool Agents, covering both atomic and real-world, open-ended workflows. It shows frontier models struggle significantly, especially on workflows. The study emphasizes that execution frameworks are crucial for performance, more so than just model capacity.

🔹 Publication Date: Published on Apr 17

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.15715
• PDF: https://arxiv.org/pdf/2604.15715
• Github: https://github.com/open-compass/GTA

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AIAgents #BenchmarkingAI #LLMs #AIWorkflows #AIResearch

🔥 EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

💡 The paper introduces EnterpriseClawBench, a benchmark for evaluating enterprise agents based on real-world sessions. Enterprise agents are increasingly used in workspaces to read files, invoke tools, and deliver business artifacts. However, existing evaluation metrics are limited, focusing on single performance scores. To address this, the authors created EnterpriseClawBench, which consists of 852 reproducible tasks derived from a large archive of workplace sessions. Each task is paired with relevant information such as fixtures, prompts, and semantic rubrics.

The benchmark is not publicly released due to the proprietary nature of the data, but the construction and evaluation protocol is made available. The authors used this benchmark to evaluate the performance of various agent configurations and found that the best configuration achieved a score of 0.663, indicating that there is still significant room for improvement.

The key contribution of this paper is the introduction of a comprehensive evaluation protocol that goes beyond single performance scores. The authors argue that enterprise agent evaluation should consider multiple factors, including harness-model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior. This approach provides a more nuanced understanding of an agent's capabilities and limitations, allowing for more effective evaluation and development of enterprise agents.

Overall, the paper highlights the need for more comprehensive evaluation metrics for enterprise agents and provides a benchmark and evaluation protocol to support this goal. The results demonstrate the challenges of developing effective enterprise agents and the importance of considering multiple factors in their evaluation.

📅 Published on Jun 22

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.23654
• PDF: https://arxiv.org/pdf/2606.23654
• Project Page: https://frontisai.github.io/EnterpriseClawBench/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#EnterpriseAgents #WorkplaceAutomation #BenchmarkingAI #ArtificialIntelligenceInBusiness #EnterpriseArtificialIntelligence