✨RiddleBench: A New Generative Reasoning Benchmark for LLMs

📝 Summary:
RiddleBench, a new benchmark of 1,737 puzzles, reveals fundamental weaknesses in state-of-the-art LLMs, including hallucination cascades and poor self-correction. Models achieve only about 60% accuracy, underscoring the need for more robust and reliable reasoning capabilities.

🔹 Publication Date: Published on Oct 28

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2510.24932
• PDF: https://arxiv.org/pdf/2510.24932

✨ Datasets citing this paper:
• https://huggingface.co/datasets/ai4bharat/RiddleBench

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLMs #GenerativeAI #AIResearch #Benchmarks #NLP

✨miniF2F-Lean Revisited: Reviewing Limitations and Charting a Path Forward

📝 Summary:
An analysis of miniF2F showed AI systems had 36% accuracy due to problem errors. Correcting these errors created miniF2F-v2, improving accuracy to 70%. High-quality benchmarks like miniF2F-v2 are crucial for evaluating formal reasoning progress.

🔹 Publication Date: Published on Nov 5

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.03108
• PDF: https://arxiv.org/pdf/2511.03108
• Github: https://github.com/roozbeh-yz/miniF2F_v2

✨ Datasets citing this paper:
• https://huggingface.co/datasets/roozbeh-yz/miniF2F_v2

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AI #FormalReasoning #Benchmarks #MachineLearning #Dataset

✨Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark

📝 Summary:
Current video model benchmarks miss assessing Chain-of-Frames CoF reasoning, crucial for world simulators. Gen-ViRe is a new benchmark that decomposes CoF reasoning into cognitive subtasks, offering the first quantitative assessment. It reveals poor reasoning depth despite impressive visual quali...

🔹 Publication Date: Published on Nov 17

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.13853
• PDF: https://arxiv.org/pdf/2511.13853

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AI #WorldSimulators #VisualReasoning #GenerativeAI #Benchmarks

✨Multimodal Evaluation of Russian-language Architectures

📝 Summary:
Mera Multi is the first open multimodal evaluation framework for Russian-language AI, addressing a lack of such benchmarks. It introduces 18 new instruction-based tasks across text, image, audio, and video, created with Russian cultural specificity and a leakage prevention methodology.

🔹 Publication Date: Published on Nov 19

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.15552
• PDF: https://arxiv.org/pdf/2511.15552
• Project Page: https://mera.a-ai.ru/en/multi
• Github: https://github.com/MERA-Evaluation/MERA_MULTIMODAL/tree/main

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MultimodalAI #RussianAI #AIEvaluation #Benchmarks #AIresearch

✨Reveal Hidden Pitfalls and Navigate Next Generation of Vector Similarity Search from Task-Centric Views

📝 Summary:
Iceberg is a new benchmark for vector similarity search VSS that evaluates methods from a task-centric view. It uncovers performance degradation, re-ranks VSS algorithms based on application-level metrics, and guides practitioners.

🔹 Publication Date: Published on Dec 15

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.12980
• PDF: https://arxiv.org/pdf/2512.12980

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#VectorSimilaritySearch #MachineLearning #DataScience #Benchmarks #Algorithms

✨Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

📝 Summary:
This paper introduces an automated framework for high-quality multilingual translation of benchmarks. It uses test-time compute scaling, specifically Universal Self-Improvement and T-RANK, to prevent semantic drift and context loss. This improves LLM evaluation accuracy beyond existing methods.

🔹 Publication Date: Published on Feb 25

🔹 Paper Links:
• arXiv Page: https://huggingface.co/collections/hannayukhymenko/recovered-in-translation-eacl26-mme
• PDF: https://arxiv.org/pdf/2602.22207
• Project Page: https://ritranslation.insait.ai/
• Github: https://github.com/insait-institute/ritranslation

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLMEvaluation #MachineTranslation #NLP #AIResearch #Benchmarks

✨CL4SE: A Context Learning Benchmark For Software Engineering Tasks

📝 Summary:
CL4SE presents a benchmark for evaluating context learning in software engineering tasks, defining four SE-specific context types. It demonstrates an average 24.7% performance improvement for LLMs across tasks like code generation and review, establishing a standardized evaluation framework.

🔹 Publication Date: Published on Feb 26

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.23047
• PDF: https://arxiv.org/pdf/2602.23047
• Project Page: https://huggingface.co/papers?q=project-specific%20context
• Github: https://github.com/Tomsawyerhu/CodeCL

✨ Datasets citing this paper:
• https://huggingface.co/datasets/tomhu/codecl

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ContextLearning #SoftwareEngineering #LLMs #CodeGeneration #Benchmarks

✨LMEB: Long-horizon Memory Embedding Benchmark

📝 Summary:
LMEB is a new benchmark for evaluating embedding models' long-horizon memory retrieval abilities, a gap in traditional benchmarks. It assesses complex memory types and reveals that performance in standard passage retrieval does not generalize to these challenging scenarios.

🔹 Publication Date: Published on Mar 13

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.12572
• PDF: https://arxiv.org/pdf/2603.12572

✨ Datasets citing this paper:
• https://huggingface.co/datasets/KaLM-Embedding/LMEB

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#EmbeddingModels #MemoryRetrieval #Benchmarks #MachineLearning #AIResearch

✨Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?

📝 Summary:
Agentic-MME introduces a process-verified benchmark for multimodal agentic capabilities. It evaluates tool usage and efficiency using real-world tasks and stepwise checkpoints, revealing models struggle with complex multimodal problem-solving.

🔹 Publication Date: Published on Apr 3

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.03016
• PDF: https://arxiv.org/pdf/2604.03016
• Project Page: https://agenticmme.github.io/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AgenticAI #MultimodalAI #AIEvaluation #AIResearch #Benchmarks

✨VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects

📝 Summary:
VEFX-Bench offers a large human-annotated video editing dataset and VEFX-Reward, a specialized model for quality assessment. This benchmark allows standardized comparison, showing current models struggle with instruction following and edit locality.

🔹 Publication Date: Published on Apr 17

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.16272
• PDF: https://arxiv.org/pdf/2604.16272
• Project Page: https://xiangbogaobarry.github.io/VEFX-Bench/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#VideoEditing #VFX #AI #ComputerVision #Benchmarks