🔥 Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games

💡 The paper introduces a new benchmark suite called RNG-Bench to evaluate the ability of multimodal large language models to reconstruct past observations and use them for decision-making in multi-step interactions. The problem addressed is that existing benchmarks do not adequately test a model's ability to recall and act on past observations, which is a crucial skill for deploying these models in real-world applications. The RNG-Bench suite consists of two games, Matching Pairs and 3D Maze, which are designed to test a model's ability to reconstruct past observations and use them to make decisions. The games have controlled difficulty parameters, including grid size, visual pattern, and observation modality, which allow for a thorough evaluation of a model's skills. The benchmark also introduces a head-to-head duel protocol to control for instance-level variance and a Memory Gap metric to distinguish between forgetting and poor decision-making. The results show that most residual errors in the models' performance are due to forgetting earlier observations rather than suboptimal decision-making. The paper also demonstrates that fine-tuning a model on optimal-policy rollouts and filtered model demonstrations can improve its performance on RNG-Bench and transfer to existing benchmarks without degrading its general multimodal capability. Overall, the paper provides a new benchmark suite and evaluation methodology for multimodal large language models, and demonstrates the importance of testing these models' ability to recall and act on past observations.

📅 Published on Jun 17

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.19338
• PDF: https://arxiv.org/pdf/2606.19338
• Project Page: https://internlm.github.io/RNGBench/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLargeLanguageModels #ControllableNonMarkovGames #RNNGBench #MultistepInteractions #NonMarkovDecisionProcesses

🔥 Kairos: A Native World Model Stack for Physical AI

💡 The paper introduces Kairos, a native world model framework designed to support physical AI applications. The problem addressed is that current world models are limited in their ability to learn from diverse experiences, maintain persistent states over time, and deploy efficiently in real-world scenarios. To address this, Kairos pioneers a native pre-training paradigm that learns from open-world videos, human behavioral data, and robot interactions, organized into a progressive developmental pathway.

The method involves a native unified architecture equipped with hybrid linear temporal attention, which captures local dynamics, mid-range dependencies, and maintains persistent global memory. This architecture is designed to limit error accumulation and guarantee state propagation across extended horizons. Additionally, Kairos incorporates a deployment-aware system co-design to support low-latency rollout generation on various hardware platforms.

The results show that Kairos achieves top-level performance on embodied world-model, long-horizon, and action-policy benchmarks, while offering a strong efficiency-capability trade-off. The experiments demonstrate that Kairos can learn from diverse experiences, maintain persistent states, and deploy efficiently in real-world scenarios. Overall, the paper positions Kairos as a cohesive operational foundation for future self-evolving physical intelligence, providing a native world model stack that can support a wide range of physical AI applications.

📅 Published on Jun 16

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.16533
• PDF: https://arxiv.org/pdf/2606.16533

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#PhysicalAI #WorldModeling #NativePreTraining #RobotLearning #TemporalAttention

🔥 GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

💡 The paper introduces GameCraft-Bench, a benchmark for evaluating the ability of coding agents to generate playable games end-to-end in a real game engine. The problem of end-to-end game generation is challenging as it requires coding agents to create complete playable games from natural language descriptions while meeting specific evaluation criteria. The authors formalize this problem and propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging.

The framework is instantiated as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. The benchmark evaluates coding agents based on three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. The authors evaluate frontier coding agents using GameCraft-Bench and find that end-to-end game generation remains highly challenging, with the strongest agent achieving only 41.46 percent and most agents scoring below 40 percent.

Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. The paper contributes to the field of game generation by providing a comprehensive evaluation framework and a benchmark for assessing the capabilities of coding agents in generating playable games. The results highlight the difficulties of end-to-end game generation and provide insights for future research in this area. The GameCraft-Bench benchmark, code, and data are made available for further research and development.

📅 Published on Jun 16

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.17861
• PDF: https://arxiv.org/pdf/2606.17861
• Project Page: https://tongxuluo.github.io/gamecraft-bench-website/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#GameDevelopmentAI #EndToEndGameGeneration #GameEngineBenchmarking #AIForGameDesign #ProceduralContentGeneration

🔥 From RAG to Memory: Non-Parametric Continual Learning for Large Language Models

💡 The paper discusses the challenges of continual learning in large language models and how current methods such as retrieval-augmented generation have limitations in mimicking human long-term memory. The authors propose a new framework called HippoRAG 2 which builds upon previous work and enhances it with deeper passage integration and more effective online use of a large language model. This approach improves performance across factual, sense-making, and associative memory tasks, addressing the deterioration in performance seen in previous methods that tried to augment vector embeddings with structures like knowledge graphs. The results show that HippoRAG 2 outperforms standard retrieval-augmented generation comprehensively, achieving a 7 percent improvement in associative memory tasks over the state-of-the-art embedding model, while also exhibiting superior factual knowledge and sense-making memory capabilities. The work contributes to non-parametric continual learning for large language models, paving the way for more effective and human-like memory capabilities in artificial intelligence systems.

📅 Published on Feb 20, 2025

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2502.14802
• PDF: https://arxiv.org/pdf/2502.14802

🤖 Models citing this paper:
• https://huggingface.co/muthuk1/graphrag-inference-hackathon

📊 Datasets citing this paper:
• https://huggingface.co/datasets/osunlp/HippoRAG_2
• https://huggingface.co/datasets/g7haha/HippoRAG_2

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ContinualLearning #LargeLanguageModels #NonParametricLearning #RetrievalAugmentedGeneration #LongTermMemory

🔥 ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?

💡 The paper proposes ImageWAM, a new framework for world action models that replaces video generation with pretrained image editing models for robot control. Traditional world action models rely on video generation, which has three major limitations: high computational costs due to dense multi-frame future tokens, wasted capacity on action-irrelevant details, and potential errors in long-horizon future imagination. The authors question the need for video generation in world action models and propose using image editing instead. ImageWAM uses pretrained image editing models to predict robot actions by focusing on action-relevant visual differences and localized visual changes. The model does not decode the target frame at inference time, but rather uses the output of the image editing model as a compact world-action context. The results show that ImageWAM outperforms standard baselines and competitive world action models without requiring additional policy pretraining, and it reduces computational costs to one sixth and latency to one quarter of video-based models. The authors also provide attention analysis that supports the effectiveness of image editing as an alternative to video-based world-action modeling. Overall, the paper demonstrates that image editing can be a more efficient and effective approach to world action modeling, achieving better performance with reduced computational costs.

📅 Published on Jun 17

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.19531
• PDF: https://arxiv.org/pdf/2606.19531
• Project Page: https://zhangwenyao1.github.io/ImageWAM/

🤖 Models citing this paper:
• https://huggingface.co/yuyangalin/ImageWAM-FLUX.2-4B-RoboTwin
• https://huggingface.co/yuyangalin/ImageWAM-FLUX.2-4B-LIBERO
• https://huggingface.co/yuyangalin/ImageWAM-FLUX.2-9B-LIBERO

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ImageEditingForRobotControl #WorldActionModels #VideoGenerationAlternatives #PretrainedImageModels #RobotControlWithImageEditing

🔥 S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

💡 The paper introduces S-Agent, a spatial reasoning framework that enhances visual language models to enable continuous 3D world understanding from multi-view imagery. The problem addressed is that existing visual language models and tool-augmented agents are limited to static and stateless inference from isolated visual observations, which is insufficient for real-world spatial intelligence.

The S-Agent method involves formulating spatial reasoning as spatio-temporal evidence accumulation, rather than isolated frame-level prediction. This is achieved by casting the visual language model as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge. The framework also includes a temporal memory mechanism, comprising scene memory and agent memory, which enables evidence integration across frames and reasoning steps.

The results show that S-Agent consistently improves both open-source and closed-source visual language models in a training-free manner. Additionally, supervised fine-tuning on S-Agent-generated spatial trajectories yields S-Agent-8B, a compact spatial agent that significantly surpasses similar-scale baselines and performs comparably to advanced closed-source models. The comprehensive experiments on multi-view and video spatial reasoning benchmarks demonstrate the effectiveness of the S-Agent framework in enhancing spatial intelligence. Overall, the paper contributes a novel spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos, which has the potential to improve real-world spatial intelligence applications.

📅 Published on Jun 18

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.20515
• PDF: https://arxiv.org/pdf/2606.20515
• Project Page: https://ropedia.github.io/S-Agent

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#SpatialReasoning #VisualLanguageModels #3DWorldUnderstanding #SpatioTemporalEvidence #ToolUseInAI

🔥 IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System

💡 The paper introduces IndexTTS, an enhanced text-to-speech system that combines the XTTS and Tortoise models to achieve improved naturalness, voice cloning, and controllable usage. The system addresses the limitations of existing text-to-speech systems, particularly in Chinese scenarios where polyphonic characters and long-tail characters can be challenging to pronounce. To overcome this, the authors propose a hybrid character-pinyin modeling approach that allows for more controllable pronunciations.

The authors also compare Vector Quantization with Finite-Scalar Quantization for codebook utilization of acoustic speech tokens, and introduce a conformer-based speech conditional encoder and BigVGAN2 to enhance voice cloning. The results show that IndexTTS achieves significant improvements in naturalness, content consistency, and zero-shot voice cloning compared to the XTTS model.

In comparison to other popular open-source text-to-speech systems, IndexTTS has a relatively simple training process, more controllable usage, and faster inference speed, while also surpassing their performance. The system is designed to be efficient and controllable, making it suitable for industrial-level applications. The authors provide demos of the system, which are available for evaluation. Overall, the paper presents a novel approach to text-to-speech synthesis that achieves state-of-the-art results and has the potential for practical applications.

📅 Published on Feb 8, 2025

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2502.05512
• PDF: https://arxiv.org/pdf/2502.05512
• Project Page: https://index-tts.github.io

🤖 Models citing this paper:
• https://huggingface.co/IndexTeam/IndexTTS-2
• https://huggingface.co/IndexTeam/Index-TTS
• https://huggingface.co/taraskurtizan/IndexTTS-2

📊 Datasets citing this paper:
• https://huggingface.co/datasets/echodict/index-tts

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/IndexTeam/IndexTTS
• https://huggingface.co/spaces/Pendrokar/TTS-Spaces-Arena
• https://huggingface.co/spaces/alexnasa/OutofLipSync

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#TextToSpeechSystems #ZeroShotLearning #VoiceCloningTechnology #ControllableSpeechSynthesis #SpeechRecognitionModels

🔥 QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks

💡 The paper introduces QUEST, a family of open deep research agents that can perform well across diverse long horizon search tasks. The problem addressed is that existing open agents often generalize poorly across different task types, while frontier systems remain proprietary. To solve this, the authors propose a training recipe that combines mid training, supervised fine tuning, and reinforcement learning. A key component of this recipe is a curated data synthesis pipeline that applies to different task types and enables synthesizing training data with verifiable rewards without human annotation. The pipeline uses unified rubric trees to generate tasks. The authors also incorporate a built in context management mechanism that enables effective long horizon reasoning and knowledge synthesis. The results show that QUEST approaches or surpasses frontier closed source agents across eight deep research benchmarks using only 8K synthesized tasks. The models, data, and training scripts are released, making it possible for others to use and build upon the work. The contributions of the paper are the proposed training recipe, the data synthesis pipeline, and the release of the QUEST models, which provide a general purpose deep research agent that can handle a wide range of tasks, including fact seeking, citation grounding, and report synthesis.

📅 Published on May 22

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.24218
• PDF: https://arxiv.org/pdf/2605.24218
• Project Page: https://osu-nlp-group.github.io/QUEST/

🤖 Models citing this paper:
• https://huggingface.co/osunlp/QUEST-35B-RL
• https://huggingface.co/osunlp/QUEST-35B-MT-Plus-SFT
• https://huggingface.co/osunlp/QUEST-9B

📊 Datasets citing this paper:
• https://huggingface.co/datasets/osunlp/QUEST-RL-Data
• https://huggingface.co/datasets/osunlp/QUEST-SFT-Data-Objective
• https://huggingface.co/datasets/osunlp/QUEST-SFT-Data-Open-ended

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/osunlp/QUEST

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#DeepResearchAgents #LongHorizonSearchTasks #SyntheticTaskGeneration #ReinforcementLearningMethods #OpenAgentArchitectures

🔥 PaperFlow: Profiling, Recommending, and Adapting Across Daily Paper Streams

💡 The paper introduces PaperFlow, a framework for scientific paper recommendation that addresses the dynamic nature of scientific reading, where user interests shift and feedback accumulates over time. The traditional approach to scientific paper recommendation evaluates static ranking over a fixed candidate set, which does not reflect real-world reading behavior. PaperFlow organizes the recommendation process into three stages: Profiling, Recommending, and Adapting.

In the Profiling stage, a structured and inspectable scholarly profile is constructed and maintained from heterogeneous evidence, even in cases of cold start where limited information is available. The Recommending stage ranks each daily paper stream through multi-signal aggregation, considering a fixed display budget. The Adapting stage updates the user state from semantically distinct feedback signals and models interest drift across days.

To evaluate PaperFlow, the authors define a longitudinal user-day benchmark that consists of 24 simulated research users, 50 daily paper streams, and 1200 user-day episodes. This benchmark also includes 20727 unique papers and 497448 episode-paper records. The authors also specify a blind human-evaluation protocol to validate the alignment between automatic metrics and expert judgments.

The results show that PaperFlow outperforms five scientific recommendation baselines in terms of oracle-based ranking, behavioral alignment with simulated reading selections, and blind human-evaluation score. This indicates that PaperFlow is effective in capturing user interests and adapting to changes in their reading behavior over time. Overall, the paper contributes to the development of a more dynamic and personalized scientific paper recommendation system that reflects real-world reading behavior.

📅 Published on Jun 5

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.07454
• PDF: https://arxiv.org/pdf/2606.07454
• Project Page: https://openraiser.github.io/PaperFlow

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ScientificPaperRecommendation #DynamicUserModeling #PersonalizedRecommendationSystems #ScholarlyProfileConstruction #AdaptiveInformationRetrieval

🔥 Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models

💡 The paper addresses the challenge of financial sentiment analysis, which is crucial for investment decision-making. Traditional natural language processing models are limited by their size and training data, resulting in poor generalization and effectiveness. Large Language Models, despite their superior performance in various NLP tasks, also face challenges in financial sentiment analysis due to the discrepancy between their pre-training objective and the task of predicting sentiment labels. Additionally, the concise nature of financial news often lacks sufficient context, which can compromise the reliability of Large Language Models' sentiment analysis.

To overcome these challenges, the authors propose a retrieval-augmented Large Language Model framework. This framework consists of two modules: an instruction-tuned Large Language Model module that ensures the model behaves as a predictor of sentiment labels, and a retrieval-augmentation module that retrieves additional context from reliable external sources. This approach enables the model to leverage external context to improve its sentiment analysis capabilities.

The authors evaluate their framework against traditional models and other Large Language Models, such as ChatGPT and LLaMA. The results show that their approach achieves a significant performance gain, with improvements in accuracy and F1 score ranging from 15% to 48%. This demonstrates the effectiveness of the proposed retrieval-augmented Large Language Model framework in enhancing financial sentiment analysis. Overall, the paper contributes to the development of more accurate and reliable financial sentiment analysis models, which can inform better investment decisions.

📅 Published on Oct 6, 2023

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2310.04027
• PDF: https://arxiv.org/pdf/2310.04027

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#FinancialSentimentAnalysis #RetrievalAugmentedModels #LargeLanguageModels #NaturalLanguageProcessing #FinancialTextAnalysis