🔥 MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization

💡 The paper introduces MobileForge, a system for adapting mobile graphical user interface agents to real target apps without requiring manual annotations. The problem addressed is that current mobile GUI agents require costly and time-consuming human-written tasks, demonstrations, or reward labels to adapt to new apps. Existing annotation-free GUI learning methods lack a unified approach to connect target-app exploration, curriculum mining, rollout execution, and feedback, and policy optimization often relies on isolated rollouts and coarse rewards.

MobileForge consists of two main components: MobileGym, which generates tasks and evaluates rollouts based on real mobile app interaction, and Hierarchical Feedback-Guided Policy Optimization, which uses trajectory outcomes, step-level process feedback, and corrective hints to update the policy. This approach allows for efficient adaptation of mobile GUI agents to new apps without requiring manual annotations.

The results show that MobileForge can adapt a mobile GUI agent to achieve 67.2 percent Pass@3 on AndroidWorld, which is close to the performance of a specialized model trained on closed data. Further adaptation using MobileForge reaches 77.6 percent Pass@3 on AndroidWorld and 41.0 percent success on the out-of-domain MobileWorld GUI-only split, establishing the strongest open-data mobile GUI agent in the evaluation. Overall, MobileForge provides a unified and efficient approach to adapting mobile GUI agents to new apps without requiring manual annotations, making it a significant contribution to the field.

📅 Published on Jun 18

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.19930
• PDF: https://arxiv.org/pdf/2606.19930
• Project Page: https://mobile-forge.github.io

📊 Datasets citing this paper:
• https://huggingface.co/datasets/lgy0404/mobileforge-exploration-trajectories
• https://huggingface.co/datasets/lgy0404/mobileforge-training-data
• https://huggingface.co/datasets/lgy0404/mobileforge-benchmark-results

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MobileGUIAgents #HierarchicalFeedbackGuidedPolicyOptimization #AnnotationFreeLearning #MobileGraphicalUserInterface #PolicyOptimizationForMobileApps