✨OmniParser for Pure Vision Based GUI Agent
📝 Summary:
OmniParser enhances GPT-4V's ability to act as a GUI agent by improving screen parsing. It identifies interactable icons and understands element semantics using specialized models. This significantly boosts GPT-4V's performance on benchmarks like ScreenSpot, Mind2Web, and AITW.
🔹 Publication Date: Published on Aug 1, 2024
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2408.00203
• PDF: https://arxiv.org/pdf/2408.00203
• Github: https://github.com/microsoft/omniparser
🔹 Models citing this paper:
• https://huggingface.co/microsoft/OmniParser
• https://huggingface.co/microsoft/OmniParser-v2.0
• https://huggingface.co/banao-tech/OmniParser
✨ Datasets citing this paper:
• https://huggingface.co/datasets/mlfoundations/Click-100k
✨ Spaces citing this paper:
• https://huggingface.co/spaces/callmeumer/OmniParser-v2
• https://huggingface.co/spaces/nofl/OmniParser-v2
• https://huggingface.co/spaces/SheldonLe/OmniParser-v2
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#GUIagents #ComputerVision #GPT4V #AIagents #DeepLearning
📝 Summary:
OmniParser enhances GPT-4V's ability to act as a GUI agent by improving screen parsing. It identifies interactable icons and understands element semantics using specialized models. This significantly boosts GPT-4V's performance on benchmarks like ScreenSpot, Mind2Web, and AITW.
🔹 Publication Date: Published on Aug 1, 2024
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2408.00203
• PDF: https://arxiv.org/pdf/2408.00203
• Github: https://github.com/microsoft/omniparser
🔹 Models citing this paper:
• https://huggingface.co/microsoft/OmniParser
• https://huggingface.co/microsoft/OmniParser-v2.0
• https://huggingface.co/banao-tech/OmniParser
✨ Datasets citing this paper:
• https://huggingface.co/datasets/mlfoundations/Click-100k
✨ Spaces citing this paper:
• https://huggingface.co/spaces/callmeumer/OmniParser-v2
• https://huggingface.co/spaces/nofl/OmniParser-v2
• https://huggingface.co/spaces/SheldonLe/OmniParser-v2
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#GUIagents #ComputerVision #GPT4V #AIagents #DeepLearning
arXiv.org
OmniParser for Pure Vision Based GUI Agent
The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as...
✨HiconAgent: History Context-aware Policy Optimization for GUI Agents
📝 Summary:
HiconAgent introduces History Context-aware Policy Optimization HCPO for GUI agents. HCPO efficiently leverages historical context using dynamic sampling and compression, achieving better performance than larger models with reduced computational cost and significant speedups.
🔹 Publication Date: Published on Dec 1
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.01763
• PDF: https://arxiv.org/pdf/2512.01763
• Github: https://github.com/JiuTian-VL/HiconAgent
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#HiconAgent #GUIAgents #AIResearch #ReinforcementLearning #ContextAwareAI
📝 Summary:
HiconAgent introduces History Context-aware Policy Optimization HCPO for GUI agents. HCPO efficiently leverages historical context using dynamic sampling and compression, achieving better performance than larger models with reduced computational cost and significant speedups.
🔹 Publication Date: Published on Dec 1
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.01763
• PDF: https://arxiv.org/pdf/2512.01763
• Github: https://github.com/JiuTian-VL/HiconAgent
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#HiconAgent #GUIAgents #AIResearch #ReinforcementLearning #ContextAwareAI
✨GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning
📝 Summary:
GUI Exploration Lab is a simulation environment to train GUI agents for screen navigation. It finds supervised fine-tuning establishes basics, single-turn reinforcement learning improves generalization, and multi-turn RL enhances exploration for superior navigation performance.
🔹 Publication Date: Published on Dec 2
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.02423
• PDF: https://arxiv.org/pdf/2512.02423
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#ReinforcementLearning #GUIAgents #AINavigation #MachineLearning #AIResearch
📝 Summary:
GUI Exploration Lab is a simulation environment to train GUI agents for screen navigation. It finds supervised fine-tuning establishes basics, single-turn reinforcement learning improves generalization, and multi-turn RL enhances exploration for superior navigation performance.
🔹 Publication Date: Published on Dec 2
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.02423
• PDF: https://arxiv.org/pdf/2512.02423
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#ReinforcementLearning #GUIAgents #AINavigation #MachineLearning #AIResearch
✨MAI-UI Technical Report: Real-World Centric Foundation GUI Agents
📝 Summary:
MAI-UI introduces a family of foundation GUI agents tackling real-world deployment challenges. It uses a self-evolving data pipeline, device-cloud collaboration, and online RL to set new state-of-the-art in GUI grounding and mobile navigation, significantly boosting performance and privacy.
🔹 Publication Date: Published on Dec 26
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.22047
• PDF: https://arxiv.org/pdf/2512.22047
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#GUIAgents #AI #ReinforcementLearning #MobileTech #HCI
📝 Summary:
MAI-UI introduces a family of foundation GUI agents tackling real-world deployment challenges. It uses a self-evolving data pipeline, device-cloud collaboration, and online RL to set new state-of-the-art in GUI grounding and mobile navigation, significantly boosting performance and privacy.
🔹 Publication Date: Published on Dec 26
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.22047
• PDF: https://arxiv.org/pdf/2512.22047
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#GUIAgents #AI #ReinforcementLearning #MobileTech #HCI
❤2
✨PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records
📝 Summary:
PersonalAlign is a new framework for GUI agents to align with implicit user intents using hierarchical memory and long-term user records. Their HIM-Agent significantly improves both execution by 15.7% and proactive performance by 7.3%.
🔹 Publication Date: Published on Jan 14
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.09636
• PDF: https://arxiv.org/pdf/2601.09636
• Project Page: https://jiutian-vl.github.io/PersonalAlign-page/
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#PersonalAlign #GUIAgents #AI #Personalization #IntelligentAgents
📝 Summary:
PersonalAlign is a new framework for GUI agents to align with implicit user intents using hierarchical memory and long-term user records. Their HIM-Agent significantly improves both execution by 15.7% and proactive performance by 7.3%.
🔹 Publication Date: Published on Jan 14
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.09636
• PDF: https://arxiv.org/pdf/2601.09636
• Project Page: https://jiutian-vl.github.io/PersonalAlign-page/
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#PersonalAlign #GUIAgents #AI #Personalization #IntelligentAgents
❤3
✨MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments
📝 Summary:
MemGUI-Bench is a new, comprehensive benchmark designed to evaluate the memory capabilities of mobile GUI agents. It addresses current benchmarks' failure to assess memory by offering a taxonomy, 128 tasks, and an automated evaluation pipeline. Experiments with state-of-the-art agents reveal sign...
🔹 Publication Date: Published on Feb 3
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.06075
• PDF: https://arxiv.org/pdf/2602.06075
• Project Page: https://lgy0404.github.io/MemGUI-Bench/
• Github: https://github.com/lgy0404/MemGUI-Bench
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#MobileAI #GUIagents #AIBenchmarking #MemoryAI #AIResearch
📝 Summary:
MemGUI-Bench is a new, comprehensive benchmark designed to evaluate the memory capabilities of mobile GUI agents. It addresses current benchmarks' failure to assess memory by offering a taxonomy, 128 tasks, and an automated evaluation pipeline. Experiments with state-of-the-art agents reveal sign...
🔹 Publication Date: Published on Feb 3
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.06075
• PDF: https://arxiv.org/pdf/2602.06075
• Project Page: https://lgy0404.github.io/MemGUI-Bench/
• Github: https://github.com/lgy0404/MemGUI-Bench
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#MobileAI #GUIagents #AIBenchmarking #MemoryAI #AIResearch
✨SlowBA: An efficiency backdoor attack towards VLM-based GUI agents
📝 Summary:
SlowBA is a novel backdoor attack targeting the response latency of VLM-based GUI agents. It induces excessively long reasoning chains using realistic pop-up window triggers, significantly increasing response length and latency while maintaining task accuracy. This reveals a new security vulnerab...
🔹 Publication Date: Published on Mar 9
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.08316
• PDF: https://arxiv.org/pdf/2603.08316
• Github: https://github.com/tu-tuing/SlowBA
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#BackdoorAttack #AISecurity #VLM #GUIagents #Cybersecurity
📝 Summary:
SlowBA is a novel backdoor attack targeting the response latency of VLM-based GUI agents. It induces excessively long reasoning chains using realistic pop-up window triggers, significantly increasing response length and latency while maintaining task accuracy. This reveals a new security vulnerab...
🔹 Publication Date: Published on Mar 9
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.08316
• PDF: https://arxiv.org/pdf/2603.08316
• Github: https://github.com/tu-tuing/SlowBA
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#BackdoorAttack #AISecurity #VLM #GUIagents #Cybersecurity
✨AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents
📝 Summary:
The paper presents AndroTMem, a framework and benchmark diagnosing interaction memory failures in long-horizon GUI agents. It proposes Anchored State Memory ASM, which uses causally linked intermediate-state anchors to overcome this bottleneck, improving task completion rates by up to 30%.
🔹 Publication Date: Published on Mar 19
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.18429
• PDF: https://arxiv.org/pdf/2603.18429
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#GUIAgents #AIMemory #AIAgents #AIResearch #HumanComputerInteraction
📝 Summary:
The paper presents AndroTMem, a framework and benchmark diagnosing interaction memory failures in long-horizon GUI agents. It proposes Anchored State Memory ASM, which uses causally linked intermediate-state anchors to overcome this bottleneck, improving task completion rates by up to 30%.
🔹 Publication Date: Published on Mar 19
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.18429
• PDF: https://arxiv.org/pdf/2603.18429
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#GUIAgents #AIMemory #AIAgents #AIResearch #HumanComputerInteraction
✨Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
📝 Summary:
Mobile GUI agents neglect user privacy personalization, as varied execution trajectories hinder standard optimization. This paper proposes Trajectory Induced Preference Optimization TIPO to address this challenge. TIPO improves persona alignment and task executability, outperforming existing meth...
🔹 Publication Date: Published on Apr 13
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.11259
• PDF: https://arxiv.org/pdf/2604.11259
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#MobileAI #PrivacyTech #Personalization #GUIAgents #MachineLearning
📝 Summary:
Mobile GUI agents neglect user privacy personalization, as varied execution trajectories hinder standard optimization. This paper proposes Trajectory Induced Preference Optimization TIPO to address this challenge. TIPO improves persona alignment and task executability, outperforming existing meth...
🔹 Publication Date: Published on Apr 13
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.11259
• PDF: https://arxiv.org/pdf/2604.11259
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#MobileAI #PrivacyTech #Personalization #GUIAgents #MachineLearning
✨Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization
📝 Summary:
This paper introduces the Turing Test on Screen to address GUI agents detectability by digital platforms. It proposes a benchmark and methods to humanize agent behavior, balancing imitability with task performance, enabling seamless coexistence in adversarial digital environments.
🔹 Publication Date: Published on Feb 24
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.09574
• PDF: https://arxiv.org/pdf/2604.09574
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#TuringTest #GUIAgents #AIHumanization #MobileAI #AISecurity
📝 Summary:
This paper introduces the Turing Test on Screen to address GUI agents detectability by digital platforms. It proposes a benchmark and methods to humanize agent behavior, balancing imitability with task performance, enabling seamless coexistence in adversarial digital environments.
🔹 Publication Date: Published on Feb 24
🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.09574
• PDF: https://arxiv.org/pdf/2604.09574
==================================
For more data science resources:
✓ https://xn--r1a.website/DataScienceT
#TuringTest #GUIAgents #AIHumanization #MobileAI #AISecurity