Awesome GUI Agent Paper List

A curated list of 590 research papers on GUI agents — models, frameworks, benchmarks, datasets, and more — spanning topics like GUI grounding, planning, memory, safety, and reinforcement learning.

🌐 Read this list on the web

The website is the recommended way to read this list:

→ osu-nlp-group.github.io/GUI-Agents-Paper-List

What it adds over the raw markdown:

• Full-text search across titles, authors, institutions, TLDRs, and keywords
• Multi-axis filtering — environment, keyword (AND/OR), author, institution, year, venue — with shareable URLs
• Per-paper detail pages with the full TLDR, all keywords, and related papers
• One-click BibTeX copy — LaTeX-paste-ready (@misc for arXiv-only, @inproceedings / @article for venue papers). Auto-generated, so please verify before citing.
• Interactive stats — quarterly publication trend by environment, top keywords / institutions / authors / venues
• Warm-paper light theme + dark theme, keyboard shortcuts (/, j/k, Esc, ?), no tracking

The structured store papers.yaml (and adjacent.yaml) is the source of truth — everything on the website and the README is generated from it.

Browse by Environment

🌐 Web (236) · 🖥️ Desktop (138) · 📱 Mobile (179) · 🖼️ General GUI (125)

Browse by Keyword

benchmark (186) · dataset (102) · framework (62) · reinforcement learning (60) · model (50)

GUI grounding (50) · safety (32) · security (24) · WebArena (20) · OSWorld (20)

long-horizon tasks (17) · reward model (15) · training-free (15) · world model (14) · planning (12)

GRPO (11) · survey (10) · memory (10) · prompt injection (10) · AndroidWorld (9)

Browse by Author

Wei Liu (14) · Graham Neubig (14) · Yu Su (14) · Huan Sun (14) · Jian Luan (13)

Mike Zheng Shou (12) · Zhuosheng Zhang (12) · Tao Yu (11) · Boyuan Zheng (11) · Shuyan Zhou (11)

Tianbao Xie (10) · Qiushi Sun (10) · Yuanchun Li (10) · Kevin Qinghong Lin (10) · Yuxiang Chai (10)

Han Xiao (10) · Kun Shao (10) · Jun Wang (10) · Zichen Ding (9) · Zhiyong Wu (9)

Contributing

We welcome contributions from the community!

• Missing a paper? Open an issue with the paper title, link, and any relevant details — we'll add it.
• Want to add papers yourself? Edit papers.yaml, run bash scripts/update_repo.sh, then submit the regenerated diff. See CLAUDE.md for the YAML schema and local update workflow.
• Spotted an error? Feel free to open an issue or PR to correct any paper metadata (authors, dates, institutions, etc.).

Recent Papers (from most recent to oldest)

This README shows the 500 most recent papers. See papers.yaml for the full structured source — including BibTeX, OpenReview / publisher / homepage / code / dataset links, and the bibtex_confirmed flag. For non-canonical adjacent papers see adjacent.yaml.

• CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

  • Linqiang Guo, Li Gu, Zihuan Jiang, Zhixiang Chi, Siobhan Reid, Ziqiang Wang, Yuanhao Yu, Wei Liu, Yang Wang, Tse-Hsun (Peter) Chen
  • 🏛️ Institutions: Concordia University, Mila, University of Toronto, McMaster University
  • 📅 Date: August 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [CoAdapt-GUI], [test-time adaptation], [workflow context], [GRPO], [AndroidWorld-Generalization]
  • 📖 TLDR: Mobile GUI agents stay brittle on applications absent from source training, so CoAdapt-GUI performs test-time adaptation that jointly updates a structured workflow context and the policy from the agent's own target-app rollouts and rewards. The workflow context retains transferable procedures, failure modes, and verification rules while excluding app-bound source details, and policy adaptation uses a task-context-matched group-relative objective over a LoRA adapter on a frozen VLM. It reaches 45.0% on AndroidWorld-Generalization against 37.5% for the reported Policy-Only TTA baseline, and lifts AndroidWorld Plus from 38.6% to 52.9%.

• Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

  • Yuke Li, Xuehan Hou
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [layout-aware matching], [coordinate hallucination]
  • 📖 TLDR: A regression-free GUI grounding pipeline lets a frozen MLLM turn an instruction into a structured visual description, then matches that description against layout-prior candidates. The paper reports gains on ScreenSpot-Pro and Mind2Web while avoiding coordinate-regression fine-tuning.

• Software Engineering for and with GUI Agent

  • Shengcheng Yu, Yuchen Ling, Junyang Xing, Quan Zhou, Chunrong Fang, Zhenyu Chen
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 10, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [survey], [software engineering], [reliability], [lifecycle]
  • 📖 TLDR: This survey reviews 336 GUI-agent papers through a software-engineering lens, covering architectures, evaluation, lifecycle concerns, and deployment gaps. It identifies recurring weaknesses in recovery, safety enforcement, auditability, testing, and maintenance.

• AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents

  • Weikai Xu, Yunren Feng, Haoxiang Lei, Kun Huang, Yuxuan Liu, Kang Zhao, Xiaolin Hu, Shuo Shang, Bo An
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [world model], [executable HTML], [mobile agents], [AppDeltaWorld]
  • 📖 TLDR: AppDeltaWorld models mobile GUI transitions as constrained executable HTML updates rather than unconstrained next-screen generation. It uses retrieved app structures and predicted next-screen content to support a mobile-agent training environment and test-time adaptation.

• Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents

  • Jiaming Wei, Zekun Wu, Adriano Koshiyama, Maria Perez-Ortiz
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [observation routing], [multimodal observation], [WebArena], [VisualWebArena]
  • 📖 TLDR: This paper measures when web agents should use text, pixels, or both. Repeated runs show that apparent oracle routing gains are heavily affected by execution noise; learned routers do not consistently beat a fixed mode, while unsolved-task routing can still reduce cost without lowering success.

• StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection

  • Zhuoxin Zhan, Akbar Rafiey, Avery Ma, Leila Pishdad, Layla El Asri
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [benchmark], [safety], [indirect prompt injection], [StepJack]
  • 📖 TLDR: StepJack studies indirect prompt injection chains in which a harmful goal is split into seemingly innocuous steps across linked pages. Its 480-example benchmark evaluates whether computer-use agents follow these multi-step attacks and releases the attack-generation pipeline.

• StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

  • Linqiang Guo, Wei Liu, Li Gu, Yang Wang, Tse-Hsun (Peter) Chen
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [reflection], [UI transitions], [structured prediction], [StepReflect]
  • 📖 TLDR: StepReflect recasts after-action reflection for mobile GUI agents as structured transition prediction over paired visual evidence. Its staged training recipe produces a local reflection model that improves several evaluated agent configurations while reducing dependence on repeated frontier-model calls.

• The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents

  • Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Wen Li
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 06, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [hindsight distillation], [mobile agents], [privileged information], [GHD]
  • 📖 TLDR: Gated Hindsight Distillation uses the next screenshot as training-only privileged information to rescore an on-policy student's actions. It distills the signal only when that view recovers a missed demonstrated action, improving mobile-agent task success over the compared GRPO baseline.

• LoginTrap: Uncovering Task-Agnostic Phishing-Style Indirect Prompt Injection Attacks against LLM-based Web Agents

  • Longtao Guo, Zelin Zhang, Kaifeng Huang, Yang Shi
  • 🏛️ Institutions: Unknown
  • 📅 Date: August 05, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [security], [indirect prompt injection], [authentication], [LoginTrap]
  • 📖 TLDR: LoginTrap is a black-box, task-agnostic attack that uses page-specific indirect instructions to make a controlled login flow appear necessary. The paper evaluates end-to-end credential-oriented attack success across web-agent configurations and defenses.

• GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

  • Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li, Yimin Deng, Jingtong Gao, Junjia Qi, Hanyu Yan, Yefeng Zheng, Xiaopeng Li, Wanyu Wang, Xian Wu, Xiangyu Zhao
  • 🏛️ Institutions: CityU, Tencent Jarvis Lab, Westlake University
  • 📅 Date: August 04, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI-Lens], [GUI grounding], [coarse-to-fine cropping], [coordinate priming], [visual verification]
  • 📖 TLDR: GUI-Lens turns G