Awesome GUI Agent Paper List

A curated list of 559 research papers on GUI agents — models, frameworks, benchmarks, datasets, and more — spanning topics like GUI grounding, planning, memory, safety, and reinforcement learning.

🌐 Read this list on the web

The website is the recommended way to read this list:

→ osu-nlp-group.github.io/GUI-Agents-Paper-List

What it adds over the raw markdown:

• Full-text search across titles, authors, institutions, TLDRs, and keywords
• Multi-axis filtering — environment, keyword (AND/OR), author, institution, year, venue — with shareable URLs
• Per-paper detail pages with the full TLDR, all keywords, and related papers
• One-click BibTeX copy — LaTeX-paste-ready (@misc for arXiv-only, @inproceedings / @article for venue papers). Auto-generated, so please verify before citing.
• Interactive stats — quarterly publication trend by environment, top keywords / institutions / authors / venues
• Warm-paper light theme + dark theme, keyboard shortcuts (/, j/k, Esc, ?), no tracking

The structured store papers.yaml (and adjacent.yaml) is the source of truth — everything on the website and the README is generated from it.

Browse by Environment

🌐 Web (221) · 🖥️ Desktop (125) · 📱 Mobile (167) · 🖼️ General GUI (123)

Browse by Keyword

benchmark (179) · dataset (102) · framework (61) · reinforcement learning (58) · model (48)

GUI grounding (48) · safety (30) · security (23) · OSWorld (20) · WebArena (18)

long-horizon tasks (16) · training-free (15) · reward model (13) · planning (12) · world model (11)

memory (10) · GRPO (10) · prompt injection (10) · survey (9) · AndroidWorld (9)

Browse by Author

Graham Neubig (14) · Yu Su (14) · Huan Sun (14) · Mike Zheng Shou (12) · Jian Luan (12)

Zhuosheng Zhang (12) · Wei Liu (11) · Boyuan Zheng (11) · Shuyan Zhou (11) · Kevin Qinghong Lin (10)

Yuxiang Chai (10) · Han Xiao (10) · Kun Shao (10) · Jun Wang (10) · Tao Yu (10)

Yuanchun Li (9) · Caiming Xiong (9) · Qiushi Sun (9) · Zhiyong Wu (8) · Difei Gao (8)

Contributing

We welcome contributions from the community!

• Missing a paper? Open an issue with the paper title, link, and any relevant details — we'll add it.
• Want to add papers yourself? Edit papers.yaml, run bash scripts/update_repo.sh, then submit the regenerated diff. See CLAUDE.md for the YAML schema and local update workflow.
• Spotted an error? Feel free to open an issue or PR to correct any paper metadata (authors, dates, institutions, etc.).

Recent Papers (from most recent to oldest)

This README shows the 500 most recent papers. See papers.yaml for the full structured source — including BibTeX, OpenReview / publisher / homepage / code / dataset links, and the bibtex_confirmed flag. For non-canonical adjacent papers see adjacent.yaml.

• Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents

  • Seoyoung Choi, Minseok Ko, Hyunseok Lee, Kunwoong Kim, Woomin Song, Chanseok Jeon, Jinwoo Shin
  • 🏛️ Institutions: Unknown
  • 📅 Date: June 12, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [visual memory], [failure modes], [experiential memory], [grounding error]
  • 📖 TLDR: This paper studies how visual memory affects GUI agents by classifying failures into cognitive failure, visual state misunderstanding, hidden operation blindness, and grounding error. It finds that full-image memory can reduce state-level failures while worsening action-level failures such as hidden operation blindness and grounding errors.

• WebChallenger: A Reliable and Efficient Generalist Web Agent

  • Jayoo Hwang, Xiaowen Zhang, Vedant Padwal
  • 🏛️ Institutions: Independent
  • 📅 Date: June 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Web]
  • 🔑 Key: [web agent], [PageMem], [selective observation], [site memory], [compound actions], [WebChallenger]
  • 📖 TLDR: WebChallenger is a web-agent framework built around PageMem, a structured page representation that supports selective observation, persistent site memory, and compound action workflows. Without additional training, it combines a 32B LLM and 7B VLM to improve performance across WebArena, VisualWebArena, Online-Mind2Web, and WorkArena.

• Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields

  • Liya Zhu, Jingzhe Ding, Jian Zhang, Jianbo Xue, Shihao Liang, Ge Zhang, Yi Zhu, Duju Zeng, Xiang Gao, Qingshui Gu, Mailun Gao, Huimin Che, Yan Zhao, Peiheng Zhou, Haojun Wang, Chaobo Xian, Lili Le, Chi Wu, Yiwei Liu, Shengda Long, Jiale Yang, Fangzhi Xu, Sijin Wu, Haodong Duan, Chao He, Zhaojian Li, Minchao Wang, Huan Zhou, Jiani Hou, Chuqian Yu, Weiran Shi, Hongwan Gao, Jiamin Chen, Guanhong Chen, Tingqin Luo, Kaiyuan Zhang, Zhixin Yao, Qing Hua, Yuhao Jiang, Jin Chen, Pu Chen, Zhenyu Hu, Xingyu Li, Zhengxuan Jiang, Meng Cao, Tianfeng Long, Haozhe Wang, Mingzhang Wang, Yichen Zhang, Yiming Dai, Chenchen Zhang, Jiaying Wang, Xinying Liu, Xingzu Liu, Lingling Zhang, Xinjie Chen, Yujia Qin, Wangchunshu Zhou, Zhiyong Wu, Yang Liu, Jiaheng Liu, Lei Zhang, Shen Yan, Wenhao Huang, Zaiyuan Wang, Xiaolong Chang
  • 🏛️ Institutions: Unknown
  • 📅 Date: June 09, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Desktop]
  • 🔑 Key: [benchmark], [long-horizon tasks], [professional workflows], [Workflow-GYM]
  • 📖 TLDR: Workflow-GYM is a benchmark for long-horizon computer-use tasks in professional software environments. It evaluates whether agents can complete domain-specific workflows through GUIs and reports that current state-of-the-art models still struggle with end-to-end professional tasks.

• Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms

  • Jiashu Yao, Heyan Huang, Daiqing Wu, Wangke Chen, Huaxi Ai, Haoyu Wen, Zeming Liu, Yuhang Guo
  • 🏛️ Institutions: BIT, THU, Beihang
  • 📅 Date: June 03, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [benchmark], [dynamic interface], [short-video platform], [observation control], [LivingScreen]
  • 📖 TLDR: LivingScreen is a benchmark for "living-screen-native" GUI agents that must act on continuously updating interfaces such as short-video platforms, where on-screen content changes between agent actions. Evaluating frontier models, it finds none reaches human cost-accuracy performance and identifies over- and under-observation as key failure modes, motivating better observation-control capabilities.

• Demo2Tutorial: From Human Experience to Multimodal Software Tutorials

  • Zechen Bai, Zhiheng Chen, Yiqi Lin, Kevin Qinghong Lin, Difei Gao, Xiangwu Guo, Xin Wang, Mike Zheng Shou
  • 🏛️ Institutions: Unknown
  • 📅 Date: June 02, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [software tutorials], [task graphs], [planning], [Demo2Tutorial]
  • 📖 TLDR: Demo2Tutorial converts screen recordings and interaction logs into structured multimodal software tutorials with parsed actions, intents, and hierarchical task graphs. The paper evaluates tutorial generation quality and shows that the resulting representations improve downstream GUI-agent planning and generalization.

• Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models

  • Athar Parvez, Muhammad Jawad Mufti, Muqaddas Gull, Omar Hammad
  • 🏛️ Institutions: Unknown
  • 📅 Date: June 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [UI annotation], [mobile UI understanding], [workflow decomposition], [MUIAnno]
  • 📖 TLDR: This paper studies automated mobile UI annotation for UI understanding, accessibility, testing, dataset construction, and GUI agents. It decomposes annotation into context-aware stages with structured prompts and schema-constrained outputs, improving precision on expert-annotated MUIAnno mobile screens.

• STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models

  • Yuhang Han, Wenzheng Yang, Yujie Chen, Xiangqi Jin, Yaojie Zhang, Siteng Huang, Linfeng Zhang
  • 🏛️ Institutions: SJTU, HKUST (GZ), ZJU
  • 📅 Date: June 01, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [efficiency], [KV cache compression], [vision-language model], [training-free], [STaR-KV]
  • 📖 TLDR: STaR-KV is a training-free KV cache compression framework for GUI vision-language models, whose cache grows linearly with interaction steps. It combines subspace-aware spatial scoring with temporal-stability re-weighting to retain salient tokens, reducing memory while preserving accuracy across GUI agent benchmarks.

• GUI-C²: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning

  • Junlong Li, Chao Hao, Lap-Pui Chau, Yi Wang
  • 🏛️ Institutions: PolyU
  • 📅 Date: May 29, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [General GUI]
  • 🔑 Key: [GUI grounding], [reinforcement learning], [difficulty-aware data curation], [coarse-to-fine refinement], [GUI-C²]
  • 📖 TLDR: GUI-C² targets data efficiency and inference overhead in RL-based GUI grounding by combining GUI-D, a difficulty-scored data curation pipeline, with area-gated coarse-to-fine visual refinement. Its improvement-aware stage rewards and simplified decision process achieve 46.4% on ScreenSpot-Pro with a 3B model while adding minimal inference time.

• UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents

  • Yuxiang Chai, Han Xiao, Xinyu Fu, Jinpeng Chen, Rui Liu, Hongsheng Li
  • 🏛️ Institutions: Unknown
  • 📅 Date: May 28, 2026
  • 📑 Publisher: arXiv
  • 💻 Env: [Mobile]
  • 🔑 Key: [ligh