PyWinAssistant: An artificial assistant – MIT Licensed | Public Release: December 31, 2023 | Complies with federal coordinations AI Standards for Complex Adaptive Systems, Asilomar AI Principles and IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems.
PyWinAssistant is the first open-source Artificial Narrow Intelligence to elicit spatial reasoning and perception as a generalist agentic framework Computer-Using-Agent that fully operates graphical-user-interfaces (GUIs) for Windows 10/11 through direct OS-native semantic interaction. It functions as a Computer-Using-Agent / Large-Action-Model, forming the foundation for a pure symbolic spatial cognition framework that enables artificial operation of a computer using only natural language, without relying on computer vision, OCR, or pixel-level imaging. PyWinAssistant emulates, plans, and simulates synthetic Human-Interface-Device (HID) interactions through native Windows Accessibility APIs, eliciting human-like abstraction across geometric, hierarchical, and temporal dimensions at an Operating-System level. This OS-integrated approach simulating spatial utilization of a computer provides a future-proof, generalized, modular, and dynamic ANI orchestration framework for multi-agent-driven automation, marking an important step in symbolic reasoning towards AGI.
Key Features: * Not relying only on Imaging Pipeline: Operates exclusively through Windows UI Automation (UIA) and programmatic GUI semantics, enabling universal workflow orchestration. * Symbolic Spatial Mapping: Hierarchical element tracking via OS-native parent/child relationships and coordinate systems. * Non-Visual Perception: Real-time interface understanding through direct metadata extraction (control types, states, positions). * Visual Perception: A single screenshot can elicit comprehension and perception with attention to detail by visualizing goal intent and environment changes in a spatial space over time, can be fine-tuned to look up for visual cues, bugs, causal reasoning bugs, static, semantic grounding, errors, corruption... * Unified Automation: Automatic element detection. Combines GUI, system, and web automation under one Python API. Eliminates context-switching between tools. * AI-Powered Script Generation: Translates natural language or demonstrations into any kind of code inside any IDE or text edit areas. * Self-Healing Workflows: Auto-adjusts to UI changes (e.g., element ID shifts). Reducing maintenance overhead, making PyWinAssistant's algorithm future-proof. * AI/ML Integration: Using NLP to generate scripts (e.g., “Automate Application” → plan of test execution steps in JSON) with self-correcting selectors. * Cross-Context Automation: Seamlessly combining GUI, web, and API workflows in a Pythonic way, unifying disjointed automation methods (GUI, API, web) into a single framework. * Accessibility: Enhancing accessibility for users with different needs, enabling voice or simple text commands to control complex actions. * Generalization: Elicits spatial cognition to understand and execute a wide range of commands in a natural, intuitive manner. * Small and compact: PyWinAssistant functions as an example algorithm of a modular and generalized computer assistant framework that elicits spatial cognition.
PyWinAssistant has its own set of reasoning agents, utilizing Visualization-of-Thought (VoT) and Chain-of-Thought (CoT) to enhance generalization, dynamically simulating actions through abstract GUI semantic dimensions rather than visual processing, making it future-proof for next-generation LLM models. By visualizing interface contents to dynamically simulate and plan actions over abstract GUI semantic dimensions, concepts, and differentials, PyWinAssistant redefines computer vision automation, enabling high-efficiency visual processing at a fraction of traditional computational costs. PyWinAssistant has achieved real-time spatial perception at an Operating-System level, allowing for memorization of visual cues and tracking of on-screen changes over time.
Released before key breakthroughs in AI for Spatial Reasoning, it predates: * Microsoft’s Visualization-of-Thought research paper (April 4, 2024) * Anthropic Claude’s Computer-Use Agent (October 22, 2024) * OpenIA ChatGPT’s Operator Computer-Using Agent (CUA) (January 23, 2025)
PyWinAssistant represents a major paradigm shift in AI and automation by pioneering pure symbolic computer interaction bridging human intent with GUI automation at an OS level through these breakthroughs: * First Agent to bypass OCR/imaging for Computer-Using-Agent GUI automation. * First Framework using Windows UIA as the primary spatial perception channel. * First System demonstrating OS-native hierarchical-temporal reasoning.
Traditional Approach:
Automation tools require scripting (e.g., AutoHotkey) or API integration (e.g., Selenium).
PyWinAssistant Breakthrough:
# True generalization for natural language directly driving UI actions
assistant("Play Daft Punk on Spotify and email the lyrics to my friend")
# The agent chooses a fitting item according to the related context to comply with user intent.
Mechanism: Combines UIAutomation’s GUI control detection with LLMs to: - Parse intent ("play", "email lyrics") - Map to UI elements (Spotify play button, Outlook compose window) - Generate adaptive workflows
PyWinAssistant Innovation: Eliminates the need for: - Predefined API integrations - XPath/CSS selector knowledge - Manual error handling
Traditional Limitation:
Tools operate in app silos (e.g., Power Automate connectors).
PyWinAssistant Innovation:
# Notes:
# The full set of steps generation from the Assistant is working flawlessly, but in-step modifier and memory-content retrieval was purposely disabled and commented into the code- [def act()](https://github.com/a-real-ai/pywinassistant/blob/6aae4e514a0dc661f7ed640181663f483972bc1e/core/driver.py#L648C1-L648C8)
# to comply with federal coordinations AI Standards for Complex Adaptive Systems, Asilomar AI Principles and IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems.
# Accurately maintains context and intent across apps using UIA tree and spatial memory: (Example for further development)
assistant("Find for the best and cheapest flight to Mexico, and also look for local hotels and suggest me on new tabs the best on cultural options")
assistant("Look for various pizza coupons for anything but pineapple, fill in the details to order and show me the results")
# PyWinAssistant is highly modular (example):
def workflow():
song = assistant(goal="get the current track") # UIA
write_action(f"Review '{song}': Great bassline!", app="Notepad") # Win32
assistant(goal="Post on twitter the written text from notepad") # Web
# The previous set of actions can be also executed by simply using natural language:
assistant(f"Get the current song playing and in notepad put the title as Review song name: Great bassline, and write about why it is a great baseline, then post it on twitter", assistant_identity="You're an expert music critic")
Key Advancements: 1. Unified Control Graph: Treats all apps as nodes in a single UIA-accessible graph 2. State Transfer: Passes data between apps via clipboard/UIA properties 3. Semantic Transfer: Passes semantics of goal intent acros all steps 4. Error Recovery: Uses agentic reasoning systems to avoid failing actions
Traditional Model:
Deterministic scripts fail on UI changes.
PyWinAssistant’s Solution:
# Adaptive element discovery
def fast_action(goal):
speaker(f"Clicking onto the element without visioning context. No imaging is required.")
analyzed_ui = analyze_app(application=ai_choosen_app, additional_search_options=generated_keywords)
gen_coordinates = [{"role": "assistant",
f"content": f"You are an AI Windows Mouse Agent that can interact with the mouse. Only respond with the "
f"predicted coordinates of the mouse click position to the center of the element object "
f"\"x=, y=\" to achieve the goal."},
{"role": "system", "content": f"Goal: {single_step}\n\nContext:{original_goal}\n{analyzed_ui}"}]
coordinates = api_call(gen_coordinates, model_name="gpt-4-1106-preview", max_tokens=100, temperature=0.0)
print(f"AI decision coordinates: \'{coordinates}\'")
Revolutionary Features:
- Semantic Search by thinking: Example synonyms("download") → ["save", "export", "↓ icon"]
- Spatial Probability: Prioritizes elements by utilizing sets of self-reasoning agents for the synthetic operation of the actions
- Spatial-Prevention: Senses and prevents possible bad actions or misaligned step execution by utilizing sets of self-reasoning agents
- Self-Healing: Automatically chooses the perfect plan to execute without failing its step reasoning, by utilizing sets of self-reasoning agents
Task: Automate to save a song on spotify GUI.
Before:
Automation required:
WinWait, Spotify
ControlClick, x=152 y=311 # Fragile coordinates
Now: Only 1 natural language command.
assistant("Like this song") # Language-first
| Shift Metrics: | Traditional Tools | PyWinAssistant |
|---|---|---|
| Learning Curve | Days, even months | Minutes |
| Cross-App Workflows | Manual Integration | Automatic |
| Maintenance Overhead | High | LLM-AutoPatch |
From Scripts to Intent:
Replaces brittle click(x,y) with human-like "understand → act" cycles.
From Silos to OS as API:
Treats the entire Windows environment as a programmable interface.
From Fixed to Adaptive:
Leverages LLMs to handle UI changes (e.g., Spotify’s 2023 UI overhaul).
From Developers to Everyone:
Makes advanced automation accessible through natural language, improving the generality quality and minimizing the overall data usage of LLM and vision models.
Has built-in assistance options to improve human utilization of a computer, with a new technical approach to User Interface and User Experience assistance and testing by spatial visualization of thought,
generalizes correctly any natural language prompt, and plans to perform correct actions into the OS with security in mind.
By directly interfacing with Windows underlying UI hierarchy, it achieves real-time spatial perception at the OS level while eliminating traditional computer vision pipelines, enabling: * 100x Efficiency Gains: Native API access. * Blind Operation: Can function on headless systems, virtual machines, or minimized windows. * Precision Abstraction: Mathematical modeling of GUI relationships rather than visual pattern matching.
Image-Free by Design (Core Architecture)
While some projects require visual processing for fundamental operation, PyWinAssistant achieves complete GUI interaction capability without an imaging pipeline through:
python
# Example of an element properties via UIA - No screenshots needed
button = uia.Element.find(Name="Submit", ControlType="Button")
print(button.BoundingRectangle) # {x: 120, y: 240, width: 80, height: 30}diff
# PyWinAssistant imaging functions like Pixel level visualization can be enabled as real-time spatial perception with memorization of visual cues and tracking of on-screen changes over time.
+ Capable of planning successful sets of highly technical steps to perform operations on a computer at an OS level, with only one screenshot.
+ Pixel level visualization.
+ Visual hash matching can be enabled for dynamic elements.
- OCR fallback / object detection for non-UIA legacy apps.
# The experimental features of OCR were added but not fully developed as it was not necessary for the current implementation as the assistant currently works too well without it.
| Key Differentiation | PyWinAssistant | Traditional Automation |
|---|---|---|
| Primary Perception | UIA Metadata | Screenshots/OCR |
| Vision Dependency | Optional Add-on | Required Core |
| Headless Ready | ✅ Native | ❌ Requires virtual display |
PyWinAssistant is limited to model's intelligence and time to inference. New advancements on LLM's are required to reach for a complete Artificial General Intelligence system with Artificial Narrow Intelligences managing it. The system's autonomous task decomposition leverages native semantic differentials rather than visual changes, visual changes can be optionally activated for real-time image corruption analysis in GUI/Screen. Long-term memory and self-learning mechanisms were designed to evolve symbolic state representations, and can be
$ claude mcp add pywinassistant \
-- python -m otcore.mcp_server <graph>