Use huggingface model to replace the original model

Process triplet chain and update database Args: start_page_id: Starting page ID Returns: List of processed triplets

Process all triplets in a single chain and merge descriptions Args: chain: Single triplet chain reasoning_chain: Triplet reasonin

Process reasoning for a single triplet Args: triplet: Triplet containing source page, element, target page and action information

ocr_bbox format: [{'type': 'text', 'bbox':[x,y], 'interactivity':False, 'content':str }, ...] boxes format: [{'type': 'icon', 'bbox':[x,y], '

Run tests.

single_human_explor function Function Overview: 1. Based on the passed action and related parameters (kwargs), call the screen_actio

Update a node property by node ID Args: node_id: Node ID value property_name: Name of the property to modify

Batch insert or update vectors Args: vectors: List of VectorData objects Returns: bool: Whether the operatio

Initialize vector store Args: api_key: Pinecone API key index_name: Index name dimension: Vector dimensio

Determines if two actions are considered to be the same. Two actions being "the same" is defined here as two actions that would result in a simil

Check element associations with shortcuts

Determine if task is completed Args: state: Execution state Returns: Updated execution state with task completion statu

Create initial execution state Args: device: Device ID Returns: Dictionary containing initial state

Delete vectors Args: ids: List of vector IDs to delete node_type: Node type (used as namespace) Returns:

Calculate the similarity of elements in two pages. Parameters: page1: str, image path of the first page page2: str, image pa

Execute operation

Main function to execute a task Args: state: Initial state task: User task device: Device ID neo4j_db: Neo4j

批量提取图像特征

提取单张图像的特征

Fall back to React mode

Generate execution template

Get the execution sequence of a composite action

获取可用的模型列表

Get the screen size (width and height) of a mobile device (Android emulator or real device). Parameters: - device (str): Specify the

获取当前模型的信息

Get page node by visual embedding ID Args: embedding_id: Visual embedding ID Returns: Dict[str, Any] or None

Get all elements on a page and their executable atomic actions

Get index statistics

Check if task is completed

Match current screen elements using visual embeddings

Understand the current page

Perform actions based on the current state Specifically, this node does two things: 1. Use LLM to understand the interface and generate r

Use huggingface model to replace the original model

Query similar vectors and parse JSON strings in the results

运行性能测试

Tool name: screen_action Tool function: Perform screen operations on mobile devices (Android emulator or real device), including tap

Call the page understanding tool interface, upload the screenshot file and receive the parsing result, save the labeled image and parsing content

设置要使用的模型

Evaluate whether to execute shortcut operations

Decide whether to execute shortcut

Decide whether to fall back to basic operations

Take a screenshot of the specified mobile device (Android emulator or real device) and save it to a directory organized by application. Para

When the number of execution steps exceeds three, use LLM and the current history of three screenshots to determine if the task is complete.

Update file list and return options