Fall back to React mode when template execution fails Args: state: Execution state Returns: Updated execution state

Capture current screen and parse elements, update state Args: state: Deployment state Returns: Updated deployment state

Fallback to semantic matching when visual matching fails

Extract features from images, supporting single or batch image processing. Input can be file paths or file streams. Parameters: imag

Format file information as option text

Update node properties in database Args: node_id: Node ID property_name: Property name property_value: Property value

Create and initialize DeploymentState object Args: task: User input task description device: Device ID max_retries:

Generic node creation function

Get all chain starting nodes (Page nodes with no incoming edges)

ocr_bbox: list of xyxy format bbox

Match current screen elements with elements in high-level action nodes using visual embedding comparison Args: state: Deployment sta

Crop elements from a page and extract features

Check if the (y,x) point is contained in each bounding box. Args: yx: The (y, x) coordinate in pixels of the point. bounding_boxes: A 2D in

This function annotates an image with bounding boxes and labels. Parameters: image_source (np.ndarray): The source image to be annot

capture_and_parse_page Function Overview: 1. Take a screenshot of the current device interface based on the device information in St

Check if high-level actions are associated with shortcuts Args: state: Execution state high_level_actions: List of high-leve

Create a Page node Args: properties: Dictionary containing page_id and other attributes

Evaluate if shortcuts meet execution conditions Args: state: Execution state shortcuts: List of shortcuts Returns:

Process the chain evolution into a high-level action node. Args: start_page_id: Starting page ID Returns: Created high-level

Execute high-level operations Args: state: Execution state shortcuts: List of shortcuts meeting execution conditions

Extract task description from the chain. Args: chain: Triplet chain Returns: Extracted task description

Format chain operations as text description. Args: chain: Triplet chain Returns: Formatted operation description text

Generate execution template based on shortcuts Args: state: Execution state shortcuts: List of shortcuts that meet execution

Get Action node by ID Args: action_id: ID of the Action node Returns: Dict[str, Any] or None: Action node in

Get all Action nodes from the database Returns: List[Dict[str, Any]]: List containing all Action node information

Get all high-level action nodes from the database Returns: List[Dict[str, Any]]: List containing all high-level action node infor

Get complete operation chain from starting node, returning triplet chain structure Args: start_page_id: ID of the starting page

Get Element node by ID Args: element_id: ID of the Element node Returns: Dict[str, Any] or None: Element nod

Convert image to base64 encoding Args: image_input: Image path string, PIL Image object, or BytesIO object Returns: str

Match user task with high-level action nodes Args: state: Execution state task: User input task description Returns:

When file is selected, update file details

Prioritize shortcuts based on page flow Args: state: Execution state shortcuts: List of shortcuts Returns: Prio

处理单个图像文件

Execute a single task Args: task: User task description device: Device ID Returns: Execution result

Export the running process to JSON

Build a comprehensive distance matrix

Calculate the normalized position distance between two bounding boxes

Determines if two drag actions are the same.

Determines if two tap actions are the same.

Ensure the index exists, create if not

Resize the bounding boxes by the given fractions. Args: annotation_positions: Array of shape (N, 4), where each row represents the (y, x,

Create Element-LEADS_TO->Page relationship

Create Action-COMPOSED_OF->Element relationship

Create Page-HAS_ELEMENT->Element relationship

Draws bounding boxes on the frame using the detections provided. Args: scene (np.ndarray): The image on which the boundi

Build workflow state graph

Clear all databases (Neo4j and Pinecone) Args: neo4j_uri: Neo4j database connection URI neo4j_auth: Neo4j authentication inf

Clear all nodes and relationships in the Neo4j graph database Args: uri: Neo4j database connection URI auth: Authentication

Clear vectors in the Pinecone vector database Args: api_key: Pinecone API key index_name: Index name namespace: Opti

Create an Action node (high-level/composite action) Args: properties: Dictionary containing action_id, description, element_seque

Create relations between high-level action and elements. Args: action_data: High-level action node data dictionary Returns:

Create an LCEL chain for generating high-level action node content.

Create a high-level action node in the database. Args: action_data: High-level action node data dictionary Returns: Created

Create an LCEL chain for evaluating whether the chain can be templated.

Create an Element node Args: properties: Dictionary containing element_id, element_type and other attributes Visual f

Create LCEL chain for merging page descriptions Returns: Description merging chain

Create LCEL chain for triplet reasoning

Process the visual features of the element and store them in the vector database Parameters: ID: str, the ID of the element in the J

Crop elements from a page and return the image byte stream. Parameters: page_path: str, image path of the page elements_json

Given the element_id (ID of the element), find the corresponding element's 'bbox' in the JSON file pointed to by state['current_page_json'] (

Batch crop multiple elements from a page and return a list of image byte streams. Parameters: page_path: str, image path of the page

Evaluate whether the chain can be templated into a high-level action. Args: chain: Triplet chain Returns: (Whether it can be

Execute screen element action Args: state: Execution state element_match: Element matching result Returns: Whet

Extract detailed information of all elements in the chain. Args: chain: Triplet chain Returns: Element detail text

Extract reasoning results of all triplets in the chain. Args: chain: Triplet chain Returns: Reasoning result text

Format file details

Generate high-level action node content. Args: chain: Triplet chain Returns: Generated high-level action node content (dicti

Get high-level action nodes related to a specific task Args: task: User task description Returns: List[Dict[

Get all JSON files and their details in the specified directory

check overlap of text and background detection box, and get_optimal_label_pos, pos: str, position of the text, must be one of 'top left', 't

Get shortcuts associated with a specific high-level action Args: action_id: High-level action ID Returns: Li

Chain stored logs into the database :param json_path: :return:

List all devices currently connected via ADB (Android Debug Bridge). Returns: - Success: Returns a list of device IDs, each represen

Merge description information of overlapping nodes in the chain Args: chain: Triplet chain merge_chain: Description merging chain

Find matching element information from the JSON file based on coordinates. If no direct match is found, return the nearest element. Args