RustAutoGUI

RustAutoGUI crate, made after Al Sweigarts library PyAutoGUI for python.

RustAutoGUI allows you to control the mouse and keyboard to automate interactions with other applications. The crate works on Windows, Linux and Macos.

Main functions:

• capture screen
• find image on screen
• move mouse to pixel coordinate
• click mouse buttons
• input keyboard string
• input keyboard command
• keyboard multiple key press
• find image on screen and move mouse to it
• detect cursor position
• and more

Table of contents

• RustAutoGUI
• Table of contents
• Achievable speed
  • Why not OpenCV?
  • Segmented template matching algorithm
• Installation
  • Lite version
• Usage:
  • Import and Initialize RustAutoGui
• Finding image on screen
  • Loading images into memory
  • Loading single image into memory
  • Loading multiple images into memory
  • Custom Image Preparation \& Storage (*_custom Functions)
  • What's different?
  • Why Use Custom Thresholds?
  • Performance Tips
  • Template matching
  • Single loaded template match
  • Multiple stored templates search
  • MacOS retina display issues:
  • Segmented vs FFT matching
• General Functions
• Mouse Functions
  • Mouse Clicks
  • Mouse Scrolls
  • Mouse Movements
  • Mouse Drags
• Keyboard Functions
• Warnings Options
• OpenCL
• OpenCL Installation
  • Linux
  • Windows
  • MacOS
• Please read before using OpenCL ⚠️⚠️
• V1 vs V2 Algorithms
• Other Info
• How does crate work
• Major changes

Achievable speed

Unlike PyAutoGUI, this library does not use OpenCV for template matching. Instead, it employs a custom multithreaded implementation. Since version 2.5, OpenCL is included, with more info further down this Readme file. While OpenCV's template matching algorithm is highly optimized, in some cases Segmented algorithm using OpenCL may be even faster. RustAutoGui aims to provide faster overall performance by optimizing the entire process of screen capture, processing, and image matching. From tests so far, the performance appears to be ~5x faster than python counterpart(on Windows). The speed will also vary between operating systems, where Windows outperforms Linux for instance.

Gif presentation (intentionally captured with phone camera):

Why not OpenCV?

OpenCV requires complex dependencies and a lengthy setup process in Rust. To keep installation simple and avoid forcing users to spend hours setting up dependencies, RustAutoGui features a fully custom template matching algorithm that minimizes computations while achieving high accuracy.

Segmented template matching algorithm

RustAutoGUI crate includes another variation of template matching algorithm using Segmented Normalized Cross-Correlation. More information: https://arxiv.org/pdf/2502.01286

Installation

Either run cargo add rustautogui

or add the crate in your Cargo.toml:

rustautogui = "2.5.0"

With OpenCL support ( ⚠️ Please read info below before using):

rustautogui = { version = "2.5.0", features = ["opencl"] }

Lite Version

rustautogui = { version = "2.5.0", features = ["lite"] }

Dev Version - allows access to private mods (immediately has opencl included)

rustautogui = { version = "2.5.0", features = ["dev"] }

For Linux additionally run:

sudo apt-get update

sudo apt-get install libx11-dev libxtst-dev

For macOS: grant necessary permissions in your settings.

Lite version

Lite version provides just keyboard and mouse function. No template matching code included

Usage:

Since version 2.2.0, RustAutoGUI supports loading multiple images in memory and searching them. It also allows loading images from memory instead of only from disk.

Loading an image does certain precalculations required for the template matching process, which allows faster execution of the process itself requiring less computations.

Import and Initialize RustAutoGui

use rustautogui;

let mut rustautogui = rustautogui::RustAutoGui::new(false); // arg: debug

Finding image on screen

Loading images into memory

Loading single image into memory

---

From file, same as load_and_prepare_template which will be deprecated

rustautogui.prepare_template_from_file( // returns Result<(), String>
   "template.png", // template_path: &str path to the image file on disk
   Some((0,0,1000,1000)), // region: Option<(u32, u32, u32, u32)>  region of monitor to search (x, y, width, height)
   rustautogui::MatchMode::Segmented, // match_mode: rustautogui::MatchMode search mode (Segmented or FFT)
).unwrap();

From ImageBuffer>

rustautogui.prepare_template_from_imagebuffer( // returns Result<(), String>
   img_buffer, // image:  ImageBuffer

> -- Accepts RGB/RGBA/Luma(black and white)
   None,  // region: Option<(u32, u32, u32, u32)> searches whole screen when None
   rustautogui::MatchMode::FFT, // match_mode: rustautogui::MatchMode search mode (Segmented or FFT)
).unwrap();

From raw bytes of encoded image

rustautogui.prepare_template_from_raw_encoded( // returns Result<(), String>
   img_raw // img_raw:  &[u8] - encoded raw bytes
   None,  // region: Option<(u32, u32, u32, u32)>
   rustautogui::MatchMode::FFT, // match_mode: rustautogui::MatchMode search mode (Segmented or FFT)
).unwrap();

Loading multiple images into memory

---

Functions work the same as single image loads, with additional parameter of alias for the image.

Load from file

rustautogui.store_template_from_file( // returns Result<(), String>
   "template.png", // template_path: &str path to the image file on disk
   Some((0,0,1000,1000)), // region: Option<(u32, u32, u32, u32)>  region of monitor to search (x, y, width, height)
   rustautogui::MatchMode::Segmented, // match_mode: rustautogui::MatchMode search mode (Segmented or FFT)
   "button_image" // alias: &str. Keyword used to select which image to search for
).unwrap();

Load from Imagebuffer

rustautogui.store_template_from_imagebuffer( // returns Result<(), String>
   img_buffer, // image:  ImageBuffer

> -- Accepts RGB/RGBA/Luma(black and white)
   None,  // region: Option<(u32, u32, u32, u32)>
   rustautogui::MatchMode::Segmented, // match_mode: rustautogui::MatchMode search mode (Segmented or FFT)
   "button_image" // alias: &str. Keyword used to select which image to search for
).unwrap();

Load from encoded raw bytes

rustautogui.store_template_from_raw_encoded( // returns Result<(), String>
   img_raw // img_raw:  &[u8] encoded raw bytes
   None,  // region: Option<(u32, u32, u32, u32)>
   rustautogui::MatchMode::Segmented, // match_mode: rustautogui::MatchMode search mode (Segmented or FFT)
   "button_image" // alias: &str. Keyword used to select which image to search for
).unwrap();

Custom Image Preparation & Storage (*_custom Functions)

---

All standard image functions have corresponding custom variants, identifiable by the _custom suffix (e.g., prepare_template_from_file_custom, store__template_from_imagebuffer_custom).

What's different?

---

These _custom functions include an extra threshold parameter. While the default segmented template matching uses an automatic threshold estimation, the custom version gives you manual control over this value.

• Threshold determines how finely the image is segmented:

• Higher threshold → Finer segmentation → More detailed image

• Lower threshold → Coarser segmentation → Faster processing

Why Use Custom Thresholds?

---

The automatic thresholding works well in many cases, but:

• It can introduce a slight performance overhead.

• In some scenarios, manual tuning of the threshold can result in significantly faster matching.

• By choosing the right threshold for each image, you can maximize performance.

• threshold is only important for Segmented match modes and has no influence on FFT match mode

💡 Internally, the threshold represents the correlation between the fast-segmented image and the original template.

Performance Tips

---

• Threshold > 0.85:

• May slow down the algorithm significantly.

• Offers diminishing returns in terms of accuracy or speed.

• Threshold < 0.3:

• Often produces similar results as 0.0 in most cases.

• Can be a good baseline for experimentation.

• What do we gain with higher threshold? -> Fast template match produces less false positives, giving less work to slow template match process and increasing speed

the algorithm does two correlation checks. First with roughly segmented image, with small number of segments, then on second finer segmented image, with higher precision and more segments. Positions found by rough image, which runs very fast, are checked with finer image. Sometimes, rough image is segmented by too small factor and leads to many false positives, which slows down algorithm due to too many checks on finer image

Example:

rustautogui.store_template_from_imagebuffer_custom( // returns Result<(), String>
   img_buffer,
   None, 
   rustautogui::MatchMode::Segmented,
   "button_image",
   0.0 
).unwrap();

Template matching

Single loaded template match

---

Find image and get pixel coordinates

let found_locations: Option<Vec<(u32, u32, f64)>> = rustautogui.find_image_on_screen(0.9).unwrap(); // arg: precision
// returns pixel locations for prepared template that have correlation higher than precision, ordered from highest correlation to lowest
// Must have prepared template before

Find image, get pixel coordinates and move mouse to location

let found_locations: Option<Vec<(u32, u32, f64)>> =  rustautogui.find_image_on_screen_and_move_mouse(0.9, 1.0).unwrap();
// args: precision , moving_time
// executes find_image_on_screen() and moves mouse to the center of the highest correlation location

IMPORTANT: Difference between linux and windows/macOS when using multiple monitors. On Windows and macOS, search for template image can be done only on the main monitor. On Linux, searches can be done on all monitors if multiple are used, with (0,0) starting from the top-left monitor.

Loop search with timeout. Searches till image is found or timeout in seconds is hit.

⚠️ Timeout of 0 initiates infinite loop

rustautogui
        .loop_find_image_on_screen(0.95, 15) // args: precision, timeout
        .unwrap();

rustautogui
        .loop_find_image_on_screen_and_move_mouse(0.95, 1.0, 15) // args: precision, moving_time and timeout
        .unwrap();

Multiple stored templates search

---

Again, functions are the same, just having alias argument

rustautogui
      .find_stored_image_on_screen(0.9,  "test2") // precision, alias
      .unwrap();

With mouse movement to location

rustautogui
      .find_stored_image_on_screen_and_move_mouse(0.9, 1.0, "test2") // precision, moving_time, alias (&str)
      .unwrap();

Loop search

⚠️ Timeout of 0 initiates infinite loop

rustautogui
        .loop_find_stored_image_on_screen(0.95, 15, "stars") // precision, timeout, alias
        .unwrap();

rustautogui
        .loop_find_stored_image_on_screen_and_move_mouse(0.95, 1.0, 15, "stars") // precision, moving_time, timeout, alias
        .unwrap();

MacOS retina display issues:

---

Macos retina display functions by digitally doubling the amount of displayed pixels. The original screen size registered by OS is, for instance, 1400x800. Retina display doubles it to 2800x1600. If a user provides a screengrab, the image will be saved with doubled the amount of pixels, where it then fails to match template since screen provided by OS api is not doubled.

It can also not be known if user is providing template from a screen grab, or an image thats coming from some other source. For that reason, every template is saved in its original format, and also resized by half. The template search first searches for resized template, and if it fails then it tries with original. For that reason, users on macOS will experience slower search times than users on other operating systems.

Segmented vs FFT matching

---

This info does not include OpenCL in comparison. More info about it below.

It is hard to give a 100% correct answer when to use which algorithm. FFT algorithm is mostly consistent, with no big variances in speed. Segmented on other hand can heavily vary and speed can be up to 10x faster than FFT, but also slower by factor of up to thousands. The best would be for users to test both methods and determine when to use which method. A general advice can be: Use segmented on smaller template images and when template is less visually complex (visual complexity is randomness of pixels in an image, for instance an image that is half white vs half black vs random noise image). FFT would probably be better when comparing