github.com/awwaiid/ghostwriter @v2025.09.27-01

MAIN IDEA

An experiment for the reMarkable that watches what you write and, when prompted either with a gesture or some on-screen content, can write back to the screen. This is an exploration of various interactions through this handwriting+screen medium.

I wrote the handwritten prompt, GPT-4o drew the Chihuahua!!!

Setup/Installation

You need an OPENAI_API_KEY (or similar for other models) environment variable set. I did this by adding it to my ~/.bashrc file on the remarkable:

# In the remarkable's ~/.bashrc or before you run ghostwriter, set one or more of your keys
export OPENAI_API_KEY=your-key-here
export ANTHROPIC_API_KEY=your-key-here
export GOOGLE_API_KEY=your-key-here

Install by getting the binary to your remarkable. On your not-remarkable (ie. your laptop):

# For the reMarkable2
wget -O ghostwriter https://github.com/awwaiid/ghostwriter/releases/latest/download/ghostwriter-rm2

# For the reMarkable Paper Pro
wget -O ghostwriter https://github.com/awwaiid/ghostwriter/releases/latest/download/ghostwriter-rmpp

# Replace this ip address with your remarkable ip address
scp ghostwriter root@192.168.1.117:

Then you have to ssh over and run it. Here is how to install and run (run these on the remarkable):

# One time -- make it executable after the initial copy
chmod +x ./ghostwriter

./ghostwriter --help # Get the options and see that it runs at all

Usage

First you need to start ghostwriter on the reMarkable. SSH into your remarkable and run:

# Use the defaults, including claude-sonnet-4-0
./ghostwriter

# Use ChatGPT with the gpt-4o-mini model
./ghostwriter --model gpt-4o-mini

Draw some stuff on your screen, and then trigger the assistant by touching/tapping the upper-right corner with your finger. In the ssh session you'll see other touch-detections and there is a log of what happens while it is processing. You should see some dots drawn during processing and then a typewritten or drawn response!

CLI Options

Models & Engines: * --model MODEL - Model to use (default: claude-sonnet-4-0) * --engine ENGINE - Engine: openai, anthropic, google (auto-detected from model) * --engine-api-key KEY - API key (or use env vars) * --engine-base-url URL - Custom API base URL

Behavior: * --prompt PROMPT - Prompt file to use (default: general.json) * --trigger-corner CORNER - Touch trigger corner: UR, UL, LR, LL (default: UR)

Tools: * --no-svg - Disable SVG drawing tool * --no-keyboard - Disable text output * --thinking - Enable model thinking (Anthropic) * --web-search - Enable web search (Anthropic)

Testing/Debug/Experiments: * --log-level LEVEL - Set log level (info, debug, trace) * --no-loop - Run once and exit * --input-png FILE - Use PNG file instead of screenshot * --output-file FILE - Save output to file * --save-screenshot FILE - Save screenshot * --save-bitmap FILE - Save rendered output * --no-submit - Don't submit to model * --no-draw - Don't draw output * --no-trigger - Disable touch trigger * --apply-segmentation - Add image segmentation for spatial awareness

Run in the background

To run in the background, start it (on the remarkable) with nohup:

nohup ./ghostwriter --model gpt-4o-mini &

(TODO: figure out how to run it on boot!)

Development

I've been developing in Ubuntu, but did get it working in OSX. Generally it goes (1) install dependencies, (2) build locally but cross-compile for the reMarkable, (3) scp it over and try it out.

• Install docker for cross-compiling
• Install Rust
• You can also follow instructions for rustup
• Or if you want to be fancy, I prefer getting it from asdf
• Or maybe apt or brew will work?
• Ubuntu
• sudo apt-get install gcc-arm-linux-gnueabihf
• OSX
• brew install arm-linux-gnueabihf-binutils
• Set up cross-rs and targets
• Get it from the current git version, especially for OSX
• cargo install cross --git https://github.com/cross-rs/cross
• rustup target add armv7-unknown-linux-gnueabihf aarch64-unknown-linux-gnu
• Then to build and scp it to your remarkable
• rm2
  • cross build --release --target=armv7-unknown-linux-gnueabihf
  • scp target/armv7-unknown-linux-gnueabihf/release/ghostwriter root@remarkable:
• rmpp
  • cross build --release --target=aarch64-unknown-linux-gnu
  • scp target/aarch64-unknown-linux-gnu/release/ghostwriter root@remarkable:
• I wrapped up that last bit into build.sh
• So I do either ./build.sh to build and send to my rm2
• Or I do ./build.sh rmpp to build and send to my rmpp

Meanwhile I have another terminal where I have ssh'd to the remarkable. I ctrl-C the current running ghostwriter there, then on my host laptop I run my build script, and then back on the remarkable shell I run ghostwriter again.

When I want to do a build for others, I tag main with like v2026.09.21-01 and that kicks off a github action that creates the latest release.

Status / Journal

• 2024-10-06 - Bootstrapping
• Basic proof of concept works!!!
• Drawing back on the screen doesn't work super well; it takes the SVG output from ChatGPT and rasterizes it and then tries to draw lots of individual dots on the screen. The Remarkable flips out a bit ... and when the whole screen is a giant black square it really freaks out and doesn't complete
• Things that worked at least once:
  • Writing "Fill in the answer to this math problem... 3 + 7 ="
  • "Draw a picture of a chihuahua. Use simple line-art"
• 2024-10-07 - Loops are the stuff of souls
• I got a rudimentary gesture and status display!
• So now you can touch in the upper-right and you get an "X" drawn. Then as the input is processed you get further crosses through the X. You have to erase it yourself though :)
• 2024-10-10 - Initial virtual keyboard setup
• I've started to learn about using the Remarkable with a keyboard, something that I hadn't done before. It's surprisingly limited ... there is basicaly one large textarea for each page with some very basic formatting
• To write in that I have to make a pretend keyboard, which we can do via rM-input-devices, and I've done basic validation that it works!
• So now I want to introduce a mode where it always writes back to the text layer and recognizes that text comes from Machine and hadwriting from Human. Not sure that I'll like this mode
• 2024-10-20 - Text output and other modes
• Slowly starting to rework the code to be less scratch-work, organized a bit
• Now introduced ./ghostwriter text-assist mode, uses a virtual keyboard to respond!
• 2024-10-21 - Binary release build
• Got a github action all set to do binary builds
• 2024-10-23 - Code shuffle
• Doing a bit of refactoring, grouping utilities into separate files
• Yesterday a new Anthropic model came out (3.5-sonnet-new) which might be better at spacial awareness on the screen, so next up is to try that out in drawing-mode
• In any case, next I want to set it up with tools so that it can contextually give back an SVG or text or start to trigger external scripts, like for TODO list management
• 2024-11-02 - Tool Time
• Switch to providing some tools -- draw_text and draw_svg
• This should make it more compatible with Anthropic?
• More immediately, this means now there is the one overall assistant and it decides to draw back keyboard text or SVG drawing
• 2024-11-07 - Claude! (Anthropic)
• More shuffling to start to isolate the API
• ... and now I added Claude/Anthropic!
• It is able to use an almost identical tool-use setup, so I should be able to merge the two
• So far it seems to like drawing a bit more, but it is not great at drawing and not much better at spacial awareness
• Maybe next on the queue will be augmenting spacial awareness through some image pre-processing and result positioning. Like detect bounding boxes, segments, etc, feed that into the model, and have the model return an array of svgs and where they should be positioned. Maybe.
• 2024-11-22 - Manual Evaluations
• Starting to sketch out how an evaluation might work
• First I've added a bunch of parameters for recording input/output
• Then I use that to record a sample input and output on the device
• Then I added support to run ghostwriter on my laptop using the pre-captured input (build with ./build.sh local)
• Next I will build some tooling around iterating on examples given different prompts or pre-processing
• And then if I can get enough examples maybe I'll have to make an AI judge to scale :)
• To help with that ... on idea is to make overlay the original input with the output but make the output a different color to make it differentiable by the judge
• So far this technique is looking good for SVG output, but it'd be nice to somehow render keyboard output locally too. That is tricker since the keyboard input rendering is done by the reMarkable app
• 2024-12-02 - Initial segmenter
• With a LOT of help from claude/copilot I have added a basic image segmenting step
• This does some basic segmenting and then gives the segment coordinates to the Vision-LLM to consider
• Only hooked it up with claude for now, need to merge those two models
• ... It helps with putting X in boxes a LOT!!

* Need to get some automation around the evaluations * The segmenter has to be explicitly enabled with --apply-segmentation and it assumes that you have either --input-png or --save-screenshot because it (dumbly) re-parses the png file * OMG this is the first time that the math prompt got even close to putting the answer where I want! It has been getting it right, but usually types the 10 with the keyboard or places it somewhere wrong. This time it actually put it where it should be!

* 2024-12-15 - Engine Unification * With the usual help from claude/copilot and some tutorials I extracted out some polymorphic engine layer for OpenAI and Anthropic backends * So now you can pass in engine and model * A lot of other codebases take a model and then do a map; maybe I'll do that based on the model name or something * I also got the prompt and tool definitions externalized (into a prompts/ directory) and unified, so each engine does whatever it needs to adjust for its own API * In theory the prompts/ files are both bundled in the executable AND overridable at runtime with a local directory, but I haven't verified that much * 2024-12-18 - System Upgrade Panic * I auto-update my remarkable, usually fine * But I just got 3.16.2.3 and ... screenshots stopped working! * So I used codexctl to downgrade. It gave me a VERY scary "SystemError: Update failed!" and then the whole system locked up! * ... but a reboot fixed it and the downgrade to 3.14.1.9 worked upon reboot * So... I'm keeping an eye out for other reports of issues on the new version * Oh yes. Now you can take prompts/general.json, rename it to james.json and go in and add "Your name is James" into the prompt. Then copy that to your reMarkable * Now run ./remarkable --prompt james.json and it has a locally modified prompt!

* 2024-12-19 -- Not Quite Local * On the internet they suggested a local-network Vision-LLM mode * Ollama has that! So I tried... * But it says that llama3.2-vision doesn't have tools :( * But Groq llama-3.2 does! * ... but it is not very good at tic-tac-toe (this is the 90b). Though it is very fast!

* Oops! I forgot to turn on segmentation. Here it is with that enabled which should give a better sense of space...

* Here are 3 runs from claude in contrast

* Well. The new ENV is OPENAI_BASE_URL, so OPENAI_BASE_URL=https://api.groq.com/openai ./ghostwriter --engine openai --model llama-3.2-90b-vision-preview for example * 2024-12-22 -- Starting to Evaluate * Starting to build out the evaluation system a bit more, including a basic script to kick it all off * Right now it is a hard-wired set of parameters which basically turn on/off segmentation and use either Claude 3.5 Sonnet or ChatGPT 4o-mini * See the initial evaluation report! * I think markdown doesn't let me lay this out how I want, so will probably switch to html (maybe turn on github site hosting for it) * This is starting to get into the terratory where it can take some time and money to execute ... running this a bunch of times and I sent like $1. Not sure how long it took. but there were 48 executions in this final report * Oh -- I think it's rather important to run each set a few times assuming there is some temperature involved * To scale this even further we of course would want to bring in a JUDGE-BOT! * Then I could say things like "my new segmentation algorithm improved output quali