MCPcopy Create free account
hub / github.com/Roy3838/Observer

github.com/Roy3838/Observer

Chat with this repo
repository ↗ · DeepWiki ↗ · release v3.3.0 ↗ · + Follow · compare 17 versions
3,185 symbols 8,359 edges 389 files 612 documented · 19% ● updated 57d agov3.0.1 · 2026-09-21★ 1,620

Browse by type

⨍ Functions 2,590◇ Types & classes 538↳ Endpoints 57
What it actually does AI analysis from the code graph — generated when you open this
loading…
README

Observer AI

Release License Discord

Observer is an agent that deploys micro-agents that monitor your screen, camera and audio, and take action when something happens. It runs on local models, so your data stays 100% private.

Observer demo: an agent that calls you when a Steam download finishes

What you can do

When X then Y.

  • When my training crashes, call me.
  • When I'm out of the queue, SMS me.
  • When I get distracted, WhatsApp me.
  • When my Steam game downloads, Telegram me.
  • When my overnight render fails, call me.
  • When I haven't made progress, WhatsApp me.
  • When an appointment is available, send me a Whatsapp.
  • When an item is back in stock, send me a Telegram.

Get started

Web app (no install): open app.observer-ai.com and download a model to run it in your browser.

Desktop app (recommended): download the latest release for macOS, Windows or Linux. It ships with llama.cpp, so it runs any GGUF+mmproj model.

Bring your own server: in the app, set Custom Model Server URL to any OpenAI-compatible endpoint (Ollama, llama.cpp, vLLM).

Then describe what you want Observer to watch, or build an agent yourself from the Micro Agents tab.

How it works

Each micro-agent runs in a loop: Sensors → Model → Code.

You are an Observer agent. If you see the Observer logo say OBSERVER, otherwise say CONTINUE.
$SCREEN
if (response.includes("OBSERVER")) {
  sendEmail("your@email.com", response, screen); // attaches the screenshot
}

Sensors

Sensor Description
$SCREEN Screenshot for multimodal models
$CAMERA Camera frame for multimodal models
$MEMORY / $MEMORY@agent_id An agent's stored text (defaults to the current agent)
$IMEMORY / $IMEMORY@agent_id An agent's stored images
$CLIPBOARD Clipboard contents
$MICROPHONE* Microphone transcription
$SCREEN_AUDIO* Transcription of a shared tab's audio
$ALL_AUDIO* Mic and screen audio mixed (good for meetings)

* Transcribed with Whisper via transformers.js.

Tools

Memory and agents (agentId is optional and defaults to the running agent) - getMemory(agentId?), setMemory(agentId?, content), appendMemory(agentId?, content) - getImageMemory(agentId?), setImageMemory(agentId?, images), appendImageMemory(agentId?, images) - startAgent(agentId?), stopAgent(agentId?) - time(), sleep(ms) - mcp(message): wakes Observer's MCP with a message, as a new turn in the open chat

Notifications - sendDiscord(webhook, message, images?, videos?) - sendTelegram(chat_id, message, images?, videos?): get your chat_id by messaging @observer_notification_bot - sendEmail(email, message, images?, videos?): must be your signed-in email - sendPushover(user_token, message, images?, title?) - call(phone_number, message)† - sendWhatsapp(phone_number, message, images?, videos?)† - sendSms(phone_number, message, images?, videos?)†: blocked for US/Canada (A2P policy) - notify(title, options): browser notification (some browsers block these) - sound(name?, volume?): local sound, no account needed

† Needs whitelisting: SMS or call +1 (863) 208-5341, or WhatsApp +1 (555) 783-4727.

Video recording - startClip(), stopClip(), markClip(label), getVideo()

Desktop app - ask(question, title?), message(message, title?), system_notify(body, title?) - overlay(body), click('left' | 'right'), celebrate()

Mobile app (iOS 16+ and Android) - lockPhone(reason?, minutes?): blocks every other app on the phone until unlockPhone() or the timeout (default 30, max 720 minutes). Calling it again restarts the timer. - unlockPhone()

Code tab variables

Variable Description
response The model's response (null with a decision model)
decision The decision model's answer (null with text models)
agentId ID of the running agent
screen, camera Captured images
imemory The agent's current image memory
images All images sent to the model
prompt The model's prompt
microphone, screenAudio, allAudio Transcriptions from this loop
const cleaned = response.replace(/<think>[\s\S]*?<\/think>/g, '').trim();
appendMemory(`[${time()}] ${cleaned}`);
if (response.includes("word")) {
  sendTelegram("12345678", cleaned);
}

Decision models

Decision models don't write text. They answer typed questions with probabilities, so response is null and the result is in decision.

Plain-text prompt: a single yes/no (noul) question.

Is there a person in the room? $CAMERA
// { type: 'noul', noul: 0.94, start: 0.12, dropped: false }
if (decision.noul > 0.8) {
  sendTelegram("12345678", "Someone is in the room", camera);
}
  • noul: probability of "yes" (0 to 1)
  • start: average noul over the first 3 iterations
  • dropped: true once noul has stayed well below start for 3 iterations (useful for "tell me when this finishes")

JSON prompt: several questions at once, keyed by id.

{
  "state": "<image:1>",
  "questions": {
    "person": { "type": "noul", "instructions": "Is there a person?" },
    "pet": { "type": "choice", "instructions": "Which pet is visible?", "criteria": { "dog": "A dog", "cat": "A cat", "none": "No pet" } },
    "mess": { "type": "score", "instructions": "How messy is the room?", "criteria": ["Clean", "A bit messy", "Very messy"] }
  }
}

Add an image sensor ($SCREEN, $CAMERA or $IMEMORY) after the JSON. <image:N> marks where each image goes in the state; without markers, images go at the top.

// decision.person → { type: 'noul', noul }
// decision.pet    → { type: 'choice', choice, probabilities, confidence }
// decision.mess   → { type: 'score', score, legend, probabilities, confidence }
if (decision.pet.choice === "dog" && decision.pet.confidence > 0.7) {
  appendMemory(`[${time()}] Dog spotted`);
}

CLI

The desktop app bundles an observe CLI. Run observe sleep 5 and Observer calls you when the command exits. To build it from source: cd app && cargo build.

Self-hosting (deprecated)

Connect with us

Join the Discord or r/ObserverAI to share agents, ask questions and report bugs. You can also follow along on YouTube, X, TikTok and Instagram.

Contributions are welcome: fork the repo, create a feature branch and open a pull request.

Sponsors

SignPath

Free code signing on Windows provided by SignPath.io, certificate by SignPath Foundation.

License

AGPLv3. Built by Roy Medina for the Observer AI community.

Extension points exported contracts — how you extend this code

browse all types & interfaces →

Core symbols most depended-on inside this repo

browse all functions →

Shape

Function 1,915
Method 675
Interface 331
Class 190
Route 57
Enum 14
Struct 3

Languages

TypeScript70%
Rust14%
Python11%
Kotlin2%
Swift2%

Modules by API surface

app/plugins/llm_engine/src/lib.rs63 symbols
app/src/utils/localLlm/NativeLlmManager.ts59 symbols
app/desktop/src/lib.rs55 symbols
app/src/utils/settings.ts47 symbols
app/src/utils/ModelManager.ts47 symbols
api/orgs.py45 symbols
api/payments.py44 symbols
app/src/components/AvailableModels.tsx42 symbols
app/src/components/AgentCard/SensorPreviewPanel.tsx41 symbols
app/src/utils/streamManager.ts40 symbols
app/src/utils/tauriStreamCapture.ts39 symbols
api/messaging.py38 symbols

Dependencies from manifests, versioned

@auth0/auth0-react2.3.0 · 1×
@choochmeque/tauri-plugin-iap-api0.7.0 · 1×
@codemirror/lang-javascript6.2.3 · 1×
@codemirror/lang-python6.1.7 · 1×
@codemirror/view6.38.1 · 1×
@datadog/browser-rum7.0.0 · 1×
@datadog/browser-rum-react7.0.0 · 1×
@fontsource/golos-text5.3.0 · 1×
@lexical/history0.25.0 · 1×
@lexical/list0.25.0 · 1×

For agents

$ claude mcp add Observer \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

Ask about this repo answers extend the page