HKUDS/CatchMe

<p align="right"> <a href="assets/readme/READMEzh.md">中文</a> · <a href="assets/readme/READMEja.md">日本語</a> · <a href="assets/readme/README_es.md">Español</a> · <b>English</b> </p>

<p align="center"> <img src="assets/catchme-logo.png" width="360" alt="CatchMe Logo"/> </p>

<h1 align="center">CatchMe: Make Your AI Agents Truly Personal</h1>

<p align="center"> <b>Capture Your Entire Digital Footprint: Lightweight & Vectorless & Powerful.</b> </p>

<p align="center"> <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue?style=flat" alt="License"></a> <img src="https://img.shields.io/badge/Python-%E2%89%A53.11-3776AB?style=flat&logo=python&logoColor=white" alt="Python"> <img src="https://img.shields.io/badge/Platform-macOS%20%7C%20Windows%20%7C%20Linux-lightgrey?style=flat" alt="Platform"> <a href="https://hkuds.github.io/CatchMe"><img src="https://img.shields.io/badge/Blog-online-orange?style=flat" alt="Blog"></a> <img src="https://img.shields.io/badge/Report-coming%20soon-lightgrey?style=flat" alt="Report"> <br> <a href="./COMMUNICATION.md"><img src="https://img.shields.io/badge/Feishu-Group-E9DBFC?style=flat-square&logo=feishu&logoColor=white" alt="Feishu"></a> <a href="./COMMUNICATION.md"><img src="https://img.shields.io/badge/WeChat-Group-C5EAB4?style=flat-square&logo=wechat&logoColor=white" alt="WeChat"></a> <a href="https://discord.gg/2vDYc2w5"><img src="https://img.shields.io/badge/Discord-Join-7289DA?style=flat-square&logo=discord&logoColor=white" alt="Discord"></a> </p>

<p align="center"> <a href="#-key-features">Features</a> &nbsp;·&nbsp; <a href="#%EF%B8%8F-how-it-works">How It Works</a> &nbsp;·&nbsp; <a href="#-llm-configuration">LLM Config</a> &nbsp;·&nbsp; <a href="#-get-started">Get Started</a> &nbsp;·&nbsp; <a href="#-cost--efficiency">Cost</a> &nbsp;·&nbsp; <a href="#-community">Community</a> </p>

<p align="center"><i>「 <b>Just do your thing. CatchMe captures everything else — stored locally to ensure privacy and security. </b> 」</i></p>

<p align="center"> <img src="assets/terminal_demo.svg" alt="CatchMe Terminal Demo"/> </p>

🦞 Makes Your Agents Truly Personal. CatchMe는 CLI 에이전트(OpenClaw, NanoBot, Claude, Cursor 등)를 위한 에이전트 호환 스킬로 제공됩니다. CatchMe를 독립적으로 실행하세요. 에이전트는 CLI 명령어를 통해서만 메모리를 조회합니다. ##

🎯 Enrich Your Personal Digital Context

<table width="100%"> <tr> <td align="center" width="25%" valign="top"> <img src="assets/usecasecoding.png" height="150" alt="Coding"/><br> <h3>💻 Personal Coding Assistant</h3> <b><i>"What was I coding in Claude Code today?"</i></b><br><br> <div align="left"> • Code session replay<br> • Recall your edited files<br> • Trace what you typed </div> </td> <td align="center" width="25%" valign="top"> <img src="assets/usecaseresearch.png" height="150" alt="Research"/><br> <h3>🔍 Personal Deep Research</h3> <b><i>"What was I reading about AI yesterday?"</i></b><br><br> <div align="left"> • Web/PDF viewed<br> • Search queries typed<br> • Reading info tracked </div> </td> <td align="center" width="25%" valign="top"> <img src="assets/usecasefiles.png" height="150" alt="Files"/><br> <h3>📁 Personal Files Manager</h3> <b><i>"Which files did I change today?"</i></b><br><br> <div align="left"> • File changes tracked<br> • Docs accessed<br> • Edits reviewed </div> </td> <td align="center" width="25%" valign="top"> <img src="assets/usecasedigital_life.png" height="150" alt="Digital Life"/><br> <h3>🧩 Digital Life Overview</h3> <b><i>"How did I spend my afternoon?"</i></b><br><br> <div align="left"> • App usage tracked<br> • Workflows replayed<br> • Activities recalled </div> </td> </tr> </table>

✨ Key Features

📹 Always-On Event Capture

• Event-Driven Recording: No timer or delays - catch mouse actions with crosshair annotation instantly.
• Comprehensive Context: Five recorders track windows, keyboard, clipboard, notifications, and files around mouse actions.

🌲 Intelligent Memory Hierarchy

• Auto-Organization: Raw streams structure into five tiers: Day → Session → App → Location → Action.
• Smart Summaries: LLM summaries at each level, transforming logs into searchable knowledge trees.

🔍 Tree-Based Retrieval

• No Vector Complexity: Skip embeddings and VDBs — our system uses tree-based reasoning for navigation.
• Top-Down Search: LLM reads summaries, selects relevant branches, and drills down to evidence.

🤖 Zero-Config Agent Integration

• One-File Setup: Drop a single skill file into any AI agent for instant integration.
• Immediate Access: CLI-based screen history queries with zero configuration required.

🪶 Ultralight & Privacy-First

• Minimal Footprint: ~0.2GB runtime RAM with efficient SQLite + FTS5 storage.
• Local & Offline: All data stays on your machine with full offline mode via Ollama/vLLM/LM Studio.

🖥️ Rich Web Interface

• Visual Exploration: Interactive timelines, memory tree navigation, and real-time system monitoring.
• Natural Conversation: Chat with your complete digital footprint using natural language.

<p align="center"> <img src="assets/web.png" width="100%" alt="CatchMe Web Dashboard"/> </p>

💡 CatchMe Architecture

CatchMe transforms raw digital activity into structured, searchable memory through three concurrent stages:

🔄 Record → Organize → Reason: Turn digital chaos into queryable memory

Capture. Six background recorders silently track your activity. They monitor window focus, keystrokes, mouse movement, screenshots, clipboard, and notifications.

Index. Raw events auto-organize into a Hierarchical Activity Tree: Day → Session → App → Location → Action. Each node gets LLM-generated summaries. Fast, meaningful recall without vector embeddings.

Retrieve. You ask a question. The LLM traverses your memory tree top-down. It selects relevant nodes and inspects raw data like screenshots or keystrokes. Then synthesizes a precise answer.

<p align="center"> <img src="assets/catchme-pipe.png" width="680" alt="CatchMe Pipeline: Capturing → Indexing → Retrieving"/> </p>

🌲 Hierarchical Activity Tree

The Activity Tree is CatchMe's memory core. It provides structured, multi-level views of your digital life. Browse high-level summaries or dive into granular details.

<p align="center"> <img src="assets/fig1activitytree.png" width="800" alt="Hierarchical Activity Tree Structure"/> </p>

🔍 Intelligent Tree Retrieval

CatchMe skips traditional vector search. Instead, the LLM directly navigates your Activity Tree. This enables complex, cross-day reasoning. Precise evidence gathering from raw activity history.

<p align="center"> <img src="assets/fig2_retrieval.png" width="800" alt="Tree-based Retrieval Process"/> </p>

📖 Learn More: Detailed design insights and technical deep-dive available in our blog.

🧠 LLM Configuration

❗️ Data Privacy Notice

• 100% Local Storage: All raw data (screenshots, keystrokes, activity trees) stays in ~/data/ and never leaves your machine.

• Offline-First Options: Local LLMs (Ollama, vLLM, LM Studio) enable fully offline operation without any cloud dependency.

• ⚠️Cloud Provider Caution: If used, cloud APIs will be used to summarize your daily activities. Untrusted endpoints may expose private data — review data policies of your provider carefully.

📋 Requirements

• Multimodal support: Your model should be able to handle text + images.

• Context window: Make sure the context window of your model exceed max_tokens limits in config.json.

• Cost control: For forced cost control, set limits via llm.maxcalls or increase filter.mousecluster_gap to reduce summarization frequency.

CatchMe requires an LLM for background summarization and intelligent retrieval. Use catchme init (in <a href="#-get-started">Get Started</a>)for guided setup or follow the manual configuration steps below.

For cloud API services:

{
    "llm": {
        "provider": "openrouter",
        "api_key": "sk-or-...",
        "api_url": null,
        "model": "google/gemini-3-flash-preview"
    }
}

For local/offline operation:

{
    "llm": {
        "provider": "ollama",
        "api_key": null,
        "api_url": null,
        "model": "gemma3:4b"
    }
}

<details> <summary><b>Supported LLM Providers</b></summary>