Record desktop demonstrations for computer-use agents.
Experimental · macOS · source-only release · MIT
AgentTrail is a local macOS app that records keyboard and pointer inputs, samples application context, and turns a demonstration into a searchable timeline. Export the raw events and grouped actions for dataset preparation, agent evaluation, or a conversation with an AI assistant.
Press Start, work in your apps, and finish the session. Inspect a shortcut or drag, follow it back to its source events, and keep bookmarks for moments worth reviewing.
The native app displaying a synthetic demonstration—not a real recording. This example uses a spreadsheet; input recording works across apps.
Privacy warning: this is an input recorder. Raw key codes can reveal typed content even with text capture disabled. Record your own authorized demonstrations using disposable data; never use it for covert monitoring. The repository contains source, synthetic test generators, and reviewed synthetic-only UI screenshots, not anyone's recordings. See Privacy and responsible use before capturing or sharing data.
Requires macOS 14 or later, Xcode 15+ or Apple Command Line Tools with the macOS 14+ SDK, Swift 5.9+, and Git. There are no downloaded package dependencies. This initial publication has no prebuilt/notarized installer; build locally:
git clone https://github.com/mctatge/AgentTrail.git
cd AgentTrail
bash scripts/build-app.sh
open dist/AgentTrail.appIf developer tools are missing, run xcode-select --install and complete Apple's installer first. Installation and troubleshooting covers permissions, updates, and removal.
After the first build, double-click Open AgentTrail.command or the app in dist/.
- Open Capture settings. Enable Input Monitoring for keys/pointer and Accessibility for context in macOS System Settings. If AgentTrail is absent, use the + button, press Command–Shift–G, navigate to this checkout's
dist/folder, and choose AgentTrail.app. Quit and relaunch after permission changes. Screen Recording and Excel Automation are optional. - Give the demonstration a name and press Start recording.
- Work in a native app or a browser. Control–Option–Command–P pauses/resumes; Control–Option–Command–M adds a bookmark. The shortcuts are listen-only and may also reach the foreground app.
- Press Finish session. Search the timeline, select an action to inspect its raw evidence, or Export a dataset.
Explore an example opens a clearly labeled synthetic spreadsheet session without recording computer input.
Start with a short disposable test: type a phrase in another app, drag, pause, resume, and finish. Confirm that the timeline contains actual key and pointer events, not only context observations. The latest capture-thread fix has automated regression coverage, but its physical-input/window-resize acceptance test is still pending. See validation status; do not assume lossless capture or production readiness.
Open a session and click Cursor trail, or select a movement/drag/click action and choose View cursor trail in its inspector. The coordinate map draws the recorded path, distinguishes movement from dragging, and marks mouse-button presses. Replay it at ½×, 1×, 2×, or 4×, scrub through time, or step between individual samples to inspect x/y coordinates, timestamps, and raw event IDs.
Synthetic cursor samples: movement in gray, dragging in red, and circles for button presses. The viewer links each sample to its coordinates, timestamp, and raw event ID.
The map preserves aspect ratio and negative coordinates. It fits the captured motion rather than placing the trail over an unregistered screenshot. Gaps, pause/resume boundaries, app switches, and idle intervals over two seconds break the path. Playback only visualizes recorded data; it never moves your actual pointer or operates an app. Longer recordings are split into navigable parts of up to 20,000 relevant records, without deleting or modifying raw events. Refresh loads newly committed samples during a recording.
| Input or observation | Included |
|---|---|
| Keyboard | Key down/up, physical key code, modifier flags, repeat state; ANSI display labels |
| Pointer | Coordinates, movement, button down/up, drag samples, button number, click count |
| Scroll | Horizontal/vertical point deltas and continuous-scroll flag |
| Time | UTC receipt timestamps, OS input timestamps, ordered database event IDs |
| Context | App and bundle ID, focus changes, sampled window and accessible element metadata |
| Clipboard | Change observations and type names; text is optional |
| Bookmarks | Timestamped notes for intent, unusual behavior, or labels |
| Coverage | Pause/resume, protected-input intervals, event-tap failures, interrupted sessions |
Optional settings enable Unicode text and accessibility values, clipboard text, window screenshots, and Excel workbook/sheet/selection sampling. Capture options are stored with each session. Input collection begins only after an explicit Start; there is no launch-at-login service.
An Excel session can contain the observed ⌘D input, a sampled B2:B10 selection, and a possible fill-down interpretation. The interpretation is labeled as an inference: observing a shortcut does not verify that the workbook changed.
The optional Excel adapter reads selection using AppleScript and asks for macOS Automation permission on first use. It does not write cells, intercept shortcuts, or replace Excel commands. The generic accessibility adapter also attempts to read Excel's name box. Cell-level accessibility varies by version and language.
OS inputs work across applications. Element labels depend on what the app exposes through macOS Accessibility. Browser canvas content and some spreadsheet grids can provide little context. This release does not include a DOM extension, browser network recorder, or universal cell-change detector. Optional screenshots provide additional visual evidence.
Each export is a new folder containing:
session.json Capture options, status, environment, and counts
events.jsonl Every committed raw input and context observation
actions.jsonl Grouped actions with source event ranges
training.jsonl Demonstration steps with explicitly unverified outcomes
timeline.md A readable timeline for review or AI attachment
sessions/... Optional screenshot attachments
The action builder groups mouse movements, drag gestures, scroll bursts, and typing. Raw data remains available. There is no automatic deletion or retention limit in this release; manage the local library and exported copies yourself.
The training format is an intermediate dataset, not a model-specific fine-tuning format or a replay program. Synthetic examples are labeled as synthetic. The recorder cannot reliably distinguish physical human inputs from generated OS inputs.
Use Use with AI in the app to copy an MCP configuration with the correct executable path, or attach an exported timeline.md to your conversation.
Example request:
Find every fill-down shortcut and drag in this session. Show the observed selection, supporting event IDs, and anything the log cannot verify.
The embedded MCP server uses stdio and exposes only list_sessions, search_actions, and get_events. It opens SQLite read-only and has no capture, modification, shell, or arbitrary SQL tool. See AI integration for configuration and paging.
AgentTrail makes no network requests. If you connect an AI client, that client's handling of returned data determines whether recording contents leave your Mac.
Connecting MCP gives that client read access to every session in the selected local library, not just the one selected in the app. Session IDs are query filters, not authorization boundaries. For a narrower scope, record into a separate --root library. Review data before allowing a client to retrieve it.
The library defaults to ~/Library/Application Support/AgentTrail/. Directories are created owner-only and the database and exported raw records use owner-only file permissions. SQLite uses WAL transactions. These controls are not encryption.
The complete committed raw log lives in the events table of library.sqlite. Use Open recording library in the sidebar to find it. While the app is open, SQLite's neighboring -wal and -shm files may contain live database state; use Export or the CLI rather than copying just the database file.
For terminal use, AgentTrail raw SESSION_ID streams every raw event as JSONL; add --follow to keep streaming newly committed events until the session finishes (Ctrl-C stops the reader). It has no 500-record cap. Redirect stdout to save a plain-text log. The GUI, CLI, and MCP server all read the same library.
Keyboard codes can reconstruct typed content even with literal text disabled. Common password-manager bundle IDs are excluded by default; the app suppresses capture during macOS secure input and recognized accessible password fields. Detection depends on the application. Pause before entering credentials or leaving a demonstration. Screenshots can contain unrelated visible information inside the captured app window.
An allowlist can restrict recording to specific app bundle IDs. Recorder controls are excluded. Pausing continues to listen for the resume shortcut but does not persist ordinary inputs. Closing the workspace leaves the visible menu-bar recorder running; Quit finishes and saves the session.
“Not recorded · AgentTrail controls (intentional)” is normal when using or resizing AgentTrail itself. It is different from “Capture gap”, which reports actual listener failures or dropped events. Since 0.2.1, input capture runs on its own thread rather than sharing the UI thread; ordinary window resizing should not disable the listener. Earlier missing inputs cannot be reconstructed from context observations.
- “Raw” means events delivered by macOS, not every physical device report. macOS can coalesce pointer motion, withhold secure inputs, or disable a stalled event tap. Trackpad magnify/rotate/swipe, pressure, touch, and IME composition lifecycle events are not implemented.
- Context and screenshots are asynchronous observations with their own timestamps. They are not guaranteed pre-action states or proof of an outcome. Foreground windows are sampled; canvas cells and custom widgets may be opaque.
- The app queues inputs and commits batches roughly every 100 ms under normal load. A crash can lose queued/uncommitted inputs. Committed raw events survive; the next launch rebuilds interrupted timelines and marks them interrupted.
- The capture-to-UI buffer and pending/queued writer stages each have an 8,000-input budget; lifecycle markers can exceed that budget. Overflow is recorded as a capture gap. Disk failures stop capture. This is an initial release, not a lossless hardware acquisition system.
- Typed/accessibility text is capped at 4,096 characters per context value, clipboard text at 16,384 characters per sample, typing summaries at 512 characters, screenshots at one request per second. Raw keystroke records retain per-event codes.
- Screenshot requests capture an available on-screen window owned by the foreground app. Apps with multiple windows can yield a different window than intended. Screenshot timing is reported as a request-to-completion interval.
- The ad-hoc signature used by default is suitable for local builds. Rebuilding or moving the app can invalidate macOS permissions. For distribution, use a stable signing identity and a separate notarization/release process.
AGENTTRAIL_SIGN_IDENTITYselects a signing identity for the build script.
swift test
bash scripts/build-app.sh
dist/AgentTrail.app/Contents/MacOS/AgentTrail --helpThe project contains a native SwiftUI/AppKit app and a Foundation/SQLite core. The architecture, event schema, and validation guide explain the boundaries. CI runs tests and packages the app on macOS.
Before proposing changes, read CONTRIBUTING.md. Before pushing, stage the intended source files and run python3 scripts/check-publication.py; it inspects Git's index for recordings, private paths, credentials, and unexpected file types. The check is also in CI and is not a substitute for reviewing the diff. Report vulnerabilities privately using SECURITY.md.
AgentTrail uses Apple's public APIs and the system SQLite library. The implementation is independent; related projects worth exploring include OpenAdapt Capture, Screenpipe, and ActivityWatch.
MIT. See LICENSE and third-party notices. This license covers the software, not rights to documents, messages, screenshots, identities, or other material a user records. No legal-compliance, anonymity, or data-rights certification is provided.