Desktop control for agents on Hyprland. Inspect a window, target an accessibility element or a screenshot point, and click, scroll or type while you keep working.
A local demo app, operated by Deskpilot with its real cursor and click feedback. Static image | 60 fps video | Recording and source
| What you need | What deskpilot gives you |
|---|---|
| Find a control | Live accessibility trees with element IDs, or a window screenshot with recorded coordinates |
| Send input | Click, scroll, hover, drag, type, key chords and private paste |
| Work out of sight | Launch your own native Wayland apps on an agent's hidden workspace |
| Reduce round trips | CLI and MCP actions, plus a sequence of known steps in one batch |
| Record a demo | A saved recipe of steps replayed as an MP4, with reading beats between them |
uv tool install deskpilotThe first input action builds and loads the plugin for your running Hyprland. Run deskpilot plugin load to
prepare it ahead of time. The build needs the tools listed under Requirements.
For Claude Code, install the agent skill:
/plugin install deskpilot --marketplace cjber/deskpilot
For MCP clients, run deskpilot mcp over stdio. In Claude Code:
claude mcp add deskpilot -- deskpilot mcpList your windows, then replace WINDOW below with one address from the output:
deskpilot windows
deskpilot elements WINDOWIn an app with an accessibility tree, choose a control from the output and replace ELEMENT with its ID:
deskpilot click WINDOW --element ELEMENTDeskpilot asks the app for that element's current position before clicking. Prefer the full SNAPSHOT:e27 form
from the reported snapshot and element ID, so a newer tree cannot reuse an old ID accidentally.
For an app without a tree, capture its window:
deskpilot shot WINDOWRead the image at the printed path, then click a point in that image with
deskpilot click WINDOW X Y --shot SHOT_ID, replacing X, Y and SHOT_ID with the point and printed shot ID.
A resized window or a point outside the recorded window is refused. Take a fresh shot to verify the result;
use wait-for WINDOW --changed SHOT_ID when the app needs time to draw it.
More actions and targeting rules | Accessibility and popup handling
Send a JSON array through a file or stdin to run up to 32 steps in one CLI process or MCP request:
deskpilot batch WINDOW --final-shot <<'JSON'
[
{"action": "key", "combo": "ctrl+a"},
{"action": "type", "text": "Hello from deskpilot"}
]
JSONThis example replaces text in the target app's current field. Run it only after inspecting that field.
To film a window with the agent cursor (brackets closing in on a landing, the marker squeezing on a press), run
deskpilot record WINDOW -o clip.mp4 --seconds 10, or -o - for PPM frames at a constant rate on stdout. It works for
hidden windows too (--hidden), and the cursor shows in the clip alone. Screenshots from shot never include it.
Supported steps are click, move, scroll, drag, key, type and paste. Element IDs and shot IDs work
on the corresponding pointer steps. --final-shot returns a capture after all steps succeed.
The entire request is validated before input starts. Each step rechecks the pinned window and uses the same refusals as a single action. A failure stops the batch and reports the completed steps and failing index; earlier actions stay done. Other agents and the user can act between steps.
A recipe is a saved sequence of steps for one window, with the beats a viewer needs, recorded in one run:
deskpilot recipe check recipes/export-an-asset.json
deskpilot recipe run recipes/export-an-asset.json --record demo.mp4The steps are the batch actions plus pause, wait and shot; a recipe may also start its own app and name the
MP4 it records. The recording is deskpilot record of the window, so the video holds the agent cursor deskpilot
draws. deskpilot recipe schema prints the format as JSON Schema. Nothing is launched or clicked until the whole
file validates, and a failed step stops the run, keeps the recording and exits 2.
Work on your own hidden window
Set an agent name and launch a native Wayland app of your own:
export DESKPILOT_AGENT=demo
deskpilot launch APPReplace APP with an installed app and use the window it reports. Add --hidden to actions on those windows,
including batch; hidden keys and text also need --window. deskpilot stop closes your agent's apps.
deskpilot show intentionally brings the agent workspace into view for you to watch or step in.
For an existing unfocused native window, deskpilot adopt ADDRESS explicitly hands only that exact window to
the agent. deskpilot release ADDRESS returns it without closing its shared application.
Hidden actions preserve your view, pointer and focus and only reach windows that agent launched or adopted. XWayland targets are refused hidden. For browser tasks, use fastbrowse.
For a private desktop with separate profiles, clipboard and X server, put --session before every command,
starting with DESKPILOT_AGENT=NAME deskpilot --session launch APP. End it with deskpilot --session stop.
Hidden-window behaviour, adoption and private desktops
eval "$(deskpilot --session env)" starts that desktop if needed and exports what reaches it (WAYLAND_DISPLAY,
HYPRLAND_INSTANCE_SIGNATURE, XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS and AT_SPI_BUS_ADDRESS), so other tools
such as grim or wf-recorder can use it; --json prints it as JSON. Your shell keeps its own profile, and stop
leaves what you started this way running.
container/Dockerfile builds an image whose --session desktop needs no GPU and no compositor around it: with
DESKPILOT_STANDALONE=1 and DESKPILOT_AGENT=agent (set in the image) Hyprland runs on its own DRM backend on the
kernel's virtual display, vkms, rendered in software by Mesa. On the host, load vkms and pass its display node in:
sudo modprobe vkms
docker build -f container/Dockerfile -t deskpilot .
docker run --rm --device /dev/dri deskpilot bash -c 'eval "$(deskpilot --session env)"; deskpilot --session launch APP'The entrypoint gives the image's agent user the display node's group and runs the command as that user, since
Hyprland refuses to run as root. The image's plugin is prebuilt against its pinned Hyprland. container/smoke.sh is
the check CI runs in it: it launches a GTK app, clicks its entry, types and reads the text back, recording a clip.
GitHub's ubuntu-24.04 runners can load vkms; a runner without the module cannot run it.
The plugin sends input to the target client directly where supported. Some clients, including XWayland, require a seat-based route that briefly holds input, restores your pointer and focus, and replays intervening user input. A private paste preserves your clipboard; apps that cannot receive one are refused.
--take-pointer, --take-focus and --show explicitly take your pointer, focus or view.
Input routes and their limits explains the client-specific behaviour.
- Hyprland, with
hyprctlonPATH. Version 0.57's Lua dispatchers and the older string ones both work. - The plugin build tools: a C++26 compiler,
make, Hyprland's headers (pkg-config hyprland, shipped with Arch'shyprlandpackage),libxcb,libxkbcommonandcairo. grimfor screenshots.wl-clipboardforclipboard.- For
--take-pointer: write access to/dev/uinput, usually through theinputgroup or a udev ACL. Check withtest -w /dev/uinput. - Python 3.12 or later and
uv.
- Command guide: actions, screenshots, accessibility, MCP and error codes.
- Recipes: a demo as data, replayed in order and recorded to an MP4.
- Input routing: dedicated seats, keymaps, agent cursors and hidden windows.
- Live calibration: isolated input checks and cleanup verification.
- Contributing: setup, checks and bug reports.
- Security: report a vulnerability privately.
- Changelog: changes by release.
