Conversation
Signed-off-by: suyao <sy20010504@gmail.com>
7 tasks
Signed-off-by: suyao <sy20010504@gmail.com>
The X11 capture path treated depth 24 as a proxy for a decodable 32-bit reply, so the ARGB (depth 32) windows Firefox and other compositing clients map always failed with `X11 pixel format is unsupported`. Both depths carry one 32-bit BGRA pixel per point, which the decoder already assumes; the check now states that directly and still rejects big-endian servers, non-32-bit depths and short replies. `Apps()` silently dropped any registered application whose accessibility name or process id could not be read, which is exactly what a sandbox policy denial looks like: the target disappeared from `listApps()` with no error and no log. The design contract requires collection state to distinguish "nothing there" from "not allowed to read", and an empty `AppList` tells an agent the desktop is empty. Skips now keep the underlying D-Bus error and reach stderr, and a desktop that registers applications but yields none returns `TARGET_UNAVAILABLE`. Both were found by driving Firefox rather than the repository fixture: the fixture is a depth-24 GTK window and always has a readable name. Verified on X11 against Firefox 142, where an unconfined caller captures a 2532x1372 PNG and a confined caller reports the AppArmor denial that hides the target. The snap policy itself is out of reach here; this only makes the condition visible, and only on the SDK path — MCP/CLI still runs runtime.py. Signed-off-by: suyao <sy20010504@gmail.com>
Signed-off-by: greycheng255 <greycheng255@gmail.com>
Signed-off-by: greycheng255 <greycheng255@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
@cherrystudio/computer-useas a typed Node/Electron-main SDK over a private native protocol, with Swift on macOS and Go on Windows/Linux. Each runtime supports application discovery, structured observation, PNG capture and one semantic left element click.Protocol v2 introduces per-app control contexts and snapshot ownership. App stop bypasses the desktop queue, cancels pending work, waits for native cleanup and invalidates old identities. macOS also exposes explicit permission onboarding with the existing drag-to-add UI. Windows owns PowerShell UIA workers through Job Objects; Linux uses Go AT-SPI/X11 without Python or cgo on the SDK path.
This is a draft for cross-platform testing. Platform npm packages, remaining actions, Wayland capture, Cherry task ownership/Tray controls and software cursors remain pending. Existing CLI/MCP entry points are retained.
Validation
windows-2025) passed Go worker/descendant cleanup, 4 native protocol tests and the real v2 counter/stop fixture. The local Windows 11 ARM64 v2 rerun remains pending after Parallels execution problems.Windows testing
Use Node 24 and Go 1.23.4 or later in an interactive Windows PowerShell session. After checking out this PR, follow the complete Windows build and fixture instructions.
The fixture verifies
Count: 0 → 1 → 2, screenshot bytes, stale/foreign reference rejection, confirmed app stop and continued access through another runtime. The Windows Go tests additionally verify worker and descendant cleanup after cancellation or owner death. Only the repository's counter fixture is manipulated.For Cherry integration, build
packages/sdkanddist/native/open-computer-use.exe, then follow companion Cherry Studio PR CherryHQ/cherry-studio#20718 and its Windows setup instructions. No npm release is required.Compatibility and limitations
The SDK and helper must use matching protocol v2 artifacts; v1 helpers are rejected at initialization. No deployed SDK package is being upgraded or published by this PR. Permission state does not authorize an Agent task, and an empty Windows/Linux permission list does not establish desktop readiness. macOS desktop validation still requires system permission grants.