Skip to content

feat(browser): drive the person's own Chrome through the Redrob extension - #257

Merged
its-janghoon merged 3 commits into
developfrom
feat/extension-browser
Nov 13, 2026
Merged

its-janghoon merged 3 commits into
developfrom
feat/extension-browser

Conversation

@its-janghoon

Copy link
Copy Markdown
Member

What's done

The browser tool now drives the person's own Chrome through the Redrob extension when that Chrome is connected to the same Redrob account. Otherwise it uses its own headless Chromium as before. The first line of every result names the browser used.

Path chosen: the existing desk relay, with no change to the extension, Redrob Desktop or the chat server. This is option 1 of the design rule:

  • The extension already joins room desk:<user id> in the desk role. It answers paired devices with the route set in its src/remote/forward.ts (POST /browser/task, GET /browser/task/:id, ...). Its AGENTS.md already names Redrob Code as a relay caller.
  • Every relay ticket is role: owner for the same room, so the engine can join as a device, the same way Redrob chat's web page reaches the extension (apps/web/lib/desk/relay-client.ts, want: 'browser').
  • The room is the account. A browser found there is signed in to the same Redrob account, so the account check comes from the relay ticket.
  • Desktop's native link (ai.redrob.desktop) was not used. Desktop exposes it only to its own webview, so using it would need a Desktop change. Registering a second native host would need an extension change and a new entry point, which the extension's AGENTS.md forbids without asking.

Code:

  • desk/extension.ts ExtensionLink: the device side of the handshake (desk.hello kind web as "Redrob Code on ", desk.pending with the six-digit code, desk.accept, sealed req/res). It runs beside the desk relay and shares its token refresh, and it is kicked on PUT /desk/token. It only listens until the first request, so Chrome gets no pairing prompt until the tool actually needs the browser. A request waits up to two minutes for Allow, once per pairing request. The device key and the paired browser are stored in the desk secret scope (browserKey, browserDesk).
  • desk/crypto.ts: Channel.device and devicePairingCode. These are the device side of the existing handshake; the desk side is unchanged.
  • POST /desk/browser on the daemon (protocol, server handler, client and SDK regenerated) sends one request of the extension's route set. It is for engines that are not the daemon (redrob run, the terminal app). All engines go through the one daemon link, so two processes never spend the same rotating refresh token.
  • GET /desk/status gains browser (off, none, found, pairing + code, ready, denied), and redrob desk status prints it.
  • browser/extension.ts: the extension takes tasks, not clicks (its AGENTS.md: nothing outside calls the layer directly). Each action becomes one short, exact task in the same conversation and tab (followUp, chatId: code-<session>, origin: code). The extension's agent replies with a reading in the tool's own shape: URL, title, text, and numbered elements [e1] button «Send». Click and type name the element from the last reading. The approval level, payments and sign-ins stay the person's step in Chrome. A question from the browser's agent ends the task and becomes the result. After an engine restart the session finds its task again through GET /browser/task?chatId=.
  • Fallback: with no daemon, relay, browser or Allow, the tool falls back to Chromium and says why. A session stays in one browser until its next open.
  • The tool is now offered where a Chromium is installed or an engine service record exists.

What's not done

  • The Windows end-to-end through the extension is not finished. On Janghoon's PC (msi-linux) the branch engine found "Chrome on Windows" (extension 0.4.15) in the room and sent the hello. The pairing request (code 286874) waited for Allow and nobody pressed it. Allow is a person's click in Chrome by design. Not merging until that run passes.
  • Screenshot in the person's Chrome: the extension's route set returns no picture. The tool says to use read there instead of showing a picture of a different browser.

What to review

  • desk/extension.ts: ready() (waiting for an announce, then the handshake, then Allow), pick() (the paired browser first), and the denied/expired handling.
  • browser/extension.ts: the task wording (READING, task()), and the fallback rules in act().
  • Whether 2 minutes is the right Allow wait inside a tool call.

Intentions

  • No new server, channel, table or entry point. The only new route is local to the daemon.
  • Desktop's native link and chat's links are untouched. Desktop's log on the PC still showed browser extension link: Chrome 0.4.15 connected, and redrob-host.exe kept running through the test.

Checks

  • New tests: test/desk/extension.test.ts (6: device/desk channel interop and the shared code; no browser; listen-only; Allow then request, then reconnect with the stored key; no Allow in time; Deny). test/browser/extension.test.ts (9: task shape, followUp and element labels, find after restart, fallback and stickiness, approval wait, question, failure, screenshot, chat id shape).
  • bun test test/browser test/desk test/tool test/server test/event-manifest.test.ts: 823 pass, 1 fail. The failure is public event manifest > contains every latest public wire type once, which fails on origin/develop too.
  • tsgo --noEmit passes in redrob, server, client, protocol and sdk/js.
  • Windows (msi-linux): redrob run with the browser tool sent the hello, waited for Allow, then fell back to the PC's Chromium with Browser: Redrob Code's own Chromium on this machine (headless). The person's Chrome is not used: waiting for Allow in Chrome on Windows (code 286874). and read "Example Domain". redrob desk status printed your Chrome (Redrob extension): waiting for Allow in Chrome on Windows (code 286874).

@its-janghoon

Copy link
Copy Markdown
Member Author

Self-review.

  • Path: the desk relay as a device, which needs no extension, Desktop or chat-server change. Every relay ticket is role: owner in desk:<user id>, and the extension answers any paired device through forward.ts. Its 403 allowlist and approval rules apply unchanged. The account check comes from the room.
  • Risk 1: the extension takes natural-language tasks, so click e5 is only as exact as its agent's reading. To narrow the gap, every action names the element line from the last reading ([e2] button «Send»), asks for exactly one action, and asks for a reply in a fixed reading shape. A failed task is an error, never a silent fallback.
  • Risk 2: the first use waits up to 2 minutes inside a tool call for Allow in Chrome. It waits only once per pairing request; later calls fall back at once and say the code.
  • Risk 3: the daemon keeps one more relay socket open while the desk is on. It listens only, sends no hello until the tool needs the browser, and reconnects with the relay's backoff.
  • Checked: refresh-token rotation. Only the daemon holds the link, and other engines call POST /desk/browser, so no second process spends the token.
  • Not merged: the Windows run through the extension needs Janghoon's Allow click in Chrome (code 286874 for this PC's engine key).

…sion

The browser tool launched its own headless Chromium, so logged-in sites and
the Redrob extension were out of reach. Now, when the person's Chrome with
the Redrob extension is connected to the same account, the tool runs there.

Path: the desk relay, with no change to the extension, Redrob Desktop or
the chat server. The extension already takes the desk role in room
desk:<user id> and answers paired devices with its forward.ts route set.
The daemon now also joins that room as a device (kind "web", "Redrob Code
on <host>"), the same handshake Redrob chat's web page uses: desk.hello,
the six-digit Allow in Chrome once, then sealed req/res frames. The room
is the account, so the browser is signed in to the same Redrob account.

- desk/extension.ts: ExtensionLink, the device side. It runs beside the
  desk relay and shares its token refresh. It only listens until the first
  request; desk.hello and the pairing notification go out when the tool
  needs the browser. The device key and the paired browser are kept in the
  desk secret scope (browserKey, browserDesk). A request waits up to two
  minutes for Allow, once per pairing request.
- crypto.ts: Channel.device and devicePairingCode, the device side of the
  existing handshake.
- POST /desk/browser on the daemon: one request of the extension's route
  set, for engines that are not the daemon (redrob run, the terminal app).
  GET /desk/status gains `browser` (off, none, found, pairing with code,
  ready, denied); `redrob desk status` prints it.
- browser/extension.ts: each action (open, read, click, type, scroll,
  wait) becomes one short task in the same conversation and tab
  (followUp, chatId code-<session>), and the extension's agent answers
  with a reading in the tool's shape (URL, title, text, numbered
  elements). Approvals, payments and sign-ins stay the person's step in
  Chrome. A question from the browser's agent ends the task and is the
  result. Screenshot is not available there (the route set has no
  picture); the tool says to use read.
- Every result starts with a line naming the browser used. Without a
  daemon, relay, browser or Allow, the tool falls back to its own
  Chromium and says why; a session stays in one browser until the next
  open.
- The tool is offered where a Chromium is installed or an engine service
  record exists.
…a token

On the Windows check the stored refresh token was stale, so the browser
link's first tickets failed and it backed off. PUT /desk/token kicked
only the desk relay. It now kicks the browser link too, and the link
keeps one ticket request in flight at a time, as the relay does.
…shake

The browser link stored its device key the first time it said hello, so
every pairing request from this engine showed the same six digits
(286874 on the Windows check) and the extension's notification kept the
same id (`pair:<code>`). An Allow given to an earlier request's prompt
could not match the request going on, and nothing in the log said what
the extension answered.

- The key is kept only once the extension sent desk.accept. Until then
  each pairing request uses its own key, kept while that request waits,
  so its code is new; an expired or denied request drops it.
- The code is known when the hello goes out, so status and the log show
  it before the extension answers.
- An extension that no longer knows an allowed key (removed in its
  Settings) gets a new request with a fresh key.
- A browser that leaves in the middle of the handshake gets the hello
  again when it is back, with the same key.
- An Allow that arrives while no tool call waits still links.
- Every step is logged with a timestamp: browser online, hello with
  name and code, waiting for Allow, allowed, denied/expired, offline.
@its-janghoon
its-janghoon force-pushed the feat/extension-browser branch from 435e279 to 41491cf Compare November 13, 2026 09:44
@its-janghoon
its-janghoon merged commit 86ae3a8 into develop Nov 13, 2026
11 checks passed
@its-janghoon
its-janghoon deleted the feat/extension-browser branch November 13, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant