Skip to content

control-plane, frontend: mirror exploring routing suggestions to Arkiv (Build on Arkiv track) - #73

Merged
OfficialAbhinavSingh merged 7 commits into
mainfrom
feat/arkiv-routing-mirror
Nov 6, 2026
Merged

OfficialAbhinavSingh merged 7 commits into
mainfrom
feat/arkiv-routing-mirror

Conversation

@OfficialAbhinavSingh

@OfficialAbhinavSingh OfficialAbhinavSingh commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Part of #72 — not closing it yet: the issue's own "Done when" needs a real write plus a working
panel on mergit.site, neither of which has happened (no funded wallet yet).

Summary

First version of this PR used the Python arkiv-sdk and claimed it worked end to end. It didn't —
a self-review checked it live against Rabdi instead of trusting the SDK's docs, and found the
Python SDK targets a different, incompatible network entirely (wrong write address, rejected query
options). arkiv/friction.md has the exact RPC errors and how to reproduce them.

Current state, rebuilt on what's actually confirmed live against Rabdi:

  • Reads: plain httpx against arkiv_query (no SDK), filtered by schema tag, plan id, and
    $owner — the first version was missing the owner filter, which left it open to anyone writing a
    spoofed entity with Mergit's own schema tag. Ordered by key (ORDER BY $key ASC).
  • Writes: a small Node sidecar (chain/arkiv/), using the official JS SDK (confirmed live to
    target Rabdi's real write address), holding the service wallet's private key — the control
    plane's own process/env never hold it. One HTTP call batches every task's decision for a plan
    into one on-chain transaction (executeBatch), serialized through an in-process queue so two
    concurrent explore calls can't race on the wallet's nonce.
  • Writes run as a BackgroundTasks entry, not awaited inline — blocking the response on the write
    (the SDK call waits for a transaction receipt) was the first version's bug; fixed.
  • GET /v0/plans/{uuid}/arkiv-suggestions reads the mirror back, owner-only, degrading to
    {enabled: true, decisions: [], error: "..."} instead of a 500 if Arkiv is unreachable.
  • Frontend: "View on Arkiv" panel in SuggestAgents.tsx, shown only after an exploring suggestion,
    reading live from the endpoint above, each entity linking straight to its own Arkiv explorer page
    (rabdi.explorer.arkiv.network/entity/<key>, confirmed real against a live entity), with a
    "matches ✓" check against what Postgres answered with for the same task.
  • The mirror is actually replayable now, not just a record of the outcome: each entity carries
    decisionId, seed and budgetMicros (via INSERT ... RETURNING on the Postgres row, not
    recomputed), matching migration 0016's own reason routing_decisions exists — "any draw can be
    replayed from its seed and the posteriors it saw."
  • No requester attribute on the entity: it was never queried, and Arkiv entities are append-only
    (only expiry removes them), so attaching a wallet address to data that outlives a requester's own
    "discard this draft" was wrong.
  • docker-compose: arkiv-sidecar service, gated by the arkiv profile (same opt-in pattern as
    keyrelay); validated with docker compose config, and the image was actually built and run
    twice (docker build + docker run, hit its real /healthz and /internal/mirror, including
    after the schema change below).
  • arkiv/{schema,friction,reach,why-arkiv,how-you-use-arkiv}.md + both READMEs, rewritten to state
    only what was actually checked live.

Review round 2 (Yash)

Caught one real, confirmed bug: server.js mapped createdEntities.map(e => e.key), but the SDK's
own type (ExecuteBatchReturnType.createdEntities: Hex[]) says those are already plain strings —
entity_keys always came back [null, ...]. Fixed. Also acted on the privacy point above
(requester dropped) and the replayability gap (decisionId/seed/budgetMicros added) — both
real, both fixed. Deferred: an arkiv_mirrors table + outbox-style retry loop for lost writes (real
engineering, not required by the track's rules, ~1 day left); deduping entities across repeated
explore clicks (not a bug — each explore call is a genuinely new Thompson draw, same as Postgres's
own append-only table); the posteriors-snapshot-instead-of-routing_decisions redesign (legitimate
question, too large a change to make now without re-verifying everything live again).

What's verified, concretely

An unfunded test wallet, run against the real Rabdi RPC from the sidecar process, the built Docker
image, and again after the schema/payload changes above (now a 2-entity batch), reaches real
on-chain gas estimation and fails there every time ("Execution error without revert data", wallet
confirmed via eth_getBalance to hold 0 tGLM) — not at encoding, not at signing, not at the RPC
connection. That's as far as this can be verified without a funded wallet. The read-side query
syntax, including ORDER BY $key ASC, was run live against the real RPC and returns correctly
(empty, since nothing matches yet).

The Arkiv panel itself was exercised in a real browser against mocked network responses (not a real
backend) by temporarily swapping the app's render root for a direct mount — reverted before
committing, never shipped; confirmed via screenshot and DOM/accessibility-tree inspection that the
real shipped component renders both the picks list and the "View on Arkiv" decisions panel
correctly.

Still needed before this is live end to end

A funded service wallet (hub.arkiv.network/faucet, wallet-connect gated — not a plain API, so
this can't be done from here) and an access key (hub.arkiv.network/access-keys), set as
ARKIV_PRIVATE_KEY/ARKIV_SIDECAR_TOKEN on chain/arkiv/'s own deploy and
ARKIV_RPC_URL/ARKIV_OWNER_ADDRESS/ARKIV_ACCESS_KEY/ARKIV_SIDECAR_URL/ARKIV_SIDECAR_TOKEN
on the control plane's. Once set, the wallet address and one live query need to be filled into
README.md and arkiv/schema.md (currently placeholders, clearly marked).

Test plan

  • uv run pytest -q — 546 passed, 1 skipped
  • uv run pytest -m postgres -q — 450 passed
  • uv run ruff check . — clean
  • uv run pyright — 0 errors (strict, mergit_cp/mergit_worker)
  • Frontend: tsc -b, eslint ., vitest run (292 passed), vite build — all clean
  • docker build chain/arkiv + docker run, hit /healthz and /internal/mirror against the
    real Rabdi RPC with an unfunded wallet (fails at gas estimation, as expected), re-checked
    after the schema fix with a 2-entity batch
  • docker compose config with the arkiv profile active
  • Real-browser render of the frontend panel (mocked network, see above)
  • A real write/read against Arkiv's live Rabdi testnet with a funded wallet

…stnet

ETHGlobal Mumbai "Build on Arkiv (existing projects)" track, swapped in for EF
(no code had been built for that pick). An exploring suggestions call already
records its draw in Postgres (routing_decisions, pre-existing); this adds a
best-effort public mirror of the same decision on Arkiv, plus a new endpoint
and frontend panel that read it back, so the write side is provably read
somewhere a user sees rather than written and ignored.

- control-plane/mergit_cp/arkiv_mirror.py: thin wrapper over the official
  arkiv-sdk (Python, pinned ==1.0.0b2 — no stable release yet), write/read,
  defensive decoding since Arkiv is a public permissionless database
- routes_routing.py: mirror write after the Postgres transaction commits
  (never fails the request), new GET /v0/plans/{uuid}/arkiv-suggestions
- frontend: "View on Arkiv" panel in SuggestAgents.tsx, reads live from the
  new endpoint, shown only after an exploring suggestion
- arkiv/{schema,friction,reach,why-arkiv,how-you-use-arkiv}.md + README
  sections for the track's qualification requirements

Needs a funded service wallet + access key (ARKIV_RPC_URL/ARKIV_PRIVATE_KEY/
ARKIV_ACCESS_KEY) on the hosted deploy before the wallet address and one live
query can be filled into README.md/arkiv/schema.md; everything else is
tested end to end against a fake Arkiv client (541 fast + 449 postgres tests
green, ruff clean, pyright strict clean on mergit_cp).
Self-review found the read side wasn't defensive like the write side:
constructing the real SDK client runs a balance check against the RPC
(arkiv-sdk's Arkiv.__init__), so a bad key or a down Rabdi testnet raised
inside GET /v0/plans/{uuid}/arkiv-suggestions uncaught, a 500 for an
unrelated-to-core-product feature. Now caught and reported as
{enabled: true, decisions: [], error: "could not reach Arkiv"}; the
frontend shows that distinctly from "nothing mirrored yet". Also hardened
_decode against a payload that's valid JSON but not an object (a bare
array), which would otherwise crash .get() the same way.

Full suite still green: 541 fast + 450 postgres, ruff/pyright clean,
292 frontend tests, tsc/eslint/build clean.
…n what is

A self-review (prompted by a second pass before the PR was declared done)
checked this track live against Rabdi for the first time instead of trusting
the SDK's docs, and found the whole write path, and the read path's query
shape, didn't work on this network:

- arkiv-sdk (Python)'s query options (`includeData`) are rejected by Rabdi
  ("unknown field includeData"), and its write target (processor address) has
  no code deployed on Rabdi (`eth_getCode` returns "0x"). It targets a
  different, incompatible Arkiv network. Confirmed live, not from reading
  source: see arkiv/friction.md for the exact RPC errors.
- Rabdi has no payload/content-type field at the protocol level at all —
  checked live on real entities. Everything is a named attribute.
- The query literal syntax (`str('...')`, unquoted `addr(0x...)`) differs
  from what the JS quickstart's prose implies.

Rebuilt on what's actually confirmed to work live:

- Reads: plain httpx against `arkiv_query`, no SDK, now also filtered by
  `$owner` (a spoofing gap the first version had — anyone can write an
  entity with our exact schema tag).
- Writes: a new small Node sidecar (chain/arkiv/), using the official JS SDK
  (confirmed, live, to target Rabdi's real write address), holding the
  service wallet's private key — the control plane's own process and env
  never hold it now, an improvement over the first version. One HTTP call
  carries every task's decision for one plan, so the sidecar sends one
  batched transaction (executeBatch), not one per task, serialized through
  an in-process queue to avoid a nonce race between concurrent explore calls.
- Writes now run as a FastAPI BackgroundTask, not awaited inline: the first
  version blocked the suggestions response on the write (the SDK's call
  waits for a transaction receipt), which this fixes.
- An unfunded test wallet, run against the real Rabdi RPC from both the bare
  sidecar and the built Docker image, reaches real gas estimation and fails
  there ("Execution error without revert data", on a wallet confirmed 0
  balance) — as far as this can be verified without a funded wallet.
- docker-compose: arkiv-sidecar service, gated by the `arkiv` profile (same
  opt-in pattern as keyrelay), config validated with `docker compose config`.
- Docs (arkiv/*.md, both READMEs) rewritten to only state what was actually
  checked live, including the wrong turns, rather than what the SDK's docs
  imply.

Verified: full control-plane suite (545 fast + 450 postgres), ruff, pyright
strict clean, frontend (292 vitest, tsc, eslint, vite build) all still green.
The Arkiv panel itself was exercised in a real browser (mocked network
responses, not a real backend) by temporarily swapping the app's render
root — reverted before this commit, never shipped.

Still blocked on a funded service wallet (hub.arkiv.network/faucet is
wallet-connect gated, not a plain API, so this can't be done from here) for
an actual successful write and a real entity key.
… page

Confirmed live: https://rabdi.explorer.arkiv.network/entity/<key> is a real
page (full entity, attributes, tx history), not a guess — checked against a
real entity already on the chain. Stronger than the generic explorer
homepage link this replaces: one click per decision goes straight to proof.
… mirror

Confirmed real:
- server.js mapped createdEntities.map(e => e.key), but the SDK's own type
  (ExecuteBatchReturnType.createdEntities: Hex[]) says those are already
  plain strings, not objects — checked against the actual .d.ts, not taken
  on faith. entity_keys always came back [null, ...]; nothing read it yet,
  so nothing broke, but a future consumer would have. Fixed.
- requester was on the entity and never queried (reads filter by $owner,
  not requester), and Arkiv entities are append-only, so it would outlive a
  requester's own "discard this draft" forever (until TTL). Dropped. Also
  kills the separate merged-account bug (mirrored principal.address, not
  plan.requester) for free, since the field no longer exists.
- The mirror wasn't actually replayable, despite why-arkiv.md's own claim:
  migration 0016 says routing_decisions exists so "any draw can be replayed
  from its seed and the posteriors it saw," but seed was never on the Arkiv
  copy. Added decisionId, seed, budgetMicros via INSERT ... RETURNING on the
  Postgres insert (not recomputed separately), plus decidedAt now comes from
  Postgres's own created_at instead of a second clock reading in the
  background task.
- Added ORDER BY $key ASC to the read query (pagination stability, per the
  docs' own note); verified live, no parse error.
- Implemented the "matches ✓" idea: the panel now compares the picks
  Postgres just answered with against what Arkiv reads back for the same
  task and shows whether they match — a real verification feature, not just
  two separate lists.

Deferred, not done (flagged to the user, their call):
- An arkiv_mirrors table + outbox-style retry loop for lost writes. Real
  engineering, not required by the track's judging rules with ~1 day left;
  best-effort stays for the hackathon.
- Deduping entities across repeated explore clicks: not a bug — each
  explore call is a genuinely new Thompson draw, and Postgres's own
  routing_decisions is append-only for the same reason.
- The posteriors-snapshot-instead-of-routing_decisions pivot: a legitimate
  design question, too large a change to make now without re-verifying
  everything live again.

Re-verified against the real Rabdi RPC (unfunded wallet): a 2-entity batch
with the new schema reaches gas estimation and fails there as before
(expected, 0 balance) — confirms the new fields and the fixed entity_keys
encode correctly, not just that they compile.

Full suite green: 546 fast + 450 postgres, ruff/pyright clean, 292 frontend
tests, tsc/eslint/build clean.
Both PR checks were failing on pure formatting — ruff format --check
(test_routes_routing.py) and prettier --check (SuggestAgents.tsx) — neither
of which I'd run, only the linters (ruff check, eslint). No logic change.

Re-verified after reformatting: ruff check/format, pyright, full control-
plane suite (546 fast + 450 postgres), tsc, eslint, vitest (292), and the
e2e specs that touch this component (e2e/live/run.spec.ts, 21/21) all green.
Found while answering "is this safe to merge": deploy/compose.controlplane.yaml
gates arkiv-sidecar behind the arkiv profile, but update.sh (the server's own
redeploy script) only activated keyrelay's profile conditionally — it would
never pass --profile arkiv, even after deploy/secrets/arkiv.env exists.
Without this, every future auto-deploy after someone sets that file up would
silently leave the sidecar off until someone ran the compose command by hand.

Confirmed safe to merge as-is: `docker compose config` with no --profile
arkiv and no deploy/secrets/arkiv.env on disk still exits 0 (the profile-
gated service and its env_file requirement are skipped entirely), so this
whole feature stays off by default until the secrets file is actually
created on the server — merging does not change prod behavior.
@OfficialAbhinavSingh
OfficialAbhinavSingh merged commit 68fd473 into main Nov 6, 2026
3 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Kanban Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant