Repository navigation
control-plane, frontend: mirror exploring routing suggestions to Arkiv (Build on Arkiv track) - #73
Merged
Merged
Conversation
…stnet
ETHGlobal Mumbai "Build on Arkiv (existing projects)" track, swapped in for EF
(no code had been built for that pick). An exploring suggestions call already
records its draw in Postgres (routing_decisions, pre-existing); this adds a
best-effort public mirror of the same decision on Arkiv, plus a new endpoint
and frontend panel that read it back, so the write side is provably read
somewhere a user sees rather than written and ignored.
- control-plane/mergit_cp/arkiv_mirror.py: thin wrapper over the official
arkiv-sdk (Python, pinned ==1.0.0b2 — no stable release yet), write/read,
defensive decoding since Arkiv is a public permissionless database
- routes_routing.py: mirror write after the Postgres transaction commits
(never fails the request), new GET /v0/plans/{uuid}/arkiv-suggestions
- frontend: "View on Arkiv" panel in SuggestAgents.tsx, reads live from the
new endpoint, shown only after an exploring suggestion
- arkiv/{schema,friction,reach,why-arkiv,how-you-use-arkiv}.md + README
sections for the track's qualification requirements
Needs a funded service wallet + access key (ARKIV_RPC_URL/ARKIV_PRIVATE_KEY/
ARKIV_ACCESS_KEY) on the hosted deploy before the wallet address and one live
query can be filled into README.md/arkiv/schema.md; everything else is
tested end to end against a fake Arkiv client (541 fast + 449 postgres tests
green, ruff clean, pyright strict clean on mergit_cp).
Self-review found the read side wasn't defensive like the write side:
constructing the real SDK client runs a balance check against the RPC
(arkiv-sdk's Arkiv.__init__), so a bad key or a down Rabdi testnet raised
inside GET /v0/plans/{uuid}/arkiv-suggestions uncaught, a 500 for an
unrelated-to-core-product feature. Now caught and reported as
{enabled: true, decisions: [], error: "could not reach Arkiv"}; the
frontend shows that distinctly from "nothing mirrored yet". Also hardened
_decode against a payload that's valid JSON but not an object (a bare
array), which would otherwise crash .get() the same way.
Full suite still green: 541 fast + 450 postgres, ruff/pyright clean,
292 frontend tests, tsc/eslint/build clean.
…n what is
A self-review (prompted by a second pass before the PR was declared done)
checked this track live against Rabdi for the first time instead of trusting
the SDK's docs, and found the whole write path, and the read path's query
shape, didn't work on this network:
- arkiv-sdk (Python)'s query options (`includeData`) are rejected by Rabdi
("unknown field includeData"), and its write target (processor address) has
no code deployed on Rabdi (`eth_getCode` returns "0x"). It targets a
different, incompatible Arkiv network. Confirmed live, not from reading
source: see arkiv/friction.md for the exact RPC errors.
- Rabdi has no payload/content-type field at the protocol level at all —
checked live on real entities. Everything is a named attribute.
- The query literal syntax (`str('...')`, unquoted `addr(0x...)`) differs
from what the JS quickstart's prose implies.
Rebuilt on what's actually confirmed to work live:
- Reads: plain httpx against `arkiv_query`, no SDK, now also filtered by
`$owner` (a spoofing gap the first version had — anyone can write an
entity with our exact schema tag).
- Writes: a new small Node sidecar (chain/arkiv/), using the official JS SDK
(confirmed, live, to target Rabdi's real write address), holding the
service wallet's private key — the control plane's own process and env
never hold it now, an improvement over the first version. One HTTP call
carries every task's decision for one plan, so the sidecar sends one
batched transaction (executeBatch), not one per task, serialized through
an in-process queue to avoid a nonce race between concurrent explore calls.
- Writes now run as a FastAPI BackgroundTask, not awaited inline: the first
version blocked the suggestions response on the write (the SDK's call
waits for a transaction receipt), which this fixes.
- An unfunded test wallet, run against the real Rabdi RPC from both the bare
sidecar and the built Docker image, reaches real gas estimation and fails
there ("Execution error without revert data", on a wallet confirmed 0
balance) — as far as this can be verified without a funded wallet.
- docker-compose: arkiv-sidecar service, gated by the `arkiv` profile (same
opt-in pattern as keyrelay), config validated with `docker compose config`.
- Docs (arkiv/*.md, both READMEs) rewritten to only state what was actually
checked live, including the wrong turns, rather than what the SDK's docs
imply.
Verified: full control-plane suite (545 fast + 450 postgres), ruff, pyright
strict clean, frontend (292 vitest, tsc, eslint, vite build) all still green.
The Arkiv panel itself was exercised in a real browser (mocked network
responses, not a real backend) by temporarily swapping the app's render
root — reverted before this commit, never shipped.
Still blocked on a funded service wallet (hub.arkiv.network/faucet is
wallet-connect gated, not a plain API, so this can't be done from here) for
an actual successful write and a real entity key.
… page Confirmed live: https://rabdi.explorer.arkiv.network/entity/<key> is a real page (full entity, attributes, tx history), not a guess — checked against a real entity already on the chain. Stronger than the generic explorer homepage link this replaces: one click per decision goes straight to proof.
… mirror Confirmed real: - server.js mapped createdEntities.map(e => e.key), but the SDK's own type (ExecuteBatchReturnType.createdEntities: Hex[]) says those are already plain strings, not objects — checked against the actual .d.ts, not taken on faith. entity_keys always came back [null, ...]; nothing read it yet, so nothing broke, but a future consumer would have. Fixed. - requester was on the entity and never queried (reads filter by $owner, not requester), and Arkiv entities are append-only, so it would outlive a requester's own "discard this draft" forever (until TTL). Dropped. Also kills the separate merged-account bug (mirrored principal.address, not plan.requester) for free, since the field no longer exists. - The mirror wasn't actually replayable, despite why-arkiv.md's own claim: migration 0016 says routing_decisions exists so "any draw can be replayed from its seed and the posteriors it saw," but seed was never on the Arkiv copy. Added decisionId, seed, budgetMicros via INSERT ... RETURNING on the Postgres insert (not recomputed separately), plus decidedAt now comes from Postgres's own created_at instead of a second clock reading in the background task. - Added ORDER BY $key ASC to the read query (pagination stability, per the docs' own note); verified live, no parse error. - Implemented the "matches ✓" idea: the panel now compares the picks Postgres just answered with against what Arkiv reads back for the same task and shows whether they match — a real verification feature, not just two separate lists. Deferred, not done (flagged to the user, their call): - An arkiv_mirrors table + outbox-style retry loop for lost writes. Real engineering, not required by the track's judging rules with ~1 day left; best-effort stays for the hackathon. - Deduping entities across repeated explore clicks: not a bug — each explore call is a genuinely new Thompson draw, and Postgres's own routing_decisions is append-only for the same reason. - The posteriors-snapshot-instead-of-routing_decisions pivot: a legitimate design question, too large a change to make now without re-verifying everything live again. Re-verified against the real Rabdi RPC (unfunded wallet): a 2-entity batch with the new schema reaches gas estimation and fails there as before (expected, 0 balance) — confirms the new fields and the fixed entity_keys encode correctly, not just that they compile. Full suite green: 546 fast + 450 postgres, ruff/pyright clean, 292 frontend tests, tsc/eslint/build clean.
Both PR checks were failing on pure formatting — ruff format --check (test_routes_routing.py) and prettier --check (SuggestAgents.tsx) — neither of which I'd run, only the linters (ruff check, eslint). No logic change. Re-verified after reformatting: ruff check/format, pyright, full control- plane suite (546 fast + 450 postgres), tsc, eslint, vitest (292), and the e2e specs that touch this component (e2e/live/run.spec.ts, 21/21) all green.
Found while answering "is this safe to merge": deploy/compose.controlplane.yaml gates arkiv-sidecar behind the arkiv profile, but update.sh (the server's own redeploy script) only activated keyrelay's profile conditionally — it would never pass --profile arkiv, even after deploy/secrets/arkiv.env exists. Without this, every future auto-deploy after someone sets that file up would silently leave the sidecar off until someone ran the compose command by hand. Confirmed safe to merge as-is: `docker compose config` with no --profile arkiv and no deploy/secrets/arkiv.env on disk still exits 0 (the profile- gated service and its env_file requirement are skipped entirely), so this whole feature stays off by default until the secrets file is actually created on the server — merging does not change prod behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #72 — not closing it yet: the issue's own "Done when" needs a real write plus a working
panel on mergit.site, neither of which has happened (no funded wallet yet).
Summary
First version of this PR used the Python
arkiv-sdkand claimed it worked end to end. It didn't —a self-review checked it live against Rabdi instead of trusting the SDK's docs, and found the
Python SDK targets a different, incompatible network entirely (wrong write address, rejected query
options).
arkiv/friction.mdhas the exact RPC errors and how to reproduce them.Current state, rebuilt on what's actually confirmed live against Rabdi:
httpxagainstarkiv_query(no SDK), filtered by schema tag, plan id, and$owner— the first version was missing the owner filter, which left it open to anyone writing aspoofed entity with Mergit's own schema tag. Ordered by key (
ORDER BY $key ASC).chain/arkiv/), using the official JS SDK (confirmed live totarget Rabdi's real write address), holding the service wallet's private key — the control
plane's own process/env never hold it. One HTTP call batches every task's decision for a plan
into one on-chain transaction (
executeBatch), serialized through an in-process queue so twoconcurrent explore calls can't race on the wallet's nonce.
BackgroundTasksentry, not awaited inline — blocking the response on the write(the SDK call waits for a transaction receipt) was the first version's bug; fixed.
GET /v0/plans/{uuid}/arkiv-suggestionsreads the mirror back, owner-only, degrading to{enabled: true, decisions: [], error: "..."}instead of a 500 if Arkiv is unreachable.SuggestAgents.tsx, shown only after an exploring suggestion,reading live from the endpoint above, each entity linking straight to its own Arkiv explorer page
(
rabdi.explorer.arkiv.network/entity/<key>, confirmed real against a live entity), with a"matches ✓" check against what Postgres answered with for the same task.
decisionId,seedandbudgetMicros(viaINSERT ... RETURNINGon the Postgres row, notrecomputed), matching migration 0016's own reason
routing_decisionsexists — "any draw can bereplayed from its seed and the posteriors it saw."
requesterattribute on the entity: it was never queried, and Arkiv entities are append-only(only expiry removes them), so attaching a wallet address to data that outlives a requester's own
"discard this draft" was wrong.
docker-compose:arkiv-sidecarservice, gated by thearkivprofile (same opt-in pattern askeyrelay); validated withdocker compose config, and the image was actually built and runtwice (
docker build+docker run, hit its real/healthzand/internal/mirror, includingafter the schema change below).
arkiv/{schema,friction,reach,why-arkiv,how-you-use-arkiv}.md+ both READMEs, rewritten to stateonly what was actually checked live.
Review round 2 (Yash)
Caught one real, confirmed bug:
server.jsmappedcreatedEntities.map(e => e.key), but the SDK'sown type (
ExecuteBatchReturnType.createdEntities: Hex[]) says those are already plain strings —entity_keysalways came back[null, ...]. Fixed. Also acted on the privacy point above(
requesterdropped) and the replayability gap (decisionId/seed/budgetMicrosadded) — bothreal, both fixed. Deferred: an
arkiv_mirrorstable + outbox-style retry loop for lost writes (realengineering, not required by the track's rules, ~1 day left); deduping entities across repeated
explore clicks (not a bug — each explore call is a genuinely new Thompson draw, same as Postgres's
own append-only table); the posteriors-snapshot-instead-of-routing_decisions redesign (legitimate
question, too large a change to make now without re-verifying everything live again).
What's verified, concretely
An unfunded test wallet, run against the real Rabdi RPC from the sidecar process, the built Docker
image, and again after the schema/payload changes above (now a 2-entity batch), reaches real
on-chain gas estimation and fails there every time ("Execution error without revert data", wallet
confirmed via
eth_getBalanceto hold 0 tGLM) — not at encoding, not at signing, not at the RPCconnection. That's as far as this can be verified without a funded wallet. The read-side query
syntax, including
ORDER BY $key ASC, was run live against the real RPC and returns correctly(empty, since nothing matches yet).
The Arkiv panel itself was exercised in a real browser against mocked network responses (not a real
backend) by temporarily swapping the app's render root for a direct mount — reverted before
committing, never shipped; confirmed via screenshot and DOM/accessibility-tree inspection that the
real shipped component renders both the picks list and the "View on Arkiv" decisions panel
correctly.
Still needed before this is live end to end
A funded service wallet (
hub.arkiv.network/faucet, wallet-connect gated — not a plain API, sothis can't be done from here) and an access key (
hub.arkiv.network/access-keys), set asARKIV_PRIVATE_KEY/ARKIV_SIDECAR_TOKENonchain/arkiv/'s own deploy andARKIV_RPC_URL/ARKIV_OWNER_ADDRESS/ARKIV_ACCESS_KEY/ARKIV_SIDECAR_URL/ARKIV_SIDECAR_TOKENon the control plane's. Once set, the wallet address and one live query need to be filled into
README.mdandarkiv/schema.md(currently placeholders, clearly marked).Test plan
uv run pytest -q— 546 passed, 1 skippeduv run pytest -m postgres -q— 450 passeduv run ruff check .— cleanuv run pyright— 0 errors (strict,mergit_cp/mergit_worker)tsc -b,eslint .,vitest run(292 passed),vite build— all cleandocker build chain/arkiv+docker run, hit/healthzand/internal/mirroragainst thereal Rabdi RPC with an unfunded wallet (fails at gas estimation, as expected), re-checked
after the schema fix with a 2-entity batch
docker compose configwith thearkivprofile active