Skip to content

Order reads by tick deadline behind a load balancer; retry sends without a nonce - #320

Merged
majesticwizardcat merged 16 commits into
mainfrom
feat/lb-stale-reads
Oct 28, 2026
Merged

majesticwizardcat merged 16 commits into
mainfrom
feat/lb-stale-reads

Conversation

@majesticwizardcat

@majesticwizardcat majesticwizardcat commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

Clients read pod through a load balancer across eventually-consistent fullnodes, so a lagging node can answer after a leading one and roll client state backwards. Ticks execute in lockstep on every node, so a tick deadline is a watermark comparable across nodes. The node now stamps every account read and push with timestamp (deadline of the last executed tick, micros), serves {withdrawals, timestamp} on new /v2/bridge/withdrawals routes (v1 unchanged), and returns error code 996 for retryable nonce rejections (node PR pod/pod#191 on Forgejo, ADR 0062). This PR is the SDK side.

TypeScript SDK

  • One helper keeps the highest watermark seen per entity and drops older pages and pushes: positions, balances, triggers, markets (stats page + push). Equal is accepted, later wins.
  • Balances are built from the pod_positions push (it carries every balances field); the REST re-fetch is gone.
  • Orders re-seed no longer overwrites rows the stream already advanced.
  • since: 0 subscriptions resubscribe with bounded backoff on rejection/close; one backoff formula shared by the sync modules.
  • Candle replays after a subscription error are deduped.
  • waitForReceipt treats a non-JSON or non-2xx response as "no receipt yet".
  • Withdrawals read /v2/bridge/withdrawals[/{account}] ({withdrawals, timestamp}), subscribe with since from the page that ended the backfill, and keep that resume point across rejections.
  • Delegated submit: no nonce ownership. Attempts wait 250ms, 500ms, then 1s apart, each send is cut off after 5s, the whole submit gives up after 10s. Retried: transport errors, cut-off sends, non-JSON/non-2xx, and node error code 996 (past/future nonce, account pending). Terminal errors are thrown at once. Re-signs with a fresh nonce only when no attempt could have been delivered; otherwise waits for the receipt and ends with a delivery-unknown error.
  • Every REST read sends the highest watermark the entity holds as the node's optional timestamp query parameter (minTimestampUs on the PodRestClient methods); the node waits for its state to reach it. A 503 with Retry-After throws PodBehindError, retried once, then thrown to the resource's own retry path; a read never falls back to no minimum.
  • pod_positions, pod_triggers and pod_markets (re)subscribe with since set to the held watermark, read when each frame is sent (ws.subscribe accepts a params function). The node waits for it before accepting and rejects with code 995 otherwise; a 995 is resubscribed after 1s without spending the retry budget.
  • The keep-highest check stays for a different reason than before: two requests for one entity are in flight together (a poll and a push, a reconnect seed and a resubscribe snapshot) and land in either order.
  • mint encodes the faucet's current mint(to, tokens, uint128 deadline) ABI; the old bool form no longer parses on the node.
  • Live test npm run test:live (gated by POD_LIVE_RPC_A/POD_LIVE_RPC_B): round-robin fetch across two fullnodes with one made deaf; asserts no store ever moves backwards and the delegated send succeeds. Run by scripts/e2e-node-switch-test.sh in the node repo.

Rust SDK

  • Simple nonce manager: every send fetches pending.
  • send_transaction retries by the same rule (backoff, 5s per-attempt cut-off, 10s budget, code 996); new PodSendError::DeliveryUnknown { tx_hash } (survives From<RpcError>); re-sign only when the caller did not fix the nonce.
  • wait_past_perfect_time returns errors instead of swallowing them, and decodes the node's null result.

Public API changes

  • TS: optional trailing minTimestampUs on the PodRestClient read methods; new PodBehindError; submit throws PodDeliveryUnknownError with txHash.
  • TS: rest.bridgeWithdrawals returns WithdrawalsPage { withdrawals, timestampUs } (was an array); optional timestampUs on PositionsSnapshot/Balances; sendRawTransaction throws on non-2xx / missing result, accepts a signal, and PodTxRevertError carries code; MintParams/client.mint lose depositToClob (the faucet always credits the CLOB balance).
  • Rust: RecommendedFillers names NonceFiller<SimpleNonceManager>; on_url bounds changed; PodSendError gains DeliveryUnknown (non-exhaustive matches need an arm; the bridge relayer gets one with the rev bump).
  • Nonce-rejection retries need a node that returns code 996 (pod/pod#191).

Tests

vitest 173 tests (+1 live, skipped without nodes), cargo test -p pod-sdk 18 + 3 doctests, fmt and clippy clean. Live test passed 3/3 against a two-fullnode devnet.

🤖 Generated with Claude Code

@majesticwizardcat
majesticwizardcat marked this pull request as draft October 19, 2026 14:35
@majesticwizardcat
majesticwizardcat force-pushed the feat/lb-stale-reads branch 2 times, most recently from 3434384 to 6313994 Compare October 23, 2026 10:42
majesticwizardcat and others added 9 commits October 23, 2026 17:35
…out a nonce

TypeScript SDK: keep the highest `timestamp`/`solution_now` seen per entity and
drop older pages and pushes (positions, balances, triggers, markets); build
balances from the positions push; re-seed of orders keeps rows the stream
already advanced; `since: 0` subscriptions resubscribe with backoff; candle
replays are deduped; `waitForReceipt` treats non-JSON/non-2xx as no receipt;
withdrawals decode `{withdrawals, timestamp}` (and the old array) and subscribe
with `since` from the backfill; the delegated `submit` retries 250ms apart for
10s, re-signs only when no attempt could have been delivered, and otherwise
waits for the receipt and ends with delivery-unknown.

Rust SDK: simple nonce manager (fetch `pending` per send); `send_transaction`
retries by the same rule with `PodSendError::DeliveryUnknown`; re-sign only
when the caller did not fix the nonce; `wait_past_perfect_time` returns errors
instead of swallowing them and decodes the node's null result.

Docs: openapi, guides, design doc and README for the new `timestamp` field,
the withdrawals body and the send retry.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…y code

Both SDKs now wait 250ms, 500ms, then 1s between send attempts (10s budget
unchanged), cut each send off after 5s so a hung node cannot consume the
budget before the receipt poll, and retry a rejection only when the node
returns error code 996 (NonceRejected) instead of matching the reason text.

TypeScript: `mint` encodes the faucet's current `mint(to, tokens, uint128
deadline)` ABI (the old bool argument no longer parses on the node; the
`depositToClob` option is gone since the faucet always credits the CLOB
balance), `PodTxRevertError` carries the error code, and a live test
(`npm run test:live`, gated by POD_LIVE_RPC_A/B) drives a round-robin fetch
across two fullnodes with one made deaf and asserts no store ever moves
backwards and the delegated send still succeeds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The delegated wallet's delivery-unknown outcome carries the tx hash as a
field so the app can poll the receipt instead of matching the message.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every REST read sends the highest watermark the entity holds as the
`timestamp` query parameter; the node waits for its state to reach it. A
503 with `Retry-After` is retried once (PodBehindError), then the read
falls back to today's path. The drop-older check stays for nodes that
predate the parameter. The live test now expects zero dropped responses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The node keeps /v1/bridge/withdrawals as the bare array and serves the
{withdrawals, timestamp} body and the ?timestamp= minimum at /v2.
rest.bridgeWithdrawals reads /v2 and, on a 404, reads the v1 array with
timestampUs 0, so the SDK still works against a node without v2.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ails instead of reading without a minimum

pod_positions, pod_triggers and pod_markets (re)subscribe with since set to the
highest timestamp the source holds, read when each frame is sent, so a
reconnect to a lagging node waits for a snapshot at or past what the client
holds. A 995 rejection is retried every second without spending the retry
budget. readAtLeast retries once after Retry-After and then throws
PodBehindError to the resource's own retry path. The keep-highest check stays:
concurrent answers for one entity land in either order.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Nodes deploy before the SDK, so the route exists by the time this SDK ships.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t once

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s the minimum block

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
majesticwizardcat and others added 2 commits October 26, 2026 13:34
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@majesticwizardcat
majesticwizardcat marked this pull request as ready for review October 27, 2026 12:11
majesticwizardcat and others added 3 commits October 27, 2026 17:38
…ps the hash when it runs out

Two hung sends used to end as the transport error, losing the hash of a
transaction that may have executed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…eipt poll

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A restarted subscription sent since: 0 and accepted older state.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@0zzy-o 0zzy-o left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a comment picked up from the bot probably worth looking at.

Comment thread ts-sdk/src/sync/sources.ts Resolved
Comment thread ts-sdk/src/sync/sources.ts Resolved
majesticwizardcat and others added 2 commits October 28, 2026 11:25
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The wire type declared a token field the node never sends and no symbols,
so the public SpotPosition.token was always undefined and the symbols on
the wire were dropped; balancesSource had to cast the push entries.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@majesticwizardcat
majesticwizardcat merged commit 689e79f into main Oct 28, 2026
10 checks passed
@majesticwizardcat
majesticwizardcat deleted the feat/lb-stale-reads branch October 28, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants