diff --git a/.llm/runs/fix-redis-watch-recovery--issue-2639/context-pack.md b/.llm/runs/fix-redis-watch-recovery--issue-2639/context-pack.md new file mode 100644 index 0000000000..49822fb65b --- /dev/null +++ b/.llm/runs/fix-redis-watch-recovery--issue-2639/context-pack.md @@ -0,0 +1,21 @@ +# Context pack + +Issue2639 on fresh main b0eb07957. Draft bootstrap; product unchanged. Need actual Redis stop/restart red test because issue was code-read only. Feature tier separate PLAN-EVAL before implementation. Restore shared live subscriptions and reread keys after any acknowledged reconnect; bounded terminal typed errors, abort/close/lease cleanup. Existing errors become catchable exports. No Redis Streams or saga transports. Full CLI smoke N/A for internal KV recovery with direct real-server coverage. + +Two genuine baseline failures reproduced: server restart after subscriber end leaves existing watch silent; never-restarted server leaves iteration pending past35seconds instead of typed failure. Production remains main. Final plan now pins30-second recovery bound, native acknowledgement before snapshot, shared live-lease restore, cancellable generations and typed error exports. Next separate feature PLAN-EVAL before implementation. + +Short-gap oracle also fails on unchanged main: native MULTI kills exactly one real pub/sub connection then SETs the envelope before any resubscription; the next batch times out. This proves reconnect state loss without a mocked backend or a publication race. Total baseline: three distinct failing runtime regressions, no bootstrap failure. Final resource label scan empty. + +PLAN-EVAL cycle1 PASS at94a7bb380, comment6927438505, artifact plan-eval.md committed for shared context. No repairs; implementation next per approved protocol. Final independent IMPL-EVAL remains mandatory. + +S2 contract committed before recovery implementation: existing typed errors are named root/Redis exports with JSDoc; outage regression uses the public type. Package59file check and doc lint zero pass. Next implement approved connection/subscription/queue recovery and state rereads; existing red tests remain the oracle. + +S2 complete: original3real-outage regressions now green; full KV250pass/0fail/2existingignored with actual-server cases enabled. Typed failure/abort/debounce/siblings/prefix/close covered. Native recovery deadline30s from first disconnect, last cause retained, stale generation fenced. New focused watch-fixed-keys.ts reduces adapter747->704lines; S2filecount18<30 drift recorded. Static61file check/lint/fmt/quality, preflight/accuracy, doc lint0 and JSR dry-run pass. Docs describe current-state recovery/typed errors. All owned resources removed. Next commit manual inputs, generate canonical consumers/freshness, independent feature IMPL-EVAL, then current required checks/readiness. No scaffold/Aspire changes: full smoke remains N/A. + +S3canonical consumers refreshed: MCP prose/export corpus/publish assets change, CLIbarrel remains current; corpus/publish freshness pass. Source atbfaa29cd6 retains250pass/2existingignored actual-backend full suite and all static/package gates. New commit adds generated carriers/evidence only. Ready for independent feature IMPL-EVAL; acceptance3boxes need linked mirroring and final ready/head checks after evaluation. + +Latest final-source correction: per-observer lifetime cancels an in-flight reread when a yielded watch is returned without a caller AbortSignal, preserving shared recovery for siblings/new leases. Genuine local red->green test added before repair. Full KV now251pass/2existingignored (all7new real-server cases enabled), check/lint/fmt/quality pass. No public doc/signature change; corpus/publish freshness recheck after commit. Independent feature IMPL-EVAL next; mark3issue boxes with linked evidence before ready transition. + +All generated consumers/freshness and final source gates complete after per-lease correction. Acceptance3boxes checked/evidence linked; PRbodyCloses2639+onefencedYAML3entries; finalDoD/evalpending. Ownership preview0ownedcommands, reviewthreads0/0. Independent feature IMPL-EVAL requested MuseSpark1.3/xhigh via OpenCode Go, separate family/session, sourceea85fb387. Handoff commit is evidence only; retain draft until PASS/currentrequiredchecks. + +Independent feature IMPL-EVAL PASS at93bf1a6bc, comment6929229737; full251/0/2 and all10focused cases independently reproduced, source/gates/acceptance allpass. Retain evaluate.md then ready transition and final head checks/threads. Product remains sourceea85fb387; new commit run evidence only. diff --git a/.llm/runs/fix-redis-watch-recovery--issue-2639/drift.md b/.llm/runs/fix-redis-watch-recovery--issue-2639/drift.md new file mode 100644 index 0000000000..09e2613601 --- /dev/null +++ b/.llm/runs/fix-redis-watch-recovery--issue-2639/drift.md @@ -0,0 +1,10 @@ +# Drift + +None yet. Initial code-read finding remains unverified until actual runtime reproduction. No implementation authorized by inference before selected PLAN-EVAL PASS. Repository same-session evaluator steering yields to T3's required fresh delegate task for each evaluation/recheck; carry prior findings in the brief. + + +## Recovery implementation details + +Native @std retry requires a finite positive integer, so the outer recovery loop uses a finite count whose minimum cumulative delay exceeds the30-second budget; jitter disabled only on this outer bound. Existing connection attempts retain jitter. The budget starts at first disconnect (a tighter bound than native-window plus30seconds), includes reconnect/subscription/snapshot retries and retains the last underlying cause. Native automatic replay of unfulfilled subscriber commands is disabled alongside autoResubscribe so released leases cannot replay stale SUBSCRIBE. Primary failed connection promises reset so later snapshot attempts can reconnect. + +Per Plan-Gate F-1 watch-point, fixed-key bookkeeping is extracted to the focused adapters/redis/watch-fixed-keys.ts module instead of enlarging the already oversized adapter. The adapter shrinks below its baseline size; no public contract/context/port or new runtime class is introduced. This adds one internal file to S2 (18maximum vs planned17, still below30), within the approved same-concern split allowance. Existing prefix discovery policy is retained, waiting for shared recovery before polling and normalizing read failures into the existing typed KV hierarchy. diff --git a/.llm/runs/fix-redis-watch-recovery--issue-2639/evaluate.md b/.llm/runs/fix-redis-watch-recovery--issue-2639/evaluate.md new file mode 100644 index 0000000000..0bc4ce35e4 --- /dev/null +++ b/.llm/runs/fix-redis-watch-recovery--issue-2639/evaluate.md @@ -0,0 +1,142 @@ +# Evaluation: PR 2644 / issue 2639 — Redis watch recovery + +## Metadata + +| Field | Value | +| ----- | ----- | +| Run ID | `fix-redis-watch-recovery--issue-2639` | +| Target | PR `2644` / issue `2639` | +| Archetype | `3 - runtime/behavior`, folding Integration-2 | +| Scope overlays | none | +| Evaluator | feature IMPL-EVAL primary Muse Spark 1.3/xhigh via OpenCode Go; observed identity matches requested route, fallback Opus 5/xhigh unused | +| Generator | Codex gpt-6.1-sol/xhigh (OpenAI family) — separate session and separate vendor family from evaluator, no substitution | +| Evaluated head | `93bf1a6bce1b5b34fea6cf32a90cef795da42a3f`, branch `fix/redis-watch-recovery`, clean tree, base `origin/main` `b0eb079575968e843b781579673c9fe2ae250e07` | +| Source head | `ea85fb387d0e379b97c598aa4a9a14dea604a6dd` — byte-identical to evaluated head across `packages/`, `plugins/`, `docs/`; delta to evaluated head is run-artifact evidence only | +| Date | 2026-11-16 | + +## Process Verification + +| Check | Result | Evidence | +| ----- | ------ | -------- | +| Plan-Gate passed before implementation | PASS | `plan-eval.md` cycle 1 PASS at `94a7bb380`, 8/8 boxes, separate-session GLM 5.3 evaluator, different vendor family from generator | +| Design section exists in worklog | PASS | `worklog.md` ## Design: state machine connected/recovering/failed/closed, shared subscriber owner, generation fences, queue terminal failure, cancellable budget | +| Commit slices match design plan | PASS | 8 commits on draft PR: S1 red tests, S2 contract then recovery, S3 consumer assets, per-lease correction, handoff evidence only | +| Each slice has a passing gate | PASS | per-slice PR comments with red baseline, green regressions, full suite and static/package gates; independently re-verified below | +| No speculative seams | PASS | changed files all traced to plan groups; no new public abstraction, no new settings/dependencies | +| Constants for finite vocabularies | PASS | `REDIS_WATCH_RECOVERY_TIMEOUT_MS`, phase union, existing error hierarchy reused | + +## Static Gates + +| Gate | Command or check | Result | Evidence | Notes | +| ---- | ---------------- | ------ | -------- | ----- | +| Narrow typecheck | scoped check wrapper `--root packages/kv --ext ts,tsx` | PASS | 61 files, 0 findings, independently re-run | — | +| Lint | scoped lint wrapper `--root packages/kv --ext ts,tsx` | PASS | 61 files, 0 findings, independently re-run | — | +| Format | scoped fmt wrapper `--root packages/kv --ext ts,tsx` | PASS | 61 files, 0 findings, independently re-run | — | +| Doc lint | doc-lint runner `--root packages/kv` | PASS | 0 diagnostics across `mod.ts`, `redis.ts`, `kvdex.ts`, testing entrypoints, independently re-run | — | +| Publish dry-run | `deno publish --dry-run --allow-dirty` in `packages/kv` | PASS | dry run complete, full file list as expected | — | +| JSR audit | `audit-jsr-package.ts --root packages/kv` | PASS | 1 finding: WARN F-JSR-7 slow-types (single pre-existing inspection warning, also reported by generator) | not introduced by this change | +| release:preflight | `deno task release:preflight` | PASS | text-imports, import-attributes, file-url-import-meta, self-imports all PASS, independently re-run | — | +| docs:accuracy | `deno task docs:accuracy` | PASS | independently re-run | — | +| Link/path check | changed imports/exports | PASS | `mod.ts`/`redis.ts` re-export existing errors; `watch-fixed-keys.ts` imports existing owners only | — | + +## Fitness Gates + +`deno task quality:gate` independently re-run: exit 0, every package `FAIL=0`. `packages/kv` row `FAIL=0 WARN=4 INFO=1`; the two F-1 size warnings name pre-existing `deno-kv.adapter.ts`/`denokv-bridge.ts`, not changed files. + +| Gate | Function | Result | Evidence | Violations | +| ---- | -------- | ------ | -------- | ---------- | +| F-1 | File-size lint | PASS | adapter `747->704` lines, new focused `watch-fixed-keys.ts` 127 lines, within approved same-concern split allowance | none new | +| F-2 | Helper-reinvention scan | PASS | timing via `@std/async` retry/delay/abortable, logging via package logger | none | +| F-3 | Layering check | PASS | `watch-subscriptions.ts`/`watch-fixed-keys.ts`/`watch-batch-queue.ts` stay in existing adapter owners, import domain/ports only | none | +| F-4 | Inheritance audit | PASS | no new base behavior; existing error hierarchy reused | none | +| F-5 | Public surface audit | PASS | only existing `KvError`/`KvConnectionError`/`KvClosedError` newly exported from two existing entrypoints with JSDoc | none | +| F-6 | JSR publishability | PASS | dry-run + audit above | one pre-existing slow-type warning | +| F-7 | Doc-score gate | PASS | doc lint 0, README and guide document recovery/typed errors | none | +| F-8 | Workspace lib override | PASS | quality:gate covers | none | +| F-9 | Permission declaration | PASS | quality:gate covers | none | +| F-10 | Test-shape audit | PASS | 7 new live server cases + 3 queue cases, no `bridge_test.ts` growth | none | +| F-11 | Forbidden-folder lint | PASS | no new `utils`/`common`/`lib` folders | none | +| F-12 | Naming-convention lint | PASS | quality:gate covers | none | +| F-13 | Saga/runtime invariants | PASS | start/stop/failure/retry/cancellation exercised by live tests | none | +| F-14 | Console-log lint | PASS | logger used, no `console.*` in runtime | none | +| F-15 | Re-export-of-upstream lint | PASS | re-exports are existing local error identities, not upstream | none | +| F-16 | Folder-cardinality lint | PASS | no new top-level folders | none | +| F-17 | Abstract-derived co-location | PASS | quality:gate covers | none | +| F-18 | Sub-barrel lint | PASS | no new sub-barrels | none | +| F-19 | Scoped source gate runners | PASS | all package evidence from scoped wrappers, not raw root CLI | none | + +## Runtime Gates + +| Gate | Validation | Result | Evidence | +| ---- | ---------- | ------ | -------- | +| Focused live recovery + queue | `packages/kv/tests/redis-watch-recovery_test.ts` + `packages/kv/tests/redis-watch-failure_test.ts` with owned Docker daemon fixture | PASS | 10 passed / 0 failed, independently re-run (7 live server cases + 3 queue cases execute, none skip) | +| Full KV suite | `packages/kv/tests/` with owned external Redis plus Docker fixture | PASS | 251 passed / 0 failed / 2 ignored, independently re-run; the 2 ignores are the pre-existing CI-only unreachable-backend test and the old externally supplied-container integration test, no acceptance case skipped | +| Watch-point races | sibling-abort-after-end, final abort during paused reread + new watch + typed close, prefix restoration + typed close, queue idle/debounce/abort precedence, returned-lease cancellation without delaying sibling/fresh watch | PASS | covered by the 10 focused cases above, all green in the independent re-run; code paths spot-checked against `watch-subscriptions.ts` generation/lifetime fences and `watch-fixed-keys.ts` snapshot suppression | +| Resource hygiene | exact-identity container removal, unique labels, final label scan | PASS | owned container removed by immutable identity after label verification; post-run scan empty; foreign resources untouched | +| scaffold.runtime | N/A | PASS | approved in plan and PLAN-EVAL: no scaffold/Aspire/executable-launch output changes; direct actual-server tests are the acceptance oracle | + +## Consumer Gates + +| Consumer | Validation | Result | Evidence | +| -------- | ---------- | ------ | -------- | +| MCP prose | `check:mcp-export-corpus` | PASS | independently re-run, exit 0 | +| Symbol corpus + publish assets | `check:publish-assets` | PASS | independently re-run, exit 0 | +| CLI asset barrel | no scaffold output change | PASS | `git diff` shows no CLI scaffold changes; changed carriers are MCP prose, symbol corpus, publish assets only | + +Note: the broader `check:agent-docs-prose` chain fails in unrelated `fresh-ui` docs generation (`--config` module resolution in the site tooling). It is pre-existing tooling drift outside this run's consumer surface; the three carriers this run actually changes are fresh per the two checks above. Not a blocking finding. + +## Anti-Pattern Check + +In-scope patterns only; others N/A. + +| AP | Status | Evidence | Notes | +| -- | ------ | -------- | ----- | +| AP-1 | CLEAR | new coverage in focused test files, no `bridge_test.ts` growth; adapter shrinks | — | +| AP-4 | CLEAR | no new abstraction; fixed-key logic stays in existing adapter concern | — | +| AP-6 | CLEAR | no base class with lifecycle behavior added | — | +| AP-10 | CLEAR | callback errors normalized once into existing KV hierarchy; terminal failure delivered, not swallowed | — | +| AP-11 | CLEAR | no hidden globals; recovery state is instance-scoped | — | +| AP-12 | CLEAR | timing via `@std/async` retry/delay with AbortSignal; no direct timers in handlers | — | +| AP-13 | CLEAR | package logger used | — | +| AP-19 | CLEAR | no new `as unknown as`/`any` suppression introduced in changed files | — | +| AP-25 | CLEAR | `Deno.*`/side effects only in adapters/tests | — | + +## Arch-Debt Delta + +| Metric | Count | Evidence | +| ------ | ----- | -------- | +| New entries | 0 | no new doctrine violation | +| Resolved entries | 0 | open `bridge_test.ts` debt not touched | +| Deepened violations | 0 | F-1 improves (adapter shrinks); no Streams/saga-transport edits (`packages/plugin-sagas-core/src/transports/` untouched) | +| Unrecorded violations | 0 | — | + +## Findings + +| Severity | Finding | Evidence | Required action | +| -------- | ------- | -------- | --------------- | +| low | Broader `check:agent-docs-prose` chain fails in unrelated `fresh-ui` site tooling | independent re-run output names the site generator, not `packages/kv` | none for this run; track site tooling separately if needed | + +No high/medium findings. All six PLAN-EVAL watch-points verified in code and live tests: typed failure survives release paths; late generations cannot resurrect released channels; snapshot bookkeeping suppresses only racing entries without versionstamp ordering; prefix discovery rides the shared lifecycle with typed failure; owner-file sizes improve; retry budget is finite with minimum cumulative delay beyond 30 s plus existing jittered attempts. + +## Acceptance + +| Issue box | Result | Evidence | +| --------- | ------ | -------- | +| After subscriber end, next reconnect restores existing watches (real stop/restart test, red on main) | PASS | focused live case green in independent re-run; final IMPL proof comment linked in PR body mapping | +| After any reconnect, watch emits current value of keys changed during the gap | PASS | atomic subscriber-kill plus write-before-resubscribe case and restart-deletion reread green in independent re-run; mapping entry 2 | +| Unrecoverable watch rejects with typed KV error and does not hang | PASS | never-restart case rejects with public `KvConnectionError` within budget; prompt abort/close and fresh watch after failure green; mapping entry 3 | + +Close-gate mapping: PR body carries `Closes #2639` plus one fenced `acceptance-evidence` block with 3 `box-index` entries pointing at the final IMPL proof comment. All three issue boxes are checked. Mapping manually verified; the dry-run mirror policy-skips only because `status:ready-merge` is (correctly) absent — that label must never be applied by the evaluator. + +## Lessons for Promotion + +| Lesson | Pattern | Applies to | Confidence | +| ------ | ------- | ---------- | ---------- | +| none | — | — | — | + +## Verdict + +| Field | Value | +| ----- | ----- | +| Verdict | `PASS` | +| Rationale | Approved scope complete; all required static/fitness/runtime/consumer gates pass or are validly N/A with evidence; no unrecorded debt; docs and run artifacts support resume; final DoD box and readiness remain intentionally open pending this verdict and required head/thread checks. | diff --git a/.llm/runs/fix-redis-watch-recovery--issue-2639/plan-eval.md b/.llm/runs/fix-redis-watch-recovery--issue-2639/plan-eval.md new file mode 100644 index 0000000000..490e3fe515 --- /dev/null +++ b/.llm/runs/fix-redis-watch-recovery--issue-2639/plan-eval.md @@ -0,0 +1,76 @@ +# PLAN-EVAL — fix-redis-watch-recovery--issue-2639 + +- Evaluated head: `94a7bb380c92f44d028fb4d21788601d47131cd5` — branch `fix/redis-watch-recovery`, clean tree, base `origin/main` `b0eb079575968e843b781579673c9fe2ae250e07`. +- Plan evaluator (requested → observed): feature PLAN-EVAL primary route GLM 5.3 provider default → observed identity GLM 5.3 via OpenCode Go, provider default, options {}. Live matrix (`agentic:matrix --role plan_eval --tier feature`) confirms primary GLM 5.3 provider default, fallback Fable 5.1 low (not used). Fallback reason: none. +- Generator: Codex gpt-6.1-sol/xhigh (OpenAI family). Vendor family and session are independent from the evaluator; no substitution was made. +- Date: 2026-11-16. Separate-session review per `.llm/harness/evaluator/plan-protocol.md` + `gates/plan-gate.md`. +- Method: read-only source inspection of the operational topic worktree plus committed red-test evidence; no containers started (existing evidence accepted per brief), no production/test modifications, nothing committed. + +## Checklist results + +| Plan-Gate item | Result | Evidence / location | +| ---------------------------------------- | ------ | ------------------- | +| Research present and current | PASS | `.llm/runs/fix-redis-watch-recovery--issue-2639/research.md`; re-baselined against fresh `main` `b0eb07957`. All load-bearing findings spot-checked against the tree (see below). | +| Decisions locked | PASS | `plan.md` "Locked recovery protocol": single lease owner retained, autoResubscribe disabled only for the dedicated subscriber, explicit SUBSCRIBE ack before reread, 30s cancellable recovery bound, generation fences, queue terminal failure, existing error hierarchy exported with JSDoc, no new settings/dependencies. Rationale stated per decision; state machine (connected → recovering → resynchronizing/connected → failed/closed) named in `worklog.md` ## Design. | +| Open-decision sweep | PASS | `plan.md` "Open decisions: None", each deferred item classified (30s + native window, no public retry option, snapshot repeat semantics, one-time ready, abort/close precedence, prefix discovery unchanged). Evaluator-run sweep found no rework-forcing open decision (see below). | +| Commit slices (< 30, gate + files each) | PASS | S1 = 7 files (six run artifacts + `packages/kv/tests/redis-watch-recovery_test.ts`) — landed at `94a7bb380` with per-slice PR comment; S2 ≤ 17 files, enumerated in `plan.md` "Concrete file groups"; S3 ≤ 14 files. Each names what it proves and its proving gate (real Redis outage suite + scoped wrappers + quality:gate/JSR gates). | +| Risk register | PASS | `plan.md` "Risks and deferred scope": seven risks with mitigations (ack-before-reread, serialization + identity fences, cancel-on-close/no-leases, current-state reread incl. deletes, bounded retry + typed queue failure, JSDoc/doc-lint for new exports, real-server baseline against fake-only proof). | +| Gate set selected | PASS | ARCHETYPE-3 (runtime/behavior) folding Integration-2: static scoped check/test/lint/fmt + publish dry-run; fitness via quality:gate (quality:scan + arch:check, F-1..F-19); runtime required → direct real Redis outage suite + full KV suite against isolated owned Redis; consumer gates required (exports change) → consumer/prose/corpus/publish freshness named. Full `scaffold.runtime` N/A justified: no scaffold output, plugin scaffolding, DB wiring, Aspire helper generation, or publish-shape change; PR carries `ci:skip-e2e` + `ci:skip-scaffold` consistent with that. | +| Deferred scope explicit | PASS | Prefix discovery, Redis Streams, saga transport changes, new abstractions, dependency/settings changes — all explicitly deferred in `plan.md` and PR body. No file under `packages/plugin-sagas-core/src/transports/` in the diff. | +| jsr-audit surface scan (package wave) | PASS | Planned public surface = existing `KvError`/`KvConnectionError`/`KvClosedError` (`packages/kv/application/errors.ts`) re-exported from `packages/kv/mod.ts` + `packages/kv/redis.ts` with JSDoc. No slow types possible (explicitly typed classes, explicit constructors; verified in source). Surface risks named: symbol JSDoc/doc-lint and publish dry-run assigned to S2/S3. | + +### Load-bearing research spot-checks (all verified against tree) + +1. `packages/kv/adapters/redis/connection.ts:127-152` — `ensureSubscriber()` has no `status === 'end'` handling (primary does at :94-97). Verified. +2. `connection.ts:298-301` — `retryStrategy` returns `null` after 5 attempts (100–1000 ms delays). Verified. +3. `connection.ts:269-271` — subscriber `error` handler only logs. Verified. +4. `packages/kv/adapters/redis.adapter.ts:530-535` — watch loop awaits `queue.next()` with no failure path; `packages/kv/adapters/redis/watch-batch-queue.ts` has no terminal failure state. Verified. +5. Error classes exist but are not exported from either entrypoint (`mod.ts`, `redis.ts`), while `getKv()` — publicly exported — throws `KvConnectionError` from `packages/kv/application/shared.ts:149`. Export rationale is contract-grounded, not cosmetic. Verified. +6. `@std/async@1.5.1` `RetryOptions.signal` and `delay({ signal })` exist — native AbortSignal cancellation of retry/delay is real. Verified via `deno doc`. +7. ioredis `^5.11.1` (pinned): `readyHandler` sets status `ready` **before** automatic resubscribe, and `autoResubscribe` is an existing option — the "await SUBSCRIBE acknowledgement, not raw ready" decision and the "disable autoResubscribe only for the dedicated subscriber" decision are both grounded in the pinned dependency. Verified in the installed ioredis 5.11.1 source/types. +8. Existing tests cover shared-subscription leases, independent releases, discovery and startup consistency, but no real outage — the only outage coverage is the new red test. Verified. +9. Baseline red evidence: three distinct failures on unchanged main product source — restart-after-end iterator hang (TimeoutError), leave-down case raising `DOMException` (deadline) instead of `KvConnectionError` within 35 s, and short-gap `CLIENT KILL TYPE pubsub` + `SET` value never reported after native reconnect. Structured raw reports (0 passed / 2 failed, then 0/1) reviewed; no mock backend; test-owned containers removed by exact identity and unique label, final label scan empty as recorded in the PLAN comment. + +## Acceptance coverage (issue #2639) + +| Issue acceptance box | Covered by | Verdict | +| -------------------- | ---------- | ------- | +| After subscriber end, next reconnect restores existing watches; proven by a real stop/restart test failing on main | `packages/kv/tests/redis-watch-recovery_test.ts` (recover=true case): real `SHUTDOWN NOSAVE` → asserts subscriber `status === 'end'` → restart with PING readiness → existing iterator must deliver `['delete', null]` → a subsequent live write must deliver through the restored subscription. Red on main. | COVERED | +| After any reconnect, a watch emits the current value of keys that changed during the gap | Short-gap case: atomic native MULTI `CLIENT KILL TYPE pubsub` + `SET` writes storage before any new subscription can be accepted; iterator must emit the gap value; also covers the ended-then-restart reread (delete/null). Red on main. | COVERED | +| Unrecoverable watch rejects with a typed KV error and does not hang | recover=false case: Redis left down; pending `next()` must reject with `KvConnectionError` within 35 s. Red on main (deadline `DOMException` proves the iterator hangs). Plan's 30 s recovery bound + native window (~1.5 s observed within the 8 s end-wait) fits the 35 s test bound. | COVERED | + +## Feasibility review (requested checks) + +- **Race/cancellation feasibility:** feasible. `RetryOptions.signal`/`delay` signal support verified; design uses one shared recovery controller with its own abort (cancelled by adapter close or zero remaining leases), generation identity fencing stale callbacks and late results, listeners installed before SUBSCRIBE ack, messages buffered through recovery, and serialization tail never held across fresh-connection retry or rereads so release/cancel stay prompt. +- **Primary connection timing:** addressed precisely — snapshot rereads reuse `getMany` under bounded retry on the same recovery signal, so a still-reconnecting primary client cannot falsely end a watch; terminal failure occurs only when the 30 s budget is exhausted. +- **Ordering semantics:** precise — suppress only snapshot entries for keys touched by live messages during the read; queued messages retain delivery; never compare opaque Redis versionstamps; delivery is at-least-once current state (may repeat unchanged values), missing keys delivered as `delete`/null. Matches the red test expectations exactly. +- **Sibling leases:** live Symbol lease identities retained; one channel subscription restored once, shared by all leases; each lease keeps its own queue, resync and typed failure; releasing one lease never removes another's subscription (existing tests retained; S2 adds recovery-context proof). +- **No hang at terminal end:** queue terminal failure wakes `next()` and rejects with the original typed error; failures interrupt debounce; abort completes cleanly; explicit adapter close rejects waiting watches with `KvClosedError` before closing sockets; ended/reconnecting sockets disconnect rather than awaiting a QUIT that cannot complete; no background unhandled rejection. +- **Deadline math:** 30 s recovery budget bounds connect/auth/readiness, subscription restoration and snapshot retries together; test bounds (10 s post-restart delivery, 35 s terminal failure) are consistent with it plus the native reconnect window. + +## Open-decision sweep (evaluator-run) + +No unflagged decision that could force rework was found. Items checked and resolved by locked invariants: recovery triggers per case (native end vs short native reconnect — both named), second-outage generation invalidation, new watch acquires joining the shared controller during recovery, release racing resubscribe (generation fences + serialized acquire/release ordering), unsubscribe on dead sockets during terminal paths (disconnect-not-QUIT + phase states), abort-vs-close precedence (abort wins for already-aborted watches), prefix watches riding the shared subscription lifecycle while their SCAN polling self-heals discovery (verified `redis.adapter.ts` watchPrefix), and never-started lazy watches needing no close notification. + +## jsr-audit surface scan + +Package wave (`packages/kv`). Planned surface: three existing error classes re-exported from two existing entrypoints with JSDoc. No slow-type risk (explicitly typed classes; no inference introduced by re-export). Named risks and slices: symbol JSDoc + `deno doc --lint` (S2/S3), publish dry-run (S3), `application/errors.ts` already inside the publish include glob. One note: `errors.ts` module doc says "Internal error helpers" — update that wording when the classes become public contract so the published docs match the surface. + +## Debt + +Open `packages/kv` debt (AP-1 god test file `bridge_test.ts`) is not deepened: new coverage goes to `redis-watch-recovery_test.ts` plus one focused lifecycle test file, not `bridge_test.ts`. No new doctrine violation identified: timing stays in `@std` primitives inside adapters, logging via the package logger, no `console.*`, no new folders/ports/globals, each concern stays in its existing owner file. No new debt entry required. + +## Verdict + +`PASS` + +All eight Plan-Gate boxes are checked; all three issue acceptance boxes are covered by committed, red-on-main tests; the design is grounded in the pinned dependencies and existing source. Implementation may begin (S2). Feature-tier loop policy: max 2 cycles — this is cycle 1 with verdict PASS. + +## IMPL-EVAL watch-points (non-blocking, from this review) + +1. Release during a failed/recovering subscription must degrade to local bookkeeping — a rejecting UNSUBSCRIBE in the generator's `finally` must not mask the typed error already delivered to the consumer. +2. A late recovery generation must not re-subscribe a channel whose final release already completed, and unsubscribe failures on dead sockets must not hang the serialization tail or surface as unhandled rejections. +3. Snapshot synthetic events must keep `previousValues`/`knownVersions` bookkeeping consistent with live messages (previousValue continuity, no opaque-versionstamp comparisons). +4. Confirm prefix-watch behavior under the shared lifecycle change: discovery still self-heals via SCAN polling, and unrecoverable outages fail prefix queues with the same typed error rather than hanging. +5. File-size fitness (F-1): recovery logic lands in existing owner files (`watch-subscriptions.ts`, `connection.ts`, `watch-batch-queue.ts`); if any approaches the size lint, split within the same concern rather than growing a monolith. +6. Whether the connection-establishment series repeats inside the 30 s budget (the stronger reading, consistent with "bounds … retries together") should be settled in code so a server returning late-but-within-budget still recovers; both readings satisfy the acceptance bounds, so this is implementation discretion, not a plan gap. diff --git a/.llm/runs/fix-redis-watch-recovery--issue-2639/plan.md b/.llm/runs/fix-redis-watch-recovery--issue-2639/plan.md new file mode 100644 index 0000000000..5619c43a69 --- /dev/null +++ b/.llm/runs/fix-redis-watch-recovery--issue-2639/plan.md @@ -0,0 +1,49 @@ +# Plan + +## SKILL + +netscript-doctrine, netscript-harness, netscript-pr, netscript-tools, netscript-deno-toolchain, jsr-audit. + +## Contract and design checkpoint + +Keep existing watch/ready API. Publish the existing typed KV errors so callers can catch failures. Subscriber recovery must restore every live lease after the old physical subscriber ends; a short native reconnect must also reread watched keys after subscription acknowledgement. State watch delivery is at least once/current state, not every intermediate mutation. An unrecoverable bounded retry rejects iteration with KvConnectionError, with original cause. Explicit adapter close rejects waiting watches with KvClosedError; consumer abort completes cleanly. Releasing one lease never removes another's subscription. + +Recovery is bounded by existing native retry settings rather than endless silent waits. Use native retry AbortSignal, cancel on no remaining leases or adapter close. No idle keyspace polling or new options. Shared subscription owner supervises reconnect and terminal failure; watch loop handles state reread and queue failure. Design is locked below after real outage reproduction; no production implementation before separate PLAN-EVAL PASS. + +## Slices + +S1 owned-server reproduction and finalized lifecycle contract/plan, <=10files (run artifacts and redis-watch-recovery_test.ts); prove actual stopped server causes ended subscriber and existing iterator hangs. Compare a short reconnect gap and terminal failure. +S2 contract exports/JSDoc, minimal connection/subscription/queue/adapter repairs plus focused lifecycle tests/docs, <=12files. Prove long/short recovery, changed/deleted keys, two independent leases, typed terminal error and prompt abort/close. +S3 package/consumer gates and generated documentation assets, <=15files; final independent evaluation and required head checks. + +## Gates + +Structured scoped check/test/lint/fmt; real Redis outage suite plus full KV suite against isolated owned Redis; quality:gate, release:preflight, docs:accuracy, JSR audit/dry-run/doc lint, consumer/prose/corpus/publish freshness if docs/exports change. Full scaffold.runtime N/A: no scaffold/Aspire/executable wiring changes; direct real-server lifecycle tests are the acceptance oracle. Independent feature PLAN-EVAL selected before implementation, then feature IMPL-EVAL. No lock churn. + +## Risks and deferred scope + +Ready-before-resubscribe -> explicit acknowledgement before reread. Concurrent lease release/reacquire -> existing serialization and identity fences retained. Abort/close during recovery -> cancel native retry and pending connections; no retained handles. Outage mutation loss -> reread current key state including deletes. Infinite silent waits -> bounded retry and typed queue failure. Public errors -> existing hierarchy only, documented JSDoc/exports and doc lint/dry-run. Avoid fake-only proof -> real server stop/restart baseline. Prefix discovery/Streams/saga transport changes deferred; shared subscription lifecycle must preserve prefix behavior. No new abstractions or dependency changes. + +## Locked recovery protocol + +Retain WatchSubscriptions as the single lease owner. Extend its internal listener record with reconnect/error callbacks; no new public watch handle. Its per-channel Subscription keeps live Symbol lease identities, current physical subscriber and named connected/recovering/failed/closed phase. One shared recovery promise/controller supervises all active channels. A generation identity fences stale connection callbacks and late recovery results. Initial acquire and final release retain serialized SUBSCRIBE/UNSUBSCRIBE ordering. Do not hold the serialization tail across fresh-connection retry or state rereads: cancellation/release must remain prompt. + +Disable native automatic resubscription only for the dedicated subscriber. On a short native reconnect, explicitly SUBSCRIBE every still-live channel and await acknowledgement before invoking watch resynchronization. On native end, ensureSubscriber clears the ended instance/promise and obtains a new shared connection with the existing three-attempt retry policy. The recovery controller's30-second deadline bounds connect/auth/readiness, subscription restoration and snapshot retries together; caller close/no remaining leases aborts it. A second outage invalidates the prior generation. Final retry failure becomes KvConnectionError with cause, delivered to all affected queues; no background unhandled rejection. Explicit adapter close sends KvClosedError before closing sockets. Abort takes precedence for a watch whose caller already aborted. + +Use native @std retry/delay AbortSignal support, plus abort-aware pending subscriber connection establishment. The manager's close cancels pending subscriber work; ended/reconnecting sockets disconnect rather than awaiting a QUIT that can never complete. A completed connection is never disconnected by a stale attempt or a single lease release while another remains. + +Fixed-key resynchronization reuses getMany, with native bounded retry under the same recovery signal so primary-client reconnect timing does not falsely end a watch. Deliver current values for all observed keys, including missing keys as delete/null. These are change batches (no initial marker) and may repeat unchanged values; document at-least-once current-state delivery, not a mutation history. Install message listeners before acknowledgement and buffer messages throughout recovery. Track keys touched by live messages during a snapshot read and suppress only their racing snapshot entries; queued messages retain delivery. Never compare opaque Redis versionstamps for order. A new reconnect generation or caller abort cancels stale resync work before it can enqueue results. + +WatchBatchQueue gains a terminal failure state that wakes next() and rejects with the original typed error. Failures also interrupt debounce; abort completes cleanly. Callback errors normalize once into the existing KV error hierarchy. Export existing KvError, KvConnectionError and KvClosedError from root and Redis entrypoints with JSDoc so callers can catch them without private imports. No new configuration or dependency. + +## Concrete file groups + +S1 (7files): six run artifacts and tests/redis-watch-recovery_test.ts. Real stopped-server/end/restart loss and never-restarted typed-failure oracle; native atomic CLIENT KILL TYPE pubsub + SET proves a publication-free short gap before any resubscription. +S2 (17files maximum): application/errors.ts; mod.ts; redis.ts; adapters/redis/connection.ts, watch-subscriptions.ts, watch-batch-queue.ts; adapters/redis.adapter.ts; tests/redis-watch-recovery_test.ts and a focused lifecycle test file; README.md; docs/site/data-persistence/kv-queues-cron.md; six run artifacts. Contract/JSDoc exports first, then adapter implementation, green regression/lease-cancellation tests and docs. Each changed concern stays in its existing owner; no package split or generic lifecycle registry. +S3 (14files maximum): six run artifacts, independent evaluate.md, generated prose/corpus/MCP+CLI publish carriers and CLI asset barrel if changed. Static/runtime/package/consumer gates prove the final state, then required checks and review threads. No doctrine debt deepening; JSR exports reuse existing explicitly typed error classes and cannot add slow inference. + +## Open decisions + +None. Bounds are30seconds plus the preceding native reconnect window; no public retry option in this fix. Recovery snapshots may repeat values, matching at-least-once state observation. Initial ready remains one-time startup acknowledgement. In-flight watch close fails with KvClosedError, caller abort completes normally. Prefix discovery is unchanged, but shared lease recovery/failure must preserve its lifecycle. Native Redis-only test commands are diagnostic CLIENT/MULTI/SET, never Streams. Production recovery only requires ordinary SUBSCRIBE and existing reads. + +Implementation refinement under approved F-1 watch-point: fixed-key behavior moves to focused adapters/redis/watch-fixed-keys.ts, reducing the existing adapter below its baseline size. S2 therefore touches at most18files, still below30. No public abstraction added. Drift records finite native retry bound and tighter disconnect-based deadline. diff --git a/.llm/runs/fix-redis-watch-recovery--issue-2639/research.md b/.llm/runs/fix-redis-watch-recovery--issue-2639/research.md new file mode 100644 index 0000000000..f6be91fb9f --- /dev/null +++ b/.llm/runs/fix-redis-watch-recovery--issue-2639/research.md @@ -0,0 +1,14 @@ +# Research + +Full issue2639 read. Main still b0eb07957 after fresh fetch. Public WatchStream lazily starts and optionally exposes ready; watch is a current-state primitive, not a durable mutation log. Existing errors KvConnectionError/KvClosedError inherit KvError but are not public exports. Connection manager already resets ended primary clients but not ended subscribers. Both use native bounded reconnect attempts; subscriptions hold the ended client and only handle message events. WatchBatchQueue has no terminal failure path. Existing lease tests cover independent releases, shared subscription and startup consistency but no real outage. + +Pinned ioredis5.11.1 readyHandler sets status ready before automatic resubscription. A correct state reread must follow SUBSCRIBE acknowledgement rather than raw ready. @std/async/retry has native AbortSignal support. Reuse these contracts and existing error classes. Runtime reproduction not yet claimed; actual owned Redis server stop/restart test comes first. + +Doctrine archetype3 (watch lifecycle), folding integration2; existing adapters/redis boundary. No new backend or abstraction. Existing KV bridge/legacy-value debt is unchanged. No saga transport or Redis Streams work. + + +## Actual baseline outcome + +Two real-server tests fail on unchanged main product source after observing native subscriber status end. Restarted Redis answers PING but the existing iterator never reports the now-missing key within10seconds. Leaving Redis down never yields KvConnectionError within35seconds: the deadline itself rejects instead. Both are genuine iterator hangs, not connection/bootstrap failures. The uniquely labelled container remains alive only to keep its endpoint fixed; Redis server itself is stopped and started with native commands. Exact container identity and label fence teardown. Both test containers removed. Native short-gap oracle is a transaction that kills the real subscriber and writes storage before any new subscription can be accepted; no mock backend. + +Short-gap oracle also fails on unchanged main: native MULTI kills exactly one real pub/sub connection then SETs the envelope before any resubscription; the next batch times out. This proves reconnect state loss without a mocked backend or a publication race. Total baseline: three distinct failing runtime regressions, no bootstrap failure. Final resource label scan empty. diff --git a/.llm/runs/fix-redis-watch-recovery--issue-2639/supervisor.md b/.llm/runs/fix-redis-watch-recovery--issue-2639/supervisor.md new file mode 100644 index 0000000000..a6ad6e6152 --- /dev/null +++ b/.llm/runs/fix-redis-watch-recovery--issue-2639/supervisor.md @@ -0,0 +1,5 @@ +# Implementation lane + +Issue2639, milestone0.0.10. Branch fix/redis-watch-recovery from freshly fetched origin/main b0eb079575968e843b781579673c9fe2ae250e07. Owner-selected Codex gpt-6.1-sol/xhigh generator. Feature tier because the shared subscriber lifecycle must restore independent leases, resynchronize state and surface terminal errors. Separate PLAN-EVAL and IMPL-EVAL use the live feature matrix, different vendor family/session. No implementation subagents. No merge, ready-merge, rebase, force push or publication. + +Final IMPL-EVAL requested route from live feature row: muse_spark_1_3/xhigh primary via OpenCode Go; concrete catalog model muse-spark-1.3-contributor with variant xhigh. Fallback Opus5/xhigh remains unused unless authorized route resolution requires it. Separate session/vendor family mandatory (OpenAI generator vs Muse evaluator). Evaluator records observed identity and exact head in evaluate.md. diff --git a/.llm/runs/fix-redis-watch-recovery--issue-2639/worklog.md b/.llm/runs/fix-redis-watch-recovery--issue-2639/worklog.md new file mode 100644 index 0000000000..aa7aaf1d60 --- /dev/null +++ b/.llm/runs/fix-redis-watch-recovery--issue-2639/worklog.md @@ -0,0 +1,53 @@ +# Worklog + +## Design + +Current-state watch contract, live lease identity and lifecycle research are in plan.md. Initial state is connected; close enters reconnecting, successful SUBSCRIBE acknowledgement enters resynchronizing/connected, exhausted recovery enters failed, release/adapter close enters closed. Shared subscription owner supervises physical subscriber; individual watch queues own caller cancellation and typed failure propagation. Redis remains external clock/transport adapter; native retry/delay own bounded timing, all cleanup must cancel it. No new handler registry, port or global state. + +Bootstrap only. No production changes or reproduction claimed. Next write/run genuine server-outage regression on unchanged main and lock plan before separate evaluation. + + +## Red runtime reproduction + +Structured real Redis tests:0passed/2failed. Both reach subscriber end after the native retry window. Recovery case restarts the actual server and confirms PING, but iterator times out rather than reporting lost state. Terminal case times out after35seconds rather than throwing KvConnectionError. No product code changed. Test fixture uses a stable container endpoint while controlling its Redis server and removes only the exact owned labelled container. Short-gap test added next using native transactional kill+write, with no timing or publication dependency. + +Short-gap oracle also fails on unchanged main: native MULTI kills exactly one real pub/sub connection then SETs the envelope before any resubscription; the next batch times out. This proves reconnect state loss without a mocked backend or a publication race. Total baseline: three distinct failing runtime regressions, no bootstrap failure. Final resource label scan empty. + + +## Independent Plan-Gate + +Separate-session GLM5.3/provider-default PLAN-EVAL [PASS](https://github.com/rickylabs/netscript/pull/2644#issuecomment-6927438505), all8boxes; observed OpenCode Go route matches requested primary and differs from OpenAI generator family. No repairs required. Acceptance3/3 covered by genuine baseline regressions. Implementation watch-points retained in plan-eval.md: preserve typed failures through release, fence stale generations, snapshot bookkeeping, prefix lifecycle, owner-file size and retries throughout the30-second budget. S2 may begin; no production implementation yet. + + +## S2 contract first + +Expose the existing KvError/KvConnectionError/KvClosedError identities from root and Redis entrypoints, with public module/class/constructor documentation. Regression test imports KvConnectionError from the published Redis entrypoint. No recovery implementation yet. Structured package check59files passes, doc lint zero, changed error-class lint/format pass. Native lifecycle tests remain intentionally red until the next implementation slice. + + +## S2 recovery implementation and regression gates + +Retained the existing subscription owner and added bounded shared recovery: native short reconnect and ended-subscriber recreation explicitly SUBSCRIBE live channels, then invoke state rereads after acknowledgement. The30-second disconnect budget includes all native connection/restoration/snapshot retry work; last cause is preserved. Attempt cancellation fences late rereads and stale acquire cannot replace the active subscriber. Final lease release cancels pending work without removing a sibling. Expired/closed sockets never wait for QUIT/UNSUBSCRIBE during terminal cleanup. Existing failed primary/subscriber promises reset for fresh attempts. + +Fixed-key state bookkeeping moved into a focused127-line adapter module, reducing redis.adapter.ts747->704lines. Snapshot reads buffer live messages, skip racing entries, preserve previousValue and never compare opaque stamps for order. Missing keys emit delete/null; recovery may repeat unchanged values. Queue failures interrupt idle/debounce waits, retain public typed error, and caller abort wins. Adapter close sends KvClosedError before closing sockets. Prefix discovery interval retained, with restored subscription and typed close/read failures. README/guide document these semantics. + +Three original real baseline regressions now pass. Additional live coverage proves sibling preservation after native end, prompt final abort during a paused reread plus fresh watch, and restored prefix subscription/typed close. Queue unit cases prove typed debounce/idle failure and abort precedence. Final full KV run against isolated owned Redis plus enabled actual-server tests:250passed/0failed/2existingignored. Package check61files, scoped lint/fmt, quality:gate, release:preflight, docs:accuracy, doc lint zero and JSR dry-run/audit pass (one existing slow-type inspection warning). Exact identities/unique labels fence every test-owned cleanup; the full-suite Redis is removed in finally, foreign resources untouched. No dependency/lock/Streams/transport changes. Next canonical consumer documentation generation and independent IMPL-EVAL. + + +## S3 canonical consumer assets + +After committing manual source/docs, generated agent prose, MCP export corpus, CLI asset barrel and publish assets in canonical order. Export generation used --allow-dirty only for the expected sole generated prose carrier, warning reviewed. Changed carriers are MCP prose, symbol corpus and publish assets; no CLI scaffold output changes. Corpus and publish freshness pass. No dependency/lock churn. All final source gates remain green; next independent feature IMPL-EVAL and current required head/review-thread gates. + + +## Final per-lease cancellation regression + +Self-review found that returning a watch from a yielded batch while its recovery reread was paused could leave the shared recovery awaiting that released observer. Added a genuine failing test against the implemented S2 head, then gave each observer a cancellable lifetime. Returning one lease now detaches its pending reread without blocking a sibling or a new lease, including callers without their own AbortSignal. Focused red->green verified. Final full KV suite251passed/0failed/2existingignored with all7new real-server cases enabled. Structured61file check/lint/fmt and quality:gate pass. Two existing skips are the CI-only unreachable-backend test and the old externally supplied-container integration test; new acceptance cases do not skip. All exact-owned containers removed. Public contract/docs and generated symbol shape unchanged; verify freshness after the private cancellation correction. + + +## Final evaluation handoff + +After the private per-lease correction, corpus/publish freshness, release:preflight, docs:accuracy and doc lint0 recheck pass. Review-thread gate0/0. Read-only leak report/teardown preview show zero owned cleanup commands; foreign resources left untouched. All3issue acceptance boxes now checked with linked final IMPL evidence and exact fenced acceptance mapping in the PR. Feature matrix confirms MuseSpark1.3/xhigh independent final route, fallback Opus5/xhigh unused. Final handoff commit updates run artifacts only; source remains ea85fb387 with251passing tests. No readiness/merge claim until final evaluator PASS and required head checks. + + +## Independent implementation verdict + +[MuseSpark1.3/xhigh IMPL-EVAL PASS](https://github.com/rickylabs/netscript/pull/2644#issuecomment-6929229737) at93bf1a6bc. Observed OpenCode Go primary route matches request, separate session/vendor from OpenAI generator; no fallback. Evaluator independently runs scoped61file gates, quality/preflight/accuracy/doc lint0/JSR/freshness, focused10/0 actual-server+queue cases and full KV251/0/2; all acceptance3/3 and6plan watch-points plus per-lease race pass. One low non-blocking unrelated site-tooling note retained in evaluate.md. Final verdict-retention commit is evidence only. Ready transition/current required-head checks and review-thread gate next; no merge/ready-merge/publication. diff --git a/docs/site/data-persistence/kv-queues-cron.md b/docs/site/data-persistence/kv-queues-cron.md index e6f7749866..6ddb70388c 100644 --- a/docs/site/data-persistence/kv-queues-cron.md +++ b/docs/site/data-persistence/kv-queues-cron.md @@ -108,6 +108,14 @@ persistence configuration. ## KV +Redis fixed-key watches restore subscriptions and reread current key values after a reconnect, +including deletes during the gap. Recovery batches can repeat values and coalesce intermediate +changes; a watch observes state rather than retaining an event history. Recovery is bounded to +30 seconds from disconnect, after which iteration raises the public `KvConnectionError`. +Closing the adapter raises `KvClosedError` for active watches; aborting a watch's own signal +completes it normally. Both error classes are exported by `@netscript/kv` and `@netscript/kv/redis`. +Prefix-watch discovery continues to use its configured polling interval. + `getKv(config?)` returns a `WatchableKv` bound to the auto-detected backend. Keys are typed tuples, values are fully typed, and `KvSetOptions.expireIn` (milliseconds) sets a per-key TTL on every adapter. Pass a `SharedKvConfig` to override provider selection on first access. diff --git a/packages/kv/README.md b/packages/kv/README.md index cfd573fb4e..0480e8b08d 100644 --- a/packages/kv/README.md +++ b/packages/kv/README.md @@ -164,6 +164,26 @@ When `createLazyKv` wraps an adapter without `ready`, its promise resolves on th batch. A readiness rejection alone never becomes an iteration error: startup abort completes cleanly, while an actual adapter failure still rejects iteration. +## Redis watch recovery + +Fixed-key Redis watches restore live subscriptions after a lost connection, including when the +driver has exhausted its native reconnect attempts. After Redis acknowledges each restored +subscription, the watch rereads its keys and emits current state. Missing keys appear as +`delete` events with null values. Recovery batches have no `initial` marker and may repeat +unchanged values. Intermediate mutations during an outage can coalesce: this is a current-state +watch, not a durable event log. No idle keyspace scan is added to fixed-key watches. + +Recovery has a 30-second total budget from disconnect, including connection attempts, +subscription acknowledgements and state rereads. A watch that cannot recover rejects iteration +with `KvConnectionError`, preserving the cause. Import `KvError`, `KvConnectionError` or +`KvClosedError` from `@netscript/kv` or `@netscript/kv/redis` to catch them. Closing an adapter +rejects its active watches with `KvClosedError`; aborting a watch's own signal completes it +normally. Releasing one watch preserves its siblings, and releasing the last watch cancels its +pending recovery. A new watch can start a fresh recovery after failed watches have ended. + +Prefix watches share subscription restoration and typed failure/close handling. Their existing +`pollInterval` still controls discovery of keys changed without a delivered publication. + ## Public surface | Entry | What it gives you | diff --git a/packages/kv/adapters/redis.adapter.ts b/packages/kv/adapters/redis.adapter.ts index d8eabfc5e3..cc4eeebbec 100644 --- a/packages/kv/adapters/redis.adapter.ts +++ b/packages/kv/adapters/redis.adapter.ts @@ -15,6 +15,7 @@ */ import { createWatchHandle } from '../application/watch-handle.ts'; +import { KvConnectionError, KvError } from '../application/errors.ts'; import { createPackageLogger } from '@netscript/logger'; import { combineAtomicValue } from '../application/atomic-combine.ts'; import { generateVersionstamp, keyHasPrefix, keyToString } from '../application/keys.ts'; @@ -44,6 +45,7 @@ import { listEntries, scanEntries } from './redis/list-entries.ts'; import { keyToRedisKey } from './redis/serialization.ts'; import { WatchBatchQueue } from './redis/watch-batch-queue.ts'; import { WatchSubscriptions } from './redis/watch-subscriptions.ts'; +import { watchFixedKeys } from './redis/watch-fixed-keys.ts'; import { DEFAULT_REDIS_NAMESPACE, DEFAULT_REDIS_URL, @@ -118,7 +120,7 @@ export class RedisKvAdapter implements WatchableKv { const options = (config.options as import('ioredis').RedisOptions | undefined) ?? {}; this.connection = new RedisConnectionManager(url, options); this.subscriptions = new WatchSubscriptions( - () => this.connection.ensureSubscriber(), + (signal) => this.connection.ensureSubscriber(signal), (message) => this.parseWatchMessage(message), ); } @@ -440,6 +442,7 @@ export class RedisKvAdapter implements WatchableKv { * Close the Redis clients owned by the adapter. */ async close(): Promise { + this.subscriptions.close(); await this.connection.close(); } @@ -467,75 +470,13 @@ export class RedisKvAdapter implements WatchableKv { keys: KvKey[], options?: WatchOptions, ): WatchStream { - return createWatchHandle((ready) => this.watchKeys(keys, options, ready)); - } - - /** Subscribe, acknowledge readiness, and deliver fixed-key batches. */ - private async *watchKeys( - keys: KvKey[], - options: WatchOptions | undefined, - ready: () => void, - ): AsyncIterable[]> { - if (options?.signal?.aborted) return; - const channel = this.getWatchChannel(); - const previousValues = new Map(); - const knownVersions = new Map(); - const watchedKeys = new Set(keys.map(keyToString)); - const queue = new WatchBatchQueue>(); - const messageHandler = (data: WatchMessage): void => { - const keyStr = keyToString(data.key); - if (!watchedKeys.has(keyStr)) { - return; - } - const versionstamp = data.versionstamp ?? generateVersionstamp(); - if (knownVersions.get(keyStr) === versionstamp) return; - knownVersions.set(keyStr, versionstamp); - queue.push({ - key: data.key, - previousValue: previousValues.get(keyStr) ?? null, - timestamp: new Date(data.timestamp), - type: data.type, - value: data.value as T | null, - versionstamp, - }); - previousValues.set(keyStr, data.value as T | null); - }; - const release = await this.subscriptions.acquire(channel, messageHandler); - try { - if (options?.signal?.aborted) return; - ready(); - // Messages received during these reads remain queued. A read must not - // overwrite the previous value already advanced by a startup message. - // Its first previousValue is null: the baseline may already contain that - // write (or a newer write), so it cannot establish the preceding value. - const entries = await this.getMany(keys); - for (const [index, key] of keys.entries()) { - const entry = entries[index]; - const keyStr = keyToString(key); - if (!knownVersions.has(keyStr)) previousValues.set(keyStr, entry?.value ?? null); - } - if (options?.initial && !options.signal?.aborted) { - yield keys.map((key, index) => { - const entry = entries[index]; - return { - key, - timestamp: new Date(), - type: entry ? 'set' : 'delete', - value: entry?.value ?? null, - versionstamp: entry?.versionstamp ?? null, - initial: true, - }; - }); - } - while (!options?.signal?.aborted) { - const events = await queue.next(options?.debounce, options?.signal); - if (events.length > 0 && !options?.signal?.aborted) { - yield events; - } - } - } finally { - await release(); - } + return createWatchHandle((ready) => + watchFixedKeys(keys, options, ready, { + channel: this.getWatchChannel(), + subscriptions: this.subscriptions, + getMany: (keys) => this.getMany(keys), + }) + ); } // --------------------------------------------------------------------------- @@ -575,7 +516,16 @@ export class RedisKvAdapter implements WatchableKv { versionstamp, }); }; - const release = await this.subscriptions.acquire(channel, messageHandler); + let release: () => Promise; + try { + release = await this.subscriptions.acquire(channel, messageHandler, { + error: (error) => queue.fail(error), + signal: options?.signal, + }); + } catch (error) { + if (options?.signal?.aborted) return; + throw error; + } try { for await (const entry of this.scanPrefix(prefix)) { const keyStr = keyToString(entry.key); @@ -632,6 +582,7 @@ export class RedisKvAdapter implements WatchableKv { const seenKeys = new Set(); + await this.subscriptions.waitForRecovery(options?.signal); for await (const entry of this.scanPrefix(prefix)) { const keyStr = keyToString(entry.key); const versionstamp = entry.versionstamp ?? generateVersionstamp(); @@ -668,6 +619,12 @@ export class RedisKvAdapter implements WatchableKv { } nextScanAt = Date.now() + pollInterval; } + } catch (error) { + if (options?.signal?.aborted) return; + throw error instanceof KvError ? error : new KvConnectionError( + 'Redis prefix watch failed.', + { cause: error }, + ); } finally { await release(); } diff --git a/packages/kv/adapters/redis/connection.ts b/packages/kv/adapters/redis/connection.ts index 04bd044f94..7e3856bb7b 100644 --- a/packages/kv/adapters/redis/connection.ts +++ b/packages/kv/adapters/redis/connection.ts @@ -63,6 +63,7 @@ export class RedisConnectionManager { private clientPromise: Promise | null = null; private subscriberPromise: Promise | null = null; private closed = false; + private readonly closing = new AbortController(); /** * @param url - Redis connection URL. @@ -108,6 +109,9 @@ export class RedisConnectionManager { } this.client = client; return client; + }).catch((error: unknown) => { + this.clientPromise = null; + throw error; }); } @@ -122,24 +126,37 @@ export class RedisConnectionManager { * A separate ioredis instance is required because Redis pub/sub locks the * connection into subscribe mode. * + * @param signal - Cancel pending connection/retry work. * @returns Connected Redis subscriber client. */ - async ensureSubscriber(): Promise { + async ensureSubscriber(signal?: AbortSignal): Promise { if (this.closed) { throw new KvClosedError('RedisKvAdapter is closed.'); } + signal?.throwIfAborted(); + if (this.subscriber?.status === 'end') { + this.subscriber = null; + this.subscriberPromise = null; + } if (this.subscriber) { return this.subscriber; } + const connectionSignal = signal + ? AbortSignal.any([signal, this.closing.signal]) + : this.closing.signal; this.subscriberPromise ??= retry( - () => this.connectSubscriberOnce(), - RETRY_OPTIONS, + () => this.connectSubscriberOnce(connectionSignal), + { ...RETRY_OPTIONS, signal: connectionSignal }, ).then(async (subscriber) => { if (this.closed) { await subscriber.quit(); throw new KvClosedError('RedisKvAdapter is closed.'); } + if (connectionSignal.aborted) { + subscriber.disconnect(); + throw connectionSignal.reason; + } this.subscriber = subscriber; return subscriber; }).catch((error: unknown) => { @@ -162,14 +179,17 @@ export class RedisConnectionManager { } this.closed = true; + this.closing.abort(new KvClosedError('RedisKvAdapter is closed.')); if (this.subscriber) { - await this.subscriber.quit(); + if (this.subscriber.status === 'ready') await this.subscriber.quit(); + else this.subscriber.disconnect(); this.subscriber = null; } if (this.client) { - await this.client.quit(); + if (this.client.status === 'ready') await this.client.quit(); + else this.client.disconnect(); this.client = null; } } @@ -248,16 +268,47 @@ export class RedisConnectionManager { * @returns Connected Redis subscriber. * @throws {KvConnectionError} When the connection attempt fails. */ - private async connectSubscriberOnce(): Promise { - const subscriber = new Redis(this.url, this.createRedisOptions()); + private async connectSubscriberOnce(signal: AbortSignal): Promise { + signal.throwIfAborted(); + const subscriber = new Redis(this.url, { + ...this.createRedisOptions(), + // The lease owner restores only channels that still have consumers, then + // rereads their state after acknowledgement. Stale commands cannot replay. + autoResubscribe: false, + autoResendUnfulfilledCommands: false, + }); try { await new Promise((resolve, reject) => { - subscriber.once('ready', () => resolve()); - subscriber.once('error', (error: Error) => reject(error)); + const cleanup = (): void => { + subscriber.off('ready', onReady); + subscriber.off('error', onError); + signal.removeEventListener('abort', onAbort); + }; + const onReady = (): void => { + cleanup(); + resolve(); + }; + const onError = (error: Error): void => { + cleanup(); + reject(error); + }; + const onAbort = (): void => { + // A native connection may already be ready while its acknowledgement + // is in flight. Send QUIT once, then guarantee the socket is closed. + if (subscriber.status === 'ready') void subscriber.quit().catch(() => {}); + subscriber.disconnect(); + cleanup(); + reject(signal.reason); + }; + subscriber.once('ready', onReady); + subscriber.once('error', onError); + signal.addEventListener('abort', onAbort, { once: true }); + if (signal.aborted) onAbort(); }); } catch (error: unknown) { subscriber.disconnect(); + if (signal.aborted) throw signal.reason; throw new KvConnectionError( `Failed to connect RedisKvAdapter subscriber to ${maskRedisUrl(this.url)}: ${ toErrorMessage(error) diff --git a/packages/kv/adapters/redis/watch-batch-queue.ts b/packages/kv/adapters/redis/watch-batch-queue.ts index 93a7289e41..b388f007b4 100644 --- a/packages/kv/adapters/redis/watch-batch-queue.ts +++ b/packages/kv/adapters/redis/watch-batch-queue.ts @@ -18,6 +18,14 @@ import { delay } from '@std/async'; export class WatchBatchQueue { #events: T[] = []; #wake: (() => void) | null = null; + readonly #failure = new AbortController(); + + /** Fail pending and future requests, including a pending debounce window. */ + fail(error: Error): void { + this.#failure.abort(error); + this.#wake?.(); + this.#wake = null; + } /** * Queue one event and wake a waiting batch request. @@ -25,6 +33,7 @@ export class WatchBatchQueue { * @param event - Event to queue. */ push(event: T): void { + if (this.#failure.signal.aborted) return; this.#events.push(event); const wake = this.#wake; this.#wake = null; @@ -48,15 +57,26 @@ export class WatchBatchQueue { signal?: AbortSignal, arrivalSignal: AbortSignal | undefined = signal, ): Promise { + if (signal?.aborted) return []; + this.#failure.signal.throwIfAborted(); if (this.#events.length === 0) { await this.#arrival(arrivalSignal); } + if (signal?.aborted) return []; + this.#failure.signal.throwIfAborted(); if (debounceMs && this.#events.length > 0 && !signal?.aborted) { - await delay(debounceMs, { signal }).catch((error: unknown) => { + const debounceSignal = signal + ? AbortSignal.any([signal, this.#failure.signal]) + : this.#failure.signal; + await delay(debounceMs, { signal: debounceSignal }).catch((error: unknown) => { // Aborted mid-window: the stream is ending, not failing. - if (!signal?.aborted) throw error; + if (!signal?.aborted) { + this.#failure.signal.throwIfAborted(); + throw error; + } }); } + if (!signal?.aborted) this.#failure.signal.throwIfAborted(); return signal?.aborted ? [] : this.#events.splice(0); } diff --git a/packages/kv/adapters/redis/watch-fixed-keys.ts b/packages/kv/adapters/redis/watch-fixed-keys.ts new file mode 100644 index 0000000000..0e863c32c9 --- /dev/null +++ b/packages/kv/adapters/redis/watch-fixed-keys.ts @@ -0,0 +1,127 @@ +/** Fixed-key Redis watch snapshots, change delivery and recovery bookkeeping. + * @module + */ +import { abortable } from '@std/async'; +import { generateVersionstamp, keyToString } from '../../application/keys.ts'; +import type { KvEntry, KvKey, WatchEvent, WatchOptions } from '../../types/common.ts'; +import type { WatchMessage } from './codec.ts'; +import { WatchBatchQueue } from './watch-batch-queue.ts'; +import type { WatchSubscriptions } from './watch-subscriptions.ts'; + +interface FixedKeyWatchContext { + readonly channel: string; + readonly subscriptions: WatchSubscriptions; + readonly getMany: (keys: KvKey[]) => Promise<(KvEntry | null)[]>; +} + +/** Subscribe before loading state and reread after each acknowledged recovery. */ +export async function* watchFixedKeys( + keys: KvKey[], + options: WatchOptions | undefined, + ready: () => void, + context: FixedKeyWatchContext, +): AsyncIterable[]> { + if (options?.signal?.aborted) return; + const previousValues = new Map(); + const knownVersions = new Map(); + const watchedKeys = new Set(keys.map(keyToString)); + const queue = new WatchBatchQueue>(); + let snapshotTouches: Set | null = null; + const messageHandler = (data: WatchMessage): void => { + const keyStr = keyToString(data.key); + if (!watchedKeys.has(keyStr)) { + return; + } + const versionstamp = data.versionstamp ?? generateVersionstamp(); + if (knownVersions.get(keyStr) === versionstamp) return; + snapshotTouches?.add(keyStr); + knownVersions.set(keyStr, versionstamp); + queue.push({ + key: data.key, + previousValue: previousValues.get(keyStr) ?? null, + timestamp: new Date(data.timestamp), + type: data.type, + value: data.value as T | null, + versionstamp, + }); + previousValues.set(keyStr, data.value as T | null); + }; + const reconnect = async (recoverySignal: AbortSignal): Promise => { + const signal = options?.signal + ? AbortSignal.any([options.signal, recoverySignal]) + : recoverySignal; + const touched = new Set(); + snapshotTouches = touched; + try { + const entries = await abortable(context.getMany(keys), signal); + signal.throwIfAborted(); + for (const [index, key] of keys.entries()) { + const keyStr = keyToString(key); + // A subscribed write arriving during the read owns this key's + // history. Its queued message must not be followed by a stale read. + if (touched.has(keyStr)) continue; + const entry = entries[index]; + queue.push({ + key, + timestamp: new Date(), + type: entry ? 'set' : 'delete', + value: entry?.value ?? null, + versionstamp: entry?.versionstamp ?? null, + previousValue: previousValues.get(keyStr) ?? null, + }); + previousValues.set(keyStr, entry?.value ?? null); + knownVersions.set(keyStr, entry?.versionstamp ?? null); + } + } catch (error) { + if (!options?.signal?.aborted) throw error; + } finally { + if (snapshotTouches === touched) snapshotTouches = null; + } + }; + let release: () => Promise; + try { + release = await context.subscriptions.acquire(context.channel, messageHandler, { + reconnect, + error: (error) => queue.fail(error), + signal: options?.signal, + }); + } catch (error) { + if (options?.signal?.aborted) return; + throw error; + } + try { + if (options?.signal?.aborted) return; + ready(); + // Messages received during these reads remain queued. A read must not + // overwrite the previous value already advanced by a startup message. + // Its first previousValue is null: the baseline may already contain that + // write (or a newer write), so it cannot establish the preceding value. + const entries = await context.getMany(keys); + for (const [index, key] of keys.entries()) { + const entry = entries[index]; + const keyStr = keyToString(key); + if (!knownVersions.has(keyStr)) previousValues.set(keyStr, entry?.value ?? null); + } + if (options?.initial && !options.signal?.aborted) { + yield keys.map((key, index) => { + const entry = entries[index]; + return { + key, + timestamp: new Date(), + type: entry ? 'set' : 'delete', + value: entry?.value ?? null, + versionstamp: entry?.versionstamp ?? null, + initial: true, + }; + }); + } + while (!options?.signal?.aborted) { + const events = await queue.next(options?.debounce, options?.signal); + if (events.length > 0 && !options?.signal?.aborted) { + yield events; + } + } + } finally { + await release(); + } +} diff --git a/packages/kv/adapters/redis/watch-subscriptions.ts b/packages/kv/adapters/redis/watch-subscriptions.ts index 422ef78876..7dff88c8fe 100644 --- a/packages/kv/adapters/redis/watch-subscriptions.ts +++ b/packages/kv/adapters/redis/watch-subscriptions.ts @@ -4,51 +4,94 @@ * @module */ +import { abortable, retry } from '@std/async'; import type { Redis } from 'ioredis'; +import { KvClosedError, KvConnectionError, type KvError } from '../../application/errors.ts'; -type Listener = (message: T) => void; +/** Total budget for restoring subscriptions and rereading state after an outage. */ +export const REDIS_WATCH_RECOVERY_TIMEOUT_MS = 30_000; + +interface Observer { + readonly lifetime: AbortController; + readonly message: (message: T) => void; + readonly reconnect?: (signal: AbortSignal) => Promise; + readonly error?: (error: KvError) => void; + readonly signal?: AbortSignal; +} interface Subscription { - readonly listeners: Map>; - readonly subscriber: Redis; - readonly handler: (channel: string, message: string) => void; + readonly listeners: Map>; + readonly lifetime: AbortController; } +type Phase = 'connected' | 'recovering' | 'failed' | 'closed'; + /** Owns one Redis subscription per channel, shared by independent watch leases. */ export class WatchSubscriptions { readonly #channels = new Map>(); #tail: Promise = Promise.resolve(); + #client: Redis | null = null; + #phase: Phase = 'connected'; + #failure: KvError | null = null; + #recovery: Promise | null = null; + #budget: AbortController | null = null; + #attempt: AbortController | null = null; constructor( - private readonly subscriber: () => Promise, + private readonly subscriber: (signal?: AbortSignal) => Promise, private readonly parse: (message: string) => T | null, ) {} /** Install a listener before SUBSCRIBE acknowledgement and return its release. */ - async acquire(channel: string, listener: Listener): Promise<() => Promise> { + async acquire( + channel: string, + listener: (message: T) => void, + options: Omit, 'message' | 'lifetime'> = {}, + ): Promise<() => Promise> { const lease = Symbol(); + const observer: Observer = { + message: listener, + ...options, + lifetime: new AbortController(), + }; + // Recovery needs the serialization tail for acknowledgements. Never hold + // that tail while waiting for the recovery itself or a fresh connection. + if (this.#phase === 'failed' && this.#channels.size === 0) { + this.#failure = null; + this.#phase = 'connected'; + } + await this.waitForRecovery(options.signal); + const client = await this.subscriber(options.signal); await this.#serialize(async () => { + if (this.#failure) throw this.#failure; + options.signal?.throwIfAborted(); + if ( + client.status !== 'ready' || + (this.#channels.size > 0 && this.#client !== null && this.#client !== client) + ) { + throw new KvConnectionError('Redis watch connection changed during acquisition.'); + } let subscription = this.#channels.get(channel); if (subscription) { - subscription.listeners.set(lease, listener); + subscription.listeners.set(lease, observer); return; } - - const subscriber = await this.subscriber(); - const listeners = new Map([[lease, listener]]); - const handler = (receivedChannel: string, message: string): void => { - if (receivedChannel !== channel) return; - const parsed = this.parse(message); - if (parsed) { for (const notify of listeners.values()) notify(parsed); } + this.#bind(client); + subscription = { + listeners: new Map([[lease, observer]]), + lifetime: new AbortController(), }; - subscription = { subscriber, listeners, handler }; - subscriber.on('message', handler); + this.#channels.set(channel, subscription); try { - await subscriber.subscribe(channel); - this.#channels.set(channel, subscription); + const subscribed = client.subscribe(channel); + await (options.signal ? abortable(subscribed, options.signal) : subscribed); } catch (error) { - subscriber.off('message', handler); - throw error; + this.#channels.delete(channel); + observer.lifetime.abort(error); + throw error instanceof KvClosedError ? error : new KvConnectionError( + 'Redis watch subscription failed.', + { cause: error }, + ); } }); @@ -56,16 +99,167 @@ export class WatchSubscriptions { return () => { if (released) return Promise.resolve(); released = true; - return this.#serialize(async () => { - const subscription = this.#channels.get(channel); - if (!subscription?.listeners.delete(lease) || subscription.listeners.size > 0) return; - this.#channels.delete(channel); - subscription.subscriber.off('message', subscription.handler); - await subscription.subscriber.unsubscribe(channel); + const subscription = this.#channels.get(channel); + if (!subscription?.listeners.delete(lease)) { + return Promise.resolve(); + } + observer.lifetime.abort(); + if (subscription.listeners.size > 0) return Promise.resolve(); + this.#channels.delete(channel); + subscription.lifetime.abort(); + const recovering = this.#recovery !== null; + if (recovering && this.#channels.size === 0) { + this.#budget?.abort(); + const old = this.#client; + this.#bind(null); + old?.disconnect(); + } + const removal = this.#serialize(async () => { + if (this.#phase === 'closed' || this.#phase === 'failed') return; + if (!this.#channels.has(channel) && this.#client?.status === 'ready') { + await this.#client.unsubscribe(channel); + } + }); + // Cancellation during recovery is local and prompt. Its queued physical + // unsubscribe precedes a future acquire without masking a stream failure. + if (recovering) { + void removal.catch(() => {}); + return Promise.resolve(); + } + return removal.catch((error: unknown) => { + if (this.#client?.status === 'ready' && this.#phase === 'connected') throw error; }); }; } + /** Wait outside the channel-operation tail before reading recovered state. */ + async waitForRecovery(signal?: AbortSignal): Promise { + if (this.#recovery) await (signal ? abortable(this.#recovery, signal) : this.#recovery); + signal?.throwIfAborted(); + if (this.#failure) throw this.#failure; + } + + /** Fail live leases before the adapter closes its physical sockets. */ + close(): void { + if (this.#phase === 'closed') return; + this.#phase = 'closed'; + this.#failure = new KvClosedError('RedisKvAdapter is closed.'); + this.#budget?.abort(this.#failure); + this.#attempt?.abort(this.#failure); + for (const subscription of this.#channels.values()) { + for (const observer of subscription.listeners.values()) { + observer.error?.(this.#failure); + observer.lifetime.abort(this.#failure); + } + subscription.lifetime.abort(this.#failure); + } + this.#channels.clear(); + this.#bind(null); + } + + readonly #message = (channel: string, message: string): void => { + const listeners = this.#channels.get(channel)?.listeners; + if (!listeners) return; + const parsed = this.parse(message); + if (parsed) { for (const observer of listeners.values()) observer.message(parsed); } + }; + + readonly #disconnected = (): void => { + this.#attempt?.abort(new KvConnectionError('Redis watch connection changed during recovery.')); + if (this.#recovery || this.#channels.size === 0 || this.#phase !== 'connected') return; + this.#phase = 'recovering'; + const budget = new AbortController(); + this.#budget = budget; + let lastFailure: unknown; + const timeout = setTimeout(() => + budget.abort( + new KvConnectionError( + 'Redis watch could not recover within 30 seconds.', + { cause: lastFailure }, + ), + ), REDIS_WATCH_RECOVERY_TIMEOUT_MS); + this.#recovery = retry(async () => { + try { + await this.#restore(budget.signal); + } catch (error) { + lastFailure = error; + throw error; + } + }, { + // Even minimum-delay retries cannot exhaust before the total deadline. + maxAttempts: Math.ceil(REDIS_WATCH_RECOVERY_TIMEOUT_MS / 100) + 1, + minTimeout: 100, + maxTimeout: 1_000, + jitter: 0, + signal: budget.signal, + }).catch((cause: unknown) => { + if (this.#phase === 'closed' || this.#channels.size === 0) return; + this.#phase = 'failed'; + this.#failure = new KvConnectionError('Redis watch recovery failed.', { cause }); + for (const subscription of this.#channels.values()) { + for (const observer of subscription.listeners.values()) observer.error?.(this.#failure); + } + }).finally(() => { + clearTimeout(timeout); + this.#recovery = null; + this.#budget = null; + this.#attempt = null; + if (this.#phase === 'recovering') this.#phase = 'connected'; + }); + }; + + async #restore(budget: AbortSignal): Promise { + const attempt = new AbortController(); + this.#attempt = attempt; + const signal = AbortSignal.any([budget, attempt.signal]); + const client = await this.subscriber(signal); + signal.throwIfAborted(); + if (client.status !== 'ready') throw new KvConnectionError('Redis subscriber is reconnecting.'); + this.#bind(client); + for (const [channel, subscription] of this.#channels) { + await this.#serialize(async () => { + if (this.#channels.get(channel) !== subscription) return; + const subscribed = AbortSignal.any([signal, subscription.lifetime.signal]); + await abortable(client.subscribe(channel), subscribed); + subscribed.throwIfAborted(); + }); + } + // Rereads are outside the serialization tail so abort/release can cancel + // them. A second disconnect aborts this attempt before snapshots enqueue. + await Promise.all( + [...this.#channels.values()].flatMap((subscription) => + [...subscription.listeners.values()].map(async (observer) => { + if (observer.signal?.aborted || observer.lifetime.signal.aborted) return; + const current = AbortSignal.any([ + signal, + subscription.lifetime.signal, + observer.lifetime.signal, + ]); + try { + await observer.reconnect?.(current); + } catch (error) { + // Returning a lease cancels only that observer, including a reread + // already captured by this attempt. Live siblings still recover. + if (!observer.lifetime.signal.aborted && !observer.signal?.aborted) throw error; + } + }) + ), + ); + signal.throwIfAborted(); + this.#phase = 'connected'; + } + + #bind(client: Redis | null): void { + if (client === this.#client) return; + this.#client?.off('message', this.#message); + this.#client?.off('close', this.#disconnected); + this.#client?.off('end', this.#disconnected); + this.#client = client; + client?.on('message', this.#message); + client?.on('close', this.#disconnected); + client?.on('end', this.#disconnected); + } + // Acquiring during the last release must wait for UNSUBSCRIBE before SUBSCRIBE. #serialize(operation: () => Promise): Promise { const pending = this.#tail.then(operation); diff --git a/packages/kv/application/errors.ts b/packages/kv/application/errors.ts index b826005eb0..5e5ffacb71 100644 --- a/packages/kv/application/errors.ts +++ b/packages/kv/application/errors.ts @@ -1,14 +1,25 @@ /** - * Internal error helpers for the KV package. + * Catchable failures from KV operations and watch streams. + * + * @module */ +/** Base class for failures reported by the KV adapters. */ export class KvError extends Error { + /** + * Create a KV failure while retaining its original cause. + * + * @param message - Description of the failed operation. + * @param options - Original cause and standard error options. + */ constructor(message: string, options?: ErrorOptions) { super(message, options); this.name = new.target.name; } } +/** The backend could not connect or a watch exhausted its recovery budget. */ export class KvConnectionError extends KvError {} +/** An operation or active watch failed because its adapter was closed. */ export class KvClosedError extends KvError {} diff --git a/packages/kv/mod.ts b/packages/kv/mod.ts index 1a0325fa7e..2c0426754d 100644 --- a/packages/kv/mod.ts +++ b/packages/kv/mod.ts @@ -40,5 +40,7 @@ export type { export { isWatchable } from './types/mod.ts'; +export { KvClosedError, KvConnectionError, KvError } from './application/errors.ts'; + export { DenoKvAdapter } from './adapters/deno-kv.adapter.ts'; export { MemoryKvAdapter } from './adapters/memory.adapter.ts'; diff --git a/packages/kv/redis.ts b/packages/kv/redis.ts index 07f1128898..d6f4a56513 100644 --- a/packages/kv/redis.ts +++ b/packages/kv/redis.ts @@ -50,4 +50,5 @@ export type { WatchPrefixOptions, WatchStream, } from './types/mod.ts'; +export { KvClosedError, KvConnectionError, KvError } from './application/errors.ts'; export { RedisKvAdapter, type RedisKvOptions } from './adapters/redis.adapter.ts'; diff --git a/packages/kv/tests/redis-watch-failure_test.ts b/packages/kv/tests/redis-watch-failure_test.ts new file mode 100644 index 0000000000..367cba64b4 --- /dev/null +++ b/packages/kv/tests/redis-watch-failure_test.ts @@ -0,0 +1,25 @@ +import { assertEquals, assertRejects } from '@std/assert'; +import { deadline } from '@std/async'; +import { KvClosedError } from '../mod.ts'; +import { WatchBatchQueue } from '../adapters/redis/watch-batch-queue.ts'; + +for (const debounce of [undefined, 30_000]) { + Deno.test(`Redis watch queue: typed failure interrupts ${debounce ? 'debounce' : 'idle wait'}`, async () => { + const queue = new WatchBatchQueue(); + if (debounce) queue.push(1); + const pending = queue.next(debounce); + queue.fail(new KvClosedError('closed')); + await assertRejects(() => deadline(pending, 500), KvClosedError, 'closed'); + await assertRejects(() => queue.next(), KvClosedError, 'closed'); + }); +} + +Deno.test('Redis watch queue: caller abort takes precedence over terminal failure', async () => { + const queue = new WatchBatchQueue(); + const controller = new AbortController(); + const pending = queue.next(undefined, controller.signal); + controller.abort(); + queue.fail(new KvClosedError('closed')); + assertEquals(await pending, []); + assertEquals(await queue.next(undefined, controller.signal), []); +}); diff --git a/packages/kv/tests/redis-watch-recovery_test.ts b/packages/kv/tests/redis-watch-recovery_test.ts new file mode 100644 index 0000000000..e52b79e3d8 --- /dev/null +++ b/packages/kv/tests/redis-watch-recovery_test.ts @@ -0,0 +1,436 @@ +import { assert, assertEquals, assertRejects } from '@std/assert'; +import { deadline, retry } from '@std/async'; +import { stub } from '@std/testing/mock'; +import { Redis } from 'ioredis'; +import { KvClosedError, KvConnectionError, RedisKvAdapter } from '../redis.ts'; +import type { KvKey } from '../mod.ts'; +import { WATCH_CHANNEL_SUFFIX } from '../adapters/redis/types.ts'; +import { encodeStoredValue } from '../adapters/redis/codec.ts'; +import { keyToRedisKey } from '../adapters/redis/serialization.ts'; + +const enabled = Deno.env.get('NETSCRIPT_TEST_REDIS_DOCKER') === '1'; + +async function docker(...args: string[]): Promise { + const result = await new Deno.Command('docker', { args, stdout: 'piped', stderr: 'piped' }) + .output(); + assertEquals(result.code, 0, new TextDecoder().decode(result.stderr)); + return new TextDecoder().decode(result.stdout).trim(); +} + +Deno.test({ + name: 'Redis watch: native reconnect rereads a value written while no subscriber exists', + ignore: !enabled, + async fn() { + const redis = await server(); + const namespace = `gap-${crypto.randomUUID()}`; + const controller = new AbortController(); + const kv = new RedisKvAdapter({ url: redis.url, namespace }); + const admin = new Redis(redis.url, { lazyConnect: true }); + const key = ['state']; + const iterator = kv.watch([key], { initial: true, signal: controller.signal }) + [Symbol.asyncIterator](); + let pending: Promise[]>> | undefined; + try { + await redis.start(); + await admin.connect(); + await kv.set(key, 'before'); + const initial = await deadline(iterator.next(), 5_000); + assert(!initial.done); + assertEquals(initial.value.map((event) => event.value), ['before']); + pending = iterator.next(); + pending.catch(() => {}); + // Atomic native commands kill the real subscriber and change storage + // before Redis can accept any new subscription. No mock or timing race, + // and no publication that could conceal a missing reconnect snapshot. + const result = await admin.multi() + .client('KILL', 'TYPE', 'pubsub') + .set(keyToRedisKey(key, namespace), encodeStoredValue('during-gap', 'gap-version')) + .exec(); + assert(result); + assertEquals(result[0], [null, 1]); + const recovered = await deadline(pending, 5_000); + assert(!recovered.done); + assertEquals(recovered.value.map((event) => event.value), ['during-gap']); + const next = iterator.next(); + await kv.set(key, 'after'); + const delivered = await deadline(next, 5_000); + assert(!delivered.done); + assertEquals(delivered.value.map((event) => event.value), ['after']); + } finally { + controller.abort(); + await Promise.allSettled([pending, iterator.return?.(), kv.close(), admin.quit()]); + admin.disconnect(); + await redis.close(); + } + }, +}); + +// The container remains alive while the actual Redis server stops and starts. +// Its published endpoint stays fixed, including with a remote Docker daemon. +async function server() { + const owner = crypto.randomUUID(); + const container = await docker( + 'run', + '-d', + '--label', + `netscript.test-owner=${owner}`, + '--publish', + '0.0.0.0::6379', + 'redis:7', + 'sh', + '-c', + 'exec sleep infinity', + ); + const start = async () => { + await docker( + 'exec', + container, + 'redis-server', + '--daemonize', + 'yes', + '--pidfile', + '/run/watch-test-redis.pid', + '--save', + '', + '--appendonly', + 'no', + ); + await retry(async () => { + assertEquals(await docker('exec', container, 'redis-cli', 'PING'), 'PONG'); + }, { maxAttempts: 100, minTimeout: 10, maxTimeout: 50 }); + }; + const stop = () => docker('exec', container, 'redis-cli', 'SHUTDOWN', 'NOSAVE'); + const binding = (await docker('port', container, '6379/tcp')).split('\n')[0]; + const port = Number(binding.slice(binding.lastIndexOf(':') + 1)); + const daemon = Deno.env.get('DOCKER_HOST'); + const hostname = daemon?.startsWith('tcp://') ? new URL(daemon).hostname : '127.0.0.1'; + return { + url: `redis://${hostname}:${port}`, + start, + stop, + async close() { + assertEquals( + await docker( + 'inspect', + '--format', + '{{index .Config.Labels "netscript.test-owner"}}', + container, + ), + owner, + ); + await docker('rm', '--force', container); + }, + }; +} + +for (const recover of [true, false]) { + Deno.test({ + name: `Redis watch: actual server outage after subscriber end ${ + recover ? 'restores the existing iterator' : 'rejects with a typed connection error' + }`, + ignore: !enabled, + async fn() { + const redis = await server(); + const controller = new AbortController(); + const kv = new RedisKvAdapter({ + url: redis.url, + namespace: `recovery-${crypto.randomUUID()}`, + }); + let subscriber: Redis | undefined; + const originalOn = Redis.prototype.on; + const on = stub(Redis.prototype, 'on', function (event, listener) { + if (event === 'message') subscriber = this; + return originalOn.call(this, event, listener); + }); + const key = ['state']; + const stream = kv.watch([key], { initial: true, signal: controller.signal }); + const iterator = stream[Symbol.asyncIterator](); + let pending: Promise[]>> | undefined; + try { + await redis.start(); + await kv.set(key, 'before'); + const initial = await deadline(iterator.next(), 5_000); + assert(!initial.done); + assertEquals(initial.value.map((event) => event.value), ['before']); + assert(subscriber, 'Observe the real subscriber installed by the adapter.'); + on.restore(); + const ended = new Promise((resolve) => subscriber!.once('end', resolve)); + pending = iterator.next(); + // Install rejection handling before exercising a future terminal failure. + pending.catch(() => {}); + await redis.stop(); + await deadline(ended, 8_000); + assertEquals(subscriber.status, 'end', 'native retry attempts really exhausted'); + if (recover) { + await redis.start(); + // No further watch() call is allowed to trigger recovery of this lease. + const result = await deadline(pending, 10_000); + assert(!result.done); + // The no-persistence test server lost its old value during the gap. + assertEquals(result.value.map((event) => [event.type, event.value]), [['delete', null]]); + const next = iterator.next(); + await kv.set(key, 'after-recovery'); + const delivered = await deadline(next, 5_000); + assert(!delivered.done); + assertEquals(delivered.value.map((event) => event.value), ['after-recovery']); + } else { + await assertRejects(() => deadline(pending!, 35_000), KvConnectionError); + await redis.start(); + const freshController = new AbortController(); + const fresh = kv.watch([key], { initial: true, signal: freshController.signal }) + [Symbol.asyncIterator](); + try { + const current = await deadline(fresh.next(), 5_000); + assert(!current.done); + assertEquals(current.value.map((event) => event.value), [null]); + } finally { + freshController.abort(); + await fresh.return?.(); + } + } + } finally { + if (!on.restored) on.restore(); + controller.abort(); + // Baseline QUIT/UNSUBSCRIBE on an ended socket can reject; preserve the + // regression verdict while always removing the positively owned server. + await Promise.allSettled([pending, iterator.return?.(), kv.close()]); + await redis.close(); + } + }, + }); +} + +Deno.test({ + name: 'Redis watch: aborting one lease after end preserves its recovering sibling', + ignore: !enabled, + async fn() { + const redis = await server(); + const kv = new RedisKvAdapter({ url: redis.url, namespace: `siblings-${crypto.randomUUID()}` }); + const first = new AbortController(); + const second = new AbortController(); + const a = kv.watch([['a']], { initial: true, signal: first.signal })[Symbol.asyncIterator](); + const b = kv.watch([['b']], { initial: true, signal: second.signal })[Symbol.asyncIterator](); + let subscriber: Redis | undefined; + const originalOn = Redis.prototype.on; + const on = stub(Redis.prototype, 'on', function (event, listener) { + if (event === 'message') subscriber = this; + return originalOn.call(this, event, listener); + }); + const pending: Promise[] = []; + try { + await redis.start(); + await kv.set(['a'], 'before-a'); + await kv.set(['b'], 'before-b'); + await deadline(a.next(), 5_000); + await deadline(b.next(), 5_000); + assert(subscriber); + on.restore(); + const ended = new Promise((resolve) => subscriber!.once('end', resolve)); + const pa = a.next(); + const pb = b.next(); + pending.push(pa, pb); + pa.catch(() => {}); + pb.catch(() => {}); + await redis.stop(); + await deadline(ended, 8_000); + first.abort(); + assertEquals((await deadline(pa, 500)).done, true); + if (a.return) await deadline(a.return(), 500); + await redis.start(); + const recovered = await deadline(pb, 10_000); + assert(!recovered.done); + assertEquals(recovered.value.map((event) => event.value), [null]); + const live = b.next(); + pending.push(live); + await kv.set(['b'], 'still-live'); + const delivered = await deadline(live, 5_000); + assert(!delivered.done); + assertEquals(delivered.value.map((event) => event.value), ['still-live']); + } finally { + if (!on.restored) on.restore(); + first.abort(); + second.abort(); + await Promise.allSettled([...pending, a.return?.(), b.return?.(), kv.close()]); + await redis.close(); + } + }, +}); + +Deno.test({ + name: 'Redis watch: final abort cancels a paused reread; a new watch and typed close still work', + ignore: !enabled, + async fn() { + const redis = await server(); + const namespace = `cancel-${crypto.randomUUID()}`; + const kv = new RedisKvAdapter({ url: redis.url, namespace }); + const admin = new Redis(redis.url, { lazyConnect: true }); + const controller = new AbortController(); + const nextController = new AbortController(); + const watch = kv.watch([['state']], { initial: true, signal: controller.signal }) + [Symbol.asyncIterator](); + const replacement = kv.watch([['state']], { initial: true, signal: nextController.signal }) + [Symbol.asyncIterator](); + const reading = Promise.withResolvers(); + const resume = Promise.withResolvers(); + let restoreGet: (() => void) | undefined; + const pending: Promise[] = []; + try { + await redis.start(); + await admin.connect(); + await kv.set(['state'], 'current'); + await deadline(watch.next(), 5_000); + const original = kv.getMany.bind(kv); + const get = stub(kv, 'getMany', async function (keys: KvKey[]) { + const entries = await original(keys); + reading.resolve(); + await resume.promise; + return entries; + }); + restoreGet = () => { + if (!get.restored) get.restore(); + }; + const pa = watch.next(); + pending.push(pa); + await admin.client('KILL', 'TYPE', 'pubsub'); + await deadline(reading.promise, 5_000); + controller.abort(); + assertEquals((await deadline(pa, 500)).done, true); + if (watch.return) await deadline(watch.return(), 500); + // The consumer ends promptly; Redis observes the socket's FIN separately. + await retry(async () => { + assertEquals(await admin.pubsub('NUMSUB', `${namespace}:${WATCH_CHANNEL_SUFFIX}`), [ + `${namespace}:${WATCH_CHANNEL_SUFFIX}`, + 0, + ]); + }, { maxAttempts: 20, minTimeout: 10, maxTimeout: 20 }); + get.restore(); + resume.resolve(); + const initial = await deadline(replacement.next(), 5_000); + assert(!initial.done); + assertEquals(initial.value.map((event) => event.value), ['current']); + const closed = replacement.next(); + pending.push(closed); + closed.catch(() => {}); + await kv.close(); + await assertRejects(() => deadline(closed, 500), KvClosedError); + } finally { + restoreGet?.(); + resume.resolve(); + controller.abort(); + nextController.abort(); + await Promise.allSettled([ + ...pending, + watch.return?.(), + replacement.return?.(), + kv.close(), + admin.quit(), + ]); + admin.disconnect(); + await redis.close(); + } + }, +}); + +Deno.test({ + name: 'Redis prefix watch: restored subscription delivers live writes and close is typed', + ignore: !enabled, + async fn() { + const redis = await server(); + const namespace = `prefix-recovery-${crypto.randomUUID()}`; + const kv = new RedisKvAdapter({ url: redis.url, namespace }); + const admin = new Redis(redis.url, { lazyConnect: true }); + const controller = new AbortController(); + const iterator = kv.watchPrefix(['events'], { signal: controller.signal, pollInterval: 60_000 }) + [Symbol.asyncIterator](); + const pending: Promise[] = []; + try { + await redis.start(); + await admin.connect(); + await kv.set(['events', 'one'], 'initial'); + await deadline(iterator.next(), 5_000); + const live = iterator.next(); + pending.push(live); + await admin.client('KILL', 'TYPE', 'pubsub'); + await retry(async () => { + assertEquals(await admin.pubsub('NUMSUB', `${namespace}:${WATCH_CHANNEL_SUFFIX}`), [ + `${namespace}:${WATCH_CHANNEL_SUFFIX}`, + 1, + ]); + }, { maxAttempts: 100, minTimeout: 20, maxTimeout: 50 }); + await kv.set(['events', 'two'], 'after'); + const delivered = await deadline(live, 5_000); + assert(!delivered.done); + assertEquals(delivered.value.value, 'after'); + const closed = iterator.next(); + pending.push(closed); + closed.catch(() => {}); + await kv.close(); + await assertRejects(() => deadline(closed, 500), KvClosedError); + } finally { + controller.abort(); + await Promise.allSettled([...pending, iterator.return?.(), kv.close(), admin.quit()]); + admin.disconnect(); + await redis.close(); + } + }, +}); + +Deno.test({ + name: 'Redis watch: returning a paused lease cancels its reread without delaying a sibling', + ignore: !enabled, + async fn() { + const redis = await server(); + const kv = new RedisKvAdapter({ url: redis.url, namespace: `return-${crypto.randomUUID()}` }); + const admin = new Redis(redis.url, { lazyConnect: true }); + const a = kv.watch([['a']], { initial: true })[Symbol.asyncIterator](); + const controller = new AbortController(); + const b = kv.watch([['b']], { initial: true, signal: controller.signal }) + [Symbol.asyncIterator](); + const reading = Promise.withResolvers(); + const resume = Promise.withResolvers(); + let restoreGet: (() => void) | undefined; + let pending: Promise | undefined; + try { + await redis.start(); + await admin.connect(); + await deadline(a.next(), 5_000); + await deadline(b.next(), 5_000); + const original = kv.getMany.bind(kv); + const get = stub(kv, 'getMany', async function (keys: KvKey[]) { + const entries = await original(keys); + // Model a reread that depends on the lost connection. Keep its promise + // unresolved while siblings recover; returning its lease must detach it. + if (keys[0][0] === 'a') { + reading.resolve(); + await resume.promise; + } + return entries; + }); + restoreGet = () => { + if (!get.restored) get.restore(); + }; + await admin.client('KILL', 'TYPE', 'pubsub'); + await deadline(reading.promise, 5_000); + if (a.return) await deadline(a.return(), 500); + // A new lease waits for shared recovery. It must not be held behind the + // returned observer's indefinitely paused reread. + const fresh = kv.watch([['c']], { initial: true, signal: controller.signal }) + [Symbol.asyncIterator](); + try { + pending = fresh.next(); + pending.catch(() => {}); + await deadline(pending, 1_000); + } finally { + controller.abort(); + await fresh.return?.(); + } + } finally { + restoreGet?.(); + resume.resolve(); + controller.abort(); + await Promise.allSettled([pending, a.return?.(), b.return?.(), kv.close(), admin.quit()]); + admin.disconnect(); + await redis.close(); + } + }, +}); diff --git a/packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts b/packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts index c51627ed0b..3c6bba5734 100644 --- a/packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts +++ b/packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts @@ -67,7 +67,7 @@ export const EMBEDDED_AGENT_DOCS_PAGES: readonly AgentDocsPage[] = [ ["pages/data-persistence/index.md","H4sIAAAAAAAA/62VTW8bOQyG7/4VBBYIGsPjWaC3njZIC7TofmSbbi/BAqYlekYdWdKSHLvOr19IsidOuwsETW+2RqIeku9L/QSvUREuIBGLE6VgaDb72DsBIaMuBjBxRyyw74kJEH4nvTXskgKmBANREnAqIIpKgMFCH/eAIKanLYLpMXQE0rsky9lsPv9AJrKV+Rxe2JFx7WkBTB7zXegX8M9I7MjC3mkPCHpIZMF4R0EvwbsdgQugPcGdRcU1CoHHA/HfL3rVJK/alp0ZDh7Xsuyc9uN66WIbSKVQt/lUc5ZtewrTXi6OSdKO+ADJj53L6Qdltx6Vap7baMkLaCwQgluCHEHiyIaWMJ+/+UJmLJUrJcmJGjQ9yQKERFwMsoB95CGnOpZl05MdPVlQZwapaUrO8+79p1KRkQQuwHAM353msGtqoCaHyckmjjtniRvsQhR1BhKaATsS0B4VeAwQA9wTx8bEsHEd+GjQwxrNQMEKjEGdh7srSY7pqWT0JXkMpd8tlpPtZWUptSllTRztWOUXA8kS/ggEq2ViJ1tcAVmnRWoxL0+hwa6ho0CMSitgOv0uIc+FVM7mxS2q6V3o4D7ao2JlMX0tzX0UflUat/FxD4YJsyZKGNdxSWgBEgGrRk7Sd1IwzcF4AkbtiXN9s4iZCIQSFsgx2WNAzICAXSaNDFjMmZ2IAbZOso/urmMwlLTYt5jgAm5KdeC2Gg+7jqk7UmXIUzUsvKYQv66EZbcjBrSYlFiWP8BOZ5TfyBhuvhFfccu0LXOdfCF1GFTxbdD7LMBnIH5thdns7uOokR16uNXItOEY9FXxdfU/GhW4RtazvxkCEJjQl52lC8ept+G4zf3PtYSXEMOTYfUIIq1MJO3PLxuDrM10eWF+G/eNRvitiK+KfOJ4F5w69O6+juTt2Z7Ebot8OJ9a313JviBMPW8mI5wD/iX53RAyMdgHwitrCxpayyS54bFY4wGrVjEjZ9NJwh9AOgo12FSW5pFUz2Af1FpnQRHsm1CW6kIMMm7Lw4U1iz07rQdPogWEz3H9bOByaTPsHqR6BL3uYyyFrVinYQ5Xo8bGOinP9mEBedw643TaUb2Vy3oTRTum2z9/PY30Z+OaQtVgNVhzurOAf6ANcd4Mq18eR8pNWMHVzTvgac8m5jF5Junj4/RkxCnU45H03xTD7v/uf//pGTcPu/Zy9i8wubahZAkAAA==","6f8dc01e6e3064c9c02bb111ba9f74b31c361d58c70cd88e5c85e37327bd718e",2404], - ["pages/data-persistence/kv-queues-cron/index.md","H4sIAAAAAAAA/7V863IbyZXmfzzFsXrDuGyhoHa7e8NkSzItUW6aksgmKWm9mh5VoioBZLOQWZ2ZBQq6ROyvfYCNjdhHmf+zbzJPsnHOyawLCF3c7RnH2BSqKuvkuX7nkvUVnL5I4Jda1tLB7yG3Rg8GVyvloBJLCfJNVQqlHTyT/jK3qvJDB35lpQSlvVxa4ZXRUIn8WiylO4Dsz1p6R3fOrjcZLIyFXOQrCUIX4KRzeL/zwsukdzORwPcvlJVToYvpwtil9HBj7DU93r0fKeXbvVrLaWHVRmr42cxdCsciXyHpxkkHRkvw20oWkBvtrcg9mI204ORGWlHCXOTXUheO3jCZiNqbaSG9zL2bTMCvJMyl8yA2QpViXkoQhai8tLCwZk3Xj1ylrASpN8oavZbap3AEb6U109zohVpCaXJRwkKUJb4NSumRixKcWEvITSHB1hqMBgGlqLypiBilobKmqHPkcQI3yq9M7WFuhc5XSi/xgTsKV1IuvMLYQM2DO53H6RXpYPC7V0ckYXD5ShZ1KYnVDgR4lV+DXwkPUgdlIK4b7Q0IXGejCmmnYqmN8ypnjTkk5hoLa2MlPSAtlMp5qaV1UFihNO2T7iaChBVlKUu4WSlkJZy+AOfx6ZUpCwd5bZ2xLoFCFnUl4VpuXULMyE2tPa5KVF5LWdHKpTEVqEKuK+OR8T+NVt5X7mA2syq/3pZi7tKl8qt6nioza9VHOCe9mxVKLK1YO1a/KW9hGtljU7dZjgeDydVKgrfK4BZwy/jWA2Ylce4//tf/joyjv5tt8k8tS/AibVn4uGXcBW/7E7tOJ4PBV1/By5XKV1BZtVZebSR4A7WTg8Fkcq7ya1QJXM2q5cq3d00mg8F7eIaq9u//BlfbSsJ7eCSZEWiN7wfvYTqdQvjvwXvITuV2uhFlLZnUbNes38PlVucra7Sp3XQhZYkvt1IUsxurfHjsgE3fJdHwXQKLUixd0mwsiSJP4bmTcHX1hGxaViu5RvNUfkuMuBEeV6KLVorcq43y25SpfbzHYWR7nAtuOzd1VUpwpblhFW/M2MpfarT0SvhVCo+FBrQ2b2Bdl17hQ7nRrl5L6w7RhNk3sPeAldBFiWt4q9B4VtJK0AL5f4G/iRLQUm0tU3hsatu4nQN4JLUhL3whC+USOHr643kC58b5pZWXPz4Jm7zquLlbG2Rv+B4uah1Isej5QDSmnsK5UJacCIigl97AQnRvKlidcd8KLX8t9LZVZnauRMwlindRo8NBsXiOA1lW1JZ8pBNLgfS8XAVqwVVCO3BeVo6JyI21suT4waq+rqR29EPSkKgcaINeydTLFfzH//w/sDaFLEF5EOi4ui8EpZ2XokAaB4+NBaXnptYF/HB1dR78hsqvHZjFgohK0HpAwCtv1XIp7Zc6j/BScheL0ty4WVjAzcYoXgtr6ZxYRnklUBq9nNpaa3LbXZ4xM0JADOR0N/XracKnkSDyHGdawtH5ScIijco3GFCoDOGbvOregMk0hoAZ4p+DuVwpDFOefVUMEuBkKSlmgZXOlBs022hkvSCJbnUtvFVvyEU6wDc01CXsGkUl5qpUHs1KeXB1VRnrOVyvzA3+phzkK+OkTgeDv9Rl+fRH/EmwQU1puaJxxXMr7JYpfihLfB8afoiDc2swhrk6X6F+XYj5XPmnP2KMo8UOMQLMDaqvQHBBSqX0wgrnbZ372kpY185TPJ/LBUY23MVCWefRgJr4mkKWWym8/BH/ORpnqAO4L6NDLA9+gZmvjaeQ7w3zGb0pb2JtNuECUUikRrKRC7rLc8hXQi9lQkYlGHrwTylGkaMAbf7f/235viWrC2yXRR8voQD8Sm5BWBnkLotfE3DCZrPseoMPsbD+/d8oxrLjXoi69D1UFfFU2qCIAIPK7WEHdtQoKHbFDU/5iqlcAtbUHhko0RKC3cJ8G27RuAtn6H+DDpGI0EgK4cVcOAka9TZGB17I+JW0QxcXdDBiEIByEmUJZoEUrtkwcLHKmlw6dwCiIeIGdZopWAn0gw2QwGctOUcrp0GjChCwkDcEhx3ZjoZCimJaSu+lxesOhrXG/aKnGI4PAxhcCeRcg0eJhVPW2GqlSuNMtdrCRgliX4oPjcYpPDMg33hptSjBSbtROcdQZSW54PeQreXa2G0UapTlOW91Si9K4SnddLqJyoch/vQFjMiO5hx1ybhkQfmIypUvt2PSPyRa6Sm/h7cT4fko+qMDGPL14ZhUGcG9XSutCMhSBiGdd+yOTl9AEVQNr7AlFnIjS1ORAUUUzsamXKNSowDyVSnHbF994gJdgTP8+Az+KqyWvqf2gUUB93eyJ9bIYH8pXAZzQ7Ch4eHRwx+OX59fnL04eXR8cc/i+v/yfknLJ3Bx/Ojk8vXzi5PZX48unh1f4Z9Jmy1E+bnXr+nB168ns/YnXuT16wlZeWWlI+/dcAD3GZnG1IZVG0/lDsNG4+3smTHkC68oqSpJk8hIUdOmU1qJnbcugGmIzIv+bYRYaZxlLb4LkTP4cRThSi1XU7+yCCCq2icM56YRywWbTuEhRZDgqedb6CSCFKaRyWFfVoYQBJYoWf8SGZjCyTr4SdzFLgSdxdg5E+tfKiKvUFbmHkSOJhH212K/PVu7/PFJP6SNHp9dwPPzR0dXx3B5enIOT84enh4/Amtupnkp1DoBlEJw5ugeDLJB5NczLfLrBB49+XGcwvF/P39y8vDkahp1CM6ePfk7O3+J6US0JgpY5+GfaaDVHQYv2GEbuoGH5KoxLOZGa+ZlWoVn0ne1LRPyRhgtPoTtn76YVqbEjKKz/dPNOf8W3QQSVjt0a7U3a+EVOX6WE6pYtEuE8wwHrFwbLwkNzvC/LkHqojJKexidvkBSkcBx0kPvbagg7a91UwhI4arWJBIUdWd7hdTmdJO+w02cYJazEWXSiuAqSGAt3lBqIF3ceMe9Zln0ls+YDFtrFF0D1S0lTAVqKuOIy3hhNCYmkDQorwlPxoJI5y2HFEMd+RSMTHs9FseyG+UkAev78Op3L48unp08++tPg/vkNIlFQ7drMsYiGlQOJpPGYjEBZseBfwUZTSaHCOOYZEYRrpcA4UuiojUpF8k0QuXG62f7rSFrzUHp5ecsohM3GViME5jXHramZnhnKoR9DM2yz5lGho6Wwqu3iDSyYTSAYYYwcI9h1LbMcIOGABOCbl3skRX7o2njZ2XRsun5xckhZI1xZdFFO3w4CxjjNdsXJ9/Ks/WwxnONwq1MXRZQogqGYg6BBG+FdpiGE3UNEhIOeWShMGus/eDvKavMydPzs4uro2dXqDRHfUWpq6UVBWKQAGJnIVrgrlsMrsm8sW7kCRlhdlHbHIG7lTCZhPxqMkkQsRGc3sm12Lmz1HIEROTOf//7+Bc9k6XwMgR55m+CbiSiDWYOiqKRRB+uUy5LlaGP2NMhs4iwNwbQQmpyXiAVGhrciC0Gdwmvzm4nikzRlyaFVL5FP2b0jLcY88GnQoulLGJYJjMicxgMskfCi3PhV1lji5W0jqBnLiloCUwju89TsD4YDLIs+9kZPXg3ALjT1IzvHAD+AHDnId7X/BPgDsf1zi8Ad47JqxZ3DqheknQvLJXGx+/wa+90rj01BV1B1er+HjeD11LUx1l4ZbjlA/3vB37izrlVa2G3kcxI3gDv+4C7w8wmxjQWS4jM7Ev2ZTYdbnp4D0dnj1lHVjK/5uiDmNH1bIwznBpROwMEY7eHrJ51BVbmWMJ24MQG7xBe0JJWVqXYMsI6Onuc7hJg1sojNHkPJ1Epk7bUxkI8ZA0vqNqTcF2FXQFZgMLyF9lbadCUYrRF/COxNMdZLEUKaiOIqioxdfcG+eaF0tImcPxG5pyG0LIoNChFrblaqHRe1gW6SgGPTI4l2VI6F5WtzfuaFR2ssaDIUDQyDISHbIbcyQ5BNm/E1obDWoKL9QnMY5vCX8RZWGfi3PGxKuWlt1KsKRhgauaVJFfu0PqXbleg83qxIPlgasVEoYwKiRgRA2WplqEGQgUYkV9rc1PKAk2KC6g3AgkkY9Og1mtZKBQCas+irN0qQRdMPCoFGuY2lEbZ7bX2nEYAkV0evTjOZtlf/kp/BNTgOmSHbSAoR6MPWnQIXlzL3m2VtMoUAXAJx5tbWnPjMCtAiewIgtSEtDNnqeZRcCkcaTiOaSRp4NBFbTI3muXU9T951/rYmZ2+GAyypfSnmxFffTDOsBRbW40YIXuJxWMU6ukmAy4JBufcQ6ttanUqt6zkXAfzWDJ2CVBFnC8satw7XeYyTHa6uZT+jOK1S+UbdLYnOoPRWpWlcjI3unBj9AFIUiXt9Fpuqd5tdKh7NRniOeqngOwS1ac43bDDyZBotHurCrmv2tYEuJBMYEWnvwSBZrpnyvc067GqMQ/Hv6aAE+nJstNNREAPEL8+NjbHqqarZK4WKm8oZ1gXE8chimKIJdyKQs5OsTBYmCEEu6cp14iOiUF3lzHcIiJ6yYCx8PwiltUp331uy97tx6HMEKBqi9B2H0QeuUrksvf4uZUL9YazE2Rrrw6JzZ2whLtW1SUnjo+UI5++zbK5MaUUmla6vFZVlw1TRFmdUmuDocjhTiZdJSRhR0Vjp8Jw0v0a+TYanel6PQ+ypZ6EN9MYurq6ziU4fDeVqdYG/axYIKTGCzdKF+YGZCkqh+n3D9hKwng33+6aA4WSgDvXhvqWonSmSWiyvDROouJmCcrFsRZnMGqKO+ME0EFciJtwBUmodSFtuSVITkkRuocmojU8bms8YZUj7D41MJ/fqtzp5kQrr0Sp3soC3xERY6Zc44Dwd3QasKyFLVICTN4NZjNAvJx6N1BUP4B3bIvwgSuEw17/bXg4GFBVDK43cA8Exopou4cwm0FFbY4elKbNNoBWli4U+uLeBgNe5nqTOulHr4a1k9YNExiKEoPWTwm8o1rkAQyP6BcsnZb4T1GslR7Ch3FDltTebhvKrjfpUvrv4+NsKIfh6ZATfbi/55XjQ1rPlDItzXJEqz5IyQunuBZv9g6Rc2eAXDzv+FXshFCttqdOZBWszOiUwkAC6yXWYVemtukuL2L7Emnz5vq1mOfMEKT4pDiAYf31ED7gL9FODuCb19/dvfv67t27xBnCjo20qZ05pYbaP0PmtPcAUXi3XNQPUscGPnqkLrLS8qaMpYPglAaENmjhUZDjBl2vWSAniGT2bKNXQyQduUE5PiagGE2MHf40HhOU78ot+y/vaKH0Wm7Tn43So+FsOP4A0/sQr6BNfMgSfiFLeHwYEXfDtdA0/iex7BOqyi4OVTL2qXGvmGN716ilR7SFoavzsPBmrfLRAODVO2TqAexfIgHE76hSXqyrA6biQdr9ER48AF2XJXz4CZOTV+/IcRzA0Ek/TD6zODLwAHoWg+vdHcN/ha9pReSAWsDod7yJ1Fyz4GYzOELp5bW1VO+mwFEKXcgCnEIExs1yUZApYcd7yznLNi9lekto15sUXXDq3e3qexLbHtwLCKMgbUm+FfNuk+BLBK7lze5zKPgd28Y3Ifeu5RZZNyRmDYNNnVmolA4woG0+UMWP0p56zvVFpnPEHVRlCBxQH90suH4GSyuqlRt39kSrfmZLM1qp3Rj98zTurr/C6B3UtjyAId10MJsRUzG1Ofjum//2p2FCDpjgCvZEtqKq2GszR8LSgS0MxRMYLqTAtmaBzFFerl3fy3234+K++oqrYG4w6DY4v7+6P8K3JxGD7ED0p1yPivdmPKP1CYiOiGAhOL1wHrNe106kMPYI8xQ73QKKy21xsW1nl0YUUIq3qty2UJo68ji55sU2ypPKNwGIkFQBK6wltdzS0LrBBTE/giy050ah4tZlQNJOlISbucE3CiMc/Xspy3DeVAgiaLaKEzYrSxma0b0qO7Eo65UkM5C6XrMacF+6P4pCMOh0g4CGmEd/BPYxoIm9CZdRekGr9/AmDlZRetURPszCvxBqFv2fzsN0Cf362xKP3lYZiCucbml05mytaKCHVSogbhh9ujg9xmKc/0hpl1IyqgWHhDL6iClVK5vb+s0/fH8E/C3eh3tU76L8g/sKobzbTYMi1VtUNGWs8tuDVsPvB/LvR+KbXMXb7ZH3cl151yB4uAff4NueijfN4BLNXwqFo0BN6xrlStuJtZwGI7fTT9SeDh0Telv3zY9kKbbd13599+5dTstKsYW59DeSCgQYSkQgczefaMn4B17fmkNQj9DkUUYHrQ0JW5hvbJLU9sGDrk29495OQr4yid2/JNbtP9AoQngH+JrGfTA5cgGJ3eoSwajXJYLbbSJo+0TjsCtsTzyh7sQlBswse9T/4dxYT/t6WDuPgzcUcbGpL1SJvTiKsylPZ4WiYNI6QvR5fiWVbec+G77EfktI2yNByuGvWMa7slTj6an1QpRONulsdOrUswO8P/qEGyuqCivgSBf2WhC2YhgwcEPleZohm29pxI3ywmzXr4Qgg12ytegFG1wcQ83/MEW4TA5tI0pVUCEMM4Uww4kXChwHxHG9E+8ga98Q9CYDURSQxcfP9DE/m8EoljTQmjFlbO95JPfew57V6Bd8ozL62FpjMxhlQzR4YYthhvsOTyWQDYvylyE1lrIhNrdvhtm4GSnq+dSM6ned+Y7AE6jKmkJvA/bybcYQu5uXBpZM5Vqosoe8uw5+D3bhpiDDqE6/xx3cCsh7UtMQuunv6MoiYEdCHNzrvv/7d+BNm1nOTbHtZpZDfoZAHWMd/iGNsRmBLy5AGeif5RuxrkqZ5mY9ROjNyw1fyjI32Poy7TR8OkwGlPfOZsGZhZIUtwNGMl2mIODb6Vrp2qNvWivc7ZgQZFCZWDT8CG0foyzSdelVSeDDygchCS2QlAP4Fibw3V2YkL/dk4UG9PEbZPtPEclsRlAlgB7sbkhymaixcRYKLURAQIlY9pdvPERMdVJgyxb9rN0+RAVue7njHa4GgCUQLDbPEw7DFcdw7z5lQfyMk7o4xufijak3SSQpxS2NDzllukTFaCfj8usZBzMOSeUWRuxM+hPB9+5xyB+njRKd4TxW7Qtzo5MwQK/0dFHSSHdzECJAvl4Q7ISW/pYZMu7KPk713i5C7PEhn7PvMNEFdYXW8Uc4mZ1Nuc7egIg2pWR4LTCZXJbtcHy0b6SnUaUeFd+/45ECLLh0NIlLEKEg8a59Ub49gD9yVYr5QTvdqwA7cl/U5UKVZ7huI/nwZi45+ZXE/nONwmk4mTTbFzhbia2yRqwdouD7e/A1zs4K1HH5RuTIkVJhj6U1Hx4jfnj+PPCRRB/C4Us5h5d0UsGlO1KlhsRUG4/BmiJJX7ZvG0m+NcXwcFfoHXz+eat/1nkLDp+sBdyDt6mZ/yxzzy6VChZvU0w6Rq/Y5DGldGusoqBrRRfxNmVhjugXdhTd3yIPTzQFUqjEFlM1R5Ob5I5Fb2AjHjahWXDM1+YSVSUEUllERetxqdG4DpQY9u4YJnt2nJDW3IrcB0DxmWnfsbvllF3BR9xt0s80Pmt5bSKLo/K3R7AYNvO4SlvA2GtsX25kA/jsPFgSO/CzGaKUzjBmM/zCZcIGZvMEQLzaTgTMZljW6E3DUBvrH52GScN6XCNp3jObiao6cDK30v+5rZh8+8dv/oBXCAHg/zWTNDRP2pmeGTZbZWIbaEdAGfFoO2LwIahzxyG1cb7hO3L79dd/+IarMzv6ozTvEJn9z9Eh7tBx5mjlUtiCuu1mATc43r47CEkDQdhh+LgSjWJ9+N1H9YRf14Q9i9kSB7TYq5tBOwjYHrqj6ZMwydfJteIkX6Qp3PERqvZqHidkB0iz8KsDGMYZm+tN2gde3ZTtAL69e3dP3tZWxjpyJ1HiNNSzs6vjnzj/8rKUawILWGLk0Zfk1uEAzl7aASSeYeoItFkmo+mnbiYTB9M5fcIUS+DZmO7ZxUK41dwIW2BfFdvLF+cP6dnLy2OKPVwE2ZPo8ZBO9rEcjeaYrpot/h7HJZZKL790gsnMcRYg4Ol2k3GM6SGdV812JyD31xjx5uaejM9udA8H9OYv2yrD7vwkHj1CNKo/PSqZxUnNkSoSfNhyEymJQHenFLpEYIKzNXRMhAyWJm90gWKhibJ2EXSpYUfn1PB0mLph4yC0ypuTNxQes7jt4m9mjjOH4bzSSBXjrDkEiu9dC13jWAcOweJZw5UssaCNauq8O4Ss1p19EeHY2aVTQ4dtedJLQUc1O/OqgJiWYQ0jv/aYJeqk495qZ4q1KzsapKH+w7KU3mgY0at27qbhCuXHaS+fWFOJieqAVkL2NzP/ga9kkYsM/mtRdq+RejS5iKAHw52YmP9s5icF1kebs3No+PiDyHGp5l9czqI71VKLMkPVxQwAqVqJjcKDap30nLoflPmwZuliv35Gx4onidoSUcjgXVPQazvuitvTDV+w4ky+oqm4IX+4SCcLyO6maXfd2M5u0rQaJeNhjaNTHQqww5TxKZT+hIdydHonUtZOWWYYxM1iwfXwOPcWN0AoO84/IXIZTCYP+/b+qQJ0cxPMsPfX/PO3lZrZ8G5VmtvRln6pmYInOsXoS5DSjjTZ2TTTLKHw8anzI3hOM4agbmX1u7uhtIrT8nRGH++4VUvdVbI4mUr1VZpbIYq6MxDxEE2YNImM3B02QScSTILOOv76iRNk1Fuj29EauAfD51cPh7i9k6NnRxDv4M7HjofEaIq9POpviyW2S3wCVJAZHq2lVbmYPZM3r/9u7PUwVqtrfaZxjKM/hYOnaruqb7AL2tFJzGfZgfNUGvbZsTyoaGsBnbYHcXmehiP93vL/y5WkliE+Rvzkk4HhEZyx42ARfHYIyhiScFbNagxReIxQtPoSTSz7C/9xiaTK5Zb2xzKkg05YwqpMqfJ+WV4hIF+oNwhNMA5qnFnGejWekVJaCpty/YudSH8OaC3ecG0sRwwT5hPMuqpRMvTKeFysdTcdlSZ1vui5s+Cx4G4KV8bv+DQmIXqmvmNqzqV5QXOh2YXMjS2+Zw1LoNY4iKnvE1uO7Fx5PC0K8X4unBeMvqJ0qD1Y2wXWsZHtOLDYjbc0rmvbU8mhMkT7gwyrvLiS0XjYCS0vtEFoo3FfccOZhSl8nY0TFgalP3SamaZFs+Diud/Sk9S+O2CCh+zobLuSFucD/jD+1xG9IfYbWbYfedgikojCzZLoMuh4WMLC5lEBmpKgY/0ke9avtIclwrxUAyDEHLMaxliVNZipud5uuUZv6KAdz7wKPCN7m2fpYPBELSRNJ4STWzS5Sz1jsVxixuM57MbzaGFwlkZz26PxeKyb0BEovTFcBYC64uZBD2GlttZUh8zYVSCpco0sjOUeclqhK8NDLweAkOKi1lkQPf2Kb3M1jWqSreE9oTfQBnZWmHld4LcPaE5hJWqHxmVswzMcxUaeov/AqEzLDx3Ny2EjJ+KUiEUa6ujELWvmDmfxMDTeFkaTQ3PJ0VaxH9ffIM6jNllsIVS5nbI09mSxTZROoANx9+Wx6PXbklSLKGPi10GIVLnrnAPeH3vbQQui6B6ESuFOiXCJJUvh5SPcyAXdizXWD01u3/mGSNTxYXfXw97W0pfHx6ePjv5++fpPR0+T8HJOVGOQO9gTttriGH4Cg5XN4LHfNapcOCGHn2lA7d5IqxY07Xij0NultyiNWUGf0GZApLOlvSVlHJ0rt9Mcw1ldfUqsv02S3aJCRCUJfbmiTcq/Dg0PpOsHogsCXTwbEgr+aF+3+g4d+BIdfywrhDVatdhtHHTn3sK1lFKFpscQLa39oZdBpN6cXJ5dhurnZ1UqEIRVVWz00H+GSaTzlog4nZYfK/r/JwqoLfUwCXtcIOLHjm/lacB4doBDuMlzHM9sOWE0VXUuaj1MYETL7JUFZm/NtCEJ5AM7VolzbdiPCBE23tN3ix8ylMT48PabyRt//N0SL+99e5iy+IJXx7HI8DstGcjpSzc2mqcoqp54laMKNXqc404lohLWyf6Pn5I8zgvuXWk0vAt/IuX7evrtsDMCiodqLIaOe/vetfPY4Y7M+NEHKfqVQGvxIC3E9mzxUsrrdkD0q6/gh/ghCozhBr+JhFOM6zV9nAzbAppLPThlRd8vww8tTCa4s8mEYNwNdcl738wJnwSLn+yYTKiI4PHzDvEDMpMJXZtMOqNcYaax/0WJXqOIu5mTCR597X3/6iMfg8JAjtFmWuI3nb7wW1ggcmuca89kDQaneCsXBpsuiluLsjxoC4ZATSY8atN+xWC+9ZiwHxGj6KW5sIzHwYobOsCGNKo1fbeCBq+bmbj4Khyfo0NC3bZl+4kM4L2NYvmX8GM8cePhuz/CqfrLmHlHH7fgEyKh2+OdLBdkU/j9t2YvtLlwxHKhEE8wt5tNxY/Y4Cg4HcTgF2BzLsYJ/kzAQqLc6ZsdNDWGXw2iw0Hx0xeYenxpVTN+W2HqiusZ/n8oWYXPMtFYDmbiXFqjwjZ9KyvWYKhQFNm6M+feHI9mDvH3dbha2vmqhDYeRzSnu7NhjMI7U1s09LWFyQQJmkxuD1+VAs8a9GevHuJhMcj29Lyz+FUc4VyNR7HjYVg6noTHYTXIxULmPh1M6cROZ6Jf4cmweH63PWK1U9xgteUPXWkQc2dK6hNiRQoHq3Fh9EOdCkP/bPTzq4cZ17uxid4O/SKImtdOaRxYRqfUQAXpcNGj8gZzimw/Wgqs5YvMmXjBtE1/rMftNPw7o55t08IBHXphuT7DYQj6tNZggAe5uT3dfACOmErHjZu89RXe1n75MXxszhr9xZ+bEl5MO8fyZitzM/UmfEfvekMhaDZGUKUqJHMKrx7S8a3m014RI/zWV/KpsKngvmrTHJuND/qDnnjWNZ7raoqnjUl1GqlxXnQwHQxeXW++lL7Gb+LY9hjrXK+Ion/8+V/oWyG4xGDw6h+RSrsEs38woCFfPEVwdH7SenaMBdP/nL7MAX0Tqm0+tTiRO0P48T7Wwd3WUzr4/wZg0gKNVAAA","d751c3f107e9558014eb82f0a215db933f50f307ae0fa48abd552367a0f1bd64",21645], + ["pages/data-persistence/kv-queues-cron/index.md","H4sIAAAAAAAA/7V865Ibx5XmfzzFMbVhXBYoUJatDXeLklskZfW0RLbYLXG9XC0rUZUA0l2ohDKz0IRIRuyvfYCNjdhHmf+zbzJPsvGdk1lVAJuSLHnGMXazUJV57vfMD+jiuyn90OhGe/o9Fc7Wg8H12njaqpUm/WpbKVN7eqLDVeHMNgw9hbXTmkwd9MqpYGxNW1XcqJX2J5T/pdbB85vzm11OS+uoUMVak6pL8tp7vO+DCnp68DKDIO8vjdMzVZezpXUrHejWuhv+vP8+IJXXg9noWenMTtf0d7vwGT1WxRqgW6892VpT2G91SYWtg1NFILvTjrzeaacqWqjiRtel5x0mE9UEOyt10EXwkwmFtaaF9oHUTplKLSpNqlTboB0tnd3w72d+a5wmXe+Ms/VG1yGjM/pROzsrbL00K6psoSpaqqrCblTpACpq8mqjqbClJtfUZGtSVKltsFsGxtS0dbZsCtB4SrcmrG0TaOFUXaxNvcIH9wxWMj5uYV2E5rN7vc95i2ww+N2LM+Yw+WKty6bSTGpPioIpbiisVSBdR2Fgqts6WFJYZ2dK7WZqVVsfTCESc8rEtY421mn+QDuqjA+61s5T6ZSpGU9+mwFSTlWVruh2bUBKuviOfMDXa1uVnorGeev8lEpdNltNN3rvp0yMwjZ1wKoM5Y3WW165snZLptSbrQ0g/PejdQhbfzKfO1Pc7Cu18NnKhHWzyIydd+KjvNfBz0ujVk5tvIjfTFCYJfK4zO9W48Fgcr3WFJyxQAEoY9cTISVT7t//1/9OhOO/WzTlUUcS/Mgoq5BQBhaC9k9gnU0Ggw8+oOdrU6xp68zGBLPTFCw1Xg8Gk8mlKW4gEljNmdU6dG9NJoPBG3oCUfu3f6Xr/VbTG3qkhRDQxjeDNzSbzSj+9+AN5Rd6P9upqtECan6s1m/oal8Xa2dr2/jZUusKmzutyvmtMyF+diKq76dJ8f2UlpVa+WmL2DSxPKNvvabr669Yp/V2rTdQTxP2TIhbFbAS/+i0KoLZmbDPBNov7jAY+R3GBWgXttlWmnxlb0XEWzV2+ocGmr5VYZ3RF6omaFuwtGmqYPBRYWvfbLTzp1BhsQ1iPWit6rLCGsEZKM9aO021Av2f4ZmqCJrqGp3RF7Zxrdk5oUe6tmyFn+nS+Cmdff3N5ZQurQ8rp6+++Soied0zc+8gKNbwDT1r6giKg+Uj1ap6RpfKODYipKJcBktL1X+pFHEG3gaav1H1vhNmMa4MzBXYu2xgcMCWIH4gz8vGsY30aqUAz/N1hJb8VtWefNBbL0AU1jldif8QUd9sde35wbQF0XiqLaySbVZr+vf/+X9oY0tdkQmkYLj6G5KpfdCqBIyDL6wjUy9sU5f05fX1ZbQbprjxZJdLBmoK7SFFL4Izq5V2v9R4xE3ZXCwre+vncQE/H4O9jjbae7VK/JpSZevVzDV1zWa7TzMhRnSIEZw+Ur8eJnwNgNhyPK01nV2eT4WlSfgGA3aV0X2zVb3TYQqM0WFG/+dpodcGbiqIrUpOgryuNPssctrbage1TUp24CRhVjcqOPOKTaQn7NBCNxXTqLZqYSoToFYmkG+2W+uCuOu1vcUz46lYW6/rbDD4vKmqr7/BIyUKNePlytYUL5xye4H4oa6wHxQ/+sGFs/BhvinWkK9narEw4etv4ON4sVN4gIWF+CoEFyxUpl465YNritA4TZvGB/bnC72EZwMWS+N8gAK1/jWjvHBaBf0N/jka55AB4GXr6MujXRDi1zawyw9W6AxrKkhs7C7+wBAyqAlsUKHu05yKtapXespKpST0kEcZvMhZDG3+3//t6L5nrYtk1+VhvAQGhLXek3I68l2Xv8bhRGTz/GaHj4RZ//av7GPFcC9VU4WDqCrFU1kbRcQwqNqf9sKOBowSU9zSVH6xWz8lZ5sAAmpoQtRbWuzjKzWw8Jb/N8oQswhKUqqgFsprqiG3yTvIQjastRv6tKCnkQQB4JOqKrJLQLgRxcBiW2cL7f0JqRaIW8i0QLBWsINtIIFvHRtHp2dRokpStNS3HA571p2aSq3KWaVD0A6/exo2NfCFpRiOT2MwuFagXBuPMglnIrHbtamst9v1nnZGMfkyfDQaZ/TEkn4VtKtVRV67nSnEhxqn2QS/oXyjN9btE1MTLy8F1RlvlNHX/NLFLgkfXPzFdzRiPVqI12Xl0iXnI6YwodqPWf4AtKlnso+gk8LzUbJHJzSU34djFmUE925jasOBLGcQ2gcv5ujiOyqjqOEX0cRS73Rlt6xAKQoXZTO+FalRDPJNpceiX4fARbgiZeTzOf1VuVqHA7GPJIpxfy97EomM+pfRVVQ3BBs1PTx7+OXjl5fPnn53/ujxswcO6//3NytefkrPHj86v3r57bPz+V/Pnj15fI0/p122kPjnX77kD1++nMy7R7LIy5cT1vKt056td0sB4JmIJtDGVVtL5U8joul1scxw+SoYTqoqliRWUkjabMYrifGuSxIYEvGSfRshVhrneRffRc8Z7ThYuDar9SysHQKIbROmEs7NUiwXdTqjh+xBoqVe7KmXCLKbBpEjXk5HF0SOIdn8kAiY0fkm2klgcRyCzpPvnKvND1sGrzROF4FUAZWI+HWx3x2oXX3z1aFLG33x9Bl9e/no7PoxXV2cX9JXTx9ePH5Ezt7OikqZzZTAhWjMYR4syKCKm3mtipspPfrqm3FGj//r5VfnD8+vZ0mG6OmTr/4mxl8jnUjaxA7rMv4zi7D602gFe2SDGXjIphpusbB1LbTMtvGb7HXjqilbI3iLtxH9i+9mW1sho+ihf7G7lGfJTACwxsOsNcFuVDBs+IVPELGklwjnJRxwemOD5mhwjv+6Il2XW2vqQKOL7wAqABxPD6L3zlWw9Dd1WwjI6LqpmSVgdQ+9Utf2Ype9BhLnyHJ2qpp2LLiOHNioV5waaJ8Q75nXPE/W8omA4ZoarGtDdccJUwlJlTjiKv0wGjMRmBuc18QvU0Gkt8sp+1DPNgWe6U6LJb7s1njNgfWn9OJ3z8+ePTl/8tfvB5+y0WQSDf2xyliHaNB4mkxajUUCLIYDf0UeTSanCOMEZIki/EEChE2SoLUpF/M0hcqt1c/v1oa8UwdTr35OI3p+UwKL8ZQWTaC9bSS8s1uEfRKa5T+nGjkMLbvX4BBp5MOkAMMcYeAditG4KgeClgMmBN11eQevxB7NWjury45M3z47P6W8Va48mWiPj/MYY7wU/ZLk2wTRHpF4qVH4tW2qkiqIYCzmcJAQnKo90nCGro2ElAeNHJV2g9oPnmciMudfXz59dn325BpCc3YoKM125VSJGCQGsfPoLYB1F4PXrN6oGwWOjJBdNK5A4O40TSYxv5pMpojYOJw+yrXEuAvXCgREbM5///v0F3+TZ/Q8Onmh7xRmJEUbQhywouXEYbjOuSxXht6jT6dCIo694UBLXbPxIm2gaHSr9nDuml48fTdRFIh+aVLI5VvYMVvPBcWUD36tarXSZXLLrEasDoNB/kgFdanCOm91caud59Cz0Oy0FNLI/vfsrE8GgzzP/+5tPXg9ILrX1ozvnRAeEN17iPfafxLdE7/ee0J07zFb1fLeCddLpv0fVqbG5/dk23u93762Jf8C0eo/T8jgtwzyOI9bxlfe8v++lS/uXTqzUW6fwEzgDfDeW2CHzCb5NGFL9MxiS+7KbHrUDPSGzp5+ITKy1sWNeB/EjP5AxyTDaRC1S4Bg3f5UxLPZktMFStievNrhDRUUL+n0tlJ7ibDOnn6RHQNgNyYgNHlD50kop12pTZh4KhJecrVnKnUVMQWsAQblL9a3ykKVkrdF/KNRmpMslj0FtxHUdlshdQ8WdAvK1NpN6fErXUgawsuCaVSpppZqoamLqilhKhU9sgVKspX2Pglbl/e1K3raoKAooWgiGKlA+RzUyU9JtzuiteFRS/CpPoE8ti38pTgLdSbJHb8wlb4KTqsNOwOkZsFoNuUe2r/yxwxdNMsl8weplQAFHpUaMSIcZWVWsQbCBRhV3NT2ttIlVEoKqLcKALKy1WQ2G10aMAHSs6wav57CBDONKgXF3MfSqJi9Tp+zFEDkV2ffPc7n+ed/5T9i1OB7YEc0EJRD6aMUnVJQN/rgta12xpYx4FJekFs5e+uRFYAjR4xgMWHpLISrRWJcRmc1PU5pJEvg0Cdpsre18Klvf4q+9okxu/huMJCgYmle6XJ2o/dtwRgUQRXGN4u2ACGIOo16NerPDokdPuKStye1hONHwBh9c18kS11p0K1sWAKA6EptgTgrJcpXsnOhGGetQrsuk1dV2hfSP3OJrVKD8aekBHCyC6YBKqb43Sl2DWHNa4J0TMcaFbM60NoAyX0PCOOJa58o6Vr66D554FLGSlxpfIuZIHvLnQUTdLRqThmpb2naNovKFJRf7GKEbGz92Dnr8oweVtYnKqQQLH6K9yvrdSnviiijbg/ZZgqdklpYx8WXiPbQg+fkzYqlwW62QmsEh9ZtIG8ZfY7ym8aiVFTo44g9QngrBar9O13I4+bhzW7OeW6e0aXTS/NqJlRnujD9IKCmBtOkx8Ji2LPKMUERLu5UlQ0G+UqHi91IXvpsnINRjYOwUf4cy8OsXOxyYUwKDw7ypS65v9B7QUsqsQFNCz9tBclpWjbQPv5ZCoH5xe5Kh6ci4Zl+BXd/Xuc02piqMpH/Y3ghgLTVjvUEHRfLctSvUVzCQirKr2DASub80qxyAA36OFPqu+q9bYgV01nUFA+X4LSN35nJO+16YuyEhuNfU0JM8OT5xS7F4J8hg/rCugJ1db/VhVmaooVcEotUuhiCFUM0EbYc9ByVq6ONt5xD3dEWblknwMDh5hLwMxAH6ah19O2z1NhhSfzWVQevP46FrpgsdTnC8Yegkd+qQh98LlIt+THIelAJR3sxLuFvzPZKShePkuzn+cLaSquaV7q6Mds+GWaI83vF/jaKZ5c/mfSFkJmdBE3cmiQ0/tfwt5XovG42i8hb7ooFO0vBU1/WpQiMvblQurHw9GLu8MOtqUt7S7pSW48C0JdoZkK3F/tjdeBgJmY+G8udc1V526bUeQFLB8HNp+CLFynOadSWF8dTgoF4pm7jLwABBtpVe04KOS2HeWhjqpbGXZUxrnLGdjQJuexq/MXuvDbBqMr8qEvskXKW3PjWAOE5jAatGuVKmK08D34wnxMytiz4geEKFr0WXaS34jKGB+ZzeDoYcF2Wbnb0gBSilaS7pzSf05YbbQfJHCPbplS68rHUnHAbDGSZm13mdRi9GDZeOz+c0lBVCJu+n9Jrroaf0PCMn6B4X+GfqtyYekhvxy1Yug5u30J2s8tWOnySPhdFOY1fx6z87ad3bDk+5fVspbPKrka86mcZW+EMawmy9xicewNQ8bJnV9GL427BgTixVogwwyjFkRiRS3QC1rZx2TEtUgMdsAV781ItCiEIID4vT2jYfDikt3iS9OSEPnr58f37L+/fv8+U4eyl5TY7vBm3dP8ZPGfcY5As2EpIE7mOERJYpH4gVevbKhWvolEacJDAC48iHzm+sUtQgkEWyzZ6MQTooAZXmVACgTexbvj9eMzJZJ9v+X96zQtlN3qf/d2aejScD8dvafYppV+gE2/zqWwoHB6fppyvpVocW/gnkewnRFVMHEQyTUoAV1R5gm/FMiC6hevqfayC3ZhiNCB68RpEPaG7l5gSMkiIVFCb7YlA8VnWf0iffUZ1U1X09nukxy9es+E4oaHXYTj9mcVBwBM60Bisd39M/5k+5BVBAbOk0e8EiczeCOPmczoD91JgLo6jUhzMeoMcQMY1VMmqhJmLvWTN+6LS2TtMu9llMMFZ8O/2f6ap8SbdqDiM1DWFOjYft6l+CcNrfXv8HRh/pNvYCdS70XuQbsjEGkadeupoa+oYBnTtL645c+LdLKTCLXCOpIdvLAcHPMlhl1LBpZVT27Uf93DiVX8GJYmUO8T4nxcJu8MVRq+pcdUJDfmlk/mciYrk+uTjj/7Ln4dTNsAcrqArt1fbrVhtoUhcOpJFksEpDZdaobFegjgm6I0/tHIfH5m4Dz6QOqwfDPot9k+uPx1h92mKQY5C9K+lIprezWVK8CdCdEQESyUJrg+ou/huJkpijzjRc9SvYr/clbe7gYrKqpIq9aOp9l0ozTMhmJ0Map/4yQXEGIgwVwk1/oqbvllsHmJBZOiUxwbxKNZ8+wSYdjNN8WVpMY/iENHhu5xl+GC3CCJ4ui9l0pWO4xAHfR4mUX5QFM9J181GxEAmIw6HoTgMutghoGHi8R+RfDGPa8vqnF7w6gfxJlJCTq96zKd5/BdCzfLw0WWcb+Knvy3xOEBVAnGD+apWZp5uDI+UiUjFiJtGP90eGaMcHN7TXOCUjLsRMaFMNmLG9fL2tcP2M/ZPAX8X79MDrrhy/iGdrdhg6KdBbY4MQTPWmbA/6ST80wj+pwn4NlcJbn8Wgt5sg28jeHpAH2G3r9WrdnSOJ4CVwTBaOzwBvjI6qZrYxsjd/B0PSMSeHe/W3/mRrtS+v+2H9+/fl7SsUnta6HCruUQFV6IimMf5RAfGP7B9pw5RPLoiSpTamLDFCds2Se0+POnr1GvpLk7ZVk5T/3maOkdveRgm7kGh4ToRkiMfI7F3+pQ0OuhT0ruNSuo6leOIFRpkX3F/7AoOM88fHT64tC4wXg8bHzD6xR4XYyXKVOgGs5/NZD4wlqWnnSGEzQtrbVw3edzSJXX8YtqeADIeT1FIvnZcZTwQ66WqvG7T2WTUuWtMeD/ZhFuntlsuh+mau30IW+EGLN1yg4inGBd7HrLkvDA/tivRyaBPu1EHzgaLw9X8N1vGn9mg7VRlSi7FIlOIU8T4ocRAKgZGz4OnvNshyk1OqiwpT58/rR/LtzmNUkkD2oyUsXvnkb7zHbGstv5OXmwrfDTKh1B45cphDrzjV1PKh2X1w5Bbm/kQ4xW3w3zcDrUd2NScK8i9CaNIE9pWDbveNtgr9rmE2P28NJJkpjfKVAeRd9/A3xG7SFtawqhex9GfvOOQ70hNo+vmv5MpSwE7APH0oL//J68p2C6zXNhy388sh/INB3US68iDLPlmBL5YgDPQv+hXCqXPrLCbIUJvWW74XFeFRfPVducxsuF0wHnvfB6NWSxJSUNqpLNVRor+NNuYukElWW8MsB1zBBlFJhUN3wPb+yBLcF0FU3Hw4fRnMQktAcoJ/Ykm9PF9mrC9vSMLjdHHb+DtP4Ul8zmHKjHoQd1fs8mExKZpPGiIohglovGkXwVKMdV5iaEB2Fm3fwgB7qYJxkdUjQGWQrDYfs9xGFYc04NPOQuSb7yuy8f4Lr2YBTtNIGVAaXwqKdMVBKObzSxu5uLMxCVVexqJMTmcSX/wQFz+OGuF6CkmAptQ2tt6Go9wmHq2rPhQQXsUJ4Z8B06w51oOUZaQ8Zj3aa783SLEHTbk5/Q7zhRSs4V2/JHO509nUmdvg4gupZTwWiGZXFXd8Yyk34CnFaUDKD55LUMtKLj0JElKELEg8brbqNif0B+lKiX0YEzvFIAjvi+bammqp1i35XzcWUpOYa0xAdGAOS0lpy36CtO9aNa2bO0BRZ88oA8xva0g4/qVKkCRyqDL16mPDLI/vPw20pFZH93hc72g53xWxmdHXOWGxKy2Ac6aPckhb39sOfmjLYenx0zvxec/r/VPertg/Gmj6AH9mNnF33URxKRyweLHDEnH6IWoPFJKv0EVBaYVJuLHTJg54idiKPrPEg3Pa3aktFV7pGqeZ4fZHKuDkaF03IlPIyBfW2juHIoj1WUStAMqtRLXCyWGB28Mp3dgPGWpecdznxD7Z4H9SO9WMzEF7zG308NM42c1r0tkcVjj3SFACZtlYKorYNypbL9cyQb0sxOJ0zQDMp8jSumNA7fjV1ImbMNsmUFJv3YzKfM5yhoH81jcxvpH57GyuJ7USNp95nO13Z54XTgd/tJVTP70x4/+gF84AsD/tbNcPNHcm98atqgKsG1ox4Ey4tFuyOVtFOeeQer8fEt3UPvlh3/4SKozR/JjasEQxP7nyJB06CRzdHqlXMnzHnZJtzhgcTyKyyNp6DC8X4hGqT78+r1yItu1bs8hWxKHlnp1c+pGUbtjnzz/FGdJe7lWmiVNMMU33gPVnZInCdkJYFZhfULDNOV1s8sOA69+ynZCf7p//468rauM9fjOrMQ83pOn14+/l/wr6EpvOFhAiVGGr6bvHE+R7KUbgZMpuh5D22Vynr/rZzLpaISkT0ixFE5n9U/PlsqvF1a5En1VtJefXT7kb6+uHrPvkSLIHYmejInl78vReJLuukXx9xjYWZl69Utn6GQSI8bTHZJpkO4hn5jOj2dw764x4uX2nVxOD/WPpxxMAHdVhuMJXhx+QzRa//Swbp5mhUemnOJjJ02kaQp0j0qhKwQmmO7ig0qssDz7VZdgC880dovApEaMLrnh6ZG6oXEQW+Xt2S92j3lCu/wXu8DUazwxNzLlOG+PIWPfjaobDHpM40zKWlcoaENMffCnlDd1Dy8GHJ1dPrd22pUng1Z8WLg3MU2IaSWskcivO+gLmfTSW+3NUfd5x6Nc3H9YVTrYmka81dHbPFxhwjg7yCc2XGLiOqDTlP+LXXwpv+SJihL8N6rq/8bi0eYiij+MbyIx/7tdnJeoj7anN6H4eKAKLNX+S8pZ/CYP1uQQXWQAgGqtdgZHJXvpOXc/OPMRyarLu+UzGVacZetKRDGD921Br+u4G2lPt3RBxZltRVtxA32kSKdLyu9nWX/d1M5u07QGnAm0wfBeDwJ0mHI5B3U44WE8nx9LkHVzvjmcuF0upR6eZnwSAhxlpwk8RC6DyeThob7/VAG6fYnm6P21//xtpWZRvHcqzd1oy2GpmZ0njGKyJYC0x00xNu00Syx8/NQJJpwUTi6oX1n9+H4sreK8Rjsf9U4t9VjI0mw011d5boUh6s9ApGNccdIkEfJ42ARGJKoEn7b99RMnINSPtu5Ga+gBDb+9fjgEeudnT84ovSGdjyMLCW+KXh73t9UK7ZIwJS7IDM822plCzZ/o25d/s+5mmKrVTf20xhjH4RQOznX3Rd+iC9qTSeSzYsBlfg99dpQHDaMWo9PuKLjM04inv7P8/3yt48ChZqMcz6bGTzBnKM4i2uzolOGSMKvmargoHGRVnbwkFcs/lz+uAKpe7Rk/4SEftUMJa2srUxyW5Q0Cch7vZFdma0zNo16NU3qm1splUv8SI3I4B7RRr6Q2ViCGifMJdrNtwBneMh1Y7MxNT6RZnJNxicOhYrHofkbXNhzZNAEhWaZDw9SejAyKJ5NzjGy68hORsCk1NUaB60+ZLGduYQLOK1N6XwrnpURfiTvcHmzcEnVskB2zin1/y4OsrjsXHytDjB/lqPJiJVvjuB00L7ZBGNGEV0I4dzSjD/PxVJjB6Q+fp+d55TyaeOm3HHDqrjdogmOefLuC0Q7zAX8Y/48R75D6jcLb93yMEVTwLG0XTQYfUJwKs2VUgKck+GIJ5r3IV3YQS8R5qTaA4NnUGGNtnUWm5g+wlRq95aOeMnWtcEr7XZplg8FXZql5OiGeHeTZce4Zq9UKGU8Qt5tORMbRbR4O7y5nwMUCHB2RqXdWqgDUbKV5cBBhZa6puQ6Zi6kAqHoDEqZyDxut2JWRoZcTQkjxrKnzyHp+it18w6OarGt4J/YGOscuArNoSty+wXMKa9V4KJd1Lc1wGAA0hf2AV+blh57n5dDISXFKikVa6PjMt0jmEWVPD6eoY3PJM6roxx0iiHnUNostlan2M+HGHVls66Wn1Atx78pjYfW7klQXUabErxchcuWudxL9bt/bDVowRA8oVgqPSoQrlCxV0I+AyDN+FzXWt21u37vFJsn4sI/18AC17PnjxxePzv529fLPZ19P4+aSqCYnd3KH2+qKY7iERYTN4uD5BiIXz2jiohBI9047s+Rpx1sDa5e9A2nKCg4BbQdEeijdWVLG6Fy1nxVwZ832p9j62zjZLyqkqGTKd6d0SfmHseEBuL5kuCjCJbMhseAP/Xqn79ALX5LhT2WFuEYnFseNg/7cW/wt41Sh7TEkTeseHGQQWbDnV0+vYvXzZ0UqAoSqKho9/J/hNMH5DoviwYb3Ff3/AxnUlXoEhDtMIOLHnm2VacB0ekVcuC0KjGd2lLA1V3WeNfVwSiNe5k5eIHtrpw2ZIW/FsGrMtaEfET1seufQLL7NwYnx6bs7szV+/958UuLO3eOUxS/YOo1Fxue8ZATnkLup0TwDqw7YazxXqGFxHvcqEVvlvD58+FOcx7zgnSuNhvfpzyx8H87+NOyNgOJYl4PreHDXXkefnR7xTD79LINdibCWn2Wl2j9dPtf6phsQ/eAD+jJdhQIfbnErF6YYNxu+Hg9tgVpKPZiy4hv0cNXHZALMJhMO4265S35wa1O8lC5dGjOZcBGBDwilK4wmE/5tMumNcsWZxsM7TQ4aRdLNnExw+PrgBrb3XEcGRw5vM6twq9gvvI2NVOGs992pwMHgAq9KYbDtongc7DnpCobETSYc9uru0VjsAxL2MyYUb1ooJ/E4OXXLRygBo9nwzSk8eN3OxKWtMD7Hx9T6bcvukhYS3Eap/MvxYzpxE+jjP9KF+XwstOPrVeSESOz2BK+rJesUbiBscWHk4iHfpUE8IdRukUrXKGEUnA9iyAZoziU/IRdVLDX4zrfG8NQY7q3iw0Hp8hWkHr+0qplu95j58maO/48lq3gxGI/lIBOX0hoXtvm2tlSD4UJRIuvRnHt7QF8oJDc8SbW0d69JbQNGNGfHs2EShfemtnjoa0+TCQCaTN4dvqoUzhoczl49xHFFyu/oeefpXiblfYPLANJxbD6ehAPZNenlUhchG8z4xE5vot/gbGI6Qd4dsToqbojYylVrNamFtxX3CVGRwmA1FoYd6lUYDk/nf3v9MJd6N5ro3dAvgqhF402NgWUYpTZU0B6LnlW3yCnyu6OlSFr5USiTfrBd0x/1uKOGf2/Us2taeOJDL8LXJxiG4MvdBgNcJSDt6fYKQiYqH3hv89YXeK27ezRed+hs/YsvPFNBzXoHQ+drezsLNt7keLNjFzQfI6gyW4A5oxcP+fhWe7lcihF+65ZyKmympK/aNsfm45PDQU+ctk7nutriaatSvUZqmhcdzAaDFze7Xwpfazcxtj1GnesFQ/SPf/8D31aDJQaDF/8IV7olhPyDAQ/54hTB2eV5Z9nhC2b/MX2ZE76VrGs+dXGidIZwfaTI4HHrKRv8f7O9XSwPVwAA","2c46b4b15fe1e199d84e96c50be9645c697208e7dc3b83cb0478b3750f1e4c4b",22287], ["pages/design-system/components/alert-dialog/index.md","H4sIAAAAAAAA/51XbW/kthH+rl8x2AN6NnpapWlaFH5LXTso3KRxcXaaFm6AHZGzEmOKVDmjXW9v73f1e39ZQXJfZONyZ98XQRzO+wyHD1/BuaUglwatb4riexSzIFDezU3oUIx3oNMeLI20YAlZSk0sYVCJ1TgjBi3MvRp4WhR359aC8l3vHTnhnw5akZ6PqioYdb+yWPO0MdIO9dT4ypGwCqaXShObxpW8YqGu2stXh/C//8LdbUsdQT0Yqyk8V6cKhEIbDVddb6kjJzko9kNQtNe0kVe+2ysdqaqtr6sOjat6VPfYEFfzQNyWg6kCNYYlrMZeD6bCmNcyJ28q/HBYFK9ewXcxZ32ghaFlUdxd9+TARtqYfcvw3ECRmWTk0Ua8+nrn0elY/a8kZvOUZdDGl4MYa2SV3btyLGhtUcxmsxq5LXZGYDBHqPUjPyNXEvuBsSFAp4EeMCaai+K23QUKYXAM0hqGv9z8I3eStLF3kjXSo4aZwlXX+yCAbpXJTBr6YDoT+41hHnwHKz8E+OEKtAmkxIfVNLks/FCYLP4OBqYbQSF4n2Ve94FQSdV6f8+vj/eMoxOw4/0j9v0nCvr6uCjoIemYD06ltvomh39wCO8KiMeIBe4WaAd6A0zy9/j3E5zufDt4zYJOY9CvD4/3Er4nlwRifzzin6NlGrM6L0bRGwgUPXmsO+sMJENwcHBydqLN4uxkHK4E0zQUvsOa7Onkba6WJksxmgmIEUunk8tIIPBBU/h6AppySxjvTie3saqYo1fonBeoCQanvaPpZLOz0T/WMwHvztPm6buDw9Oz7P/B5DpuZhdITw7fQ3V2UiW/ewg+esOCMvDk7F0O/f1J1Z+dVGeHx8X7XUP+Lfiei+LP5CigkM5VdXm6zTQ5D9orKMuf2Tsoyz6YBQrNwC8opN7k1vQ9aTCP5sYULmJPDG7UstBHYzHNaFyUNQH80hnXgHFCYY6Kjre2USSYehBi6JHjoQh+aFpYthQINCmLgfS0KNZwvXQUYJ1igTXcrnqCNVynvKOFNVzu6wDrYg1lWcIvfos1zEaVTwmawRpm4xZIBJZgXBN/v/ewho+JxvZIfxfbZFy0xupA7jnioz76fCVqxPthDf8k/riKUY++MAEKnSL7IclPGl1gMOgk/U80zXGwMoF/rWEyul0nz1LF5j+5ChPusopOP0+yI20w/dXeW0L3fKkr9bGqfVJHnHAvNrxJ0/XnyHoXxS5adE1O1kF04Qg2Og7h9AwW3uhnKsujKyt6qexF6prPke3J6U2DvSxxhrG2pF8sqCwy/1Jrx0H7La1qj0HHgfUtrWANOTNPB1K0cIt1ddOaufz6Fuuo6WKlLGXkmECBcdB5HWdbZP+GFfapVjlhCV8EYvGBthfXRjj7cuvvyXFR3FCHToyKV6QEVJIuDj6CGQ9pGlcKg569gZnQg1TzZjYtioubG1ADi+/SPKcghuJQn1Mgp0hvoIZdHcGsLB2XdbrGopbRslwaLe2OuLWzW5Q96lTBHTG66G3ZYL+jjQZA2cYL6YM7PNRx+n5oK4a125jvjVnCaLzkDq3dUbtBSD9ejYWiAxZXu3VAbYYnoW1pOZgdmVvUfll2e0buUVH5my+eEn7/hPDlk/VXT9Z/2K1jrE8CyiQJfpTnJZmm3VGnqV8utu2Bm+scxdQJDBfFuVLEvFlCjyIU3BHc/Xh+VZ6/vTqHZjA6tuUeoy+Xy+nyt1MfmurH86sqclXYN9VGmDOAzPixOiyKMjkQffXOruCt97IFvAz9UFujtqghH8spbJ5nNap7HVFBoH8PJhCD80APEhAur/96HFfKeiYwyjvwAfwgbDSVyhp1D9pwZ5jRTqMP+WwFUpTg9eP33NZiPovJHrFwRHqK7DFsBhJsBgxDCiXip3ypTuG870GhtdFnTplGXjnVBu98PPY+3EOg+LRKbBRKv3QRAZXbSWKcsoMmhj8NIt6Bjddtxb1xER/1GOKj4QKdd0ahhfHjdG6sreY+UBMSZktzADCklwlTWJA+zmhP+Qj20mmLAJrHaqpMDihtAoeYUB40EVwaBZYaVCsIuIRAGgJ2/TTPIkjIP06RFF8KDOoVvPryq999ESOMV1KVSqXjG0lMR5CeJzsRt510VTxc5OQY6uCXTKGc+5Dn0qaaGxneNk1NLS6MD9OiuEyv6/083PjVYbiPjyuLLnoW8xLx+2PIm+HtFPJ7LiJdrP0iYtWI3mqCZYuS4DKDeI2rafF/ZA4mgFUQAAA=","c58a0d74f3e8ec5354967b63d84a423426737e05940ecfd10d83993682f0c797",4181], diff --git a/packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts b/packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts index 7b902d1ebf..037ff3a615 100644 --- a/packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts +++ b/packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts @@ -187,11 +187,11 @@ export const EXPORT_SURFACE_ROWS: readonly (readonly [string, string, string, st ["fresh-ui","./tokens","rVhtbxs3Ev4rAyFAUkAN0gL3xck5SGy19V1jG5bh3iEJCmo5KxHmknskV5ZQ3H8/zJC7y5Wo1Af0k7V8GT6ceeaZoT9/nn24vf39/uafi+vff7r6dbGczWfCe3ThJ7VuxL19RHMtGvSz+WzVKS15hL4+dkoH/prNZxJr0aXPpe1cxRsu0au16dcsdq11AaOF2XyG/D0e87EzUuNsPuOhC6s1VkFZWlpYVA9D43m87EE4JVa8RjUnTri9WzxcLX47uPcdequ36KLB2XwWpy8X18ur+yteEkdu7xbLxf34vbz4ZfEpzsdjlJY3LWH3k7E79J0O/dCFNQF34aYLbTcMXqLxKuzHT6e2Inkhjtiqa9AMGz5hEFIEMXyr3fGmW4cehy39PfvvZbXBBoevjZD26djG6Gb+fBC6w9nX+eeZwxodGvZY3Zk+ZtlPZQK6WhyuyIf/miXZcI4q7Fv8oJUg8NuRHvmSbPjUmfnv3GL++zl7n2Pzz9dMPA/HSdy7AorZ/Koa0sufQWcejX0y353B1irJCrA3FQwmssR/Jdz6Tb7j1tlGeXx3yPLzHDmMYgG4C2ikh4kcwB/gUEhr9B5cpKc8g5Fob8dpzzxchr3G92fgg1NmfTR9Y/T+/RmsrNUoDPw3d0hBq159lw4bST5iz1Qsh/mCYlEA8GJLgE+AfyHRV06xNpTQv2DveArL+8HLDH/Ec9JxJwAlPEfjlfcPwhUmWhE2w+HZ+AZ3Jcy10Holqsdhburussy/UqbtQk6kgkyPdz4oCfmtRyYXsDVW4kjwiVNivvtTXj6Ekx+5RUchOoMvsx/f/Pi31z+8+TLLHVvIrSOG+injpj47Km6n8vUkaSelMMduRFMiQ+ROAWxl27IHKytxuTdBFEnxJzxnTvozuEsD7+6wsk6+iwvn/XHn51O/lAv6MZfSr1zjoVzz6d4wLfzw92x/ZY0PcNAIlA847A3yralDyDeOsTpqGiZJLdwawygGeZ463Cp88qPQTSc97zvyvd2ic0qSek46imOKvi9wtAQ7Kn6Oulb6W+GNf8/PJ3TibqhItXiXBCb2MiUsk4YqR1PYP6E4tT/9Vfkjm3X4nw59QJk6s95p8SvPIK6Pp+88lsDEaqZebmxCvHy2b8UmCw4u30cR/oDPj7jv4/71lL5NmsfcWfZRV5si32LBKFGq8v5ZlSHfIgKurdsXT5J05ZGggwNy2yQCJ7sAnj1ZQY8b5dwBj8pIkvZG7Q5l3ZbK5ZOicnk8jGq9CWdgumaFLudbKyr8JrRE8QyTkoUTtDDrTqxLcl5rsbXFUAXi9/KEovDkJytLJtdO7E9MiapCE05MRignJqPQDKlVKCOFpCu5rJfwYpk+PhbDc4rPtJFQtUL3f27jXrYjht04iW4a9THFo+p8K70PX2Ulwnpe8yzO2rr2GP5dmFnp7vSGfxXpTD/7iVJs+vJx3OO7s2n08sgnPfuGwye6dySq3HpnLqUHE/0XwtSWijy1L9/bJ4Ny6AVBSTRBBYUehJHw2+Ijt0seVlhbh6mVBetAGR+E1hyK1+mfIRJwiy6rGH1NY4WA5eWc8suA6wwsL6G2rhHB018QWoNErbbouCl9/cVcW7i6AVtVnfOwQYdv4eLXK8bViupRrBFih+WB+BQ2qBzYWPqkclgF6xR6Are8fOmHV1VsvUDRSBCK7k8IuKdi6wTaCR/mIJ7EHmpnG1g5++TRwYr7LTZ6v0HwosE0JvueAYRDEFuhdHSpAQFtt9LKb3CETgcJ8JWoa6sl2bsxCJf3Fz/HOvoWKtu01quAqVdku1uhlRTkWnZp2CARU1VK6FjGRW8qhgplNDdncyIogjRslfywA2e7gC89WKfWyggNjTCqRh/I1oU1W3QBBFvQyHgSXbwRrd/YAMFG6OkxME+upacQV1Pywviw6yGOLT10Hj2oFEpRBbVFfrrA00ZpTLEjc4Le/jEADwes3UPTV3MfxN4TGbySyau9E40EQzQFiS0/w61Jsb+6ZLMX7Pf0gOi3DS7DnfKBgPQJHCPZhY11TPimtYa6kNgNkUE29NL3yeM3osU5PGIbQJmIgjbYmoG+9PwUga2txKrTwu3jXWPcoTPBdZxbwQnj2SJfmWA4rKi35WwagpRuwDSmG7A3BnEh47P57MN4n0hjqISBFdFXK4OU8WOfL4btNP6P5c01tJYlj6x9kFJRTIVOLowljDTFIMo+G6rE15EXCQrlVXiyk1U4dMej1vCb5y2Itu1JQb+5K/DMSLqsx0aYoCoICkm0kkIlawxX6141VvuAnkKI0VsO18oHpLhynwWXitkq3L4Xr57Ko+QNfmzRNV2ItYqCQ+yu1Q4lYF0T7beY+nKWqK5N+Zoq/qCdLMDMATTSEm/7EtgrWLz/96mwpwvHsPMNY8jXznYtBawPHs1dY1iyiI65Iy1FygaoNsKs8+wZjCezXFuhUTtw2AhlKLsiFvL0kfvjpnhl9oRP0RA75cGGDToIG2FSyAeNHTyTSwwMfh64uLKdkSlfRndGkNFkwh3bBMoW1RKzON997P2irvWwpWqSZg0hZHtkp6+7UZgJpVYoYcVNJFRCayrn9DunBPGG3AtXN2wkk6bOj9vHuLCOZHf7+vV/","idLAvkAMGAWysmnMvDtbPhfxbanTQya2Dd0AevG9pr4=",6323,32], - ["kv",".","3FhRb9s4Ev4rA+MAJzjHxe3e3oOTJs22XVyRdrdosu1DHMC0NLZ5lkiBpOS4Qf77YYaURCpOWyxwL/eSWNRoODOc+b4Z3t6OLp0uZfZ6g9l2NAlPH2onnNSqW/iEti7caDLKCm3xqqFfBoXD9+Lrnh/foNJXzWUuKodmNBmt0V1mTjb40ehG5u0ay/L/j8Jt/O9PYtcuf8Jc2tdaKcxo/9+MLt8qeiftVfNOSSdFIb9izitfhMs2YlngaDK6at4qZ/b86wr9//fSuj8q0mP5ObLkqrnG5N2104b0fMBSm33siEEbzL7eCIP5VfNaq5VcjyajzgB+zU9vG1Sufeg34MePBlfyfrB47QyKcnQ3uR1J5dCsREZ2uH2Fl4UUJBevr2rFkUl/ZoWw9tm3f/Hnj5jz3Hr8+4Bxz4n+yO9+yyRmEOUxPMAW9zPgVDiFBo2lkDtRVjOo1VbpnYLHoAvSlIeXrUS8L8RlAA+gtzNYal2gUKn+ixlYZ6Ras35h9yqD1nEIxXN0PIOPRpfS4lmjZX4eBQfisjrKONMuZpCm3vEM0tTjEENSgiDLqsASlbOxMDxARqaaOnPaHG2bixl/N71qJlAJt+kcOD6FbdOFix6CzdEaO5RHoTAocq2KPdi6qrRxvkYjgbbcKQadmjW64fMHofbJmh3I5Figw2RpI2zyXEibfiP4EJMl9iBZub3el0tdTPns3khbaYt3icSOfHq6coV7+3TVF32yjsrWBv+oUCXLlS4KL/16I9Qan1X2Xq43bof094AV9JtM+UNlsWOcj12WPUHnSHIg9800bNM4yrAkmzukT/UP6qJLiu9vFHJ1uMlh3njGqQGXkFTIz1Sqc+rIEjsQnjBNHFOJaIMgk9JK8CLw0dlNjyfnQ1gSRY0zuPkuQLH8M7iUsBw8QMUZ0u1RyFK6ixmoulyiOQXrhKHn8BpV3j8YJDvwIqpWggppHapsf3HQsDZ9nrWup1l4ALyvpMF3qjOIVSXiHFe8d6hyC5d9CVKI4YGOOgnpUR/QKEnatx41epFJG/MgMAHtbeMg9KYO8blDm4O7BRFCn4PvQzTPPSCl5of9Z+k5Hs+86+8cGnL8rMum8/MOHZ/EwXZ+HQxFAL+MCLIXnUAZiO/g5zHtnUdoOfD9WcxM5Pi0PVUN2qwfI6vjmF+cNmKNM/ggqjNPWBOg7BFr5FidR8IMjGjsQBrd2RFSvxbolHu3Y3h5DmxurMGX1ZeDevoQd+Kc6BzVf6OoZvCrVMLs6ffZ2+5Va+YTEu2/vpElmosZfEJXG3Wzr/DMontHBdOI4vzH+Pb/lUazAkXMW6dg5dfUCKqKgbtLXafEeAqiqop92/+lWxvpfDolywZL3RxYL+ShVWEtGveE7hmK+2zgxjXh4VNQ2snVvs26IZkPSDTMKYeayx5hU15lvvAAzgjYovlp2gmSw7m0f5piuPS7KNFWIsPohd3K6hpNIzN8I22mGzREHiEjB4gfF3uL+i0LPHw/uzk1vgGEMcDHE1nbQPvh6+zmPMmyVGHCqE8UJjPdE9zuYYWg+5Dv4eV3u4S+SzTYSF3bz7Tek/IpECPHj7LE0FC8EQ6faTJOQfpG6GIGztR4yMSev3vZ7gSsXCteuVxq4675qS+zqPMwYvftLEgi2SXDwALqkVv4i9uaraxCRzfYhPuU5KzjUPddC82Sn32EgCkSaos5LPcB7SDTZSUMngiVn9idqEBX6CvXTrtrko5Mo7dt8npti5Dd0xZEF/R1mCsNwzzmsDK6BKHavTtlJPuaoBaEyj0AgtsgWC5rEIFNrRMOWZaHSRCwiAptAW4jaDOriwbtIQVawUoa6ygI07l6VQkjSuJiAg1/GqJo00F+DS43aIzM0TKgSswTNXP1yrsHl1CIr3tYabMTJqdRud12KbKtD1Nk09VnKOQKs33GFz00AJwoQbNL991KG1i8UuhsZmTlXmybBVmN94LaClgsFs7OFbcSsG3gJYidkM7PvbpCRah52gq0Sl+Cwl06Ux9tG5JbLBZ8ZuwNWZrVxqByxR4ED1Udqk5AroZh2gjLyUS8m0/7sNxscPg5kF+qLooFLHHFE0eiy2cOnWIcsPSsrVTrAp1Wf+EcuQS6QxRZhtamBvdnFMKWBoaiRy+JTqIoSUWqJ+QeSeWBJjCHHFeiLtzhsGgTi8a6B4EwYgeLMCkuQBLmqQxht0FvVrC6C7P0dydw9fl/EqRa5WiKPWV6a3Rr01y9chujdxa+tMYNc0Ba6gPaL9lTkYMAHnnJzDDztvdOjB1RVgKqRhqtqLuO4noJSpvST8BB15+f3vsYaYWQCQXL+GwIThgYdxt0GzwIPJTbS0TVRytJ8QVRzCI6hzY3Ae+ldfbJFqIdtPvxgL5LMzzTyhmRuf7s/Ee+i+hO32n6XWHmvmFQGD+pWfVbcVsAlx/f8TUx7k+YjwGpyxvg9YJ77QVD88I3zQzuH7VxPL5ukVGvFI4RSxRFVDh2OldzdbORloJa8eUCMaTy+VfsPQM5SZp20m26DL/CPe/TsiSjYUszsR0howaM9AzUwgaLisx6TjNPEqz4V2HZuRCbMJrF5zJXl0XRedofJzjyt28ErIbf0V0zikMlsq1Yo+VUNFgKqeaKKULlJ2KttHUyO4DzsiTn4GFwH/roT2mc8MT4lIzzwJ9e8hzEf0L/uXrxAn4VNuZlO1eeUmwXmttxbdHY8QTG//jp5/HdBB5AiRJnML4sZIZjeOw5h0Q7WvI61gd00AexTBjeDomxlXSlQEfsW9i54qzj7+mmzbqQxnoV9HGy9HdIrd47eDw+hoe5Ah7HdYHTQq+P+OPplppi/5OPnzZ/DPt/oTTF+6qQmfQNR20yhFIosfbnf8Sw9tP0578fz1rfaoIFsJmuMP/2UYRY9JIh8lvcU8DHbNCYJF+8gMva6VI4mfli4qHbVz4rIAii7PHk/lkXwsmibzG4ZdoiVpbrNhdOEI1VRhPeQ8n3GaGVQxcXFTdjXDoOrbPEYtKSesypuZe5JxRTqye15jmoZlhPy7RFNfriLbfK7XpbSgwRBgObeBjLRCWWspBOooebAAmWBIsTGhlALy11157kVoxZmb+OngZ46vcYTmyEfNLjZmVkKcw+El5pM1fcGWahMeaYKsS8t7OucuF4q2ermq+Uv1/NUZ/nL7G7qmDyAGICuZIZ3xH4aHs3D5UJDWmWw9FM/bXF7dP6jlb++cu/xnd3bdGQvlgTKfIqg0BaVwu67f3bA0tQeT0Gy/JZt0pTzeOCPAJ4jOqNPaP8ZKe47wAR6hmOpMqKmhtuKicSOX7O2djXcPVyO8Z7zGo+unHnW2L421YgnOMsdeNl98iF6R14bEuOJ2HAUjrXFqZQIR0xTxJxSElROxCOxxPTt4XaKyUWvQwlIsPo7uvHD/uiAFsvfZb5mxaRS4XW19Bv1PedVMK4fc9woXuDBcnuF1O4ptser511hH6eI08nIt3pXMW7WBAGeVKawhtpMOsmE6pW3+uSHs+HIDKaYwvMPbBO5yqMWdGMROOWr06pFJrejzDFTwiburFQ+5aI21riXY7OKvo8bvqUDjqCp3UFKyGL2nDXLhSgMDQe0SXBXBn8D2aOUccbcBpFxqATktEh6ZzDbuNueGKAMnOFxmgDS9yIRmozhZuNNPnwPErRdmtUD4WQCvz1WXve09Hd3X8BAAD//wMA","HUC43n1Q4+n3WFlMYqo0wHzO/TilpbbWKg6gFEYgHn4=",8461,27], + ["kv",".","3VlRbxs5Dv4rRHBAEpzj4nbv7sFJk2bbHq5Iu1s02fYhDmB5hrZ1npEGksaOG+S/H0hJM9LESRcL3Mu9tB4NhyIp8vtI5fb24NLpWhZvV1isD0bh6VPrhJNadQtf0LaVOxgdFJW2eLWhXwaFw4/i+44f36HSV5vLUjQOzcHoYInusnByg5+N3sgyrrEs//9ZuJX//UVs4/IXLKV9q5XCgvb/l9H1e0XvpL3afFDSSVHJ71jyyjfhipWYV3gwOrjavCXLyvfGaNrqatNr6dfeK2d2/le3doV+5aO07reG5C0/J3Zfba4xe3fttKFdP2GtzS5126ANTl6vhMGS7VjI5cHooDOXX/PT+w0qFx/6Dfjxs8GFvB8sXjuDoj64G90eSOXQLERBdrhdg5eVFCSXri9axRHIfxaVsPbZt3/yZ1Qa/0/NiGvPmfnc+j4dyZbPif6R3/2WWSwhqQZ4gDXuJsApcgobNJaOwom6mUCr1kpvFTwGXZAXDryOEum+kBYTPIBeT2CudYVC5fovJmCdkWrJ+oXdqQKi4xBK8Oh4Ap+NrqXFs42W5XkSHEiL86jgDLyYQJ6SxxPIU5JDDFkhg6ybCmtUzqbC8AAFmWrawmlztN5cTPi78dVmBI1wq86B41NYb7pw0UOwOVljh8okFAZFqVW1A9s2jTbOV3oiEEGDYtCpWaIbPn8Sapet2YFMiRU6zJZWwmbPlbT5N4IPMVtiD7KV2+tdPdfVmM/unbSNtniXSWzJp6crV7izT1c9GGTrqGxr8LcGVbbc6Kry0m9XQi3xWWUf5XLltkj/7rGCfpMpv6kidYzzscuyJxifSA7kXkzDmMZJhmXZ3PFFrn9QF11S/HijkKvDTfazzzNODRiJpEJ+5lKdU0eWWIPwhOnjmEpEGwRp99ZhxmjwABz67lVObvF1jzSB685ueiQ6HwKaqFqcwM1L0Bb3i7ukVV+jtWKJsdJHoD1XXUyAxQN1HfcYyRs/A40ZAcMDNJyknbGVrKW7mIBq6zmaU7BOGHoOr1GV/YNBcggvEsAgu6V1qIrdxR7w7tn+Wev6DgAeAO8bafCD6gx6En0+YsB7h6q0cNmjAJ0yPFC2ZWdz1J9MkqfxrQeuXmQUDy8IpLFPTR1SRAd4e3cLIgSAe9+HaJ57TMzND/tP8nM8nnjXPzg05PhZl5bn5x1AP4mD7fzaG4qAvwVxdC86gjpw797PU+Y9TwB74PuzsJ3JJaUx6AD/GF8epxTntOEq+iSas1hJlD1iiRyr80SYsRmNHUijOztCaiUDo3NbeQyvz4HNTTX4svq2V08f4k6cE52j+m8UzQR+kUqYHf0+e9+9imY+4fH+6xtZo7mYwBd0rVE3uwbPLLoPVDAbUZ3/Mcr/f2XyokKRUucpWPk9N4KqYuDuXLc5N5+CaJpqF1vQfGsjnU+nbNlgrTd71iu5b1VYi8Y96TgYivts4N45awVOQWknF7uYdcN+YsDjYYTa19/2CJtTO/OFB3BGwIjmp3kzSg6X0v5uquHSr6JG24gCkxd2LZtrNBtZ4DtpC71BQ+QRMnKA+GmxR9SPLPDw4+zm1HgBCFOAT4fF2MP7ufDs5jzLslxhxqhPFGbj5hPc7mGFoHuf7+HlD9uNvlE1uJG6tV9pvSflUyBGTh9ljaEzeSccPtOtnIL0vdjFBJxpcZ+JPX/3st0JWLlUvHI518Zd81NfZknnYcT25SzIItklw8ACatMj/KVtzVo2oakcbMJ9SnbWaaj7roXG2a8+QsAUCa3FEua7gHZQ6LoRBk+EKk/sVjSgG/SVa8fdfU9HpsnbmLxe2yxk9ziC6Iy+DqOtYZjHEhZG1yBU3LtTRrLc3oJQpQdAcCsEy2UNIrCpdcIhy/I8CwJmSaHNwK0EbWZ1tUG7T4FWsJDGOgrCeKreNMKImriYQMOfhqhiOsjvweUNGiNLtAyoEstMzVS98e7BJVTi+w4W2myFKWlaj9vORbH2YUpsuvoKlVxgsSv4xopmkBMlaHzqvltoA7M3Cp0tjGzcq/VmRlbjvaC2AmazmbNTxa0ErDfwGsRWSOdHb92gItQ8jQJR6WtQuM3H+qP1huRmsxmfGXtDlhatMahctQPBc12HqiOQi2GYVsJyMhHvluM+LDcrHH4O5Jdqq2oGc1zw0JPp8plDp5gGLD9rK9WyQqfVnzhHLoHuEEVRoLW5wf0ZhbDlgaHo0UuikyRKUpHqEblHUmWgCSyhxIVoK7c/LNqkoqnuQSCM2MIsDKszkIR5qkDYrtCbFazuwiz99Q1cff2fBKlVJZpqR5kejY42TdUbtzJ6a+FbNG6YA9JSHxC/ZE9FCQJ46iYzw0Qbr74YO5KsBFQbabSi7jqJ6yUobWo/hAddv3/56GOkFUIhFMzTsyE4YWDcrtCtcC/wUG7PEVUfrSzFZ0Qxs+QcYm4C3kvr7JMtRJz1+/GAvsszvNDKGVG4/uz8R76L6E7fafrdYOFeMCiMn9Ss+q24LYDLzx/IuMsE1ykbw1F5mYWQFYEXFqK1CNLZLixbKnnu8gmUKSUY6QjCdVuVfL7hIFlt0Ij3K9FaIg5SZtD3UjBvyyU60nSFuxPuDwCp6xzwx4x7/xlTxcw38Uw2vwiL4McwAk6yuzVIG/Q0Ra73Rc0M91kbx1P4Ghm8a+H4e1FVmehUTdXNSlrKjYavaYjolS+jaueJ1EnStJVu1RXqFe7YvEj2DOqRLRPzY5M6INZnGANWWDXBg72aeSDq47LuQhomzDS9puqyqjpP+6wER/72/YzV8Cu6ayYjaESxFku0XFEGayHVVIXzPxFLpa2TxR66kjU5Bw+Dm+VHf7iHGd0dnpJxnr/y67K9NEYkNlWvXsEvwqbthZ0qz4y2C83tYWvR2MMRHP7tp58P70bwAErUOIHDy0oWeAiPPXWSaMeuXsdyjw76IJUJM+g+MbaSbkboiH0nPlWcdfw93VlaF7JfL4I+Tpb+KizqvYPH42N4mCrgWwVd4bjSyyP+eLym3t7/5OOnzR/D/t8oTfG+qWQhfd/UmgKhFkos/fkfMTr/NP75r8eT6FtL6Aa20A2WLx9FiEUvGSK/xh0F/JANOiTJV6/gsnW6Fk4WvpgYVTyAsQJCUsoe36N81ZVwsuo7Je781oiN5bothRPExo3RRFtQ87VM6EjRpUXFPSWXjkPrLOGUtKQeS5pRZOlR0bTqSa15Km2ZnfIyjeBMX7znjj+ux1JiiDCYIW0hGjGXlXQSPdwESCAIE9UJTT6g55aGBI/UC8aswl/sjwM89XsMB08CTOnhvzGyFmaXCC+0mSpucIvQ33NMFWLZ29k2pXC81bNVzZfzP67mpF31fw7oqoI5EIjQ5EIWfNXho+3d3FcmNGtaDsdm7G9fbp/Wd7Ly93/88/DuLhYN6Us1kSKvMgjkdTWjS+u/PLAElddjsKycdKs0nD3OyCOAx6Te2DPKT3aK2ycQoZ7hSKqianluoHIikePnnE19DTdIt4d4j0XLR3fY+ZYZ/j4KhHOc5G687h65ML0Dj7HkeKAHrKVzsTCFCumIZZaIQ0pKuppwPJ6YXhaKN2MsehlKRIYbCF8//s5CVGDbuc8yf2EkSqnQ+hr6F7WvJ40wbtczXGhCYUayu9kYrunSymtnHWEs4cjTiUh3OlXpLhaEQR74xvBOGupwOu1hAqXmOfZDoqBxvMLSA+t4qsK0mIx6NDX66pRKoen9CJcRI8KmbrrVvrPj7px4l6OzSD5Pe1elg47gadvE9oj7MgUoDE15dNcxVQb/gwV3ZsGA0yQyBp2QjA7ZABB2O+xmQN9KThXyn4rmuBIbqc0YblbSlMPzqEVs8qgeKiEV+FvAeN7jg7u7/wI=","7Eef6iYOVnq4iYHYfwXg1jhGQNjrLTFz2twc+D4AGzc=",8861,30], ["kv","./kvdex","1Fltb9tGEv4rA/WDqYCVA7ToB9lW4ti5nOGm6cVOekAUgCtyJG1F7rK7S9mqIeB+xP3C+yWHmeXLUpKdFAEOuC8xtZydnZ2XZ55hPn0apDrPMXVSq0E8SA0Kh7+gs6mRpbucDeLBxe7au5Kk7SAeXKLS504XMr1YYrrqrbwr0YhaLa1ery90UUj32hhtdtbeo61y1y6+Vn9UWGH/HFp2ZvNWbGbYLr1Btyt1jZtfhemU/Sytu3JkS3AsLe5upLUbJF8Egjf7B9w4I1N3jZt25Tfh0mUntlpneD+IB9f09yZdYiH417vZ75iSYdfrX41eywzpnEJnmA/iASsRsxyv16+MzBY4+Bx/Gswr1cQmeJTKoZmLFL/x2W1KPM+lsF+5/i17A/NDkfDZ4BwNql3xNBfW9rwBXdKe3l6psnIx3L6rXFk5wHuHKrNwvf4o8grphY9M++ai3Vy/Oa33TiaRMIvnY6jUSuk7FYP2Ai/GrZbhGF5VMs/Q/G3v7O6sySAeCLtRKXQm75bR6W1ga5spk8jy3zHchsc/VoTDMfxqdCEtnr5HVxl1uynxlDNwMgkj8KgGeICyzsYX7c1h29u7U+fwACvcjGGnHk5gjcaSTieK8su6WoSAB0hJbzQajXoBGI7BLaU9gZRhIgpuW0tMTiDDHB1Ghyxq96NHlGhNKXEwvoeQp91eiPuD+mNQY5jJhVSuk5Xqq2Utukdkv2Roh0udsqr4yoMPRKSHzvAAejWGucgtPiHrUbsWdqbC3QSwzki1OKih72h4oCiKzYsxqKqYoTmhBLNX8w8qw1yu0WC2k5yEG7DbF05v4ayRmnSPB87vGgcln1ZWWocq3Tx+St1XntQa9pq+LU2lnxMosMgsx1Z0/x6TCdlVGavNk34MGhk8QC4L6QInegUvGg0nYJAihC/GMNM6R6FODl/+BGbUjm7kn9iq23VI2C+f9EqXq/AAeF9Kg1fqUbVt3j6pM2y58ABG3HV3Yo0t7jIWnt56dG3j4H9e4lwqSWKTqK6wsQfjtjPUqDwcw6VwYiYstmuN3QF6w9me5k7K9//etdqOBz1O0BrP5OD08dbG/QfOmgY0iZwRys61KV6MIcqEoz7CQkM4mzRiwzG89Yr7/WvSNFvYIyN1lThTUbij1XocygxPYIEMz23+LNC9FWrTWyO4C3/XyB0u5dL2ZQS3it7SHR3cW0lzbft6GsDf1Y3qH3vrn242xUznI27Yl9KW2uLnQAK2TD58AwUBCu8CAgJ30i2lAgFZnSDkKYf3bjRVL/FeFGWOkCSJs1Mli1IbgkwObByoiX2iwhbmRhcwHbzUeS6PeXE6CHb+brV6rVKdUekcED5GeinV4pgkaetUcQJ+sEh1+jBVAJVFo0SBDTLQmlhgU5NTtaVtHHFYreEMxJ2QjmtvpEtU1+to2ElkMzjz5kesnbJjtY7psSEzvA61ejsOLh75HPdvJtFwyPu8iYcE6RqTaBg3KgFk9gaVR9IxRJzoqdmUTo+MUJkuPny4uowatQDsHzTj0JPBa6kymaJtTW5s8e6aDkojC2E200G7o3bddGAx1Srjl827rX/YDunvdqroIUmSqXpZCiMKnwcBI/ULo/Z9jUqhRL1EMoYpH5yH6Tjz7LQlnqN2hANR51ibqJI6NaHPTKQrzGC2AbdEEKmTa4TkpWrY4vFqnUxVwxNHFPrbJYI/H7NAo4V5lecboJTLQKSpNpSM4DSrrlVkdWKAUNlU2aqk5LbBNWwMrTcpJHiPNq7BAHRDHGlJZSDyHLRbUt76C85RuMqg7fxYH+df1z+yFqMh6p3MMbBxkwrDvWj47iBy0Gs0RmZoYa5Ny6PBcluUWoUxesT3T4BEiA4+UeqCPwrq/egk3LFL8rsdYSy7nb0S9kW+p8MX9WP1+AAhkJyEMALbsFD3C8tvPKor6qgtKFZx1Ma/ebHlv1sq1e2QbfcGZ7MRGzcSWRY15hyd5zLFo9gr++E58BZ/2Qcwnrpu20u3OuZSZa82N83ZV5R60ZFYkKofnpOOJEkG8aDhHhT3h5e5VCsCkn3nban+Pnpa7Kcc3rKXySR2vrPY1rJbCgezaj5HY6GonN/CuU89dCEcWnB6qgJLgvY88qdtgUatwFZm8VuuZp+cWuUbCrSFxKJLYkh8i058nSVsfwJMxpSgMqe7cHlOVSFoXVFOjyCxVZEcJ4VU9K+4T0AYhFJYixm4pdHVYtlggshE6dDEcLeU6ZKCVMykohstseD2ymLWaYOZn4wY1Nj8JpRawVzIvDKH39kqTdGyl8PIJS1PSOgVM3CwS1EG2DbbQMLsJoFjfvK8hn95upLsqfXytExEtqS5gQDREiBi6bxW9jnJEJQy+fhe1pMBcxXgX8Su2QcCEs/nEwKaEo3b7J0b2NNS87UwUii3L9HbydzMb2QCTgMY2wz/+de/QYBBkXF6uIqgSs9ZgG6279SaoLG2HmvaAz/OPf6ewtEOKVSLx5x6RGm9iAfvp7gV+zXep1jfzq3+j9nUTz8ekmMTrtcffvoxkLVopMjln5hRKqjFoY3+oh7eH2FUHqn/IpX7n5Kv1Rqo9MgJcP2xY0NOwwzpXM47Bh6iMJSHBp2RuKaflKu7BKMb+1jzPtFgpaSfcn4vy3336FdCl+8tk+CBszPXo4NzIl22xIrbSYM1Bgk4iNIFxVDbVlazXNrlDiKdK9B+WqVZRkhFF+byv/7oEbimXtKCdBbzOYj2FSkIK5/MzITJag5D5urK8Q1pg8y4nTEUNMQq3zRkCgS0M23Hafer/68XJdTuzs4dD/dY1+nxMfyiTSFyb29TlqSbp2Y4g1761ntuWyu/sO1dIR3vjWFa/4dLdu6mg0nMwpMoItN5jPEAMBoxN4l3TCZAJbPJgu1wJ7U7n3V2dV8VRFlSQBngJH9C4GA4Ddp/aWhyu025fgw5RahrWxBwmHNse+VEGZ34Lzc1feBvOXhfYuowm6o2+fxoIS3M/HcHVKTP1uz5iwMCTQNMdqaq41ZkwZ02K++VUhhULt8weQehNruDjp+GaDCJ3mMm7fEbYRS6GN5ioc0mBk3TFQ0YDXWxQ7b7u+++g0u0cqEgw1Tyl1Ba/x6ePWsohKIvcDW5sKCqPH/2jFusv6D3iIWEP7G334FJjuPf/8BKy7BNGKUKaS0FlT6cjuoz62bvKR0VokFRMJTx/00smrH52TO4Mz4p6JAk5I61jphM03yCgGS0y0SIY1GQG8rSWNB8vkl6RLV/QitTe4L0sOMb93Lp5sRANiCp6AskPvP2w8+3V8ev//n6wkPgb+e3F3//nkAzg4vzm8aEho20TheQvEeR0ek37JHmYD7O+rIgIwIjyQRco3K1flbaZLb3a/2my+5eRBtrasKZHCfB5yjPjq2rZrY25Y77Ty8z/b6niM8C3fX68MTXmxL3v+59YUrcpVbp1w6mDQqyu/yJLf1ha6NuKqsL/oyBbc/EKFDR7TnElbyeQ3xJOiy+PMLy1OqpC02cPDfSzPf5838BAAD//wMA","0J0vZdFVqaxYOmLVxnPCmcjKSmUt4ui8u7oXs+IJt6A=",7982,23], - ["kv","./redis","tVZtb+I4EP4rI74EtAh2t9KdLvS6RW11H7hqq+1J+wGQMjhDsHDsrO0Ecqj//WSHlzhlP61OQoJ55t3PzLTzeW9qVc7Zw4bYtjc8Ss+lRcuVPAPfyJTC9oa9WfUkra79rxk1339zY78Wzt54+ZVC0SpNvWHvG6XczKppioUlfQEutt/Rsg2uBM2qk/RUkbQnoWP5omnN9x3w1WrCvLccznua1qRJMpf8//jNBBqXl0tLeo2/GC+oGEJWuniLn67qzNRFceGsjTXstZGQx7YmYDRQHLn1TwEhw8DzQlBO0hpoMQsHYEoaq0tmle6jzj7GUMqtVDs5mIAmTJUUtTOSxFzSuAn8cAaeUWJGumUsMSdTIKMYjNVcZi2dKVeGad7UHzelvLaxWw89kzGY0d0E0D/kP8hFDC9a5dzQbaV4ehcELQqlrfGuMayUEoRyAhnZ/uDczwRMR05JkKUA2qAJ5IzsM8o6wAQ3nbgMZTP/AdyUHkC0J1Zamr7XWF0/qDznYWTXoqc1DeAfJZV0mruwYCaUCVuav9b5SokRmlqyR24KZWgZWOzcu71HZlSb9+iVRjOy/ukfNigliUBXlCvBjVdlFIYrUBtq090hh6mU3jUPb+0lP035cSHgAKUWXy5zdx7FFqYa2y9ukpnS6W2jGJ4y3PkUl8UKD2EHP53EDnxtQa+eyY66dTB7w+ufR+7qXZWW0qZ7wOOKr5WG5F6SbbZpvK2S0UIu5ItWFU/JAMK6FAKSVkfJ5TD4YYIVsi2lsKqPwftKL+RfqCXZwQguaw+533vnCNz4VcrQ1WTVQh7uBZdbx99PrsXbBLZUgyHNUfB/m9SCV2SAy4VMtPMaB+qRNa6de9qjqxiSJLFmIXnuVh8O3Xv3BmutcoiC9xj7uNHEvYq/e7Ct4E+QtOu49/0gxRB5h3g8Foqh2Chj499ufv8jgrfBZCFxh9yFGLnDMo9KQ9pEQ4g+fb6JlkM4+PGLIZqmePRIkqTVwtO+EJxxC5qMKjWj9qP2H0kq+Dy6+TDwfq7ZJmFpuMx+WvmVwrZUu3qiCkVJkTMYj2FaWpWj5QyFO/DubKSw25AEw1RBQHvuUiZJcvrvoPVX4LREfubshk4zOOrO63L5Hw==","Tb0CysivATg1FyvV6lQ5gTLeGdHR0ghNqEw/PbI9U98=",2262,15], + ["kv","./redis","xVZbb+pGEP4rI14MKoL2RGpVk5MTlER9SKNGJ5X6AEhe1oNZsZ51d9cGF+W/V7vG4HU4Og99qMQD881l5/LNwGIxmFuVC/6wRb4bjE/SS2mZFYrOwFc0pbSD8eC5epDKYPqktdKNrIiQO+sL9kRW1823M/aMDfK7MPaPwtkbL79hKFqlcTAefMVUmOdqnrLCor4AF9u/mOVbtpb4XLXSU4VkW6Fn+apxIw498M1qZPlgNV4MNG5QI3H3+P/1nUtmXG6CLOoN+4/xgqognHMf70y8rzrP/qLosyDQfOBDV9syI8Cu2DVs6SIhb7qagEGB4sQl31YIGQUiLyTmSNZAh0lwBK7IWF1yq/SQ6ezHGErakdrTaAYaWapI1s7oVGXcBL6U/cKIZag7xsRyNAXjGIOxWlDW0ZlybbgWTf5xk8pbF7v10AsawzK8mwHzQ/mTCRnDq1a5MHhbKZHeBUGLQmlrvGsMa6UkMppBhnY4OtczA9OTU5RoMYC2zARyhvaFUR1gUpheXM6o2bcAblIPIDwgLy3OP2qsrh9UnoswsivRjzUN4L9LLLHlcJgwd0wNkMVbna+VnDBTE38UplAGV4HF3vXtI3KlpAytb/LDlhGhDHRFuZbCeFWGYVIF0wa7g+2NgasUP5QJ793T0PL5RH04QqnllwvDzqTrYKqx/eI4y5VObxvFuH3hzj9xWaHwxPbw9tj24GurePUA99SdUzwYf+fzKFzi69Ji2rQB2GmrN0pDck9omwWa7qpksqQlvWpViRQNMNiUUkLSKS253ALPH1gzvsMU1vUp+FDpJf3GNKEdTeCy6ZD7VXeOIIzfnoy5nKxa0vFeCtq5QX7jQLzPYIc1GNSCSfFP87QUFRoQtKREO69poJ5Y48q5xwNzGUOSJNYsSeRu2+HYP3HvsNEqhyjox9THjWauK/7Uwa6Cz0C477kPPaNiiLxDPJ1KxZncKmPjn29++TWC99FsSWzPhAsxcbdkEZUGtYnGEP306SZajeHoeRhDNE/ZySNJkk4JT4dCCi4saDSq1By7TR0+Iin4NLn5YeT9XLHNg6URlH0z8yuJ7bB2+UQVkyVGzmA6hXlpVc6s4Ey6m+5/02C/RQLDVYGAB+GeTJKk/QPSOfztNnnO2S22HJz0+bpa/Qs=","QlL5e8iv4Y+h0AWltybI8mHBbMStMLG1RcnuJIWEFAM=",2407,18], ["kv","./testing","xVZRj9s2DP4rRJ7ugFz6nmDbFdfuYbe2w3VbH+oCZiQm1iJTrignlxX33wfKTuLkkq4YCgxIezYlkR8/fqT88ePoZQq1M3cVmdVo3L+9aRMmF3hveCBpfRqNRyYSJnpDdYjb+/VLi02iOBqP7tevOcVtfrqn7u+vTtK7Rv1Ifn9Px68pRDo83QVOEc1gy/MoseUOz8mZbums8QMmU+Hc0/16sDCwHr89R5EXX6+J9+dO1n6LtHCPJ8b3KRLWo0/jj6NIC4rERnO99Lxo2fSEf8v2S8+OE8UFfs39+cdvcfLvz0fZwrGwTu0DiZ0u7cW2AwhnVXd1PYUzEtk7O0hyaOvEObQcy3S4ciTYo4VeunuqdrYT+cAXiIQ2sN8CY01TkBQdL2cHc40rmsLVNfzwI7S84rDhwapbcoj00xTmIXhChqcjJM/T31N2qVWuQgdtegHy9RTWwdkTV9/FyZk2Ozi63IMHZ4fEj/v3UIbLXr57KYZL7JJDf7FMRyPkxHxOYGfHysnyYMCMxv3vZzQpxC20QhbmW7Bhw5L3QSJJApuKGFJFW2AiCwgmo3V8U2clwf2fgJ2WJgXfRkptZHgJTBtYB4/JedptgFRhAlc3nmriJOoXmnbunVE3pqd/ske3/+0qsggRvtx6xysTLJ2R2VN/+IGWThLFLsYr4qARSsz6vrouocHo0nYfM3tGPkJa8AnUTnnooeyjTg7+DDY4d96l7aTggh/oc+siCQQmWFMUhZ+wbqChCCbUtUtj2ESXkvIbgNYUt1DriCMLpGMIkG3BHaNk4UqC0h/qBiPdINsboQSBwSWQ1hgiK9czKKWtyxdl7Vj/x8dSj8wdO14WvHGpyqmIwrewRt/SDISiQ+/+1jZQSkwbI3HqccpMkYAPIp5ECi7nbuk4ld1xAUyAvAVLTapUBPSIShyUZZkksxhigi/nZ/L44tyBJ1jEUEMxumVKYqJr0ovV+oUKU7MZzZTpi1PrS8HQ920x6sRajMZqHHbthXtiXPDT9azgsiwLvm0wYt2XX6DfBIu+d5Qc66Tx2M2JyTP9GeTAzqBXEWbqD8Lr2uxYfllCd0F1A3cPf7wag3c553EOlmmHhbOkeptC49HxzS/v373tKzIGx8a3Ntc8zP8ikwSkwoYseLciIDZBV6HGuKIoY6hbSXk8wRzNClo2FfKS7GTQNAU7OTRB4H0flLMdcjnp8SzOKrTeAnoJWuuCjzv4bPWeJt+V9nKjAzD36RneoRxcAuVREX5XN7sjcdfUOcfsUvtWx4PBGLc56FGzh0W2aZ9TwflYE4NtDVldqLskyLs1xV037amfdXEQuneIZMityRa8qXSq5hQDS1tTBCcwp7QhYpgrMBI19a7JarV0P9NjlwdJ6sZBaDUVp+rq2fCoNGfMPYTS0jy0bKgE1kEFNoZGAOFzS60C6hB2c6IDPQ8xu9R55Sll0nTs5KtlFxc5pIpih3icuYTa2ZtduEnBr2iBrU89Bzf58hRDjNEFAfQb3IqqaAIf9KZCqJCtJ6DHJggJlKrpbTnW8AVXGJlEADVjAf2HRq9tT3aZBTuDDGnjhFS7TSShuO7he1qi2YIkjKltCkbvwwYVKLwll1PZt4fGdTkYhwjC2EgVdFI3eRo62cnJ5iyNx0jPt+0H9g555x/K/gNiCim2VOoNgtYWvHNwU2t39eUfMDaB3ytyAzwkCefeiQrmAFlv/YI5qFrrTIX0jNrsoavv3i1EqtGxDKv03y6CM19j8K3XwLmPxf/1Evjqh8sZtLuPl/736dM/","NeVppXmwMfo5010zZqFoGR5u+HhY6mui+BmRJqsIblg=",3948,20], diff --git a/packages/mcp/src/publish-assets.generated.ts b/packages/mcp/src/publish-assets.generated.ts index 53e321fddf..90db8cd126 100644 --- a/packages/mcp/src/publish-assets.generated.ts +++ b/packages/mcp/src/publish-assets.generated.ts @@ -307,7 +307,7 @@ export const MCP_EMBEDDED_DOCS_PROVENANCE: 'pages/web-layer/vite/index.md', 'pages/why/index.md', ], - sourceBytes: 3259362, + sourceBytes: 3260004, documentCount: 286, - sha256: 'c8e3cf122102e77625d4f3030770ba90294b977cdf91d0009cfd4dcc4a78cfe4', + sha256: '1b21739625cdaed1c605790ea7b6cae6e589da39350a86fa23217540e81536bd', };