Skip to content

test(core): add reproducible large usage-ledger benchmark - #417

Closed
jcs130 wants to merge 1 commit into
Pal-AI-Lab:mainfrom
jcs130:test/usage-ledger-regression-20261116
Closed

jcs130 wants to merge 1 commit into
Pal-AI-Lab:mainfrom
jcs130:test/usage-ledger-regression-20261116

Conversation

@jcs130

@jcs130 jcs130 commented Nov 16, 2026

Copy link
Copy Markdown

Large ledgers need a repeatable HTTP and event-loop regression workload. This adds a manual benchmark, deterministic fixture tests, and documentation; it changes no production implementation or dependencies.

Workload

pnpm bench:usage starts isolated processes for selected source checkouts using the real WebApp routes, UsageLog, aggregateUsage and JSONL event store. It covers first/cached storage scans, all-time/30-day queries, repeated and concurrent queries, preloaded aggregation, and append during a query.

Outputs include phase timings; independent HTTP status probes at 25 ms and 500 ms; external event pulses and queued-handler latency; timer gaps and GC; peak RSS and sampled heap; source/input hashes; full-response fingerprints; and correctness gates. No Persona, model or World starts. The runner can generate deterministic ledgers or take a read-only snapshot supplied locally by its operator. Raw data remains in the selected local output directory.

Synthetic verification

The generated workload has 334,000,640 bytes and 130,469 deterministic records, with three repetitions per version/case. Official pre-fix bd4e35e and v0.2.16 9569f55 produce matching full-response fingerprints and independent call/token totals. Source data hashes remain unchanged. On this generated workload, v0.2.16's preloaded aggregation median is 246.4 ms and concurrent-query status latency reaches 420.8 ms. These are environment-specific observations, not universal thresholds; --max-status-ms enables an explicit host-specific gate.

The disposable boundary fixture also exposes duplicate-ID count drift: after warming count(), appending an existing recordId increments the cached count even though readAll() deduplicates it. The manual benchmark preserves this as a failed invariant rather than accepting it as expected behavior. This PR contains no production fix for that issue.

Validation

  • Full suite: 5,224 passed, 4 skipped, zero failures.
  • Root and WebUI TypeScript checks pass.
  • Final 16 MB CLI smoke run: response consistency and input immutability pass; the known count invariant is reported as failed.
  • First testing under a Windows non-ASCII checkout hit Vite URL-decoding errors; the full suite was rerun in a hash-matched ASCII checkout. Small manifest/job reads use the repository's BOM decoder.

Documentation and commands are in docs/usage-benchmark.md. No private-ledger measurement results, raw records, deployment configuration, or live identities are included in this PR description.

Measure real usage and storage HTTP routes, independent status and event probes, phase timings, memory, and result consistency across source checkouts. Keep boundary failures visible and provide deterministic synthetic ledgers plus read-only private snapshots.
@Phantivia

Copy link
Copy Markdown
Member

Thanks @jcs130 for the benchmark. Its main result is useful to us: on the 334 MB synthetic ledger the responses of bd4e35e and v0.2.16 match byte for byte, which confirms that #415 did not change any usage figure.

We are closing this PR without merging. For us a harness like this is a comparison we run by hand when the ledger code changes, and we keep those outside the repository. In the repository its 634 lines would have to follow every change to the WebApp constructor, UsageLog and aggregateUsage, and the tests it adds to pnpm test check the harness rather than Cortico.

It has already been used once: for #421 we ran it from your branch to measure before and after the fix in #424, and the numbers are in #421.

On the cached count: the drift exists inside UsageLog (an append increments the count at src/core/usage-log.ts:63 while reads deduplicate by id at :88-90), but no caller appends an id that is already on disk. Model-call records take the attempt id, a fresh crypto.randomUUID() per attempt (src/providers/transport/response-http.ts:115); usage a World reports has no id and gets a random one on append (src/core/usage-log.ts:46). We are leaving it as it is.

@Phantivia Phantivia closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants