Skip to content

fix: estimate tokens at the measured 2.0 chars/token (figures were 44% low), and stop a failed or Ctrl+C'd script leaving its transcript recording the window - #98

Merged
kellymichels merged 3 commits into
masterfrom
fix/token-ratio-and-transcript-leak
Oct 10, 2026
Merged

kellymichels merged 3 commits into
masterfrom
fix/token-ratio-and-transcript-leak

Conversation

@kellymichels

Copy link
Copy Markdown
Collaborator

Two fixes to output tracking, one commit each so either can be reverted alone. Both commits pass the suite on their own.

1. Token estimates were about 44% low (12f286e)

Every script's ~N tokens est. footer, token-count.ps1 and the ztokens records divided output characters by 3.5. That was a prose rule of thumb that was never measured. Anthropic's count_tokens on 8.9M characters of real z-script output gives:

Model chars/token
Claude Opus 5.5 / Fable 5.1 / Sonnet 5.5 / Haiku 5.5 (one tokenizer) 1.97
short, non-deploy script output alone 2.03
the previous tokenizer 2.62
  • ZHelpers.ps1: one named constant, $global:ZCharsPerToken = 2.0. Each record carries its basis, est. chars/2.0.
  • token-count.ps1 reads that constant instead of keeping its own 3.5.
  • TOKEN_SAVINGS.md (and .txt): every measured figure is recomputed from the same captured character counts (×1.75). An active dev day is ~46,300 tokens, not ~26,500; a cached zdeploy is ~1,420, not ~810. The measurement note is rewritten. Prices are now the current models: Haiku 5.5, Sonnet 5.5, Opus 5.5, Fable 5.1. The raw-orchestration estimates were judged in tokens, so they are unchanged, apart from the full-rebuild floor, which can't be below the output it contains.
  • tests/TokenEstimate.Tests.ps1 (new): checks the ratio stays in the measured band and the footer and record agree. It fails at 3.5.

2. An error or Ctrl+C left the transcript recording the window (d9cded9)

Tracking ended only on paths that reached Stop-ZTracking. After a terminating error or Ctrl+C, the transcript in %TEMP% kept capturing every later command and its output in that PowerShell window. It was never deleted, and the run was never recorded.

  • All 16 tracked scripts run their body in try { … } finally { Stop-ZTracking -IfActive }, which runs on an error, an exit and Ctrl+C. The body is not re-indented, so each script's diff is three lines.
  • -IfActive is silent when the script already stopped tracking itself, so the footer prints once.
  • Start-ZTracking first clears leftovers (Clear-ZTrackLeftovers). It handles this window's own unstopped transcript, which is held open and named by the Process ID: in its header, and files whose process has ended, after an hour's grace. A live run's file is held open, can't be deleted, and is skipped. It never stops another window's run, or a transcript started by hand.
  • tests/TranscriptLeak.Tests.ps1 (new) contains:
    • a parse-tree check that every script calling Start-ZTracking wraps the rest in that finally;
    • throw, exit and already-stopped runs in child processes with an isolated TMP;
    • both kinds of leftover, including a file held open by another process.

Tested

  • Invoke-Pester .\tests with TMP isolated: 305 passed, 0 failed (1 skipped). The commit-1 state alone passes too: 296 passed, 0 failed.
  • Mutation check, on the private copy: removing the try from one script, the cleanup call, the -IfActive guard or the hour's grace each fails exactly its test.
  • CHECKSUMS.txt is refreshed in each commit (zchecksums -Update), and the plain-text twins are regenerated with scripts/plaintext_twins.py. --check reports them in sync.
  • The diff and the new tests were scanned for private paths, hosts and project names.

The release bump comes after merge, as usual.

🤖 Generated with Claude Code

…unmeasured 3.5

Every script's "~N tokens est." footer, token-count.ps1 and the ztokens
records divided output characters by 3.5, a prose rule of thumb that was never
measured. Anthropic's count_tokens on 8.9M characters of real z-script output
gives 1.97 characters per token on Claude Opus 5.5, Fable 5.1, Sonnet 5.5 and
Haiku 5.5 (one shared tokenizer), 2.03 on short non-deploy output, and 2.62 on
the previous tokenizer. So every figure was about 44% low.

- ZHelpers.ps1: one named constant, $global:ZCharsPerToken = 2.0, with the
  measurement in a comment; records carry the basis "est. chars/2.0".
- token-count.ps1 reads that constant instead of keeping its own 3.5.
- TOKEN_SAVINGS.md: every measured figure recomputed from the same captured
  character counts (x1.75; an active dev day is ~46,300 tokens, not ~26,500),
  a rewritten measurement note, and current prices (Haiku 5.5, Sonnet 5.5,
  Opus 5.5, Fable 5.1). Raw-orchestration estimates were judged in tokens and
  are unchanged, apart from the full-rebuild floor.
- tests/TokenEstimate.Tests.ps1: the ratio stays in the measured band, and
  the footer and record agree; it fails at 3.5.
…er leaves its transcript recording the window

Start-ZTracking starts a transcript, and only Stop-ZTracking stopped and
deleted it, on the paths that reached it. A terminating error or Ctrl+C left
the transcript in %TEMP% capturing every later command and its output in that
PowerShell window, never deleted, and the run never recorded.

- Every tracked script (16 here) runs its body in
  `try { ... } finally { Stop-ZTracking -IfActive }`. A finally block runs on
  an error, an exit and Ctrl+C. The body is not re-indented, so each script's
  diff is three lines.
- Stop-ZTracking -IfActive is silent when the script already stopped tracking
  itself, so a normal run's footer and trailer print once.
- Start-ZTracking first clears leftovers (Clear-ZTrackLeftovers): this
  window's own unstopped transcript, held open and named by the Process ID in
  its header (never another window's run, nor a transcript started by hand),
  and files whose process has ended, after an hour's grace. A live run's file
  is held open and cannot be deleted, so it is skipped.
- tests/TranscriptLeak.Tests.ps1: a parse-tree check that every script that
  calls Start-ZTracking wraps the rest in that finally; throw, exit and
  already-stopped runs in child processes with an isolated TMP; both kinds of
  leftover, including one held open. TokenEstimate.Tests.ps1 now isolates TMP
  too, because Start-ZTracking sweeps it.
…'s Stop cannot turn their stderr into a test failure

CI failed two TranscriptLeak tests with "RemoteException: boom". The
powershell shell in GitHub Actions runs with $ErrorActionPreference = 'Stop',
and under 2>&1 Windows PowerShell turns each line a child process writes to
stderr into an error record - here the fixture's own deliberate throw - which
Stop then made terminate the test. Invoke-Isolated now sets Continue for the
child call, local to the function. Reproduced under Stop before the fix; the
suite passes under Stop after (305 passed). CHECKSUMS.txt refreshed.
@kellymichels
kellymichels merged commit a969c20 into master Oct 10, 2026
1 check passed
@kellymichels
kellymichels deleted the fix/token-ratio-and-transcript-leak branch October 10, 2026 14:04
kellymichels added a commit that referenced this pull request Nov 1, 2026
…p comment in 15 scripts, so they match their source (#100)

The Oct 31 zpublish_zscripts run copied this from evo.scripts and left it
uncommitted, as the publisher always does. The nightly test run skips a
dirty tree, so evo.zscripts went unmeasured on 2026-11-01.

PR #98 published this comment without the number, while the private source
has carried it since its own fix, so every publish staged it again. Other
files here already cite evo.scripts numbers, so this follows the same
practice. Comment only; CHECKSUMS.txt refreshed by the publisher.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant