[aiev]

Problem

countTokens() from @anthropic-ai/tokenizer builds and frees a WASM tokenizer on every call (~70 ms, see #638) and its encode() runs synchronously on the host's event loop with super-linear cost on huge inputs (see #659). DCP counts a lot of text per chat-messages transform: uncached tool outputs, summaries, the system prompt, and the whole-session breakdown behind /dcp.

On a long session (1.5k messages, 7.8 MB) that blocked the server for seconds per pass and ended with the clients' health check (2 s timeout × 3 attempts) declaring the service unresponsive — three times in one afternoon, always during compression — plus a SIGKILL of the service after a stalled stream. The compaction never finished and was retried in a loop every ~2 minutes ("the DCP compact keeps restarting the service"):

• one countTokens over the session text: ~3.1 s (1 MB ~0.5 s, 100 KB ~0.25 s)
• server at ~100% CPU for 47–79 s during a compress pass, /api/info unanswered for 9 s
• counting 8 MB blocked the event loop for 299 consecutive 10 ms timer ticks in-process

What changes

lib/token-utils.ts only:

1. Reuse one tokenizer (~70 ms → ~0.2 ms per call). Credit to fix: reuse one tokenizer instead of building one per countTokens call #639 (and feat: reclaimable cost map, in-place recondense, and long-session freeze fix #621), which propose the same idea — happy to fold these commits into fix: reuse one tokenizer instead of building one per countTokens call #639 instead if you prefer a single PR.
2. Memoize exact counts (LRU bounded by entry count and total characters). The same summaries, system prompt and tool outputs are re-counted on every transform, and again after MAX_TOOL_CACHE_SIZE evictions re-count tool parts.
3. Bound the work per call: above DCP_TOKEN_COUNT_MAX_EXACT_CHARS (default 16k characters) count head/middle/tail samples and extrapolate. Inputs containing a long single-character run (≥ 512) are sampled regardless of size, because their encode cost grows super-linearly (measured ~20 ms at 4k → ~315 ms at 16k characters, warm).
4. estimateTokensBatch counts each item instead of joining every text into one string.

Measured on the same box (missed ticks = 10 ms ticks lost while the call blocks the event loop):

input	before (ms / ticks)	after (ms / ticks)
100 KB	245.6 / 24	121.3 / 12 (first call in process: one-time tokenizer build + JIT)
1 MB	504.3 / 50	5.1 / 0
8 MB	2997.3 / 299	22.7 / 2

Sampled estimates vs exact counts (prose / JSON / code / half-prose-half-base64, 32 KB–8 MB): −0.6% … +0.4%.

Tests

tests/token-counting.test.ts gains:

• reference equality with the package's countTokens for small inputs (NFKC, emoji, special tokens, a 1k single-char run);
• a shared-tokenizer guard (200 distinct counts < 6 s — ~14 s if a tokenizer is built per call);
• bounds: 4 MB of realistic text and a 2 MB single-character run, both < 1 s;
• cache: 500 repeats of the same text < 1 s;
• sampled accuracy < 10% with exact counting forced off via DCP_TOKEN_COUNT_MAX_EXACT_CHARS=0.

npm test (123 passed), npm run typecheck, npm run format:check and npm run check:package pass on this branch.

Benches (added, all numbers above reproducible)

npx tsx scripts/bench-token-count.ts 100 1024 8192   # legacy vs patched, wall time + missed ticks
npx tsx scripts/bench-token-pathological.ts          # degenerate single-character runs
npx tsx scripts/bench-token-accuracy.ts              # sampled vs exact, 32 KB–8 MB

Relation to existing work

• fix: reuse one tokenizer instead of building one per countTokens call #639 / feat: reclaimable cost map, in-place recondense, and long-session freeze fix #621 already implement the shared tokenizer; this PR keeps that and adds the content cache and the per-call bound, which neither has.
• Count tokens in a killable child process so a slow encode cannot block OpenCode #660 moves counting to a child process. This PR stays on the event loop and bounds the cost instead — smaller surface, no IPC or async rework. If you prefer Count tokens in a killable child process so a slow encode cannot block OpenCode #660's shape, we are glad to drop ours and contribute only the measurements and benches.
• Env knobs: DCP_TOKEN_COUNT_MAX_EXACT_CHARS (0 = sample everything, a huge value = count everything exactly) and DCP_TOKEN_DEBUG=1 (log each count with its path and size, for measurement).

Known limitations

• Above the threshold counts are approximate (sampled) — that is the price of the bound. Below it they stay exact and cached. The threshold was picked from the measurements above.
• The one-time tokenizer build (~70 ms) still happens once per process.
• An aggregate sweep still costs proportional to the text it must count; what changes is that no single call can block the host, and repeats hit the cache.

Disclosure

This PR was written with AI assistance (diagnosis, patch, tests and benches). I reviewed the diff and validated the behaviour myself:

• npm test (123 passed), npm run typecheck, npm run format:check pass on this branch.
• I ran the plugin from this branch in my daily driver (OpenCode 2.x, the same 1.5k-message session that used to be killed mid-compression): the compress pass completes, the server stays up, and the log has no Background service is unresponsive warning and no kill since the change (they only appear during compression before it).
• Every number in this PR is reproducible with the bench scripts added here; the sampled estimator's error is measured, not assumed.
• The diff is deliberately small: one file in lib/ plus tests and benches. No new dependencies, no behaviour change below the exact-count limit.