v0.3.0 — content-addressed storage (205MB → 30MB, 6.8x)

[imsobear]

Summary

Cut on-disk capture size by ~7x by content-addressing the repeated
parts of every request.

Heavy hitters in a typical capture (system prompt, tools array, each
messages[]/input[] item) are now written once to
<projectId>/blobs/<aa>/<sha256>.json and referenced by hash from a
small v2 manifest line in log.jsonl. Hot blobs (tools, system,
prefix-history items) get referenced by every iteration but stored
exactly once.

Measured on real captured data

11 projects in ~/.agentmind/projects/, repacked in place:

Project	Agent	Iters	Before	After	Shrink
ac90562a	claude-code	1112	126.2MB	7.1MB	18.6x
b22fd262	claude-code	392	28.1MB	2.1MB	13.4x
475fc34f	codex-cli	120	20.8MB	5.8MB	3.6x
5df6e381	codex-cli	46	10.6MB	5.8MB	1.8x
4eb642b5	claude-code	36	5.4MB	1.2MB	4.5x
3e3b5b3c	claude-code	18	1.4MB	244KB	5.7x
(small projects, 1-22 iters)			~13MB	~5MB	~2.6x avg
TOTAL		1762	205.5MB	30.0MB	6.8x

Codex shrinks less than Claude because each function_call /
function_call_output item in input[] is unique per turn — the win
comes mostly from dedup'ing tools + instructions. Long Claude
sessions are where this lands hardest because tools[] (124KB per call
in dogfood data) and system (27KB) are byte-identical across 1000+
iterations.

Format

Per-project folder (was: flat <id>.jsonl):

~/.agentmind/projects/<projectId>/
  log.jsonl                 ← v2 manifests (and any unmigrated v1 lines)
  blobs/<aa>/<sha256>.json  ← content-addressed; git-style sharded

v2 interaction manifest (one line in log.jsonl):

{
  "type": "interaction", "v": 2,
  "interactionId": "...", "messageId": "...", "index": 3,
  "agentType": "claude-code",
  "request": {
    "meta": { "model": "claude-sonnet-4-6", "max_tokens": 32000, "stream": true, ... },
    "systemRef": "sha256:abc...",
    "toolsRef":  "sha256:def...",
    "historyKey": "messages",
    "historyRefs": ["sha256:...", "sha256:...", ...]
  },
  "requestHeaders": {...},
  "response": {...},     // inline (unique per request)
  "sseEvents": [...]     // inline (unique per request)
}

v1 lines (no v field) still read inline — coexistence is fully
supported, so a partially-migrated file works.

Migration

Three layers, all automatic + idempotent on Storage init:

1. Layout migration — flat <id>.jsonl files renamed to
   <id>/log.jsonl. The old smoke-codex.mjs mixed-agent migration test
   was updated to assert the new layout.
2. Lazy v1 → v2 repack on read — loadProject notices any v1
   interaction lines, hashes their inline content into blobs, and
   atomically rewrites log.jsonl with v2 manifests + the original
   project/message rows. Skipped if the file was modified in the last
   5 seconds (quiet-window guard against clobbering an active capture).
3. Pre-existing legacy migrations (sessions/ rename,
   cwd-only-id split, mixed-agent split) all preserved.

Verification

• pnpm typecheck clean.
• pnpm smoke (now includes scripts/smoke-storage.mjs with 3
  scenarios: roundtrip+dedup, legacy migration, quiet-window guard) —
  all green.
• Deep-equal check on real data: every interaction's
  request / response / sseEvents / headers / metadata was
  compared between /tmp/agentmind-backup/ (pre-migration) and the
  v2-unpacked output of Storage.loadProject. Result: 143
  interactions across 11 projects, 0 mismatches (modulo JSON key
  order, which is normalised away).

Risks evaluated, in priority order

Risk	Mitigation in this PR
Repack stomps an active capture	REPACK_QUIET_MS = 5_000 skips repack when log.jsonl was just touched
Dangling blob refs after crash	Blobs land BEFORE manifest line; crash = harmless orphan blob (GC sweeps later)
JSON key-order shifts kill dedup	SDKs stable within a release; future bump to canonical hash is read-compatible
Cold-load read latency from many small files	Per-load BlobCache: same hot blob is read at most once per project load
Concurrent writers racing on same blob	rename is atomic; identical contents make the race no-op-safe
Cross-platform inode pressure	256 shards (2-char hex) keep any single directory under ~few-hundred entries even on huge corpora
Dashboards / scripts greping <id>.jsonl externally	Format is documented in the storage.ts header; jsonl stays utf-8 readable

Future work, intentionally out of scope

• agentmind-cli --gc subcommand: walk all manifests, sweep orphan
  blobs. gcBlobs(root, used) is already exported, just needs a CLI
  flag + a "no live captures running" assertion.
• Cross-project blob dedup (currently per-project): would save more on
  duplicated Claude tool definitions across repos, but per-project
  isolation makes rm -rf <id>/ cleaner. Defer.

Test plan

• pnpm typecheck
• pnpm smoke (incl. new storage smoke)
• Deep-equal vs backup on all 11 captured projects
• CI green