Integrate main v0.7.3 graph with schema v4 - #157
Merged
theDakshJaitly merged 6 commits intoAug 29, 2026
Merged
Conversation
* fix(graph): isolate per-project compiler crashes, gate semantic diagnostics - wrap each project's ts.createProgram: a TS-internal assertion from one malformed source no longer aborts the whole corpus; the project's files fall back to tree-sitter extraction - keep poisoned files out of the inferred program; isolate it the same way - force skipLibCheck/noEmit on discovered projects - make the per-file getSemanticDiagnostics pass opt-in: parse_status never depended on it, and it is the environment-sensitive cost behind the 40x rebuild variance and check RSS ballooning in #140 - bump extractor version to typescript-5.9-v2 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f * fix(graph): free web-tree-sitter trees; tree-sitter fallback grammars Trees live in the Emscripten WASM heap and are never reclaimed by JS GC; every parse leaked for the life of the process (twice per file: extraction + fingerprint tokens). Free them at the same boundary that narrows the concrete Tree to the frozen TSTree. Also load grammars for compiler-language files the compiler could not stage, so the new crash-isolation fallback actually extracts them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f * fix(check): read-only grounding runtime; add mex graph repair mex check silently re-staged the entire corpus (TypeScript programs included) whenever any source mtime changed - a drift check paying full rebuild wall-clock and RSS (#140 obs 2). check now opens the last published graph read-only and reports how many source files it is behind instead. sync/setup/ground keep the writer path. mex graph repair checkpoints a stranded WAL (killed writer) and runs integrity_check without rebuilding (#140 obs 3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f * test(graph): cover #140 fixes; align grounding tests with read-only check - end-to-end crash isolation: a throwing programFactory still yields a complete build with tree-sitter nodes (engine threads CompilerExtractionOptions for fault injection) - read-only runtime: byte-identical store after check, staleness counted - repair: SIGKILLed writer strands a real WAL; repair checkpoints it and the stranded write survives - grounding integration/migration/e2e tests refresh the graph before expecting GROUNDING_DRIFT, matching check's new read-only contract Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f * test: explicit timeouts for graph-building tests; list graph repair in help Full-build tests ran at ~80% of the 5s default on Windows and flaked under parallel suite load; give them explicit budgets. Local failures are now a strict subset of main's pre-existing set on this machine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f * fix(graph): ordinal-disambiguate duplicate identities in TS-family tree extractor The tree-sitter TS/JS walker never handled two same-identity declarations in one file (python/rust already did) because compiler extraction always covered those files. The #140 crash-isolation fallback exposed it: staging the TypeScript repo aborted on 'duplicate node id'. Port the same ordinal pattern; ordinal 0 keeps byte-identical ids, so existing graphs are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f * perf(graph): process TypeScript projects sequentially, one program alive at a time The extractor held every project's ts.Program + TypeChecker simultaneously through drafts, identity, imports, and references. Peak RSS was the SUM of all programs — the dominant remaining memory term on multi-tsconfig repos (#140 memory-ceiling ask). Restructured into capture + finish: - projects are processed in ownership-priority order (most specific config dir first — the same comparator the per-file probe-then-sort applied), so greedy claiming picks the identical owner without comparing live programs - everything AST- or checker-derived (positions, texts, symbol declaration locations, polymorphism, invocation proofs) is captured while a project's program is alive; the program is then released - cross-project linkage already flowed through declaration-location strings (path:offset:kind), stable across programs for the same bytes; only those string lookups into the corpus-wide location-to-id map are deferred to a compiler-free finishing pass that replays the exact id/status/candidate/ skip/sort/hash logic of the concurrent implementation Measured on a 92-tsconfig repo (3,254 files): peak RSS 5.17 GB -> 2.11 GB (-59%), wall 448s -> 309s (-31%), output byte-identical (55,069 nodes / 93,213 edges; single-tsconfig repo also byte-identical at 22,645 / 46,820). One deliberate crash-path refinement: a healthy, more specific project's completed extraction is no longer discarded when a less specific project containing the same file crashes later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f * perf(graph): schema v3 — compact fingerprint/LSH encoding, ~40% smaller stores The v2 encoding made the fingerprint subsystem 40-50% of every graph store: minhash as JSON text (~600 B where 256 B suffice), 32 LSH rows per node each repeating the full TEXT node id plus a 64-char hex band hash, duplicated again by idx_lsh (#140 storage follow-up). v3 stores the minhash as a 256-byte big-endian BLOB, keys LSH rows on a VACUUM-stable INTEGER ref, truncates band hashes to int64 (a truncation collision merely adds one LSH candidate, which full minhash scoring then rejects - it can never remove or reorder a true candidate), and lets the composite primary key serve as the only index (WITHOUT ROWID, idx_lsh gone). Decoded fingerprints, serialized mh:64 grounding anchors, and every reconciler outcome are unchanged. v2 stores migrate LOSSLESSLY on any writer open (mex graph / sync / ground) without being marked rebuild-required - existing groundings keep working immediately; read-only commands on an unmigrated store report the standard rebuild guidance until then. Covered by a round-trip test that hand-encodes a real store back to v2 and asserts identical fingerprints and LSH hits after migration. Measured (identical node/edge/fingerprint/LSH-row counts throughout): - 494-file repo: 269.8 MB -> 162.9 MB (-40%); fingerprint tables 124 MB -> 17 MB - 3,254-file repo: 700.1 MB -> 451.1 MB (-36%); fingerprint tables 289 MB -> ~38 MB Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f * chore(release): 0.7.3 Version bump, changelog, release notes, and README updates for the graph performance and recovery release: read-only check, sequential compiler program release, schema v3 compact fingerprint/LSH encoding, mex graph repair, TypeScript program crash isolation, and the tree-sitter WASM leak fix. Also teaches the evaluator integrity harness the v3 fingerprint encoding. Its determinism hash and schema assertion were pinned to v2, and under v3 the lsh_buckets node_id column no longer exists, so the generic column filter would have silently dropped node identity from the hashed content exactly where the encoding changed. Bucket rows now join back through the fingerprint ref so the hash keeps meaning 'which node sits in which band bucket' (the storage ref itself is deliberately not hashed), the minhash is projected through hex() so BLOB and legacy TEXT sketches hash identically in form, and the schema-version assertion follows the schema to 3. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YQ1wQdymBBK9ZgsXpBC4f --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
theDakshJaitly
marked this pull request as ready for review
August 29, 2026 03:41
theDakshJaitly
merged commit Aug 29, 2026
8e9521c
into
integration/human-team-memory-v1
7 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Compatibility
Ordinary reads never migrate. Explicit repair can losslessly upgrade recognized v2 and complete v3 lineages; v1, partial, ambiguous, or malformed stores require a safe rebuild. The released 0.7.3 notes remain historical, while integration-only A-E and schema-v4 work stay under Unreleased.
Local verification
CI expectations
Required Node 22, Node 24, macOS/Windows storage portability, Hub browser, and pinned release-performance checks must pass before merge.
Targets integration/human-team-memory-v1 only. No main PR, tag, deployment, setup UI, Relay, Playbook, or Catch Up work is included.