feat: add sharded frame-tree analysis and lightweight cache reuse

.github/scripts/clean_wheel_agent_acceptance.py

@@ -197,6 +197,52 @@ def _dataset_query(
     return scope
 
 
+def _event_parity(crossing: list[dict[str, Any]], reference: list[dict[str, Any]]) -> None:
+    """Compare stable event values and validate dataset-only ordered evidence."""
+    fields = (
+        "id",
+        "action",
+        "address",
+        "size",
+        "stream",
+        "allocated",
+        "active",
+        "reserved",
+        "callstack",
+    )
+    _require(
+        len(crossing) == len(reference) == 2
+        and all(all(key in row for key in fields) for row in [*crossing, *reference]),
+        "Cross-shard real events have missing rows or fields",
+    )
+    _require(
+        [{key: row[key] for key in fields} for row in crossing]
+        == [{key: row[key] for key in fields} for row in reference],
+        "Cross-shard real events differ from standalone",
+    )
+    for row in crossing:
+        frames = row.get("frames")
+        _require(
+            row.get("frames_status") == "ordered"
+            and isinstance(frames, list)
+            and all(
+                isinstance(frame, dict)
+                and isinstance(frame.get("filename"), str)
+                and type(frame.get("line")) is int
+                and isinstance(frame.get("name"), str)
+                for frame in frames
+            ),
+            "Cross-shard ordered frame evidence is incomplete",
+        )
+        _require(
+            "\n".join(
+                f"{frame['filename']}:{frame['line']} {frame['name']}" for frame in reversed(frames)
+            )
+            == row["callstack"],
+            "Cross-shard ordered frames differ from standalone stack text",
+        )
+
+
 def _sharded_acceptance(
     pt_snap: Path,
     fixture: Path,
@@ -358,10 +404,7 @@ def cli(label: str, *args: object) -> dict[str, Any]:
             scope.get("slice_indices") == [slices[0]["index"], slices[1]["index"]],
             "Event range did not cross shards",
         )
-        _require(
-            crossing["rows"] == reference["rows"] and len(crossing["rows"]) == 2,
-            "Cross-shard real events differ from standalone",
-        )
+        _event_parity(crossing["rows"], reference["rows"])
         _require(
             crossing.get("total_is_exact") is True and crossing.get("total") == 2,
             "Cross-shard exact total is invalid",

.github/workflows/AGENTS.md

@@ -97,12 +97,11 @@ The required installed-wheel case creates a clean venv without system packages
 and runs its actual `pt-snap` console script from isolated HOME/CWD. Its trusted,
 test-owned input has exactly 1,000,000 events; the default exporter produces two
 full 500,000-event shards. All installed catalog entries are classified: currently
-11 dataset-supported templates succeed with installed CLI/API equivalence and
-independent known-result checks, while `event_frames` and
-`active_memory_frame_tree_at_event` return explicit unsupported-dataset errors.
-Both report commands run: `peak-memory` succeeds and `memory-tree` preserves the
-corresponding dataset rejection. The allocation/free-request/free-completion
-boundary assertions remain covered, alongside static occupancy, offset pages,
+12 dataset-supported templates succeed with installed CLI/API equivalence and
+independent known-result checks; `event_frames` remains the explicit
+unsupported-dataset query. Both `peak-memory` and `memory-tree` reports succeed.
+The allocation/free-request/free-completion boundary assertions remain covered,
+alongside static occupancy, offset pages,
 terminal survivors and unchanged source/manifest/member hashes. A second required
 nightly case separately imports 500,002 events and proves the original final
 short shard has exactly two real events, with the crossing completed lifetime

README.md

@@ -124,6 +124,11 @@ byte units, allocated/active/reserved counters, expected output and partial repo
 - [Replayable pickle splitting](docs/en/splitting.md): one split strategy and a new output directory; separate from dataset import.
 - [Complete-dataset addressing](docs/en/sharded-snapshotdb.md#complete-dataset-focus-and-addressing-p1), [global built-in support](docs/en/querying.md#dataset-global-built-in-support-p3), and [recorded interoperability acceptance](docs/en/interop-acceptance.md): advanced contracts and evidence, not prerequisites for a first report.
 
+Current compatible exports use parser v3; the pinned v2 revision is a historical
+schema reference, with no verified v3 GUI consumer revision yet. Compatible reuse
+checks source identity, member inventory and required structure, without comparing
+database content digests. See the protocol contract above for limits and evidence.
+
 ## Built-in Query Templates & Reports
 
 Use `pt-snap query --list` or `pt-snap capabilities --json` to inspect all 13 built-in templates and their parameter/output schemas:

README_zh.md

@@ -121,6 +121,10 @@ allocated/active/reserved 指标、预期输出和报告不完整的情况。
 - [可回放 pickle 拆分](docs/zh/splitting.md)：只选一种拆分策略并使用新目录，与数据集导入不同。
 - [完整数据集寻址](docs/zh/sharded-snapshotdb.md#完整数据集-focus-与寻址p1)、[全局内建支持](docs/zh/querying.md#数据集全局内建支持p3)和[已记录的互操作验收](docs/zh/interop-acceptance.md)：按需查看高级合同与证据，不是第一份报告的前置条件。
 
+当前兼容导出使用 parser v3，固定的旧 v2 revision 仅为历史 schema 参考，尚无已核验的
+v3 GUI 消费端固定版本。兼容复用检查源身份、成员清单及必要结构，不比较数据库内容摘要。
+限制与验收证据见上方协议合同。
+
 ## 内置查询模板与报告
 
 可执行 `pt-snap query --list` 或 `pt-snap capabilities --json` 查看全部 13 个内置查询模板及其参数与输出契约：

docs/en/database.md

@@ -381,8 +381,8 @@ format and rebuilds the cache; read-only analysis never migrates a database.
 ### Standalone frame capability boundary
 
 Format 3 applies only to standalone imports. Native sharded imports remain format 2
-and compatible exports remain format 1; neither claims the standalone frame-tree
-capability. `callstack_frame_manifest(callstackId INTEGER PRIMARY KEY,
+and compatible exports remain format 1. A separate version-2 deduplicated frame
+extension supports dataset trees; see [querying](querying.md#deduplicated-dataset-frame-extension-version-2). `callstack_frame_manifest(callstackId INTEGER PRIMARY KEY,
 frameCount INTEGER NOT NULL)` records the expected length of every captured stack,
 including zero. Frame fields must be text filename/name and an integer line.
 `event_frames` returns no rows for absent, known-empty or corrupt arrays, never a
@@ -468,16 +468,20 @@ P0 compatibility-v1 table order, inline layout and manifest semantics are unchan
 `trace_entry_<device>` tables with the ninth column `callstack TEXT`, not views,
 with the exact positional block/dictionary schema. It removes staging's unprefixed
 `callstack` table and writes `pt_snap_metadata.import_format_version=1`, retaining
-isolated `pt_snap_block_reference` text lifecycle sources. Native standalone and
-native datasets keep their interning and v2 metadata. Structured ordered frames
-are not emitted or inferred from text.
+isolated `pt_snap_block_reference` text lifecycle sources. Native datasets keep their text interning and v2 metadata, plus the separate
+version-2 deduplicated frame extension; standalone imports use format 3. Frames are never inferred from text.
 
 Inline stack bytes scale with the sum of stack text **over every event**, not
 unique stacks; conversion temporarily holds native and inline tables and may leave
 SQLite free pages. Capacity does not bound blocks, loading RSS or total disk.
-Compatible datasets additionally carry `ptSnap.identity`, metadata, omitted devices
-and full member hashes. Recognized whole-artifact metadata is available only after
-all attestation checks; external v1 datasets still need no metadata.
+Compatible exports additionally carry `ptSnap.identity`, metadata, omitted devices
+and a structural member inventory with empty digest values. Historical digest
+fields are accepted as inventories; compatible import reuse compares member names,
+not digest values, alongside source identity and required read-only structure.
+It does not scan arbitrary row contents, so structurally valid content edits can
+be reused. Recognized metadata does not attest database content integrity.
+Queries and explicit validators perform independent admission checks; external v1
+datasets still need no metadata. Native dataset content verification is unchanged.
 
 For finalized compatibility datasets ONLY, bare validation and cached analysis
 use `mode=ro&immutable=1` after rejecting all live/dangling sidecars and aliases

docs/en/dataset-acceptance.md

@@ -9,10 +9,10 @@ fixture is not evidence that a producer, collector, or exporter ran.
 compatibility-v1 and native-v2 layouts; the nightly profile extends the same bank
 to 200 seeds. The original eight regression seeds and twenty query shapes remain.
 Each seed now runs 69–73 query cases per layout (2,896 paired comparisons for the
-20-seed bank; 28,864 for 200), covering all eleven dataset-supported templates.
-The two standalone-only templates, `event_frames` and
-`active_memory_frame_tree_at_event`, have separate explicit rejection tests and
-never count as successful differential comparisons.
+20-seed bank; 28,864 for 200), covering eleven text-query contracts. Dataset frame trees have a separate
+real-import single-DB differential suite in `test_dataset_frame_tree.py`, including
+recursion, cross-shard sources, corruption, budgets and read-only conservation.
+Only `event_frames` remains an explicit dataset rejection.
 
 Both layouts use the same independently generated 80-event history and a
 layout-matched standalone SQL oracle. Capacities 2/3/4/5/10/16 produce 5–40
@@ -131,10 +131,10 @@ separate empty/static/dynamic/mixed regressions across both layouts and reducers
 - A trusted, test-owned snapshot with 1,000,000 alloc/free-request/free-completion
   events goes through the clean-venv installed `pt-snap import --format msinsight`
   with **no capacity override**, producing two full 500,000-event members. All
-  installed catalog templates are classified: 11 dataset-supported templates
-  have CLI/API parity and independent result checks; `event_frames` and
-  `active_memory_frame_tree_at_event` retain explicit dataset rejection. The
-  `peak-memory` report succeeds; `memory-tree` retains its dataset rejection.
+  installed catalog templates are classified: 12 dataset-supported templates
+  have CLI/API parity and independent result checks; only `event_frames`
+  retains explicit dataset rejection. Both `peak-memory` and `memory-tree`
+  reports succeed, with root/self occupancy conservation checked.
 - A second required installed-wheel case preserves the original 500,002-event
   workload and its 500,000 + 2 member layout. It verifies the crossing completed
   lifetime, final survivor, pages and point-event results. Both installed cases

docs/en/interop-acceptance.md

@@ -4,7 +4,9 @@
 
 ## Evidence layers and fixed target
 
-The ONLY upstream target is **Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5**.
+The historical v2 acceptance target was **Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5**.
+This revision and `targetRevision` are historical schema/identity references, not
+a verified consumer for current v3 exports.
 Protocol/schema validation, isolated source-function checks, synthetic CI,
 reading an actual original-producer artifact, and actual GUI/C++ consumption are
 DIFFERENT evidence layers. None implies the next. **The bounded real GUI/C++ and
@@ -23,7 +25,27 @@ are separately CC BY 4.0, not automatically covered by this project's MIT lineag
 A port needs snapshot provenance/PR decisions and retained notices. Executable
 pickle trust is a SEPARATE fixture review; no new committed pickle is added here.
 
-## Forward chain: trusted source → physical export → real GUI
+## Current v3 consumer acceptance status
+
+No matching v3 consumer revision/build has been verified yet; current exports
+have no claimed GUI reuse support. The coordinated consumer must match
+`cacheHash = SHA256(b"mem_snapshot_parser_v3" + raw_pickle_bytes)` and treat
+`parserCache.members` as a structural inventory with empty member values,
+without requiring historical DB content digests. A `targetRevision` match alone
+does not establish these behaviors.
+
+Before claiming support, pin the consumer revision/build and verify actual cache
+reuse without producer rerun or unintended artifact changes. Use a separately
+trusted pickle copy and a new adjacent cache directory, preserving original
+artifacts. Do not open current v3 caches with the pinned old v2 GUI: it can
+reparse and rewrite the cache. The following steps record the historical v2
+procedure and are not instructions for accepting current CLI exports.
+
+## Historical v2 forward chain: trusted source → physical export → real GUI
+
+The pinned historical acceptance below used `mem_snapshot_parser_v2`. Current
+exports use `mem_snapshot_parser_v3`; the historical GUI result does not verify
+the new parser version. Re-export v2 caches to a new output directory for v3 reuse.
 
 1. Review/trust the source separately: pickle is executable input, not a sandbox.
    Record exact source SHA256, size, selected device and capacity. Default import

docs/en/querying.md

@@ -99,6 +99,7 @@ across devices and there is no shared cross-device event order.
 | `memory_peak` | Independent allocated/active/reserved maxima over real events; earliest real ID wins ties; `start_id/end_id` may cross shards |
 | `allocator_gap` | The same independently selected global events, with each gap computed from counters **at that event** |
 | `active_blocks_at_event`, `active_memory_callstack_at_event` | Route a real event to its owner and resolve full cross-shard sources before grouping/top_n (details below) |
+| `active_memory_frame_tree_at_event` | Event-owner live set with cross-shard allocation frames, grouped by complete caller paths; root/self totals conserve occupancy; no top_n pruning |
 | `preexisting_live` | Owner-event preexisting occupancy, not a leak count |
 | `block` | Merge proved allocation-event lifetimes and stable negative tokens once, then apply filters/order/page; latest containing-shard state is an observation, not state at arbitrary E |
 | `freed_block_lifetime` | Deduplicated captured allocations with a proved `free_completed` source; event-ID distance buckets, not time |
@@ -216,7 +217,7 @@ shard. Event/allocation pages apply SQL filters and ordering per shard, merge at
 `offset + limit` candidates per shard, and count matches separately. Range evidence
 is measured before non-range filters. Ordinary text-stack statistics merge full
 per-shard groups before global thresholds; averages use summed sizes and counts.
-Canonical ordered-frame statistics retain the bounded source-reading path because
+Version-1 canonical ordered-frame statistics retain the bounded source-reading path because
 formatted text is not their identity. Small block/leak/lifetime reductions use read-only
 attached shards for proof, conflicts and latest observations before pagination;
 the attachment limit (normally ten) counts selected observation shards plus any
@@ -372,9 +373,9 @@ those merge semantics merely by using a built-in name.
 
 ### Optional ordered-frame reader contract
 
-This reader-only pt-snap extension is new, not an existing original-producer
-feature. Current native/compatible writers emit **no** ordered-frame tables.
-Declare `extensions.ptSnapOrderedFrames={"version":1}` in the manifest and use
+This pt-snap extension is independent of original-producer capabilities. New
+native/compatible imports write the deduplicated version-2 extension described
+below. The version-1 reader remains supported. Declare `extensions.ptSnapOrderedFrames={"version":1}` in the manifest and use
 isolated actual tables in each covered source shard:
 
 - `pt_snap_frame_coverage(eventId INTEGER PRIMARY KEY, frameCount INTEGER)`
@@ -779,7 +780,7 @@ and `semantics_version` so that lookup stays tied to the rows.
 
 ## Frame Memory Breakdown
 
-After a standalone import with this version, inspect the original ordered frame array or
+After an import with this version, inspect the original ordered frame array or
 decompose live memory after an event:
 
 ```bash
@@ -830,15 +831,44 @@ node ID to distinguish identical labels. Lists include every match and support
 Tab then Enter/Space. Parent frame moves up one level; Reset zoom returns to the
 root without clearing the search. Clear the search field to remove the results.
 
-Frame-tree templates and `report memory-tree` support standalone format-3 databases
-only. Dataset roots are rejected explicitly; the analyzer never selects a shard
-implicitly. Direct native shards have no standalone frame tables: re-import the
-original snapshot as a standalone database to use these templates. Existing
-native `pt_snap_frame`/coverage readers remain unchanged. `--timeout` uses the
-shared query deadline. Text and HTML renderers reject incomplete or nonconserving
-trees. Known-empty stacks have an `empty_callstack` synthetic leaf;
-missing/corrupt stacks retain the `missing` leaf. `event_frames` returns no rows
-for missing events/stacks, known-empty stacks or invalid frame arrays.
+`active_memory_frame_tree_at_event` and `report memory-tree` support standalone
+format-3 databases and complete native/compatible datasets. Query the dataset root:
+the event owner supplies live blocks and allocation owners supply frames, including
+earlier slices. Local IDs and display text never identify cross-shard caller paths.
+`frame_id` is query-local; `node_id` hashes the full ordered path of
+`(filename, line, name)` triples. `scope.source_coverage.ordered_frames_complete`
+reports frame evidence separately from output row completeness. Missing,
+unknown-version or corrupt evidence keeps its bytes in `missing`; re-import the
+original snapshot to capture frames. Known-empty arrays have an `empty_callstack`
+leaf. Work budgets and the shared deadline cover the full operation; failures
+return no partial tree. Text and HTML renderers reject truncated/nonconserving trees.
+`event_frames` remains standalone-only; use dataset `event` with `id` for its
+covered original `frames`. Direct native shards lack standalone frame tables;
+query the dataset root for cross-shard allocation sources.
+
+### Deduplicated dataset frame extension (version 2)
+
+New imports declare `extensions.ptSnapOrderedFrames={"version":2}` and store:
+
+| Table | Columns |
+| --- | --- |
+| `pt_snap_frame_value` | `id INTEGER PRIMARY KEY, frameJson TEXT` |
+| `pt_snap_stack` | `stackId INTEGER PRIMARY KEY, frameCount INTEGER` |
+| `pt_snap_stack_frame` | `stackId INTEGER, frameIndex INTEGER, frameId INTEGER, PRIMARY KEY(stackId,frameIndex)` |
+| `pt_snap_event_stack` | `eventId INTEGER PRIMARY KEY, stackId INTEGER` |
+
+Each canonical raw frame JSON (including extra typed fields) and each distinct
+ordered frame-ID sequence is stored once per shard. Repeated frames retain
+positions; empty stacks have an explicit count zero. Only real events receive
+references. These independent IDs preserve native text interning and compatible
+inline schemas. Native extension cache identity is 2; compatible export identity
+is 3, preventing reuse of old text-only exports as frame-capable artifacts.
+Readers support versions 1/2 and validate exact schemas, contiguous positions,
+counts, references and finite JSON with nesting depth at most 128. Tree frames
+require text filename/name and integer line; invalid fields degrade to missing.
+No text reconstruction occurs. `callstack_analysis` reduces version-2 events by
+local stack and text in SQLite before resolving representative arrays and merging
+global identities; repeated stacks do not require fetching every event into Python.
 
 The Python API uses the same template and contracts: