Skip to content

Add validated snapshot-only metadata export - #154

Open
arkmish wants to merge 2 commits into
armishra/zk-snapshot-summaryfrom
armishra/zk-snapshot-export
Open

arkmish wants to merge 2 commits into
armishra/zk-snapshot-summaryfrom
armishra/zk-snapshot-export

Conversation

@arkmish

@arkmish arkmish commented Sep 28, 2026 •

Copy link
Copy Markdown

Description

Add OfflineAuditExporter, a machine-readable snapshot metadata exporter for automated size, fanout, namespace, and session analysis. It uses ZooKeeper's native decoder, avoids live ensemble queries, and does not export znode payloads or credential-bearing ACL identities.

bin/zkOfflineAudit.sh --snapshot-file snapshot.1 --output-dir new-export

The command produces four versioned files:

File Contents
nodes.ndjson Paths, payload lengths, immediate-child counts, response-size estimates, persisted metadata, node types, and non-secret ACL review flags
namespaces.ndjson Complete top-level customer namespace totals and exact-namespace quota metadata availability
sessions.ndjson Usage summaries for owners unambiguously classified as normal ephemeral sessions
manifest.json Source identity, decoder settings, checksums, record counts, and explicit completion/uncertainty fields

Traversal is deterministic depth-first preorder, keeping each subtree contiguous without retaining all output records. The native DataTree is still loaded into memory.

The exporter validates snapshot format/seals and source identity, rejects existing destinations and source/output aliases, and supports an explicit output-byte budget. It publishes the completion manifest atomically only after output files close successfully and the source is rechecked.

Unknown capture times and ambiguous owner encodings remain unknown. TTL/container owners are not blindly counted as sessions. A completed export is not proof of current state, session liveness, backup completeness, or transaction consistency.

Tests

Validated OfflineAuditExporterTest, SnapshotComparerTest, SnapshotRecursiveSummaryTest, SnapStreamTest, and EphemeralTypeTest on JDK 11 with Java 8 API targeting:

Tests run: 74, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

Regressions cover actual native serialization, compressed/corrupt inputs, protocol response-byte boundaries, owner/configuration uncertainty, deep traversal, source mutation, filesystem aliases, output limits, and failure-safe publication.

Actual Java 8 VM and Windows execution were not available.

  • Local code review completed

Changes that Break Backward Compatibility (Optional)

No client protocol, persistence format, recovery, ACL, or quota-enforcement changes. Existing snapshot-tool CLI behavior is preserved.

The new command requires a new output directory and rejects malformed or unsupported input. It is strictly snapshot-only: no transaction-log replay or capture-time inference is performed.

Documentation (Optional)

Updated zookeeper-docs/src/main/resources/markdown/zookeeperTools.md with the CLI, exact output schemas, decoder configuration, resource requirements, and evidence limitations.

🤖 Generated with GitHub Copilot CLI

Stream versioned node, namespace and true-session metadata without values or credential-bearing ACL IDs. Validate the selected source, enforce optional output budgets and publish a checksummed manifest atomically last.

Reuse the D1 native snapshot loader and expose iterative preorder traversal without changing the summary CLI. Keep capture and owner uncertainty explicit; do not replay transaction logs or infer recovery completeness.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Carry current branch-3.6 through the existing summary dependency without importing unrelated feature stacks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request snapshot audit Metadata-based snapshot audits, exports, and analysis

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant