Skip to content

feat(pdf): classic xref, indirect lengths and object bounds - #812

Open
sefanty17 wants to merge 10 commits into
T-Auto:mainfrom
sefanty17:work/pdf-container
Open

sefanty17 wants to merge 10 commits into
T-Auto:mainfrom
sefanty17:work/pdf-container

Conversation

@sefanty17

@sefanty17 sefanty17 commented Nov 9, 2026 •

Copy link
Copy Markdown

PR A1

Part of #811 — slice A1/7. Stack base: main at 033aba3; latest main 88ecc72 is merged as 7bd7c76.

The working A2..A7 stack lives in sefanty17/dsh-ops PRs #7..#13; this upstream PR is the A1 contribution.

What was wrong

ops_read_pdf assumed one classic cross-reference table and direct object values. Files with indirect /Length, /Size or /Count failed before any text was read.

The first review found four regressions: a missing or null /Contents became a parse error, the range verifier checked only the gap before the first object, the xref/trailer metadata no longer counted toward the file-wide budget, and a literal % such as (100%) on the same physical line as a later header was mistaken for a comment. The restored container suite did not pin the gap or blank-page cases.

The change

  • Parse classic xref tables and objects, with the existing syntax whitelist.
  • Resolve indirect /Length, /Size and /Count in either direction.
  • Lex every gap between constructs and before the xref table; only whitespace and % comments may appear. A comment that hides a header is refused, while a % inside a literal string stays data.
  • Count the xref section and trailer toward the 100,000-value metadata budget, and check the container deadline in the xref loop.
  • Keep /ObjStm and /XRef refused in this slice; the xref/object-stream slice relaxes them when it reads them.
  • Bound object parsing: 16 MiB file, 10,000 xref entries/objects, 1 MiB encoded stream, 20 s container sub-deadline inside the 30 s read deadline. The 8 MiB decoded xref/object-stream budget lands with slice A2.

Deliberately not done here: xref streams, object streams, the producer offset tolerance and the 8 MiB decoded xref/object-stream budget (slice A2).

Validation

  • one CI required run triggered by pull_request; the latest run is action_required (maintainer approval), URL below
  • npm run verify output from the approved run
  • the affected checks go red when this implementation is removed (gap, comment, blank-page, ledger tests)
  • docs/CHANGELOG.md ## [Unreleased] entry added
  • behaviour change mirrored in the bilingual docs
  • no new dependency, no change to package.json version or optionalDependencies
  • no model-facing description or parameter change in this slice
$ cargo test -p ops-documents --locked --manifest-path native/Cargo.toml
test result: ok. 30 passed; 0 failed
test result: ok. 3 passed; 0 failed (container_matrix)

$ node test/run.mjs pdf
pdf-host: all 5 checks passed
pdf-worker: all 15 checks passed
2 suite(s) passed

$ node test/run.mjs docs-facts manifest
docs-facts: all 7 checks passed
manifest: all 15 checks passed
2 suite(s) passed

$ npm run check:size
check-file-size: 825 code file(s) checked, all at or under 1000 lines (longest: native/crates/ops-search/src/search.rs (930))

$ gh api repos/T-Auto/dsh-ops/actions/runs/43542505665 --jq '.event, .conclusion'
pull_request
action_required

$ gh api repos/T-Auto/dsh-ops/actions/runs/43263831112 --jq '.head_sha, .event, .conclusion'
596d9f2...
pull_request
success

The run for the current head 7bd7c76 is action_required: a maintainer must approve it before it can run. The previous head 596d9f2 had a green pull_request run; these review fixes are the delta.

Environment

  • Windows 11 x64; Node v24.12.0; Rust 1.98.1.
  • Based on origin/main 033aba3; latest main 88ecc72 is merged as 7bd7c76.

@sefanty17 sefanty17 changed the title feat(pdf): resolve indirect length, size and count references [1/7] feat(pdf): classic xref, indirect lengths and object bounds Nov 9, 2026
@sefanty17 sefanty17 changed the title [1/7] feat(pdf): classic xref, indirect lengths and object bounds feat(pdf): classic xref, indirect lengths and object bounds Nov 9, 2026
@T-Auto

T-Auto commented Nov 11, 2026

Copy link
Copy Markdown
Owner

Review:请求修改

审查 head:30f431f;对照 main:8f79848。本轮为源码、测试差异及远端 CI 审查,未运行构建或测试。间接 Length/Size/Count 有价值,但以下回归需在本 slice 修复。

[P1] 合法空白页被拒绝

native/crates/ops-documents/src/pdf_pages.rs:181–192 用必填 get(Contents) 替代 optional lookup,并删除 Value::Null 分支。没有 /Contents 或 /Contents null 的页从空内容成功变为解析错误。现有 test/pdf-worker.test.mjs 已包含这两类正例;新增 Count 单测只遍历 pages,未读取 content。

请恢复 missing/null 返回空内容;用真实 worker 覆盖 missing、null、空数组、正常单流/多流和非法数组成员。

[P1] 范围检查遗漏对象间与 xref 前的未索引字节

pdf_container.rs:393–418 的 verify_ranges 仅检查第一个对象之前的 gap,再检查范围/重叠,没有检查后续每段 gap。完整 Size 或不重叠不能证明对象区完整解释,未索引构造/垃圾可能被忽略,破坏既有 no-recovery 契约。

请对排序后的每个 gap 做词法验证,只允许空白/注释。首部、中间、xref 前的未索引对象/垃圾应拒绝;空白/注释应接受;保留重叠、错误身份和注释覆盖对象头反例。

[P1] 删除已有防线测试,未等价迁移

pdf_container_tests.rs 从 339 行降至 14 行,删去 duplicate/escaped keys、xref generation/count/footer、未索引/overlap、全局节点/嵌套/字符串预算、逐字节截断/变异 panic-free、zlib 截断/校验和/尾随数据/解压上界等。新增 container_matrix.rs 的三个测试不是等价覆盖,common builder 也不是断言。

请恢复仍适用的旧测试,仅调整本次有意变化的间接 Length 等规则;搬迁须保留对应行为断言。

[P2] 注释判断误拒字符串中的百分号

pdf_container.rs:129–139 检查同一物理行前部是否含 %,会把 literal string 内的 (100%) 误当注释,例如同一行后接 endobj 2 0 obj。请改为 token-aware 边界判断,加入合法正例并保留真正注释引入对象的拒绝。

[P2] 预算承诺与实现不一致

正文宣称 8 MiB xref、30 秒;classic_section 无对应 section 字节预算或循环 deadline 检查;container 独立为 20 秒,后续层另为 30 秒,不能视为同一文件级上限。trailer 节点也没有计入从零开始的 Loader.nodes 总账。总文件/entry 限制仍在,但不是这些保证。

请统一/说明分层预算并补 at-limit/one-over 和跨 trailer/object 节点总账断言,或收窄本 slice 承诺。

合并门槛

修复回归并恢复安全矩阵;native workspace 中运行 cargo test -p ops-documents、真实 worker PDF suites、完整 verify/size;缺 worker skip 不算通过。当前 head pull_request CI 为 action_required,须有真实通过记录;补 Unreleased CHANGELOG 与双语行为文档。不要把本 slice 的回归交给未来 A2 修补。

Verify every gap between constructs and before the xref, counting only whitespace and comments, so an unindexed object or a comment that hides a header is never recovered. Count the xref section and trailer toward the file-wide metadata budget and check the container deadline inside the xref loop. Keep /ObjStm and /XRef refused until the xref-stream slice reads them.
Assert missing, null, empty-array and list-member page contents in the container unit tests and in the real worker suite, including the invalid member that must fail as a parse error.
Document indirect stream lengths, per-gap lexical verification, the xref/trailer share of the metadata budget, and the 20-second container sub-deadline inside the 30-second read deadline. Record the same bounds in the Unreleased changelog.
@sefanty17

Copy link
Copy Markdown
Author

CI status: the new head 7bd7c76 has a pull_request run in action_required (https://github.com/T-Auto/dsh-ops/actions/runs/43542505665); the previous head 596d9f2 passed (https://github.com/T-Auto/dsh-ops/actions/runs/43263831112). Could a maintainer approve the new run so the review fixes get a real CI record? The affected local suites (cargo 30+3, pdf 5+15, docs-facts/manifest 7+15, size 825) pass on the merged tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants