v0.1.29: real-world PDFs that open on v0.1.23 now fail (CCITT, annotation appearance, ICCBased, CID font) — reproduced in the official demo

[sebgoubier]

First, thank you for the work released in v0.1.24 through v0.1.29 — replacing pdf.js/pdf-lib/jszip with the native engine is a big win on my side, and I want to lead with the measurement that made me upgrade:

On a 147 MB dense architectural plan, version v0.1.23 peaked at ~5.5 GB in the hepr-dense-pdf worker and never finished; your demo running v0.1.29 completed the same file at ~2.9 GB. Same bytes, same options, same dense-vector-worker path. Zero runtime dependencies and a 518 KB → 192 KB dense worker are excellent too.

Unfortunately I can't ship the upgrade yet: most of the real drawings I test with open on v0.1.23 and fail on v0.1.29, and all of the failures below reproduce in the official demo (https://soadzoor.github.io/Highly-Efficient-PDF-Renderer), so this isn't something about my integration.

Failures observed on v0.1.29 (all fine on v0.1.23)

Four different documents, four different errors:

1. CCITT Group 4 (scanned drawing)

[hepr] dense PDF fast path fallback: unsupported-content:
XObject /I1 is missing, is an image, or was not validated as a supported Form XObject.

then

CCITT vertical mode places a changing element outside the row.

2. Annotation appearance stream (MEP installation drawing)

Failed to render PDF: Visible /Square annotation 0 has no usable normal appearance;
native synthesis is not implemented.

3. ICC-based colour space (architectural specification exported from Revit)

Failed to render PDF: ICCBased color conversion requires a caller-owned ICC transform resolver.

4. CID font metadata (MEP standardisation document)

Failed to render PDF: CID font /CIDSystemInfo /Ordering must contain printable ASCII.

What they seem to have in common

Each one looks like a construct pdf.js tolerated and the native engine now rejects outright rather than degrading:

• a CCITT G4 decoder edge case in 2D vertical mode;
• an annotation with no usable /N appearance stream, where pdf.js synthesises one;
• an ICCBased colour space with no host-provided transform, where pdf.js falls back to the alternate space or the component count;
• a /CIDSystemInfo /Ordering string that isn't printable ASCII — malformed by spec, but very common in the wild.

Points 3 and 4 in particular are cases where the spec allows a documented fallback (/Alternate or /N for ICCBased; ignoring a malformed /Ordering), so failing the whole render may be stricter than it needs to be.

Question

Is the intent for these to be hard errors, or would a lenient path be in scope — either a per-feature fallback, or an option like strict: false that degrades (skip the annotation, use the alternate colour space, ignore malformed font metadata) instead of aborting the document?

For anything document-shaped, a page that renders slightly imperfectly is far more useful than a document that won't open at all, and v0.1.23 gave me that behaviour through pdf.js.

Environment

• @soadzoor/hepr v0.1.29 (regression from v0.1.23)
• Chrome, Windows 11
• Reproduced both in my own app and in the official demo, with the same files
• three.js 0.184, WebGL backend, default options (vectorLod: "auto", textLod: "auto")

Sample files

I'm attaching a reproducer for #1 below. The other three are real drawings I use as test data and don't own, so I'd rather not republish them here — happy to send them privately, or to try to cut them down to minimal reproducers if that's more useful to you.

[sebgoubier]

Housing_concrete_Domestic water_Network components_20240611.pdf

This is a small device-schedule drawing (125 KB, one page). It renders on
v0.1.23 and fails on v0.1.29, both in our app and in the official demo, with:

[hepr] dense PDF fast path fallback: unsupported-content:
XObject /I1 is missing, is an image, or was not validated as a
supported Form XObject.
CCITT vertical mode places a changing element outside the row.

[sebgoubier]

Follow-up on this issue: I went through all four items on real documents, found three more causes along the way, and opened seven pull requests. Since they are not all the same kind of change, here is what each one actually is — including the two you should push back on if you disagree.

Genuine defects — the code computes or classifies something wrong

#4 — CCITT b1 selection. This is the one from the top of this issue, and my original reading of it was wrong. The stream is conformant. findReferenceTransition returns the first changing element at or after a0, while T.4 4.2.1.3.2 defines b1 as the first one strictly to the right of a0. When the reference row changes colour exactly on a0, b1 comes back as a0 and any V_L(n) puts a1 behind it — so the throw was a correct reaction to a wrong b1. Proof rather than absence-of-error: with b1 fixed, that stream decodes 264/264 rows, consumes all 4819 encoded bytes, and lands exactly on its EOFB.

#10 — BaseFontBlend read as Multiple Master. Top DICT operator 12 23 is BaseFontBlend, a delta; the font that triggered this carries [408, -397]. It is not a Multiple Master marker and implies no blend axes. 12 20 SyntheticBase still rejects, which is substantive.

#9 — Repeated stitching Bounds. Bounds [0.14286, 0.14286] is an empty subdomain, not a disordered array. Your own selection — findIndex(bound => input < bound) — makes that segment unreachable, which is also why it cannot divide by zero. Note this validation exists twice, in nativeFunctions.ts and heprFunctionEvaluator.ts; fixing one only moves the message to the other.

Strictness beyond the specification, on inert bytes

#7 — A single trailing EOL byte. Two unrelated PDFs: complete zlib streams (8321 and 4075 bytes produced) followed by one 0x0d / one 0x0a. That is the EOL marker 7.3.8.1 places before endstream and excludes from /Length; the producers counted it. Accepted only after re-decoding without it proves the same output length, so a truncated stream still cannot be rescued. Your existing trailing-NUL case still rejects, deliberately.

#5 — NUL-padded /CIDSystemInfo. /Ordering arrives as "Identity\0\0" — a fixed-width string. Trimming rather than tolerating matters here: keeping "Japan1\0\0" would resolve no collection and silently degrade instead of failing loudly.

An addition inside your existing design

#6 — /Square appearance synthesis. The annotation that motivated it has /Border [0 0 0], no /C and no /IC — it paints nothing, yet failed the whole page. I followed the precedent already in nativeAppearanceSynthesis.ts, where a zero-width /Link border returns null as explicitly non-painting. The policy on resolveAnnotationAppearance is untouched: visible unsupported annotations still fail. /Circle is the twin in 12.5.6.8 and I left it out on purpose — say the word and I will add it.

A proposal, not a fix — push back freely

#8 — ICCBased without a resolver. This one changes behaviour three of your tests assert, and I say so at the top of the PR. What pushed me to open it anyway: BuildHepFromPdfOptions exposes no way to supply an iccTransformResolver or iccKernel, so the contract the error message asks for cannot be met from the public API — every ICCBased document fails through buildHep with no opt-in. 8.6.5.5 already defines the alternate space for exactly this, and parseIcc already resolves and validates it. A resolver that is present and fails still raises. If you would rather keep the strict contract, the other fix is to expose the resolver in the public options, and I am happy to write that instead.

The conclusion, which matters more than the seven PRs

Across eleven real documents, these fixes open three. The other eight get further and then stop, on limits of the rendering path rather than on format conformance:

• A selective source paint identity is ambiguous or missing.
• Legacy vector output cannot prove an arbitrarily clipped image underlay. (twice)
• Legacy vector output supports only fill and invisible text, not rendering mode 2
• Image 0 is not a directly renderable RGBA8 underlay.
• Visible annotation appearances are not yet representable by the direct VectorScene path.
• Gradient 0 cannot meet its color tolerance within the subdivision limit.
• ExtGState /GS1 /UCR2 is unsupported by native compositing.

Individually these all look reasonable. Together they are the actual blocker, and one observation stands out: compilePage and compileVectorPage have different limits, and neither picks up what the other refuses — so a document can be rejected by the path it happened to take. That is a design question, not a bug, which is why I have not touched any of it.

So: is there an intended last-resort path when a compilation path refuses a construct — and would you want the two paths to fall back to one another? That answer decides whether the remaining documents are reachable at all, far more than any further patch from me would.

I left /BG /BG2 /UCR /UCR2 /HT alone for the same reason. They are inert for sRGB output, but three tests assert their refusal and you reject /TR too, which genuinely does affect colour — so that reads as a deliberate policy, and it is yours to change, not mine.

Happy to split, rework, or close any of these.

[soadzoor]

Thank you for your feedback, and the PRs to make this tool better!

I'd like to clarify: the ability to open malformed/non-standard PDFs (even with visual glitches) is more preferrable to me than throwing an error when such a PDF is attempted to open. I made the necessary modifications to reflect this direction.

I believe I merged all your PRs (or closed some, but harvested the ideas from those as well with a bit of a different solution).

Please try the newest version (0.1.30) with your test PDFs, and let me know whether they render fine now (just like with 0.1.23). If not, please reach out to me, send me the problematic PDFs if you can (directly you can reach me at soadzoor at gmail), and I'll try my best to fix the problems as soon as I can.

Thanks again!