Skip to content

Add ignoreWhitespace search flag - #800

Open
LazyCompiler wants to merge 1 commit into
embedpdf:mainfrom
LazyCompiler:feature/search-ignore-whitespace
Open

Add ignoreWhitespace search flag#800
LazyCompiler wants to merge 1 commit into
embedpdf:mainfrom
LazyCompiler:feature/search-ignore-whitespace

Conversation

@LazyCompiler

@LazyCompiler LazyCompiler commented Sep 8, 2026

Copy link
Copy Markdown

Summary

Adds an opt-in ignoreWhitespace flag to SearchQuery. A literal query drops whitespace on both sides of the fold instead of collapsing it, so invoice finds the letter-spaced i n v o i c e that OCR'd scans and tracked-out headings produce, and total amount finds totalamount.

Live demo: https://lazycompiler.github.io/embedpdf-ignore-whitespace-demo/ — a static build of this branch (@embedpdf/engine, wasm in a worker) against the fixture that ships with the change. Search invoice, then tick ignoreWhitespace, then wholeWord; try total amount.

Design

  • A fold option, not a new matcher. FoldOptions.dropWhitespace skips whitespace units in foldText; the existing map already carries hits back to exact original ranges, so a match spans the page's whitespace. The default fold is unchanged (no SEARCH_FOLD_VERSION bump). Like matchCase / matchDiacritics the flag is a non-default fold: SearchReader re-folds the raw page text per query and canonicalSearchQuery passes the query through.
  • wholeWord boundaries on the original text. With whitespace dropped, i n v o i c e 42 folds to invoice42, so a boundary check on the folded plane would reject a genuine whole word. Under ignoreWhitespace, isWholeWordHit reads the boundaries off haystack.original around the mapped range; the default path is untouched.
  • Literal-only. regex: true + ignoreWhitespace is rejected by validateSearchQuery with a new issue ignore-whitespace-with-regex, mirroring matchDiacritics (regex runs on the raw text plane; a pattern spells its own \s*).
  • Identity / wire. Included in the cursor query key (searchCursor.ts, CloudDocumentSearchService.queryIdentity), the zod SearchQuerySchema, the search token (SearchTokenSchema + encode/decode), and the cloud search route (ignoreWhitespace=true).
  • Not exposed in the viewer-chrome search panel here (same stance as matchDiacritics); happy to add a toggle if wanted.

Tests

  • Unit (engine-core): fold (drop + decomposition-born whitespace), literal (spaced page text, spaced needle, glued words, line wraps, matchCase and wholeWord interplay, whitespace-only needle), regex validation, canonicalSearchQuery, token round-trip.
  • Conformance (runSearchConformance): the flag can only add hits over the default fold and the space-free needle finds the same hits; the regex combo is rejected with InvalidArg.
  • Real engine: packages/engine/main/test/search-ignore-whitespace.test.ts runs against test/fixtures/letter_spaced_text.pdf — a 2 KB fixture authored by a committed deterministic generator (fixtures/generate-letter-spaced-fixture.mjs, documented in fixtures/README.md, following the action_payloads.pdf pattern). It covers letter-spaced hits on both pages with drawable geometry, a word split across a line break, a spaced needle finding a glued word, wholeWord boundaries from the original text, and matchCase composition. Happy to drop the fixture and keep only the unit + conformance tests if you'd rather not grow the fixture set.
  • Verified locally against the wasm runtime: the full @embedpdf/engine suite (45 files / 332 tests) and engine-core (436 tests) pass; engine-services and @cloudpdf/engine typecheck.

Changeset included (minor: @embedpdf/engine-core, @embedpdf/engine-services, @cloudpdf/engine, @cloudpdf/server).

A literal query with ignoreWhitespace drops whitespace on both sides of
the fold instead of collapsing it, so "invoice" finds the letter-spaced
"i n v o i c e" that OCR'd scans and tracked-out headings produce, and
"total amount" finds "totalamount". Hits span the original text
including the dropped whitespace; with wholeWord the boundaries are read
off the original text, since the folded plane has no word gaps left.

Like matchDiacritics the flag is literal-only: regex + ignoreWhitespace
is rejected with InvalidArg (ignore-whitespace-with-regex). The flag is
part of the cursor/query identity, rides the search token, and is
accepted by the cloud search route as ignoreWhitespace=true.

Adds letter_spaced_text.pdf (authored by a committed deterministic
generator) and a local-engine test that exercises the flag against real
page text: letter-spaced hits on both pages, a word split across a line
break, a glued word found by a spaced needle, wholeWord boundaries from
the original text, and matchCase composition.
@vercel

vercel Bot commented Sep 8, 2026

Copy link
Copy Markdown

@LazyCompiler is attempting to deploy a commit to the CloudPDF Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant