Add ignoreWhitespace search flag - #800
Open
LazyCompiler wants to merge 1 commit into
Open
Conversation
A literal query with ignoreWhitespace drops whitespace on both sides of the fold instead of collapsing it, so "invoice" finds the letter-spaced "i n v o i c e" that OCR'd scans and tracked-out headings produce, and "total amount" finds "totalamount". Hits span the original text including the dropped whitespace; with wholeWord the boundaries are read off the original text, since the folded plane has no word gaps left. Like matchDiacritics the flag is literal-only: regex + ignoreWhitespace is rejected with InvalidArg (ignore-whitespace-with-regex). The flag is part of the cursor/query identity, rides the search token, and is accepted by the cloud search route as ignoreWhitespace=true. Adds letter_spaced_text.pdf (authored by a committed deterministic generator) and a local-engine test that exercises the flag against real page text: letter-spaced hits on both pages, a word split across a line break, a glued word found by a spaced needle, wholeWord boundaries from the original text, and matchCase composition.
|
@LazyCompiler is attempting to deploy a commit to the CloudPDF Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
ignoreWhitespaceflag toSearchQuery. A literal query drops whitespace on both sides of the fold instead of collapsing it, soinvoicefinds the letter-spacedi n v o i c ethat OCR'd scans and tracked-out headings produce, andtotal amountfindstotalamount.Live demo: https://lazycompiler.github.io/embedpdf-ignore-whitespace-demo/ — a static build of this branch (
@embedpdf/engine, wasm in a worker) against the fixture that ships with the change. Searchinvoice, then tickignoreWhitespace, thenwholeWord; trytotal amount.Design
FoldOptions.dropWhitespaceskips whitespace units infoldText; the existingmapalready carries hits back to exact original ranges, so a match spans the page's whitespace. The default fold is unchanged (noSEARCH_FOLD_VERSIONbump). LikematchCase/matchDiacriticsthe flag is a non-default fold:SearchReaderre-folds the raw page text per query andcanonicalSearchQuerypasses the query through.wholeWordboundaries on the original text. With whitespace dropped,i n v o i c e 42folds toinvoice42, so a boundary check on the folded plane would reject a genuine whole word. UnderignoreWhitespace,isWholeWordHitreads the boundaries offhaystack.originalaround the mapped range; the default path is untouched.regex: true+ignoreWhitespaceis rejected byvalidateSearchQuerywith a new issueignore-whitespace-with-regex, mirroringmatchDiacritics(regex runs on the raw text plane; a pattern spells its own\s*).searchCursor.ts,CloudDocumentSearchService.queryIdentity), the zodSearchQuerySchema, the search token (SearchTokenSchema+ encode/decode), and the cloud search route (ignoreWhitespace=true).matchDiacritics); happy to add a toggle if wanted.Tests
engine-core):fold(drop + decomposition-born whitespace),literal(spaced page text, spaced needle, glued words, line wraps,matchCaseandwholeWordinterplay, whitespace-only needle),regexvalidation,canonicalSearchQuery, token round-trip.runSearchConformance): the flag can only add hits over the default fold and the space-free needle finds the same hits; the regex combo is rejected withInvalidArg.packages/engine/main/test/search-ignore-whitespace.test.tsruns againsttest/fixtures/letter_spaced_text.pdf— a 2 KB fixture authored by a committed deterministic generator (fixtures/generate-letter-spaced-fixture.mjs, documented infixtures/README.md, following theaction_payloads.pdfpattern). It covers letter-spaced hits on both pages with drawable geometry, a word split across a line break, a spaced needle finding a glued word,wholeWordboundaries from the original text, andmatchCasecomposition. Happy to drop the fixture and keep only the unit + conformance tests if you'd rather not grow the fixture set.@embedpdf/enginesuite (45 files / 332 tests) andengine-core(436 tests) pass;engine-servicesand@cloudpdf/enginetypecheck.Changeset included (minor:
@embedpdf/engine-core,@embedpdf/engine-services,@cloudpdf/engine,@cloudpdf/server).