fix: scope semantic search in the query, not in its results (#323) - #340
Open
ProfSynapse wants to merge 3 commits into
Open
fix: scope semantic search in the query, not in its results (#323)#340ProfSynapse wants to merge 3 commits into
ProfSynapse wants to merge 3 commits into
Conversation
`search content --semantic` with `paths` set could return zero results while matching notes sat in the requested folder, with nothing in the response to say the scope was the cause. Three cuts ran before `paths` was ever consulted. The tool asked the service for `limit * 2`; `NoteEmbeddingService` answered by taking `ORDER BY distance LIMIT limit * 3` over the whole vault and re-ranking that; the tool then sliced the survivors back to `limit`. At the default limit of 10 the scope was applied to a 60-row global window, so any note ranking below it was unreachable no matter how well it matched. Raising the multipliers only makes that rarer, and passing a huge limit trades the bug for a whole-vault re-rank. The scope has to reach the query, so `semanticSearch` now takes optional `pathPrefixes` and confines the candidate scan with `WHERE em.notePath LIKE ? ESCAPE '\'`. The prefixes are escaped: `_` is a LIKE wildcard and `_Base/` is a real folder name. `searchContent` reduces each `paths` entry to its fixed prefix — a literal path is one already, a glob is cut at the last folder boundary before its first metacharacter. Anything that does not reduce (a leading wildcard, a bare `/`) leaves the query unscoped, and the existing post-filter stays as the second pass that does the real matching. The parameter is optional and defaults to the previous unscoped query, so the other caller (`noteSearch.searchNotes`) is unaffected. One consequence worth stating: with the scope in the query, an empty candidate set now means the folder holds no embedded notes, so that returns an empty result rather than the "issue with the vector database" error, which would otherwise be a false alarm the fix itself introduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C6aSoCAS5gNoew6n9qv6DJ
`searchContent` matched a `paths` entry with a bare `startsWith`, which is a string test rather than a folder test. Scoping to `_Base` therefore also returned everything under `_Baseball/`, and the extra results are indistinguishable from legitimate hits — the caller sees more results, not wrong ones. `isWithinPathScope` compares at a separator instead: a path matches when it IS the scope, or when it continues past it with a `/`. A trailing slash on the scope means the same thing, and the empty scope that `"/"` normalizes to still means the whole vault. Both branches of the tool use it, since the keyword filter carried the same comparison. The semantic branch's SQL prefix from the previous commit is deliberately left unanchored — it is a superset that stops the candidate window being spent out of scope, and this filter is the pass that decides membership. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C6aSoCAS5gNoew6n9qv6DJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #323.
The problem is bigger than the issue says
There are three truncations, not one. At the default
limit: 10:searchContent.tsover-fetcheslimit * 2NoteEmbeddingService.semanticSearchrunsORDER BY distance LIMIT limit * 3with noWHEREclause at allranked.slice(0, limit), then thepathsfilter, then another sliceSo: 60 global candidates → rerank → 20 → then apply scope → 10. If the scoped notes rank below the global top 60, the caller gets zero results and no indication why.
This is also why raising the multiplier never fixed it — the unscoped cut sits underneath.
The fix
NoteEmbeddingService.semanticSearchtakes an optionalpathPrefixesand addsWHERE em.notePath LIKE ? ESCAPE '\'(OR'd) to the candidate scan.EmbeddingServicepasses it through.searchContentreduces eachpathsentry to its fixed prefix — a glob is cut at the last/before the first metacharacter — and an entry that cannot reduce leaves the query unscoped with the post-filter in charge.The SQL prefix is deliberately a superset; the post-filter still decides membership.
%,_and\are escaped, which matters here because_is a LIKE wildcard and_Base/is a real folder name in this vault.Second commit: the literal path match was an unanchored
startsWith, so scoping to_Basealso matched_Baseball/. NewisWithinPathScopeinpathUtils.ts, applied to both the semantic post-filter and the keyword pre-filter.One deliberate behaviour change: a scoped query with no candidates now returns
success: true, results: []instead of an "issue with the vector database" error, which would otherwise be a false alarm this fix introduced.The test has the failing shape
tests/unit/SearchContentSemanticScope.test.tsdrives the real chain — tool →EmbeddingService→NoteEmbeddingService, real limits, real re-rank, real post-filter. Only the embedding engine and SQLite are faked, the latter a small interpreter for the exact query issued, honouring LIKE wildcards and the escape character.80 decoy notes outrank everything scoped, putting the three
_Base/notes at global ranks 82–84, past the 60-row window. A guard test asserts that property from the fixture itself, so it cannot silently drift into the window and stop meaning anything._Baseball/roster.mdsits closer than every_Base/note, so an anchoring leak surfaces at the top.Verified against pre-fix source (
git checkout <base> -- src/): 8 failures, including the headline case returning exactly[]— the silent zero from the issue.Verification
npx jest tests/unit— 327 suites, 4319 tests passingnpm run buildclean (lint, mobile-import gate, tsc, CLI + connector)noteSearch.tscalls with two arguments and a test pins that an unscoped call still issues aWHERE-less query;NoteEmbeddingQueryAdapter.test.tsreads the query buffer at param index 0, so the LIKE params were appended before the limit rather than prependedKnown limitation, pinned by a test rather than papered over
A glob with a wildcard in its first segment (
**/ledger.md) cannot reduce to a prefix and is still subject to the global window. Closing that needs a whole-vault re-rank, which changes ranking.🤖 Generated with Claude Code
https://claude.ai/code/session_01C6aSoCAS5gNoew6n9qv6DJ
Generated by Claude Code