Add SentinelLocalIndex.from_texts() to build an index in one line - #35
Merged
Conversation
4 tasks
wxiao0421
force-pushed
the
feat/persist-corpus-and-deterministic-load
branch
from
July 29, 2026 20:07
d64f37e to
cb513d9
Compare
wxiao0421
force-pushed
the
feat/index-from-texts
branch
from
July 29, 2026 20:07
983dd9b to
d261206
Compare
wxiao0421
force-pushed
the
feat/index-from-texts
branch
from
July 30, 2026 16:53
d261206 to
9241569
Compare
Base automatically changed from
feat/persist-corpus-and-deterministic-load
to
main
July 30, 2026 20:38
wxiao0421
force-pushed
the
feat/index-from-texts
branch
from
July 30, 2026 20:54
9241569 to
564d4d5
Compare
vcai4071
approved these changes
Jul 30, 2026
vcai4071
left a comment
Contributor
There was a problem hiding this comment.
Looks good overall. Some comments that may be worth addressing.
| ) | ||
| ) | ||
|
|
||
| if neg_to_pos_ratio is not None: |
Contributor
There was a problem hiding this comment.
This seems to be repeated code with subsample? Can we reuse some code here?
Contributor
Author
There was a problem hiding this comment.
Thanks for the suggestion. Fixed .
| **encoding_kwargs, | ||
| ) | ||
| ) | ||
| negative_embeddings = torch.tensor( |
Contributor
There was a problem hiding this comment.
would it be worth downsampling before embedding to save some cost?
Contributor
Author
There was a problem hiding this comment.
Thanks for the suggestion. Fixed it improve performance.
Building an index by hand takes eight steps, two of which fail silently when skipped: omit normalize_embeddings=True and the similarity maths quietly returns wrong numbers, and omit the corpus and explanations degrade to row numbers. No crash, no warning either way. Doing those steps inside the library, where they are tested, means a caller cannot forget a step they never have to write. Also corrects the README's save-format section, which described corpus.json as optional and written only when the index has a corpus. Both became false when that file started being written unconditionally, and the file's contents are what is optional now. Co-authored-by: Cursor <cursoragent@cursor.com>
…ogic Two review points on from_texts, addressed together because the fix for one decides the shape of the other. from_texts encoded every negative it was given and only then applied the ratio, so anything above the ratio was paid for and thrown away. Encoding is per-text and is the only expensive step here: at a 1:1 ratio against 1,000 positives, a caller passing 100,000 negatives paid to encode 99,000 rows that never reached the index. The choice of which negatives to keep depends only on the positive count and the ratio, both known before encoding, and encoding is per-text, so selecting first yields the same index for a fraction of the cost. Selecting first means selecting texts rather than embedding rows, so the piece worth sharing with subsample() is the choice of positions, not the copy. That is now _choose_indices, used by both, with _select_subset built on top of it for the embedding case. _select_subset also moves to module level, since it never used self and a classmethod cannot reach an instance method. Consolidating matters beyond tidiness: keeping each embedding beside its own text is the one rule in this module that fails silently when broken, so it should live in one place. Adds tests asserting that only the kept negatives ever reach the encoder, and that the no-ratio path still encodes everything. The first fails on the previous encode-then-discard order. _apply_negative_ratio has a third copy of this logic, left alone deliberately: it mutates in place and warns rather than raising, so folding it in would mean re-reviewing merged load behaviour from this PR. Reported in review by vcai4071. Co-authored-by: Cursor <cursoragent@cursor.com>
wxiao0421
force-pushed
the
feat/index-from-texts
branch
from
August 3, 2026 22:19
564d4d5 to
093a174
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Building an index today is eight manual steps (the README's "Creating a New Index" recipe), and two of them fail silently when skipped:
normalize_embeddings=True→ the similarity maths quietly returns wrong numbers.Neither raises. Neither warns. You just get subtly wrong or unexplainable results, and nothing tells you why. That is the real cost here, not keystrokes: doing these steps inside the library, where they are tested, means a caller cannot forget a step they never have to write.
Verified end to end — the resulting index scores correctly, its embeddings are unit-length, and its explanations name real sentences that survive a save/load round trip:
Implementation details worth reviewing
get_sentence_transformer_and_scaling_fn. Droppingscale_fnis a silent scoring bug for models like E5.DEFAULT_ENCODING_KWARGS, now used by both the constructor andfrom_texts, rather than writingnormalize_embeddings=Truein a second place. Two copies of a default eventually disagree, and this one disagreeing would be invisible.neg_to_pos_ratiovia the shared_take_rowshelper from Persist the corpus and make index loading deterministic #32, with the same private-torch.Generatorseeding convention used everywhere else in the library.positive_texts="some text"would otherwise be encoded one character at a time: confusing, slow, and entirely silent.What breaks if this is wrong
normalize_embeddingsstopped being applied, every index built this way would score incorrectly with no error anywhere. Asserted directly by checking the embedding norms are 1.0, rather than just checking the kwarg was passed.scale_fnwere dropped, scores change silently for scaled models.Nothing existing changes behaviour: this is a new classmethod, plus a refactor that moves an existing default into a named constant without altering its value.
Docs
README's Creating a New Index now leads with the one-liner. The manual recipe is kept directly below under "Advanced: building an index manually" for anyone who needs custom encoding, a different backend, or precomputed vectors — with its two silent traps called out explicitly so the advanced path is safer too.
Test plan
12 new tests.
neg_to_pos_ratiodownsamples and the surviving negatives keep their own textpytest tests/— 79 passed (67 on the base branch)flake8 --config=.flake8output byte-identical to base: zero new findingsdocs/check_docs_sync.pyreports in syncThe validation tests run without loading a model, since they raise before any encoding; the rest are marked
integrationin line with the existing suite.Note on CI:
.github/workflows/test.ymlonly triggers on pull requests targetingmain, so stacked PRs show no checks. Run locally against Python 3.10.