Skip to content

Add SentinelLocalIndex.from_texts() to build an index in one line - #35

Merged
wxiao0421 merged 2 commits into
mainfrom
feat/index-from-texts
Aug 3, 2026
Merged

Add SentinelLocalIndex.from_texts() to build an index in one line#35
wxiao0421 merged 2 commits into
mainfrom
feat/index-from-texts

Conversation

@wxiao0421

Copy link
Copy Markdown
Contributor

Stacked on #32 (→ #31). Base branch is feat/persist-corpus-and-deterministic-load.

This one branches off the stack rather than sitting on top of it. Its only real dependency is the _take_rows helper from #32, so basing it there lets it merge as soon as #32 lands, without waiting on #33 and #34.

Summary

Building an index today is eight manual steps (the README's "Creating a New Index" recipe), and two of them fail silently when skipped:

  • Forget normalize_embeddings=True → the similarity maths quietly returns wrong numbers.
  • Forget to pass the corpus → explanations degrade to row numbers.

Neither raises. Neither warns. You just get subtly wrong or unexplainable results, and nothing tells you why. That is the real cost here, not keystrokes: doing these steps inside the library, where they are tested, means a caller cannot forget a step they never have to write.

index = SentinelLocalIndex.from_texts(
    positive_texts=[...],
    negative_texts=[...],
    neg_to_pos_ratio=5.0,
    seed=42,
)

Verified end to end — the resulting index scores correctly, its embeddings are unit-length, and its explanations name real sentences that survive a save/load round trip:

built in one call: 3 pos / 3 neg
normalized: True
corpus kept: True

explanation names real sentences, not row numbers:
   [+] 'they are invading our nation'
   [+] 'our people are being replaced'

after save+load, corpus survives: True

Implementation details worth reviewing

  • Keeps both return values of get_sentence_transformer_and_scaling_fn. Dropping scale_fn is a silent scoring bug for models like E5.
  • Hoists the encoding defaults into DEFAULT_ENCODING_KWARGS, now used by both the constructor and from_texts, rather than writing normalize_embeddings=True in a second place. Two copies of a default eventually disagree, and this one disagreeing would be invisible.
  • Always passes the corpus through — half the point of the method.
  • Applies neg_to_pos_ratio via the shared _take_rows helper from Persist the corpus and make index loading deterministic #32, with the same private-torch.Generator seeding convention used everywhere else in the library.
  • Validates input before any encoding happens, including the bare-string case. A string is iterable, so positive_texts="some text" would otherwise be encoded one character at a time: confusing, slow, and entirely silent.

What breaks if this is wrong

  • If normalize_embeddings stopped being applied, every index built this way would score incorrectly with no error anywhere. Asserted directly by checking the embedding norms are 1.0, rather than just checking the kwarg was passed.
  • If scale_fn were dropped, scores change silently for scaled models.
  • If the corpus were not carried through, explanations silently revert to the row-number behaviour that Persist the corpus and make index loading deterministic #32 exists to fix.

Nothing existing changes behaviour: this is a new classmethod, plus a refactor that moves an existing default into a named constant without altering its value.

Docs

README's Creating a New Index now leads with the one-liner. The manual recipe is kept directly below under "Advanced: building an index manually" for anyone who needs custom encoding, a different backend, or precomputed vectors — with its two silent traps called out explicitly so the advanced path is safer too.

Test plan

12 new tests.

  • One call produces a usable index that scores text
  • The corpus is kept automatically
  • Embeddings are unit-length by default (catches a silent normalization regression)
  • neg_to_pos_ratio downsamples and the surviving negatives keep their own text
  • Same seed → identical index
  • The result saves and reloads with explanations intact
  • Bare string instead of a list raises, for both arguments
  • Empty list raises, for both arguments
  • Non-positive ratio raises
  • pytest tests/ — 79 passed (67 on the base branch)
  • flake8 --config=.flake8 output byte-identical to base: zero new findings
  • docs/check_docs_sync.py reports in sync

The validation tests run without loading a model, since they raise before any encoding; the rest are marked integration in line with the existing suite.

Note on CI: .github/workflows/test.yml only triggers on pull requests targeting main, so stacked PRs show no checks. Run locally against Python 3.10.

@wxiao0421
wxiao0421 force-pushed the feat/persist-corpus-and-deterministic-load branch from d64f37e to cb513d9 Compare July 29, 2026 20:07
@wxiao0421
wxiao0421 force-pushed the feat/index-from-texts branch from 983dd9b to d261206 Compare July 29, 2026 20:07
@wxiao0421
wxiao0421 force-pushed the feat/index-from-texts branch from d261206 to 9241569 Compare July 30, 2026 16:53
Base automatically changed from feat/persist-corpus-and-deterministic-load to main July 30, 2026 20:38
@wxiao0421
wxiao0421 force-pushed the feat/index-from-texts branch from 9241569 to 564d4d5 Compare July 30, 2026 20:54

@vcai4071 vcai4071 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. Some comments that may be worth addressing.

Comment thread src/sentinel/sentinel_local_index.py Outdated
)
)

if neg_to_pos_ratio is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be repeated code with subsample? Can we reuse some code here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Fixed .

**encoding_kwargs,
)
)
negative_embeddings = torch.tensor(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be worth downsampling before embedding to save some cost?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Fixed it improve performance.

wxiao0421 and others added 2 commits August 3, 2026 15:14
Building an index by hand takes eight steps, two of which fail silently when
skipped: omit normalize_embeddings=True and the similarity maths quietly returns
wrong numbers, and omit the corpus and explanations degrade to row numbers. No
crash, no warning either way. Doing those steps inside the library, where they
are tested, means a caller cannot forget a step they never have to write.

Also corrects the README's save-format section, which described corpus.json as
optional and written only when the index has a corpus. Both became false when
that file started being written unconditionally, and the file's contents are what
is optional now.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ogic

Two review points on from_texts, addressed together because the fix for one
decides the shape of the other.

from_texts encoded every negative it was given and only then applied the ratio,
so anything above the ratio was paid for and thrown away. Encoding is per-text
and is the only expensive step here: at a 1:1 ratio against 1,000 positives, a
caller passing 100,000 negatives paid to encode 99,000 rows that never reached
the index. The choice of which negatives to keep depends only on the positive
count and the ratio, both known before encoding, and encoding is per-text, so
selecting first yields the same index for a fraction of the cost.

Selecting first means selecting texts rather than embedding rows, so the piece
worth sharing with subsample() is the choice of positions, not the copy. That is
now _choose_indices, used by both, with _select_subset built on top of it for the
embedding case. _select_subset also moves to module level, since it never used
self and a classmethod cannot reach an instance method. Consolidating matters
beyond tidiness: keeping each embedding beside its own text is the one rule in
this module that fails silently when broken, so it should live in one place.

Adds tests asserting that only the kept negatives ever reach the encoder, and
that the no-ratio path still encodes everything. The first fails on the previous
encode-then-discard order.

_apply_negative_ratio has a third copy of this logic, left alone deliberately: it
mutates in place and warns rather than raising, so folding it in would mean
re-reviewing merged load behaviour from this PR.

Reported in review by vcai4071.

Co-authored-by: Cursor <cursoragent@cursor.com>
@wxiao0421
wxiao0421 force-pushed the feat/index-from-texts branch from 564d4d5 to 093a174 Compare August 3, 2026 22:19
@wxiao0421
wxiao0421 merged commit ce3b641 into main Aug 3, 2026
4 checks passed
@wxiao0421
wxiao0421 deleted the feat/index-from-texts branch August 3, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants