Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

## Unreleased

- Search upgraded from TF-IDF cosine to **Okapi BM25** (term-frequency
saturation + document-length normalization, normalized to [0, 1] so the
graph-entity boost keeps its weight). A note spamming one query term no
longer outranks a note covering all of them, and short focused notes
beat long ones with the same match. New docs page:
[how search works](docs/how-search-works.md).

- Library API: `import graphier; v = graphier.open("~/notes")` — entities,
relations, Datalog queries, conflicts, timeline, `to_networkx()`, and
matplotlib plotting (`plot_graph()`, `plot_timeline()`) via the
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ backend/graphier/
extraction.py entity + relation extraction over a masked shadow of each note
graph.py vault-wide graph: nodes, edges, sentence evidence, domain parsing
enrichment.py Datalog inference, conflicts, suggestions, PageRank
search.py TF-IDF + graph-boosted hybrid search
search.py BM25 + graph-boosted hybrid search
history.py git-backed snapshots and graph-at-commit replay
timeline.py dated events across all sources
demo.py first-run demo corpus
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,11 @@ aggregation, and a deterministic enrichment layer:
it, plus the conflicts and inferences it participates in.
- **One-click linking** — accept a link suggestion and the `[[wiki-link]]` is
written into your Markdown for you.
- **Hybrid search** — TF-IDF ranking over note bodies, graph-boosted: a note
- **Hybrid search** — BM25 ranking over note bodies, graph-boosted: a note
that *mentions the entity you searched for* outranks one that merely shares
vocabulary, and matching entities surface as chips that open their entity
pages.
pages. Deterministic and fully explained in
[docs/how-search-works.md](docs/how-search-works.md).
- **Time travel** — the vault is a git repo. Take a snapshot anytime; the
graph view's timeline selector replays the knowledge graph exactly as it
was at any snapshot — the past still knows what the present forgot.
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ for the bigger ones so we can agree on shape.
inferred-edge overlay
- [x] User-programmable rules (```datalog blocks)
- [x] Sentence-level provenance + entity pages + one-click linking
- [x] Hybrid search (TF-IDF + graph boost)
- [x] Hybrid search (BM25 + graph boost)
- [x] Time travel (git snapshots, graph-at-commit replay)
- [x] Domains: vault-defined entity types + typed relation templates
- [x] Live query notes (```query blocks, Datalog queries)
Expand Down
2 changes: 1 addition & 1 deletion backend/graphier/mcp.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ def read_note(self, note_id: str) -> dict[str, Any]:
TOOL_DEFS = [
{
"name": "search_vault",
"description": "Hybrid search over the vault: TF-IDF ranking boosted by the "
"description": "Hybrid search over the vault: BM25 ranking boosted by the "
"knowledge graph. Returns ranked note hits with snippets, plus entities "
"matching the query.",
"inputSchema": {
Expand Down
50 changes: 31 additions & 19 deletions backend/graphier/search.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,15 @@
"""Hybrid vault search: lexical TF-IDF + knowledge-graph awareness.

Pure numpy TF-IDF with cosine ranking, then graph-boosted: a note that
mentions an entity whose name matches the query outranks a note that
merely shares vocabulary. Matching entities are returned alongside note
hits so the UI can offer their entity pages directly.
"""Hybrid vault search: lexical BM25 + knowledge-graph awareness.

Pure-Python Okapi BM25 (term-frequency saturation + document-length
normalization), then graph-boosted: a note that mentions an entity whose
name matches the query outranks a note that merely shares vocabulary.
Matching entities are returned alongside note hits so the UI can offer
their entity pages directly.

BM25 raw scores are unbounded, so each score is divided by the query's
best achievable score — every term saturated to its ceiling — mapping it
into [0, 1]. That keeps the flat entity boost and the relevance floor
meaningful regardless of query length.
"""

from __future__ import annotations
Expand All @@ -17,6 +23,11 @@

_ENTITY_BOOST = 0.35

# Okapi BM25 constants: _K1 controls how quickly repeated terms stop
# adding score, _B how strongly long documents are penalized.
_K1 = 1.5
_B = 0.75


def _tokens(text: str) -> list[str]:
return _TOKEN_RE.findall(text.lower())
Expand Down Expand Up @@ -50,30 +61,31 @@ def search(
for note_id in node["notes"]:
boosted_notes[note_id] = boosted_notes.get(note_id, 0.0) + _ENTITY_BOOST

# ---- TF-IDF cosine over note bodies ----
# ---- BM25 over note bodies ----
doc_tokens = {note_id: _tokens(content) for note_id, content in notes.items()}
doc_freq: Counter = Counter()
for toks in doc_tokens.values():
doc_freq.update(set(toks))
n_docs = len(notes)
avg_len = sum(len(toks) for toks in doc_tokens.values()) / n_docs or 1.0

def idf(term: str) -> float:
return math.log((n_docs + 1) / (doc_freq[term] + 1)) + 1

def vectorize(toks: list[str]) -> dict[str, float]:
counts = Counter(toks)
total = len(toks) or 1
return {t: (c / total) * idf(t) for t, c in counts.items()}
df = doc_freq[term]
return math.log(1 + (n_docs - df + 0.5) / (df + 0.5))

query_vec = vectorize(query_tokens)
query_norm = math.sqrt(sum(w * w for w in query_vec.values())) or 1.0
query_terms = set(query_tokens)
best_possible = sum(idf(t) * (_K1 + 1) for t in query_terms) or 1.0

scored = []
for note_id, toks in doc_tokens.items():
doc_vec = vectorize(toks)
dot = sum(query_vec[t] * doc_vec.get(t, 0.0) for t in query_vec)
doc_norm = math.sqrt(sum(w * w for w in doc_vec.values())) or 1.0
score = dot / (query_norm * doc_norm) + boosted_notes.get(note_id, 0.0)
counts = Counter(toks)
length_norm = _K1 * (1 - _B + _B * len(toks) / avg_len)
raw = sum(
idf(t) * counts[t] * (_K1 + 1) / (counts[t] + length_norm)
for t in query_terms
if counts[t]
)
score = raw / best_possible + boosted_notes.get(note_id, 0.0)
if score > 0.01:
scored.append((score, note_id))
scored.sort(reverse=True)
Expand Down
28 changes: 28 additions & 0 deletions backend/tests/test_history_search.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,34 @@ def test_search_entity_match_boosts_and_returns_entity(client):
)


def test_search_term_coverage_beats_repetition(tmp_path):
c = TestClient(create_app(vault_dir=str(tmp_path)))
for title, body in [
("Spam", "# Spam\n\n" + "compiler " * 40),
("Plan", "# Plan\n\nThe compiler roadmap was finalized this week."),
]:
note_id = c.post("/api/notes", json={"title": title}).json()["id"]
c.put(f"/api/notes/{note_id}", json={"content": body})
result = c.get("/api/search", params={"q": "compiler roadmap"}).json()
assert [r["id"] for r in result["results"]][0] == "plan"


def test_search_length_normalization_and_score_range(tmp_path):
c = TestClient(create_app(vault_dir=str(tmp_path)))
filler = " ".join(f"word{i}" for i in range(300))
for title, body in [
("Short", "# Short\n\nThe roadmap."),
("Long", f"# Long\n\nThe roadmap. {filler}"),
]:
note_id = c.post("/api/notes", json={"title": title}).json()["id"]
c.put(f"/api/notes/{note_id}", json={"content": body})
result = c.get("/api/search", params={"q": "roadmap"}).json()
ids = [r["id"] for r in result["results"]]
assert ids[0] == "short"
# normalized BM25: lexical score is bounded by 1 (no entity boost here)
assert all(0 < r["score"] <= 1 for r in result["results"])


def test_search_empty_query_and_no_hits(client):
assert client.get("/api/search", params={"q": " "}).json()["results"] == []
assert client.get("/api/search", params={"q": "zzzqqqxxx"}).json()["results"] == []
Expand Down
87 changes: 87 additions & 0 deletions docs/how-search-works.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# How search works

Graphier's vault search is deliberately boring in the best way: no
embeddings, no model calls, no index server. It is a small, deterministic
ranking function — the same query over the same vault returns the same
results, every time, offline. This page explains exactly what happens when
you type a query, so you can predict it, tune it, or replace it.

Everything below lives in one file:
[`backend/graphier/search.py`](../backend/graphier/search.py) (~120 lines,
stdlib only).

## The pipeline

A query goes through four steps:

1. **Tokenize** — lowercase, split on runs of `[a-z0-9]+`. Punctuation,
Markdown syntax, and case disappear; `"Ada Lovelace!"` becomes
`["ada", "lovelace"]`. The same tokenizer is applied to every note body
(documents like PDFs contribute their extracted text).

2. **Score notes lexically with BM25** — the classic Okapi BM25 ranking
function, computed from scratch on each request:

```
score(note) = Σ over query terms t of
idf(t) · tf(t) · (k1 + 1) / (tf(t) + k1 · (1 − b + b · |note| / avg_len))
```

with `k1 = 1.5` and `b = 0.75` (the standard defaults), and the BM25
inverse document frequency
`idf(t) = ln(1 + (N − df(t) + 0.5) / (df(t) + 0.5))` where `N` is the
number of notes and `df(t)` how many contain `t`.

Two properties make BM25 the right default over plain TF-IDF:

- **Term-frequency saturation** (`k1`): the 40th occurrence of
"compiler" in a note adds almost nothing. A note that spams one query
term cannot outrank a note that covers *all* the query terms.
- **Length normalization** (`b`): a term match in a 20-word note means
more than the same match buried in a 2,000-word note.

The raw score is then divided by the query's **best achievable score**
(every term saturated to its ceiling), mapping the lexical component
into `[0, 1]` regardless of query length.

3. **Boost by the knowledge graph** — this is the hybrid part. Every graph
entity whose name contains the query (or vice versa, token-normalized)
contributes a flat **+0.35** to every note it appears in. Because the
lexical score is normalized to at most 1.0, the boost is a strong,
predictable signal: a note that *mentions the entity you searched for*
reliably outranks one that merely shares vocabulary. Matched entities
are also returned alongside the results, which is what powers the
entity chips above the hit list in the app.

4. **Cut, sort, snippet** — notes scoring below `0.01` are dropped, the
top 10 survive, and each hit gets a ~130-character snippet windowed
around the first query-term occurrence in the raw note text.

## What it deliberately is not

- **Not embeddings / semantic search.** Vector search is powerful but
unexplainable and nondeterministic across model versions. Graphier's
thesis is that a personal knowledge base should be auditable; optional
semantic search may arrive later as exactly that — optional
([roadmap](../ROADMAP.md)).
- **Not a persistent index.** The vault is small by web-search standards
(thousands of notes, not millions of pages), so scoring on request keeps
the implementation transparent and always in sync with the files — there
is no index to corrupt or rebuild.
- **Not fuzzy.** Typos don't match. The graph boost softens this in
practice: entity names matched via the graph often rescue queries that
lexical scoring alone would miss.

## Tuning it

All the knobs are module constants in `search.py`:

| Constant | Default | Effect of raising it |
| --- | --- | --- |
| `_K1` | `1.5` | repeated terms keep adding score for longer |
| `_B` | `0.75` | long notes are penalized harder |
| `_ENTITY_BOOST` | `0.35` | graph mentions dominate lexical relevance more |

The search endpoint is `GET /api/search?q=…`, the MCP tool is
`search_vault`, and the library exposes the same ranking through the HTTP
API. All three share this one implementation.
Loading
Loading