diff --git a/CHANGELOG.md b/CHANGELOG.md index dcbf322..0f58f1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ ## Unreleased +- Search upgraded from TF-IDF cosine to **Okapi BM25** (term-frequency + saturation + document-length normalization, normalized to [0, 1] so the + graph-entity boost keeps its weight). A note spamming one query term no + longer outranks a note covering all of them, and short focused notes + beat long ones with the same match. New docs page: + [how search works](docs/how-search-works.md). + - Library API: `import graphier; v = graphier.open("~/notes")` — entities, relations, Datalog queries, conflicts, timeline, `to_networkx()`, and matplotlib plotting (`plot_graph()`, `plot_timeline()`) via the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 19af5b4..fc14695 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -44,7 +44,7 @@ backend/graphier/ extraction.py entity + relation extraction over a masked shadow of each note graph.py vault-wide graph: nodes, edges, sentence evidence, domain parsing enrichment.py Datalog inference, conflicts, suggestions, PageRank - search.py TF-IDF + graph-boosted hybrid search + search.py BM25 + graph-boosted hybrid search history.py git-backed snapshots and graph-at-commit replay timeline.py dated events across all sources demo.py first-run demo corpus diff --git a/README.md b/README.md index acf30a8..84b846d 100644 --- a/README.md +++ b/README.md @@ -44,10 +44,11 @@ aggregation, and a deterministic enrichment layer: it, plus the conflicts and inferences it participates in. - **One-click linking** — accept a link suggestion and the `[[wiki-link]]` is written into your Markdown for you. -- **Hybrid search** — TF-IDF ranking over note bodies, graph-boosted: a note +- **Hybrid search** — BM25 ranking over note bodies, graph-boosted: a note that *mentions the entity you searched for* outranks one that merely shares vocabulary, and matching entities surface as chips that open their entity - pages. + pages. Deterministic and fully explained in + [docs/how-search-works.md](docs/how-search-works.md). - **Time travel** — the vault is a git repo. Take a snapshot anytime; the graph view's timeline selector replays the knowledge graph exactly as it was at any snapshot — the past still knows what the present forgot. diff --git a/ROADMAP.md b/ROADMAP.md index 22181e9..7a23f79 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -13,7 +13,7 @@ for the bigger ones so we can agree on shape. inferred-edge overlay - [x] User-programmable rules (```datalog blocks) - [x] Sentence-level provenance + entity pages + one-click linking -- [x] Hybrid search (TF-IDF + graph boost) +- [x] Hybrid search (BM25 + graph boost) - [x] Time travel (git snapshots, graph-at-commit replay) - [x] Domains: vault-defined entity types + typed relation templates - [x] Live query notes (```query blocks, Datalog queries) diff --git a/backend/graphier/mcp.py b/backend/graphier/mcp.py index 6003f9e..f1888be 100644 --- a/backend/graphier/mcp.py +++ b/backend/graphier/mcp.py @@ -126,7 +126,7 @@ def read_note(self, note_id: str) -> dict[str, Any]: TOOL_DEFS = [ { "name": "search_vault", - "description": "Hybrid search over the vault: TF-IDF ranking boosted by the " + "description": "Hybrid search over the vault: BM25 ranking boosted by the " "knowledge graph. Returns ranked note hits with snippets, plus entities " "matching the query.", "inputSchema": { diff --git a/backend/graphier/search.py b/backend/graphier/search.py index 2ec30a5..0ba209c 100644 --- a/backend/graphier/search.py +++ b/backend/graphier/search.py @@ -1,9 +1,15 @@ -"""Hybrid vault search: lexical TF-IDF + knowledge-graph awareness. - -Pure numpy TF-IDF with cosine ranking, then graph-boosted: a note that -mentions an entity whose name matches the query outranks a note that -merely shares vocabulary. Matching entities are returned alongside note -hits so the UI can offer their entity pages directly. +"""Hybrid vault search: lexical BM25 + knowledge-graph awareness. + +Pure-Python Okapi BM25 (term-frequency saturation + document-length +normalization), then graph-boosted: a note that mentions an entity whose +name matches the query outranks a note that merely shares vocabulary. +Matching entities are returned alongside note hits so the UI can offer +their entity pages directly. + +BM25 raw scores are unbounded, so each score is divided by the query's +best achievable score — every term saturated to its ceiling — mapping it +into [0, 1]. That keeps the flat entity boost and the relevance floor +meaningful regardless of query length. """ from __future__ import annotations @@ -17,6 +23,11 @@ _ENTITY_BOOST = 0.35 +# Okapi BM25 constants: _K1 controls how quickly repeated terms stop +# adding score, _B how strongly long documents are penalized. +_K1 = 1.5 +_B = 0.75 + def _tokens(text: str) -> list[str]: return _TOKEN_RE.findall(text.lower()) @@ -50,30 +61,31 @@ def search( for note_id in node["notes"]: boosted_notes[note_id] = boosted_notes.get(note_id, 0.0) + _ENTITY_BOOST - # ---- TF-IDF cosine over note bodies ---- + # ---- BM25 over note bodies ---- doc_tokens = {note_id: _tokens(content) for note_id, content in notes.items()} doc_freq: Counter = Counter() for toks in doc_tokens.values(): doc_freq.update(set(toks)) n_docs = len(notes) + avg_len = sum(len(toks) for toks in doc_tokens.values()) / n_docs or 1.0 def idf(term: str) -> float: - return math.log((n_docs + 1) / (doc_freq[term] + 1)) + 1 - - def vectorize(toks: list[str]) -> dict[str, float]: - counts = Counter(toks) - total = len(toks) or 1 - return {t: (c / total) * idf(t) for t, c in counts.items()} + df = doc_freq[term] + return math.log(1 + (n_docs - df + 0.5) / (df + 0.5)) - query_vec = vectorize(query_tokens) - query_norm = math.sqrt(sum(w * w for w in query_vec.values())) or 1.0 + query_terms = set(query_tokens) + best_possible = sum(idf(t) * (_K1 + 1) for t in query_terms) or 1.0 scored = [] for note_id, toks in doc_tokens.items(): - doc_vec = vectorize(toks) - dot = sum(query_vec[t] * doc_vec.get(t, 0.0) for t in query_vec) - doc_norm = math.sqrt(sum(w * w for w in doc_vec.values())) or 1.0 - score = dot / (query_norm * doc_norm) + boosted_notes.get(note_id, 0.0) + counts = Counter(toks) + length_norm = _K1 * (1 - _B + _B * len(toks) / avg_len) + raw = sum( + idf(t) * counts[t] * (_K1 + 1) / (counts[t] + length_norm) + for t in query_terms + if counts[t] + ) + score = raw / best_possible + boosted_notes.get(note_id, 0.0) if score > 0.01: scored.append((score, note_id)) scored.sort(reverse=True) diff --git a/backend/tests/test_history_search.py b/backend/tests/test_history_search.py index 5ca1023..114e65d 100644 --- a/backend/tests/test_history_search.py +++ b/backend/tests/test_history_search.py @@ -33,6 +33,34 @@ def test_search_entity_match_boosts_and_returns_entity(client): ) +def test_search_term_coverage_beats_repetition(tmp_path): + c = TestClient(create_app(vault_dir=str(tmp_path))) + for title, body in [ + ("Spam", "# Spam\n\n" + "compiler " * 40), + ("Plan", "# Plan\n\nThe compiler roadmap was finalized this week."), + ]: + note_id = c.post("/api/notes", json={"title": title}).json()["id"] + c.put(f"/api/notes/{note_id}", json={"content": body}) + result = c.get("/api/search", params={"q": "compiler roadmap"}).json() + assert [r["id"] for r in result["results"]][0] == "plan" + + +def test_search_length_normalization_and_score_range(tmp_path): + c = TestClient(create_app(vault_dir=str(tmp_path))) + filler = " ".join(f"word{i}" for i in range(300)) + for title, body in [ + ("Short", "# Short\n\nThe roadmap."), + ("Long", f"# Long\n\nThe roadmap. {filler}"), + ]: + note_id = c.post("/api/notes", json={"title": title}).json()["id"] + c.put(f"/api/notes/{note_id}", json={"content": body}) + result = c.get("/api/search", params={"q": "roadmap"}).json() + ids = [r["id"] for r in result["results"]] + assert ids[0] == "short" + # normalized BM25: lexical score is bounded by 1 (no entity boost here) + assert all(0 < r["score"] <= 1 for r in result["results"]) + + def test_search_empty_query_and_no_hits(client): assert client.get("/api/search", params={"q": " "}).json()["results"] == [] assert client.get("/api/search", params={"q": "zzzqqqxxx"}).json()["results"] == [] diff --git a/docs/how-search-works.md b/docs/how-search-works.md new file mode 100644 index 0000000..1e0843e --- /dev/null +++ b/docs/how-search-works.md @@ -0,0 +1,87 @@ +# How search works + +Graphier's vault search is deliberately boring in the best way: no +embeddings, no model calls, no index server. It is a small, deterministic +ranking function — the same query over the same vault returns the same +results, every time, offline. This page explains exactly what happens when +you type a query, so you can predict it, tune it, or replace it. + +Everything below lives in one file: +[`backend/graphier/search.py`](../backend/graphier/search.py) (~120 lines, +stdlib only). + +## The pipeline + +A query goes through four steps: + +1. **Tokenize** — lowercase, split on runs of `[a-z0-9]+`. Punctuation, + Markdown syntax, and case disappear; `"Ada Lovelace!"` becomes + `["ada", "lovelace"]`. The same tokenizer is applied to every note body + (documents like PDFs contribute their extracted text). + +2. **Score notes lexically with BM25** — the classic Okapi BM25 ranking + function, computed from scratch on each request: + + ``` + score(note) = Σ over query terms t of + idf(t) · tf(t) · (k1 + 1) / (tf(t) + k1 · (1 − b + b · |note| / avg_len)) + ``` + + with `k1 = 1.5` and `b = 0.75` (the standard defaults), and the BM25 + inverse document frequency + `idf(t) = ln(1 + (N − df(t) + 0.5) / (df(t) + 0.5))` where `N` is the + number of notes and `df(t)` how many contain `t`. + + Two properties make BM25 the right default over plain TF-IDF: + + - **Term-frequency saturation** (`k1`): the 40th occurrence of + "compiler" in a note adds almost nothing. A note that spams one query + term cannot outrank a note that covers *all* the query terms. + - **Length normalization** (`b`): a term match in a 20-word note means + more than the same match buried in a 2,000-word note. + + The raw score is then divided by the query's **best achievable score** + (every term saturated to its ceiling), mapping the lexical component + into `[0, 1]` regardless of query length. + +3. **Boost by the knowledge graph** — this is the hybrid part. Every graph + entity whose name contains the query (or vice versa, token-normalized) + contributes a flat **+0.35** to every note it appears in. Because the + lexical score is normalized to at most 1.0, the boost is a strong, + predictable signal: a note that *mentions the entity you searched for* + reliably outranks one that merely shares vocabulary. Matched entities + are also returned alongside the results, which is what powers the + entity chips above the hit list in the app. + +4. **Cut, sort, snippet** — notes scoring below `0.01` are dropped, the + top 10 survive, and each hit gets a ~130-character snippet windowed + around the first query-term occurrence in the raw note text. + +## What it deliberately is not + +- **Not embeddings / semantic search.** Vector search is powerful but + unexplainable and nondeterministic across model versions. Graphier's + thesis is that a personal knowledge base should be auditable; optional + semantic search may arrive later as exactly that — optional + ([roadmap](../ROADMAP.md)). +- **Not a persistent index.** The vault is small by web-search standards + (thousands of notes, not millions of pages), so scoring on request keeps + the implementation transparent and always in sync with the files — there + is no index to corrupt or rebuild. +- **Not fuzzy.** Typos don't match. The graph boost softens this in + practice: entity names matched via the graph often rescue queries that + lexical scoring alone would miss. + +## Tuning it + +All the knobs are module constants in `search.py`: + +| Constant | Default | Effect of raising it | +| --- | --- | --- | +| `_K1` | `1.5` | repeated terms keep adding score for longer | +| `_B` | `0.75` | long notes are penalized harder | +| `_ENTITY_BOOST` | `0.35` | graph mentions dominate lexical relevance more | + +The search endpoint is `GET /api/search?q=…`, the MCP tool is +`search_vault`, and the library exposes the same ranking through the HTTP +API. All three share this one implementation. diff --git a/docs/launch-posts.md b/docs/launch-posts.md new file mode 100644 index 0000000..2020d0a --- /dev/null +++ b/docs/launch-posts.md @@ -0,0 +1,165 @@ +# Launch posts — ready to adapt and publish + +Drafts, not scripts: adjust anything that doesn't sound like you. The one +rule that matters everywhere: be present in the comments for the first +day, answer fast, and concede real limitations without being asked twice. + +--- + +## Show HN + +**Title** (pick one; the first is safest): + +> Show HN: An Obsidian-like app where the knowledge graph builds itself (no LLM) + +> Show HN: Graphier – notes that extract their own knowledge graph, with provenance + +**URL:** https://github.com/ioteverythin/Graphier + +**First comment** (post immediately after submitting): + +> Hi HN — I built Graphier because I wanted Obsidian's graph to actually +> *mean* something. In Obsidian, the graph shows links you drew by hand. +> Graphier extracts a typed knowledge graph from what you write — people, +> orgs, dates, relations like "founded_by" — and every edge can quote the +> exact sentence it came from. +> +> The contrarian part: there's **no LLM anywhere in the pipeline**. Same +> vault in, same graph out, every time, offline. Extraction is +> pattern-based, inference is a Datalog reasoner, and the whole graph +> rebuilds from your Markdown files (plus PDFs/DOCX/HTML you drop in). +> The tradeoff is real: regex-tier extraction is noisier than an LLM on +> free prose. What you get in exchange is determinism and provenance — +> the app can never tell you something it can't point to in your own +> files. +> +> The features I'm most fond of: +> +> - Your notes program the system: a ```domain block defines entity +> types, ```datalog blocks add inference rules ("founders of companies +> that acquired others are empire builders"), ```query blocks render +> live dashboards. +> - Conflict detection: two documents disagreeing about who founded a +> company gets flagged with both sentences quoted. +> - Time travel: the vault is a git repo; the graph view can replay what +> you knew at any snapshot. +> - An MCP server, so Claude/Cursor can query the *graph* instead of +> doing RAG over raw text — with citations. +> - It's also a Python library: `graphier.open("~/notes").plot_graph()`. +> +> Try it in one command (seeded demo included): +> `docker run -p 8000:8000 -e GRAPHIER_DEMO=1 ghcr-or-build...` — or +> `pip install graphier && graphier setup`. +> +> Deliberately out of scope: cloud sync, mobile, and any required AI. +> Deliberately in scope next (based on what you tell me): Obsidian vault +> import, OCR, conflict-resolution workflow. +> +> Would love tough questions — especially from people who've tried to +> make extraction-based PKM work before and hit the noise wall. + +**Predictable questions — have answers ready:** + +- *"Regex NER? Won't that be noisy?"* — Yes, and it's the honest trade. + Domain blocks let you define precise types for what you care about; the + full-Semantica install swaps in ML extractors behind the same interface; + and everything shows confidence + evidence so noise is visible, not + silent. +- *"Why not embeddings?"* — Embeddings are unexplainable and + nondeterministic; this project's thesis is that a personal knowledge + base should be auditable. Optional semantic search is on the roadmap as + exactly that — optional. +- *"Obsidian has plugins for X"* — Probably true; the difference is the + typed graph + reasoner + provenance core, which a plugin can't retrofit. +- *"Is this a Semantica demo?"* — It uses three engine components from + Semantica (MIT); the vault model, provenance, domains/rules/queries, + and all UI are Graphier's. There's a comparison table in the README. + +--- + +## r/ObsidianMD + +**Title:** +> I built an experiment: what if the graph view built itself? (no AI, every edge cites its sentence) + +**Body:** + +> Long-time graph-view lover, always slightly disappointed that it only +> shows links I manually made. So I built Graphier — an Obsidian-*like* +> app (plain Markdown vault, local-first, your files) where a +> deterministic extractor builds a typed graph as you write: people, +> orgs, projects, dates, relations like "founded by". Click any edge and +> it quotes the exact sentence that created it. +> +> It's not an Obsidian replacement — you'd miss your plugins, mobile, +> and the editor polish. It's a different answer to the question "what +> should a notes graph *be*?" Things it does that I haven't seen +> elsewhere: fenced code blocks that define your own entity types and +> inference rules, contradiction detection between notes, and replaying +> the graph as it was at any git snapshot. +> +> Demo GIF in the repo; runs in one Docker command with seeded content. +> Would genuinely love this community's take — especially "here's a case +> where the extraction falls flat" reports. [link] + +--- + +## r/PKMS + +**Title:** +> Graphier: a PKM where notes compile into a queryable knowledge graph (typed entities, Datalog, provenance) + +**Body:** + +> The pitch in one line: Dataview queries file metadata you wrote; +> Graphier queries *facts extracted from your prose* — and can also +> derive new ones with rules you write in your notes. +> +> ```datalog +> empire_builder(P, B) :- rel(C, P, founded_by), rel(B, C, acquired_by) +> ``` +> +> …put that in any note and "Ada founded Acme; Acme acquired Widget" +> across two different files derives "Ada ↔ Widget", citing the rule and +> both sentences. Also: user-defined entity types via regex, a +> chronology view that orders every dated fact across Markdown + PDF + +> DOCX sources, and conflict detection when files disagree. +> +> No LLM, no cloud, MIT. Curious how this lands with people deep in +> Tana/Capacities-style typed notes — this is the "types without manual +> tagging" experiment. [link] + +--- + +## r/selfhosted + +**Title:** +> Graphier — self-hosted notes that build their own knowledge graph. One container, no external services, no AI calls. + +**Body:** + +> Single container, single volume, zero external dependencies at +> runtime — no LLM APIs, no telemetry, nothing phones home. Your notes +> are plain Markdown in the volume; the knowledge graph, search, +> timeline, and inference are all derived locally and rebuild from the +> files. `docker run -p 8000:8000 -e GRAPHIER_DEMO=1 -v graphier:/vault …` +> gets you a seeded demo. MIT, FastAPI + React, 60+ tests. [link] + +--- + +## Short blurbs + +**lobste.rs** (tag: `practices`, `web`): submit the repo link with title +"Graphier: an Obsidian-like workspace where the knowledge graph builds +itself (no LLM)" — no self-promo text needed, be in comments. + +**X / Mastodon thread opener:** + +> Obsidian's graph shows links you drew. I built one where the graph +> builds itself — typed entities, extracted relations, inference rules +> you write inside your notes, and every single edge can quote the +> sentence it came from. No LLM anywhere. 🧵 [GIF] + +**awesome-lists PRs:** `awesome-selfhosted` (Note-taking category), +`awesome-mcp-servers` (knowledge/search category — cite the nine tools), +`awesome-knowledge-management`.