From 37672869a21317d1922138cabb91f86972d8e795 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 15:55:14 +0000 Subject: [PATCH 1/2] Add ready-to-post launch copy Show HN title + first comment (with prepared answers to the four predictable objections: extraction noise, why-not-embeddings, Obsidian-has-plugins, is-this-a-Semantica-demo), tailored posts for r/ObsidianMD, r/PKMS, and r/selfhosted, and short blurbs for lobste.rs, X/Mastodon, and awesome-list PRs. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01QVbxD1gd3gsC6zXKE6L3ya --- docs/launch-posts.md | 165 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 docs/launch-posts.md diff --git a/docs/launch-posts.md b/docs/launch-posts.md new file mode 100644 index 0000000..2020d0a --- /dev/null +++ b/docs/launch-posts.md @@ -0,0 +1,165 @@ +# Launch posts — ready to adapt and publish + +Drafts, not scripts: adjust anything that doesn't sound like you. The one +rule that matters everywhere: be present in the comments for the first +day, answer fast, and concede real limitations without being asked twice. + +--- + +## Show HN + +**Title** (pick one; the first is safest): + +> Show HN: An Obsidian-like app where the knowledge graph builds itself (no LLM) + +> Show HN: Graphier – notes that extract their own knowledge graph, with provenance + +**URL:** https://github.com/ioteverythin/Graphier + +**First comment** (post immediately after submitting): + +> Hi HN — I built Graphier because I wanted Obsidian's graph to actually +> *mean* something. In Obsidian, the graph shows links you drew by hand. +> Graphier extracts a typed knowledge graph from what you write — people, +> orgs, dates, relations like "founded_by" — and every edge can quote the +> exact sentence it came from. +> +> The contrarian part: there's **no LLM anywhere in the pipeline**. Same +> vault in, same graph out, every time, offline. Extraction is +> pattern-based, inference is a Datalog reasoner, and the whole graph +> rebuilds from your Markdown files (plus PDFs/DOCX/HTML you drop in). +> The tradeoff is real: regex-tier extraction is noisier than an LLM on +> free prose. What you get in exchange is determinism and provenance — +> the app can never tell you something it can't point to in your own +> files. +> +> The features I'm most fond of: +> +> - Your notes program the system: a ```domain block defines entity +> types, ```datalog blocks add inference rules ("founders of companies +> that acquired others are empire builders"), ```query blocks render +> live dashboards. +> - Conflict detection: two documents disagreeing about who founded a +> company gets flagged with both sentences quoted. +> - Time travel: the vault is a git repo; the graph view can replay what +> you knew at any snapshot. +> - An MCP server, so Claude/Cursor can query the *graph* instead of +> doing RAG over raw text — with citations. +> - It's also a Python library: `graphier.open("~/notes").plot_graph()`. +> +> Try it in one command (seeded demo included): +> `docker run -p 8000:8000 -e GRAPHIER_DEMO=1 ghcr-or-build...` — or +> `pip install graphier && graphier setup`. +> +> Deliberately out of scope: cloud sync, mobile, and any required AI. +> Deliberately in scope next (based on what you tell me): Obsidian vault +> import, OCR, conflict-resolution workflow. +> +> Would love tough questions — especially from people who've tried to +> make extraction-based PKM work before and hit the noise wall. + +**Predictable questions — have answers ready:** + +- *"Regex NER? Won't that be noisy?"* — Yes, and it's the honest trade. + Domain blocks let you define precise types for what you care about; the + full-Semantica install swaps in ML extractors behind the same interface; + and everything shows confidence + evidence so noise is visible, not + silent. +- *"Why not embeddings?"* — Embeddings are unexplainable and + nondeterministic; this project's thesis is that a personal knowledge + base should be auditable. Optional semantic search is on the roadmap as + exactly that — optional. +- *"Obsidian has plugins for X"* — Probably true; the difference is the + typed graph + reasoner + provenance core, which a plugin can't retrofit. +- *"Is this a Semantica demo?"* — It uses three engine components from + Semantica (MIT); the vault model, provenance, domains/rules/queries, + and all UI are Graphier's. There's a comparison table in the README. + +--- + +## r/ObsidianMD + +**Title:** +> I built an experiment: what if the graph view built itself? (no AI, every edge cites its sentence) + +**Body:** + +> Long-time graph-view lover, always slightly disappointed that it only +> shows links I manually made. So I built Graphier — an Obsidian-*like* +> app (plain Markdown vault, local-first, your files) where a +> deterministic extractor builds a typed graph as you write: people, +> orgs, projects, dates, relations like "founded by". Click any edge and +> it quotes the exact sentence that created it. +> +> It's not an Obsidian replacement — you'd miss your plugins, mobile, +> and the editor polish. It's a different answer to the question "what +> should a notes graph *be*?" Things it does that I haven't seen +> elsewhere: fenced code blocks that define your own entity types and +> inference rules, contradiction detection between notes, and replaying +> the graph as it was at any git snapshot. +> +> Demo GIF in the repo; runs in one Docker command with seeded content. +> Would genuinely love this community's take — especially "here's a case +> where the extraction falls flat" reports. [link] + +--- + +## r/PKMS + +**Title:** +> Graphier: a PKM where notes compile into a queryable knowledge graph (typed entities, Datalog, provenance) + +**Body:** + +> The pitch in one line: Dataview queries file metadata you wrote; +> Graphier queries *facts extracted from your prose* — and can also +> derive new ones with rules you write in your notes. +> +> ```datalog +> empire_builder(P, B) :- rel(C, P, founded_by), rel(B, C, acquired_by) +> ``` +> +> …put that in any note and "Ada founded Acme; Acme acquired Widget" +> across two different files derives "Ada ↔ Widget", citing the rule and +> both sentences. Also: user-defined entity types via regex, a +> chronology view that orders every dated fact across Markdown + PDF + +> DOCX sources, and conflict detection when files disagree. +> +> No LLM, no cloud, MIT. Curious how this lands with people deep in +> Tana/Capacities-style typed notes — this is the "types without manual +> tagging" experiment. [link] + +--- + +## r/selfhosted + +**Title:** +> Graphier — self-hosted notes that build their own knowledge graph. One container, no external services, no AI calls. + +**Body:** + +> Single container, single volume, zero external dependencies at +> runtime — no LLM APIs, no telemetry, nothing phones home. Your notes +> are plain Markdown in the volume; the knowledge graph, search, +> timeline, and inference are all derived locally and rebuild from the +> files. `docker run -p 8000:8000 -e GRAPHIER_DEMO=1 -v graphier:/vault …` +> gets you a seeded demo. MIT, FastAPI + React, 60+ tests. [link] + +--- + +## Short blurbs + +**lobste.rs** (tag: `practices`, `web`): submit the repo link with title +"Graphier: an Obsidian-like workspace where the knowledge graph builds +itself (no LLM)" — no self-promo text needed, be in comments. + +**X / Mastodon thread opener:** + +> Obsidian's graph shows links you drew. I built one where the graph +> builds itself — typed entities, extracted relations, inference rules +> you write inside your notes, and every single edge can quote the +> sentence it came from. No LLM anywhere. 🧵 [GIF] + +**awesome-lists PRs:** `awesome-selfhosted` (Note-taking category), +`awesome-mcp-servers` (knowledge/search category — cite the nine tools), +`awesome-knowledge-management`. From 81de59d6e7422c67e2fda5f374f314826489a615 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 22 Aug 2026 11:40:59 +0000 Subject: [PATCH 2/2] Upgrade search to BM25 and document the ranking pipeline Replace TF-IDF cosine with Okapi BM25 (k1=1.5, b=0.75): term-frequency saturation stops one spammed term from outranking full query coverage, and length normalization lets short focused notes beat long ones with the same match. Raw scores are divided by the query's best achievable score, keeping the lexical component in [0, 1] so the +0.35 graph-entity boost and the relevance floor keep their calibrated weight. Add docs/how-search-works.md explaining the full pipeline (tokenize, BM25, graph boost, snippet) with the tuning knobs, and update every TF-IDF reference. Two new tests pin the BM25 properties. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01QVbxD1gd3gsC6zXKE6L3ya --- CHANGELOG.md | 7 +++ CONTRIBUTING.md | 2 +- README.md | 5 +- ROADMAP.md | 2 +- backend/graphier/mcp.py | 2 +- backend/graphier/search.py | 50 ++++++++++------ backend/tests/test_history_search.py | 28 +++++++++ docs/how-search-works.md | 87 ++++++++++++++++++++++++++++ 8 files changed, 159 insertions(+), 24 deletions(-) create mode 100644 docs/how-search-works.md diff --git a/CHANGELOG.md b/CHANGELOG.md index dcbf322..0f58f1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ ## Unreleased +- Search upgraded from TF-IDF cosine to **Okapi BM25** (term-frequency + saturation + document-length normalization, normalized to [0, 1] so the + graph-entity boost keeps its weight). A note spamming one query term no + longer outranks a note covering all of them, and short focused notes + beat long ones with the same match. New docs page: + [how search works](docs/how-search-works.md). + - Library API: `import graphier; v = graphier.open("~/notes")` — entities, relations, Datalog queries, conflicts, timeline, `to_networkx()`, and matplotlib plotting (`plot_graph()`, `plot_timeline()`) via the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 19af5b4..fc14695 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -44,7 +44,7 @@ backend/graphier/ extraction.py entity + relation extraction over a masked shadow of each note graph.py vault-wide graph: nodes, edges, sentence evidence, domain parsing enrichment.py Datalog inference, conflicts, suggestions, PageRank - search.py TF-IDF + graph-boosted hybrid search + search.py BM25 + graph-boosted hybrid search history.py git-backed snapshots and graph-at-commit replay timeline.py dated events across all sources demo.py first-run demo corpus diff --git a/README.md b/README.md index acf30a8..84b846d 100644 --- a/README.md +++ b/README.md @@ -44,10 +44,11 @@ aggregation, and a deterministic enrichment layer: it, plus the conflicts and inferences it participates in. - **One-click linking** — accept a link suggestion and the `[[wiki-link]]` is written into your Markdown for you. -- **Hybrid search** — TF-IDF ranking over note bodies, graph-boosted: a note +- **Hybrid search** — BM25 ranking over note bodies, graph-boosted: a note that *mentions the entity you searched for* outranks one that merely shares vocabulary, and matching entities surface as chips that open their entity - pages. + pages. Deterministic and fully explained in + [docs/how-search-works.md](docs/how-search-works.md). - **Time travel** — the vault is a git repo. Take a snapshot anytime; the graph view's timeline selector replays the knowledge graph exactly as it was at any snapshot — the past still knows what the present forgot. diff --git a/ROADMAP.md b/ROADMAP.md index 22181e9..7a23f79 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -13,7 +13,7 @@ for the bigger ones so we can agree on shape. inferred-edge overlay - [x] User-programmable rules (```datalog blocks) - [x] Sentence-level provenance + entity pages + one-click linking -- [x] Hybrid search (TF-IDF + graph boost) +- [x] Hybrid search (BM25 + graph boost) - [x] Time travel (git snapshots, graph-at-commit replay) - [x] Domains: vault-defined entity types + typed relation templates - [x] Live query notes (```query blocks, Datalog queries) diff --git a/backend/graphier/mcp.py b/backend/graphier/mcp.py index 6003f9e..f1888be 100644 --- a/backend/graphier/mcp.py +++ b/backend/graphier/mcp.py @@ -126,7 +126,7 @@ def read_note(self, note_id: str) -> dict[str, Any]: TOOL_DEFS = [ { "name": "search_vault", - "description": "Hybrid search over the vault: TF-IDF ranking boosted by the " + "description": "Hybrid search over the vault: BM25 ranking boosted by the " "knowledge graph. Returns ranked note hits with snippets, plus entities " "matching the query.", "inputSchema": { diff --git a/backend/graphier/search.py b/backend/graphier/search.py index 2ec30a5..0ba209c 100644 --- a/backend/graphier/search.py +++ b/backend/graphier/search.py @@ -1,9 +1,15 @@ -"""Hybrid vault search: lexical TF-IDF + knowledge-graph awareness. - -Pure numpy TF-IDF with cosine ranking, then graph-boosted: a note that -mentions an entity whose name matches the query outranks a note that -merely shares vocabulary. Matching entities are returned alongside note -hits so the UI can offer their entity pages directly. +"""Hybrid vault search: lexical BM25 + knowledge-graph awareness. + +Pure-Python Okapi BM25 (term-frequency saturation + document-length +normalization), then graph-boosted: a note that mentions an entity whose +name matches the query outranks a note that merely shares vocabulary. +Matching entities are returned alongside note hits so the UI can offer +their entity pages directly. + +BM25 raw scores are unbounded, so each score is divided by the query's +best achievable score — every term saturated to its ceiling — mapping it +into [0, 1]. That keeps the flat entity boost and the relevance floor +meaningful regardless of query length. """ from __future__ import annotations @@ -17,6 +23,11 @@ _ENTITY_BOOST = 0.35 +# Okapi BM25 constants: _K1 controls how quickly repeated terms stop +# adding score, _B how strongly long documents are penalized. +_K1 = 1.5 +_B = 0.75 + def _tokens(text: str) -> list[str]: return _TOKEN_RE.findall(text.lower()) @@ -50,30 +61,31 @@ def search( for note_id in node["notes"]: boosted_notes[note_id] = boosted_notes.get(note_id, 0.0) + _ENTITY_BOOST - # ---- TF-IDF cosine over note bodies ---- + # ---- BM25 over note bodies ---- doc_tokens = {note_id: _tokens(content) for note_id, content in notes.items()} doc_freq: Counter = Counter() for toks in doc_tokens.values(): doc_freq.update(set(toks)) n_docs = len(notes) + avg_len = sum(len(toks) for toks in doc_tokens.values()) / n_docs or 1.0 def idf(term: str) -> float: - return math.log((n_docs + 1) / (doc_freq[term] + 1)) + 1 - - def vectorize(toks: list[str]) -> dict[str, float]: - counts = Counter(toks) - total = len(toks) or 1 - return {t: (c / total) * idf(t) for t, c in counts.items()} + df = doc_freq[term] + return math.log(1 + (n_docs - df + 0.5) / (df + 0.5)) - query_vec = vectorize(query_tokens) - query_norm = math.sqrt(sum(w * w for w in query_vec.values())) or 1.0 + query_terms = set(query_tokens) + best_possible = sum(idf(t) * (_K1 + 1) for t in query_terms) or 1.0 scored = [] for note_id, toks in doc_tokens.items(): - doc_vec = vectorize(toks) - dot = sum(query_vec[t] * doc_vec.get(t, 0.0) for t in query_vec) - doc_norm = math.sqrt(sum(w * w for w in doc_vec.values())) or 1.0 - score = dot / (query_norm * doc_norm) + boosted_notes.get(note_id, 0.0) + counts = Counter(toks) + length_norm = _K1 * (1 - _B + _B * len(toks) / avg_len) + raw = sum( + idf(t) * counts[t] * (_K1 + 1) / (counts[t] + length_norm) + for t in query_terms + if counts[t] + ) + score = raw / best_possible + boosted_notes.get(note_id, 0.0) if score > 0.01: scored.append((score, note_id)) scored.sort(reverse=True) diff --git a/backend/tests/test_history_search.py b/backend/tests/test_history_search.py index 5ca1023..114e65d 100644 --- a/backend/tests/test_history_search.py +++ b/backend/tests/test_history_search.py @@ -33,6 +33,34 @@ def test_search_entity_match_boosts_and_returns_entity(client): ) +def test_search_term_coverage_beats_repetition(tmp_path): + c = TestClient(create_app(vault_dir=str(tmp_path))) + for title, body in [ + ("Spam", "# Spam\n\n" + "compiler " * 40), + ("Plan", "# Plan\n\nThe compiler roadmap was finalized this week."), + ]: + note_id = c.post("/api/notes", json={"title": title}).json()["id"] + c.put(f"/api/notes/{note_id}", json={"content": body}) + result = c.get("/api/search", params={"q": "compiler roadmap"}).json() + assert [r["id"] for r in result["results"]][0] == "plan" + + +def test_search_length_normalization_and_score_range(tmp_path): + c = TestClient(create_app(vault_dir=str(tmp_path))) + filler = " ".join(f"word{i}" for i in range(300)) + for title, body in [ + ("Short", "# Short\n\nThe roadmap."), + ("Long", f"# Long\n\nThe roadmap. {filler}"), + ]: + note_id = c.post("/api/notes", json={"title": title}).json()["id"] + c.put(f"/api/notes/{note_id}", json={"content": body}) + result = c.get("/api/search", params={"q": "roadmap"}).json() + ids = [r["id"] for r in result["results"]] + assert ids[0] == "short" + # normalized BM25: lexical score is bounded by 1 (no entity boost here) + assert all(0 < r["score"] <= 1 for r in result["results"]) + + def test_search_empty_query_and_no_hits(client): assert client.get("/api/search", params={"q": " "}).json()["results"] == [] assert client.get("/api/search", params={"q": "zzzqqqxxx"}).json()["results"] == [] diff --git a/docs/how-search-works.md b/docs/how-search-works.md new file mode 100644 index 0000000..1e0843e --- /dev/null +++ b/docs/how-search-works.md @@ -0,0 +1,87 @@ +# How search works + +Graphier's vault search is deliberately boring in the best way: no +embeddings, no model calls, no index server. It is a small, deterministic +ranking function — the same query over the same vault returns the same +results, every time, offline. This page explains exactly what happens when +you type a query, so you can predict it, tune it, or replace it. + +Everything below lives in one file: +[`backend/graphier/search.py`](../backend/graphier/search.py) (~120 lines, +stdlib only). + +## The pipeline + +A query goes through four steps: + +1. **Tokenize** — lowercase, split on runs of `[a-z0-9]+`. Punctuation, + Markdown syntax, and case disappear; `"Ada Lovelace!"` becomes + `["ada", "lovelace"]`. The same tokenizer is applied to every note body + (documents like PDFs contribute their extracted text). + +2. **Score notes lexically with BM25** — the classic Okapi BM25 ranking + function, computed from scratch on each request: + + ``` + score(note) = Σ over query terms t of + idf(t) · tf(t) · (k1 + 1) / (tf(t) + k1 · (1 − b + b · |note| / avg_len)) + ``` + + with `k1 = 1.5` and `b = 0.75` (the standard defaults), and the BM25 + inverse document frequency + `idf(t) = ln(1 + (N − df(t) + 0.5) / (df(t) + 0.5))` where `N` is the + number of notes and `df(t)` how many contain `t`. + + Two properties make BM25 the right default over plain TF-IDF: + + - **Term-frequency saturation** (`k1`): the 40th occurrence of + "compiler" in a note adds almost nothing. A note that spams one query + term cannot outrank a note that covers *all* the query terms. + - **Length normalization** (`b`): a term match in a 20-word note means + more than the same match buried in a 2,000-word note. + + The raw score is then divided by the query's **best achievable score** + (every term saturated to its ceiling), mapping the lexical component + into `[0, 1]` regardless of query length. + +3. **Boost by the knowledge graph** — this is the hybrid part. Every graph + entity whose name contains the query (or vice versa, token-normalized) + contributes a flat **+0.35** to every note it appears in. Because the + lexical score is normalized to at most 1.0, the boost is a strong, + predictable signal: a note that *mentions the entity you searched for* + reliably outranks one that merely shares vocabulary. Matched entities + are also returned alongside the results, which is what powers the + entity chips above the hit list in the app. + +4. **Cut, sort, snippet** — notes scoring below `0.01` are dropped, the + top 10 survive, and each hit gets a ~130-character snippet windowed + around the first query-term occurrence in the raw note text. + +## What it deliberately is not + +- **Not embeddings / semantic search.** Vector search is powerful but + unexplainable and nondeterministic across model versions. Graphier's + thesis is that a personal knowledge base should be auditable; optional + semantic search may arrive later as exactly that — optional + ([roadmap](../ROADMAP.md)). +- **Not a persistent index.** The vault is small by web-search standards + (thousands of notes, not millions of pages), so scoring on request keeps + the implementation transparent and always in sync with the files — there + is no index to corrupt or rebuild. +- **Not fuzzy.** Typos don't match. The graph boost softens this in + practice: entity names matched via the graph often rescue queries that + lexical scoring alone would miss. + +## Tuning it + +All the knobs are module constants in `search.py`: + +| Constant | Default | Effect of raising it | +| --- | --- | --- | +| `_K1` | `1.5` | repeated terms keep adding score for longer | +| `_B` | `0.75` | long notes are penalized harder | +| `_ENTITY_BOOST` | `0.35` | graph mentions dominate lexical relevance more | + +The search endpoint is `GET /api/search?q=…`, the MCP tool is +`search_vault`, and the library exposes the same ranking through the HTTP +API. All three share this one implementation.