perf: serve partial symbol lookup from a Qdrant full-text index

README.md

@@ -283,7 +283,7 @@ Tests live under `tests/`:
 | Tool                    | Description                                                                                          |
 |-------------------------|------------------------------------------------------------------------------------------------------|
 | `search_code`           | Hybrid (dense + BM25) search by query, with optional filters for language, service, symbol type      |
-| `find_symbol`           | Look up a symbol by name — exact match, or case-insensitive substring when `exact=false`             |
+| `find_symbol`           | Look up a symbol by name — exact match, or case-insensitive token match when `exact=false`          |
 | `find_usages`           | Find code that references a given symbol name (semantic search, then excludes the definition itself) |
 | `get_code_context`      | Fetch the full source of a file — or a specific symbol within it — directly from GitHub              |
 | `reindex`               | Trigger code indexing of one or all services (incremental by default; `force` to re-embed)           |
@@ -293,6 +293,13 @@ Tests live under `tests/`:
 | `list_indexed_services` | List indexed services with chunk and file counts, languages, and last-indexed time                   |
 | `index_stats`           | Show Qdrant collection statistics and configured services                                            |
 
+`find_symbol(exact=false)` matches against a full-text index over the symbol name's camelCase/snake_case tokens, so
+`order` or `ord` finds `placeOrderRequest` in ~2 ms regardless of collection size. Mid-token fragments (`rder`) still
+match, but fall back to a client-side scan that is linear in collection size. Collections indexed before this field
+existed use that same fallback until reindexed — and because change detection skips unchanged files, populating the
+field needs a **force** reindex (`POST /reindex {"force": true}`), which re-embeds every symbol. See
+[docs/retrieval-rrf.md](docs/retrieval-rrf.md#name-lookup-find_by_name).
+
 ## MCP Prompts
 
 | Prompt                   | Arguments | Description                                                                                                                                                                                 |

docs/retrieval-rrf.md

@@ -79,17 +79,39 @@ Queries Qdrant with a keyword filter on the `symbol_name` payload field:
 FieldCondition(key="symbol_name", match=MatchValue(value=name))
 ```
 
-Returns up to 20 exact matches via a scroll operation. Additional filters for `symbol_type` and `service` are stacked into the same `must` list. No vectors are fetched.
+Returns up to 20 exact matches via a scroll operation. `symbol_name` carries a `KEYWORD` payload index, so this filter is served by Qdrant. Additional filters for `symbol_type` and `service` are stacked into the same `must` list. No vectors are fetched.
 
-### Substring mode (`exact=False`, default)
+### Partial mode (`exact=False`, default)
 
-Qdrant has no native text-contains index for partial name matching. The implementation falls back to a **client-side substring scan**:
+Matching is **token-aware and case-insensitive**, served server-side by a full-text payload index.
 
-1. Scroll the collection in batches of 200 points
-2. For each point, check whether `name.lower()` appears in `payload['symbol_name'].lower()`
-3. Collect up to 50 matches, then stop
+At index time, `_symbol_to_payload()` stores a derived `symbol_name_tokens` field built by `symbol_name_tokens()`. It contains three things: the original identifier, its camelCase/PascalCase/snake_case subwords (via the same `split_code_identifiers()` helper that feeds BM25), and *suffix joins* — the identifier re-joined from each subword boundary onwards. That field carries a `TEXT` index with the `PREFIX` tokenizer (`lowercase=True`, token length 2–30), and `find_by_name` queries it with a single `MatchText`-filtered scroll returning up to 50 matches.
 
-This is **O(N)** in collection size — it scans every indexed symbol in the collection (or service subset, if filtered). On large codebases with hundreds of thousands of symbols, this can be slow.
+**Why suffix joins.** A `PREFIX` index only matches a query that prefixes a stored token. Storing just the identifier and its subwords means `GetWebAuthnSession` is reachable by `GetWeb` (prefix of the whole name) and by `Authn` (a subword), but *not* by `WebAuth` — that query spans `Web` + `Authn` and is not anchored at the start of the name. The old client-side substring scan found it, so without the joins this would be a silent recall regression: `WebAuth` returned 12 of 18 real matches on a live collection. Emitting `WebAuthnSession`, `AuthnSession`, `Session` as their own tokens makes every subword-boundary query a real index hit. Joins are capped at the first `MAX_SUFFIX_JOIN_SUBWORDS` (8) subwords and skipped entirely for names containing whitespace — markdown headings, CSS selector lists and dependency coordinates are prose, whose words are already separate tokens.
+
+Matching a query against `placeOrderRequest`, measured against a real Qdrant:
+
+| Query | Matches | Latency | Why |
+| --- | --- | --- | --- |
+| `order` | ✅ | ~2 ms | full subword token — indexed lookup |
+| `ord`, `plac`, `reques` | ✅ | ~2 ms | `PREFIX` tokenizer indexes every token prefix |
+| `orderRequest` | ✅ | ~2 ms | suffix join — indexed lookup |
+| `place order` | ✅ | — | `MatchText` requires all query tokens to match |
+| `rder`, `quest` | ✅ | O(N) | mid-token: no index hit, served by the client-side fallback (see below) |
+
+Results are then ranked exact name → prefix → remainder, because Qdrant returns scrolled points in point-id order and would otherwise bury the exact hit.
+
+**Mid-token queries are not served by the index, and cost O(N) client-side.** A fragment that is not a prefix of any stored token (`rder` inside `placeOrderRequest`) produces **zero** rows from Qdrant — the `PREFIX` tokenizer indexes token prefixes only, and Qdrant does not silently scan on your behalf. Such queries land in `_find_by_name_scanning()` below. Qdrant offers no n-gram tokenizer, so arbitrary-substring matching cannot be made sublinear. What issue [#72](https://github.com/GoodbyePlanet/semcode/issues/72) removed is the client-side scan for *token and subword-boundary* queries, which are the overwhelming majority.
+
+**Fallback.** When the full-text filter returns zero results, `_find_by_name_scanning()` runs the pre-#72 behaviour — scroll in batches of 200 and substring-match `symbol_name` in Python. Two distinct cases reach it, indistinguishable from each other: a mid-token fragment, and a collection indexed before `symbol_name_tokens` existed (where `MatchText` on the absent field matches nothing). A genuinely unmatched query such as `zzz` also triggers it, and is the worst case — it scans the entire collection to return nothing. Measured cost of that scan: 0.9 s at 50k symbols, 8.8 s at 250k, linear thereafter.
+
+**Migrating an existing collection.** The payload indexes are created on every startup, including for collections that predate them, so no collection drop is needed. But populating `symbol_name_tokens` requires the points to be rewritten, and incremental indexing compares Git blob SHAs and skips unchanged files — a plain `make index-code` on an unchanged repo reports `{"files": 0, "chunks": 0, "skipped": N}` and leaves the field empty. Use a **force** reindex:
+
+```bash
+curl -X POST http://localhost:8090/reindex -H 'Content-Type: application/json' -d '{"force": true}'
+```
+
+This re-embeds every symbol, so it costs a full pass of embedding-provider calls (709 symbols took ~4 minutes on a hosted provider). Until it runs, partial lookups keep working via the fallback scan.
 
 ---
 
@@ -117,7 +139,7 @@ Each result includes: symbol name and type, RRF score, file location (path + lin
 find_symbol(name: str, symbol_type: str | None, service: str | None, chunk_tier: str | None, exact: bool = False) -> str
 ```
 
-Name-based lookup via `store.find_by_name()`. Does not use vectors or RRF. Supports filtering by `chunk_tier` (`"method"` or `"class"`) in addition to `symbol_type` and `service`. Returns up to 20 (exact) or 50 (substring) matches. Each result includes: name, type, location, package, parent class, and source (first 800 characters).
+Name-based lookup via `store.find_by_name()`. Does not use vectors or RRF. Supports filtering by `chunk_tier` (`"method"` or `"class"`) in addition to `symbol_type` and `service`. Returns up to 20 (exact) or 50 (partial) matches, exact names first. Each result includes: name, type, location, package, parent class, and source (first 800 characters).
 
 ### `find_usages`
 
@@ -174,7 +196,9 @@ public OrderResult processOrder(OrderRequest request) {
 
 **RRF constant is not configurable** — Qdrant's `k=60` default is used. There is no way to adjust this via configuration. The choice of `k` affects how strongly RRF rewards documents appearing in both lists versus only one. A lower `k` amplifies the benefit of appearing in both; a higher `k` makes the fusion more uniform.
 
-**Substring scan is O(N)** — `find_by_name` with `exact=False` scans the entire collection client-side. On a codebase with 500,000 indexed symbols, every partial-name lookup scrolls through all symbols in batches. A Qdrant full-text index on `symbol_name` would solve this but is not currently implemented.
+**Mid-token queries are still O(N), on the client** — `find_by_name` with `exact=False` is served by the `symbol_name_tokens` full-text index, but only a query prefixing the identifier, one of its subwords, or one of its suffix joins is a real index hit (~2 ms, flat from 50k to 250k symbols). A mid-token fragment such as `rder` is not a prefix of any stored token, so Qdrant returns nothing and `_find_by_name_scanning()` pages the collection over the wire instead — 0.9 s at 50k symbols, 8.8 s at 250k. Qdrant offers no n-gram tokenizer, so there is no index that would make arbitrary-substring matching sublinear. Note that a query matching *no* symbol pays this same full scan.
+
+**Token semantics differ from substring matching** — now that lookups are token-aware, `Auth` no longer matches `oauth_db` or `oauth2-session`: `auth` is not a prefix of the token `oauth2`. This is intentional, and differs from the pre-#72 substring scan. Subword-boundary queries (`WebAuth` → `GetWebAuthnSession`) *are* matched, via suffix joins.
 
 **`find_usages` depends on dense quality** — the "code that uses or references X" query wrapper is a heuristic. If the dense model doesn't associate the phrasing with caller patterns, results will be poor. There is no static call-graph analysis; the tool is entirely retrieval-based.
 

server/embeddings/code_tokenizer.py

@@ -2,6 +2,11 @@
 
 import re
 
+# Suffix joins are only emitted for the first N subwords of an identifier. The
+# joins are O(k^2) in total characters for k subwords, and their value drops off
+# fast — nobody starts a lookup nine subwords into a name.
+MAX_SUFFIX_JOIN_SUBWORDS = 8
+
 
 def split_code_identifiers(text: str) -> str:
     """Split camelCase/PascalCase/snake_case into subwords; keep originals alongside.
@@ -14,3 +19,34 @@ def split_code_identifiers(text: str) -> str:
     expanded = expanded.replace("_", " ")
     expanded = expanded.replace("-", " ")
     return text + "\n" + expanded
+
+
+def symbol_name_tokens(name: str) -> str:
+    """Build the text indexed behind `find_symbol(exact=False)`.
+
+    Extends `split_code_identifiers()` with *suffix joins* — the identifier
+    re-joined from each subword boundary onwards. Without them, a `PREFIX`
+    full-text index only matches a query that prefixes the whole identifier or
+    one single subword, so `WebAuth` would miss `GetWebAuthnSession`: the query
+    spans `Web` + `Authn` but is not anchored at the start of the name. Emitting
+    `WebAuthnSession` as its own token makes that a real index hit.
+
+    Suffix joins are skipped for names containing whitespace (markdown headings,
+    CSS selector lists, dependency coordinates). Those are prose rather than
+    concatenated identifiers — their words are already separate tokens, so joins
+    would add nothing but index bulk.
+    """
+    base = split_code_identifiers(name)
+    if re.search(r"\s", name):
+        return base
+
+    subwords = base.split("\n", 1)[1].split()
+    if len(subwords) < 2:
+        return base
+
+    capped = subwords[:MAX_SUFFIX_JOIN_SUBWORDS]
+    joins = ["".join(capped[i:]) for i in range(1, len(capped))]
+    # dict.fromkeys dedupes while preserving order; a join can repeat the
+    # original name (snake_case) or a lone trailing subword.
+    extra = [j for j in dict.fromkeys(joins) if j not in {name, *subwords}]
+    return base + ("\n" + " ".join(extra) if extra else "")

server/indexer/pipeline.py

@@ -14,12 +14,13 @@
 from server.embeddings import get_embedding_provider
 from server.embeddings.base import EmbeddingProvider
 from server.embeddings.bm25 import BM25SparseProvider, get_sparse_embedding_provider
+from server.embeddings.code_tokenizer import symbol_name_tokens
 from server.indexer.cleanup import prune_orphaned_services
 from server.indexer.github_source import fetch_blob_content, list_github_files
 from server.parser.base import CodeSymbol, ParseError
 from server.parser.registry import parse_file
 from server.state import get_reindex_lock, get_service_registry
-from server.store.qdrant import QdrantStore
+from server.store.qdrant import SYMBOL_TOKENS_FIELD, QdrantStore
 from server.store.service_registry import ServiceRegistry, load_effective_services
 
 logger = logging.getLogger(__name__)
@@ -136,6 +137,7 @@ def _symbol_to_payload(
 ) -> dict[str, Any]:
     return {
         "symbol_name": symbol.name,
+        SYMBOL_TOKENS_FIELD: symbol_name_tokens(symbol.name),
         "symbol_type": symbol.symbol_type,
         "language": symbol.language,
         "service": service_name,

server/store/qdrant.py

@@ -11,6 +11,7 @@
     Fusion,
     FusionQuery,
     HnswConfigDiff,
+    MatchText,
     MatchValue,
     OptimizersConfigDiff,
     PayloadSchemaType,
@@ -20,11 +21,38 @@
     SparseIndexParams,
     SparseVector,
     SparseVectorParams,
+    TextIndexParams,
+    TextIndexType,
+    TokenizerType,
     VectorParams,
 )
 
 from server.config import settings
 
+# Payload field holding the tokenized form of symbol_name (original identifier plus
+# its camelCase/snake_case subwords). Backed by a full-text index so partial-name
+# lookups are served by Qdrant instead of a client-side scan.
+SYMBOL_TOKENS_FIELD = "symbol_name_tokens"
+
+FUZZY_MATCH_LIMIT = 50
+
+
+def _rank_by_name(points: list[ScoredPoint], name: str) -> list[ScoredPoint]:
+    """Exact name matches first, then prefix matches, then the rest.
+
+    Qdrant returns scrolled points in point-id order, which would otherwise bury an
+    exact hit underneath incidental partial matches.
+    """
+    name_lower = name.lower()
+
+    def rank(point: ScoredPoint) -> int:
+        symbol_name = (point.payload.get("symbol_name") or "").lower()
+        if symbol_name == name_lower:
+            return 0
+        return 1 if symbol_name.startswith(name_lower) else 2
+
+    return sorted(points, key=rank)
+
 
 def _symbol_point_id(
     service: str, file_path: str, symbol_name: str, start_line: int
@@ -43,6 +71,9 @@ async def ensure_collection(self) -> None:
         exists = await self._client.collection_exists(self._collection)
         if exists:
             await self._validate_dimensions()
+            # Payload indexes are created unconditionally so that indexes added
+            # in later versions also reach collections created before them.
+            await self._create_payload_indexes()
             return
         await self._client.create_collection(
             collection_name=self._collection,
@@ -85,13 +116,26 @@ async def _create_payload_indexes(self) -> None:
             "chunk_tier",
             "parent_name",
             "file_path",
+            "symbol_name",
         ]
         for field in keyword_fields:
             await self._client.create_payload_index(
                 collection_name=self._collection,
                 field_name=field,
                 field_schema=PayloadSchemaType.KEYWORD,
             )
+        # PREFIX tokenizer so a partial query ("Ord") matches a full token ("Order").
+        await self._client.create_payload_index(
+            collection_name=self._collection,
+            field_name=SYMBOL_TOKENS_FIELD,
+            field_schema=TextIndexParams(
+                type=TextIndexType.TEXT,
+                tokenizer=TokenizerType.PREFIX,
+                min_token_len=2,
+                max_token_len=30,
+                lowercase=True,
+            ),
+        )
 
     async def upsert_chunks(
         self,
@@ -283,10 +327,47 @@ async def find_by_name(
             )
             return list(results)
 
+        token_filter = Filter(
+            must=[
+                *must,
+                FieldCondition(key=SYMBOL_TOKENS_FIELD, match=MatchText(text=name)),
+            ]
+        )
+        results, _ = await self._client.scroll(
+            collection_name=self._collection,
+            scroll_filter=token_filter,
+            limit=FUZZY_MATCH_LIMIT,
+            with_payload=True,
+            with_vectors=False,
+        )
+        matches = list(results)
+        if not matches:
+            # Two distinct cases reach here, both indistinguishable from an empty
+            # MatchText result:
+            #   1. The collection predates SYMBOL_TOKENS_FIELD, so the filter runs
+            #      against an absent field and matches nothing until a force reindex.
+            #   2. A mid-token fragment ("rder", "asskey"). The PREFIX tokenizer only
+            #      indexes token *prefixes*, so Qdrant returns nothing for these —
+            #      it does not resolve them server-side.
+            matches = await self._find_by_name_scanning(name, base_filter)
+        return _rank_by_name(matches, name)
+
+    async def _find_by_name_scanning(
+        self, name: str, base_filter: Filter | None
+    ) -> list[ScoredPoint]:
+        """Substring fallback: scrolls the collection and matches in Python.
+
+        Pre-#72 behaviour, retained for collections indexed before
+        SYMBOL_TOKENS_FIELD existed and for mid-token fragments, which the
+        PREFIX index cannot serve. O(N) in collection size, and unlike the
+        indexed path it pages every payload over the wire — measured at 8.8 s
+        for a no-match query over 250k symbols, so it is a real cliff on large
+        collections, not a rounding error.
+        """
         name_lower = name.lower()
         matches: list[ScoredPoint] = []
         offset = None
-        while len(matches) < 50:
+        while len(matches) < FUZZY_MATCH_LIMIT:
             batch, offset = await self._client.scroll(
                 collection_name=self._collection,
                 scroll_filter=base_filter,
@@ -295,9 +376,11 @@ async def find_by_name(
                 with_payload=True,
                 with_vectors=False,
             )
-            for r in batch:
-                if name_lower in (r.payload.get("symbol_name") or "").lower():
-                    matches.append(r)
+            matches.extend(
+                r
+                for r in batch
+                if name_lower in (r.payload.get("symbol_name") or "").lower()
+            )
             if offset is None:
                 break
         return matches