You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
runtime.SearchCatalog matching assumes two properties that only hold for English-shaped text:
Words are whitespace- or case-delimited.normalizeSearchText splits on non-alphanumeric runes and camelCase boundaries. Text written without inter-word spacing (Chinese, Japanese, Thai, Khmer, Lao) collapses a whole phrase into one token, so a keyword query is neither equal to nor a prefix of that token and scores nothing.
Morphology is English suffix inflection. The stemming fallback strips -s, -ing, -ed. Other morphologies get no equivalent normalization.
The first property is the blocking one. For a catalog whose spec text is not English, search is not degraded, it is empty, and the documented agent loop (search "<intent>" --json -> commands show) loses its entry point.
This is pre-existing and script-general, not a regression from #161. Chinese is the instance that was measured because a real catalog was available.
Evidence
Measured against a generated CLI built from a real 2476-command swagger catalog whose summaries are Chinese, comparing the ranker before and after #161 in the same process:
45 sampled two-character keyword queries derived from the catalog's own summaries return zero results, 100% of the time, both before and after feat: add layered lexical search ranking #161.
Hand-written realistic intents (集群列表, 创建用户, 节点列表, 查询告警规则) all return zero results.
Mixed-script queries succeed only through their Latin tokens: 获取集群 GPU 配置 returns 20 results, every one matched on gpu.
Whole-phrase queries do work, because the query token is byte-identical to the indexed token. Only partial and keyword queries fail, which is what an agent actually issues.
The same root cause has a narrow English symptom: terms glued into a single token, such as acronym runs or unsplittable path segments like sriovnodesresources, are unreachable by any sub-term.
Direction
Add a sub-token matching layer that is script-agnostic rather than a CJK special case: substring or character n-gram containment inside an indexed token, scored strictly below the exact, prefix, and stem layers.
The hard part is the gate, not the score. A sub-token hit must be able to surface a command on its own where no finer tokenization exists, while a sub-token hit inside an English word must not, or the infix false positives removed in #161 come back (get must not reach list-widgets through widgets). Deciding this from the tokenizability of the indexed text, rather than from a declared language, keeps the runtime free of per-CLI locale configuration.
Acceptance criteria
A single keyword drawn from non-space-delimited text returns the commands whose text contains it, ranked below any command that matches a full token.
The relevance benchmark in pkg/runtime/search_test.go gains non-Latin fixture commands and keyword queries, reported with the same top-1, top-3, and MRR method.
No per-CLI locale, language, or tokenizer configuration.
Compatibility
Ranking behavior only. Catalog JSON shape, generated command shape, and the search -> commands show agent workflow must not change.
Problem
runtime.SearchCatalogmatching assumes two properties that only hold for English-shaped text:normalizeSearchTextsplits on non-alphanumeric runes and camelCase boundaries. Text written without inter-word spacing (Chinese, Japanese, Thai, Khmer, Lao) collapses a whole phrase into one token, so a keyword query is neither equal to nor a prefix of that token and scores nothing.-s,-ing,-ed. Other morphologies get no equivalent normalization.The first property is the blocking one. For a catalog whose spec text is not English,
searchis not degraded, it is empty, and the documented agent loop (search "<intent>" --json->commands show) loses its entry point.This is pre-existing and script-general, not a regression from #161. Chinese is the instance that was measured because a real catalog was available.
Evidence
Measured against a generated CLI built from a real 2476-command swagger catalog whose summaries are Chinese, comparing the ranker before and after #161 in the same process:
集群列表,创建用户,节点列表,查询告警规则) all return zero results.获取集群 GPU 配置returns 20 results, every one matched ongpu.The same root cause has a narrow English symptom: terms glued into a single token, such as acronym runs or unsplittable path segments like
sriovnodesresources, are unreachable by any sub-term.Direction
Add a sub-token matching layer that is script-agnostic rather than a CJK special case: substring or character n-gram containment inside an indexed token, scored strictly below the exact, prefix, and stem layers.
The hard part is the gate, not the score. A sub-token hit must be able to surface a command on its own where no finer tokenization exists, while a sub-token hit inside an English word must not, or the infix false positives removed in #161 come back (
getmust not reachlist-widgetsthroughwidgets). Deciding this from the tokenizability of the indexed text, rather than from a declared language, keeps the runtime free of per-CLI locale configuration.Acceptance criteria
getdoes not reachlist-widgets, and no-match queries stay empty.pkg/runtime/search_test.gogains non-Latin fixture commands and keyword queries, reported with the same top-1, top-3, and MRR method.Compatibility
Ranking behavior only. Catalog JSON shape, generated command shape, and the
search->commands showagent workflow must not change.