Skip to content

feat(runtime): make catalog search work for non-English specs #162

Description

@samzong

Problem

runtime.SearchCatalog matching assumes two properties that only hold for English-shaped text:

  1. Words are whitespace- or case-delimited. normalizeSearchText splits on non-alphanumeric runes and camelCase boundaries. Text written without inter-word spacing (Chinese, Japanese, Thai, Khmer, Lao) collapses a whole phrase into one token, so a keyword query is neither equal to nor a prefix of that token and scores nothing.
  2. Morphology is English suffix inflection. The stemming fallback strips -s, -ing, -ed. Other morphologies get no equivalent normalization.

The first property is the blocking one. For a catalog whose spec text is not English, search is not degraded, it is empty, and the documented agent loop (search "<intent>" --json -> commands show) loses its entry point.

This is pre-existing and script-general, not a regression from #161. Chinese is the instance that was measured because a real catalog was available.

Evidence

Measured against a generated CLI built from a real 2476-command swagger catalog whose summaries are Chinese, comparing the ranker before and after #161 in the same process:

  • 45 sampled two-character keyword queries derived from the catalog's own summaries return zero results, 100% of the time, both before and after feat: add layered lexical search ranking #161.
  • Hand-written realistic intents (集群列表, 创建用户, 节点列表, 查询告警规则) all return zero results.
  • Mixed-script queries succeed only through their Latin tokens: 获取集群 GPU 配置 returns 20 results, every one matched on gpu.
  • Whole-phrase queries do work, because the query token is byte-identical to the indexed token. Only partial and keyword queries fail, which is what an agent actually issues.

The same root cause has a narrow English symptom: terms glued into a single token, such as acronym runs or unsplittable path segments like sriovnodesresources, are unreachable by any sub-term.

Direction

Add a sub-token matching layer that is script-agnostic rather than a CJK special case: substring or character n-gram containment inside an indexed token, scored strictly below the exact, prefix, and stem layers.

The hard part is the gate, not the score. A sub-token hit must be able to surface a command on its own where no finer tokenization exists, while a sub-token hit inside an English word must not, or the infix false positives removed in #161 come back (get must not reach list-widgets through widgets). Deciding this from the tokenizability of the indexed text, rather than from a declared language, keeps the runtime free of per-CLI locale configuration.

Acceptance criteria

  • A single keyword drawn from non-space-delimited text returns the commands whose text contains it, ranked below any command that matches a full token.
  • The feat: add layered lexical search ranking #161 precision cases still hold: get does not reach list-widgets, and no-match queries stay empty.
  • The relevance benchmark in pkg/runtime/search_test.go gains non-Latin fixture commands and keyword queries, reported with the same top-1, top-3, and MRR method.
  • No per-CLI locale, language, or tokenizer configuration.

Compatibility

Ranking behavior only. Catalog JSON shape, generated command shape, and the search -> commands show agent workflow must not change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions