Skip to content

feat(config): allow a per-task model, so edge typing can use one that is good at it (#515) - #517

Merged
jasonssdev merged 1 commit into
mainfrom
feat/515-per-task-model
Aug 10, 2026
Merged

feat(config): allow a per-task model, so edge typing can use one that is good at it (#515)#517
jasonssdev merged 1 commit into
mainfrom
feat/515-per-task-model

Conversation

@jasonssdev

Copy link
Copy Markdown
Owner

Closes #515.

Why

model: in openkos.yaml is global — every verb that calls a model uses it. The eight-model sweep in #516 found that a single global value cannot express what the measurements say: the best relation typer measured is the worst extractor measured.

model edge-typing accuracy extraction subject recall (large-03 EN / small-04 ES)
gemma2:27b 0.81 0.24 / 0.00
qwen3:8b (default) 0.44 0.81 / 0.76

A +0.37 was sitting in a config value that could not be collected without moving the extraction pipeline evals/extraction_cap/ tuned on qwen3:8b — and no harness would have caught that.

What ships

model: qwen3:8b        # unchanged default for everything
models:
  edge_typing: gemma2:27b

Valid keys: extraction, adjudication, edge_typing, volatility_typing, contradiction. Additive — a task the map does not name keeps model:, so a workspace that does not opt in behaves exactly as before.

Keyed by task, never by verb: suggest_edge_types is used by both curate's Structure stage and standalone suggest-relations, and a per-verb key would let the two drift onto different models — the drift #385's design already prevents by routing both through one write core.

Three decisions the issue's principles settled but did not spell out

  1. models: validates rather than degrading. Its two passthrough precedents (volatility_windows, type_tiers) silently degrade a malformed value. This one refuses an unknown key or a non-string/blank value at read time, because curate/relate: allow a per-task model, so edge typing can use one that is good at it #515's rejection of a silent fallback does not distinguish a model that is missing from a name that is malformed — a typo like edge_types: would keep writing relation types from a model nobody chose.
  2. The cost gate discloses the model on a separate line, only when it differs from the global default. A suffix on cost_line would have rewritten the literal the curate-command spec pins byte-identical ("Below-Cap Cost-Line Output Is Byte-Identical To Pre-Change Behavior") for every workspace, including those that never opted in. The disclosure gap only exists when the models differ.
  3. query and curate's locality probe name no task. query has no harness, so curate: confidence-threshold auto-acceptance needs the suggesters to expose confidence first #508's rule forbids picking a model for it; locality is a property of the host, not of any task.

The invariant that moved

curate now tracks Ollama availability per model rather than per run. One failed connection no longer settles reachability for models it never contacted — Structure failing for want of gemma2:27b says nothing about Metadata's model, and skipping Metadata on that basis refuses work that would have succeeded.

The deliberate cost, stated rather than discovered: against a genuinely dead server a run pays one failed connection per distinct model instead of one per run. Clients are cached by model so stages sharing a tag share one connection. The curate-command spec moves with the code rather than being quietly broken (two new requirements, one corrected sentence).

A named model that is not installed fails only the stage that named it, with an ollama pull remediation naming that model — never the global default, which would send the operator to pull something already installed while the missing one stayed missing.

A defect this caught on the way

tests/unit/cli/test_chat_timeout_wiring.py identifies a chat client by syntax (ast.Attribute named model). Migrating a site to config.resolve_task_model(...) makes it an ast.Call, which the detector did not match: the site vanished from seen (2 → 1) and both the chat_timeout and max_generation_tokens guards would have silently stopped protecting it while every assertion still passed. Caught by a test written before the migration; the detector now recognizes ast.Call and walks ast.IfExp.

That is also why curate.py writes model=config.resolve_task_model(ctx.cfg, stage.task) in the client construction despite holding an equal local — a bare ast.Name is indistinguishable there from the liveness probes' own model= locals, which must not be governed by those two settings. It is commented at the call site.

Scope note

This does not close #513. The shipped default stays qwen3:8b at 0.44; gemma2:27b is offered in the commented template, not packaged, since it costs a 15.6 GB pull. #515 is the mechanism #513 needed, not its resolution.

Evidence

  • Suite 4052 → 4079 (27 new tests), ruff check, ruff format --check, and mypy . all clean.
  • Strict TDD: every slice red before green, with mutations verified on resolve_task_model's fallback and the unknown-key guard.
  • Only edge_typing has a harness; the other four keys are accepted because restricting the schema would be arbitrary, and docs/cli.md says plainly which has evidence behind it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MraEQooNmnhKbUqaQ2xU3A

… is good at it (#515)

`model:` in `openkos.yaml` is global: every verb that calls a model uses
it. The eight-model sweep in #516 found that a single global value cannot
express what the measurements say -- the best relation typer measured is
the worst extractor measured. `gemma2:27b` scores 0.81 on
`evals/edge_typing/`'s 17-edge fixture against the configured default
`qwen3:8b`'s 0.44, and on the same corpus collapses extraction to 0.24
subject recall on the long English fixture and 0.00 on the Spanish one.

A new optional `models:` map overrides `model:` for one task at a time.
Every task it does not name keeps the global default, so a workspace that
does not opt in behaves exactly as before.

Keyed by task, never by verb: `suggest_edge_types` is used by both
`curate`'s Structure stage and standalone `suggest-relations`, and a
per-verb key would let the two drift onto different models -- the drift
#385's design already prevents by routing both through one write core.

Three decisions the issue's principles settled but did not spell out:

- `models:` validates rather than degrading. Its two passthrough
  precedents (`volatility_windows`, `type_tiers`) silently degrade a
  malformed value; this one refuses an unknown key or a non-string/blank
  value at read time. #515's rejection of a silent fallback does not
  distinguish a model that is missing from a name that is malformed --
  a typo would keep writing relation types from a model nobody chose.
- The cost gate discloses the model on a SEPARATE line, and only when
  the stage resolves something other than the global default. A suffix
  on `cost_line` would have rewritten the literal the `curate-command`
  spec pins byte-identical, for every workspace including those that
  never opted in.
- `query` and `curate`'s locality probe name no task. `query` has no
  harness, so #508's rule forbids picking a model for it; locality is a
  property of the host, not of any task.

`curate` now tracks availability per model rather than per run. One
failed connection no longer settles reachability for models it never
contacted: Structure failing for want of `gemma2:27b` says nothing about
Metadata's model. The deliberate cost is one failed connection per
distinct model against a dead server; clients are cached by model so
stages sharing a tag share one connection. The `curate-command` spec
moves with the code rather than being quietly broken.

A named model that is not installed fails only the stage that named it,
with an `ollama pull` remediation naming THAT model -- never the global
default, which would send the operator to pull something already
installed while the missing one stayed missing.

Only `edge_typing` has a harness today. The other four keys are accepted
because restricting the schema would be arbitrary, and the docs say
plainly which has evidence behind it.

Suite 4052 -> 4079.

Closes #515
@jasonssdev
jasonssdev merged commit 0b52d2a into main Aug 10, 2026
6 checks passed
@jasonssdev
jasonssdev deleted the feat/515-per-task-model branch August 10, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant