feat(config): allow a per-task model, so edge typing can use one that is good at it (#515) - #517
Merged
Merged
Conversation
… is good at it (#515) `model:` in `openkos.yaml` is global: every verb that calls a model uses it. The eight-model sweep in #516 found that a single global value cannot express what the measurements say -- the best relation typer measured is the worst extractor measured. `gemma2:27b` scores 0.81 on `evals/edge_typing/`'s 17-edge fixture against the configured default `qwen3:8b`'s 0.44, and on the same corpus collapses extraction to 0.24 subject recall on the long English fixture and 0.00 on the Spanish one. A new optional `models:` map overrides `model:` for one task at a time. Every task it does not name keeps the global default, so a workspace that does not opt in behaves exactly as before. Keyed by task, never by verb: `suggest_edge_types` is used by both `curate`'s Structure stage and standalone `suggest-relations`, and a per-verb key would let the two drift onto different models -- the drift #385's design already prevents by routing both through one write core. Three decisions the issue's principles settled but did not spell out: - `models:` validates rather than degrading. Its two passthrough precedents (`volatility_windows`, `type_tiers`) silently degrade a malformed value; this one refuses an unknown key or a non-string/blank value at read time. #515's rejection of a silent fallback does not distinguish a model that is missing from a name that is malformed -- a typo would keep writing relation types from a model nobody chose. - The cost gate discloses the model on a SEPARATE line, and only when the stage resolves something other than the global default. A suffix on `cost_line` would have rewritten the literal the `curate-command` spec pins byte-identical, for every workspace including those that never opted in. - `query` and `curate`'s locality probe name no task. `query` has no harness, so #508's rule forbids picking a model for it; locality is a property of the host, not of any task. `curate` now tracks availability per model rather than per run. One failed connection no longer settles reachability for models it never contacted: Structure failing for want of `gemma2:27b` says nothing about Metadata's model. The deliberate cost is one failed connection per distinct model against a dead server; clients are cached by model so stages sharing a tag share one connection. The `curate-command` spec moves with the code rather than being quietly broken. A named model that is not installed fails only the stage that named it, with an `ollama pull` remediation naming THAT model -- never the global default, which would send the operator to pull something already installed while the missing one stayed missing. Only `edge_typing` has a harness today. The other four keys are accepted because restricting the schema would be arbitrary, and the docs say plainly which has evidence behind it. Suite 4052 -> 4079. Closes #515
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #515.
Why
model:inopenkos.yamlis global — every verb that calls a model uses it. The eight-model sweep in #516 found that a single global value cannot express what the measurements say: the best relation typer measured is the worst extractor measured.large-03EN /small-04ES)gemma2:27bqwen3:8b(default)A +0.37 was sitting in a config value that could not be collected without moving the extraction pipeline
evals/extraction_cap/tuned onqwen3:8b— and no harness would have caught that.What ships
Valid keys:
extraction,adjudication,edge_typing,volatility_typing,contradiction. Additive — a task the map does not name keepsmodel:, so a workspace that does not opt in behaves exactly as before.Keyed by task, never by verb:
suggest_edge_typesis used by bothcurate's Structure stage and standalonesuggest-relations, and a per-verb key would let the two drift onto different models — the drift #385's design already prevents by routing both through one write core.Three decisions the issue's principles settled but did not spell out
models:validates rather than degrading. Its two passthrough precedents (volatility_windows,type_tiers) silently degrade a malformed value. This one refuses an unknown key or a non-string/blank value at read time, because curate/relate: allow a per-task model, so edge typing can use one that is good at it #515's rejection of a silent fallback does not distinguish a model that is missing from a name that is malformed — a typo likeedge_types:would keep writing relation types from a model nobody chose.cost_linewould have rewritten the literal thecurate-commandspec pins byte-identical ("Below-Cap Cost-Line Output Is Byte-Identical To Pre-Change Behavior") for every workspace, including those that never opted in. The disclosure gap only exists when the models differ.queryandcurate's locality probe name no task.queryhas no harness, so curate: confidence-threshold auto-acceptance needs the suggesters to expose confidence first #508's rule forbids picking a model for it; locality is a property of the host, not of any task.The invariant that moved
curatenow tracks Ollama availability per model rather than per run. One failed connection no longer settles reachability for models it never contacted — Structure failing for want ofgemma2:27bsays nothing about Metadata's model, and skipping Metadata on that basis refuses work that would have succeeded.The deliberate cost, stated rather than discovered: against a genuinely dead server a run pays one failed connection per distinct model instead of one per run. Clients are cached by model so stages sharing a tag share one connection. The
curate-commandspec moves with the code rather than being quietly broken (two new requirements, one corrected sentence).A named model that is not installed fails only the stage that named it, with an
ollama pullremediation naming that model — never the global default, which would send the operator to pull something already installed while the missing one stayed missing.A defect this caught on the way
tests/unit/cli/test_chat_timeout_wiring.pyidentifies a chat client by syntax (ast.Attributenamedmodel). Migrating a site toconfig.resolve_task_model(...)makes it anast.Call, which the detector did not match: the site vanished fromseen(2 → 1) and both thechat_timeoutandmax_generation_tokensguards would have silently stopped protecting it while every assertion still passed. Caught by a test written before the migration; the detector now recognizesast.Calland walksast.IfExp.That is also why
curate.pywritesmodel=config.resolve_task_model(ctx.cfg, stage.task)in the client construction despite holding an equal local — a bareast.Nameis indistinguishable there from the liveness probes' ownmodel=locals, which must not be governed by those two settings. It is commented at the call site.Scope note
This does not close #513. The shipped default stays
qwen3:8bat 0.44;gemma2:27bis offered in the commented template, not packaged, since it costs a 15.6 GB pull. #515 is the mechanism #513 needed, not its resolution.Evidence
ruff check,ruff format --check, andmypy .all clean.resolve_task_model's fallback and the unknown-key guard.edge_typinghas a harness; the other four keys are accepted because restricting the schema would be arbitrary, anddocs/cli.mdsays plainly which has evidence behind it.🤖 Generated with Claude Code
https://claude.ai/code/session_01MraEQooNmnhKbUqaQ2xU3A