Fix/the config cannot name a provider that does not exist - #292
Merged
evkir merged 4 commits intoSep 21, 2026
Merged
Conversation
CYBERAI_LLM_PROVIDER=gemini produced a config whose provider was the literal string "gemini". from_env passed str into a field declared Literal, and cyberai/core/config.py was outside [tool.mypy] files, so the checker never saw the assignment either. The name was answered at call time, with api_key_for resolving a credential for a provider that does not exist: a run asked to use one vendor could reach another under a key nobody named. _env_provider narrows at the boundary instead of raising, per the rule already stated on _env_bool: a value outside the declared set is nobody having chosen, and garbage in a variable must not abort a scan at startup. Nine call sites reach from_env, one of them inside an MCP tool where an exception is not a message anybody reads. The declared set is the Literal itself, read through get_args, so the checker and the runtime reader answer from one list. A hand-written copy would be a rule that can disagree with itself; the architecture test that compares the credential map to the Literal now covers this reader too. The module enters the typing scope in the same commit, which is what makes the assignment visible to CI. Two more errors went with it: phase_models was an unparameterised dict, and save() had no return annotation. Measured before the change: adding the file to the scope costs three errors and one module, not an import closure. Measured after: 101 of 172 modules, 280 errors outside, drift none. Both crossing counts fell -- 22 modules reaching 29 became 20 reaching 28 -- because the edge moved inward, not because an import went away. docs/architecture/typing-scope.md says so, and its per-module table was remeasured while it was open: async_agent.py read 17 and reads 12. Mutation: five mutants, five killed. The fallback needed an assertion where the branch and the default differ -- from_env passes "openai", which is also the field default, so a reader that ignored its argument passed every test until one read with a different default.
Two boundaries were left after the environment reader. --provider assigned its string onto config.llm.provider, a field declared Literal, and the router put RoutingConfig.air_gapped_provider, declared str, into a replace() on the same field. Both files were outside [tool.mypy] files, so neither assignment was visible to CI. The two boundaries get different answers on purpose. The environment reader narrows silently: a stale variable in a .env file must not abort a scan. A flag was typed seconds ago by someone who can read the reply, so click.Choice refuses it and names the set. One declared list stands behind both -- the Choice is built from the Literal, not from a second list written by hand. The router's credential line was typed wrong in a way that mattered. api_key was inferred str from the cloud branch while api_key_for returns str | None, and None is the correct answer for the air-gapped provider, which takes no key. The test asserting exactly that has passed since the day it was written; the annotation was the part that disagreed. Also in this commit, because the checker reaches them together: Provider moved above RoutingConfig so both dataclasses can name it, and air_gapped_provider carries the type instead of str. Measured before the change: two errors in model_router.py, one in __main__.py, three modules and no import closure. Measured after: 103 of 172 modules, 277 errors outside, drift none. The crossing counts rose to 22 reaching 31 -- the opposite direction from the previous commit on the same day, because an entry point brings its own imports to the edge. __main__.py reaches cli/audit_verify.py, cli/detector_eval.py and cli/scope.py, which nothing else in the scope touches. The page says so. Mutation: four mutants, four killed, and one of them only by mypy. Widening air_gapped_provider back to str leaves every test green -- a type is not executed, so pytest cannot see it. That mutant would have survived both channels before this commit, which is the argument for the scope entry rather than for the annotation alone.
…epted validate_exploit_scope is public API, re-exported from agents.exploit, and declared authorized_scope: List[str] = None with attack_paths the same way. PEP 484 prohibits the implicit Optional and mypy rejects it by default, so the two lines were two errors inherited by everyone who imported the module -- and it sat outside [tool.mypy] files, so nobody was ever shown them. The behaviour does not move. The body has always read both arguments through a falsiness check: `if authorized_scope` for the scope, `attack_paths or []` for the paths. None was accepted before and is accepted now; the declaration was the part that disagreed with the code under it. Measured rather than assumed: the tail said nine call sites in four test files. There are more than twenty, and two of them are production -- orchestrator.py:174 and :396, both passing positional lists that are never None. The public signature was wrong for callers who do not exist yet, which is the only kind of caller a library API can be wrong for. The guard reads the signature, not a call. A test that passed None and got a result would have passed before this commit too, because the body already handled it; only inspect.signature plus get_type_hints can see the defect. A control asserts the rule is not vacuous by naming the two arguments that default to None. Measured after: 104 of 172 modules, 275 errors outside, drift none. Only one crossing count moved, reached 31 to 30: this module imports nothing outside the scope, so it was never a crosser and simply stopped being reached. A leaf pays one counter where an entry point pays both. Mutation: four mutants, three killed, one survived by design. Rewriting Optional[List[...]] as List[...] | None survives, and should: the two spell one type. Restoring the implicit Optional dies in both channels, and removing the `or []` guard dies in both as well.
…undary Risk 24: configuration naming a provider that does not exist. The row names a test per boundary -- the environment reader and the CLI parser -- rather than one test for both, because they answer differently on purpose and a single row would hide which half is guarded. Per the lesson risk 20 taught, each named test measures behaviour: what the config ends up holding and what the parser does with an unknown name, not whether a field exists. The README said `# openai | anthropic` in the configuration example while `ollama` has been a declared provider throughout, is named in the example command two hundred lines above, and is the one provider deliberately without a credential. The page disagreed with itself; the comment now names all three. The environment table says what an unrecognised value does. This is the cheap half of the refusal-behaviour gap: a reader who sets CYBERAI_LLM_PROVIDER to something the tool does not know had no page telling them what happens, and now has one sentence per boundary -- the variable falls back and the run continues, the flag is refused and the names are listed. The general case, a page describing every refusal and when it fires, is still not written. No numbers moved: the badges already read 2970 and 104/172 before this commit, and the scripts confirmed rather than wrote them.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
evkir
deleted the
fix/the-config-cannot-name-a-provider-that-does-not-exist
branch
September 21, 2026 06:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
How it was measured
Checklist
ruff format --check cyberai/ tests/andruff check cyberai/ tests/passpytest -W ignore::DeprecationWarning -m "not slow and not smoke"passes