Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,13 @@ both halves move when modules are added. It has its own tool:

python3 scripts/mypy_badge.py

Adding a module to `[tool.mypy] files` costs its own errors and moves the two
boundary counters the scope page states, sometimes in directions the module's
shape does not predict. Read the price before paying it, with the declared
flags and without editing the manifest:

python3 scripts/scope_price.py cyberai/core/cache.py

## Lint
ruff check cyberai/ --fix

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)
![License](https://img.shields.io/badge/license-Apache_2.0-blue)
![Version](https://img.shields.io/badge/version-v1.7.0-brightgreen)
![Tests](https://img.shields.io/badge/tests-2983%20collected-brightgreen)
![Mypy](https://img.shields.io/badge/mypy-strict%3A%20104%2F172%20modules-blue)
![Tests](https://img.shields.io/badge/tests-2988%20collected-brightgreen)
![Mypy](https://img.shields.io/badge/mypy-strict%3A%20107%2F172%20modules-blue)
![LLM](https://img.shields.io/badge/LLM-OpenAI%20%7C%20Anthropic%20%7C%20Ollama-blueviolet)
![Air-Gapped](https://img.shields.io/badge/air--gapped-ready-success)

Expand Down
4 changes: 2 additions & 2 deletions blog/launch-post-draft.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ not exist yet, it says so.
CyberAI is a multi-agent offensive-security platform: eight agents (recon,
intel, exploit, report, planner, mcp-scan, redteam, web3) run a typed, audited
pipeline over a shared knowledge base.
2983 tests collected under the gated selection run before every commit, with the
2988 tests collected under the gated selection run before every commit, with the
slow and smoke tests deselected there and run separately, `mypy --strict`
clean over 104 of 172 modules, Apache-2.0.
clean over 107 of 172 modules, Apache-2.0.

It is not a wrapper that pipes nmap output into a chat model. Three things make
it a different category of tool.
Expand Down
4 changes: 2 additions & 2 deletions cyberai/agents/web3/immunefi_severity.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

from __future__ import annotations

from typing import TYPE_CHECKING, List
from typing import TYPE_CHECKING, Any, Dict, List

if TYPE_CHECKING:
from .slither_tool import SlitherFinding
Expand Down Expand Up @@ -147,7 +147,7 @@ def classify(finding: "SlitherFinding") -> str:
return _IMPACT_FALLBACK.get((finding.impact, finding.confidence), "Insight")


def classify_all(findings: List["SlitherFinding"]) -> List[dict]:
def classify_all(findings: List["SlitherFinding"]) -> List[Dict[str, Any]]:
"""Classify findings, attaching an `immunefi_severity` field, sorted high→low."""
rows = []
for f in findings:
Expand Down
2 changes: 1 addition & 1 deletion cyberai/bench/apps/_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ def _query(path: str) -> Dict[str, str]:
class BenchHandler(BaseHTTPRequestHandler):
"""Routes are supplied by each app as {(method, path): handler}."""

routes: Dict[tuple, Callable[["BenchHandler"], Any]] = {}
routes: Dict[tuple[str, str], Callable[["BenchHandler"], Any]] = {}

def log_message(self, fmt: str, *args: Any) -> None: # keep output quiet
pass
Expand Down
2 changes: 1 addition & 1 deletion cyberai/core/exploit_memory.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ def build_record(
}


def _cosine(a: Counter, b: Counter) -> float:
def _cosine(a: Counter[str], b: Counter[str]) -> float:
common = set(a) & set(b)
if not common:
return 0.0
Expand Down
28 changes: 24 additions & 4 deletions docs/architecture/typing-scope.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Typing scope

`mypy --strict` reads 104 of 172 modules in the package. The other 68 hold 275
`mypy --strict` reads 107 of 172 modules in the package. The other 65 hold 272
errors and are not checked.

The scope is a list of named modules, so a module that passes strictly stays
Expand All @@ -13,7 +13,7 @@ module that could be declared and is not becomes a failing CI step rather than
a quiet omission.

Not checked is stronger than it sounds, and the boundary is the reason. Of
the 104 modules in the scope, 22 import a module outside it at module level,
the 107 modules in the scope, 22 import a module outside it at module level,
and between them they reach 30 such modules. mypy follows those imports to
resolve names and does not report what it finds there: measured on 2026-09-17
by appending an unannotated function to `cyberai/core/config.py`, which was
Expand Down Expand Up @@ -68,8 +68,28 @@ before the fact, which is what the numbers here are for.
A third module went in the same day, `agents/exploit/safety_validator.py`,
whose public signature declared two implicit Optionals. Only one counter
moved: it imports nothing outside the scope, so it never was a crosser, and
reached fell from 31 to 30. A leaf pays one counter and an entry point pays
both, which is the shape of the rule rather than a number to memorise.
reached fell from 31 to 30.

Those three moves were once summarised here as a rule -- a leaf pays one
counter, an entry point pays both. `scripts/scope_price.py` projected all 68
undeclared modules on 2026-09-23 and the summary does not survive it. The
moves fall into eleven classes, not two, and `core/cache.py` is the plainest
refutation: it imports nothing outside the scope, so by that rule it was a
leaf paying one counter, and it moves both. `agents/intel/epss_client.py` is
the only module in the scope that reaches it and reaches nothing else, so
declaring cache.py retires a crosser and a reached module at once.

What the projection shows instead is two independent sums. A candidate
enters the crosser set if it imports past the edge itself, and it removes
every module whose only crossing was to reach it; `integrations/phantom_grid.py`
removes two, which no rule about leaves permits. It adds its own targets to
the reached set and removes itself if anything already reached it;
`core/orchestrator.py` adds two while joining no crossing of its own. Because
the two sums are independent, the observed pairs run from -2 and -1 up to +1
and +5, and the largest class is neither: twenty modules move nothing at all,
costing only the errors they carry. The price of a module is therefore read
per module and before the fact, which is what the tool is for and what this
paragraph no longer claims to predict.

## How the set was drawn

Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -136,13 +136,15 @@ files = [
"cyberai/agents/web3/etherscan.py",
"cyberai/agents/web3/foundry_poc.py",
"cyberai/agents/web3/halmos_tool.py",
"cyberai/agents/web3/immunefi_severity.py",
"cyberai/agents/web3/judge.py",
"cyberai/agents/web3/poc_synth.py",
"cyberai/agents/web3/property_synth.py",
"cyberai/agents/web3/slither_tool.py",
"cyberai/bench/__init__.py",
"cyberai/bench/agent_engine.py",
"cyberai/bench/apps/__init__.py",
"cyberai/bench/apps/_server.py",
"cyberai/bench/ctf.py",
"cyberai/bench/ctf_loader.py",
"cyberai/bench/cve_bench.py",
Expand All @@ -166,6 +168,7 @@ files = [
"cyberai/core/config.py",
"cyberai/core/cost_tracker.py",
"cyberai/core/egress_guard.py",
"cyberai/core/exploit_memory.py",
"cyberai/core/llm_usage.py",
"cyberai/core/model_router.py",
"cyberai/core/phase_map.py",
Expand Down
222 changes: 222 additions & 0 deletions scripts/scope_price.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
#!/usr/bin/env python3
"""What a module costs before it is added to `[tool.mypy] files`.

Three times on 2026-09-21 the boundary counters moved as modules entered the
scope, and three times the expected numbers in the guard were corrected after
the fact by rerunning it and reading the failure. That works and teaches
nothing: the price was paid first and read afterwards. The prose on the scope
page states the rule -- a leaf pays one counter, an entry point pays both --
and the rule was written from those three moves rather than measured against
a fourth.

This reads the price first. For a candidate module it reports the errors the
module itself brings under the declared flags, and the two boundary counters
as they would stand with the module inside the scope. Nothing in the tree is
edited to get there: the scoped run is driven by a config file written to a
temporary directory, so `pyproject.toml` is never touched and an interrupted
run leaves no half-applied scope behind.

The counters are counted the way the guard counts them, from the import graph
rather than from the checker, because that is the measurement the guard will
compare against. A number produced by a different method would agree with the
guard by luck and diverge without warning.

A run that produces no verdict line is not a clean module. `mypy` absent from
the environment writes nothing to standard output, and every count taken from
that silence reads as zero errors on a module nobody checked.
"""

from __future__ import annotations

import argparse
import ast
import pathlib
import re
import subprocess
import sys
import tempfile
import tomllib

_ROOT = pathlib.Path(__file__).resolve().parents[1]
_PYPROJECT = _ROOT / "pyproject.toml"
_PACKAGE = _ROOT / "cyberai"

_ERROR = re.compile(r"^(?P<module>[^:]+\.py):\d+: error")
_VERDICT = re.compile(r"^(Found \d+ error|Success: no issues found)", re.MULTILINE)


def settings() -> dict[str, object]:
loaded: dict[str, object] = tomllib.loads(_PYPROJECT.read_text(encoding="utf-8"))["tool"][
"mypy"
]
return loaded


def declared_scope(config: dict[str, object]) -> set[pathlib.Path]:
declared = config["files"]
if not isinstance(declared, list):
raise TypeError("[tool.mypy] files is not a list; the scope cannot be resolved")
resolved: set[pathlib.Path] = set()
for entry in declared:
path = _ROOT / str(entry)
if path.is_dir():
resolved.update(path.rglob("*.py"))
else:
resolved.add(path)
return resolved


def _resolve(dotted: str) -> pathlib.Path | None:
base = _ROOT / pathlib.Path(dotted.replace(".", "/"))
for candidate in (base.with_suffix(".py"), base / "__init__.py"):
if candidate.exists():
return candidate
return None


def _imported_names(module: pathlib.Path) -> list[str]:
tree = ast.parse(module.read_text(encoding="utf-8"))
names: list[str] = []
for node in ast.walk(tree):
if isinstance(node, ast.Import):
names.extend(alias.name for alias in node.names)
elif isinstance(node, ast.ImportFrom):
if node.level:
package = module.parent
for _ in range(node.level - 1):
package = package.parent
dotted = str(package.relative_to(_ROOT)).replace("/", ".")
if node.module:
dotted = f"{dotted}.{node.module}"
else:
dotted = node.module or ""
names.append(dotted)
names.extend(f"{dotted}.{alias.name}" for alias in node.names)
return names


def crossings(declared: set[pathlib.Path]) -> tuple[set[pathlib.Path], set[pathlib.Path]]:
"""Modules importing past the edge, and the modules they reach."""
crossers: set[pathlib.Path] = set()
reached: set[pathlib.Path] = set()
for module in sorted(declared):
for dotted in _imported_names(module):
if not dotted.startswith("cyberai"):
continue
target = _resolve(dotted)
if target is not None and target not in declared:
crossers.add(module)
reached.add(target)
return crossers, reached


def _config_text(config: dict[str, object], files: list[str]) -> str:
lines = ["[tool.mypy]"]
for key in ("python_version", "strict", "ignore_missing_imports"):
if key not in config:
continue
value = config[key]
if isinstance(value, bool):
lines.append(f"{key} = {str(value).lower()}")
else:
lines.append(f'{key} = "{value}"')
lines.append("files = [")
lines.extend(f' "{entry}",' for entry in files)
lines.append("]")
return "\n".join(lines) + "\n"


def run_scoped(config: dict[str, object], files: list[str]) -> subprocess.CompletedProcess[str]:
"""Run the checker over an arbitrary file set without editing the tree."""
with tempfile.TemporaryDirectory() as tmp:
written = pathlib.Path(tmp) / "mypy.toml"
written.write_text(_config_text(config, files), encoding="utf-8")
return subprocess.run(
[
sys.executable,
"-m",
"mypy",
"--config-file",
str(written),
"--cache-dir",
str(pathlib.Path(tmp) / "cache"),
],
cwd=_ROOT,
capture_output=True,
text=True,
check=False,
)


def errors_in(output: str, module: pathlib.Path) -> int:
relative = module.relative_to(_ROOT).as_posix()
return sum(
1
for line in output.splitlines()
if (match := _ERROR.match(line)) and match.group("module") == relative
)


def price(config: dict[str, object], module: pathlib.Path) -> dict[str, int] | None:
"""Errors the module brings and where the two counters land with it inside."""
declared = [str(entry) for entry in config["files"]] # type: ignore[union-attr]
candidate = module.relative_to(_ROOT).as_posix()
completed = run_scoped(config, [*declared, candidate])
if not _VERDICT.search(completed.stdout):
return None
scope = declared_scope(config)
before_crossers, before_reached = crossings(scope)
after_crossers, after_reached = crossings(scope | {module})
return {
"errors": errors_in(completed.stdout, module),
"crossers": len(after_crossers),
"reached": len(after_reached),
"delta_crossers": len(after_crossers) - len(before_crossers),
"delta_reached": len(after_reached) - len(before_reached),
}


def outside(config: dict[str, object]) -> list[pathlib.Path]:
return sorted(set(_PACKAGE.rglob("*.py")) - declared_scope(config))


def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument(
"module", nargs="*", help="paths under cyberai/; default: the whole outside"
)
args = parser.parse_args()
config = settings()
if args.module:
candidates = [(_ROOT / entry).resolve() for entry in args.module]
else:
candidates = outside(config)
scope = declared_scope(config)
crossers, reached = crossings(scope)
print(f"scope: {len(scope)} modules, {len(crossers)} crossers reaching {len(reached)}")
print(f"{'module':<52} {'errors':>6} {'crossers':>9} {'reached':>8}")
for module in candidates:
if not module.exists():
print(f"{module.relative_to(_ROOT).as_posix():<52} {'absent':>6}")
return 2
if module in scope:
print(f"{module.relative_to(_ROOT).as_posix():<52} {'in':>6}")
continue
measured = price(config, module)
if measured is None:
print(
"the checker returned no verdict: this environment was not measured",
file=sys.stderr,
)
return 2
name = module.relative_to(_ROOT).as_posix()
print(
f"{name:<52} {measured['errors']:>6} "
f"{measured['crossers']:>4} {measured['delta_crossers']:>+4} "
f"{measured['reached']:>4} {measured['delta_reached']:>+4}"
)
return 0


if __name__ == "__main__":
raise SystemExit(main())
5 changes: 3 additions & 2 deletions scripts/typing_scope_drift.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
#!/usr/bin/env python3
"""Report modules that pass `mypy --strict` while sitting outside the scope.

`[tool.mypy] files` names five directories and seventy-two individual modules.
`[tool.mypy] files` names five directories and eighty-five individual modules
(measured 2026-09-23; the count moves as modules are declared).
A directory keeps its guarantee open: a module added inside one is checked from
the moment it lands. A named module does not. A sibling dropped next to one of
the seventy-two passes strictly, is never checked, and nothing says so. The
the named ones passes strictly, is never checked, and nothing says so. The
declared scope then understates what the repository holds, and it understates
it more with every module added.

Expand Down
Loading
Loading