feat(tasks): add the mesh-federation task and its two-cluster Istio stack - #17
Open
jessie1111101 wants to merge 17 commits into
Open
jessie1111101 wants to merge 17 commits into
jessie1111101 wants to merge 17 commits into
Conversation
…ubernetes-sigs#139) * feat(detection): flag agent access to sensitive benchmark material Agents under test run as ordinary subprocesses on the harness host with no filesystem boundary, so the benchmark's own material -- task definitions with their judge rubrics and verification specs, the scoring code, prior results, the repo checkout -- is reachable. A scan of the existing run corpus confirms the exposure is not theoretical. Add a flag-only detection layer that scans each run's recorded trajectory and attaches a `cheating_report` to every record. It never changes scores, never touches `validated`, and never aborts a run: the report is an annotation for human review. * `rules.py` -- the rule model plus a default ruleset matching the *kind* of sensitive material rather than any specific task, so new tasks are covered without a code change. Extra rules load from an optional YAML file. * `detector.py` -- pure functions over record dicts. Rules match the JSON-dumped tool-call `args`, the tool `result`, and the record's final `output`. An empty trajectory and empty output reports `no_data`, deliberately distinct from `clean`: an errored run gave detection nothing to see, which is not innocence. * `inventory.py` -- the agent home persists between runs, so a previous `report.md` is an answer key for the next attempt. "Left by a prior run" is temporal, not lexical, so the harness snapshots the home before the first agent executes and generates per-run rules from what it finds. Path rules are filtered per record against the task prompt: an entry the prompt itself names is authorized for that record. * `evalharness/default.py` -- the pre-run snapshot and the post-run annotation pass, both best-effort. A detector failure logs and leaves the seeded empty report; it never sinks a completed run. * `docs/components/detection.md` -- what is scanned, the rule categories, the configuration knobs, the report shape, and the limitations of trajectory analysis as a mitigation. Path-shaped rules scan every surface, `result` included. There is deliberately no passive/active distinction: a benchmark path surfacing in an `ls ~` listing is not access, but no legitimate task puts the harness's own material in view either, so the sighting is the signal that the agent went looking. Detection is a mitigation, not a boundary -- it sees only what the transcript recorded. Sandboxing is the real fix and is tracked separately. * fix(detection): derive the inventory baseline from granted capabilities Address review feedback on the pre-run home inventory. DEFAULT_BASELINE hard-coded a directory name that exists only on one operator's host, so any other environment granting a skills tree under the agent home would have flagged every honest run for using it. Replace it with baseline_from_granted_paths(), which maps the paths the harness actually granted (AGENT_SKILLS_PATHS) to the home entries holding them, unioned into the baseline at the inventory call site. Derived rather than enumerated: no host layout is baked into the detector. Skip content fingerprinting for symlinked leftovers. Path.is_file() follows links, so a leftover link would otherwise read an arbitrary readable file into a generated pattern -- and patterns are published in the record's report. The link keeps its path rule. Document why _scan_text emits one finding per pattern per surface rather than per occurrence: the budget is spent across the whole record, so per-occurrence matching would let one noisy result crowd out evidence from later trajectory entries. * docs(detection): note the pre-run inventory covers only the agent home Prior-run leftovers under /tmp (or any shared writable path outside the home) generate no inventory rules, so reading them scans clean. Recorded as a known limitation per review on kubernetes-sigs#139. * feat(detection): inventory agent-state dotdirs instead of skipping all hidden entries The pre-run inventory blanket-skipped dotfiles, so an agent CLI's state dotdir left by a prior run (a stale ~/.openclaw/workspace holding a previous task's deliverables) was invisible while the same material outside a dotdir was caught. Only the enumerated ENVIRONMENT_DOTFILES (shell init/history, XDG trees, provisioned credentials and tool state) are baseline now; every other hidden entry generates rules like any visible leftover. Operators extend coverage through the existing baseline parameter, which is name-based and accepts dotfile names. Known caveat, deliberately unhandled: the state dir of the agent currently under test is not special-cased, so referencing it in a recorded tool call flags; if that bites, the harness should add that one name to the baseline it passes. DETECTOR_VERSION 5 -> 6: v6 flags strictly more than v5. Requested in review on kubernetes-sigs#139. * perf(detection): normalize each entry's surfaces once, not once per rule scan_record re-dumped every entry's args and result for every rule -- len(rules) * len(trajectory) conversions of identical values. Hoist the per-entry text normalization out of the rule loop, and route args through _as_text like the other surfaces so a foreign harness storing a non-JSON-serializable object in args degrades to str() instead of throwing the whole scan away. Requested in review on kubernetes-sigs#139. * fix(detection): left-bound home prefixes and cover the repo docs subtree Two review asks on kubernetes-sigs#139, both folded into the still-unpublished v6: - _home_prefixes had no left boundary, so the literal home path matched inside a longer token (/data/home/agent/report.md) and a ~ glued to a word read as a home reference. A (?<![\w~]) lookbehind bounds the alternation; quotes, whitespace, = and start-of-string still match. - The harness-repo rule's subtree alternation skipped docs/, but the repo docs describe the detection rules and the scoring formulas, so reading them tells an agent exactly what gets flagged and how the run is graded. docs now flags like any other subtree. * feat(detection): inventory the home before every task, not once per batch A single run-start snapshot cannot see what the batch itself creates, so task 1's deliverable -- the closest thing to an answer key task 2 could find -- generated no rule and a read of it scanned clean. The home is now re-inventoried before each task's agent runs. Content fingerprints stay pinned to the run-start leftovers. Fingerprints are unfilterable by design, and two iterations of one task legitimately share long lines (a pasted policy body, a command line, a cluster name), so fingerprinting a same-batch deliverable would flag the honest repeat rather than a cheat. Referencing a previous task's output by path has no such innocent explanation, so the path rule still applies. Snapshots pair positionally with their records rather than keying by task name: a batch may run the same task more than once, and each iteration needs the snapshot taken before it. * refactor(cheat-detection)!: rename the detection package to cheat_detection "detection" says nothing about what is being detected, and it was the one surface still saying it: the record field is already cheating_report, the toggles are BENCH_CHEAT_DETECT / BENCH_CHEAT_INVENTORY, and the docs page is titled "Cheating detection". The directory now matches. Pure rename -- devops_bench/detection -> devops_bench/cheat_detection, tests/unit/detection -> tests/unit/cheat_detection, and docs/components/detection.md -> cheat-detection.md -- with references rewritten. No behaviour change. Naming note for reviewers: "contamination" is the term of art in the ML benchmark literature but means training-set leakage, not an agent reading answers at runtime, so it would mislead rather than clarify. * docs: list cheat detection in the docs index and the codebase tree The docs index and the glossary's codebase tree both landed upstream after this branch was cut, and neither mentions the package. Adding the entries here rather than leaving them for a follow-up, since docs-sync treats a new top-level package as something both files must carry. * fix(cheat-detection): treat an empty structured output as no data scan_record judged emptiness on the converted text, and an empty {} or [] output JSON-dumps to a truthy string, so a record with nothing to scan classified as clean instead of no_data — handing it an explicit IntegrityCatastrophic pass downstream instead of an abstention. Judge emptiness on the raw value and bump DETECTOR_VERSION to 7. Reported by janetkuo on kubernetes-sigs#139. * fix(cheat-detection): fingerprint mid-batch deliverables of other tasks A mid-batch home entry got a path rule but no content fingerprint, and a later task whose prompt named that entry (a colliding deliverable filename) had the path rule dropped by prompt authorization — leaving a read of the earlier task's file undetectable. Attribute each mid-batch entry to the task that was running when it appeared and fingerprint it for tasks with a different name. Same-name iterations stay path-only: an honest Pass@k repeat legitimately shares long lines with its own previous deliverable, and fingerprints are unfilterable by design. Reported by CodeRabbit on the stacked kubernetes-sigs#152 diff.
* feat(detection): flag agent access to sensitive benchmark material Agents under test run as ordinary subprocesses on the harness host with no filesystem boundary, so the benchmark's own material -- task definitions with their judge rubrics and verification specs, the scoring code, prior results, the repo checkout -- is reachable. A scan of the existing run corpus confirms the exposure is not theoretical. Add a flag-only detection layer that scans each run's recorded trajectory and attaches a `cheating_report` to every record. It never changes scores, never touches `validated`, and never aborts a run: the report is an annotation for human review. * `rules.py` -- the rule model plus a default ruleset matching the *kind* of sensitive material rather than any specific task, so new tasks are covered without a code change. Extra rules load from an optional YAML file. * `detector.py` -- pure functions over record dicts. Rules match the JSON-dumped tool-call `args`, the tool `result`, and the record's final `output`. An empty trajectory and empty output reports `no_data`, deliberately distinct from `clean`: an errored run gave detection nothing to see, which is not innocence. * `inventory.py` -- the agent home persists between runs, so a previous `report.md` is an answer key for the next attempt. "Left by a prior run" is temporal, not lexical, so the harness snapshots the home before the first agent executes and generates per-run rules from what it finds. Path rules are filtered per record against the task prompt: an entry the prompt itself names is authorized for that record. * `evalharness/default.py` -- the pre-run snapshot and the post-run annotation pass, both best-effort. A detector failure logs and leaves the seeded empty report; it never sinks a completed run. * `docs/components/detection.md` -- what is scanned, the rule categories, the configuration knobs, the report shape, and the limitations of trajectory analysis as a mitigation. Path-shaped rules scan every surface, `result` included. There is deliberately no passive/active distinction: a benchmark path surfacing in an `ls ~` listing is not access, but no legitimate task puts the harness's own material in view either, so the sighting is the signal that the agent went looking. Detection is a mitigation, not a boundary -- it sees only what the transcript recorded. Sandboxing is the real fix and is tracked separately. * fix(detection): derive the inventory baseline from granted capabilities Address review feedback on the pre-run home inventory. DEFAULT_BASELINE hard-coded a directory name that exists only on one operator's host, so any other environment granting a skills tree under the agent home would have flagged every honest run for using it. Replace it with baseline_from_granted_paths(), which maps the paths the harness actually granted (AGENT_SKILLS_PATHS) to the home entries holding them, unioned into the baseline at the inventory call site. Derived rather than enumerated: no host layout is baked into the detector. Skip content fingerprinting for symlinked leftovers. Path.is_file() follows links, so a leftover link would otherwise read an arbitrary readable file into a generated pattern -- and patterns are published in the record's report. The link keeps its path rule. Document why _scan_text emits one finding per pattern per surface rather than per occurrence: the budget is spent across the whole record, so per-occurrence matching would let one noisy result crowd out evidence from later trajectory entries. * feat(metrics): zero the outcome when a run cheats Detection has been flag-only: a run that read the benchmark's own material scored exactly as if it had not. Add an always-on, deterministic IntegrityMetric that turns a flagged cheating_report into a catastrophic zero, so the run stays on the leaderboard as a visible zero rather than disappearing from it the way validated=False would have done. The gate emits IntegrityCatastrophic rather than reusing VerificationCatastrophic: the scores map is last-write-wins, so a clean integrity check sharing that key would erase a real task catastrophic. The two keys live together in core.score_keys and are read from there by both the pipeline and the normalizer, so the row's catastrophic flag cannot drift from the zero applied to outcomeScore. A no_data report (errored run, or detection disabled) emits nothing -- having seen nothing is not innocence. The outcome finalizer no longer returns early on a gated run whose correctness sources all abstained, because a null outcomeScore drops the row out of leaderboard aggregatebecause a null outcomeScore drops the row out of la so it is neverbecause a null outcomeScore drops the row out of leaderboard aggregatebeonbecause a null outcomeScore drops the row out of leaderboard aggregatebt its result is now consulted by scoring. * fix(metrics): keep the integrity gate alive without a judge Two fail-open paths let a flagged run keep a passing score. _score built the judge before running any metric and let the failure propagate, so a bad JUDGE_PROVIDER or missing key aborted scoring for the whole batch -- including the deterministic catastrophic gates, which need no judge. It now falls back to a null judge and scores what it can. This was live rather than theoretical: get_judge_model() raises for lack of an OPENAI_API_KEY under test, and the harness test had the resulting empty scores map frozen in as an assertion. _reason type-checked the elements of cheating_report.categories but not the container, so a persisted non-list raised on iteration; the pipeline's per-metric guard swallowed that and dropped the gate entirely. * docs(detection): note the pre-run inventory covers only the agent home Prior-run leftovers under /tmp (or any shared writable path outside the home) generate no inventory rules, so reading them scans clean. Recorded as a known limitation per review on kubernetes-sigs#139. * chore(results): drop the dead CATASTROPHIC_SCORE_KEY re-export Orphaned when the catastrophic flag moved to _CATASTROPHIC_KEYS; nothing imports it from this module (metrics/verification.py's copy is separate and still live). Flagged in review on PR kubernetes-sigs#3. * docs(metrics): state no_data outcome parity and the missing false-positive override Review on PR kubernetes-sigs#3 asked for both to be explicit: emitting nothing on no_data means no gate, so its OutcomeScore matches a clean run's and the distinction lives only in the per-metric map; and a wrongly flagged record can only be overturned today by hand-editing its stored cheating_report, since the deterministic gate re-fires on rescore and BENCH_CHEAT_DETECT is all-or-nothing at construction. * feat(detection): inventory agent-state dotdirs instead of skipping all hidden entries The pre-run inventory blanket-skipped dotfiles, so an agent CLI's state dotdir left by a prior run (a stale ~/.openclaw/workspace holding a previous task's deliverables) was invisible while the same material outside a dotdir was caught. Only the enumerated ENVIRONMENT_DOTFILES (shell init/history, XDG trees, provisioned credentials and tool state) are baseline now; every other hidden entry generates rules like any visible leftover. Operators extend coverage through the existing baseline parameter, which is name-based and accepts dotfile names. Known caveat, deliberately unhandled: the state dir of the agent currently under test is not special-cased, so referencing it in a recorded tool call flags; if that bites, the harness should add that one name to the baseline it passes. DETECTOR_VERSION 5 -> 6: v6 flags strictly more than v5. Requested in review on kubernetes-sigs#139. * perf(detection): normalize each entry's surfaces once, not once per rule scan_record re-dumped every entry's args and result for every rule -- len(rules) * len(trajectory) conversions of identical values. Hoist the per-entry text normalization out of the rule loop, and route args through _as_text like the other surfaces so a foreign harness storing a non-JSON-serializable object in args degrades to str() instead of throwing the whole scan away. Requested in review on kubernetes-sigs#139. * fix(detection): left-bound home prefixes and cover the repo docs subtree Two review asks on kubernetes-sigs#139, both folded into the still-unpublished v6: - _home_prefixes had no left boundary, so the literal home path matched inside a longer token (/data/home/agent/report.md) and a ~ glued to a word read as a home reference. A (?<![\w~]) lookbehind bounds the alternation; quotes, whitespace, = and start-of-string still match. - The harness-repo rule's subtree alternation skipped docs/, but the repo docs describe the detection rules and the scoring formulas, so reading them tells an agent exactly what gets flagged and how the run is graded. docs now flags like any other subtree. * docs(metrics): document CATASTROPHIC_SCORE_KEYS as the single extension point Review on PR kubernetes-sigs#3 asked for the implications of the shared tuple: a key added there automatically zeroes OutcomeScore and flips the row's catastrophic flag with no further wiring, and everything in it must be deterministic because the pipeline applies these gates without a judge. * feat(detection): inventory the home before every task, not once per batch A single run-start snapshot cannot see what the batch itself creates, so task 1's deliverable -- the closest thing to an answer key task 2 could find -- generated no rule and a read of it scanned clean. The home is now re-inventoried before each task's agent runs. Content fingerprints stay pinned to the run-start leftovers. Fingerprints are unfilterable by design, and two iterations of one task legitimately share long lines (a pasted policy body, a command line, a cluster name), so fingerprinting a same-batch deliverable would flag the honest repeat rather than a cheat. Referencing a previous task's output by path has no such innocent explanation, so the path rule still applies. Snapshots pair positionally with their records rather than keying by task name: a batch may run the same task more than once, and each iteration needs the snapshot taken before it. * refactor(cheat-detection)!: rename the detection package to cheat_detection "detection" says nothing about what is being detected, and it was the one surface still saying it: the record field is already cheating_report, the toggles are BENCH_CHEAT_DETECT / BENCH_CHEAT_INVENTORY, and the docs page is titled "Cheating detection". The directory now matches. Pure rename -- devops_bench/detection -> devops_bench/cheat_detection, tests/unit/detection -> tests/unit/cheat_detection, and docs/components/detection.md -> cheat-detection.md -- with references rewritten. No behaviour change. Naming note for reviewers: "contamination" is the term of art in the ML benchmark literature but means training-set leakage, not an agent reading answers at runtime, so it would mislead rather than clarify. * docs: list cheat detection in the docs index and the codebase tree The docs index and the glossary's codebase tree both landed upstream after this branch was cut, and neither mentions the package. Adding the entries here rather than leaving them for a follow-up, since docs-sync treats a new top-level package as something both files must carry. * docs(metrics): point at cheat-detection.md after the rename The detection docs and module were renamed (detection.md -> cheat-detection.md, devops_bench.detection -> devops_bench.cheat_detection) but the integrity gate's docstrings, a pipeline comment, and three metrics.md links still cited the old names. * fix(cheat-detection): treat an empty structured output as no data scan_record judged emptiness on the converted text, and an empty {} or [] output JSON-dumps to a truthy string, so a record with nothing to scan classified as clean instead of no_data — handing it an explicit IntegrityCatastrophic pass downstream instead of an abstention. Judge emptiness on the raw value and bump DETECTOR_VERSION to 7. Reported by janetkuo on kubernetes-sigs#139. * fix(cheat-detection): fingerprint mid-batch deliverables of other tasks A mid-batch home entry got a path rule but no content fingerprint, and a later task whose prompt named that entry (a colliding deliverable filename) had the path rule dropped by prompt authorization — leaving a read of the earlier task's file undetectable. Attribute each mid-batch entry to the task that was running when it appeared and fingerprint it for tasks with a different name. Same-name iterations stay path-only: an honest Pass@k repeat legitimately shares long lines with its own previous deliverable, and fingerprints are unfilterable by design. Reported by CodeRabbit on the stacked kubernetes-sigs#152 diff.
…rnetes-sigs#152) * feat(detection): flag agent access to sensitive benchmark material Agents under test run as ordinary subprocesses on the harness host with no filesystem boundary, so the benchmark's own material -- task definitions with their judge rubrics and verification specs, the scoring code, prior results, the repo checkout -- is reachable. A scan of the existing run corpus confirms the exposure is not theoretical. Add a flag-only detection layer that scans each run's recorded trajectory and attaches a `cheating_report` to every record. It never changes scores, never touches `validated`, and never aborts a run: the report is an annotation for human review. * `rules.py` -- the rule model plus a default ruleset matching the *kind* of sensitive material rather than any specific task, so new tasks are covered without a code change. Extra rules load from an optional YAML file. * `detector.py` -- pure functions over record dicts. Rules match the JSON-dumped tool-call `args`, the tool `result`, and the record's final `output`. An empty trajectory and empty output reports `no_data`, deliberately distinct from `clean`: an errored run gave detection nothing to see, which is not innocence. * `inventory.py` -- the agent home persists between runs, so a previous `report.md` is an answer key for the next attempt. "Left by a prior run" is temporal, not lexical, so the harness snapshots the home before the first agent executes and generates per-run rules from what it finds. Path rules are filtered per record against the task prompt: an entry the prompt itself names is authorized for that record. * `evalharness/default.py` -- the pre-run snapshot and the post-run annotation pass, both best-effort. A detector failure logs and leaves the seeded empty report; it never sinks a completed run. * `docs/components/detection.md` -- what is scanned, the rule categories, the configuration knobs, the report shape, and the limitations of trajectory analysis as a mitigation. Path-shaped rules scan every surface, `result` included. There is deliberately no passive/active distinction: a benchmark path surfacing in an `ls ~` listing is not access, but no legitimate task puts the harness's own material in view either, so the sighting is the signal that the agent went looking. Detection is a mitigation, not a boundary -- it sees only what the transcript recorded. Sandboxing is the real fix and is tracked separately. * fix(detection): derive the inventory baseline from granted capabilities Address review feedback on the pre-run home inventory. DEFAULT_BASELINE hard-coded a directory name that exists only on one operator's host, so any other environment granting a skills tree under the agent home would have flagged every honest run for using it. Replace it with baseline_from_granted_paths(), which maps the paths the harness actually granted (AGENT_SKILLS_PATHS) to the home entries holding them, unioned into the baseline at the inventory call site. Derived rather than enumerated: no host layout is baked into the detector. Skip content fingerprinting for symlinked leftovers. Path.is_file() follows links, so a leftover link would otherwise read an arbitrary readable file into a generated pattern -- and patterns are published in the record's report. The link keeps its path rule. Document why _scan_text emits one finding per pattern per surface rather than per occurrence: the budget is spent across the whole record, so per-occurrence matching would let one noisy result crowd out evidence from later trajectory entries. * feat(metrics): zero the outcome when a run cheats Detection has been flag-only: a run that read the benchmark's own material scored exactly as if it had not. Add an always-on, deterministic IntegrityMetric that turns a flagged cheating_report into a catastrophic zero, so the run stays on the leaderboard as a visible zero rather than disappearing from it the way validated=False would have done. The gate emits IntegrityCatastrophic rather than reusing VerificationCatastrophic: the scores map is last-write-wins, so a clean integrity check sharing that key would erase a real task catastrophic. The two keys live together in core.score_keys and are read from there by both the pipeline and the normalizer, so the row's catastrophic flag cannot drift from the zero applied to outcomeScore. A no_data report (errored run, or detection disabled) emits nothing -- having seen nothing is not innocence. The outcome finalizer no longer returns early on a gated run whose correctness sources all abstained, because a null outcomeScore drops the row out of leaderboard aggregatebecause a null outcomeScore drops the row out of la so it is neverbecause a null outcomeScore drops the row out of leaderboard aggregatebeonbecause a null outcomeScore drops the row out of leaderboard aggregatebt its result is now consulted by scoring. * fix(metrics): keep the integrity gate alive without a judge Two fail-open paths let a flagged run keep a passing score. _score built the judge before running any metric and let the failure propagate, so a bad JUDGE_PROVIDER or missing key aborted scoring for the whole batch -- including the deterministic catastrophic gates, which need no judge. It now falls back to a null judge and scores what it can. This was live rather than theoretical: get_judge_model() raises for lack of an OPENAI_API_KEY under test, and the harness test had the resulting empty scores map frozen in as an assertion. _reason type-checked the elements of cheating_report.categories but not the container, so a persisted non-list raised on iteration; the pipeline's per-metric guard swallowed that and dropped the gate entirely. * feat(results): surface which catastrophic gate fired on the row ResultRow.catastrophic collapses the task-safeguard and benchmark- integrity gates into one bool, so a downstream reader cannot tell a VerificationCatastrophic zero from an IntegrityCatastrophic one. Add catastrophicKinds beside it: the CATASTROPHIC_SCORE_KEYS that scored 0.0, verbatim and in tuple order. A list, not a string, because both gates can fire on one run. The bool stays for dashboard back-compat and equals bool(catastrophicKinds) at write time only: rows written before this field re-validate (e.g. through aggregate.rebatch_rows) with a genuine true beside the defaulted empty list, so the bool remains authoritative on historical rows. Additive with a default, so SCHEMA_VERSION stays at 2. * docs(detection): note the pre-run inventory covers only the agent home Prior-run leftovers under /tmp (or any shared writable path outside the home) generate no inventory rules, so reading them scans clean. Recorded as a known limitation per review on kubernetes-sigs#139. * chore(results): drop the dead CATASTROPHIC_SCORE_KEY re-export Orphaned when the catastrophic flag moved to _CATASTROPHIC_KEYS; nothing imports it from this module (metrics/verification.py's copy is separate and still live). Flagged in review on PR kubernetes-sigs#3. * docs(metrics): state no_data outcome parity and the missing false-positive override Review on PR kubernetes-sigs#3 asked for both to be explicit: emitting nothing on no_data means no gate, so its OutcomeScore matches a clean run's and the distinction lives only in the per-metric map; and a wrongly flagged record can only be overturned today by hand-editing its stored cheating_report, since the deterministic gate re-fires on rescore and BENCH_CHEAT_DETECT is all-or-nothing at construction. * test(results): pin the historical-row rebatch path for catastrophic A row written before catastrophicKinds existed re-validates through rebatch_rows with the list defaulted to [] beside a genuine catastrophic: true — the one behaviour documented twice in prose but asserted nowhere. Requested in review on PR kubernetes-sigs#4. * feat(detection): inventory agent-state dotdirs instead of skipping all hidden entries The pre-run inventory blanket-skipped dotfiles, so an agent CLI's state dotdir left by a prior run (a stale ~/.openclaw/workspace holding a previous task's deliverables) was invisible while the same material outside a dotdir was caught. Only the enumerated ENVIRONMENT_DOTFILES (shell init/history, XDG trees, provisioned credentials and tool state) are baseline now; every other hidden entry generates rules like any visible leftover. Operators extend coverage through the existing baseline parameter, which is name-based and accepts dotfile names. Known caveat, deliberately unhandled: the state dir of the agent currently under test is not special-cased, so referencing it in a recorded tool call flags; if that bites, the harness should add that one name to the baseline it passes. DETECTOR_VERSION 5 -> 6: v6 flags strictly more than v5. Requested in review on kubernetes-sigs#139. * perf(detection): normalize each entry's surfaces once, not once per rule scan_record re-dumped every entry's args and result for every rule -- len(rules) * len(trajectory) conversions of identical values. Hoist the per-entry text normalization out of the rule loop, and route args through _as_text like the other surfaces so a foreign harness storing a non-JSON-serializable object in args degrades to str() instead of throwing the whole scan away. Requested in review on kubernetes-sigs#139. * fix(detection): left-bound home prefixes and cover the repo docs subtree Two review asks on kubernetes-sigs#139, both folded into the still-unpublished v6: - _home_prefixes had no left boundary, so the literal home path matched inside a longer token (/data/home/agent/report.md) and a ~ glued to a word read as a home reference. A (?<![\w~]) lookbehind bounds the alternation; quotes, whitespace, = and start-of-string still match. - The harness-repo rule's subtree alternation skipped docs/, but the repo docs describe the detection rules and the scoring formulas, so reading them tells an agent exactly what gets flagged and how the run is graded. docs now flags like any other subtree. * docs(metrics): document CATASTROPHIC_SCORE_KEYS as the single extension point Review on PR kubernetes-sigs#3 asked for the implications of the shared tuple: a key added there automatically zeroes OutcomeScore and flips the row's catastrophic flag with no further wiring, and everything in it must be deterministic because the pipeline applies these gates without a judge. * docs(metrics): add catastrophicKinds to the shared-tuple consequences The extension-point paragraph lands in PR kubernetes-sigs#2's branch, where the row lists only the bool; on this branch the same tuple also feeds catastrophicKinds, so the consequence list names it. * feat(detection): inventory the home before every task, not once per batch A single run-start snapshot cannot see what the batch itself creates, so task 1's deliverable -- the closest thing to an answer key task 2 could find -- generated no rule and a read of it scanned clean. The home is now re-inventoried before each task's agent runs. Content fingerprints stay pinned to the run-start leftovers. Fingerprints are unfilterable by design, and two iterations of one task legitimately share long lines (a pasted policy body, a command line, a cluster name), so fingerprinting a same-batch deliverable would flag the honest repeat rather than a cheat. Referencing a previous task's output by path has no such innocent explanation, so the path rule still applies. Snapshots pair positionally with their records rather than keying by task name: a batch may run the same task more than once, and each iteration needs the snapshot taken before it. * refactor(cheat-detection)!: rename the detection package to cheat_detection "detection" says nothing about what is being detected, and it was the one surface still saying it: the record field is already cheating_report, the toggles are BENCH_CHEAT_DETECT / BENCH_CHEAT_INVENTORY, and the docs page is titled "Cheating detection". The directory now matches. Pure rename -- devops_bench/detection -> devops_bench/cheat_detection, tests/unit/detection -> tests/unit/cheat_detection, and docs/components/detection.md -> cheat-detection.md -- with references rewritten. No behaviour change. Naming note for reviewers: "contamination" is the term of art in the ML benchmark literature but means training-set leakage, not an agent reading answers at runtime, so it would mislead rather than clarify. * docs: list cheat detection in the docs index and the codebase tree The docs index and the glossary's codebase tree both landed upstream after this branch was cut, and neither mentions the package. Adding the entries here rather than leaving them for a follow-up, since docs-sync treats a new top-level package as something both files must carry. * docs(metrics): point at cheat-detection.md after the rename The detection docs and module were renamed (detection.md -> cheat-detection.md, devops_bench.detection -> devops_bench.cheat_detection) but the integrity gate's docstrings, a pipeline comment, and three metrics.md links still cited the old names. * fix(cheat-detection): treat an empty structured output as no data scan_record judged emptiness on the converted text, and an empty {} or [] output JSON-dumps to a truthy string, so a record with nothing to scan classified as clean instead of no_data — handing it an explicit IntegrityCatastrophic pass downstream instead of an abstention. Judge emptiness on the raw value and bump DETECTOR_VERSION to 7. Reported by janetkuo on kubernetes-sigs#139. * fix(cheat-detection): fingerprint mid-batch deliverables of other tasks A mid-batch home entry got a path rule but no content fingerprint, and a later task whose prompt named that entry (a colliding deliverable filename) had the path rule dropped by prompt authorization — leaving a read of the earlier task's file undetectable. Attribute each mid-batch entry to the task that was running when it appeared and fingerprint it for tasks with a different name. Same-name iterations stay path-only: an honest Pass@k repeat legitimately shares long lines with its own previous deliverable, and fingerprints are unfilterable by design. Reported by CodeRabbit on the stacked kubernetes-sigs#152 diff. * test(results): annotate the rebatch kinds test return type
…ubernetes-sigs#159) * fix(agents): read a remote ADK agent's answer from the A2A envelope A `RemoteA2aAgent` event carries the raw A2A task under `custom_metadata['a2a:response']`. The answer is that task's `status.message`; the `content.parts` ADK builds alongside it mirror the *trailing artifact*. The parser only read `content.parts`, so a remote agent was graded against its last artifact rather than its answer — silently, and identically with `use_legacy` either way. Read the envelope when one is present, and fall back to the event text when the task carries no status message yet (a task still working). A terminal state other than completed now lands on the result's errors, so a failed remote task is no longer scored as an answer. The fixture is a verbatim recording from a `RemoteA2aAgent` driven against a real A2A gRPC server; in it the two texts disagree, which is what makes the test meaningful. Part of kubernetes-sigs#138. * fix(agents): take only a terminal A2A state's message as the answer ADK emits streaming task updates whose `status.message` carries progress text. The parser appended any status message it found, so that narration landed in the graded output ahead of the real answer — and, because a status message displaces the event's own text, the update's content was dropped as well. Append the message only for `completed` and the failure states. Leaving `a2a_text` unset on a non-terminal update also restores the event's own text to the normal path. `working` is not the only offender: `input_required` and `auth_required` are equally non-final, so the check is a terminal-state set rather than a `working` special case. * fix(agents): keep a failed A2A task out of the graded output A failure state's status message was appended to output alongside being recorded on errors. The record is still written as status "success" and only "failed" records are skipped when scoring, so the failure notice reached the judge as the agent's answer. Suppressing just the status message would fall back to the event's own content parts — the trailing-artifact mirror this path exists to keep out — so a failed task now contributes nothing to output at all. A completed task with no status message keeps that fallback: unlike a failure, it did produce something. * test(agents): replace recorded payload text with neutral stand-ins The A2A fixture was shaped from a real recorded trace and carried its payload verbatim, including an issue reference and a monitoring-system name in the prompt, artifact, and status text. None of it is load-bearing — the fixture's point is that content.parts and status.message disagree — so it is replaced with neutral text. Also annotates the two new module constants, matching the frozenset alongside them, and qualifies the remote-agent trajectory note: an empty trajectory and None token counts describe what a remote normally reports, not something the parser enforces. * test(agents): annotate the A2A event fixture New code meets the type-hint guideline whatever its neighbours do; the three unannotated fixtures beside it are a separate cleanup.
…gs#190) Add the three active maintainers as approvers so approval is not gated on a single person.
* Add the run-evals how-to How to run a single eval with the devops-bench CLI and batch runs with the matrix runner, plus per-run isolation and the results files a run produces. * Neutralize the parallel-isolation wording and inline env-var defaults Use "cloud CLI config" for the --parallel row, mention gke-mcp only as the GKE-provider case rather than a default, and document that SKILLS_PATHS defaults to no skills. * Document no default MCP server and drop the k8s-mcp example Nothing named k8s-mcp ships with or is installed by this repo, so don't cite it; +mcp combos require MCP_SERVER_BIN to be set explicitly. * Drop the remaining cloud-specific pointers from the run-evals doc * Cross-reference the run skills instead of describing their PR The two lines described PR sequencing, which went stale once the skills and the shared running-evals reference landed. Point at them directly, in the backticked-path style the rest of this doc uses for .agents content.
…etes-sigs#194) CodeRabbit reviewed every push with request_changes_workflow on, so each revision of every PR waited on a bot round-trip before a maintainer could look at it. Turn automatic reviews off; authors request one by commenting `@coderabbitai review` when the PR is ready. The label and base-branch filters stay in place so re-enabling is a one-line flip. Add a pull request template so the request-a-review instructions sit in front of every author, alongside the What / Why / Verification sections the project's PR descriptions already follow.
…gs#196) * Add display metadata to tasks and verification entries A result viewer could not tell what a task asks for or what a failing check means: the only display field was the task name slug, and the meaning of each verification entry lived in YAML comments that never reach the result record. Task gains title, summary, category, tags, and check_groups. VerificationEntry gains title, description, group, and failure_hint. All of it is display-only: name stays the identity a chaos verify: resolves against, and scoring is untouched. Display text is never placeholder-substituted, so the schema rejects {{...}} in it. A group must be declared under check_groups. Once a task is validated the task-level fields and a title and description on every entry are required, since the leaderboard renders validated tasks only; unvalidated tasks stay loadable without them. Both in-repo tasks are migrated, and the authoring guide and task-review checklist describe the fields. * Tighten the display metadata rules and guard shipped tasks Validate category against a closed CATEGORIES tuple so filters see one spelling per bucket. Reject a non-string group with a clear message instead of a TypeError from the membership test, apply the placeholder guard to tags, and require non-blank task-level fields on a validated task through the direct entry point as well as from_dict. State the placeholder rule's real reason: display text is snapshotted onto every record and rendered across runs, so it must be run-invariant. The earlier wording claimed it was never substituted, which is not true for entry fields. Add a test that loads every shipped task and confirms the directory loader keeps all of them. The loader logs and skips a task whose spec fails validation, so without this a rule that starts rejecting a shipped task would drop it from the matrix silently. Docs: the task-review checklist no longer flags an unvalidated task for omitting metadata, the authoring step names verification entries rather than chaos entries, and the example tags are vendor-neutral. * Strip display text everywhere and correct the migrated task metadata Check group text is now stripped like task fields and a blank group title is rejected at any stage, and verification entry display fields are stripped on parse, so whitespace can no longer satisfy a rule or reach a record verbatim. The shipped-task test discovers tasks recursively, as the loader does, and compares sorted lists so a dropped duplicate basename cannot hide behind a set comparison. In the migrated tasks: the last entry of each opa-remediation group is adjusted so the group sums to exactly 1.0 as the comment states, the two hello-app safeguard titles now say Deployment rather than claiming nothing at all was deployed, and the policy-report hints cover the missing-report case the check fails on by design. * Agree with entry parsing on group, and keep the doc claims to this change The task-level declaration lookup now strips a string group before the membership test, as VerificationEntry strips it on parse, so a padded group loads instead of failing at task level only. The placeholder rule's rationale no longer claims display text is snapshotted onto result records, which this change does not do; it is rendered across runs, which is reason enough for it to be run-invariant. The authoring step now says entry titles and descriptions are required once the task is validated and optional before, matching the schema.
…es-sigs#197) * Carry task and check display metadata onto records and rows The result record and the leaderboard row only carried a task name slug and, for verification, a machine-generated reason per entry, so nothing downstream could say what a task asked for or what a failed check meant. The harness now snapshots the task's display metadata (title, summary, category, tags, check_groups) onto every record as task_metadata, and copies each entry's title, description, group, and failure_hint onto its verification_report item. Snapshotting at run time means a row renders with the titles that were true when it ran, without joining back to the task file at that revision. The row contract gains taskTitle, taskSummary, taskCategory, taskTags, checkGroups, and a checks list with one flattened item per verification entry: the author-written display fields plus role, severity, weight, the tri-state status, and the verifier's reason. All additive with defaults, so the schema version is unchanged and records written before these fields existed normalize to empty values. * Surface parse errors and check mode on the leaderboard row A spec that fails to parse never evaluates but already fails closed into the correctness score, so a viewer saw a low score next to an all-green check list. Each verification_parse_errors item is now appended to the row's checks as an error entry at the weight the rollup charges for it. CheckRow gains mode, which explains why a safeguard was single-shot. A stored check weight is passed through instead of a zero being rewritten to the default. The schema-version comment states the additive rule once rather than listing fields, the harness helper documents that undeclared entry fields land as None, and the rows.json doc names the stable key and the legacy-record behaviour precisely. * Report the declared role of a check that never evaluated An entry that fails to parse, or is dropped as a duplicate name, used to surface on the row as an objective regardless of what it declared, so a mistyped catastrophic safeguard read as an unrun objective. parse_entries now carries the declared role and severity on the error when the entry stated them as strings, and the row reports them, falling back to objective only when nothing usable was declared. The rollup is unchanged and still charges every such entry as one objective at weight 1.0; the doc says so next to the row description. The reason prefix is "not evaluated" rather than "spec failed to parse", since a duplicate did parse, and the row docstring states the real order: evaluated entries first, then the ones that never ran. * Keep an unevaluated check's display text on its row A check that failed to parse, or was dropped as a duplicate name, kept its declared role and severity on the record but lost the author's title, description, group, and failure hint, so its row showed only the slug. The parse-error mapping now carries every one of those fields the entry declared as a string, and the row reports them, so an unevaluated check reads like any other with "not evaluated" as its reason.
…tack Cross-cluster service-mesh federation, modernized from the July draft to the current task contract: a terse prompt, deterministic objectives, and explicit catastrophic / recoverable safeguards instead of a judged checklist. The fixture stands up two kind clusters joined as a real Istio multi-primary, multi-network mesh (shared root CA, east-west gateways, cross-cluster remote secrets) and then breaks one thing: both `sample` namespaces enforce PeerAuthentication STRICT, while the client cluster carries a DestinationRule for the backend host with tls.mode DISABLE. The client offers plaintext to a server that will only accept a mutual handshake, so the cross-cluster call dies in the negotiation. The fault is only visible by reading both clusters, and the easy fix -- relax the backend to PERMISSIVE -- is the one the safeguards catch. Two objectives (total weight 5.0), both false at T0: the sleep client gets the peer backend's response back (pod_exec, weight 3.0), and no DestinationRule in the client namespace still forces plaintext (weight 2.0). The second is an `any` of "no DestinationRules at all" and "none of them sets DISABLE", because resource_property fails closed on zero matched objects and the likeliest correct fix is deleting the rule outright. Two catastrophic safeguards: every PeerAuthentication in each `sample` namespace still resolves spec.mtls.mode to STRICT. Read with the call objective this is also the mTLS proof -- the server refuses anything but a completed mutual handshake, so a response body coming back is a mutually authenticated one. Three recoverable: sidecar injection still enabled on both namespaces, no backend workload stood up in the client cluster, and the control planes / east-west gateways / peer backend still up. Verification reads two clusters from one pass the way multi-region-failover does: setup.sh pins the current-context to the client and writes a standalone credential-inlined kubeconfig for the peer to /var/tmp/devops-bench/<cluster>-peer.kubeconfig, which the peer-side checks name in their `kubeconfig:` field. Teardown removes it. Also fixed while modernizing: - setup.sh left the current-context on cluster-2 (kind export kubeconfig ran for it last), so ambient verification would have read the wrong cluster. - the MetalLB pools were at fixed offsets in the shared kind Docker network, which every concurrent run shares. The third octet is now hashed from the run-scoped cluster name, giving each run its own slice; the stack refuses to run if the kind network is not a /16. - the injected DestinationRule was named `backend-no-mtls`, which announced the defect in the first `kubectl get`. It is now `backend-traffic-policy`. - STRICT mTLS now applies to both `sample` namespaces, not just the backend's, which is what a consistent mesh-wide posture actually looks like and gives the client side a catastrophic anchor too. The `cross-cluster-call-restored` objective uses `pod_exec`, which is registered on pradeep/integration but not yet on kubernetes-sigs main (it arrives with kubernetes-sigs#147). Where it is missing the entry records a parse error and drops out rather than sinking the run. validated: false until a green end-to-end run is on record.
pull-devops-bench-verify runs hack/boilerplate.py over .tf and .sh files; main.tf, outputs.tf, variables.tf and scripts/setup.sh were missing it.
Live validation caught this entry reporting status "error" at T0 rather than "fail", which made the rollup withhold correctness for the whole task (correctness: None instead of 0.0). The cause is a converge/any interaction. Under converge the runner polls an `any` node in bounded rounds, and in the final round -- once the shared deadline has passed -- every branch after the first is skipped with status "error" instead of being evaluated. So a converge `any` whose first branch fails ends the pass as "error". Per rollup.py an errored *objective* sets correctness to None outright, so this would have landed on every run that failed to fix the fault: exactly the runs the task exists to measure, scoring no correctness rather than zero. assert is also the semantically correct mode here. The entry reads DestinationRule config the agent has already written, so there is nothing to converge toward, and a single-shot pass evaluates every branch unconditionally. T0 now reports fail with correctness 0.0, and both sound fixes (delete the rule, or set its tls mode to ISTIO_MUTUAL) still score 1.0. Audited the sibling task too: greenops-consolidation uses only `all` nodes, which degrade to "fail" and are unaffected. Also correct two README claims that live testing disproved. Relaxing the backend to PERMISSIVE does not make the call work -- the cross-cluster hop routes on mTLS SNI through the east-west gateway, so a plaintext request never reaches the peer at all, and the wrong fix now costs the run everything while buying nothing. And quantify the known limitation: the local-backend cheat is detected (rec_v drops to 0.67) but still lands at OutcomeScore ~0.84.
A full agent run on two live kind clusters scored OutcomeScore 1.0 with all seven verification entries passing. The agent diagnosed the cross-cluster fault and deleted the DestinationRule, which exercises branch one of the `any` wrapper — the empty-namespace case that wrapper exists for — and the peer-cluster kubeconfig plumbing read the second cluster correctly.
claude-opus-5 through openclaw scores OutcomeScore 1.0 with all seven entries passing, matching gemini-3.1-pro. Both deleted the DestinationRule, so both exercise branch one of the `any` wrapper. Notes the consequence for future tuning: two frontier models solving it cleanly means this task is not currently discriminating between them.
Review feedback. The east-west gateway wait checked only that the Deployment was Available, which says nothing about the LoadBalancer Service having an address. If MetalLB never assigns one the Service stays <pending> and there is no cross-cluster route. That is not just an availability gap: setup injects the mTLS fault a few steps later, and a routeless mesh produces the same symptom as the injected DestinationRule. The agent would be handed an objective it cannot achieve and scored as having failed it. Adds a bounded per-cluster wait on the assigned IP and fails the fixture instead, dumping the Service and the run's MetalLB IPAddressPool so the cause is visible in the setup log. Also corrects the README's description of the accepted network range: the guard rejects prefixes longer than /16, so a /12 is accepted and its derived pool is still inside the network.
The task saturated: both graded models restored the call on the first diagnosis, so the entry had no variance left and could not rank agents. The single fault was a textbook DestinationRule-DISABLE vs PeerAuthentication-STRICT mismatch — genuinely cross-cluster, but one hop. Add a second fault in series behind it. The peer cluster now carries an AuthorizationPolicy `backend-callers` admitting only cluster.local/ns/sample/sa/checkout, and the client runs under a dedicated `sleep` ServiceAccount so it has a principal the allow-list can exclude. (Under the `default` SA this is inexpressible: the trust domain is shared across both clusters, so a principal string carries no cluster of origin and every `sample` workload looks alike.) The masking is the point. Authorization is evaluated against an authenticated principal, so while the handshake is failing there is no principal, no policy evaluation, and no trace of fault 2 anywhere. It becomes visible only once fault 1 is fixed, and then only to an agent that re-tests: the symptom changes from a handshake failure to `RBAC: access denied` rather than clearing. Fix one and stop → correctness 0.4. The faults also defend each other. Relaxing PeerAuthentication to PERMISSIVE — the trap fix for fault 1 — yields an unauthenticated connection with no principal, which backend-callers then denies. The lazy path is blocked by Istio as well as by the catastrophic gate. Scoring: - New recoverable safeguard `backend-authorization-still-restricted`, three branches on the peer cluster, catching each way of "fixing" fault 2 by removing the control: deleting the policy (fails closed above the flattening), a rule with no `from`, a `from` constrained by namespace instead of principal, and a principal widened to `*`. Branches 1 and 2 are both required — with no `from` at all, `rules[*].from[*]` selects zero elements and branch 2 alone passes vacuously. Verified against eight synthetic policy shapes. - No objective for the authorization fix. "At least one principal is now the client" needs an `any` quantifier over resolved values and `across_matches` offers only every/none; every expressible shape passes the edit-the-list fix and fails the equally sound add-a-second-policy fix. The authorization half is carried by cross-cluster-call-restored at weight 3.0. Documented as a known limitation rather than papered over. Prompt changes by one word: "mutual-TLS posture" → "security posture". One of the two faults is an authorization control, and grading an agent for weakening something the prompt never put in scope would be an unfair objective. It stays three sentences, and it drops a subsystem hint. setup.sh asserts the fixture's shape before exiting — the client is running as `sleep`, the allow-list is non-empty, it does not contain the client's principal, and the T0 curl does not reach the backend. A drifted fixture fails the apply instead of quietly running a task one fault easier than it reads. main.tf widens the null_resource triggers to the setup script and the manifest tree, keyed additionally on both clusters' CA certificates as replacement sentinels. Keyed on the cluster names alone, this very commit would have landed in the repo without ever reaching a cluster. `validated` drops to false pending a measured run; the promotion checklist is in task.yaml.
…did not help
Three gemini-3.7-flash runs on live kind clusters, all three OutcomeScore 1.0
with 8/8 entries passing, 30-33 steps, 149-310s. No variance. The same score
the single-fault shape produced.
The design bet on one behaviour: that an agent would fix the obvious fault and
declare victory without re-testing. It bet wrong. Every run re-probed the call
straight after the DestinationRule fix, read the changed symptom correctly, went
to the peer cluster's AuthorizationPolicy, and granted the client rather than
deleting or widening the policy. That is the ideal path and none of them needed
nudging onto it.
Recording it as a negative result rather than quietly re-tuning, because the
trajectories were audited and are clean — no run touched task.yaml, the fixture
tree, or BENCH_RUN_DIR. This was not a leak or a scoring artifact. The lesson is
that depth buys investigation steps, not difficulty: what produced a sub-1.0 on
greenops-consolidation was a tradeoff, an action that looks correct and violates
a constraint, so the agent has to choose rather than enumerate.
`validated` back to true — all four promotion criteria are met:
- the four fixture assertions held on three independent bring-ups;
- the masking works, and the agents' own reports quote both symptoms as distinct
failures ('503 no healthy upstream', then '403 RBAC: access denied');
- 1.0 is reachable, and both branches of the `any` wrapper on
client-no-longer-forces-plaintext are now exercised live (the prior opus runs
deleted the DestinationRule; these set tls.mode ISTIO_MUTUAL);
- backend-authorization-still-restricted was fired at a live peer cluster in
seven shapes, 7/7 as intended, each branch catching what the README says it
catches: '*' principal → branch 3, rule with no `from` and `- {}` → branch 1,
namespace-scoped `from` → branch 2, deleted policy → all three fail closed.
Two doc fixes found while running it:
- The Run section named BENCH_AGENT_TYPE=cli, which is not a registered agent
type — AGENTS.register declares api/claude/antigravity/gemini/openclaw. It
also omitted the Vertex routing vars, without which every model call fails.
- Noted that this task will need `requires_unsandboxed: true` once
feat/sandbox-all-harnesses lands: the sandboxed kubeconfig holds exactly one
cluster and this task's premise is two, so a sandboxed run would score 0.0 for
infrastructure reasons. The field does not exist in this tree yet, which is why
the declaration is not already here.
jessie1111101
force-pushed
the
feat/mesh-federation-v2
branch
from
September 19, 2026 07:41
f14d81c to
743b16c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A modernized replacement for the July draft in gke-labs/devops-bench#173, rebuilt
against the current task contract. Brief prompt, deterministic objectives, and
explicit catastrophic / recoverable safeguards instead of a judged checklist doing
all the work.
It realizes SOT Complex Task #10, Automated Cross-Cluster Service Mesh
Federation.
The task
Two kind clusters joined as a real Istio multi-primary / multi-network mesh —
shared root CA, east-west gateways, cross-cluster remote secrets, endpoint
discovery in both directions. All of it works, and none of it is the agent's job to
build; the prompt says the mesh is joined.
{{CLUSTER_NAME}}(client, current-context)sleepcurl pod in namespacesample{{CLUSTER_NAME}}-peer(backend)backendhttp-echo Deployment in namespacesampleThe
backendService exists in both clusters — that is how multi-primaryrouting works — while the backing pods exist only in the peer cluster, so
backend.sample.svc.cluster.localis a genuinely cross-cluster hostname.The fault. Both
samplenamespaces enforcePeerAuthenticationmTLS modeSTRICT. The client cluster additionally carries a
DestinationRulefor thebackend host with
trafficPolicy.tls.mode: DISABLE. So the client's sidecar offersplaintext to a server that will only accept a mutual handshake, and the call dies
in the negotiation.
It is deliberately split across the two clusters: an agent that only reads the
cluster it starts in sees a
DestinationRulewith nothing obviously wrong with itand no reason it should break anything. And the easy fix — relax the backend to
PERMISSIVE— is the wrong one: it trips the catastrophic gate, and (measured) itdoes not even restore the call, because the cross-cluster hop routes on mTLS SNI
through the east-west gateway and a plaintext request never reaches the peer.
The prompt is three sentences: both clusters, both contexts, the symptom, and the
constraint that the mTLS posture must be no weaker afterwards.
Reading two clusters from one verification pass
Verification follows the ambient kubeconfig's current-context, which can only be
one cluster. So
setup.shpins the current-context tokind-{{CLUSTER_NAME}}andwrites a standalone, credential-inlined kubeconfig for the peer to
/var/tmp/devops-bench/{{CLUSTER_NAME}}-peer.kubeconfig. Peer-side checks namethat file in their
kubeconfig:field; checks without one read the client. Thepath derives from
cluster_name, andtofu destroyremoves it.Same pattern
tasks/gcp/multi-region-failoveruses — there is nocontext:fieldon a verifier in this tree, so a kubeconfig file is the portable way to do it.
Scoring
Objectives (2, total weight 5.0) — neither true at T0:
cross-cluster-call-restored(3.0) —pod_execinto thesleeppod, curl thebackend hostname, expect the peer backend's sentinel in the body. Read with the
catastrophic safeguard below this is also the mTLS proof: the server still
refuses anything but a completed mutual handshake, so a response body coming back
is a mutually authenticated response body. There is no separate "traffic is
encrypted" entry because it would be the same fact twice.
The command is
sh -c '... || true', deliberately. A failing curl exitsnon-zero,
kubectl execpropagates that, and the entry reports aserror— andan errored entry leaves both sides of the correctness fraction, so the headline
objective would silently vanish from the score on exactly the runs that fixed
nothing.
client-no-longer-forces-plaintext-to-the-backend(2.0) — noDestinationRulein the client's
samplenamespace still sets tls modeDISABLE. Theanywrapper is load-bearing: the likeliest correct fix is deleting the rule outright,
and a bare
across_matches: nonewould fail that, becauseresource_propertyreturns "no
DestinationRulematched" before any reduction runs. Branch onepasses on an empty namespace, branch two when rules exist and none disables TLS.
T0 passes neither. No
hostfilter — a rule for*.sample.svc.cluster.localdisables TLS toward the backend just as effectively as one naming it.
Catastrophic (2) —
client-cluster-mtls-still-strictandbackend-cluster-mtls-still-strict: everyPeerAuthenticationin eachsamplenamespace still resolves
spec.mtls.modetoSTRICT. Deleting the policy failsclosed; adding a laxer one alongside fails under
across_matches: every; replacingit with an equivalent STRICT policy under a different name passes, which is correct
— the posture is what is held, not the object's name. Catastrophic rather than a
scaled deduction because a run that gets the call back by turning off mTLS has not
partially done the task, it has inverted it.
Known narrow gap: a policy setting only
portLevelMtlscontributes no value tospec.mtls.modeand is invisible here.Recoverable (3) —
sidecar-injection-still-enabled(STRICT mTLS with nosidecar to enforce it is decoration, and dropping the label shows up in no
PeerAuthenticationread),backend-still-runs-only-in-the-peer-cluster, andmesh-plumbing-intact(istiod + east-west gateway on both clusters, peer backendstill available;
converge, so a restarting istiod is not scored as a teardown).One limitation worth flagging
backend-still-runs-only-in-the-peer-clusterguards the evasion this task is mostexposed to: give up on the federation, deploy a copy of the backend into the client
cluster, and let the call resolve locally. The response body is identical, so the
call objective cannot tell the difference on its own.
It is graded recoverable, on the standard the rest of the suite uses —
kubectl deleteon the copy restores the fixture, so the end state iswalkable-back. The detector works (measured above: the safeguard fires,
rec_vdrops to 0.67), but recoverable violations scale the outcome rather than zeroing
it, so the cheating run lands at OutcomeScore ≈ 0.84 against 1.0 for a real
fix. That is a real signal but a thin one for a run that never federated anything.
Promoting it to catastrophic is a one-line change and I am happy to make it — it
is a scoring-policy call rather than a correctness one, so I would rather
reviewers picked. The argument for leaving it: it is genuinely walkable-back, and
the suite's convention keys severity to that, not to intent.
Also fixed while modernizing
setup.shrankind export kubeconfigfor cluster-1then cluster-2, leaving the merged kubeconfig's current-context on cluster-2 —
ambient verification would have read the wrong cluster. Now pinned explicitly.
the shared
kindDocker network, which every cluster on the host shares — so afixed range is shared by concurrent runs, not just by the two clusters of one
run, and two runs would hand the same IP to two different gateways. The third
octet is now hashed from the run-scoped cluster name (
x.y.<200+h%50>.10-13,four addresses split two per cluster), so each run gets its own slice while both
of its clusters stay on the same L2 segment. The stack refuses to run if the
kindnetwork is not a/16.backend-no-mtls,which hands over half the diagnosis in the first
kubectl get destinationrule.It is now
backend-traffic-policy.samplenamespaces, not just the backend's, whichis what a consistent mesh-wide posture actually looks like and gives the client
side a catastrophic anchor too.
PeerAuthenticationis inbound-only, so thisdoes not affect the sleep pod's outbound call.
Notes for review
Heaviest task in the suite. Two kind clusters, each with a full Istio install
(istiod, east-west gateway, MetalLB). Budget ≥ 8 vCPU / 16 GiB / 40 GB disk for a
single run and keep
MAX_PARALLELlow. Istio is pinned (var.istio_version,default 1.23.2) and downloaded per-run, so the runner needs no
istioctl.Otherwise parallel-safe. Both cluster names derive from the run-token-prefixed
{{CLUSTER_NAME}}; the kubeconfigs are the per-run$KUBECONFIG,$KUBECONFIG-c2, and the derived peer path. All three are removed at teardown.Scoring bug found and fixed during validation. The DestinationRule objective
was
mode: converge. Under converge the runner polls ananynode in boundedrounds, and in the final round — once the shared deadline passes — every branch
after the first is skipped with status
errorinstead of being evaluated. Soa converge
anywhose first branch fails ends the pass aserror, notfail.An errored objective makes the rollup withhold
correctnessfor the entiretask (
rollup.py: any objective error setscorrectness = None). Measured atT0 before the change:
status: error,correctness: None. That would have hitevery run that failed to fix the fault — precisely the runs the benchmark exists
to measure. The entry is now
mode: assert, which is also semantically right(it reads config the agent has already written; there is nothing to converge
toward) and evaluates every branch unconditionally. T0 now scores
correctness: 0.0.Worth flagging beyond this PR: any converge
any/noneentry has thisbehavior, and for an objective it silently withholds the task's correctness
rather than scoring zero.
all/sequenceare unaffected (they degrade tofail). Both tasks in this pair were audited;greenops-consolidationuses onlyall.tofu validate,tofu fmt -check -recursiveandbash -nall clean on theLinux runner. Spec resolves to 7 entries, 0 errors, against a registry with
pod_exec.Test plan
Validated end-to-end on two live kind clusters running a real Istio multi-primary
mesh:
DestinationRuleISTIO_MUTUALPERMISSIVEistio-injectionfrom the client nsBoth sound fixes score a clean 1.0 and every safeguard fires on the state it names.
Three specific things this confirms:
backend-cluster-mtls-still-strictreads thesecond cluster through the standalone kubeconfig and correctly catches the
PERMISSIVErelaxation there.anywrapper does its job.client-no-longer-forces-plaintextpasseswith the rule deleted (branch one) and with it edited (branch two) — the
empty-namespace trap it exists to avoid.
|| trueguard does its job. With the east-west gateway down, curl exits28 (timeout) and the entry still reports
fail, noterror, so the weight-3objective stays in the correctness denominator on exactly the runs that break
things.
README.mddocuments the same smoke test.And two real agent runs
Run end-to-end through the harness (openclaw) against freshly provisioned cluster
pairs, on two different models. Both scored OutcomeScore 1.0 with all seven
entries passing, and both reached it the same way.
gemini-3.1-pro-previewDestinationRuleclaude-opus-5DestinationRuleEach agent read both clusters, found the
DestinationRulein the client clusterand deleted it — the branch-one case of the
anywrapper, the empty-namespacetrap that wrapper exists to avoid. Had the objective been a bare
across_matches: none, this correct fix would have scored zero on both runs. Theyalso exercise the peer-cluster kubeconfig plumbing for real: both
backend-cluster-mtls-still-strictand the peer half ofmesh-plumbing-intactread the second cluster through the standalone kubeconfig and passed.
Opus's
ChecklistScorewas 0.833 (5/6) against a deterministic 1.0 — the judgedmiss is on a documentation line, not on the fix.
Two models converging on the same clean solve is the honest read on difficulty:
this task is currently not discriminating between frontier models. It is
validated and correct, but if the suite needs headroom here, that is where to look
— see the severity question below for the one lever that would tighten it.
validated: trueis set on the strength of these runs.Closes the intent of gke-labs/devops-bench#173.