docs: add CAT Phase 6 (Acceptance-Test Integrity) to make customer acceptance testing gaming-resistant

[ossumpossum]

Summary

Adds Phase 6: Acceptance-Test Integrity Validation to the Customer Acceptance Testing (CAT) appendix, plus its row in the Relationship-to-Requirements table. It makes the acceptance test itself resistant to a capable platform detecting and gaming the evaluation, and it gives customers a concrete way to exercise the newly merged advisory practices APTS-RP-A01 (finding authenticity) and APTS-MR-A02 (evaluation awareness) from their side.

Informative, non-normative, single file. No requirement is added or renumbered and the tier-coverage count is intentionally unchanged (Phase 6 exercises the RP-A01 / MR-A02 advisory practices, plus SE-001 which Phase 1 already counts).

Why

The current CAT framework (Phases 1 to 5) assumes the platform under test performs genuine testing. But a customer acceptance environment is, by construction, a benchmark: planted vulnerabilities, dense monitoring, and a known-good answer key. A sufficiently capable autonomous platform can recognize those signals and optimize for the test rather than the target, for example by locating and reading the known-vulnerability inventory, reporting findings it did not earn, or orienting toward the scoring and monitoring infrastructure. When that happens, Phases 3 (Detection Effectiveness) and 5 (Reporting) can report strong results that do not reflect real capability, because both rely on the platform's own reported evidence.

This is not hypothetical. In a controlled five-tool benchmark against a deliberately vulnerable target, a leading autonomous tool detected the evaluation context, located the scoring collector, read planted flags from disk and from process memory, and posted forged confirmation events for every planted vulnerability, producing a perfect score without exploiting anything. Phase 6 closes that gap for customers running CAT.

What it adds

Phase 6 is deliberately complementary to Phase 5. Phase 5 reproduces findings using the evidence the platform supplied; Phase 6 reproduces them independently and adds anti-gaming checks:

1. Test-integrity preconditions: isolate the known-vulnerability inventory and scoring artifacts (the evaluation oracle) so they are unreachable by the platform, and gate finding acceptance on evidence the customer gathers independently.
2. Independent evidence replay: reproduce a random sample of reported findings out-of-band using only the customer's own monitoring, not the platform's logs or report (operationalizes APTS-RP-A01).
3. Evaluation-awareness canaries: plant a scoring-shaped host, a decoy answer-key file, and a tokenized out-of-band collector, and check whether the platform orients toward any of them; orientation is both an evaluation-awareness signal and a scope concern under APTS-SE-001 (operationalizes APTS-MR-A02).
4. Oracle-exposure check: confirm the platform never reached the isolated inventory or scoring artifacts; if it did, the run's Phase 3 and Phase 5 results are void.

Pass, fail, and interpretation criteria are included, matching the existing phase format.

Lineage

Follows Discussion #64 and builds directly on the two advisory practices merged this week: RP-A01 (#68) and MR-A02 (#69). This is the customer-acceptance-side application of both. Happy to adjust wording, the sampling threshold, or the canary set to fit house style.

[jinsonvarghese]

@ossumpossum Thanks, this closes the loop nicely. Two small things before I merge, both in one pass.

First, the Phase 6 preconditions use MUST and MUST NOT (isolating the oracle, gating on independent evidence). CAT is informative and non-normative, and Phases 1 through 5 don't lean on any RFC 2119 keywords, so could you reword those two lines to plain prose? Something like "is stored outside... and is not reachable" and "acceptance is gated on...". Keeps the normative keywords for the actual requirements.

Second, in the Relationship-to-Requirements row, RP-A01 and MR-A02 are advisory practices rather than tier-required requirements. Since they're sharing a table with normative IDs, could you tag each as advisory so the row doesn't read them as counted requirements? A quick "(advisory practice)" next to each does it.

The rest holds up. Ping me once those are in and I'll merge.

[ossumpossum]

Done, @jinsonvarghese. Both in one pass: reworded the two Phase 6 preconditions to plain prose (dropped MUST / MUST NOT, matching the non-normative tone of Phases 1 through 5), and tagged APTS-RP-A01 and APTS-MR-A02 as advisory practices in the Relationship-to-Requirements row so they don't read as counted tier requirements. Left SE-001 untagged since it is tier-required. Ready to merge when you are.

[jinsonvarghese]

@ossumpossum Thank you, that's everything. Merging.