Does the ethics pipeline an AI agent runs on actually change what it decides — and do the specific values in it matter, or only the scaffolding that carries them?
That question is the current work. RATCHET is where the experiment that answers it is designed, pre-registered and run.
📄 Read the plain-English overview →
CIRIS agents put every decision through a pipeline: several reasoning stages plus four conscience faculties that can veto an action. That costs roughly twenty extra model calls per thought. Is it doing anything, and does what it says matter?
TORQUE compares four agents that differ only in the values document they carry — CIRIS's own, a different real value system, a neutral one with the values drained, and a blank one — on 540 ethics questions they all see in the same order.
It is an equivalence experiment. It asks whether the difference is smaller than a declared bound, not whether a difference exists. That is not modesty; it is what the instrument turned out to support, and the measurements that decided it are below.
TORQUE set out to ask three questions. Two of them the instrument cannot answer, and finding that out cost about six dollars in probe runs rather than the $167 the full design would have.
"Does the pipeline change behaviour?" — withdrawn. Comparing pipeline arms against a plain model is confounded twice: the benchmark's own harness runs every item as a fresh conversation while ours runs ten in a row, and the pipeline arms carry a position effect the plain arms do not. Not recoverable by collecting more data.
"Does it hold when you stop paying?" — dropped, and this one is the name. Torque is a force you keep applying; the plan was to withdraw the pipeline mid-conversation and watch whether behaviour drifted back. Then prompt capture showed the agent receives no conversation history at all in this harness — the retrieval path returned empty on every call. You cannot withdraw a force from a system that was never carrying it. A fix landed in the agent (v2.9.14), but the measurements behind the current design predate it, so the contrast is dropped rather than quietly rerun.
The name stays. It describes the question honestly, including the part that is still out of reach.
| measurement | value | what it bounds |
|---|---|---|
| movable range | 12.9% | swapping the entire values document changes 12.9% of verdicts. No contrast between these arms can detect more. |
| gold-label floor | 4.1% | fraction of benchmark answers that are wrong on inspection. Bounds any accuracy claim; largely cancels in a paired comparison. |
| position skew | pipeline-only | accuracy depends on where a question sits in the conversation — but identically across all four arms, so it cancels between them. |
The movable range was measured twice, by two different methods, on two different agent versions: 9.2% from a discriminating-set probe, 12.9% from direct verdict agreement. Both say the same thing — the manipulation moves about one item in eight, and an experiment claiming a large effect on this instrument would be claiming something the instrument cannot see.
Status: running. 540 items, four contrasted arms plus two reference arms, $24. Every stake is marked survives / fires / void when it lands — including the ones that go against us.
TORQUE_FINAL.yaml |
the design: arms, three contrasts, bounds, and every input to the sample size |
experiments/torque/ |
corpora, tooling, probes |
| CIRISOntology#5 | independent review — approved conditional on two items, both discharged |
| RATCHET#20 | the position-effect investigation, including four eliminated hypotheses |
Comparing two value systems means building a second corpus that differs only in values — same length, structure, register and procedural content. Two attempts at writing one failed, the second worse than the first: asked to repair its own drift, it folded the extra content into the lines it claimed to be restoring.
That is not carelessness. An author asked to rewrite a document while leaving most of it alone will improve the neighbouring sentences, because that is what writing is. Checking afterwards is a race the reviewer loses to a fluent author.
So nothing is rewritten. The corpus is built mechanically — see
partition.py,
substitute_terms.py,
detect_residue.py:
- Split the document line by line into "states a value" and "does not." Review it, freeze it — that split is the public record of what changed.
- Substitute names globally and mechanically. Nobody authors a name, so no author can diverge on one.
- Author meanings in isolation — one line and the alt source, never the surrounding document. Nothing adjacent to improve, because it is not visible.
- Assemble and assert byte-identity on every unchanged line. A test, not a review, and it cannot be talked around: a line with content folded into it fails automatically.
Result on the main corpus: the intervention is 49 lines of 1,153, verified, with the frozen partition published alongside so anyone can see exactly what was varied and what was held.
Every claim carries one of four labels and we do not round up. "Validated" is not one of them.
| label | means |
|---|---|
| proved | follows deductively; no experiment can confirm or refute it |
| measured | observed, with the domain stated |
| open | testable, not yet tested |
| wager | we are betting on it, and saying so |
Grammar: CIRISOntology.
RATCHET began as a computational implementation of the Coherence Ratchet framework (Book IX of the CIRIS Covenant). Two results from it are load-bearing, at very different levels of confidence.
k_eff = k / (1 + ρ(k−1)) — the Kish design effect. proved: an identity
from survey statistics. It needs no experimental validation and can receive none —
computing it and comparing it against itself is arithmetic, not evidence.
Machine-checked proofs are in formal/ (Lean 4).
k_eff is sound as diversity bookkeeping — a correlation discount — and is
used that way here.
Whether ρ predicts fragility in a real system — open. Untested here and everywhere else in the corpus. The bridge from a proved identity to a measured prediction has not been built.
See CCA_PAPER/ for the paper and its correction history, and
KNOWN_LIMITATIONS.md for the theoretical boundaries that
are boundaries rather than bugs.
git clone https://github.com/CIRISAI/RATCHET.git
cd RATCHET
python -m venv .venv && source .venv/bin/activate
pip install numpy scipy pydantic cryptography z3-solver matplotlib pytest
pytest simulation/ -vThe campaign tooling under experiments/torque/ needs only the standard library
plus pyyaml.
experiments/torque/ the TORQUE campaign — design, corpora, tooling
docs/ the plain-English overview (GitHub Pages)
formal/ Lean 4 proofs
ratchet/engines/ detection, geometric, complexity, federation engines
CCA_PAPER/ the paper and its corrections
simulation/ standalone simulations and tests
Production AI safety without defence in depth; high-stakes deployment as a sole safety mechanism; unsupervised operation. This is a research testbed.
Looking for the agent itself? CIRISAgent
is the runtime under test — including its safety battery in tests/safety/. This
repository is the measurement apparatus, not the system being measured.