Skip to content

M8: optional Jev/System One escalation (--escalate jev) - #68

Merged
hexuria merged 10 commits into
mainfrom
instinct/m8-jev
Oct 17, 2026
Merged

hexuria merged 10 commits into
mainfrom
instinct/m8-jev

Conversation

@hexuria

@hexuria hexuria commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • JevEscalation in gpui-agent-policy behind new optional jev feature — port of aui-cli/src/typesafe.rs onto our ActionSpace. One blocking::Client::system_one call per abstain: an operation choice head plus one <kind>_target head per element-bearing kind, keyed by id region so composed replies are valid offered ActionIds.
  • Still behind EscalatingPolicy: local Instinct decides first; Jev runs only on Abstain — abstain never becomes "top candidate wins". The text tier stays command-escalation-only.
  • CLI --escalate jev (conflicts with --escalate-cmd). Missing TYPESAFE_API_KEY is a clear config error; a binary built without jev names the missing feature. TYPESAFE_MODEL / TYPESAFE_BASE_URL / TYPESAFE_DEFAULT_MODEL handled by the SDK.
  • Strict validation, all fail-closed → escalation: → Blocked: off-menu choice, uncovered/extra probability keys, sum ≠ 1, out-of-range/non-finite probabilities, non-argmax choice, transport errors, empty operation menu.
  • typesafe-sdk 0.3 optional — cargo tree shows no typesafe in default or instinct builds (extends the M1.7 guard).
  • Requests sent Compression::None so scripted endpoints stay std-only.
  • bench/arms.toml gains arm a4 (--escalate jev). A4 ran live (jev-latest): 75/80 → ambiguity guard → a4+guard wrong=0; recorded in RESULTS.md.
  • §8 docs sweep: PROTOCOL.md (since/compact/fingerprints/revision/expect_fingerprint/features), INTEGRATING.md (host fingerprint-gate contract), PERF.md (M2–M8 numbers), RECIPES.md (instinct resolve), README.md (run/locate/act + flags).

Test plan

  • 11 new jev.rs unit tests — build_questions coverage (heads, id-region keys, criterion label text, controls-only space), validate_choice (off-menu, uncovered, extra keys, bad sums, out-of-range incl. boundary matrix + NaN, non-argmax, argmax accept), keys_for, element_kind_of, model_non_empty, plus a std-only in-crate mock System One server asserting request model/state/questions
  • 7 run_jev.rs integration tests over a std-only mock — menu pick dispatches, Instinct choice never calls Jev, non-argmax/bad-sum/uncovered/off-menu → blocked, missing key → clear error; request asserts model from TYPESAFE_MODEL
  • cargo test --workspace green in default, instinct, and jev configs (25 suites under jev)
  • cargo clippy --workspace --all-targets --features jev -- -D warnings clean; also default + instinct configs; cargo doc clean under -D warnings
  • cargo tree under default/instinct has no typesafe
  • Mutation gate: cargo mutants -p gpui-agent-policy --features jev -f jev.rs → 40 caught / 0 missed / 8 unviable
  • A4 live run (jev-latest, n=5 × 16): 75/80 — Jev coin-flipped T08's identical-row menu (5 wrong) → ambiguity guard (7cdfa37): identical (kind,label) contenders short-circuit before any endpoint → a4+guard 5/5, wrong=0. RESULTS.md §M8 + §Final summary.

Generated with Devin

@hexuria

hexuria commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

A4 live + bonus fix (2026-10-19) — --escalate jev against real System One (jev-latest): 75/80, all 5 wrong on T08 (Jev coin-flipping identical-row menus). New ambiguity guard (commit 7cdfa37): identical (kind,label) contenders short-circuit before any endpoint — a model sees the same identical options, so a pick is a coin flip, not evidence. Rerun: a4+guard 5/5, wrong=0. Mutation gate now also covers escalate.rs: 38 mutants, 28 caught, 0 missed.

@hexuria

hexuria commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

Two-axis review remediation — all findings addressed in 64c43f9

A Standards + Spec review of origin/main...HEAD (the combined M2–M8 stack) found contract bugs now fixed:

Spec conformance

  • choose/jev request actions now carry state ({value,checked,states} from UiNode) per M7.2 — OfferedAction::state() + NodeState::to_json(), covered by choose_request_actions_carry_state.
  • act replies conform to the M4.5 envelope: {"done":{op,id,text?},"rev":N,"diff":{…}} via post-dispatch snapshot since=<pre-rev>; abstains and blocked/wait controls emit {"abstain":{reason,candidates}}.
  • DeterministicTextResolver is spec-strict: multiple quoted literals without label disambiguation → TextError::Ambiguous (dropped the untyped[0] fallback).
  • Runner stale bound aligned to two re-decides (M5.3 "at most twice"), matching act.
  • M2.5 proptest added (tests/props.rs): a Choice always names an offered action and never a dangerous one — abstain can never produce a dispatchable op. Committed regression seed guards the duplicate-id shrink.
  • k declared in the act tool schema.

Standards

  • CI now compiles the feature surface: cargo test under instinct and jev + jev clippy; bench job runs r1 (instinct resolver) and a2 (local policy) arms — previously every mcp_instinct/run_* test file compiled to nothing in CI.
  • bench/run.py no longer sends a raw token when the host issues no challenge.
  • INSTINCT_VERSION_REQ const + manifest pin-test replaces the "0.1" literal in receipts.
  • SECURITY.md documents --escalate-cmd env inheritance (GPUI_AGENT_TOKEN, TYPESAFE_API_KEY reach the child).
  • lib.rs contract wording carves jev out of "no floats, no network, no model" explicitly.
  • PRD gains §11 "Deviations" addendum recording every deliberate divergence (opt-in defaults, ambiguity guard, spec-strict text resolver, jev f64 carve-out, stale-bound interpretation).

Verified: fmt + clippy -D warnings ×3 configs; workspace tests ×3 (0 failures); escalate.rs mutation gate 38 mutants / 0 missed; A2 re-run at the specced n=20 (320/320, wrong=0, p50 ≈49 ms — RESULTS.md updated); live spot-checks a3 10/10 + a4 6/6 (incl. T08), wrong=0.

Still gated on M0.4 (human crates.io publish) before [patch.crates-io] can drop — unchanged per #69.

codeitlikemiley and others added 10 commits October 18, 2026 03:03
Port of aui-cli/src/typesafe.rs onto our ActionSpace: Instinct decides
first; JevEscalation runs only on Abstain behind EscalatingPolicy. One
blocking System One call per abstain — an `operation` head plus one
`<kind>_target` head per element-bearing kind, keyed by id region so
the composed reply re-enters the same closed-menu validator as M7.

Fail-closed throughout: off-menu choices, uncovered/extra probability
keys, bad sums, out-of-range or non-argmax answers, transport errors,
and missing TYPESAFE_API_KEY all surface as escalation:* → Blocked.
Requests go uncompressed so scripted endpoints stay std-only.

typesafe-sdk 0.3 is an optional dep; default/instinct builds carry no
typesafe in cargo tree. --escalate conflicts with --escalate-cmd; the
text tier stays command-only. A4 bench arm added; not run (no key).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- 40/40 jev.rs mutants caught (was 13 missed): keys_for / element_kind_of /
  model_non_empty coverage, range-check boundary matrix (inclusive 0/1,
  negative-vs-sum isolation, NaN), criterion label text, and a std-only
  in-crate System One mock asserting request model/state/questions.
- PROTOCOL.md: snapshot since/compact/fingerprints, revision op,
  TreeDiff shape, expect_fingerprint gate, hello.features.
- INTEGRATING.md: host-side expect_fingerprint contract + checklist.
- PERF.md: M2–M8 numbers. RECIPES.md: instinct resolve. README.md:
  run --goal, locate/act MCP, feature flags.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A4 live data: on T08 (two identical "Buy milk" rows, refuse task)
Jev picked one of the identical delete targets — a coin flip that
counts as wrong_action. No endpoint can resolve identical (kind,
label) contenders; the menu itself is ambiguous. EscalatingPolicy
now returns the local abstain without calling the fallback — the
model faces the same identical options, so paying for its guess
violates abstain-over-guess. Same fix protects --escalate-cmd.

Also fix bench LLM harness for hosts that ship the v2 challenge
handshake (HMAC auth, never the raw token), pydantic tool-use
serialization in the byte counter, and explicit api_key for
anthropic>=0.75 credentials-file discovery.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
All previously blocked gates now have measured data:

- A0 baseline (never recorded before): 65/80, 13,409 in_tok p50,
  $0.00609/passed. T14 = driver quits the app on a refuse task.
- GATE-T PASS: A1 65/80 wrong=0 at 4,550 in_tok p50 = 34% of A0.
- GATE-E PASS: A3 80/80 wrong=0, 61ms p50, 15 escalations total,
  $0.0000306/passed (0.5% of A0).
- A4: 75/80 — Jev coin-flipped T08 identical-row menu (5 wrong,
  pre-guard binary); with the ambiguity guard T08 is 5/5 wrong=0.
- escalate.rs mutation gate: 38 mutants, 28 caught, 0 missed.

New tests: ambiguity guard (2), subprocess round-trip via /bin/sh
stubs (3, unix-gated), choose_request wire shape, text boundary 256.
bench report skips non-benchmark side files and tolerates sparse
rows (harness-error rows lack wall_ms).

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…rage

Standards fixes:
- ci.yml: test/clippy under instinct + jev features; bench job runs r1 and
  a2 arms so the M2-M8 test surface actually compiles in CI
- bench/run.py: drop raw-token fallback on challenge-less hosts (PROTOCOL
  forbids tokens on the wire)
- runner.rs: INSTINCT_VERSION_REQ const + manifest pin-test (was "0.1" literal)
- run_cmd.rs: remove dead json!(receipt) + unused import
- SECURITY.md: document --escalate-cmd env inheritance (GPUI_AGENT_TOKEN,
  TYPESAFE_API_KEY reach the child endpoint)
- lib.rs: carve jev (blocking network + f64 reply validation) out of the
  crate-level "no floats, no network, no model" contract wording

Spec fixes:
- OfferedAction gains NodeState {value,checked,states}; choose/jev request
  actions now carry state per PRD M7.2 (wire + regression tests)
- act replies conform to M4.5: {"done":{op,id,text?},"rev":N,"diff":{...}}
  via post-dispatch snapshot since=<pre-rev>; abstains and blocked/wait
  controls emit {"abstain":{reason,candidates}}
- tests/props.rs: proptest — a Choice always names an offered action and
  never a dangerous one (M2.5 abstain-never-dispatches); committed
  regression seed guards the duplicate-id shrink
- runner stale bound aligned to two re-decides (M5.3 "at most twice")
- text.rs: multi-literal no-label-match returns TextError::Ambiguous
  instead of the untyped[0] fallback (spec-strict)
- mcp.rs: declare k in the act tool schema
- PRD §11 addendum documents every deliberate deviation
- RESULTS.md: A2 re-run at n=20 — 320/320, wrong=0, wall p50 ~49ms

Verified: fmt + clippy -D warnings x3 configs, workspace tests x3
(0 failures), escalate.rs mutation gate 38 mutants / 0 missed, live
spot-checks a3 10/10 + a4 6/6 wrong=0.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@hexuria
hexuria changed the base branch from instinct/m7-escalate to main October 17, 2026 19:03
@hexuria
hexuria merged commit 8b3a2eb into main Oct 17, 2026
5 checks passed
@codeitlikemiley
codeitlikemiley deleted the instinct/m8-jev branch October 17, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants