feat(scheduler): the power axis — the scheduler refuses work it cannot power (bp-154) - #41
Merged
Conversation
…robe bp-154 Item 1 (dn-supervision-and-liveness Amendment A1; issue #12). §2.7 gave the supervisor exactly one resource axis to refuse on — memory. This is the sensor for a second: the machine's remaining energy. `scheduler/power.py` is deliberately shaped like `scheduler/presence.py` rather than invented — an injectable probe (so the gate is testable with no hardware), a floor, and the idiom that matters: a fail-CLOSED default. `Presence` reads `assume_present_when_unknown = True`; this reads `assume_discharging_when_unknown = True`. An unreadable battery is treated as discharging, because the failure mode being designed against is the machine dying and a sensor that failed open would re-create it exactly when the system is least healthy (A1.7's named falsifier). `None` is an ORDINARY state here, not a theoretical one, so all four paths to it are carried from the probe precedent and tested: an absent `pmset` (`shutil.which`), a failed or hung exec (explicit `timeout=5`, plus the `(OSError, SubprocessError)` catch), unreadable output, and — added because an exception escaping into `tick` would be a new crash path rather than a guard — a probe that raises. One named asymmetry, deliberate and defaulted off: an unknown PERCENTAGE does not put the sensor below the floor. Fail-closed on the discharging question costs the heavy lanes; fail-closed on the floor would refuse every tier, so a host with no battery to read (a desktop, CI, any non-macOS worker) would dispatch nothing ever — the guard against an outage would be the outage. `halt_when_percent_unknown` exists, off, so the decision is visible. Verification: `uv run pytest tests/unit/test_power.py -q` → 22 passed. No `pmset` subprocess runs in the unit tests (every probe is injected), and the real probe is proven not to run at import or by construction — the call counter is asserted in both directions so the test cannot pass against a dead probe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U
bp-154 Item 2 (dn-supervision-and-liveness A1.1). The supervisor gains a THIRD
refusal predicate beside `blocked_tiers()` (the foreground gate) and
`model_blocked_tiers()` (the single-model rule): on battery, shed the heavy
lanes.
⚑ The amendment's one load-bearing pin is honored: the power rule is its OWN
predicate and is never folded into `blocked_tiers()`. That method's own
docstring already gives the reason about the model rule — "two different reasons
to refuse a tier, conflated into one predicate, is how a reader later cannot
tell which rule refused a job" — and a power refusal answers a different
question ("is there energy?") from a presence refusal ("is the owner here?").
`model_blocked_tiers` was the precedent for a sibling; this is the third. Each
of the three now points at the other two, so all are findable from any one.
`HEAVY_TIERS` is READ, never reshaped — one shed vocabulary, not two (A1's
parked selector decision; `load_key` was rejected as the default because its
interaction with this set is undesigned). `tests/integrity/test_shadow_
isolation.py`, which imports `HEAVY_TIERS` and is deliberately out of scope,
stays green untouched.
This commit is INERT by construction: the predicate computes a set nobody
consumes yet. `tick` is untouched, so no dispatch decision changes. Item 3's
whole content is "and now it is actually consumed" — separated precisely because
a guard nobody calls is the finding-0187 shape (deleting bp-105's sweep call
left 85/85 green).
`Supervisor.power` is defaulted like `presence`, so every existing construction
site gets the guard with no edit. Because that default FAILS CLOSED, tests that
dispatch a heavy tier must inject an on-AC sensor rather than inherit the host's
real battery — `tests/fixtures/power.py` is the one place those doubles live,
with the rule about which to use (injecting on-AC into a test meant to exercise
a power refusal would hide the feature).
Verification: `uv run pytest tests/integration/test_supervisor.py -q` → 29
passed (28 pre-existing, unchanged in behaviour; the foreground gate is asserted
byte-identical under all four presence × power combinations, and a source-shape
test proves the two predicates read their own sensor and no other's).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U
bp-154 Item 3 (dn-supervision-and-liveness A1.1/A1.4). Item 2 landed the predicate inert; this is the commit whose whole content is "and now it is actually consumed". The union in `tick` gains one term — `| self.power_blocked_tiers()` — and nothing else changes: no new queue API, no second enforcement point, the ONE claim site pin intact. ⚑ The falsifier is executed, not asserted. With the new term deleted from the union, exactly one test reds — `test_a_heavy_job_is_NOT_claimed_while_discharging_and_IS_once_back_on_AC` — while all 22 sensor tests and all 5 predicate tests stay green. That is the finding-0187 shape caught in the act: a guard nobody calls passes every test written about the guard (deleting bp-105's sweep call left 85/85 green), so the test that matters tests the COMPOSITION. Non-vacuity is asserted inline before the claim: the foreground gate is open, the model rule is unarmed, and the floor is not reached (55%), so the power term is provably the only rule that can have refused the job. The same queue is then drained by an on-AC supervisor and the held job runs — the rule defers, it never drops. A companion test pins the other direction: adding a term must not weaken the terms already there (owner present + on AC still refuses the heavy lane). Enforcement tier, honestly: tier 5 with a tier-4 test, identical to what §2.7 claims for the memory ceiling. Power is a sampled reading of the physical world, so tier 1 is unreachable and claiming it would be the overclaim §0's ladder names as the foot-gun. Verification: `uv run pytest tests/integration/test_supervisor.py -q` → 31 passed; `uv run ruff check scheduler/ tests/integration/test_supervisor.py` → clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U
…the ledger clean bp-154 Item 4 (dn-supervision-and-liveness A1.2). Below ~20% while discharging, the answer stops being "shed the heavy lanes" and becomes "start nothing at all": `tick` refuses every tier, including the light ones the shed leaves alone. The refusal sits BEFORE `claim`, and that placement is the whole content of "close the ledger clean". No RUNNING row is minted, so nothing is left behind for a machine that may not survive to close it, and a following run's orphan sweep finds nothing to reclaim and nothing to strand-fail. Claiming here would reproduce the Aug 1 shape exactly — the machine died mid-run and run #39 came up in recovery. The hold for AC is the ABSENCE of dispatch, not a wait loop. `run` returns 0 on the first tick and hands control back; the launcher's existing conditional sleep ("sleep only when the drain came back idle") is the duty cycle, and the next tick re-reads the battery, so mains returning resumes dispatch with no intervention and no state to reset. An in-process sleep/wait was the rejected alternative in A1's parked hold-for-AC decision: it would hold the supervisor lock while doing nothing and would itself drain the battery it protects. That falsifier is now checkable rather than argued — a counting probe proves a ten-tick drain request costs exactly ONE reading. Nothing is killed. This bounds what is STARTED; a job already in flight runs to its own completion or checkpoint (A1.4's honest limit — in-flight energy bounding is finding-0178's machinery, not this), and a job that finished before the floor was reached is left DONE, asserted. Deleting the floor branch reds four tests; the fixtures are non-vacuous by construction (the floor test enqueues a LIGHT tier, so the discharging shed provably does not cover it). Verification: `uv run pytest tests/integration/test_supervisor.py -q` → 35 passed; `uv run ruff check scheduler/ tests/` → clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U
… built bp-154 Item 5. `Supervisor.power` is defaulted like `presence` so the live daemon gets the guard with no wiring — and because that default FAILS CLOSED, a default-constructed supervisor in a test decides heavy-tier dispatch from the machine the suite runs on: the developer's actual battery on macOS, and "no `pmset` therefore discharging" on CI's ubuntu runner. Repaired by injection, exactly as each of these already injects `Presence(idle_probe=...)` rather than reading real HID idle time: `test_cron.py`, `test_research_cron.py` (both dispatch synthesis-tier jobs), `test_chat_sensor_wiring.py` and `test_scheduler_live.py` (pinned tier — the shed does not reach them, but the floor would on a draining laptop). ⚑ Proven necessary, not assumed. With the probe forced unreadable to simulate the CI runner, `test_cron_jobs_are_gated_during_foreground_then_run_in_a_trough` fails `assert trough.run() == 2` with 0 — the trough half of the foreground test never runs, because the synthesis tier is shed for want of a battery reading. With the injection restored and the same simulation in place, the whole model-free tier is green but for the three known-red worktree-enforcement tests (issue #13). Note the scope: `test_cron.py`, `test_research_cron.py` and `test_chat_sensor_wiring.py` are NOT in bp-154's write_scope. §5 anticipated the class of repair — "may observe a supervisor constructed without a power sensor; carried so a default-construction change can be repaired in-session" — but enumerated only three files; the real set of real-Supervisor construction sites in tests is six. Flagged for review rather than quietly absorbed. Item 5's own falsifier is respected: `on_ac()` restores the intended subject of a test that was never about power. No test meant to exercise a power refusal is given one — those live in `test_supervisor.py` and inject their own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U
…orded The seal entry carries the read map, the Follow-through block, the gate results per leg exactly as observed, and the two in-build decisions with their warrant: unknown-percentage does not halt (a named field, not an omission), and the hold for AC is the absence of dispatch rather than a process exit (filed as #40 with its evidence — ThrottleInterval 10 s × preflight's uncosted ~120 s Ollama probe would out-drain the idle hold). Also recorded: the write_scope overrun on three test files (§5 anticipated the repair class but enumerated three of the six real-Supervisor construction sites), and that `tests/e2e/test_scheduler_live.py`'s earlier red was the known flake — it failed on the live model returning an empty string, one assertion AFTER `run() == 1` and `state == DONE` had already passed, and it passed in the gate run.
…tion The module docstring enumerates what the loop does at each step, and it is the first thing a reader meets. It listed the foreground check and the RAM ceiling but not the axis that now refuses dispatch outright — an omission that would let a later reader conclude presence and memory are the only reasons a job is refused. Adds step 0 (the floor, ahead of the claim) and the power half of step 1's shed. Comment-only; no behaviour, no test changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements bp-154 —
dn-supervision-and-livenessAmendment A1, the power axis. Thesupervisor could see memory and could see the owner; it could not see the battery. Now it can.
closes #12
Why (the warrant, glossed inline)
Issue #12 (
energy: the scheduler cannot see the battery, migrated from frozendocs/findings/finding-0279.md) records three measured emergencies, not the two its titlenames: Jul 24 — drained to 1% during the v1.18.0 deploy night, the embedder starved under
critical-battery throttle,
code_syncwedged, and the daemon died unwitnessed and stayed dead threedays; Jul 28 — 100%→8% in 2h40m during the revival's backlog drain, caught at the wire; Aug 1
— fatal, the machine died mid-run and run #39 came up in recovery.
The battery hardware is healthy and stays healthy: Condition Normal, 95% maximum capacity, 128
cycles, re-measured 2026-08-05. The drain is load, not degradation. The scheduler was the defect.
§2.7 had given the supervisor exactly one resource axis to refuse on — memory (non-negotiable #8,
"the scheduler refuses breaching work"). That principle was never memory-specific; it had simply
never been given a second axis.
What changed
scheduler/power.py(new)pmset -g batt, a ~20% floor,assume_discharging_when_unknown = Truescheduler/supervisor.pypower: Powerfield (defaulted),power_blocked_tiers(), the floor branch intick, one term added to the union at the claim sitetests/unit/test_power.py(new)pmsetsubprocess runstests/fixtures/power.py(new)on_ac()/on_battery()/unreadable(), one home for the doublestests/integration/test_supervisor.pytests/integration/test_cron.py,test_research_cron.py,test_chat_sensor_wiring.py,tests/e2e/test_scheduler_live.pySupervisorconstruction siteFive commits, one per plan item, deliberately reviewable in isolation.
⚑ The one hard pin held: three predicates, three questions
The power rule is its own predicate, never folded into
blocked_tiers(). That method's owndocstring already gave the reason about the single-model rule — "two different reasons to refuse a
tier, conflated into one predicate, is how a reader later cannot tell which rule refused a job" —
and a power refusal answers a different question ("is there energy?") from a presence refusal ("is
the owner here?").
model_blocked_tiers()was the precedent for a sibling; this is the third. Eachnow points at the other two. They compose only at the ONE claim site, by union.
HEAVY_TIERSis read, never reshaped — one shed vocabulary, not two. The tripwire(
tests/integrity/test_shadow_isolation.py, which imports it and is deliberately out of scope) isuntouched and green.
⚑ Fail closed, and one named asymmetry
Noneis an ordinary state here, not a theoretical one: an absentpmset, a failed or hung exec,unreadable output, or a probe that raises. All four funnel to "discharging", because the failure
being designed against is the machine dying and a sensor that failed open would re-create it exactly
when the system is least healthy.
The one asymmetry is named and defaulted off rather than left implicit: an unknown percentage
does not put the sensor below the floor. Fail-closed on the discharging question costs the heavy
lanes; fail-closed on the floor refuses every tier, so a host with no battery to read (a desktop,
CI, any non-macOS worker) would dispatch nothing ever — the guard against an outage would be the
outage.
halt_when_percent_unknownexists, off, asserted in both positions.The floor: refuse before
claim, and that is the clean closeBelow ~20% while discharging,
tickreturns beforequeue.claim. Norunningrow is minted, sonothing is stranded for a machine that may not survive to close it, and a following run's
sweep_orphansfinds nothing to reclaim or strand-fail — asserted, because the falsifier is the Aug1 shape. The hold for AC is the absence of dispatch:
run()returns 0, the launcher's existingconditional sleep is the duty cycle, and the next tick re-reads the battery. Nothing in the diff kills
an in-flight job.
How it was verified — the falsifiers were executed, not asserted
| self.power_blocked_tiers()from the union reds exactly one test —test_a_heavy_job_is_NOT_claimed_while_discharging_and_IS_once_back_on_AC(1 failed, 52 passed) —while every sensor and predicate test stays green. That is the finding-0187 shape caught in the act
(deleting bp-105's sweep call once left 85/85 green), and it is why Items 2 and 3 are separate
commits: Item 2 lands the predicate inert, Item 3's whole content is "and now it is consumed".
purpose, so the discharging shed provably does not cover it — the test cannot pass on the shed's
behaviour.
costs exactly ONE battery reading.
simulate the CI runner (
ubuntu-latest, nopmset),test_cron_jobs_are_gated_during_foreground_then_run_in_a_troughfailsassert trough.run() == 2with 0. With the injections in place under the same simulation the whole model-free tier is green
but for three known-red worktree tests.
The local gate, each leg run separately, counts exactly as observed
uv run ruff check .uv run scripts/check_imports.pyuv run mypy core agents eval ops scheduler scriptsuv run mypy(argless)uv run python -m ops.type_gatepsutilreportuv run pytest -qThe five reds are the three known-red-locally classes and nothing else: the finding-0103
core-self-containment ratchet,
tests/e2e/test_dream_v2_live.py, and threetests/integration/test_worktree_enforcement.pycases (#13 — green in CI).tests/e2e/test_scheduler_live.py(the known flake, and in this plan's write scope) passed inthe gate run; an earlier full run failed it on
assert captured["text"].strip()withrun() == 1and
state == DONEalready passing — the job dispatched and the live model returned an empty string,which a power refusal cannot cause (a refusal fails one assertion earlier).
For the reviewer to rule on
tests/integration/test_cron.py,test_research_cron.pyandtest_chat_sensor_wiring.pyare not in bp-154 §5's list. §5anticipated exactly this repair class — "may observe a supervisor constructed without a power
sensor; carried so a default-construction change can be repaired in-session" — but enumerated
three files where the real set of real-
Supervisorconstruction sites in tests is six. Therepairs are one line each and are proven necessary above.
records "stop cleanly; launchd KeepAlive restarts and re-evaluates". This PR holds by not
dispatching instead, because
ThrottleIntervalis 10 s and every respawn pays preflight'suncosted ~120 s Ollama probe (frozen finding-0195) — a restart loop at the floor would spend the
reserve the floor exists to protect.
ops/lifecycle/launcher.pyis also outside this plan'swrite_scope. Consequence stated honestly: the queue ledger is clean, the run ledger is
unchanged, so a death at 0% still produces a recovery run — with no stranded jobs in it.
§2.7 claims for the memory ceiling. Power is a sampled reading of the physical world, so tier 1 is
unreachable and claiming it would be the overclaim §0's ladder names as the foot-gun.
already running — Jul 24's
code_backfillwas in flight when the throttle hit, so this would nothave prevented that emergency outright. In-flight bounding is finding-0178's machinery.
Ready to deskcheck on real hardware: unplug and watch a synthesis-tier job stay QUEUED while the
light lanes drain, then plug in and watch it go. The code path is what the tests exercise; the
deskcheck is what proves the probe reads what we think it reads on the live daemon. (Checked by
reading rather than executing:
/usr/bin/pmsetis-rwxr-xr-x root:wheel,-g battneeds noprivilege, and neither plist overrides PATH — so launchd's default PATH reaches it.)
Context: #40 (parked,
route:orchestrator), #13 (the locally-red enforcement suite),bp-153 — this plan is cheap insurance for that rebuild, whose Jul 24 wedge was itself caused by
a battery drain.
🤖 Generated with Claude Code
https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U