diff --git a/doc/rfc/submitqueue/speculation.md b/doc/rfc/submitqueue/speculation.md index e69de29b..27f4adc6 100644 --- a/doc/rfc/submitqueue/speculation.md +++ b/doc/rfc/submitqueue/speculation.md @@ -0,0 +1,392 @@ +# Speculation + +A merge queue that verifies one change at a time is as slow as its slowest build. Speculation pipelines it: while a batch's conflicting predecessors are still unresolved, build the batch against the futures they might produce, so that when reality resolves, a matching build is already done. This RFC designs speculation for SubmitQueue end to end: how futures are represented, which ones get built under a bounded CI budget, what is persisted, how decisions stay correct on an eventually consistent, transactionless substrate, and where every piece lives in the repo. + +In one paragraph: work enters as **batches** (changes verified and merged as a unit). Batches that touch the same code **conflict** and form chains — a batch's **dependencies** are the conflicting batches ahead of it, and it cannot safely merge until they resolve. Speculation builds a batch early against a guess at how its dependencies will resolve; each guess is a **path** (which dependencies land, which do not), and the batch a path builds is its **head**. Every change to a queue runs a **pass**: it reads the current facts, applies fixed rules, asks one pluggable extension (the **Speculator**) which paths to build, and writes the result. Only the paths actually chosen for building are stored (a **path set** per head); the rest are recomputed each pass, never stored. A batch merges only after its dependencies have (merged, or been ruled out) and a matching build passed — optimistic shortcuts are future work. + +## Problem + +Conflicting queued changes form dependency chains: a batch's build is valid only against a particular resolution of the batches ahead of it, so without speculation the queue serializes on its slowest build. Four constraints shape the design: + +- **The future space is exponential** — N unresolved predecessors give up to 2^N futures; whatever plans over them must never materialize the space. +- **CI is bounded** — which futures build is a queue-wide rationing decision under a concurrent-build budget, not a per-batch one. +- **The substrate is eventually consistent** — per-record CAS, no transactions; at-least-once, unordered delivery. Every decision must converge from stale reads, duplicates, crashes, and racing consumers. +- **Merges are irreversible** — getting speculation wrong may waste builds; it must never cause a wrong merge. + +Four structural facts make a lean design possible: + +- **The paths are derivable, not data** — a batch's possible paths follow from its dependency list, recomputable in microseconds, so storing them buys nothing. +- **Events only shrink the set** — when a dependency lands, paths that bet against it die; when one fails, paths that bet on it die. Each event filters, never rebuilds. +- **The best paths come out cheaply, in order** — when a path's score is a product of per-dependency odds, the best and next-best can be produced one at a time, without listing them all. +- **Real futures are far fewer than 2^N** — if dependencies depend on each other, a path that bets a batch lands while its prerequisite does not is nonsense, never considered; a straight chain of N has only N+1 real futures. + +## The speculation pass + +A pass runs whenever the queue changes. Three events trigger it — a **new batch** (via score), a **build result** (buildsignal), a **merge result** (mergesignal) — plus cancel and a self-arming tick backstop; each publishes a dirty signal keyed by the queue name. The dirty signal is an internal queue contract — payload in `submitqueue/core/messagequeue`, topic key in `submitqueue/core/topickey`. The loop: + +``` + dirty(queue) — "run a pass" — is published after: + - a new batch is scored in (score) + - a build result lands (buildsignal) + - a merge result lands (mergesignal) + - a cancel, a DLQ recovery, or the self-arming tick + │ + │ keyed by queue name, so a queue's passes run one at a time + ▼ + speculate (orchestrator/controller/speculate) — all controller code, + │ except step 3 (the one call out to the extension) + │ + │ 1 read the queue's live batches, their path sets, executions, and builds + │ — once; never re-read mid-pass + │ 2 rules update each path's status from its build result; + │ cancel paths a resolved dependency has ruled out; + │ apply the verdicts — merge a ready batch, fail a dead one, + │ conclude a cancelled one (defined below) + │ 3 ask Speculator.Speculate(batches, in-flight) ◀── the only extension call + │ → paths to build, paths to preempt + │ 4 check validate that output: drop builds it shouldn't propose + │ (refuted, incoherent, already-failed), reject passed-path cancels + │ 5 write record each head's decisions, send build/cancel messages, re-arm + │ the tick (a head whose write loses is re-planned next pass) + │ + │ build / cancel (path ID, attempt) + ▼ + build (orchestrator/controller/build) + reserve → BuildRunner.Trigger(token) → record the build ID + │ + ▼ + CI runs; the result lands back at buildsignal — the loop closes +``` + +Four properties make this work: + +**Pure function of what it read.** The pass reads the facts once — batches, builds, path sets, execution records — and never re-reads mid-pass. It uses no limits itself (those belong to the Speculator) and never retries (a failed build is final). No pass depends on an earlier one, so trigger messages may be lost, duplicated, or reordered without harm: re-running from the current facts always converges. + +**Deterministic, so replay and shadow testing are free.** Given the same inputs, the default produces the same output. Shadow mode runs the live Speculator and a candidate one on the same inputs and compares them. Replay uses the per-pass log line the controller writes each pass (a digest of the inputs and the decisions) — written and never read back by the system, so nothing extra is stored. + +**One pass per queue, normally.** The message framework handles one message at a time per key; keying on the queue name runs a queue's passes one at a time, different queues in parallel. It is best-effort — if a pass runs past the message's visibility timeout the message is redelivered and a second pass can overlap — so the design tolerates two concurrent passes through per-batch conditional writes (compare-and-swap: write only if the row is unchanged), with the timeout set above the worst-case pass. + +**The pass is the only writer of the path sets.** Every path set is written only by the pass. The one exception: the **execution record** is written by the build stage, which creates the build and owns the record linking a path to it — keeping that out of the path entry leaves the path entry with a single writer. Every other stage writes its own data and signals the queue to run a pass. + +### Verdicts and other duties + +Beyond selection, the pass computes every **verdict** — a rule-bound batch outcome with one correct answer, from persisted facts, never the Speculator's to compute or veto: + +- **Merge finalization (strict).** A batch merges only after every dependency ahead of it has resolved — merged, or ruled out — and it has a passed build for exactly that outcome. The pass then moves it to Merging and hands it to runway (the service that performs merges). Down a chain, each batch waits for its predecessors to merge, so a chain lands one at a time. (The merge in runway is idempotent — a base-design obligation: retrying a change that already merged amounts to an empty merge, which runway skips while still reporting success, so a lost hand-off is always safe to re-send.) +- **No-viable-path failure.** A batch fails when no future can still pass it — every way its dependencies might resolve already has a failed build, so it cannot succeed no matter what happens ahead of it. The controller checks the *full* space of outcomes, not just the paths the wired extension chose to build, so a narrow extension can leave a batch waiting but can never wrongly fail one. +- **Cancel.** When a batch is cancelled, the pass cancels its in-flight paths and, once they have all stopped, marks it Cancelled. + +Three smaller duties: + +- **Merging supervision** — if the hand-off to runway is lost, the next pass re-sends it, and the outcome returns via mergesignal. A batch reaches Merging only once its dependencies are settled, so it cannot be invalidated mid-merge. +- **No dependent fanout** — a merge is a single dirty signal, not a message to each dependent. The next pass recomputes every batch against the new reality: dependents that bet against the merged batch are cancelled, those that bet on it keep building. +- **Optimistic hand-off is deferred** — the pass merges a batch only after its predecessors actually merge; pipelining that is a future extension. + +### Triggers and the tick + +**Each trigger needs a unique message ID.** The queue drops a message whose ID it has seen before (even one already consumed), so a reused ID silently loses the wakeup. Every producer builds its ID from something that changes each time — a build result, a batch's new state version, a merge result — so miss a producer and that change is only picked up at the next tick. + +**The tick is a periodic self-wakeup.** So a queue keeps making progress even when no event fires, each pass schedules a delayed trigger for the next fixed time slot. Its ID is (queue name, slot), so many passes scheduling the same slot collapse to a single wakeup rather than a storm. An empty queue stops ticking; a new batch starts it again. The tick also bounds how stale pull-only inputs get — notably any live limit an impl reads (budget, depth), which takes effect at the next event or tick. + +**Every event is its own pass.** Ten events are ten passes; a pass with nothing to change just re-arms the tick. Batching them is a possible later optimization, harmless because any run of passes reaches the same result from the same facts. + +## Persistence: path sets and executions + +The persistence model inverts: **persist paths admitted, not possibilities.** The stores are extension contracts in `submitqueue/extension/storage` (a path set store and an execution record store), key/value-shaped like the rest of that package. + +- **The path entry** — created when the pass chooses a path to build, keyed by a hash of the path's content. Holds the path, its status, an attempt count, and a version. Written only by the pass. It carries no build reference — that lives in the execution record, which keeps the path entry to a single writer. +- **The execution record** — links a path (and attempt) to its CI build. The build stage writes it in two steps: a **reservation** before the build is triggered (so no build can exist that no record points to), then the **build ID** once the build starts. The trigger carries a token derived from (path, attempt), so a retry after a crash lands on the same build rather than starting a second one. The pass reads this record to learn each build's outcome. This is why the build runner must honor that token (see Open questions). +- **The path set** — one row per head, holding that head's chosen paths and their statuses (pending, building, passed, failed, cancelling, cancelled) under one version. Settled entries are kept a while, so a re-run does not collide with an old build. It never holds unchosen paths — those stay virtual. The live heads come from the batch store's existing by-state listing, and each path set is a direct read by batch ID; when a head finishes it drops out of that listing and its row is later garbage-collected by the speculate controller (see Path identity). + +Passed entries stay in the row until the batch finishes, so merge finalization finds them in one read. + +The stored shapes (`submitqueue/entity`; field lists may change): + +```go +// SpeculationPathEntry is the stored record of one chosen path, keyed by a +// hash of its content. It has no build reference (that is the execution +// record's) and no score (a score is transient — meaningful only within a +// single pass — so it is never stored). +type SpeculationPathEntry struct { + ID string // primary key; hash of the path's content + Path SpeculationPath // head + which dependencies it bets land / are ruled out, in queue order + Status SpeculationPathStatus // pending | building | passed | failed | cancelling | cancelled (for the current attempt) + Attempt int // run count, >= 1; builds and messages key on (ID, Attempt) + Version int // for compare-and-swap writes + CreatedAt int64 // ms + UpdatedAt int64 // ms +} + +// SpeculationPathSet is one row per head: that head's chosen paths. Finished +// entries linger (so a re-run cannot collide with an old build) until the +// row is garbage-collected with its head. +type SpeculationPathSet struct { + BatchID string // primary key; the head batch + Paths []SpeculationPathEntry // the head's chosen paths, live and recently finished + Version int // for compare-and-swap writes +} + +``` + +Unchosen paths are never stored: they are fully recomputable, storing them would mean rewriting many rows every time a dependency resolves (with no transaction to keep them consistent), and a stored copy can drift from reality while recomputing cannot. + +### Path identity + +A path's ID is a hash of its content — the head plus which dependencies it bets on. So the same guess always produces the same ID, which is what makes crash recovery simple: if a pass crashes and re-runs, it recomputes the same IDs, re-writing an entry is a harmless no-op, and it never starts a second build for a path it already has. A change that gets re-batched becomes new batches and a fresh path automatically. + +Each path also carries an **attempt** count, kept on the record rather than in the ID: a path cancelled while still viable can be re-run at the next attempt, but paths that failed or were ruled out never re-run. + +**Garbage collection** belongs to the speculate controller (`orchestrator/controller/speculate`) — the path set's single writer, deletion included: a path and its builds are deletable only once the batch and all its builds have finished, and a pass that observes a head past that point (retention elapsed) deletes its row. No extension ever writes. + +## Extensions + +**One extension: the Speculator.** Every pipeline-stage decision in this repo is one extension (`conflict.Analyzer`, `scorer.Scorer`, `buildrunner.BuildRunner`); the speculate stage's is *which paths to build, and which running ones to cancel*, so it gets one. Everything else stays controller code — reconciling facts, cancelling ruled-out paths, the verdicts, and checking the extension's output. Swapping the Speculator changes which paths run; it cannot change whether a batch merges or fails. (Letting the extension decide those would be unsafe — a limited plugin could "fail" a batch whose futures it never looked at.) + +**The contract** is `Speculate(batches, inflight) → []Speculation`. In: the queue's live batches (each with its prior score, dependency list, and state) and the in-flight path sets, one per head (each holding that head's chosen paths and their current statuses). Out: a list of build/cancel actions, nothing else. Budget, depth bound, and clock are injected when the Speculator is built, not passed in. An implementation may read extra data (also injected), which only weakens replay/shadow testing, never correctness, since the controller validates the output either way. + +### The default Speculator + +Composed in wiring from three swappable interfaces. The Speculator is the unit of *replacement* (swap it whole for a joint optimizer or a learned model); PathScorer, Generator, and Allocator are the units of *tuning*. Its entire body: open a candidate stream per head, then hand the streams and the in-flight sets to the Allocator. + +- **PathScorer → `Score(path, deps)`** — scores a path as the product of its dependencies' landing chances, where a dependency's chance is its `Batch.Score` (forced to 1 or 0 once it resolves). Built per queue, stateless: the path's dependency batches come in through the argument, so there is no per-pass instance. A replacement may score paths differently. It is the Generator's dependency and only caller — it prices candidates so the Generator can order them; in-flight paths are never scored. +- **Generator → `CandidateStream`** — for one head, produces candidate paths best-first, on demand (the caller pulls the next only when it wants it, so nothing is computed past what is used). It also caps how deep it plans — **depth** is the number of unresolved dependencies a head's paths range over — so one deep chain can't blow up a pass: a head beyond the cap is skipped this pass and enters as its dependencies resolve, and deep paths are low-value anyway. The depth cap is the impl's own input (static config, or a live signal for load-shedding). +- **Allocator → `[]Speculation`** — spends the build budget (the queue's cap on concurrent builds) across heads: it takes the highest-scored candidate from any head and funds it until the budget is full, favoring paths that have waited longer. The budget is the impl's own input, and it is deliberately the single rationing lever — there is no score floor: the default spends every slot it has on the best remaining candidates, however long the shot. Nothing in the contract forces slots full (a thriftier impl may fund fewer), but that is an impl's own policy, not a second knob in the design. A building or cancelling path charges the budget (a cancelling build holds CI until observed terminal); passed, failed, and cancelled paths hold none. It scores nothing — candidates arrive scored, and it settles in-flight by matching IDs against what it admits: a path already building is kept; the sticky default leaves every viable in-flight running and fills only free slots, while a preempting impl cancels in-flight that miss the top-budget set. + +### Extension APIs + +The method shapes are the design decision; exact field lists may change. All four interfaces live in one `submitqueue/extension/speculation` package (implementations in subpackages), each built per queue in the wiring layer, with its limits (and any extra read-only inputs) injected there. + +```go +// Speculator — the one extension the controller calls. It receives the +// queue's live batches and its in-flight path sets, one per head (statuses +// already reconciled by the controller), and returns a list of actions on +// paths. Budget, depth bound, and clock are injected at construction, not +// passed here. An implementation deterministic over these two arguments and +// its own limit and clock samples gets replay and shadow testing for free. +type Speculator interface { + Speculate(ctx context.Context, batches []entity.Batch, inflight []entity.SpeculationPathSet) ([]Speculation, error) +} + +// Speculation is one proposed action on one path — the whole of the +// Speculator's decision surface, returned as a plain slice (a value, no +// callbacks into stores). A kept path simply has no entry; the controller +// enacts each Speculation and writes every status. +type Speculation struct { + Path entity.SpeculationPath // the path acted on; the controller derives its ID and mint-vs-resurrect + Action PathAction // Build | Cancel +} + +// PathAction is closed to the two side-effecting verbs the Speculator may +// propose. There is deliberately no Merge or Fail action — verdicts are the +// controller's and cannot be expressed here. The zero value is not a verb: +// a zero-valued Speculation is invalid, and the controller's check step +// rejects it rather than silently starting a build. +type PathAction int + +const ( + PathActionUnknown PathAction = iota // zero-value sentinel; never valid, rejected by the controller's check step + Build // start (or resurrect) a build for the path + Cancel // preempt this in-flight path (refutation cancels are the controller's, not this) +) +``` + +```go +// PathScorer scores one path against its dependencies. Built per queue in +// wiring, stateless, and pure: Score takes a path and that path's dependency +// batches (which carry each dependency's current Batch.Score) and returns a +// value — the per-pass data flows through the argument, so there is no per-pass +// instance to build. The default score is the product of the dependencies' +// landing chances — each dependency's Batch.Score, forced to 1 or 0 once it +// resolves. A replacement may score paths any way it likes (deps carries the +// batches, so a richer impl can read features too). It is the Generator's +// dependency (injected there) and its only caller: the Generator uses it to +// order candidates. In-flight paths are never scored — the Allocator settles +// their fate by ID, not by a number. +type PathScorer interface { + Score(path entity.SpeculationPath, deps []entity.Batch) float64 +} +``` + +```go +// Generator — the ordered source of futures, one head at a time. Returning +// a CandidateStream is the decision that makes laziness contractual: the consumer +// pulls, and the producer never learns why pulling stopped. It holds the +// PathScorer (injected at construction) and uses it to order candidates, so +// Open needs only the head and its dependencies. +type Generator interface { + // Open starts the head's candidate stream over its dependencies, + // best-first by the PathScorer's score. + Open(ctx context.Context, head entity.Batch, deps []entity.Batch) (CandidateStream, error) +} + +type CandidateStream interface { + // Head is the batch this stream produces candidates for; the Allocator + // reads its priority and wait time when ordering across heads. + Head() entity.Batch + + // Next yields the next-best candidate; ok = false means the head's + // space is exhausted. Candidates descend in score, never repeat, + // and never contradict a resolved fact. + Next(ctx context.Context) (c Candidate, ok bool, err error) +} + +type Candidate struct { + Path entity.SpeculationPath // the path the Generator produced + Score float64 // the score the Generator computed while ordering, carried within the + // pass so the Allocator ranks on it directly. Transient, not storage: + // stored entities keep no score (scores go stale across passes); within + // one pass the value is fresh. +} +``` + +```go +// Allocator — divides the build budget (the queue's cap on concurrent +// builds) among competing paths, ranking globally across heads by candidate +// score, aged by wait. It merges the per-head candidate streams, pulling each +// only as deep as the budget makes relevant; each stream knows its head, +// so an admission attributes back to it. Candidates arrive already scored; +// in-flight arrives unscored (just IDs and statuses) and is never scored — the +// Allocator matches each in-flight path against the admitted set by ID: one +// already building is kept (not re-triggered), and one the best-first pull +// never reaches is, by that ordering, below the cutoff. The sticky default +// keeps every viable in-flight build and only fills free slots; a preempting +// impl instead cancels in-flight that miss the top-budget set. Budget and +// clock are injected at construction, not passed. +type Allocator interface { + Allocate(ctx context.Context, inflight []entity.SpeculationPathSet, streams []CandidateStream) ([]Speculation, error) +} +``` + +**The default's `Speculate`, in outline** (the Generator and Allocator meet only here; the PathScorer lives inside the Generator): + +```go +// error handling elided. +func (s *defaultSpeculator) Speculate(ctx context.Context, batches []entity.Batch, inflight []entity.SpeculationPathSet) ([]Speculation, error) { + var streams []CandidateStream + for _, head := range liveHeads(batches) { + st, _ := s.gen.Open(ctx, head, deps(head, batches)) // Generator scores + orders candidates with its own PathScorer; caps its own depth + streams = append(streams, st) + } + return s.allocator.Allocate(ctx, inflight, streams) // matches in-flight by ID, funds the best candidates within its budget +} +``` + +## Four passes, by example + +Queue `q`: `B2` depends on `B1`, `B3` on `B1` and `B2` (a chain), `B4` independent. Priors: p(B1)=.90, p(B2)=.80. Budget 3, depth bound 2. Notation `[base]→head`: a path bets that the dependencies in `[base]` land and the rest are ruled out; its score is the product of those odds (p for a bet-to-land, 1−p for a bet-against) — the chance this exact path is what actually happens. An impossible bet (B2 landing without B1) is never generated. (Scores here are sharpened with build evidence; the plain default just uses each batch's prior — same result in this example.) + +``` +pass 1 — dirty: batches scored into an empty queue +────────────────────────────────────────────────── +inputs live {B1,B2,B3,B4} · no path sets yet — no records, no builds +scores p(B1)=.90 p(B2)=.80 (priors only — no evidence yet) +generate B1 ▸ []→B1 1.0 B4 ▸ []→B4 1.0 + B2 ▸ [B1]→B2 .90 B3 ▸ [B1,B2]→B3 .72 (B3 has 2 unresolved deps — + (one pull per head; deeper candidates exactly at the depth bound; + stay unpulled, unyielded) at bound 1 it would wait) +allocate budget 3 ▸ ✔ []→B1 1.0 ✔ []→B4 1.0 ✔ [B1]→B2 .90 │ ✘ [B1,B2]→B3 .72 — no + slot; stays virtual: no record, no status, nothing to clean up later +enact create records p1,p4,p2 (attempt 1) · in-flight {p1,p4,p2} · 3 builds · re-arm tick +``` + +How the Generator found B3's best future — the whole trick in one table. Each dependency is a two-way pick with a price: on — it lands (worth p) — or off — ruled out (worth 1−p). A future is one pick per dependency; its score is the product of its picks: + +``` + pick for B1 pick for B2 score + on .90 / off .10 on .80 / off .20 + [B1,B2]→B3 .90 × .80 = .72 + [B1]→B3 .90 × .20 = .18 + []→B3 .10 × .20 = .02 + [B2]→B3 .10 × .80 = (never exists: B2 needs B1) +``` + +Three things to read off the table: + +- **Best future = the bigger pick in every column.** .90>.10, .80>.20 → `[B1,B2]`, in one step, no enumeration — bet on a dependency exactly when its landing chance is ≥ .5; argmax arithmetic. +- **Next-best = flip the cheapest pick.** Swapping B2 (.80→.20) costs less than B1 (which drags B2 out too), so `[B1]` .18 is second, `[]` .02 last. Every swap shrinks the product, so the stream descends automatically. +- **The table is the naive alternative** — building, persisting, and scoring every row up front; here only pulled rows are scored. Pass 1 pulled B3 once, the budget filled, no other row was yielded. A stream holds a small frontier and expands on `Next()`, so work tracks pulls × dependencies, never the table. + +Even under a deep budget the work stays bounded: a stream yields only coherent, depth-capped futures, in descending score, so pulling stops when the budget fills or the streams run dry — never the 2^N space. + +``` +pass 2 — dirty: buildsignal, []→B1's build passed +───────────────────────────────────────────────── +inputs records p1 building→(FACT: build passed), p4 building, p2 building +scores p(B1)=.97 (its build passed — evidence sharpens the prior) p(B2)=.80 +verdicts p1 passed + empty base ⇒ B1 finalized: CAS Merging, publish merge + (strict rule satisfied vacuously — no predecessors to wait on) +generate B3 ▸ [B1,B2]→B3 now .97×.80 = .78 (yesterday's reject, rescored) +allocate p1 passed frees its slot — passed paths hold no CI; the entry stays + in B1's path set ▸ ✔ [B1,B2]→B3 .78 +enact p1 → passed (in-flight set shrinks) · B1 → Merging + + merge publish · create p3 (attempt 1) · in-flight {p4,p2,p3} · 1 build · + re-arm tick +``` + +``` +pass 3 — dirty: mergesignal, B1 Succeeded (B4 passed and finalized meanwhile) +────────────────────────────────────────────────────────────────────────────── +inputs B1 terminal · in-flight {p2,p3,p5} — the elided B4 pass spent its freed + slot on B3's next-best candidate [B1]→B3 ≈ .19 as p5 (no floor: a free + slot goes to the best remaining candidate, however long the shot) · + p2,p3,p5 building · p4 passed, settled in B4's path set (B4 in Merging) +scores B1 = certainty (resolved fact, no longer a probability) p(B2)=.80 +collapse p2 [B1]→B2: B1's pick settled to fact, score .90→1.0; a fresh + derivation of the same future writes the same Base [B1] — same path + ID, covered by identity, no rebuild + p3 [B1,B2]→B3: score .72→.80 · p5 [B1]→B3: .19→.20, the same way +gc B1 terminal ⇒ p1's records satisfy the GC rule (record and executions + deletable; retention is TTL tuning) +allocate budget full — p2,p3,p5 charge all three slots ⇒ nothing to pull +enact nothing to write — in-flight unchanged {p2,p3,p5} · no builds · re-arm tick +``` + +``` +pass 4 — dirty: buildsignal, B2's build failed +────────────────────────────────────────────── +inputs in-flight {p2,p3,p5} · p2 building→(FACT: build failed), p3,p5 building · + p4 passed, settled in B4's path set +verdicts B2's only live path failed — a failed path is final; CI already spent + any internal retries — and its coherent space holds no other + assignment ⇒ controller verdict: B2 Failed +scores B2 = ruled out (resolved fact) +refute p3 [B1,B2]→B3 holds B2 in Base; B2 ruled out ⇒ REFUTED — a fact contradicts + it ⇒ RULE cancel (controller; Speculator not consulted): p3 → cancelling, + cancel published + p5 [B1]→B3 bet exactly this — score collapses to 1.0; already building, kept +allocate budget 3, charged 2 — p3's cancelling build holds CI until observed + terminal, p5 builds on ⇒ one slot free · B3's stream is spent ([B1,B2] + is refuted, []→B3 contradicts B1's merge) ⇒ nothing to fund +enact p2 failed (settles in B2's path set) · p3 cancelling · 1 cancel · + in-flight {p3,p5} · re-arm tick +``` + +The refutation cost one wasted build, never a stall: the losing bet p3 is cancelled while p5 — the future that actually happened — has been building since the elided B4 pass. Note p5 `[B1]→B3` is a *different* path from p3 (a different dependency set, a different hash, a separate record), not a resurrection (which re-runs the *same* path at attempt+1). + + +## Future extensions + +Three relaxations the architecture is shaped to receive — each additive, each with its own failure analysis, none in scope now. + +### Bypassing a large diff (full-coverage early merge) + +If a batch has passed builds covering *every* way its dependencies could resolve, its result is the same whichever way they go — so it can merge now, ahead of them. The classic case: a small change stuck behind a slow one is built both with and without it, both pass, and it merges immediately. This adds a second merge rule in the controller and asks the Allocator to fund the extra long-shot builds; runway and the merge stage are unchanged. Blockers: the number of combinations growing with depth, and the fact that the skipped dependency's base shifts under it — so its own later merge needs a re-check. + +### Optimistic merge + +Hand a batch to merge while its predecessors are still merging, instead of waiting for each — saving about one merge round-trip per link in a chain. Future work because it pulls the pass into merge-timing details it currently avoids (is a predecessor definitely merged or only probably; holding a batch that is waiting; undoing a hand-off that turns out wrong), each needing its own failure analysis — and undoing a hand-off is a strictly stronger demand than the idempotent retry the base design already requires. + +### Conflict relaxation (Floodgate) + +Score conflicts by strength and drop the weakest, so loosely-conflicting changes stop depending on each other and run in parallel — a smaller dependency set means fewer paths and shallower chains. To the pass this is invisible; it plans over whatever dependency list it is handed. Relaxation belongs in conflict analysis, not the Speculator: what counts as a dependency is a *correctness* input the pass computes verdicts over, so it cannot be a swappable extension's call. Its one obligation: a dropped conflict that turns out to be real must be caught at merge time, with limited and reversible fallout. + +(Tracks a separate in-flight design; treat the specifics as a moving target.) + +## Open questions + +- **Depth-bound value.** How the per-head depth bound is chosen — fixed, budget-derived, or load-shed under CI pressure — and whether it is per-queue config or a limit-style extension. +- **Runner token adoption.** How each BuildRunner backend delivers *same token, same build* (native idempotency keys, metadata search, a lookup API), and whether lookup-by-token becomes a first-class interface method. +- **Score model.** Whether a dependency's landing chance stays its independent `Batch.Score` or grows correlation awareness (shared targets, authors, build-system incidents); today a path's score is just the product of independent batch scores. +- **Explain surface.** An orchestrator-local debug RPC is the floor; whether "why is this batch waiting?" should reach the gateway status APIs is a product call. +- **Tick cadence.** The slot interval trades staleness against idle passes — a per-queue config value nothing else depends on. +- **Adjacent, pre-existing.** The cancel-vs-completed-push race (a batch recorded Cancelled after its changes physically landed) predates this design and needs its own fix.