Skip to content

fix(v2): cancel active drains and fence session deletes - #41

Merged
alderpath merged 6 commits into
Reliary:mainfrom
jrzmurray:fix/v2-drain-cancel-pr
Nov 14, 2026
Merged

alderpath merged 6 commits into
Reliary:mainfrom
jrzmurray:fix/v2-drain-cancel-pr

Conversation

@jrzmurray

Copy link
Copy Markdown
Contributor

What this changes

V2 interrupt now cancels the registered provider drain, reports its actual interruption result, and keeps later work ordered behind terminal publication. The bounded drain admission and per-session deletion fence prevent queued work or an in-flight turn from being stranded or writing into a deleted session. This change covers K-V2-DRAIN-CANCEL and K-SESSION-DELETE-FENCE.

Issue

Closes #

Verification

  • cargo fmt --check — passed
  • cargo clippy --workspace -- -D warnings — passed
  • v2_admission integration target — 22 passed
  • drain and deletion race tests — 15 passed
  • coordinator tests — 5 passed
  • uv run --with jsonschema bash ./scripts/check-guards.sh — passed
  • ./scripts/check-matrix.sh — passed
  • cargo test --workspace — blocked on this macOS host by seven Linux cgroup tests and one /proc RSS test
  • cargo test -p ocserve-http — 57 passed, 1 unrelated symlink containment test failed on this host

Linux CI remains the full-workspace gate.

Keep registered drains cancellable through claim release and terminal publication, and fence session deletion against new work. Bound drains, reservation waiters, prompt-lock waiters, and delete fences. Closes K-V2-DRAIN-CANCEL and K-SESSION-DELETE-FENCE (V2-PLAN P3).

Focused v2 admission (22), drain and delete races (15), coordinator (5), fmt, clippy, guards, and matrix passed. Full workspace tests have macOS host failures in Linux cgroup and /proc assumptions; the ocserve-http suite also has an unrelated symlink-containment failure on this host.
Amendment to the contributor's branch (credited @jrzmurray), from the review.

Three members of one class (blocking work on an executor that must stay
free), plus two waits with no ceiling:

  * check_v1_session_admission ran session_exists (open_reader + SELECT)
    inline on EVERY v1 prompt — /message, /prompt_async, /command,
    /summarize, and the v2 create. It had also REPLACED the run_blocking call
    prompt_async already had, so the change was a net loss of offload. It is
    now async and goes through run_blocking. (F1 class, PERF-10X.)
  * lock_session_wait branched on the error MESSAGE string
    ("too many concurrent prompts") to detect capacity. A wording change
    would silently fall into the 1 Hz retry branch while holding a drain
    slot. Now matched on status + name.
  * scheduler::step awaited reserve_drain_slot_wait inside the single serial
    tick loop, so one slow drain stalled every other schedule. It now uses
    the non-blocking reservation and returns without marking the row, so the
    next pass retries.
  * cancel_drain bounded its watch wait; cancel_drains_for_delete bounded the
    WHOLE teardown. The first attempt bounded only the inner loop, and the
    outer re-cancel loop still spun forever — the regression test caught it.
    Both record ocserve_v2_drain_cancel_timeout_total.
  * the interrupt re-arm read session_exists(...).unwrap_or(false), so a read
    error was treated as "session gone" and silently dropped the work. It
    propagates.

Docs: MEMORY states the drain cap is a COUNT bound and gives the real
worst case (min(128 slots, 64 prompt locks) x history); PLAN gains
D-V2-INTERRUPT-STATUS (one handler cannot return v1 204 and v2 200);
TESTING and TRACEABILITY gain the class and its negative control.

Verified: 929 workspace tests, clippy -D under CARGO_BUILD_WARNINGS=deny,
guards, matrix, docs. Negative control for the delete bound: delete the
outer deadline and the request hangs to the 90 s test timeout.
@alderpath

Copy link
Copy Markdown
Collaborator

Thanks — the core insight is right and the delete fence is the correct shape. I approved the workflows (fork runs had never executed) and pushed two commits onto your branch; tell me if you would rather have them as a separate PR.

What I changed, and the evidence

  1. Admission was back on the async worker. check_v1_session_admission called session_exists (an open_reader + SELECT count(*)) inline on every v1 prompt — and it had replaced the run_blocking call prompt_async already had, so the PR was a net loss of offload. It is now async and routes the query through run_blocking. This is the F1 convoy class the repo already fixed once; five routes were affected.

  2. The delete wait needed TWO bounds. Your cancel_drain waits on completion_rx.changed(); I bounded that, then wrote a test with a fabricated registry entry that never confirms, and it still hung for 90 s. Reason: cancel_drains_for_delete is a loop that re-cancels, so bounding only the inner call just pays the budget repeatedly. Both loops now share one deadline (OCSERVE_DRAIN_CANCEL_SECS, default 30 s) and record ocserve_v2_drain_cancel_timeout_total. The test is delete_returns_when_a_registered_drain_never_confirms_cancellation; its negative control (remove the outer deadline) hangs to the timeout.

  3. Capacity detection was string matching. error.message == "too many concurrent prompts" — a wording change would silently drop into the 1 Hz retry branch while holding a drain slot. Now matched on status + name.

  4. The scheduler awaited capacity in its serial loop. One slow drain stalled every other schedule, for every session. It now uses the non-blocking reservation and returns without marking the row, so the next pass retries.

  5. unwrap_or(false) on the interrupt re-arm. A read error became "session gone" and the re-armed steer work was dropped silently. It propagates now.

On the MEMORY claim: I rewrote it as an explicit count bound with the real worst case, min(128 slots, 64 prompt locks) x history, because MAX_PROMPT_LOCKS is what actually caps concurrent turns.

One contract note, recorded not fixed: /api/session/{id}/interrupt is in both contracts — v1 says 204, v2 says 200 — and it is not in the v2-only table, so a v2-disabled server now answers 200 where its v1 contract says 204. No shipped v1 client calls it (1.19.5's v1 SDK uses /session/{id}/abort), so I logged it as D-V2-INTERRUPT-STATUS rather than splitting the handler. If you want the split, say so.

The 1,400-line inline test module stays: integration tests cannot see the #[cfg(test)] seams, so moving it would mean rewriting six race tests. Reviewing the seams is the better trade.

The new knob was read in code but absent from the env contract, so the guard
that exists to catch exactly this failed CI rather than the build passing
with an undocumented knob. Group is 'Cgroups and lifecycle' (drain teardown),
default 30 s, 0 = the old unbounded wait. docs/ENVIRONMENT.md regenerated.

Already covered by K-OFFLOAD-ADMISSION in TRACEABILITY.md; this is the
documentation half of the same change.
…ot a latency delta

The point lookup measures 2.1 us, so no p50 improvement is claimed; the
evidence is behavioural (the delete regression test). Recording this stops a
future reader from inferring a perf win that was never measured.
Kept both sides in CHANGELOG, PLAN section 18 and TESTING 1.6: the permission
rows from Reliary#40 and the drain/delete rows from this branch are independent.
@alderpath
alderpath merged commit 4580085 into Reliary:main Nov 14, 2026
15 checks passed
alderpath added a commit to jrzmurray/ocserve that referenced this pull request Nov 14, 2026
The Reliary#41 branch made check_v1_session_admission async; main's Reliary#42 branch added
a call site that was written against the sync signature. Caught by
check-matrix (workspace must build before the test-existence check runs),
which is exactly what that fail-closed behaviour is for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants