Skip to content

XERK-261: cap the per-host queued-command queue so a flood can't 413 a host out of the fleet - #618

Merged
xerhab merged 1 commit into
mainfrom
XERK-261
Sep 1, 2026
Merged

XERK-261: cap the per-host queued-command queue so a flood can't 413 a host out of the fleet#618
xerhab merged 1 commit into
mainfrom
XERK-261

Conversation

@xerhab

@xerhab xerhab commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

XERK-261: cap the per-host queued-command queue

Fixes XERK-261 (qa-found, pre-existing, medium).

The bug

queueCommand(key, cmd) in turma/server.js appended to a host's a.commands with no bound. Every append re-serializes and SSE-broadcasts the whole agent record, and the queue drains only when the host heartbeats. So an operator hammering any command-queuing route (the ticket's repro: ~1316 queued model-source commands) against an offline host piled commands up until the record crossed AGENT_RECORD_MAX (8 MiB) — after which that host's own heartbeats are 413'd forever, because the drain needs a beat and the beat is what gets refused. Records live 7 days, so it is a durable self-lockout (needs the hub user's own creds, so self-DoS, not privilege).

The fix (all in queueCommand)

Bound the queue after the push, oldest dropped first (a queue thousands deep for an offline host is stale by the time it returns; dropping preserves every caller's cmdId return contract, which a 429-refuse would break):

  • Count cap AGENT_COMMAND_QUEUE_MAX (256, positiveEnv-overridable) — holds the common small-command flood with a length check, no serialization.
  • Byte trim (agentRecordSize(a) > AGENT_RECORD_MAX) — the general invariant the ticket asked for: no hub-side write leaves a record past the ceiling. Covers fat payloads (a spawn label is up to 100k) and a record already near the ceiling that one more command would tip over. Always keeps the just-enqueued command.
  • The single agentRecordSize measure doubles as the recordBytes update, keeping the aggregate registry budget honest for an offline host that never re-measures on its own beat.
  • Trim events log through logCommandTrim, throttled one line/min (same discipline as logRegistryFull — the flood that trips it is the traffic that would flood the log).

queueCommand is a low-frequency path (operator actions, ≤ one ticket dispatch per host per beat), so one serialization per call is cheap on a normal (<0.3 MiB) record.

Server-only; no client-facing contract change, so no Android/glasses parity work applies. Docs: new "queued-command cap (XERK-261)" section in .claude/rules/turma-limits.md.

How it was verified

  • Unit: two new queued-command cap (XERK-261) cases in server.test.js (count cap drops oldest + host still beats; fat payloads never grow the record past the ceiling). Full suite 702/0.
  • Adversarial QA — PASS, end-to-end against a real booted hub with an A/B control:
    • Flood of 200 × ~100k-label spawns at an offline host → queue held at 83, served record 7.92 MiB (< 8 MiB), oldest dropped / newest kept. Small-command flood (600) → held at exactly 256, newest kept.
    • Post-flood heartbeat returns 200 on the fixed hub. On parent commit 1fcdca1 the identical flood grew the record to 19.08 MiB and the next beat returned 413 agent record too large — the lockout reproduced, then shown fixed.
    • Legitimate low-volume queuing unaffected (3 commands ride the next beat in order; acks drain them).
    • registry-cap / registry-restore / cache-budget suites 36/0. No feature or security defects.
    • Accepted residuals (both unreachable via HTTP, neither a regression): a single command large enough to leave the record over-ceiling after byte-trim requires a base record already ~8 MiB (self-locks regardless); byte-trimming a tracked cmdId (migration/spawnTicket/history) needs 200+ fat commands flooded to one host faster than it drains — those cmdIds are only queued to online hosts, which drain one/beat and never approach the caps.

…a host out of the fleet

queueCommand appended to a host's command queue with no bound. Every append
re-serializes and SSE-broadcasts the whole record, and the queue only drains
when the host heartbeats — so an operator hammering a control (e.g.
/model-source) against an OFFLINE host piled up commands until the record
crossed AGENT_RECORD_MAX, after which that host's own heartbeats 413 forever
(the drain needs a beat, and the beat is what gets refused). Records live 7
days, so the lockout is durable self-DoS.

Bound the queue in queueCommand, oldest dropped first:
 - AGENT_COMMAND_QUEUE_MAX (256, positiveEnv): the common small-command flood,
   held with a length check.
 - a byte trim (agentRecordSize > AGENT_RECORD_MAX): the general invariant the
   ticket asks for — no hub-side write leaves a record past the ceiling —
   covering fat payloads (a spawn label is up to 100k) and a record already near
   the ceiling that one more command would tip over. Always keeps the
   just-enqueued command.

The single measure doubles as the recordBytes update, keeping the aggregate
budget honest for an offline host that never re-measures on its own beat. Trim
log is throttled one line/min like logRegistryFull.

Tests: two queued-command-cap cases in server.test.js (count cap drops oldest
and the host still beats; fat payloads never grow the record past the ceiling).
@xerhab
xerhab merged commit 72bdb8c into main Sep 1, 2026
5 checks passed
@xerhab
xerhab deleted the XERK-261 branch September 1, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant