Conversation
…a host out of the fleet queueCommand appended to a host's command queue with no bound. Every append re-serializes and SSE-broadcasts the whole record, and the queue only drains when the host heartbeats — so an operator hammering a control (e.g. /model-source) against an OFFLINE host piled up commands until the record crossed AGENT_RECORD_MAX, after which that host's own heartbeats 413 forever (the drain needs a beat, and the beat is what gets refused). Records live 7 days, so the lockout is durable self-DoS. Bound the queue in queueCommand, oldest dropped first: - AGENT_COMMAND_QUEUE_MAX (256, positiveEnv): the common small-command flood, held with a length check. - a byte trim (agentRecordSize > AGENT_RECORD_MAX): the general invariant the ticket asks for — no hub-side write leaves a record past the ceiling — covering fat payloads (a spawn label is up to 100k) and a record already near the ceiling that one more command would tip over. Always keeps the just-enqueued command. The single measure doubles as the recordBytes update, keeping the aggregate budget honest for an offline host that never re-measures on its own beat. Trim log is throttled one line/min like logRegistryFull. Tests: two queued-command-cap cases in server.test.js (count cap drops oldest and the host still beats; fat payloads never grow the record past the ceiling).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
XERK-261: cap the per-host queued-command queue
Fixes XERK-261 (qa-found, pre-existing, medium).
The bug
queueCommand(key, cmd)inturma/server.jsappended to a host'sa.commandswith no bound. Every append re-serializes and SSE-broadcasts the whole agent record, and the queue drains only when the host heartbeats. So an operator hammering any command-queuing route (the ticket's repro: ~1316 queuedmodel-sourcecommands) against an offline host piled commands up until the record crossedAGENT_RECORD_MAX(8 MiB) — after which that host's own heartbeats are 413'd forever, because the drain needs a beat and the beat is what gets refused. Records live 7 days, so it is a durable self-lockout (needs the hub user's own creds, so self-DoS, not privilege).The fix (all in
queueCommand)Bound the queue after the push, oldest dropped first (a queue thousands deep for an offline host is stale by the time it returns; dropping preserves every caller's
cmdIdreturn contract, which a 429-refuse would break):AGENT_COMMAND_QUEUE_MAX(256,positiveEnv-overridable) — holds the common small-command flood with a length check, no serialization.agentRecordSize(a) > AGENT_RECORD_MAX) — the general invariant the ticket asked for: no hub-side write leaves a record past the ceiling. Covers fat payloads (a spawnlabelis up to 100k) and a record already near the ceiling that one more command would tip over. Always keeps the just-enqueued command.agentRecordSizemeasure doubles as therecordBytesupdate, keeping the aggregate registry budget honest for an offline host that never re-measures on its own beat.logCommandTrim, throttled one line/min (same discipline aslogRegistryFull— the flood that trips it is the traffic that would flood the log).queueCommandis a low-frequency path (operator actions, ≤ one ticket dispatch per host per beat), so one serialization per call is cheap on a normal (<0.3 MiB) record.Server-only; no client-facing contract change, so no Android/glasses parity work applies. Docs: new "queued-command cap (XERK-261)" section in
.claude/rules/turma-limits.md.How it was verified
queued-command cap (XERK-261)cases inserver.test.js(count cap drops oldest + host still beats; fat payloads never grow the record past the ceiling). Full suite 702/0.labelspawns at an offline host → queue held at 83, served record 7.92 MiB (< 8 MiB), oldest dropped / newest kept. Small-command flood (600) → held at exactly 256, newest kept.1fcdca1the identical flood grew the record to 19.08 MiB and the next beat returned 413agent record too large— the lockout reproduced, then shown fixed.registry-cap/registry-restore/cache-budgetsuites 36/0. No feature or security defects.cmdId(migration/spawnTicket/history) needs 200+ fat commands flooded to one host faster than it drains — those cmdIds are only queued to online hosts, which drain one/beat and never approach the caps.