Two small findings from running the brainstem on a box where a pulse costs more than the default interval.
1. The default cadence can run the pulse back-to-back
The loop sleeps max(5.0, PULSE_SEC - elapsed), so a pulse can never overlap the next one, but when
the pulse's own work exceeds PULSE_SEC, the effective period becomes duration + 5s and the daemon
runs essentially continuously, with only a 5-second idle gap.
Observed on this machine (Arch, kernel 6.18-lts, ollama serving embeddings on CPU):
-
SIA_PULSE_SEC=60 (the unit's default): load average 2.44, llama-server pinned at 60.5% CPU,
sustained for hours. Pulses fired back-to-back.
-
Deriving the cost from the after-fix duty cycle: at 900 s the load settled to 0.40 and
llama-server to ~10% idle. Duty went from ~0.97 to ~0.16, so a single pulse costs roughly 145 s
of work (this figure is derived, not directly measured, because the pulse does not log its own duration).
-
At 900 s the pulses land at a clean 15-minute cadence:
11:11:36 pulse 13621: 614 events, state thinking
11:26:35 pulse 13622: 614 events, state thinking
11:41:35 pulse 13623: 609 events, state thinking
11:56:35 pulse 13624: 607 events, state thinking
12:11:31 pulse 13625: 604 events, state thinking
Local mitigation (not a proposal for the default): a systemd drop-in setting
Environment=SIA_PULSE_SEC=900. The knob is read at sia-brainstem.py:53
(PULSE_SEC = int(os.environ.get("SIA_PULSE_SEC", "60"))), so it is intended; it is just not
something an operator would know to look for, and 60 s is aggressive for a machine where a pulse
costs minutes.
Ask: log the pulse's own duration and warn (or back off) when it exceeds the interval. A pulse
that cannot finish inside its period is invisible today, because the log line only reports the event count.
2. The brainstem ingests its own log
cursors.json carries a source cursor named sia.lines, pointing at the brainstem's own log file, which is the file each pulse appends its summary line to. So every pulse manufactures at least one new record
for the next pulse to consider, and sync_needed can be satisfied by the daemon's own output rather
than by anything that changed on the machine. Current offset on this box is 13,718 bytes (a fresh log;
it grows with uptime).
It is not the dominant cost (the pulse's embedding work in item 1 is), but it is a feedback loop in a
system whose whole job is tracking what actually happened, and it is cheap to close by excluding the
brainstem's own log from the source list.
What we ruled out
Reported so nobody re-walks them:
- A refused record wedging the cursor. Hypothesis was that a record the tail refuses to consume
blocks the cursor forever. Measured: zero refusals or invalid records in 24 h of journal output.
sync_needed stuck true. The key is absent from memo.json, so .get("sync_needed", False)
reads False. It is not stuck.
The tail machinery itself (sialib._stable_tail_chunk) looks careful: device/inode fingerprints,
rotation detection, bounded reads, per-record refusal handling. No complaint there.
Environment
Rig, Arch, kernel 6.18-lts, SIA 1.7.3. Corpus 104 files / 6.4 MB, journal at part-11.
The pulse transaction lives in sialib._pulse_transaction.
Two small findings from running the brainstem on a box where a pulse costs more than the default interval.
1. The default cadence can run the pulse back-to-back
The loop sleeps
max(5.0, PULSE_SEC - elapsed), so a pulse can never overlap the next one, but whenthe pulse's own work exceeds
PULSE_SEC, the effective period becomesduration + 5sand the daemonruns essentially continuously, with only a 5-second idle gap.
Observed on this machine (Arch, kernel 6.18-lts, ollama serving embeddings on CPU):
SIA_PULSE_SEC=60(the unit's default): load average 2.44,llama-serverpinned at 60.5% CPU,sustained for hours. Pulses fired back-to-back.
Deriving the cost from the after-fix duty cycle: at 900 s the load settled to 0.40 and
llama-serverto ~10% idle. Duty went from ~0.97 to ~0.16, so a single pulse costs roughly 145 sof work (this figure is derived, not directly measured, because the pulse does not log its own duration).
At 900 s the pulses land at a clean 15-minute cadence:
Local mitigation (not a proposal for the default): a systemd drop-in setting
Environment=SIA_PULSE_SEC=900. The knob is read atsia-brainstem.py:53(
PULSE_SEC = int(os.environ.get("SIA_PULSE_SEC", "60"))), so it is intended; it is just notsomething an operator would know to look for, and 60 s is aggressive for a machine where a pulse
costs minutes.
Ask: log the pulse's own duration and warn (or back off) when it exceeds the interval. A pulse
that cannot finish inside its period is invisible today, because the log line only reports the event count.
2. The brainstem ingests its own log
cursors.jsoncarries a source cursor namedsia.lines, pointing at the brainstem's own log file, which is the file each pulse appends its summary line to. So every pulse manufactures at least one new recordfor the next pulse to consider, and
sync_neededcan be satisfied by the daemon's own output ratherthan by anything that changed on the machine. Current offset on this box is 13,718 bytes (a fresh log;
it grows with uptime).
It is not the dominant cost (the pulse's embedding work in item 1 is), but it is a feedback loop in a
system whose whole job is tracking what actually happened, and it is cheap to close by excluding the
brainstem's own log from the source list.
What we ruled out
Reported so nobody re-walks them:
blocks the cursor forever. Measured: zero refusals or invalid records in 24 h of journal output.
sync_neededstuck true. The key is absent frommemo.json, so.get("sync_needed", False)reads
False. It is not stuck.The tail machinery itself (
sialib._stable_tail_chunk) looks careful: device/inode fingerprints,rotation detection, bounded reads, per-record refusal handling. No complaint there.
Environment
Rig, Arch, kernel 6.18-lts, SIA 1.7.3. Corpus 104 files / 6.4 MB, journal at
part-11.The pulse transaction lives in
sialib._pulse_transaction.