Skip to content

fix: restart resumes the process loop and the pulse emitter - #90

Open
jbiskur wants to merge 2 commits into
mainfrom
fix/restart-resumes-process-loop
Open

fix: restart resumes the process loop and the pulse emitter#90
jbiskur wants to merge 2 commits into
mainfrom
fix/restart-resumes-process-loop

Conversation

@jbiskur

@jbiskur jbiskur commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Problem

restart() calls stop(true), which clears running, empties the buffer and stops the pulse emitter. The fetch loop revives itself from restartTo:

this.running = true
return this.loop()

But the process loop is a while (this.running) loop and only start() called startProcessLoop(). A pump restarted while it was delivering keeps pulling events and never delivers or checkpoints again — and stops pulsing, so it looks dead to the control plane while its host still reports healthy.

This is the defect written up on 2026-04-14 in Usable fragment 28875215-003d-4770-9d44-8e3488d25246.

Production impact

It stalled two data-pathways pathways for 26 days (2026-08-02 → 2026-08-28):

Pathway Tenant Effect
417b8dd9-b6a3-4ce9-b254-6ea12ca97d57 memory-mesh 26-day backlog, drained in 2.4 h at 98% DB CPU
5a32bd1c-9d0c-4aef-a813-4fee60280f8e flowcore-saas nothing delivered, assignment reported active

A scoped replay asked for one flow type; the consumer restarted every source, and every restarted pump went quiet.

Change

  • ensureProcessLoop() starts the delivery loop whenever the pump is running with a processor and no loop is live. The fetch loop calls it once per iteration, so a loop that exits for any reason comes back within one iteration.
  • processLoopRunning guards against a second loop when the existing one is only parked in reserve() — two loops would race over the same buffer.
  • A loop that exits while the pump is already running again (the restart landed mid-batch) restarts itself.
  • The restart path calls pulseEmitter.start() again. It is already idempotent.

Tests

Three new cases in data-pump-restart.test.ts, all of which fail on the previous behaviour:

  1. delivery continues after a restart() issued from inside the handler
  2. no second process loop when one is only parked (an event is delivered once, not twice)
  3. the pulse emitter is started again

Suite: 57 pass / 0 fail. tsc --noEmit and prettier --check clean.

🤖 Generated with Claude Code

jbiskur and others added 2 commits June 12, 2026 10:33
`restart()` calls `stop(true)`, which clears `running`, empties the buffer and
stops the pulse emitter. The fetch loop revives itself from `restartTo`, but the
process loop exits on `running` and only `start()` brought it back. A pump
restarted while it was delivering kept pulling events and never delivered or
checkpointed again — and stopped pulsing, so it looked dead to the control
plane while its host reported healthy.

This is the defect described on 2026-04-14 in Usable fragment
28875215-003d-4770-9d44-8e3488d25246. It stalled two production data-pathways
pathways for 26 days (2026-08-02 → 2026-08-28):
417b8dd9-b6a3-4ce9-b254-6ea12ca97d57 and 5a32bd1c-9d0c-4aef-a813-4fee60280f8e.

- `ensureProcessLoop()` starts the delivery loop whenever the pump is running
  with a processor and no loop is live. The fetch loop calls it every
  iteration, so a loop that exits for any reason comes back within one
  iteration instead of leaving a pump that pulls but never delivers.
- `processLoopRunning` guards against a second loop when the existing one is
  only parked in `reserve()`. Two loops would race over the same buffer.
- A loop that exits while the pump is running again — the restart landed
  mid-batch — restarts itself.
- The restart path starts the pulse emitter again. `PulseEmitter.start()` is
  already idempotent.

Tests: delivery continues after a restart issued from inside the handler; no
second loop when one is parked; the pulse emitter is started again. All three
fail on the previous behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant