fix(sandbox): a dev server killed while running crashed, it did not fail to start

[pedrofrxncx]

Root cause

orchestrator.go OnTaskExit mapped every non-zero, unintentional dev-script exit to PhaseStartFailed — regardless of the phase it happened in.

restartablePhase (main.go:406) allows only running | starting | crashed. So start-failed permanently disarms devwatch, and nothing else calls RestartDev. OnTaskExit also sets status.State = "error", and stepStart skips starting unless status is running — so a config re-apply won't revive it either. The sandbox stays dark for the life of the pod while still reporting ready.

Why it fires constantly

An agent has no endpoint to restart the dev server — RestartDev is called only by devwatch. So it uses pkill, which it legitimately must to pick up discovery.config.js or FASTSTORE_THEME (both read once at boot). yarn reports the signal as exit code 1, and the daemon reads a deliberate restart of a perfectly healthy server as a failure to start.

Observed live on tenant-electrolux-prod-jg6qk:

[probe] server responded on port 3000 (status 200)
error Command failed with signal "SIGTERM"
task exit  task=dev status=exited exit_code=1
[orchestrator] dev script exited with code 1
lifecycle transition  from=running to=start-failed

At the time of writing that pod — and 3 of the other 7 tenant-electrolux-prod-* pods — had nothing listening on 3000. Two prod runs (ELEC-244, ELEC-245) then spent 25 of 55 and ~15 of 82 tool calls killing, rebuilding and re-polling the dev server by hand, next to a watchdog that had been switched off.

Change

DevExitPhase(current) — extracted so the decision is unit-testable:

exited from	phase	restartable?
running	crashed	yes — devwatch respawns
crashed	crashed	yes — devwatch owns the bound
starting	start-failed	no — it really did fail to start
anything else	start-failed	no

Recovery chain verified end to end in code: devwatch → ActionRestart → RestartDev → clearCrashError (error→running) → stopDevTask → enqueue(StepStart), whose status guard now passes.

crashed must re-enter itself rather than latch, or one retry would disarm the watchdog and defeat MaxRestarts — repeated failures stay bounded by devwatch's own ActionGiveUp, which transitions to start-failed deliberately.

Testing

internal/setup/devexit_test.go pins the table above. go build ./..., go vet ./..., gofmt -l clean; setup, devwatch, lifecycle suites pass.

Relation to #7023

#7023 makes the probe distinguish "answering 5xx" from "healthy". It is not the fix for this — the watchdog is disarmed before the probe's opinion is ever consulted. This PR is the one that matters; #7023 is a complementary net and can be merged, deferred, or closed independently.

---

Summary by cubic

Fixes the sandbox dev server lifecycle so a dev process killed while running is treated as a crash (restartable) instead of a startup failure, which permanently disarmed the watchdog and left the sandbox dark.

• Exits from running or crashed now transition to crashed; exits from starting still become start-failed.
• start-failed is not restartable, so misclassifying a running server's death as a startup failure disabled devwatch for the pod's lifetime.
• Repeated failures stay bounded by devwatch's MaxRestarts and ActionGiveUp because crashed re-enters itself.

^{Written for commit b36713f. Summary will update on new commits.}