Services are stopped with a bare SIGKILL. There's no SIGTERM grace
period, so nothing Bates manages ever gets to shut down cleanly.
source/lib/bates/app.ex:568:
defp stop_service(state, service_name, service_state) do
pid = service_state.pid
:exec.kill(pid, :sigkill)
ref = Process.monitor(pid)
receive do
{:DOWN, ^ref, :process, ^pid, _reason} -> :ok
after
5_000 -> :ok
end
This is the only stop path — bates down, the dashboard stop button,
and restart all route through it.
Why it matters
- Rails/Puma never runs its shutdown hooks. In-flight requests are
dropped and PID files are left behind.
- PostgreSQL gets no chance at a clean shutdown, so the next start
runs crash recovery. The addon already carries a workaround for the
fallout: source/lib/bates/addons/postgresql.ex:32-34 removes a stale
postmaster.pid when no live process holds it. That prologue exists
because of this bug.
- Anything holding a lock, a socket, or a temp file leaks it on every
stop.
Proposal
Send SIGTERM, wait a grace period, then escalate to SIGKILL. The
monitor plumbing is already in place at app.ex:570-575 — it just waits
on the wrong signal. A 5-second default is probably too short for
Postgres under load; 10 seconds is a better starting point, and it's
worth considering whether the grace period should be configurable per
service in the TOML.
Also needs checking as part of this: :exec.kill/2 targets the pid of
the shell wrapper that ProcessInvocation.compile/1 builds, not the
application process it execs. The exec in the compiled command means
the shell replaces itself, so the signal should reach the right process
— but this should be verified for the multi-line prologue case
(asdf, direnv, and the Postgres addon all prepend prologue lines),
and confirmed to reach grandchildren such as Puma workers.
Acceptance
- A service that traps
SIGTERM and exits cleanly is reported as down,
not crashed.
- A service that ignores
SIGTERM is still killed after the grace
period.
- PostgreSQL shuts down cleanly enough that the following start doesn't
run recovery.
- The stale-
postmaster.pid prologue in the addon can be evaluated for
removal once this lands (it may still be worth keeping for the case
where the daemon itself is killed).
Flagged while evaluating a possible Go rewrite (see REWRITE.md). Worth
fixing here rather than reproducing the behavior in a port.
Services are stopped with a bare
SIGKILL. There's noSIGTERMgraceperiod, so nothing Bates manages ever gets to shut down cleanly.
source/lib/bates/app.ex:568:This is the only stop path —
bates down, the dashboard stop button,and restart all route through it.
Why it matters
dropped and PID files are left behind.
runs crash recovery. The addon already carries a workaround for the
fallout:
source/lib/bates/addons/postgresql.ex:32-34removes a stalepostmaster.pidwhen no live process holds it. That prologue existsbecause of this bug.
stop.
Proposal
Send
SIGTERM, wait a grace period, then escalate toSIGKILL. Themonitor plumbing is already in place at
app.ex:570-575— it just waitson the wrong signal. A 5-second default is probably too short for
Postgres under load; 10 seconds is a better starting point, and it's
worth considering whether the grace period should be configurable per
service in the TOML.
Also needs checking as part of this:
:exec.kill/2targets the pid ofthe shell wrapper that
ProcessInvocation.compile/1builds, not theapplication process it
execs. Theexecin the compiled command meansthe shell replaces itself, so the signal should reach the right process
— but this should be verified for the multi-line prologue case
(
asdf,direnv, and the Postgres addon all prepend prologue lines),and confirmed to reach grandchildren such as Puma workers.
Acceptance
SIGTERMand exits cleanly is reported asdown,not
crashed.SIGTERMis still killed after the graceperiod.
run recovery.
postmaster.pidprologue in the addon can be evaluated forremoval once this lands (it may still be worth keeping for the case
where the daemon itself is killed).
Flagged while evaluating a possible Go rewrite (see
REWRITE.md). Worthfixing here rather than reproducing the behavior in a port.