Skip to content

Stop services with SIGTERM before escalating to SIGKILL #35

Description

@tylerhunt

Services are stopped with a bare SIGKILL. There's no SIGTERM grace
period, so nothing Bates manages ever gets to shut down cleanly.

source/lib/bates/app.ex:568:

defp stop_service(state, service_name, service_state) do
  pid = service_state.pid
  :exec.kill(pid, :sigkill)

  ref = Process.monitor(pid)

  receive do
    {:DOWN, ^ref, :process, ^pid, _reason} -> :ok
  after
    5_000 -> :ok
  end

This is the only stop path — bates down, the dashboard stop button,
and restart all route through it.

Why it matters

  • Rails/Puma never runs its shutdown hooks. In-flight requests are
    dropped and PID files are left behind.
  • PostgreSQL gets no chance at a clean shutdown, so the next start
    runs crash recovery. The addon already carries a workaround for the
    fallout: source/lib/bates/addons/postgresql.ex:32-34 removes a stale
    postmaster.pid when no live process holds it. That prologue exists
    because of this bug.
  • Anything holding a lock, a socket, or a temp file leaks it on every
    stop.

Proposal

Send SIGTERM, wait a grace period, then escalate to SIGKILL. The
monitor plumbing is already in place at app.ex:570-575 — it just waits
on the wrong signal. A 5-second default is probably too short for
Postgres under load; 10 seconds is a better starting point, and it's
worth considering whether the grace period should be configurable per
service in the TOML.

Also needs checking as part of this: :exec.kill/2 targets the pid of
the shell wrapper that ProcessInvocation.compile/1 builds, not the
application process it execs. The exec in the compiled command means
the shell replaces itself, so the signal should reach the right process
— but this should be verified for the multi-line prologue case
(asdf, direnv, and the Postgres addon all prepend prologue lines),
and confirmed to reach grandchildren such as Puma workers.

Acceptance

  • A service that traps SIGTERM and exits cleanly is reported as down,
    not crashed.
  • A service that ignores SIGTERM is still killed after the grace
    period.
  • PostgreSQL shuts down cleanly enough that the following start doesn't
    run recovery.
  • The stale-postmaster.pid prologue in the addon can be evaluated for
    removal once this lands (it may still be worth keeping for the case
    where the daemon itself is killed).

Flagged while evaluating a possible Go rewrite (see REWRITE.md). Worth
fixing here rather than reproducing the behavior in a port.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions