From 646e692551b78a984c3efd09c56be1f4502b9c6e Mon Sep 17 00:00:00 2001 From: Luis Guzman Date: Fri, 4 Sep 2026 23:51:01 -0600 Subject: [PATCH 1/8] K2GO-386 feat(dashboard): proot-owned log rotation triggered by dash-node MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit proot has no systemd/cron, so /etc/cron.daily/logrotate never runs — logrotate is installed but nothing triggers it, and a service log (php-fpm, dash-node) can grow until the device hits ENOSPC. Own our rotation instead of the RPi-oriented inherited config: - tools/setup-proot-logging.sh: idempotent configurator. Moves the inherited nginx/php-fpm snippets out of /etc/logrotate.d (we override them; a duplicate path fails logrotate), installs /etc/logrotate.d/k2go (copytruncate + size-based, so no daemon reopen signal is needed under proot and a file that grows is rotated regardless of schedule), covering php-fpm, nginx, calibre-web, dash-node; validates with logrotate -d. kiwix writes no log; kolibri self-rotates — both left alone. - dash-node (static/dashboard): startLogRotation() installs the config once at startup (light: mv + write + validate, no rotation) and triggers logrotate every 10 min. No rotation at boot (first pass at +10 min), keeping the fragile boot moment clear. - ADR-386 documents the full disk-fill containment strategy and this layer. Typecheck/tests run in the on-device dashboard build (tsc + smoke test). --- .../ADR-386-unattended-disk-containment.md | 173 ++++++++++++++++++ static/dashboard/server.ts | 4 + static/dashboard/sockets/log-rotate.ts | 63 +++++++ tools/setup-proot-logging.sh | 84 +++++++++ 4 files changed, 324 insertions(+) create mode 100644 controller/docs/ADR-386-unattended-disk-containment.md create mode 100644 static/dashboard/sockets/log-rotate.ts create mode 100644 tools/setup-proot-logging.sh diff --git a/controller/docs/ADR-386-unattended-disk-containment.md b/controller/docs/ADR-386-unattended-disk-containment.md new file mode 100644 index 000000000..e29f55a2b --- /dev/null +++ b/controller/docs/ADR-386-unattended-disk-containment.md @@ -0,0 +1,173 @@ +# ADR-386 — Unattended disk-fill containment & maintainability (prevent · contain logs · reap-restart from the app) + +**Status:** proposed — design only, no production code. Lands as small diffs a maintainer verifies. This +is the single home for the K2GO-386 disk-fill work; if a layer's detail grows it splits into a lettered +delta (`ADR-386a`, `ADR-386b`) rather than a separate ADR. + +**Scope note (form):** genericized per the ADR authoring convention — no personal names, no device +identifiers. "the box" = the Debian userland under proot; "the device" = the Android target; "upstream" += Internet-in-a-Box (iiab/iiab). + +--- + +## 1. Why this exists (the stakes, stated plainly) + +A single misbehaving box process can **saturate the device's disk in hours** — faster on a faster phone +— until it hits `ENOSPC`. When that happens the server breaks *and the user's phone is left full and +unusable for anything else*. We have measured this class directly: an orphaned `php-fpm` busy-looping +into its log at ~600 MB–1.3 GB/min, and unbounded service logs reaching ~1.8 GB each with nothing ever +trimming them. + +Our users are **not technical**. The product's promise is a server that just works on a phone. So the +goal of this effort is not to detect-and-blame; it is to make the system **robust and simple to +administer**: + +- **resolve these situations unattended**, through **parameters** (thresholds/intervals we can tune), not + through a human running commands on the box; +- **keep the system alive** — recover by containing or restarting, not by denying service and walking + away (a monitor that spends battery only to leave the phone stopped in a bad state is worse than none); +- **report to the developers** the situations that warrant attention, so we learn what happens in the + field without the user having to notice or act. + +Everything below serves that goal. This is a maintainability effort as much as a bug fix. + +## 2. The problem, as facts + +Three failure shapes, all the same root class (a box process consuming disk faster than anything reclaims +it), plus the environment that makes them hard: + +- **Vector A — a runaway *real* log file:** visible on disk, grows without bound. +- **Vector B — deleted-but-open:** the process holds an *unlinked* fd; disk is consumed but there is **no + file** to find or rotate. Only **closing the fd (stop/restart the process)** frees it. +- **Speed:** the php busy-loop is a *firehose* (≫ any gentle rotation interval); most logs are a slow drip. + +Environment facts that shape every decision: + +- **proot has no systemd and no running cron** → `/etc/cron.daily/logrotate` **never runs**; logrotate is + installed but never triggered (root cause of the unbounded logs). +- **Upstream configs assume a Raspberry Pi** (systemd, always-on, cron, signal-based `postrotate`). Well + made for that world; unusable in ours. +- **An orphaned service is off proot's ptrace**, so an in-box healer cannot reach it — device-proven, only + an **outside-the-rootfs** actor (the Android app) can stop it. This is why the last layer lives in the app. + +## 3. The strategy — three layers, one home + +| Layer | Owns | Failure it handles | Where it runs | +|---|---|---|---| +| **L1 Prevent** | php-fpm does not run idle | removes the *cause* of the known orphan-loop on a default build | rootfs (ansible/patch) | +| **L2 Contain logs** | bounded, proot-correct rotation | Vector A, steady/moderate growth | in-box (dash-node-triggered) | +| **L3 App backstop** | reap + **restart to keep alive**, and report | Vector B + the fast firehose; the net for anything L2 misses | Android app (outside rootfs) | + +They are complementary, not redundant: L1 stops one cause, L2 bounds ordinary growth cheaply, L3 is the +outside net for the acute and the invisible. Reporting (§7) spans all three. + +## 4. Layer 1 — Prevent (php-fpm not idle) + +The idle `php-fpm` enabled unconditionally by the nginx role is the orphan that busy-loops. We move +php-fpm ownership to the roles that use it (Matomo enables its own), so a default build never runs it → +it cannot orphan. Carried as `tools/upstream-patches/0002-php-fpm-role-ownership.patch` (WIP upstream); +PR #543. **Note:** the rootfs build sources patches from `main`, so this takes effect only once merged +to `main` — a branch bake still reads `main`'s patches (verified). + +## 5. Layer 2 — Contain logs (K2Go owns its proot logging) + +We **own a centralized, proot-correct log-rotation config set** for the services we run, discarding the +RPi-oriented inherited snippets (learn from them, write ours): + +- **`copytruncate`, never a postrotate signal** — proot cannot drive `systemctl`/`invoke-rc.d`, and a + failed reopen is exactly Vector B. copytruncate truncates in place; the writer keeps its fd. +- **Size-based** (`size 100M`, `rotate 3`, `compress`, `delaycompress`, `missingok`, `notifempty`, + `su root root`) — our threat is a file that grows, not a calendar. +- **Trigger = dash-node** (the box's always-up process), **every 10 min, no run at boot** — boot is the + heaviest/most fragile moment (Python services starting; phantom-process-killer risk). Accepted tradeoff: + a <10-min session never rotates; the size cap + L3 bound it. + +**The config set (verified on-device against the pdsm wrappers + live `/var/log`):** + +| Service | Log | In the set | +|---|---|---| +| php-fpm | `/var/log/php8.4-fpm.log` | **yes** — override inherited snippet (add size cap, copytruncate) | +| nginx | `/var/log/nginx/*.log` | **yes** — override (drop `invoke-rc.d`, copytruncate) | +| calibre-web | `/var/log/calibre-web.log` (pdsm redirects here) | **yes** — added (`missingok`) | +| dash-node | `/var/log/dash-node.log`, `/var/log/dash-rebuild.log` | **yes** — added | +| kiwix | — none (`kiwix-serve --daemon`, no redirection) | **excluded** (no file to rotate) | +| kolibri | `/library/kolibri/logs/*.txt` (+ `archive/`) | **excluded** (self-managed, bounded ~64 KB) | + +**Install:** an idempotent `tools/setup-proot-logging.sh` (ansible-role-shaped) moves the inherited +nginx/php-fpm snippets out of `/etc/logrotate.d` (we override them; a duplicate path fails logrotate), +writes `/etc/logrotate.d/k2go`, and validates with `logrotate -d`. dash-node runs it **once at +startup** — this is light (a couple of `mv`s, one file write, a parse-only validate; **no rotation, no +Python, nothing that risks the phantom-process killer**), which is distinct from the rotation itself +(§4, on the timer, never at boot). Phase 2 bakes it into the rootfs as a role with a pdsm-owned trigger. + +## 6. Layer 3 — App-side backstop (reap + restart, not deny) + +The outside-the-rootfs net for what L2 cannot catch: the **fast firehose** (fills faster than a 10-min +rotation) and **Vector B** (no file to rotate; must stop the holding process). Device-proven: only the +app can stop an off-proot orphan. + +**Decisions (direction; detailed mechanism is the next design step, possibly `ADR-386a`):** + +- **Targeted, not blind.** Act on the offending vector (a specific runaway log/service), keeping the rest + of the system up. NOT the rejected "disk full → stop everything → stay down → user fixes it." +- **Restart to keep alive.** A fresh service under a fresh proot does not busy-loop, so the recovery is + **reap + reclaim + relaunch through the lifecycle owner** (the ADR-5343 reconciler / desired-state), so + the box comes back — unattended. Stopping-and-staying-down is a last resort only, never the default. +- **Parametric.** Thresholds and cadence are parameters we tune (floor, growth-rate, intervals), not + hard-coded cliffs; the primary directive right now is **disk free space / abnormal growth**. +- **Single surface.** Watching disk pressure / abnormal log growth catches *any* vector, not just php. + +**Open (design next):** the exact leading-indicator (growth-rate vs absolute), the targeted-vs-full +decision, and how relaunch coordinates with the reconciler. The existing `feat/K2GO-386-disk-guard` +slice (device-verified) is the scaffold to reorient from "stop + stay-down" to this. + +## 7. Reporting to developers (unattended, cadence-based) + +When any layer contains or recovers a situation, it should tell us — so field robustness is measured, not +guessed — **without bothering the user**: + +- **Optional and cadence/rate-based:** a rare anomaly → a low-cadence digest (daily/weekly/monthly); a + recurring one (e.g. hourly) → escalate ("we're noticing X; send a report?"). A frequency/rate rule sets + the cadence. +- **Reuse the delivery backbone** (`DeliveryManager` / the debug-delivery path) rather than a new channel. +- The user is *informed*, not *tasked*: a report goes to developers; recovery already happened. + +## 8. Lifecycle (who sets it, who clears it, what if a process dies) + +- **L2 config:** installed by dash-node at startup (idempotent — writes only when changed; the snippet + move is a no-op once done), re-asserted every dash-node start; no persistent marker to strand. +- **L2 trigger:** dash-node's 10-min timer; owns nothing but the timer. If dash-node dies, logs only grow + while services are up and the reconciler owns bring-up; if dash-node *wedges*, rotation stalls — the one + Phase-1 liveness dependency, bounded by L3, removed in Phase 2 (pdsm-owned trigger). +- **L3:** a poller for the life of a box-up session (started once, stopped on teardown); recovery routes + through the ADR-5343 desired-state owner, so no second source of "should the box be up." +- **State:** logrotate's own status file + `rotate N`/`compress` bound disk; nothing we add persists + unbounded. + +## 9. Forks considered and rejected + +- **Run cron/crond in the box** — Phase 1 rejects it (dash-node can schedule); revisit only for a + dash-node-independent trigger (Phase 2 pdsm). +- **Keep upstream snippets, just trigger them** — rejected: no size caps + proot-broken postrotate signals + (re-creates Vector B). +- **A light logrotate pass at boot** — rejected for now (boot lightness wins; cap + L3 bound the gap). +- **Bake L2 into the rootfs now** — deferred to Phase 2, not rejected. +- **L3 = stop-and-stay-down** — rejected: denies service to a non-technical user (the whole point of §1). + +## 10. Verification (per layer) + +| Layer | Check | Expected | +|---|---|---| +| L1 | rootfs built with the merged patch | php-fpm installed, **not enabled**, not running on a default build | +| L2 | `logrotate -d` after install; a log grown past `size`, then a trigger | parses clean; truncated in place; writer keeps writing; no orphaned deleted-but-open file | +| L2 | short (<10 min) session | no boot rotation (by design); bounded next long session | +| L3 | fast fill + Vector B (synthetic) | contained/recovered without denying service; system back up; a report enqueued | + +## 11. Consequences + +- Disk-fill is handled unattended across its vectors, and the phone is kept usable — matching §1. +- We carry a small owned config set instead of inherited RPi drift; new services get one block, one place. +- One liveness dependency remains in Phase 1 (dash-node as L2 trigger); named, bounded by L3, removed in + Phase 2. +- Field occurrences are reported to developers, so we tune the parameters from real data rather than + guesses. diff --git a/static/dashboard/server.ts b/static/dashboard/server.ts index feac9f5c9..42e15d4fe 100644 --- a/static/dashboard/server.ts +++ b/static/dashboard/server.ts @@ -11,6 +11,7 @@ import './sockets/books.exec'; import './sockets/kolibri.exec'; import { apiRouter } from './routes'; import { startServiceHeal } from './sockets/service-heal'; +import { startLogRotation } from './sockets/log-rotate'; const app = express(); const server = http.createServer(app); @@ -40,6 +41,9 @@ server.listen(PORT, '127.0.0.1', () => { try { jobs.reconcileOnBoot(); } catch (e) { console.error('[jobs] reconcile failed', e); } // ADFA-5343 (ADR-5343a §10): the box heals its own content-service tree in-proot. try { startServiceHeal(); } catch (e) { console.error('[service-heal] start failed', e); } + // K2GO-386 (ADR-386 §5): proot has no cron — dash-node triggers logrotate on a timer so the + // K2Go-owned rotation config actually runs (no rotation at boot; first pass at +10 min). + try { startLogRotation(); } catch (e) { console.error('[log-rotate] start failed', e); } }); // ========================================== diff --git a/static/dashboard/sockets/log-rotate.ts b/static/dashboard/sockets/log-rotate.ts new file mode 100644 index 000000000..028023985 --- /dev/null +++ b/static/dashboard/sockets/log-rotate.ts @@ -0,0 +1,63 @@ +// sockets/log-rotate.ts — K2GO-386 / ADR-386 (Layer 2: contain logs) +// +// proot has no systemd/cron, so nothing runs logrotate on its own. dash-node — the box's always-up +// process — installs the K2Go-owned rotation config once (light: it moves snippets + writes one file +// + validates; it does NOT rotate) and then triggers `logrotate` on a fixed timer so the config +// actually rotates. There is NO rotation at boot (setInterval fires first at +interval): boot is the +// heaviest, most fragile moment under proot and we keep it clear. Everything here is best-effort — a +// failure logs and the next tick retries; nothing throws into the caller. +import { execFile } from 'child_process'; +import * as path from 'path'; + +// Rotate every 10 min (ADR-386 §5). Wide enough to be nearly free; narrow enough to bound growth. +const INTERVAL_MS = 10 * 60 * 1000; + +// The configurator ships in the repo under tools/; on an installed box it lives in the clone that +// rebuild-dashboard.sh / dev-push-dashboard.sh keep current (default /opt/iiab-android). +const CLONE_DIR = process.env.K2GO_CLONE_DIR || '/opt/iiab-android'; +const CONFIGURATOR = path.join(CLONE_DIR, 'tools', 'setup-proot-logging.sh'); + +let timer: NodeJS.Timeout | null = null; + +// Install the rotation config once (idempotent, no rotation). Best-effort. +function installConfigOnce(): void { + execFile('sh', [CONFIGURATOR], { timeout: 30_000 }, (err, stdout, stderr) => { + if (err) { + console.error('[log-rotate] config install failed:', err.message, (stderr || '').trim()); + } else if ((stdout || '').trim()) { + console.log('[log-rotate] config:', stdout.trim()); + } + }); +} + +// Run one logrotate pass over the whole config (the K2Go blocks use copytruncate + size, so this is +// cheap unless a file actually exceeds its cap). Best-effort. +function runLogrotateOnce(): void { + execFile('logrotate', ['/etc/logrotate.conf'], { timeout: 60_000 }, (err, _stdout, stderr) => { + if (err) { + console.error('[log-rotate] logrotate run failed:', err.message, (stderr || '').trim()); + } + }); +} + +/** + * Start the periodic log-rotation trigger. Idempotent; call once from server.ts at listen time. + * Installs the config immediately (light), then rotates every INTERVAL_MS — the FIRST rotation is at + * +INTERVAL_MS, never at boot (ADR-386 §5). + */ +export function startLogRotation(): void { + if (timer) return; + installConfigOnce(); + timer = setInterval(runLogrotateOnce, INTERVAL_MS); + if (typeof timer.unref === 'function') timer.unref(); // don't keep the process alive just for this + console.log( + `[log-rotate] scheduled: logrotate every ${INTERVAL_MS / 60000} min (config installed; no rotation at boot)`, + ); +} + +export function stopLogRotation(): void { + if (timer) { + clearInterval(timer); + timer = null; + } +} diff --git a/tools/setup-proot-logging.sh b/tools/setup-proot-logging.sh new file mode 100644 index 000000000..4f9d33f32 --- /dev/null +++ b/tools/setup-proot-logging.sh @@ -0,0 +1,84 @@ +#!/bin/sh +# tools/setup-proot-logging.sh — K2GO-386 / ADR-386 (Layer 2: contain logs) +# +# Install K2Go's proot-correct log-rotation config for the box, idempotently. +# +# WHY: proot has no systemd and no running cron, so /etc/cron.daily/logrotate NEVER runs — logrotate +# is installed but nothing triggers it, and a service log can grow until the device hits ENOSPC. The +# inherited (Raspberry-Pi-oriented) snippets also lack size caps and use postrotate signals that fail +# under proot (a failed reopen = the deleted-but-open failure). We own our rotation instead. +# +# This script only INSTALLS the config; it does NOT rotate. The rotation itself is triggered on a +# timer by dash-node (static/dashboard/sockets/log-rotate.ts), the box's always-up process. Run this +# inside the proot box (where /etc/logrotate.d exists); it is safe to run on every dash-node start. +# +# What it does (all idempotent): +# 1. Move the inherited nginx + php-fpm snippets OUT of /etc/logrotate.d — we override them, and two +# snippets listing the same path make logrotate fail with "duplicate log entry". +# 2. Write /etc/logrotate.d/k2go (copytruncate + size-based; proot-correct — no reopen signal). +# 3. Validate the whole config with `logrotate -d` (parse only); fail loudly if broken. +set -eu + +LR_D="/etc/logrotate.d" +OVERRIDDEN="/etc/logrotate.d.k2go-overridden" # OUTSIDE LR_D, so logrotate never reads it +K2GO_CONF="$LR_D/k2go" + +[ -d "$LR_D" ] || { echo "[k2go-logging] $LR_D not found (not inside the box?) — nothing to do" >&2; exit 0; } +command -v logrotate >/dev/null 2>&1 || { echo "[k2go-logging] logrotate not installed — skipping" >&2; exit 0; } + +# 1) Override the inherited snippets for services we now own (moved aside, reversible). +mkdir -p "$OVERRIDDEN" +for snip in nginx php8.4-fpm; do + if [ -f "$LR_D/$snip" ]; then + mv -f "$LR_D/$snip" "$OVERRIDDEN/$snip" + echo "[k2go-logging] overrode inherited snippet: $snip (moved to $OVERRIDDEN)" + fi +done + +# 2) Write our config, only if it changed (idempotent). +NEW="$(mktemp)" +cat > "$NEW" <<'EOF' +# K2GO-386 / ADR-386 — K2Go-owned log rotation for the proot box. DO NOT edit by hand; +# managed by tools/setup-proot-logging.sh. +# +# proot has no systemd/cron: dash-node triggers `logrotate` on a timer. Every block uses +# copytruncate (no daemon reopen signal — a failed reopen under proot is the deleted-but-open +# failure) and rotates on SIZE (our threat is a file that grows, not a schedule). This overrides +# the inherited nginx/php-fpm snippets (moved to /etc/logrotate.d.k2go-overridden) and adds the +# ones that never shipped (calibre-web, dash-node). kiwix writes no dedicated log; kolibri +# self-rotates under its KOLIBRI_HOME — both are deliberately left alone. +/var/log/php*-fpm.log +/var/log/nginx/*.log +/var/log/calibre-web.log +/var/log/dash-node.log +/var/log/dash-rebuild.log +{ + su root root + size 100M + rotate 3 + compress + delaycompress + missingok + notifempty + copytruncate +} +EOF + +if [ -f "$K2GO_CONF" ] && cmp -s "$NEW" "$K2GO_CONF"; then + echo "[k2go-logging] $K2GO_CONF already up to date" + rm -f "$NEW" +else + cp -f "$NEW" "$K2GO_CONF" + chmod 0644 "$K2GO_CONF" + rm -f "$NEW" + echo "[k2go-logging] installed $K2GO_CONF" +fi + +# 3) Validate (parse only; does NOT rotate). Fail loudly so a broken config never ships silently. +if logrotate -d /etc/logrotate.conf >/dev/null 2>&1; then + echo "[k2go-logging] logrotate config validates OK" +else + echo "[k2go-logging] ERROR: logrotate config failed validation:" >&2 + logrotate -d /etc/logrotate.conf 2>&1 | tail -20 >&2 + exit 1 +fi From 4255914efbabdc2568b65e99323ffd000312ef7e Mon Sep 17 00:00:00 2001 From: Luis Guzman Date: Sat, 5 Sep 2026 00:20:33 -0600 Subject: [PATCH 2/8] K2GO-386 feat(dashboard): install log rotation at deploy, not at dash-node boot; bump 1.3.0-dev.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move the K2Go log-rotation config install out of dash-node startup into the three deploy paths — iiab-android install_iiaboa_dashboard (rootfs build → a clean R2 rootfs ships preconfigured), rebuild-dashboard.sh and dev-push-dashboard.sh (updates re-assert it). Reconfiguring on every boot would be needless churn; installing on build/update is more defensive. dash-node keeps only the 10-min logrotate trigger (no rotation at boot). Bump dash-node 1.2.12 -> 1.3.0-dev.1 (-> 1.3.0 at merge): not a REST-surface change, but the version bump is how a new dash-node behavior is delivered to devices (the update pill; no ansible role yet), so a behavior change that must reach the field gets a bump. --- .../ADR-386-unattended-disk-containment.md | 16 +++++--- iiab-android | 5 +++ static/dashboard/CHANGELOG.md | 1 + static/dashboard/package.json | 2 +- static/dashboard/server.ts | 5 ++- static/dashboard/sockets/log-rotate.ts | 39 +++++-------------- tools/dev-push-dashboard.sh | 3 ++ tools/rebuild-dashboard.sh | 2 + 8 files changed, 35 insertions(+), 38 deletions(-) diff --git a/controller/docs/ADR-386-unattended-disk-containment.md b/controller/docs/ADR-386-unattended-disk-containment.md index e29f55a2b..6d22ed1a5 100644 --- a/controller/docs/ADR-386-unattended-disk-containment.md +++ b/controller/docs/ADR-386-unattended-disk-containment.md @@ -95,10 +95,13 @@ RPi-oriented inherited snippets (learn from them, write ours): **Install:** an idempotent `tools/setup-proot-logging.sh` (ansible-role-shaped) moves the inherited nginx/php-fpm snippets out of `/etc/logrotate.d` (we override them; a duplicate path fails logrotate), -writes `/etc/logrotate.d/k2go`, and validates with `logrotate -d`. dash-node runs it **once at -startup** — this is light (a couple of `mv`s, one file write, a parse-only validate; **no rotation, no -Python, nothing that risks the phantom-process killer**), which is distinct from the rotation itself -(§4, on the timer, never at boot). Phase 2 bakes it into the rootfs as a role with a pdsm-owned trigger. +writes `/etc/logrotate.d/k2go`, and validates with `logrotate -d`. It runs **at deploy time, not at +dash-node startup** — reconfiguring on every boot would be needless churn (and updating the config is +exactly a deploy concern). The three deploy paths call it: the **rootfs build** (`iiab-android`'s +`install_iiaboa_dashboard`), so a clean R2 rootfs ships preconfigured; and the two update paths +(`rebuild-dashboard.sh`, `dev-push-dashboard.sh`). Since the version bump that carries a dash-node +change is delivered through the dashboard-update mechanism, an update that reaches a device also +re-asserts this config. Phase 2 folds it into a rootfs ansible role with a pdsm-owned trigger. ## 6. Layer 3 — App-side backstop (reap + restart, not deny) @@ -134,8 +137,9 @@ guessed — **without bothering the user**: ## 8. Lifecycle (who sets it, who clears it, what if a process dies) -- **L2 config:** installed by dash-node at startup (idempotent — writes only when changed; the snippet - move is a no-op once done), re-asserted every dash-node start; no persistent marker to strand. +- **L2 config:** installed at deploy time (rootfs build + rebuild/dev-push), idempotent — writes only + when changed, the snippet move is a no-op once done; re-asserted on every update, not every boot. No + persistent marker to strand. - **L2 trigger:** dash-node's 10-min timer; owns nothing but the timer. If dash-node dies, logs only grow while services are up and the reconciler owns bring-up; if dash-node *wedges*, rotation stalls — the one Phase-1 liveness dependency, bounded by L3, removed in Phase 2 (pdsm-owned trigger). diff --git a/iiab-android b/iiab-android index faffc1478..4b4bea80a 100644 --- a/iiab-android +++ b/iiab-android @@ -512,6 +512,11 @@ install_iiaboa_dashboard() { /usr/local/bin/pdsm enable dash-node /usr/local/bin/pdsm restart dash-node + # K2GO-386 (ADR-386): install K2Go-owned log rotation for the box (proot has no cron, so a service + # log can otherwise fill the device). Run at deploy, not at dash-node boot. Non-fatal. + log "Configuring log rotation (setup-proot-logging)..." + sh "${K2GO_OPT_DIR}/tools/setup-proot-logging.sh" || warn "log-rotation setup failed (non-fatal)" + # Setup nginx log "Configuring Nginx for the dashboard..." cp "${dash_src}/dash-node-nginx.conf" "/etc/nginx/conf.d/dash-node-nginx.conf" diff --git a/static/dashboard/CHANGELOG.md b/static/dashboard/CHANGELOG.md index 72fc339ee..0d84d7e02 100644 --- a/static/dashboard/CHANGELOG.md +++ b/static/dashboard/CHANGELOG.md @@ -4,6 +4,7 @@ One line per version, newest first. Every REST-facing change bumps the version i (the app surfaces it via `/system/dashboard/update-check` and the "Update available" pill), so this file is the human record of what each bump enables. Keep entries short: `version - change (TICKET)`. +- **1.3.0-dev.1** - Proot log rotation, dash-node-triggered (K2GO-386, ADR-386). proot has no systemd/cron, so `/etc/cron.daily/logrotate` never runs — logrotate was installed but never triggered, and a service log (php-fpm, dash-node) could grow until the device hit ENOSPC. dash-node now runs `logrotate /etc/logrotate.conf` every 10 min (no rotation at boot; `timer.unref`), and the K2Go-owned config `/etc/logrotate.d/k2go` (copytruncate + `size 100M`, proot-correct — no reopen signal — overriding the RPi-oriented nginx/php-fpm snippets and adding calibre-web + dash-node; kiwix has no log, kolibri self-rotates) is installed at deploy by `tools/setup-proot-logging.sh` (rootfs build + rebuild/dev-push), not at boot. Not a REST-surface change; the version bump is the delivery mechanism for the new dash-node behavior (no ansible role yet). WIP → 1.3.0 at merge. (K2GO-386) - **1.2.12** - Dashboard-update card back end (ADFA-5339, Phase 1 server half). New read-only `GET /system/dashboard/rebuild/log`: the last ~200 lines of `/var/log/dash-rebuild.log`, for the card's expandable "Details" (no file yet = empty log, not an error). `POST /system/dashboard/rebuild` now accepts `{ site: true }`: it refreshes the served landing page in the SAME run via `site-updater.sh`, from the same clone the rebuild's git fetch+reset refreshes, in finalize AFTER the core swap verifies live — so the site matches the new source. The site is a separate, versionless artifact: it never touches the reported version, and a site failure is logged, never a rollback of the (already-verified) core update. Both localhost-only. (ADFA-5339) - **1.2.11** - `/auth/:service/session` mints the session **for the agent that asks** (ADFA-5361). Calibre-Web (Flask-Login) binds a session to a fingerprint of the User-Agent, so a session minted under dash-node's own agent was rejected on the WebView's first request: the identity was dropped, the `remember_token` deleted, and the card opened as the anonymous Guest — the "logged in as Admin" flash comes from the injected session and renders even then, which is why the auto-login looked like it worked. The route now forwards the caller's `User-Agent` through the whole login handshake (every request, not just the POST — the fingerprint is established on the first one), for Calibre-Web and Kolibri alike. The callers that consume the session themselves (downloads runner, `removeBook`) are unchanged. No User-Agent on the request degrades to the previous behaviour, logged. Same ticket: the books runner's private copy of the Calibre-Web login is gone — it never got the ADFA-5043 `remember_me` and was the drift this whole bug rode in on — so `getCalibreSession` is the one source. (ADFA-5361) - **1.2.10** - In-proot content-service recovery (ADFA-5343, ADR-5343a §10). New `POST /system/service/:svc/restart`: runs `pdsm restart ` in the one living proot to recover a content service wedged after an environment relaunch (orphaned off proot → `epoll_wait` ENOSYS), for the supported upstream services (mirrors `pdsm_installed_services`; `dash-node` excluded). A server-side watcher auto-heals a present-but-wedged content service (404 = not installed → left alone), cooldown-bounded; the app's future module-card Retry is the manual backstop hitting the same endpoint. Loopback-only, like all of `/k2go-api`. (ADFA-5343) diff --git a/static/dashboard/package.json b/static/dashboard/package.json index 03c376746..40bbe6828 100644 --- a/static/dashboard/package.json +++ b/static/dashboard/package.json @@ -1,6 +1,6 @@ { "name": "dashboard-console", - "version": "1.2.12", + "version": "1.3.0-dev.1", "description": "", "main": "index.js", "scripts": { diff --git a/static/dashboard/server.ts b/static/dashboard/server.ts index 42e15d4fe..461194c0e 100644 --- a/static/dashboard/server.ts +++ b/static/dashboard/server.ts @@ -41,8 +41,9 @@ server.listen(PORT, '127.0.0.1', () => { try { jobs.reconcileOnBoot(); } catch (e) { console.error('[jobs] reconcile failed', e); } // ADFA-5343 (ADR-5343a §10): the box heals its own content-service tree in-proot. try { startServiceHeal(); } catch (e) { console.error('[service-heal] start failed', e); } - // K2GO-386 (ADR-386 §5): proot has no cron — dash-node triggers logrotate on a timer so the - // K2Go-owned rotation config actually runs (no rotation at boot; first pass at +10 min). + // K2GO-386 (ADR-386 §4): proot has no cron — dash-node triggers logrotate on a timer so the + // K2Go-owned rotation config (installed at deploy time) actually runs (no rotation at boot; + // first pass at +10 min). try { startLogRotation(); } catch (e) { console.error('[log-rotate] start failed', e); } }); diff --git a/static/dashboard/sockets/log-rotate.ts b/static/dashboard/sockets/log-rotate.ts index 028023985..3e10e987b 100644 --- a/static/dashboard/sockets/log-rotate.ts +++ b/static/dashboard/sockets/log-rotate.ts @@ -1,35 +1,19 @@ // sockets/log-rotate.ts — K2GO-386 / ADR-386 (Layer 2: contain logs) // // proot has no systemd/cron, so nothing runs logrotate on its own. dash-node — the box's always-up -// process — installs the K2Go-owned rotation config once (light: it moves snippets + writes one file -// + validates; it does NOT rotate) and then triggers `logrotate` on a fixed timer so the config -// actually rotates. There is NO rotation at boot (setInterval fires first at +interval): boot is the -// heaviest, most fragile moment under proot and we keep it clear. Everything here is best-effort — a -// failure logs and the next tick retries; nothing throws into the caller. +// process — triggers `logrotate` on a fixed timer so the K2Go-owned rotation config actually rotates. +// The config itself is INSTALLED at deploy time (tools/setup-proot-logging.sh, run by the rootfs +// install and by rebuild/dev-push), NOT here and NOT at boot — re-configuring on every dash-node start +// would be needless churn (ADR-386 §5). There is NO rotation at boot either (setInterval fires first +// at +interval): boot is the heaviest, most fragile moment under proot and we keep it clear. +// Best-effort — a failure logs and the next tick retries; nothing throws into the caller. import { execFile } from 'child_process'; -import * as path from 'path'; -// Rotate every 10 min (ADR-386 §5). Wide enough to be nearly free; narrow enough to bound growth. +// Rotate every 10 min (ADR-386 §4). Wide enough to be nearly free; narrow enough to bound growth. const INTERVAL_MS = 10 * 60 * 1000; -// The configurator ships in the repo under tools/; on an installed box it lives in the clone that -// rebuild-dashboard.sh / dev-push-dashboard.sh keep current (default /opt/iiab-android). -const CLONE_DIR = process.env.K2GO_CLONE_DIR || '/opt/iiab-android'; -const CONFIGURATOR = path.join(CLONE_DIR, 'tools', 'setup-proot-logging.sh'); - let timer: NodeJS.Timeout | null = null; -// Install the rotation config once (idempotent, no rotation). Best-effort. -function installConfigOnce(): void { - execFile('sh', [CONFIGURATOR], { timeout: 30_000 }, (err, stdout, stderr) => { - if (err) { - console.error('[log-rotate] config install failed:', err.message, (stderr || '').trim()); - } else if ((stdout || '').trim()) { - console.log('[log-rotate] config:', stdout.trim()); - } - }); -} - // Run one logrotate pass over the whole config (the K2Go blocks use copytruncate + size, so this is // cheap unless a file actually exceeds its cap). Best-effort. function runLogrotateOnce(): void { @@ -42,17 +26,14 @@ function runLogrotateOnce(): void { /** * Start the periodic log-rotation trigger. Idempotent; call once from server.ts at listen time. - * Installs the config immediately (light), then rotates every INTERVAL_MS — the FIRST rotation is at - * +INTERVAL_MS, never at boot (ADR-386 §5). + * Rotates every INTERVAL_MS — the FIRST rotation is at +INTERVAL_MS, never at boot (ADR-386 §4). + * The rotation config is installed separately, at deploy time (ADR-386 §5). */ export function startLogRotation(): void { if (timer) return; - installConfigOnce(); timer = setInterval(runLogrotateOnce, INTERVAL_MS); if (typeof timer.unref === 'function') timer.unref(); // don't keep the process alive just for this - console.log( - `[log-rotate] scheduled: logrotate every ${INTERVAL_MS / 60000} min (config installed; no rotation at boot)`, - ); + console.log(`[log-rotate] scheduled: logrotate every ${INTERVAL_MS / 60000} min (no rotation at boot)`); } export function stopLogRotation(): void { diff --git a/tools/dev-push-dashboard.sh b/tools/dev-push-dashboard.sh index acc2564ed..d9fb9e532 100755 --- a/tools/dev-push-dashboard.sh +++ b/tools/dev-push-dashboard.sh @@ -43,6 +43,9 @@ echo "[dev-push] deploying nginx vhost to $NGINX_CONF_DIR..." cp -f "$DEST/dash-node-nginx.conf" "$NGINX_CONF_DIR/dash-node-nginx.conf" chmod 0600 "$NGINX_CONF_DIR/dash-node-nginx.conf" +echo "[dev-push] configuring log rotation (setup-proot-logging)..." +sh "$CLONE_DIR/tools/setup-proot-logging.sh" || echo "[dev-push] warn: log-rotation setup failed (non-fatal)" + echo "[dev-push] restarting dash-node + nginx..." /usr/local/bin/pdsm restart dash-node /usr/local/bin/pdsm restart nginx diff --git a/tools/rebuild-dashboard.sh b/tools/rebuild-dashboard.sh index 04fb586b0..c1fddcfe7 100755 --- a/tools/rebuild-dashboard.sh +++ b/tools/rebuild-dashboard.sh @@ -165,6 +165,8 @@ if verify_live; then # nginx reads /etc/nginx/conf.d, not /library/dashboard, so mirror the vhost then reload nginx. [ -f "$LIVE/dash-node-nginx.conf" ] && { cp -f "$LIVE/dash-node-nginx.conf" "$NGINX_CONF_DIR/dash-node-nginx.conf"; chmod 0600 "$NGINX_CONF_DIR/dash-node-nginx.conf"; } /usr/local/bin/pdsm restart nginx >>"$LOG" 2>&1 || log "warn: pdsm restart nginx returned non-zero" + # K2GO-386 (ADR-386): re-assert K2Go-owned log rotation on every update (proot has no cron). + sh "$CLONE_DIR/tools/setup-proot-logging.sh" >>"$LOG" 2>&1 || log "warn: log-rotation setup failed (non-fatal)" # ADFA-5339: optionally refresh the served landing page, from the SAME clone the git fetch+reset # above just refreshed, so it matches the new source. Runs only here — after the core swap has # verified live — and is best-effort: the site is a separate, versionless artifact, so a failure is From 046a3f7518839f241853cd2ed51be360b794954c Mon Sep 17 00:00:00 2001 From: Luis Guzman Date: Sat, 5 Sep 2026 00:45:37 -0600 Subject: [PATCH 3/8] K2GO-386 fix(dashboard): code-review fixes for proot log rotation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Double-pass code review on the L2 diff: - setup-proot-logging.sh: validate the effective config and ROLL BACK to the exact pre-run state on failure (snapshot k2go + track moved snippets), so a validation failure never leaves a broken config that breaks all rotation. Move php*-fpm by glob (was hardcoded php8.4-fpm) to match the config's /var/log/php*-fpm.log — one source for 'which php', survives a version bump. Resolve logrotate by absolute path. - log-rotate.ts: call /usr/sbin/logrotate by absolute path (dash-node's runtime PATH may omit /usr/sbin -> ENOENT every tick). - server.ts: gracefulShutdown clears the timer via stopLogRotation() (explicit lifecycle). Tracked (not fixed): a package update recreating a default snippet between deploys (Phase-2 pdsm re-assert), and copytruncate of dash-rebuild.log mid-rebuild (size cap bounds it). --- static/dashboard/server.ts | 5 ++- static/dashboard/sockets/log-rotate.ts | 6 +++- tools/setup-proot-logging.sh | 48 ++++++++++++++++++-------- 3 files changed, 43 insertions(+), 16 deletions(-) diff --git a/static/dashboard/server.ts b/static/dashboard/server.ts index 461194c0e..87fe409d0 100644 --- a/static/dashboard/server.ts +++ b/static/dashboard/server.ts @@ -11,7 +11,7 @@ import './sockets/books.exec'; import './sockets/kolibri.exec'; import { apiRouter } from './routes'; import { startServiceHeal } from './sockets/service-heal'; -import { startLogRotation } from './sockets/log-rotate'; +import { startLogRotation, stopLogRotation } from './sockets/log-rotate'; const app = express(); const server = http.createServer(app); @@ -53,6 +53,9 @@ server.listen(PORT, '127.0.0.1', () => { const gracefulShutdown = (signal: string) => { console.log(`\n[System] Received ${signal}. Starting graceful shutdown...`); + // K2GO-386: clear the log-rotation timer (who starts it: startLogRotation; who clears it: here). + try { stopLogRotation(); } catch (e) { console.error('[log-rotate] stop failed', e); } + server.close(() => { console.log('[System] HTTP server closed. No longer accepting connections.'); console.log('[System] Cleanup complete. Exiting safely.'); diff --git a/static/dashboard/sockets/log-rotate.ts b/static/dashboard/sockets/log-rotate.ts index 3e10e987b..58aa73657 100644 --- a/static/dashboard/sockets/log-rotate.ts +++ b/static/dashboard/sockets/log-rotate.ts @@ -14,10 +14,14 @@ const INTERVAL_MS = 10 * 60 * 1000; let timer: NodeJS.Timeout | null = null; +// logrotate lives in /usr/sbin, which dash-node's runtime PATH (set by its pdsm wrapper) may not +// include — call it by absolute path so a reduced PATH can't turn every tick into an ENOENT. +const LOGROTATE_BIN = '/usr/sbin/logrotate'; + // Run one logrotate pass over the whole config (the K2Go blocks use copytruncate + size, so this is // cheap unless a file actually exceeds its cap). Best-effort. function runLogrotateOnce(): void { - execFile('logrotate', ['/etc/logrotate.conf'], { timeout: 60_000 }, (err, _stdout, stderr) => { + execFile(LOGROTATE_BIN, ['/etc/logrotate.conf'], { timeout: 60_000 }, (err, _stdout, stderr) => { if (err) { console.error('[log-rotate] logrotate run failed:', err.message, (stderr || '').trim()); } diff --git a/tools/setup-proot-logging.sh b/tools/setup-proot-logging.sh index 4f9d33f32..0fc60006e 100644 --- a/tools/setup-proot-logging.sh +++ b/tools/setup-proot-logging.sh @@ -10,29 +10,45 @@ # # This script only INSTALLS the config; it does NOT rotate. The rotation itself is triggered on a # timer by dash-node (static/dashboard/sockets/log-rotate.ts), the box's always-up process. Run this -# inside the proot box (where /etc/logrotate.d exists); it is safe to run on every dash-node start. +# inside the proot box (where /etc/logrotate.d exists), at DEPLOY time (rootfs build + rebuild/dev-push). # # What it does (all idempotent): -# 1. Move the inherited nginx + php-fpm snippets OUT of /etc/logrotate.d — we override them, and two +# 1. Move the inherited nginx + php*-fpm snippets OUT of /etc/logrotate.d — we override them, and two # snippets listing the same path make logrotate fail with "duplicate log entry". # 2. Write /etc/logrotate.d/k2go (copytruncate + size-based; proot-correct — no reopen signal). -# 3. Validate the whole config with `logrotate -d` (parse only); fail loudly if broken. +# 3. Validate the whole config with `logrotate -d` (parse only); on failure ROLL BACK to the prior +# state (never leave a broken config that would break ALL rotation) and fail loudly. set -eu LR_D="/etc/logrotate.d" OVERRIDDEN="/etc/logrotate.d.k2go-overridden" # OUTSIDE LR_D, so logrotate never reads it K2GO_CONF="$LR_D/k2go" +# Resolve logrotate by absolute path (a deploy shell may have a reduced PATH without /usr/sbin). +LOGROTATE="$(command -v logrotate 2>/dev/null || echo /usr/sbin/logrotate)" [ -d "$LR_D" ] || { echo "[k2go-logging] $LR_D not found (not inside the box?) — nothing to do" >&2; exit 0; } -command -v logrotate >/dev/null 2>&1 || { echo "[k2go-logging] logrotate not installed — skipping" >&2; exit 0; } +[ -x "$LOGROTATE" ] || { echo "[k2go-logging] logrotate not installed ($LOGROTATE) — skipping" >&2; exit 0; } -# 1) Override the inherited snippets for services we now own (moved aside, reversible). +# --- snapshot for rollback (restore EXACTLY the pre-run state on validation failure) -------------- +PREV_K2GO_BAK="" +if [ -f "$K2GO_CONF" ]; then PREV_K2GO_BAK="$(mktemp)"; cp -f "$K2GO_CONF" "$PREV_K2GO_BAK"; fi +MOVED="" # basenames this run moved aside, so rollback restores only those + +rollback() { + if [ -n "$PREV_K2GO_BAK" ]; then cp -f "$PREV_K2GO_BAK" "$K2GO_CONF"; else rm -f "$K2GO_CONF"; fi + for base in $MOVED; do mv -f "$OVERRIDDEN/$base" "$LR_D/$base" 2>/dev/null || true; done +} + +# 1) Override the inherited snippets for services we now own. nginx is a fixed name; php*-fpm is a +# glob so a php-version bump (php8.5-fpm) is covered too — it MUST match the config's php*-fpm.log +# glob below, or the un-moved snippet would collide with our block. mkdir -p "$OVERRIDDEN" -for snip in nginx php8.4-fpm; do - if [ -f "$LR_D/$snip" ]; then - mv -f "$LR_D/$snip" "$OVERRIDDEN/$snip" - echo "[k2go-logging] overrode inherited snippet: $snip (moved to $OVERRIDDEN)" - fi +for snip in "$LR_D/nginx" "$LR_D"/php*-fpm; do + [ -f "$snip" ] || continue + base="$(basename "$snip")" + mv -f "$snip" "$OVERRIDDEN/$base" + MOVED="$MOVED $base" + echo "[k2go-logging] overrode inherited snippet: $base (moved to $OVERRIDDEN)" done # 2) Write our config, only if it changed (idempotent). @@ -74,11 +90,15 @@ else echo "[k2go-logging] installed $K2GO_CONF" fi -# 3) Validate (parse only; does NOT rotate). Fail loudly so a broken config never ships silently. -if logrotate -d /etc/logrotate.conf >/dev/null 2>&1; then +# 3) Validate the WHOLE effective config (parse only; does NOT rotate). On failure, roll back to the +# pre-run state so a broken config never breaks all rotation, then fail loudly. +if "$LOGROTATE" -d /etc/logrotate.conf >/dev/null 2>&1; then echo "[k2go-logging] logrotate config validates OK" + rm -f "$PREV_K2GO_BAK" else - echo "[k2go-logging] ERROR: logrotate config failed validation:" >&2 - logrotate -d /etc/logrotate.conf 2>&1 | tail -20 >&2 + echo "[k2go-logging] ERROR: logrotate config failed validation — rolling back:" >&2 + "$LOGROTATE" -d /etc/logrotate.conf 2>&1 | tail -20 >&2 + rollback + rm -f "$PREV_K2GO_BAK" exit 1 fi From 5ff44e9847362edfc30eea2c8fa647ff66bdb69e Mon Sep 17 00:00:00 2001 From: Luis Guzman Date: Sat, 5 Sep 2026 01:17:37 -0600 Subject: [PATCH 4/8] K2GO-386 chore(tools): add -h/--help usage to the touched shell scripts Add a focused usage() + `case "${1:-}" in -h|--help)` early-exit to the four shell scripts this branch touches (setup-proot-logging.sh, rebuild-dashboard.sh, dev-push-dashboard.sh, iiab-android), so their usage can be consulted without opening the file. Uses a hand-written usage block (not `grep '^#'`) so inline comments and the config heredoc don't leak into the help. Additive: existing args (branch, clone_dir, -f, set-hostname) fall through unchanged; --help works before iiab-android's root check. --- iiab-android | 11 +++++++++++ tools/dev-push-dashboard.sh | 14 ++++++++++++++ tools/rebuild-dashboard.sh | 15 +++++++++++++++ tools/setup-proot-logging.sh | 14 ++++++++++++++ 4 files changed, 54 insertions(+) diff --git a/iiab-android b/iiab-android index 4b4bea80a..10616af16 100644 --- a/iiab-android +++ b/iiab-android @@ -19,6 +19,17 @@ log() { printf "${BLU}[iiab]${RST} %s\n" "$*"; } warn() { printf "${YEL}[iiab] WARNING:${RST} %s\n" "$*" >&2; } die() { printf "${RED}[!] ERROR${RST}: %s\n" "$*" >&2; exit 1; } +usage() { + cat <<'USAGE' +iiab-android — K2Go rootfs installer. Runs the IIAB install under proot and installs the K2Go +dashboard, services, content, and log rotation. Run as ROOT inside the proot box; normally invoked +by tools/rootfs-builder/build-iiab-rootfs.sh, not by hand. + +Usage: iiab-android [-h|--help] +USAGE +} +case "${1:-}" in -h|--help) usage; exit 0 ;; esac + #----------------------------- # Safety checks #----------------------------- diff --git a/tools/dev-push-dashboard.sh b/tools/dev-push-dashboard.sh index d9fb9e532..804cce993 100755 --- a/tools/dev-push-dashboard.sh +++ b/tools/dev-push-dashboard.sh @@ -17,6 +17,20 @@ # deploy dash-node-nginx.conf to /etc/nginx/conf.d (nginx does not read /library/dashboard). set -eu +usage() { + cat <<'USAGE' +dev-push-dashboard.sh — push an updated dashboard from a local clone into the INSTALLED rootfs and +restart the service, WITHOUT the ~2h rootfs rebuild (ADFA-4839). Run from INSIDE the proot. + +Usage: sh dev-push-dashboard.sh [CLONE_DIR] + CLONE_DIR clone to deploy from (default: the repo this script lives in) + +Syncs static/dashboard -> /library/dashboard (preserving node_modules), installs log rotation, +deploys the nginx vhost, rebuilds (yarn build), and restarts dash-node + nginx. +USAGE +} +case "${1:-}" in -h|--help) usage; exit 0 ;; esac + CLONE_DIR="${1:-$(cd "$(dirname "$0")/.." && pwd)}" SRC="$CLONE_DIR/static/dashboard" DEST="/library/dashboard" diff --git a/tools/rebuild-dashboard.sh b/tools/rebuild-dashboard.sh index c1fddcfe7..850773bd5 100755 --- a/tools/rebuild-dashboard.sh +++ b/tools/rebuild-dashboard.sh @@ -26,6 +26,21 @@ set -u # is how the app's detached REST call passes it). CLONE_DIR is the optional $2 — the install location is # almost always the same, so it defaults; pass it only if it moved. # sh tools/rebuild-dashboard.sh [clone_dir] +usage() { + cat <<'USAGE' +rebuild-dashboard.sh — rebuild ONLY the dash-node REST API from the on-device clone, no rootfs +rebuild. Blue-green + verify-before-commit (ADFA-5011). + +Usage: sh rebuild-dashboard.sh [BRANCH] [CLONE_DIR] + BRANCH git branch to fetch + reset --hard from origin (default: $K2GO_BRANCH, else main) + CLONE_DIR clone location to build from (default: /opt/iiab-android) + +Run inside the proot box. Fetches origin/, builds in a staging dir, smoke-tests it, +atomically swaps the dist in, restarts dash-node, and verifies live (rolls back on failure). +USAGE +} +case "${1:-}" in -h|--help) usage; exit 0 ;; esac + BRANCH="${1:-${K2GO_BRANCH:-main}}" CLONE_DIR="${2:-/opt/iiab-android}" SRC="$CLONE_DIR/static/dashboard" diff --git a/tools/setup-proot-logging.sh b/tools/setup-proot-logging.sh index 0fc60006e..f806b5797 100644 --- a/tools/setup-proot-logging.sh +++ b/tools/setup-proot-logging.sh @@ -20,6 +20,20 @@ # state (never leave a broken config that would break ALL rotation) and fail loudly. set -eu +usage() { + cat <<'USAGE' +setup-proot-logging.sh — install K2Go-owned log rotation for the proot box (K2GO-386 / ADR-386). + +Usage: sh setup-proot-logging.sh [-h|--help] + +Takes no arguments. Run inside the proot box, at DEPLOY time (rootfs build + rebuild/dev-push). +It overrides the inherited nginx/php-fpm logrotate snippets, installs /etc/logrotate.d/k2go +(copytruncate + size-based), validates with `logrotate -d`, and rolls back on failure. Idempotent. +dash-node triggers the rotation itself on a 10-min timer; this script only installs the config. +USAGE +} +case "${1:-}" in -h|--help) usage; exit 0 ;; esac + LR_D="/etc/logrotate.d" OVERRIDDEN="/etc/logrotate.d.k2go-overridden" # OUTSIDE LR_D, so logrotate never reads it K2GO_CONF="$LR_D/k2go" From cf9b8abf2e92b092bdc52891c8f752e03235cf2b Mon Sep 17 00:00:00 2001 From: Luis Guzman Date: Sat, 5 Sep 2026 09:26:01 -0600 Subject: [PATCH 5/8] K2GO-386 feat(dashboard): firehose guard before logrotate (in-box L3 half) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A firehose (a runaway logging ~GB/min) must never reach logrotate: copytruncate would COPY a multi-GB file (doubling disk, pegging CPU on a weak phone) and still not stop the writer. So dash-node's tick now runs, IN ORDER on ONE timer: 1. firehose guard — truncate any /var/log[/nginx]/*.log past ~1 GiB IN PLACE (reclaim, no copy/compress; a firehose log is garbage). Deterministic order (guard->rotate), one clock, so there is no second timer to drift. 2. logrotate — rotate the remaining moderate logs. This is the in-box half of L3 (ADR-386 §6): dash-node can DETECT + RECLAIM but cannot stop an off-proot orphan (device-proven); a recurring firehose is flagged (occurrence count) for the app-side reap, which is the other half, still to build. Truncating each tick bounds the disk even for an unstoppable orphan (headroom > rate*interval), buying time for the app-side stop. --- .../ADR-386-unattended-disk-containment.md | 18 +++- static/dashboard/CHANGELOG.md | 2 +- static/dashboard/sockets/log-rotate.ts | 82 +++++++++++++++---- 3 files changed, 85 insertions(+), 17 deletions(-) diff --git a/controller/docs/ADR-386-unattended-disk-containment.md b/controller/docs/ADR-386-unattended-disk-containment.md index 6d22ed1a5..5eec9a5e3 100644 --- a/controller/docs/ADR-386-unattended-disk-containment.md +++ b/controller/docs/ADR-386-unattended-disk-containment.md @@ -80,7 +80,12 @@ RPi-oriented inherited snippets (learn from them, write ours): `su root root`) — our threat is a file that grows, not a calendar. - **Trigger = dash-node** (the box's always-up process), **every 10 min, no run at boot** — boot is the heaviest/most fragile moment (Python services starting; phantom-process-killer risk). Accepted tradeoff: - a <10-min session never rotates; the size cap + L3 bound it. + a <10-min session never rotates; the size cap + the guard below bound it. +- **A firehose guard runs FIRST, in the same tick** (deterministic order, one clock — no second timer to + drift): before logrotate, dash-node truncates any log past a firehose threshold (~1 GiB) **in place**, + so logrotate never has to copy a multi-GB runaway (a copy doubles disk and pegs CPU on a weak phone, and + still would not stop the writer). This is the in-box half of L3 (§6); a recurring firehose is flagged for + the app-side reap. **The config set (verified on-device against the pdsm wrappers + live `/var/log`):** @@ -109,6 +114,17 @@ The outside-the-rootfs net for what L2 cannot catch: the **fast firehose** (fill rotation) and **Vector B** (no file to rotate; must stop the holding process). Device-proven: only the app can stop an off-proot orphan. +**L3 splits into two halves at the proot boundary — this matters:** +- **In-box (dash-node) — DETECT + RECLAIM.** dash-node can *see* a firehose log and *truncate* it in + place (reclaim, no copy) — but it **cannot stop an off-proot orphan** (an in-box kill does not reach it, + device-proven; the same limitation that parked the K2GO-381 in-box healer). This half is already coupled + to L2's tick as the firehose guard (§5): it protects L2 and bounds the disk (each tick truncates the + runaway back, so even an unstoppable orphan cannot reach ENOSPC as long as headroom > rate×interval), + and it emits the recurring-firehose signal. +- **App-side (Android) — STOP.** The only actor that can reap an off-proot orphan. It is driven by the + in-box recurring-firehose signal (a log that keeps refilling after truncation = an orphan) or by disk + pressure, and it reaps + reports. This is the half that is still to be built. + **Decisions (direction; detailed mechanism is the next design step, possibly `ADR-386a`):** - **Targeted, not blind.** Act on the offending vector (a specific runaway log/service), keeping the rest diff --git a/static/dashboard/CHANGELOG.md b/static/dashboard/CHANGELOG.md index 0d84d7e02..d10aaa820 100644 --- a/static/dashboard/CHANGELOG.md +++ b/static/dashboard/CHANGELOG.md @@ -4,7 +4,7 @@ One line per version, newest first. Every REST-facing change bumps the version i (the app surfaces it via `/system/dashboard/update-check` and the "Update available" pill), so this file is the human record of what each bump enables. Keep entries short: `version - change (TICKET)`. -- **1.3.0-dev.1** - Proot log rotation, dash-node-triggered (K2GO-386, ADR-386). proot has no systemd/cron, so `/etc/cron.daily/logrotate` never runs — logrotate was installed but never triggered, and a service log (php-fpm, dash-node) could grow until the device hit ENOSPC. dash-node now runs `logrotate /etc/logrotate.conf` every 10 min (no rotation at boot; `timer.unref`), and the K2Go-owned config `/etc/logrotate.d/k2go` (copytruncate + `size 100M`, proot-correct — no reopen signal — overriding the RPi-oriented nginx/php-fpm snippets and adding calibre-web + dash-node; kiwix has no log, kolibri self-rotates) is installed at deploy by `tools/setup-proot-logging.sh` (rootfs build + rebuild/dev-push), not at boot. Not a REST-surface change; the version bump is the delivery mechanism for the new dash-node behavior (no ansible role yet). WIP → 1.3.0 at merge. (K2GO-386) +- **1.3.0-dev.1** - Proot log rotation, dash-node-triggered (K2GO-386, ADR-386). proot has no systemd/cron, so `/etc/cron.daily/logrotate` never runs — logrotate was installed but never triggered, and a service log (php-fpm, dash-node) could grow until the device hit ENOSPC. dash-node now runs, every 10 min (no work at boot; `timer.unref`), a firehose guard THEN `logrotate /etc/logrotate.conf`: the guard truncates any log past ~1 GiB in place first (a runaway ~GB/min that logrotate would otherwise copy — doubling disk + pegging CPU on a weak phone), so L2 never meets a firehose; a recurring firehose is flagged for the future app-side reap (ADR-386 §6). The K2Go-owned config `/etc/logrotate.d/k2go` (copytruncate + `size 100M`, proot-correct — no reopen signal — overriding the RPi-oriented nginx/php-fpm snippets and adding calibre-web + dash-node; kiwix has no log, kolibri self-rotates) is installed at deploy by `tools/setup-proot-logging.sh` (rootfs build + rebuild/dev-push), not at boot. Not a REST-surface change; the version bump is the delivery mechanism for the new dash-node behavior (no ansible role yet). WIP → 1.3.0 at merge. (K2GO-386) - **1.2.12** - Dashboard-update card back end (ADFA-5339, Phase 1 server half). New read-only `GET /system/dashboard/rebuild/log`: the last ~200 lines of `/var/log/dash-rebuild.log`, for the card's expandable "Details" (no file yet = empty log, not an error). `POST /system/dashboard/rebuild` now accepts `{ site: true }`: it refreshes the served landing page in the SAME run via `site-updater.sh`, from the same clone the rebuild's git fetch+reset refreshes, in finalize AFTER the core swap verifies live — so the site matches the new source. The site is a separate, versionless artifact: it never touches the reported version, and a site failure is logged, never a rollback of the (already-verified) core update. Both localhost-only. (ADFA-5339) - **1.2.11** - `/auth/:service/session` mints the session **for the agent that asks** (ADFA-5361). Calibre-Web (Flask-Login) binds a session to a fingerprint of the User-Agent, so a session minted under dash-node's own agent was rejected on the WebView's first request: the identity was dropped, the `remember_token` deleted, and the card opened as the anonymous Guest — the "logged in as Admin" flash comes from the injected session and renders even then, which is why the auto-login looked like it worked. The route now forwards the caller's `User-Agent` through the whole login handshake (every request, not just the POST — the fingerprint is established on the first one), for Calibre-Web and Kolibri alike. The callers that consume the session themselves (downloads runner, `removeBook`) are unchanged. No User-Agent on the request degrades to the previous behaviour, logged. Same ticket: the books runner's private copy of the Calibre-Web login is gone — it never got the ADFA-5043 `remember_me` and was the drift this whole bug rode in on — so `getCalibreSession` is the one source. (ADFA-5361) - **1.2.10** - In-proot content-service recovery (ADFA-5343, ADR-5343a §10). New `POST /system/service/:svc/restart`: runs `pdsm restart ` in the one living proot to recover a content service wedged after an environment relaunch (orphaned off proot → `epoll_wait` ENOSYS), for the supported upstream services (mirrors `pdsm_installed_services`; `dash-node` excluded). A server-side watcher auto-heals a present-but-wedged content service (404 = not installed → left alone), cooldown-bounded; the app's future module-card Retry is the manual backstop hitting the same endpoint. Loopback-only, like all of `/k2go-api`. (ADFA-5343) diff --git a/static/dashboard/sockets/log-rotate.ts b/static/dashboard/sockets/log-rotate.ts index 58aa73657..ce8d87882 100644 --- a/static/dashboard/sockets/log-rotate.ts +++ b/static/dashboard/sockets/log-rotate.ts @@ -1,25 +1,70 @@ // sockets/log-rotate.ts — K2GO-386 / ADR-386 (Layer 2: contain logs) // // proot has no systemd/cron, so nothing runs logrotate on its own. dash-node — the box's always-up -// process — triggers `logrotate` on a fixed timer so the K2Go-owned rotation config actually rotates. -// The config itself is INSTALLED at deploy time (tools/setup-proot-logging.sh, run by the rootfs -// install and by rebuild/dev-push), NOT here and NOT at boot — re-configuring on every dash-node start -// would be needless churn (ADR-386 §5). There is NO rotation at boot either (setInterval fires first -// at +interval): boot is the heaviest, most fragile moment under proot and we keep it clear. -// Best-effort — a failure logs and the next tick retries; nothing throws into the caller. +// process — drives log containment on a fixed timer. Each tick, IN ORDER: +// 1. firehose GUARD: truncate any pathologically huge .log IN PLACE, before logrotate can try to +// copy it. logrotate's copytruncate would COPY a 6 GB runaway log (doubling disk, pegging CPU on +// a weak phone) and still not stop the writer — so we reclaim it in seco first (no copy, no gzip). +// A firehose log is garbage (repeated error spam); we discard it. This is the in-box half of L3 +// (ADR-386 §6): dash-node can DETECT + RECLAIM, but it CANNOT stop an off-proot orphan (device- +// proven — an in-box kill does not reach it); STOPPING that is app-side. A recurring firehose is +// flagged for that app-side reap. +// 2. logrotate: rotate the remaining MODERATE logs (copytruncate + size-based, cheap now). +// Coupling the guard to the SAME timer as logrotate gives a deterministic order (guard→rotate) with +// ONE clock — so L2 never meets a firehose, and there is no second timer to drift out of sync. +// +// The config itself is INSTALLED at deploy time (tools/setup-proot-logging.sh), not here and not at +// boot. There is NO work at boot either (setInterval fires first at +interval): boot is the heaviest, +// most fragile moment under proot and we keep it clear. Everything here is best-effort. import { execFile } from 'child_process'; +import * as fs from 'fs'; +import * as path from 'path'; // Rotate every 10 min (ADR-386 §4). Wide enough to be nearly free; narrow enough to bound growth. const INTERVAL_MS = 10 * 60 * 1000; -let timer: NodeJS.Timeout | null = null; - // logrotate lives in /usr/sbin, which dash-node's runtime PATH (set by its pdsm wrapper) may not // include — call it by absolute path so a reduced PATH can't turn every tick into an ENOENT. const LOGROTATE_BIN = '/usr/sbin/logrotate'; -// Run one logrotate pass over the whole config (the K2Go blocks use copytruncate + size, so this is -// cheap unless a file actually exceeds its cap). Best-effort. +// A .log past this size is a firehose (a runaway ~GB/min), not a normal log (which are KB–MB). logrotate +// would copy it; we truncate it in place instead. Parametric — tune per device headroom / interval. +const FIREHOSE_BYTES = 1024 * 1024 * 1024; // 1 GiB +// The dirs logrotate's k2go config covers; the guard scans the same surface for *.log files. +const LOG_DIRS = ['/var/log', '/var/log/nginx']; + +// Consecutive-truncation count per path, so a RECURRING firehose (an orphan the box cannot stop, that +// just refills) is flagged for the app-side reap (ADR-386 §6). In-memory; cleared when the log is sane. +const firehoseStreak = new Map(); + +let timer: NodeJS.Timeout | null = null; + +/** Pre-logrotate guard: truncate any firehose-sized .log IN PLACE so logrotate never has to copy it. + * Reclaims instantly (no copy, no compress). Best-effort per file. */ +function guardFirehoseLogs(): void { + for (const dir of LOG_DIRS) { + let entries: string[]; + try { entries = fs.readdirSync(dir); } catch { continue; } + for (const name of entries) { + if (!name.endsWith('.log')) continue; + const p = path.join(dir, name); + try { + const size = fs.statSync(p).size; + if (size <= FIREHOSE_BYTES) { firehoseStreak.delete(p); continue; } + fs.truncateSync(p, 0); // reclaim in place — no copy, no compress + const n = (firehoseStreak.get(p) || 0) + 1; + firehoseStreak.set(p, n); + console.warn( + `[log-rotate] FIREHOSE: truncated ${p} (${size} B) in place, occurrence #${n}` + + (n >= 2 ? ' — recurring; likely an off-proot orphan, needs app-side reap (ADR-386 L3)' : ''), + ); + } catch { /* best-effort per file */ } + } + } +} + +/** Run one logrotate pass over the whole config (the K2Go blocks use copytruncate + size, so this is + * cheap unless a file actually exceeds its cap). Best-effort. */ function runLogrotateOnce(): void { execFile(LOGROTATE_BIN, ['/etc/logrotate.conf'], { timeout: 60_000 }, (err, _stdout, stderr) => { if (err) { @@ -28,16 +73,23 @@ function runLogrotateOnce(): void { }); } +/** One tick: the firehose guard first (so L2 never meets a firehose), then logrotate the moderate logs. */ +function tick(): void { + try { guardFirehoseLogs(); } catch (e) { console.error('[log-rotate] firehose guard failed', e); } + runLogrotateOnce(); +} + /** - * Start the periodic log-rotation trigger. Idempotent; call once from server.ts at listen time. - * Rotates every INTERVAL_MS — the FIRST rotation is at +INTERVAL_MS, never at boot (ADR-386 §4). - * The rotation config is installed separately, at deploy time (ADR-386 §5). + * Start the periodic log-containment trigger. Idempotent; call once from server.ts at listen time. + * Runs guard+logrotate every INTERVAL_MS — the FIRST run is at +INTERVAL_MS, never at boot (ADR-386 §4). */ export function startLogRotation(): void { if (timer) return; - timer = setInterval(runLogrotateOnce, INTERVAL_MS); + timer = setInterval(tick, INTERVAL_MS); if (typeof timer.unref === 'function') timer.unref(); // don't keep the process alive just for this - console.log(`[log-rotate] scheduled: logrotate every ${INTERVAL_MS / 60000} min (no rotation at boot)`); + console.log( + `[log-rotate] scheduled: firehose guard + logrotate every ${INTERVAL_MS / 60000} min (none at boot)`, + ); } export function stopLogRotation(): void { From f116e52638c332849e4a0153429fd5bb35d83703 Mon Sep 17 00:00:00 2001 From: Luis Guzman Date: Sat, 5 Sep 2026 09:48:41 -0600 Subject: [PATCH 6/8] K2GO-386 refactor(dashboard): extract testable updateStreaks; fix firehose streak leak Double-pass code-review fixes on the firehose guard, verified with the real toolchain (tsc strict --noEmit exit 0; node --test 4/4 passing): - Extract the streak logic into a pure exported updateStreaks(firehosing, prev) and add sockets/log-rotate.test.ts (first-time / recurrence / drop-on-absent / empty). The guard now rebuilds the streak map from ONLY the paths firehosing this pass, so a deleted or renamed log no longer leaves a stale entry. Verified NOT issues (checked, not assumed): the try/catch definite-assignment on 'entries' compiles clean under strict tsc; guarding on statSync().size (logical) is correct because logrotate's copytruncate copies by logical size. --- static/dashboard/package.json | 2 +- static/dashboard/sockets/log-rotate.test.ts | 33 +++++++++++++++++++++ static/dashboard/sockets/log-rotate.ts | 33 +++++++++++++++------ 3 files changed, 58 insertions(+), 10 deletions(-) create mode 100644 static/dashboard/sockets/log-rotate.test.ts diff --git a/static/dashboard/package.json b/static/dashboard/package.json index 40bbe6828..fc4442b24 100644 --- a/static/dashboard/package.json +++ b/static/dashboard/package.json @@ -4,7 +4,7 @@ "description": "", "main": "index.js", "scripts": { - "test": "node --require ts-node/register --test sockets/maps.socket.test.ts sockets/rolling-log.test.ts sockets/kolibri.session.test.ts sockets/credentials.test.ts sockets/net-retry.test.ts sockets/services.test.ts", + "test": "node --require ts-node/register --test sockets/maps.socket.test.ts sockets/rolling-log.test.ts sockets/kolibri.session.test.ts sockets/credentials.test.ts sockets/net-retry.test.ts sockets/services.test.ts sockets/log-rotate.test.ts", "test:db": "node --require ts-node/register --test sockets/jobs.test.ts", "typecheck": "tsc --noEmit", "build": "tsc", diff --git a/static/dashboard/sockets/log-rotate.test.ts b/static/dashboard/sockets/log-rotate.test.ts new file mode 100644 index 000000000..3d5b3b980 --- /dev/null +++ b/static/dashboard/sockets/log-rotate.test.ts @@ -0,0 +1,33 @@ +/// +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { updateStreaks } from './log-rotate'; + +test('updateStreaks: a first-time firehose starts at occurrence 1', () => { + const next = updateStreaks(new Set(['/var/log/php8.4-fpm.log']), new Map()); + assert.equal(next.get('/var/log/php8.4-fpm.log'), 1); +}); + +test('updateStreaks: a recurring firehose increments from its previous count', () => { + const prev = new Map([['/var/log/php8.4-fpm.log', 2]]); + const next = updateStreaks(new Set(['/var/log/php8.4-fpm.log']), prev); + assert.equal(next.get('/var/log/php8.4-fpm.log'), 3); +}); + +test('updateStreaks: a path not firehosing this pass is dropped (no stale/leaked entry)', () => { + // gone.log firehosed before but is not in this pass (deleted, renamed, or now sane) → must not linger. + const prev = new Map([ + ['/var/log/gone.log', 5], + ['/var/log/dash-node.log', 1], + ]); + const next = updateStreaks(new Set(['/var/log/dash-node.log']), prev); + assert.equal(next.has('/var/log/gone.log'), false); + assert.equal(next.get('/var/log/dash-node.log'), 2); + assert.equal(next.size, 1); +}); + +test('updateStreaks: an empty firehose set clears everything (a calm tick leaks nothing)', () => { + const prev = new Map([['/var/log/a.log', 3]]); + const next = updateStreaks(new Set(), prev); + assert.equal(next.size, 0); +}); diff --git a/static/dashboard/sockets/log-rotate.ts b/static/dashboard/sockets/log-rotate.ts index ce8d87882..96c4fd239 100644 --- a/static/dashboard/sockets/log-rotate.ts +++ b/static/dashboard/sockets/log-rotate.ts @@ -34,14 +34,25 @@ const FIREHOSE_BYTES = 1024 * 1024 * 1024; // 1 GiB const LOG_DIRS = ['/var/log', '/var/log/nginx']; // Consecutive-truncation count per path, so a RECURRING firehose (an orphan the box cannot stop, that -// just refills) is flagged for the app-side reap (ADR-386 §6). In-memory; cleared when the log is sane. -const firehoseStreak = new Map(); +// just refills) is flagged for the app-side reap (ADR-386 §6). In-memory; only firehosing paths carry +// forward each pass (see updateStreaks), so a deleted/renamed log leaves no stale entry. +let firehoseStreak = new Map(); + +/** Pure: the updated streak counts given the paths firehosing THIS pass and the previous counts — each + * firehosing path +1, everything else dropped (so nothing leaks when a log disappears). No I/O; + * unit-tested (log-rotate.test.ts). */ +export function updateStreaks(firehosing: Set, prev: Map): Map { + const next = new Map(); + for (const p of firehosing) next.set(p, (prev.get(p) || 0) + 1); + return next; +} let timer: NodeJS.Timeout | null = null; /** Pre-logrotate guard: truncate any firehose-sized .log IN PLACE so logrotate never has to copy it. * Reclaims instantly (no copy, no compress). Best-effort per file. */ function guardFirehoseLogs(): void { + const firehosing: Array<{ p: string; size: number }> = []; for (const dir of LOG_DIRS) { let entries: string[]; try { entries = fs.readdirSync(dir); } catch { continue; } @@ -50,17 +61,21 @@ function guardFirehoseLogs(): void { const p = path.join(dir, name); try { const size = fs.statSync(p).size; - if (size <= FIREHOSE_BYTES) { firehoseStreak.delete(p); continue; } + if (size <= FIREHOSE_BYTES) continue; fs.truncateSync(p, 0); // reclaim in place — no copy, no compress - const n = (firehoseStreak.get(p) || 0) + 1; - firehoseStreak.set(p, n); - console.warn( - `[log-rotate] FIREHOSE: truncated ${p} (${size} B) in place, occurrence #${n}` + - (n >= 2 ? ' — recurring; likely an off-proot orphan, needs app-side reap (ADR-386 L3)' : ''), - ); + firehosing.push({ p, size }); } catch { /* best-effort per file */ } } } + // Update the streaks in one pass: only paths that firehosed now carry forward (no stale entries). + firehoseStreak = updateStreaks(new Set(firehosing.map((f) => f.p)), firehoseStreak); + for (const { p, size } of firehosing) { + const n = firehoseStreak.get(p) || 1; + console.warn( + `[log-rotate] FIREHOSE: truncated ${p} (${size} B) in place, occurrence #${n}` + + (n >= 2 ? ' — recurring; likely an off-proot orphan, needs app-side reap (ADR-386 L3)' : ''), + ); + } } /** Run one logrotate pass over the whole config (the K2Go blocks use copytruncate + size, so this is From cfd02e227d4ef79f43a052bc9187ba830ed18fcd Mon Sep 17 00:00:00 2001 From: Luis Guzman Date: Sat, 5 Sep 2026 09:56:45 -0600 Subject: [PATCH 7/8] K2GO-386 docs(diskguard): L3 must confirm before acting on a stale clock-driven signal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture the design rule Luis raised: a clock/calendar-driven signal always leaves gaps, so the actor must file the edges. Because dash-node cannot push (pull-only, verified) and every layer here is clock-driven, a reported firehose can be a tick — or a whole app restart — stale by the time the app reads it (user feels heat, closes + reopens the app, the firehose is gone, but an old dash-node.log line still says "under a busy-loop"). ADR-386 §6 now states: the L3 escalation signal is LIVE state from a /system endpoint (never a parsed dash-node.log line, which persists / a restore can copy in); firehoseStreak stays in-memory ON PURPOSE (resets on restart → self- freshens); and the app RE-PROBES to confirm the firehose is happening now before any destructive action — never acting on the report alone (ADR-5343's stale-state rule at runtime). Also comment log-rotate.ts to mark the warn diagnostic-only, not the signal. --- .../docs/ADR-386-unattended-disk-containment.md | 16 ++++++++++++++++ static/dashboard/sockets/log-rotate.ts | 7 +++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/controller/docs/ADR-386-unattended-disk-containment.md b/controller/docs/ADR-386-unattended-disk-containment.md index 5eec9a5e3..2b379f6ad 100644 --- a/controller/docs/ADR-386-unattended-disk-containment.md +++ b/controller/docs/ADR-386-unattended-disk-containment.md @@ -135,6 +135,22 @@ app can stop an off-proot orphan. - **Parametric.** Thresholds and cadence are parameters we tune (floor, growth-rate, intervals), not hard-coded cliffs; the primary directive right now is **disk free space / abnormal growth**. - **Single surface.** Watching disk pressure / abnormal log growth catches *any* vector, not just php. +- **Confirm before acting — the signal is a HINT, never a command (file the clock's edges).** Because the + channel is **pull, not push** (dash-node cannot call the app; verified — Express loopback, socket.io + retired), and every layer here is clock-driven, any reported state can be a tick — or a whole app + restart — **stale** by the time the app reads and acts on it. So: + - **The escalation signal is LIVE state, not a logged line.** The app reads dash-node's *current* + in-memory firehose state from a `/system/...` endpoint — it must **never parse a `[FIREHOSE]` line out + of `dash-node.log`**, because a log persists: an old line, or a log copied in by a restore, would fire a + false alarm. dash-node's `firehoseStreak` is **in-memory on purpose** — it resets when dash-node (or the + box) restarts, so the exact "user feels heat → closes the app → reopens it, and the firehose is now + gone" case comes back **clean**, with no stale state to act on. + - **The app re-probes before any destructive action.** On reading the hint it runs a **fresh check** — + is the firehose happening *right now* (disk pressure now, a log growing fast now)? — and reaps only on + confirmation. It never acts on the report alone. (ADR-5343's rule — do not act on stale/unowned state — + at runtime; the same "verify, don't suppose" we apply to code, applied to live state.) + - **The signal carries a timestamp; a stale one is ignored.** Freshness is part of the contract, not an + assumption. **Open (design next):** the exact leading-indicator (growth-rate vs absolute), the targeted-vs-full decision, and how relaunch coordinates with the reconciler. The existing `feat/K2GO-386-disk-guard` diff --git a/static/dashboard/sockets/log-rotate.ts b/static/dashboard/sockets/log-rotate.ts index 96c4fd239..d499299eb 100644 --- a/static/dashboard/sockets/log-rotate.ts +++ b/static/dashboard/sockets/log-rotate.ts @@ -34,8 +34,11 @@ const FIREHOSE_BYTES = 1024 * 1024 * 1024; // 1 GiB const LOG_DIRS = ['/var/log', '/var/log/nginx']; // Consecutive-truncation count per path, so a RECURRING firehose (an orphan the box cannot stop, that -// just refills) is flagged for the app-side reap (ADR-386 §6). In-memory; only firehosing paths carry -// forward each pass (see updateStreaks), so a deleted/renamed log leaves no stale entry. +// just refills) is flagged for the app-side reap (ADR-386 §6). In-memory ON PURPOSE: it resets when +// dash-node (or the box) restarts, so a resolved-by-restart firehose comes back clean with no stale +// state (ADR-386 §6, "confirm before acting"). The console.warn below is DIAGNOSTIC/historical — the +// L3 escalation signal must be LIVE state read from a /system endpoint, never a parsed dash-node.log line. +// Only paths firehosing this pass carry forward (see updateStreaks), so a deleted/renamed log never lingers. let firehoseStreak = new Map(); /** Pure: the updated streak counts given the paths firehosing THIS pass and the previous counts — each From 3f85e21e049c8b559201a83e9cd1779c392ba691 Mon Sep 17 00:00:00 2001 From: Luis Guzman Date: Sat, 5 Sep 2026 10:28:13 -0600 Subject: [PATCH 8/8] K2GO-386 docs(diskguard): note the guard-timer-resets-on-restart edge (observed on device) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On-device testing surfaced a real edge: dash-node's 10-min guard timer resets on every restart, so anything that restarts it more often than the interval — a rebuild, a respawn loop, or a desired=DOWN vs pdsm-supervisor flap (a lingering WatchdogEnable=false from a prior Barrier-2 stop, in this case) — blinds the guard. It is the '(a clock-driven mechanism leaves gaps)' hazard made concrete. ADR-386 §8 records it, why the app-side disk-pressure backstop must stay the ultimate net, and why the Phase-2 pdsm-owned trigger (independent of dash-node's process lifetime) is the durable fix. --- controller/docs/ADR-386-unattended-disk-containment.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/controller/docs/ADR-386-unattended-disk-containment.md b/controller/docs/ADR-386-unattended-disk-containment.md index 2b379f6ad..51c830c59 100644 --- a/controller/docs/ADR-386-unattended-disk-containment.md +++ b/controller/docs/ADR-386-unattended-disk-containment.md @@ -175,6 +175,14 @@ guessed — **without bothering the user**: - **L2 trigger:** dash-node's 10-min timer; owns nothing but the timer. If dash-node dies, logs only grow while services are up and the reconciler owns bring-up; if dash-node *wedges*, rotation stalls — the one Phase-1 liveness dependency, bounded by L3, removed in Phase 2 (pdsm-owned trigger). + - **Edge (observed on-device 2026-09-05): the timer resets on every dash-node restart.** The interval + fires only after dash-node runs *uninterrupted* for the full interval, so anything that restarts it more + often than 10 min — a rebuild, a crash/respawn loop, or a desired=DOWN vs pdsm-supervisor flap (which is + what a lingering `WatchdogEnable=false` from a prior Barrier-2 stop caused in testing) — **blinds the + guard**. This is exactly the "a clock-driven mechanism leaves gaps" hazard (§6). Normal operation runs + dash-node for hours, so it fires; but it is why (a) the disk-pressure app-side backstop must remain the + ultimate net if the in-box guard is starved, and (b) the Phase-2 pdsm-owned trigger (a supervisor loop + independent of dash-node's process lifetime) is the durable fix. - **L3:** a poller for the life of a box-up session (started once, stopped on teardown); recovery routes through the ADR-5343 desired-state owner, so no second source of "should the box be up." - **State:** logrotate's own status file + `rotate N`/`compress` bound disk; nothing we add persists