Scheduled refresh: bound the tagging step, say what it costs, and scan runs first

[mgoldey]

Implements docs/superpowers/specs/2026-08-23-scheduled-refresh-design.md,
which has sat at status: proposed since 2026-08-23, and flips it to
accepted with both open questions resolved in the document.

The failure it prevents was live

attest tag in the generated refresh script was unbounded. At the measured
2.3s/item a 1000-item backlog is a ~40-minute run against an hourly tick — so
with flock -n, every subsequent wakeup logs SKIP: previous run still holding lock, and the ingest half, the deterministic part that must succeed, never
runs either. A slow best-effort step starving a fast mandatory one.

Three changes

Bound the tagging step. The script now calls attest tag --limit "$TAG_BUDGET", computed in-script from three named variables that each carry
their measurement in a comment rather than arriving as a bare integer:

TAG_SECONDS_PER_ITEM=2.3   # measured on this machine (see design spec)
TAG_BUDGET_MULTIPLIER=0.5  # leave the next tick a free lock even at full budget
TAG_INTERVAL_SECONDS=3600  # must match the cron schedule above

782 items at hourly — half the interval, so a run that hits its budget still
leaves the next tick a free lock. REFRESH_INTERVAL_SECONDS is new and
step_schedule uses it too, so the interval and the "17 * * * *" cron literal
cannot drift apart.

Say what a run will cost, before it runs. run_tagging printed nothing
until it finished. It now announces to stderr — deliberately, because the
refresh script redirects stdout to /dev/null, so stderr is the only place a
cron run's output survives.

The estimate is derived, not the constant. estimate_seconds_per_item reads a
50-row trailing window of item_features.tagged_at, takes consecutive deltas,
and excludes any delta more than 10x the median as a boundary between two
separate runs rather than a slow item. That exclusion is the whole trick: one
hour-long cron gap left in would swamp every genuine ~2s per-item delta.

MEASURED against the live database's 11,972 tagged rows: 2.558 s/item —
close to but independent of the hardcoded 2.3, which is the point, since a 12B
model on a slower box is a different number. Edge cases verified: an injected
hour-long gap still yields 2.000; identical timestamps and a single row fall
back to the named constant rather than dividing by zero.

Scan runs first. attest runs scan is deterministic, needs no model, and
completes in ~1s on 1045 runs — yet it was absent from the refresh while both
model-dependent steps were present. It now runs first, ahead of ingest, gated on
RESEARCH_ROOT being set and pointing at something that exists.

The spec's "must succeed" and "skip when unconfigured" are reconciled by making
the gate the escape hatch rather than the exit code: unconfigured skips
silently, configured-and-broken is fatal exactly like ingest, because it is
exactly as deterministic and model-free.

Resolved open questions

• Per-item estimate source. item_features.tagged_at deltas, no new schema.
  The spec proposed created_at, which does not exist — the real column is
  tagged_at TEXT NOT NULL DEFAULT (datetime('now')), and the wording is
  corrected in the document.
• Prompting for RESEARCH_ROOT during install. Declined. ddd560b exists
  precisely because --check reported BROKEN over optional wiring a self-hoster
  never asked for; a prompt for an optional env var repeats that mistake in a
  different costume. Discoverability is still a real problem — a line in
  --check output would inform without nagging — but that is a separate change.

Verification

• tests/test_install.py test_install_e2e.py test_install_skills.py test_features.py test_docs_site.py test_architecture.py → 169 passed, 1 skipped
• Estimator checked against 11,972 live rows and six synthetic edge cases
• ruff format, ruff check, ty, uv.lock, complexity ratchet, bandit,
  xenon all Passed; the full-suite pytest hook was killed by host memory
  pressure mid-run rather than by a failure, so the scoped run above stands in
• tests/test_golden_paths.py::test_every_output_block_of_an_offline_path_is_real
  was reported as a pre-existing failure during development. It is not: 13
  passed on a clean tree and 13 passed with these changes applied. The
  original failure was host load (15–49 load average), which is exactly the trap
  docs/measurement-lessons.md warns about.

🤖 Generated with Claude Code