fix(backup): detect missing daily attempts - #820
Conversation
|
Warning Review limit reachedNext included review available in 40 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
BrainLayer ratchetEvery Value below was measured by this run. A row this machine cannot measure says
🟢 GREEN measured, within budget · 🔴 RED measured, out of budget — a finding to clear before merge · ⚪ n/a not measurable on this machine, never guessed. No RED rows. Measured on Linux/x86_64 · measured |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_5c8fda72-f21b-43d1-ba25-36ca85172637) |
|
@coderabbitai review @codex review Please review the latest head — brainlayerCodex-9451c648 (worker) · codex/gpt-5.6-sol |
|
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review Retry after GitHub now reports PR head — brainlayerCodex-9451c648 (worker) · codex/gpt-5.6-sol |
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| "error": str(exc), | ||
| "traceback": traceback.format_exc(), | ||
| } | ||
| _append_terminal_failure(log_path, result) |
There was a problem hiding this comment.
🟡 Medium brainlayer/jsonl_backup.py:528
When run_backup has already uploaded and verified the archive, an exception from the post-backup _enqueue_run_summary call is recorded as a new status="failed" receipt, so the last log line falsely reports a critical backup failure. Handle summary-enqueue errors separately and preserve the verified backup receipt as the terminal backup status.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/jsonl_backup.py around line 528:
When `run_backup` has already uploaded and verified the archive, an exception from the post-backup `_enqueue_run_summary` call is recorded as a new `status="failed"` receipt, so the last log line falsely reports a critical backup failure. Handle summary-enqueue errors separately and preserve the verified backup receipt as the terminal backup status.
| return None | ||
| return finish_slow(stage, f"health-check exceeded {config.max_duration_seconds:.0f}s during {stage}") | ||
|
|
||
| result.jsonl_backup, jsonl_backup_issue = inspect_jsonl_backup_health( |
There was a problem hiding this comment.
🟠 High brainlayer/health_check.py:1229
run_health_check can exceed max_duration_seconds or consume excessive memory before reaching any slow-check handling. inspect_jsonl_backup_health reads and splits the entire append-only log synchronously at line 1229, so a large forever_files backup result blocks the existing deadline checks; bound or make this inspection deadline-aware.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/health_check.py around line 1229:
`run_health_check` can exceed `max_duration_seconds` or consume excessive memory before reaching any slow-check handling. `inspect_jsonl_backup_health` reads and splits the entire append-only log synchronously at line 1229, so a large `forever_files` backup result blocks the existing deadline checks; bound or make this inspection deadline-aware.
| except FileNotFoundError: | ||
| status = JsonlBackupHealth(state="missing", detail=f"attempt log does not exist: {resolved}") | ||
| return status, HealthIssue("jsonl_backup_attempt_missing", "critical", status.detail) | ||
| except OSError as exc: |
There was a problem hiding this comment.
🟠 High brainlayer/health_check.py:857
A non-UTF-8 attempt log raises UnicodeDecodeError and aborts run_health_check, so no jsonl_backup_attempt_invalid issue or health state is emitted. Catch the decoding error with the existing unreadable-log path so the receipt is classified as invalid.
- except OSError as exc:
+ except (OSError, UnicodeDecodeError) as exc:🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/health_check.py around line 857:
A non-UTF-8 attempt log raises `UnicodeDecodeError` and aborts `run_health_check`, so no `jsonl_backup_attempt_invalid` issue or health state is emitted. Catch the decoding error with the existing unreadable-log path so the receipt is classified as invalid.
| try: | ||
| payload = json.loads(lines[-1]) | ||
| except json.JSONDecodeError as exc: | ||
| status = JsonlBackupHealth(state="invalid", detail=f"latest attempt receipt is malformed JSON: {exc}") | ||
| return status, HealthIssue("jsonl_backup_attempt_invalid", "critical", status.detail) | ||
| if not isinstance(payload, dict): | ||
| status = JsonlBackupHealth(state="invalid", detail="latest attempt receipt is not a JSON object") | ||
| return status, HealthIssue("jsonl_backup_attempt_invalid", "critical", status.detail) |
There was a problem hiding this comment.
🟡 Medium brainlayer/health_check.py:864
A trailing non-JSON diagnostic on the shared stdout/stderr log makes inspect_jsonl_backup_health report jsonl_backup_attempt_invalid even when the preceding line is a fresh successful receipt. Because the function unconditionally parses lines[-1], scan backward for the latest JSON receipt instead of requiring the literal final log line to be JSON.
- try:
- payload = json.loads(lines[-1])
- except json.JSONDecodeError as exc:
- status = JsonlBackupHealth(state="invalid", detail=f"latest attempt receipt is malformed JSON: {exc}")
- return status, HealthIssue("jsonl_backup_attempt_invalid", "critical", status.detail)
- if not isinstance(payload, dict):
- status = JsonlBackupHealth(state="invalid", detail="latest attempt receipt is not a JSON object")
+ payload = None
+ for line in reversed(lines):
+ try:
+ candidate = json.loads(line)
+ except json.JSONDecodeError:
+ continue
+ if isinstance(candidate, dict):
+ payload = candidate
+ break
+ if payload is None:
+ status = JsonlBackupHealth(state="invalid", detail="attempt log contains no JSON receipt object")🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/health_check.py around lines 864-871:
A trailing non-JSON diagnostic on the shared stdout/stderr log makes `inspect_jsonl_backup_health` report `jsonl_backup_attempt_invalid` even when the preceding line is a fresh successful receipt. Because the function unconditionally parses `lines[-1]`, scan backward for the latest JSON receipt instead of requiring the literal final log line to be JSON.
| if backup_status == "created" and verified is True: | ||
| return JsonlBackupHealth(state="verified_bundle", **common), None |
There was a problem hiding this comment.
🟡 Medium brainlayer/health_check.py:909
The created branch reports verified_bundle after only local archive verification, so run_backup(upload=False) is considered healthy even though no backup was uploaded. Remove this branch so only a verified upload or verified no-op is healthy.
- if backup_status == "created" and verified is True:
- return JsonlBackupHealth(state="verified_bundle", **common)
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/health_check.py around lines 909-910:
The `created` branch reports `verified_bundle` after only local archive verification, so `run_backup(upload=False)` is considered healthy even though no backup was uploaded. Remove this branch so only a verified upload or verified no-op is healthy.
| normalized_now = now.astimezone(UTC) if now.tzinfo is not None else now.replace(tzinfo=UTC) | ||
| age_seconds = max(0.0, (normalized_now - attempted_at).total_seconds()) |
There was a problem hiding this comment.
🟡 Medium brainlayer/health_check.py:888
A future-dated receipt is treated as zero age, so a verified backup remains healthy until that timestamp is reached even if no new attempts occur. The max(0.0, ...) clamp hides producer clock errors; reject or alarm on attempted_at values later than now before evaluating freshness.
normalized_now = now.astimezone(UTC) if now.tzinfo is not None else now.replace(tzinfo=UTC)
- age_seconds = max(0.0, (normalized_now - attempted_at).total_seconds())
+ age_seconds = (normalized_now - attempted_at).total_seconds()
+ if age_seconds < 0:
+ status = JsonlBackupHealth(
+ state="invalid",
+ status=backup_status,
+ attempted_at=attempted_at.isoformat(),
+ age_seconds=age_seconds,
+ verified=verified,
+ archive=archive,
+ detail="latest attempt receipt is dated in the future",
+ )
+ return status, HealthIssue("jsonl_backup_attempt_invalid", "critical", status.detail)🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @src/brainlayer/health_check.py around lines 888-889:
A future-dated receipt is treated as zero age, so a verified backup remains healthy until that timestamp is reached even if no new attempts occur. The `max(0.0, ...)` clamp hides producer clock errors; reject or alarm on `attempted_at` values later than `now` before evaluating freshness.
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
HOLD: do not merge this PR before #815. Mandatory order is #815 → #820 → #819. After #815 merges, this branch must be rebased onto post-#815
— brainlayerCodex-9451c648 (worker) · codex/gpt-5.6-sol |
Co-Authored-By: brainlayerCodex-9451c648 (worker) running gpt-5.6-sol <noreply@anthropic.com>
085ea06 to
db8d611
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_c7c4921f-9458-477b-b051-07aa4a1774bf) |
|
Post-#815 rebase handoff: READY, not merged.
The HOLD/draft state is cleared. Per lane ownership, Etan/lead merges. — brainlayerCodex-9451c648 (worker) · codex/gpt-5.6-sol |
1 participant
Summary
attempted_atRoot-cause evidence
This PR intentionally does not re-enable the disabled LaunchAgent or alter retention. It adds source/test detection only; merge does not prove the detector is installed or live.
Verification
BRAINLAYER_PREPUSH_SCOPE=changed-only git push: 67 focused Python tests, 3 MCP tests, 40 isolated tests, 1 Bun test, and the FTS shell regression all passed.ulimit -n 8192: 5,189 passed; 3 unrelated order-dependent failures intest_retro_self_pollution_quarantine.py; all three pass when rerun alone.Closes #818
— brainlayerCodex-9451c648 (worker) · codex/gpt-5.6-sol
Note
Medium Risk
Scheduled health checks will now fail critically when the default backup attempt log is missing or stale, which changes operator alerting behavior without modifying backup scheduling itself.
Overview
Adds independent detection of missing or bad JSONL backup runs to
brainlayer health-check, so a job that never starts cannot look healthy.The backup job now writes a timezone-aware
attempted_aton every terminal receipt (successful no-op/upload, and failures caught inmain()), and appends timeout/exception outcomes to the same JSONL attempt log via_append_terminal_failure.run_health_checkreads the latest log line, classifies it (verified_bundle,no_op,failed,missing,invalid,stale), and raises critical issues when the receipt is absent, malformed, unverified, failed, or older than 36 hours (overridable withBRAINLAYER_JSONL_BACKUP_MAX_AGE_SECONDS/BRAINLAYER_JSONL_BACKUP_LOG_PATH). Legacy receipts withoutattempted_atinfer time from archive name at 05:00 local. Results surface onHealthCheckResult.jsonl_backupand ops docs are updated.Reviewed by Cursor Bugbot for commit db8d611. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add JSONL backup receipt health check to detect missing daily attempts
attempted_atstamp, including no-op results and terminal failures written byjsonl_backup.maininspect_jsonl_backup_healthin health_check.py parses the last nonblank log line, classifies it (verified no-op, verified bundle, failed, stale, missing, malformed), and returns a critical issue when the receipt is absent, undated, older than the configured threshold, or not a recognized successrun_health_checknow invokes the inspector and adds any returned issue to the aggregate result, so a bad receipt can fail the overall health checkBRAINLAYER_JSONL_BACKUP_LOG_PATHandBRAINLAYER_JSONL_BACKUP_MAX_AGE_SECONDS(one-hour minimum); a legacy archive-date fallback covers older receipts withoutattempted_at_jsonl_backup_attempt_timerejects naive (non-timezone-aware) timestamps and unrecognized archive names; any existing receipt withoutattempted_atmust match the legacy archive-date format or it will be classified as undated/criticalMacroscope summarized db8d611.