Skip to content

Add feed health observability and critical-lane alerts - #8

Closed
MEHuggins wants to merge 14 commits into
mainfrom
agent/feed-health-observability
Closed

Add feed health observability and critical-lane alerts#8
MEHuggins wants to merge 14 commits into
mainfrom
agent/feed-health-observability

Conversation

@MEHuggins

Copy link
Copy Markdown
Contributor

What changed

  • Adds scripts/feed_health.py, an independent post-build auditor that:
    • distinguishes current-run rewrites from preserved stale outputs
    • probes every configured source for HTTP status, redirects, content type, bot challenges, and candidate-link discovery
    • classifies lanes as HEALTHY, DEGRADED, STALE_LAST_KNOWN_GOOD, EMPTY, CRITICAL_STALE, or CRITICAL_EMPTY
    • preserves last_nonempty_at and counts consecutive nonproductive runs
    • generates health/feed-health.json, health/feed-health.md, and a public health/index.html dashboard
    • adds a feed-health link to the generated root index
  • Adds scripts/check_feed_health.py to publish GitHub Actions warnings/errors and fail the run when a critical lane is empty or stale.
  • Extends generated-output validation to verify health-report schema, area coverage, dashboard presence, and root-index linking.
  • Makes gzip output deterministic with gzip -n, eliminating timestamp-only .gz churn.
  • Adds unit tests for critical-empty, stale-last-known-good, degraded-source, RSS-discovery, and bot-challenge behavior.

Root cause addressed

The existing workflow treats structurally valid empty feeds as healthy and silently preserves old files when a lane produces zero usable items. Fresh .gz timestamps also obscure this because compression rewrites every archive even when the underlying feed did not run successfully.

Operational impact

After merge, each scheduled collection will commit a machine-readable and human-readable health report. Critical lanes such as agentic AI, quantum security, Elon Musk, robotics, AI policy, workforce change, and planetary warning will produce explicit GitHub Actions errors when empty or stale. The report is committed before the final policy gate, so diagnostics remain available even when the run is marked failed.

Validation performed

  • python -m py_compile passed for all new and modified Python files.
  • python -m unittest discover -s tests -p "test_*.py" passed: 5 tests.
  • The modified workflow parsed successfully as YAML.
  • End-to-end source probing has not been executed in this environment because outbound runtime network access is unavailable; the first manual workflow dispatch should be treated as the live integration test.

Safety

  • Draft PR only.
  • No changes were made to main.
  • The existing feed collector was not rewritten; health auditing is isolated as a post-build layer.

Copy link
Copy Markdown
Contributor Author

Superseded by clean Draft PR #10. PR #8 served as the live integration branch and proved the health system could rebuild all 111 areas with zero critical and zero empty lanes, but its workflow-generated feed outputs made the review too large. The branch and test history are being retained for evidence; this PR should not be merged.

@MEHuggins MEHuggins closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant