fix: durable commit retries — journal, replay, and /v1/events fallback (v0.5.0)

AUDIT.md

@@ -1,6 +1,7 @@
 # Cycles Protocol v0.1.25 — Client (Python) Audit
 
-**Date:** 2026-07-10 (unreleased — `TENANT_CLOSED` + `LIMIT_EXCEEDED` error-code support. `TENANT_CLOSED` per runtime spec v0.1.25.13 (`cycles-protocol-v0.yaml`, runcycles/cycles-protocol#125): `ErrorCode.TENANT_CLOSED` enum member, `TenantClosedError` subclass wired into the lifecycle error-code→exception mapping (reservation-creation surfaces), `CyclesProtocolError.is_tenant_closed()` helper. `LIMIT_EXCEEDED` per runtime spec v0.1.25.12 (revision 2026-07-04, HTTP 429 rate limiting): enum-only member matching the `BUDGET_FROZEN`/`BUDGET_CLOSED` pattern, classified retryable at both the enum and exception layers (429 is transient; previously it fell through to `UNKNOWN`, which happened to be retryable, so semantics are unchanged — now typed). Enum reordered to mirror spec declaration order. Both purely additive; previously both codes fell through the `ErrorCode.from_string` forward-compat path to `UNKNOWN`. See the dated entries at the end of this file. 398 tests pass at 100% coverage.),
+**Date:** 2026-07-27 (v0.5.0 — durable commit retries: on-disk pending-commit journal with next-run replay, bounded atexit flush, and `POST /v1/events` recovery for commits that land after reservation expiry; async retry-task GC fix; `retry_enabled=False` now journals instead of silently dropping. Review hardening: per-identity journal partitioning (tenant-keyed when configured — rotation-safe), 429 transient with `Retry-After` incl. first-attempt commits (no more release-on-429), 401/403 retained, `0700`/`0600` journal permissions, process-wide flush deadline; round 3: first-attempt 401/403 journaled (not released), Retry-After persisted across restarts, PBKDF2 30k rounds, unique journal temp files. See the dated entry below. 491 tests pass at 100% coverage.),
+2026-07-10 (v0.5.0 — `TENANT_CLOSED` + `LIMIT_EXCEEDED` error-code support. `TENANT_CLOSED` per runtime spec v0.1.25.13 (`cycles-protocol-v0.yaml`, runcycles/cycles-protocol#125): `ErrorCode.TENANT_CLOSED` enum member, `TenantClosedError` subclass wired into the lifecycle error-code→exception mapping (reservation-creation surfaces), `CyclesProtocolError.is_tenant_closed()` helper. `LIMIT_EXCEEDED` per runtime spec v0.1.25.12 (revision 2026-07-04, HTTP 429 rate limiting): enum-only member matching the `BUDGET_FROZEN`/`BUDGET_CLOSED` pattern, classified retryable at both the enum and exception layers (429 is transient; previously it fell through to `UNKNOWN`, which happened to be retryable, so semantics are unchanged — now typed). Enum reordered to mirror spec declaration order. Both purely additive; previously both codes fell through the `ErrorCode.from_string` forward-compat path to `UNKNOWN`. See the dated entries at the end of this file. 398 tests pass at 100% coverage.),
 2026-07-09 (README + docstring transport-error documentation fix, no version bump — see the dated entry at the end of this file. `CyclesTransportError` is exported but never raised by the SDK; README and its docstring now describe the actual `status == -1` surfacing.),
 2026-07-03 (integration-test-only, no version bump — `test_health_check` now probes the public `/actuator/health/readiness` endpoint instead of aggregate `/actuator/health`, which requires `X-Admin-API-Key` since cycles-server v0.1.25.45 and fails closed with 500 when the server has no admin key configured. The old assertion had failed the org nightly Full-Stack Integration every night since 2026-06-28. No library code change.),
 2026-05-22 (v0.4.3 — `expires_from`/`expires_to` and `finalized_from`/`finalized_to` ISO-8601 window-filter passthrough on `list_reservations` per `cycles-protocol-v0.yaml` revision 2026-05-22; closes the Python-client side of runcycles/cycles-server#162. No code change — `**query_params` already forwards arbitrary kwargs. Added sync + async regression tests; unlike `from`/`to` the new param names are plain kwargs (no Python-reserved-word workaround needed). 393 tests pass at 100% coverage.),
@@ -12,6 +13,43 @@
 
 ---
 
+## 2026-07-27 — Durable commit retries (journal + /v1/events fallback)
+
+Pending commits no longer exist only in memory: the retry engines journal
+each one to disk (`~/.runcycles/commit-journal`, config/env overridable)
+before retrying, replay survivors on the next run, and flush bounded at
+interpreter exit. A commit answered `RESERVATION_EXPIRED` — where the server
+has already returned the reserved budget to the pool — is recovered via
+`POST /v1/events` (spec-conformant `EventCreateRequest`, commit idempotency
+key reused, recovery markers in `metadata`). Also fixes the async engine's
+unreferenced-task GC hazard and the silent drop under `retry_enabled=False`.
+Post-review (PR #89) hardening, round 1: journal records are partitioned
+into per-identity subdirectories so co-located clients with different
+credentials never replay or 401-discard each other's records and replay
+claims cannot cross identities; HTTP 429 / `LIMIT_EXCEEDED` on retried
+attempts is transient (record retained, `Retry-After` honored) instead of
+a terminal discard; the atexit flush enforces one process-wide
+`retry_flush_timeout` deadline instead of per-engine. Round 2: a
+rate-limited *first* commit attempt now schedules a retry in all four
+lifecycle variants instead of releasing the reservation (a release
+returned budget for spend that already happened); the identity fingerprint
+uses the configured tenant when set, so API-key rotation no longer orphans
+pending records, and 401/403 retains the journal entry instead of
+discarding it; journal directories/files are created `0700`/`0600` where
+supported. Round 3: first-attempt 401/403 also journals instead of
+releasing (same class as the 429 gap, all four variants); the `Retry-After`
+floor is persisted as an absolute `not_before_ms` and restored on replay;
+PBKDF2 rounds reduced 600k → 30k (~20 ms cold, cache 256 — input is a
+high-entropy machine credential, rounds only defend the weak-key
+fallback); journal temp files get unique per-writer names so concurrent
+processes cannot publish each other's partial writes. Fleet self-review
+round: ASCII-explicit sanitizer + pinned cross-SDK fingerprint vectors
+(interop with TS/Java identity dirs), blank-tenant normalization, 1-hour
+clamp on honored Retry-After and restored floors, status-410 expired
+trigger, unclassifiable-4xx retention (never release/discard on codeless
+or unknown-code responses), base-dir permissions, stale-temp reaping.
+506 tests pass at 100% coverage.
+
 ## 2026-07-26 — Python publishing workflow maintenance
 
 Dependabot PRs #82–#86 update the SHA-pinned PyPI trusted-publishing action to

CHANGELOG.md

@@ -5,7 +5,26 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.5.0] - 2026-07-27
+
+Durable commit retries. Previously a commit that failed transiently lived only in an in-memory daemon thread (or an unreferenced asyncio task): a process exit — even a clean one — dropped it, and once the reservation's grace period elapsed the server's expiry sweep returned the reserved budget to the pool, permanently under-counting spend that had already happened. Pending commits are now journaled to disk before retry, replayed on the next run, flushed (bounded) at interpreter exit, and — when the reservation has already expired — recovered via `POST /v1/events`, the spec's post-hoc direct-debit endpoint.
+
+### Added
+
+- `runcycles.journal`: file-per-commit `CommitJournal` (atomic write, idempotent replay). Config: `journal_enabled` (default `True`), `journal_dir` (default `~/.runcycles/commit-journal`), `retry_flush_timeout` (default 10 s); env `CYCLES_JOURNAL_ENABLED`, `CYCLES_JOURNAL_DIR`, `CYCLES_RETRY_FLUSH_TIMEOUT`. Records are partitioned into per-identity subdirectories (directories `0700`, files `0600` where supported) keyed by a non-secret PBKDF2-HMAC-SHA256 fingerprint of the server plus principal — the configured `tenant` when set (rotation-safe: any same-tenant credential can settle the records), else the API key — so clients with different servers or principals sharing a journal directory never replay each other's records, and one identity's replay claim cannot starve another's. The first engine created per identity replays surviving entries; corrupt files are renamed `*.corrupt` for operator triage.
+- Event fallback: when a commit (first attempt or retry) returns `RESERVATION_EXPIRED`, the SDK posts the spend to `/v1/events` reusing the commit's idempotency key, with `metadata.recovered_reservation_id` / `metadata.recovery_reason` markers and no `overage_policy` (spec default `ALLOW_IF_AVAILABLE` never rejects). Applies to the `@cycles` lifecycles and both streaming context managers. `RESERVATION_FINALIZED` is still treated as settled.
+- `flush()` on both retry engines; a process-wide `atexit` hook flushes sync engines under one shared `retry_flush_timeout` deadline (not per engine) so daemon retry threads aren't killed mid-backoff on clean exit and shutdown time stays bounded regardless of engine count.
+- Rate-limit awareness end to end: HTTP 429 / `LIMIT_EXCEEDED` on the *first* commit attempt schedules a retry instead of releasing the reservation (a release would return budget for spend that already happened) in all four lifecycle variants, passing the server's `Retry-After` into the engine; on retried commit/event attempts the journal entry is retained and the next attempt waits at least `Retry-After` (consistent with `ErrorCode.is_retryable`). The `Retry-After` floor is persisted in the journal record as an absolute `not_before_ms`, so a restart during a long server-mandated wait does not replay into the window early.
+- Authentication failures (401/403) on any commit attempt — first or retried — and on event fallbacks journal the spend instead of releasing or discarding it, so spend recorded during a key misconfiguration or rotation window replays once credentials are fixed.
+
+### Fixed
+
+- Self-review hardening (fleet-wide adversarial review): filename sanitization is ASCII-explicit, matching the TS/Java SDKs, so sibling SDKs sharing a tenant identity directory can always discard records this SDK wrote (and vice versa); the two cross-SDK PBKDF2 fingerprint vectors are now pinned in this suite; a whitespace-only `tenant` falls back to the key principal (matching Java); honored `Retry-After` values and restored journal floors are clamped to 1 hour; HTTP 410 triggers the expired/event-fallback path even when the response body was mangled in transit; a 4xx with no recognizable protocol error code (proxy error pages, forward-compat future codes) is no longer treated as a genuine rejection — the journal entry is retained and the reservation is never released; the base journal directory is also permission-tightened and stale temp files from crashed writers are reaped after 1 hour.
+- With `retry_enabled=False`, failed commits were dropped with only a warning; they are now journaled for replay (the old drop behavior remains only when the journal is also disabled).
+- `AsyncCommitRetryEngine` created retry tasks without holding a reference, so a pending retry could be garbage-collected mid-flight; task references are now held until completion.
+- Commit retries exhausting, or landing after expiry, no longer lose the spend record silently: the journal entry is retained (transient exhaustion) or the event fallback records it (expiry).
+
+---
 
 `TENANT_CLOSED` + `LIMIT_EXCEEDED` error-code support. `TENANT_CLOSED` implements the runtime spec v0.1.25.13 revision of `cycles-protocol-v0.yaml` ([runcycles/cycles-protocol#125](https://github.com/runcycles/cycles-protocol/pull/125)): servers return HTTP 409 `error=TENANT_CLOSED` on reservation create/commit/release/extend when the owning tenant is CLOSED (mirrors governance spec Rule 2). `LIMIT_EXCEEDED` closes the same class of gap for the runtime spec v0.1.25.12 revision (2026-07-04): HTTP 429 rate-limit responses carry `error=LIMIT_EXCEEDED` plus `Retry-After` / `X-RateLimit-Reset` headers.
 

README.md

@@ -227,9 +227,52 @@ CyclesConfig(
     retry_initial_delay=0.5,
     retry_multiplier=2.0,
     retry_max_delay=30.0,
+    retry_flush_timeout=10.0,
+    journal_enabled=True,
+    journal_dir=None,  # None → ~/.runcycles/commit-journal
 )
 ```
 
+### Commit durability
+
+A commit records spend that has already happened, so the SDK never lets one
+exist only in memory. Every commit scheduled for background retry is first
+journaled to disk and removed only on a terminal outcome. Records live under
+`journal_dir` (default `~/.runcycles/commit-journal`) in a per-identity
+subdirectory (directories `0700`, files `0600` where supported) keyed by a
+non-secret fingerprint of the server plus principal — the configured
+`tenant` when set (stable across API-key rotation; any same-tenant
+credential can settle the records), otherwise the API key. Clients using
+different servers or principals on the same machine never replay each
+other's records. Without a tenant configured, rotating the API key orphans
+pending records under the old fingerprint directory; records are plain
+JSON, so moving them into the new identity directory is safe — replay is
+idempotent:
+
+- **Process exit**: an `atexit` hook waits up to `retry_flush_timeout` seconds
+  (one process-wide budget shared across all engines) for in-flight retries;
+  anything unfinished stays journaled and is replayed automatically the next
+  time the process creates a client lifecycle.
+- **Rate limiting**: HTTP 429 / `LIMIT_EXCEEDED` responses are transient
+  everywhere — a rate-limited *first* commit attempt is scheduled for retry
+  (never released, which would return budget for spend that already
+  happened), the journal entry is kept, and the next attempt waits at least
+  the server's `Retry-After`. The floor is persisted as an absolute
+  timestamp, so a restart mid-wait still honors it.
+- **Authentication failures**: 401/403 on any commit attempt — first or
+  retried — journals the spend (never releases it) and stops the current
+  run's attempts, so spend recorded during a key misconfiguration or
+  rotation window replays once credentials are fixed.
+- **Reservation expired before the commit landed**: the server has already
+  returned the reserved budget to the pool, so the SDK re-records the spend
+  via `POST /v1/events` (the protocol's post-hoc direct-debit endpoint),
+  tagging the event metadata with `recovered_reservation_id` for
+  reconciliation. Commit and event requests both carry idempotency keys, so
+  replays across restarts (or from multiple processes sharing a journal
+  directory) are exactly-once.
+- Set `journal_enabled=False` (or `CYCLES_JOURNAL_ENABLED=false`) to opt out
+  and restore fire-and-forget behavior.
+
 ### Default client / config
 
 Instead of passing `client=` to every `@cycles` decorator, set a module-level default:

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "runcycles"
-version = "0.4.3"
+version = "0.5.0"
 description = "Python AI agent budget control — enforce LLM cost limits, tool permissions, and multi-tenant policies before agent actions execute."
 readme = "README.md"
 license = "Apache-2.0"

runcycles/config.py

@@ -31,6 +31,13 @@ class CyclesConfig:
     retry_initial_delay: float = 0.5
     retry_multiplier: float = 2.0
     retry_max_delay: float = 30.0
+    # Bounded wait (seconds) at interpreter exit for in-flight commit retries.
+    # 0 disables the wait; journaled entries replay on the next run either way.
+    retry_flush_timeout: float = 10.0
+
+    # Durable journal for pending commits (survives process restarts)
+    journal_enabled: bool = True
+    journal_dir: str | None = None  # None → ~/.runcycles/commit-journal
 
     @classmethod
     def from_env(cls, prefix: str = "CYCLES_") -> CyclesConfig:
@@ -40,7 +47,8 @@ def from_env(cls, prefix: str = "CYCLES_") -> CyclesConfig:
         CYCLES_APP, CYCLES_WORKFLOW, CYCLES_AGENT, CYCLES_TOOLSET,
         CYCLES_CONNECT_TIMEOUT, CYCLES_READ_TIMEOUT, CYCLES_RETRY_ENABLED,
         CYCLES_RETRY_MAX_ATTEMPTS, CYCLES_RETRY_INITIAL_DELAY,
-        CYCLES_RETRY_MULTIPLIER, CYCLES_RETRY_MAX_DELAY.
+        CYCLES_RETRY_MULTIPLIER, CYCLES_RETRY_MAX_DELAY,
+        CYCLES_RETRY_FLUSH_TIMEOUT, CYCLES_JOURNAL_ENABLED, CYCLES_JOURNAL_DIR.
         """
         base_url = os.environ.get(f"{prefix}BASE_URL", "")
         api_key = os.environ.get(f"{prefix}API_KEY", "")
@@ -66,4 +74,7 @@ def from_env(cls, prefix: str = "CYCLES_") -> CyclesConfig:
             retry_initial_delay=float(os.environ.get(f"{prefix}RETRY_INITIAL_DELAY", "0.5")),
             retry_multiplier=float(os.environ.get(f"{prefix}RETRY_MULTIPLIER", "2.0")),
             retry_max_delay=float(os.environ.get(f"{prefix}RETRY_MAX_DELAY", "30.0")),
+            retry_flush_timeout=float(os.environ.get(f"{prefix}RETRY_FLUSH_TIMEOUT", "10.0")),
+            journal_enabled=os.environ.get(f"{prefix}JOURNAL_ENABLED", "true").lower() == "true",
+            journal_dir=os.environ.get(f"{prefix}JOURNAL_DIR"),
         )