effect-broker

Retries can turn one intended charge, refund, email, or production change into several, so effect-broker records stable intent, enforces the target's real safety contract, and stops in an explicit unknown state when a safe retry cannot be proved.

Run the crash proof

git clone https://github.com/prathamesh-git9/effect-broker.git
cd effect-broker
python -m pip install -e ".[dev]"
python -m examples.crash_matrix.runner

The final command uses SQLite-backed subprocesses and produces this result:

scenario | failpoint | target_count | broker_status | result
--- | --- | ---: | --- | ---
idempotent | after_target_commit_before_receipt | 1 | succeeded | PASS
reconcilable | after_target_commit_before_receipt | 1 | succeeded | PASS
unsafe | after_target_commit_before_receipt | 1 | manual_review | PASS
ALL PASS

What the tests prove

Validation for this README was run against committed code at 61906b2 with Python 3.13.5: ruff check . passed, and pytest reported 68 passed and 11 skipped.

• The crash matrix kills a worker with os._exit(137) after the durable target commits but before the broker records a receipt. Recovery runs the same lease sweep a deployment runs, and leaves one target effect for the idempotent and reconcilable cases; the unsafe case stops in manual_review instead of redispatching.
• A dispatch that outlives one lease period is not reclaimed while its worker keeps the lease renewed, and is reclaimed once the renewals stop. Recovery bumps the version, so a resurrected worker's receipt loses the compare-and-swap.
• Concurrent recovery processes are version-fenced, and a stale worker cannot apply its transition after another process wins.
• Deterministic tests cover operation-key deduplication, payload-conflict rejection, state-machine transitions, lease renewal and expiry, cancellation, reconciliation, API status codes, CLI behavior, and redaction in metrics and spans.
• The in-memory and SQLite stores pass the shared conformance cases. The eleven skips are the Postgres conformance/crash cases because no Postgres test DSN was configured for this local run; CI runs them against a real Postgres 16.

Not implemented / not proven

• This is not an unconditional exactly-once system. If a target supplies neither idempotency nor authoritative reconciliation, the broker may stop without retrying and require manual review.
• The repository does not ship production adapters for payment, email, ticketing, or infrastructure providers. The executable examples use local test targets; a deployment must supply adapters and contracts for its real systems.
• The local result above does not prove the Postgres path or behavior under a real network partition; those tests require EFFECT_BROKER_TEST_DSN or DATABASE_URL.
• It is not an agent planner, workflow engine, task queue, or LLM gateway.

How the safety boundary works

The caller supplies a stable semantic operation_key. The broker stores that intent with an immutable payload hash and a pinned tool contract, derives one downstream idempotency key for every attempt, and routes ambiguous outcomes by the contract's safety class. It never hashes arguments to guess business intent: two legitimate $10 refunds need two different operation keys.

Safety class	Required target contract	Honest guarantee
transactional	Target mutation and broker receipt share one database transaction.	One committed effect per operation key while that shared transaction is used.
idempotent	The target honors a stable key, rejects payload drift, and retains the result for the declared recovery horizon.	One observable effect while that contract remains true.
reconcilable	An authoritative lookup by business key settles within a declared bound.	Recovery converges when the lookup is truthful; otherwise it stops for review.
unsafe	No idempotency and no authoritative lookup.	No broker-issued duplicate retry, but possible omission; no exactly-once claim.

An eventually consistent search can prove that an effect exists but cannot prove that it does not. Likewise, a lease only coordinates workers; it does not make a remote side effect atomic. Those limitations are represented in the state machine rather than hidden behind retry language.

Library example

import asyncio

from effect_broker.config import Settings, build_broker
from effect_broker.models import EffectRequest

broker = build_broker(Settings(contracts_path="examples/contracts.yaml"))
result = asyncio.run(
    broker.submit(
        "tenant-1",
        EffectRequest(
            operation_key="agent-run:run_42:step:refund_01",
            tool="charge",
            arguments={"amount_usd": 49.99, "order_id": "A-10428"},
            requested_by="support-agent",
        ),
    )
)
print(result.effect.status)  # prepared; a worker performs dispatch

Submitting the same operation key and payload returns the same effect. Reusing the key with a different payload is a conflict.

Runtime surfaces

effect-broker serve
effect-broker worker
effect-broker reconciler
effect-broker doctor

• HTTP: submit and inspect effects, replay receipts without dispatch, cancel before dispatch, trigger reconciliation, filter by status, and record an operator resolution for manual_review.
• CLI: serve, worker, reconciler, submit, inspect, list, cancel, contracts validate, doctor, and crash-demo. There is no CLI switch that forces an unsafe redispatch.
• MCP: management tools and a mutating-tool proxy that requires an operation_key; safety class and retry policy remain code-controlled.

The core includes SQLite and Postgres stores, a contract registry, dispatcher, reconciler, CAS-fenced cancellation, Prometheus metrics, and the crash-injection harness. See docs/NEXT_PROJECT_SPEC.md for the remaining design boundary.

License

MIT. See LICENSE.