[shawnburke]

Summary

Adds a failure hook to the Go and Python handler SDKs (WithFailureHook / on_failure) that fires for every failed invocation. The hook receives the original message and a reason that separates handler failures from Axon failures. A customer can then route failed webhooks to a store of their own choosing and replay them later. Before this change, those messages failed silently. Tracks COR2-1098.

The three failure cases asked for, and what the hook receives for each:

Scenario	Who detects it	reason	source
Handler raises / returns an error / panics	SDK	HANDLER_ERROR	APPLICATION
Handler runs past its timeout	SDK	TIMEOUT	APPLICATION, or RUNTIME if it was waiting on a Cortex API call or for a free handler
Invocation never answered (client hung or gone)	Agent	TIMEOUT	RUNTIME
Agent queue full (webhook got 503)	Agent	NO_RESOURCES	RUNTIME, caller_rejected=true
Expired in the agent queue before any client took it	Agent	NO_RESOURCES	RUNTIME

Each failure is reported once, by whichever side saw it.

Python usage

from cortex_axon import AxonClient, FailureAction, FailureSource
from cortex_axon.handler import cortex_webhook


@cortex_webhook(id="employee-sync")
def employee_sync(context):
    update_records(context.args["body"])   # raising here is a HANDLER_ERROR


def on_failure(info, failure):
    dead_letter_topic.publish({
        "invocation_id": info.invocation_id,
        "handler": info.handler_name,
        "attempt": info.attempt,
        "reason": failure.reason.name,      # HANDLER_ERROR / TIMEOUT / NO_RESOURCES
        "source": failure.source.name,      # APPLICATION / RUNTIME
        "caller_rejected": failure.caller_rejected,
        "message": info.args,               # body, content-type, url
    })
    if failure.source == FailureSource.RUNTIME and info.attempt < 3:
        return FailureAction.RETRY             # optional: agent re-queues it


AxonClient(scope=globals(), on_failure=on_failure).run()

Design

• Protocol (additive only, per AGENTS.md):
  • New DispatchEvents server stream and RetryInvocation RPC.
  • New FailureReason / FailureSource enums and InvocationFailure / DispatchEvent messages.
  • DispatchRequest.failure_events.
  • DispatchHandlerInvoke.attempt and .deadline.
• Failure events have their own stream. The Python SDK stops reading Dispatch while all its handler slots are busy, which is exactly when the agent has failures to report. Events are read on a separate thread and run on a separate pool.
• Failure events are never dropped. The agent keeps an unbounded queue per dispatch id; axon_handler_failure_events_pending shows a consumer that has fallen behind. The agent only queues events for clients that set failure_events, so older SDKs see no change and the queue can't grow for them. An event whose send fails is put back at the front of the queue.
• One report per failure. The agent waits 5s past an invocation's timeout for the client's report before reporting the failure itself, so a client that enforces its own timeout gets there first. completeInvocation now checks and removes the outstanding request atomically, so two reports can't both win.
• Absolute deadline. A saturated client reads invocations out of the gRPC stream buffer long after they were sent. Without a deadline it would run invocations the agent had already reported as failed, and a hook replaying them would process them twice. I reproduced this against a real agent: under saturation, 109 of 110 stale invocations still ran. With the deadline, both SDKs skip work past it and cap a handler's timeout at it. The agent normally runs as a localhost sidecar, so clock skew isn't a concern. An older agent leaves the field unset and the SDKs behave as before.
• Retry re-queues through the agent, so it respects the dispatch queue, with attempt+1 and a hardcoded cap of 5 attempts. The hook sees attempt and decides when to stop.
• Saturated webhooks still return 503. The hook is notified either way, with caller_rejected=true, so a replay system can avoid double-delivering a message the sender will also retry.
• No new config.
• Python timeouts. A thread can't be stopped, so at the deadline the SDK reports the timeout itself, runs the hook, and drops anything the handler does afterwards.
• Go fix. invokeHandler used to let a timed-out handler keep writing to the report while it was being sent, a race that was already on main. The outcome now comes back over a channel, and logs go through a lock.

Compatibility

Agent	SDK	Behavior
new	old	Unchanged. Nothing is queued, and the old SDK ignores the unknown fields.
old	new	Hooks still fire for SDK-side failures. DispatchEvents / RetryInvocation return Unimplemented: the SDK logs once and stops asking; it does not redial in a loop.

Known limits

• If the handler process is wedged, hooks can't run. Events wait in the agent and are delivered when the process reconnects.
• Events live in agent memory, so an agent restart loses any that haven't been delivered.
• Delivery to the SDK is best-effort: an event whose send fails is queued again, but there is no ack after the SDK has received it.
• Under SDK-side saturation, invocations the agent already sent come back as TIMEOUT/RUNTIME ("did not report") rather than NO_RESOURCES. The source is still right.
• A durable agent-side sink would close the restart gap and is worth a follow-up.

Test plan

• Agent:
  • unit tests: failure queue ordering, requeue, 10k events with none dropped, queue-full / expired-in-queue events, no events without a subscriber, retry attempts, the retry cap, and lookup after re-registration;
  • e2e over real gRPC: unreported → TIMEOUT → retry → attempt 2; a client-reported failure is not repeated; retry rejected at the cap;
  • webhook 503 emits caller_rejected.
  • go test ./... passes.
• Go SDK (go test -race ./...):
  • reason/source mapping for error, panic, timeout, timeout during an API call, and timeout capped by the deadline;
  • retry; a panicking hook is contained; the hook doesn't run on success;
  • event stream → hook → retry;
  • Unimplemented agent is not redialed;
  • the failure_events flag.
• Python SDK (26 tests, ruff clean):
  • the same mapping; the hook doesn't run on success;
  • a slow handler is APPLICATION and its late result is dropped;
  • an expired invocation is not started;
  • a deadline-capped timeout is RUNTIME;
  • hook errors are contained;
  • 20 events reach the hook while every handler slot is held;
  • an older agent is handled.
• Manual end-to-end: real agent binary (DRYRUN) with the Python SDK and the HTTP webhook endpoint:
  • Raising handler: HANDLER_ERROR/APPLICATION, then a successful retry as attempt 2.
  • Slow handler: one TIMEOUT/APPLICATION, and the late result was dropped.
  • 110 webhooks against 1 handler slot: each message reached the hook exactly once, and 109 stale invocations were skipped instead of run.

🤖 Generated with Claude Code

[shawnburke]

What this means for handler authors

The problem

When a webhook message failed to process, it disappeared without a trace. A message could fail because:

• the handler raised;
• it ran out of time;
• Axon was too busy to hand it to a handler at all.

Your code got no signal. The only way to find out was to notice later that data hadn't updated. Even when you did notice, you couldn't tell whether your handler or Axon was to blame.

The solution

You can now register a failure hook with the Axon SDK. Axon calls it for every message that fails, including the ones that never reached your handler. It passes:

• the original message: the webhook body, content type and URL, plus the handler name and an invocation id;
• why it failed: one of
  • HANDLER_ERROR: your handler raised.
  • TIMEOUT: it didn't finish in time.
  • NO_RESOURCES: Axon had no free capacity to run it.
• whose fault it was:
  • APPLICATION: your code.
  • RUNTIME: Axon itself, or a Cortex API call it was waiting on.

What the hook does is up to you: save the message to a topic, a table or a file, raise an alert, or ask Axon to try it again. Axon doesn't store or replay anything itself; your hook decides what happens.

Example

from cortex_axon import AxonClient, FailureAction, FailureSource
from cortex_axon.handler import cortex_webhook


# A normal webhook handler. If it raises, the failure hook is called.
@cortex_webhook(id="employee-sync")
def employee_sync(context):
    update_records(context.args["body"])


# Called for every message that fails.
def on_failure(info, failure):
    # Keep the failed message so it can be replayed later.
    dead_letter_topic.publish({
        "invocation_id": info.invocation_id,
        "handler": info.handler_name,
        "attempt": info.attempt,
        "reason": failure.reason.name,   # HANDLER_ERROR, TIMEOUT or NO_RESOURCES
        "source": failure.source.name,   # APPLICATION or RUNTIME
        "message": info.args,            # the original webhook: body, content-type, url
    })

    # Optionally, have Axon try Axon-side failures again (up to 5 attempts).
    if failure.source == FailureSource.RUNTIME and info.attempt < 3:
        return FailureAction.RETRY


AxonClient(scope=globals(), on_failure=on_failure).run()

Good to know

• Busy periods: when Axon is too busy to accept a webhook, the sender still gets a 503 so it can resend. Your hook is told too, with failure.caller_rejected = True, so you can avoid replaying a message that will also be resent.
• Every failure, once: the hook gets each failed message exactly once, even under heavy load.
• Retries: returning FailureAction.RETRY queues the message again as a new attempt. Returning nothing leaves it alone.
• Timeouts in Python: a handler that runs past its timeout is reported as a TIMEOUT at the deadline. Anything it does after that is ignored.
• Upgrade both sides: to see failures that only Axon can detect, such as a full queue or a handler that never answered, upgrade both the Axon agent and the SDK. With an older agent, the hook still fires for failures inside your handler.
• Go: the same hook is available as axon.WithFailureHook.