[shawnburke]

Context

Scoping: The goal of this is to send more context about why calls might fail from the agent back to the server. It's not a functional change for how the tunnel works, but requires fishing context gathering calls through the flows. Its bloated by a proto change and a bunch of new structs.

So, after some customer testing of grpc-tunnel

• Its working well
• There are some issues with stream management and timeouts that need some iteration.

So two PRs coming:

=> This one - now that we have our own stack we can start adding more context. Here we tell the server side WHY a call failed from the agents perspective so we can diagnose these classes of issues from the backend.

=> Upcoming - improve some stream management semantics for better load balancing.

Why

Production shows two dispatch failure shapes that the logs can't tell apart today. Both end as kind: caller_canceled, with nothing about stream state and nothing from the agent:

• 2s /__axon/info probe failures. A token's only stream on one relay pod was busy with another call, so the probe waited for a stream until the dispatcher's 2s client gave up (acquireWait ≈ elapsed ≈ 2s, no callId).
• 60s timeouts on an upstream-facing relay. The call got a stream and reached the agent, and the agent was still waiting on the upstream for headers when the caller gave up. A few never got a stream at all, because slow calls held every stream on that pod.

This PR is the first of a series. It makes every failure explain itself so we can tell "the upstream was too slow" from "slow upstreams ate all our streams". Fixing the per-pod stream sizing that causes the starvation comes next.

What changes

Server: DispatchError explains itself

• Phase: acquire (never got a stream) or awaiting_response (the call was with the agent).
• Streams: the token's state on this server at the moment of failure:
  • total streams, busy streams, and dispatches waiting for one
  • the split per agent instance
  • the oldest calls holding streams, with method, path (query string stripped) and age
• Cause (dispatch.FailureCause): streams_exhausted, upstream_slow, upstream_error, agent_at_capacity, agent_unresponsive, route_rejected, agent_shutdown, tunnel_error, no_tunnel, …

Agent: explains its own failures (CallCancel fields 3–7, all optional, wire-compatible both ways)

• Every early end reports the call's stage (QUEUED_FOR_CAPACITY, ROUTING, AWAITING_UPSTREAM_HEADERS, STREAMING_BODY), a cause, time in that stage, total time, and the upstream host.
• That includes the agent's answer to a cancel sent by the server.
• Stage tracking lives in agent/server/grpctunnel/call_progress.go.

Server holds a cancelled call open for the agent's answer (AGENT_REPORT_GRACE, default 1s; 0 disables)

• When the server gives up on a call the agent is still running, it keeps the call registered and its stream busy until the agent's CallCancel arrives or the grace period runs out.
• Fixes an existing bug: today the stream is released at once. The server can then dispatch onto it while the agent is still running the old call, and the agent's reply is dropped as "frame for unknown call".
• Late CallData during the hold fails fast instead of blocking the stream's receive loop. A stream that dies during the hold ends it immediately.

Surfacing the cause

• Response: the response goes out immediately with X-Axon-Dispatch-Failure: <cause> and X-Axon-Dispatch-Phase. The 404 broker-signature response is left byte-identical.
• Log: the Dispatch failed line is written after the agent answers, so it records the final cause plus agentStage, agentCause, upstreamHost, holders and tokenInstances.
• Log level: a caller cancel caused by something on our side (streams_exhausted, upstream_slow, …) is now logged at Warn instead of Info.
• Metric: new dispatch.failures{tenant_id,integration,alias,cause} counter. The previous tagged metrics.DispatchError helper reused the untagged dispatch.errors name with different labels, so Prometheus panics when it's called. That's likely why it was never wired up.
• README.relay.md documents the causes.

Test plan

• server/dispatch/failure_diagnostics_test.go: reproduces both production shapes:
  • a probe starved by a slow call (deadline and caller-cancel variants)
  • a slow upstream timing out after send
  • the hold-for-report cases: agent reports, agent silent, agent completes late, stream closes mid-hold, no grace
• server/dispatch/failure_cause_test.go: table tests for the cause mapping
• server/adapters/dispatch_failure_cause_test.go: end-to-end log fields and response headers for streams_exhausted, upstream_slow, agent_unresponsive and agent_at_capacity
• agent/server/grpctunnel/call_report_test.go: the agent's stage and cause for a server cancel, the agent's own deadline, an upstream error, a capacity queue timeout, and a rejected route
• go test -race ./... in server/ and in agent/server/grpctunnel
• After deploy: the 2s probe failures should log cause=streams_exhausted with holders, and the 60s failures should log cause=upstream_slow with upstreamHost

🤖 Generated with Claude Code

[keithfz]

i think if something fails mid read here there might be a case where the stream gets reused.

E.g. dispatch -> cancelCall -> recvDone = true

[shawnburke]

Good catch, confirmed. A request-body failure cancels the call and holds it while the agent answers, and then Dispatch cancelled it a second time with notifyAgent=false. That second cancel treated "don't notify the agent" as "the agent is done", so it set recvDone and released the stream while the agent was still running the call. Fixed in 8ac580c. cancelCall never releases a held call now: only the agent's own final frame, the grace timer, a failed cancel send, or the stream closing ends the hold. There's a repro test in TestDispatch_RequestBodyFailureHoldsStreamUntilAgentReports.

[shawnburke]

You found a bug!
Reward: 🏆

[keithfz]

one thing that might be worth double checking on but LGTM