Skip to content

Circuit breaking for multi-relays and metrics instrumentation - #5

Merged
TarasKovalenko merged 1 commit into
mainfrom
relay-circuit-breaker-and-metrics
Aug 23, 2026
Merged

TarasKovalenko merged 1 commit into
mainfrom
relay-circuit-breaker-and-metrics

Conversation

@TarasKovalenko

Copy link
Copy Markdown
Owner

Two gaps that show up once this library is actually carrying production traffic.

Circuit breaker

Failover with retry still pays the full cost of a dead relay on every request - each one waits for the timeout before moving on. The breaker remembers the outcome:

services.AddMultiRelay<IPaymentProvider>(config => config
        .AddRelay<PrimaryProvider>()
        .AddRelay<BackupProvider>()
        .WithStrategy(RelayStrategy.Failover)
        .WithRetry(maxAttempts: 3, delay: TimeSpan.FromMilliseconds(100))
        .WithCircuitBreaker(failureThreshold: 2, breakDuration: TimeSpan.FromSeconds(30)))
    .Build();

After two consecutive failures the primary is skipped outright; once the break elapses exactly one trial request goes through, and the circuit closes again only if that succeeds. From the new example, with the primary down:

Request 1 -> charged 19.99 via backup (primary calls so far: 1)
Request 2 -> charged 19.99 via backup (primary calls so far: 2)
Request 3 -> charged 19.99 via backup (primary calls so far: 2)   <- skipped, circuit open
Request 4 -> charged 19.99 via backup (primary calls so far: 2)
=== Primary recovers, break elapses ===
Request 5 -> charged 19.99 via primary                             <- trial succeeded, circuit closed

Details worth reviewing:

  • composes with WithRetry - retries run first, only an exhausted relay counts as one failure
  • applies to Failover/FirstSuccessful, the strategies where a per-relay failure is observed; broadcast and parallel are untouched
  • every transition derives from TimeProvider, so tests use FakeTimeProvider rather than sleeping
  • MultiRelay<T>.GetCircuitSnapshots() exposes live state for a health endpoint
  • off unless you call WithCircuitBreaker, so nothing changes for existing registrations

Metrics

There were traces but nothing to alert on. Chains, multi-relay operations and circuit transitions now publish on a Meter named Relay:

builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddSource(RelayDiagnostics.SourceName))
    .WithMetrics(m => m.AddMeter(RelayDiagnostics.MeterName));
Instrument Kind
relay.adapter_chain.executions / .duration Counter / Histogram, tagged with kind, result type, step count, outcome
relay.multi_relay.operations / .duration Counter / Histogram, tagged with strategy, operation, relay count, outcome
relay.circuit_breaker.transitions Counter, tagged with relay type and target state
relay.circuit_breaker.open_circuits UpDownCounter - alarm on > 0
relay.circuit_breaker.skipped Counter

Every instrument is checked for listeners before a tag list is built, so nothing is allocated when nobody is subscribed.

Notes

  • Test parallelisation is disabled for the assembly. The meter is process-wide, so a MetricCollector in one test would otherwise pick up measurements from tests running next to it - that was a real intermittent failure, not a hypothetical. The suite still runs in ~190 ms.
  • Microsoft.Extensions.* test packages moved to 10.0.11 to avoid a downgrade warning pulled in by Microsoft.Extensions.Diagnostics.Testing.
  • 341 tests pass (17 new), all 19 examples run, build is clean with warnings-as-errors and the trim/AOT analyzers on. No breaking changes; version 2.1.0.

Two things production applications ask for that the library could not answer.

A circuit breaker for multi-relays. Retry and failover alone still pay the
full cost of a relay that is down: every request waits for it to time out
before moving on. WithCircuitBreaker(failureThreshold, breakDuration) now
skips a relay that failed that many times in a row, and lets exactly one
trial request through once the break elapses. It composes with WithRetry -
the retries run first, and only an exhausted relay counts as a failure - and
applies to the Failover and FirstSuccessful strategies, where a per-relay
failure is actually observed. Every transition is derived from TimeProvider,
so tests drive it with a FakeTimeProvider instead of sleeping, and
GetCircuitSnapshots() exposes live state for a health endpoint.

Metrics. The library emitted traces but nothing you could alert on. Chains,
multi-relay operations and circuit transitions now publish to a Meter named
Relay, so a single AddMeter(RelayDiagnostics.MeterName) gives you execution
counts, durations, circuit transitions, the number of open circuits and the
relays skipped because of them. Instruments are checked for listeners before
any tag list is built, so nothing is allocated while nobody is watching.

Test parallelisation is disabled for the suite: the meter is process-wide,
so a MetricCollector would otherwise observe measurements from tests running
alongside it. The whole suite still finishes in under a second.
@TarasKovalenko
TarasKovalenko merged commit ae281bb into main Aug 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant