Repository navigation
Circuit breaking for multi-relays and metrics instrumentation - #5
Merged
Merged
Conversation
Two things production applications ask for that the library could not answer. A circuit breaker for multi-relays. Retry and failover alone still pay the full cost of a relay that is down: every request waits for it to time out before moving on. WithCircuitBreaker(failureThreshold, breakDuration) now skips a relay that failed that many times in a row, and lets exactly one trial request through once the break elapses. It composes with WithRetry - the retries run first, and only an exhausted relay counts as a failure - and applies to the Failover and FirstSuccessful strategies, where a per-relay failure is actually observed. Every transition is derived from TimeProvider, so tests drive it with a FakeTimeProvider instead of sleeping, and GetCircuitSnapshots() exposes live state for a health endpoint. Metrics. The library emitted traces but nothing you could alert on. Chains, multi-relay operations and circuit transitions now publish to a Meter named Relay, so a single AddMeter(RelayDiagnostics.MeterName) gives you execution counts, durations, circuit transitions, the number of open circuits and the relays skipped because of them. Instruments are checked for listeners before any tag list is built, so nothing is allocated while nobody is watching. Test parallelisation is disabled for the suite: the meter is process-wide, so a MetricCollector would otherwise observe measurements from tests running alongside it. The whole suite still finishes in under a second.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two gaps that show up once this library is actually carrying production traffic.
Circuit breaker
Failover with retry still pays the full cost of a dead relay on every request - each one waits for the timeout before moving on. The breaker remembers the outcome:
After two consecutive failures the primary is skipped outright; once the break elapses exactly one trial request goes through, and the circuit closes again only if that succeeds. From the new example, with the primary down:
Details worth reviewing:
WithRetry- retries run first, only an exhausted relay counts as one failureFailover/FirstSuccessful, the strategies where a per-relay failure is observed; broadcast and parallel are untouchedTimeProvider, so tests useFakeTimeProviderrather than sleepingMultiRelay<T>.GetCircuitSnapshots()exposes live state for a health endpointWithCircuitBreaker, so nothing changes for existing registrationsMetrics
There were traces but nothing to alert on. Chains, multi-relay operations and circuit transitions now publish on a
MeternamedRelay:relay.adapter_chain.executions/.durationrelay.multi_relay.operations/.durationrelay.circuit_breaker.transitionsrelay.circuit_breaker.open_circuits> 0relay.circuit_breaker.skippedEvery instrument is checked for listeners before a tag list is built, so nothing is allocated when nobody is subscribed.
Notes
MetricCollectorin one test would otherwise pick up measurements from tests running next to it - that was a real intermittent failure, not a hypothetical. The suite still runs in ~190 ms.Microsoft.Extensions.*test packages moved to 10.0.11 to avoid a downgrade warning pulled in byMicrosoft.Extensions.Diagnostics.Testing.