fix(scoring): resolve indexers once per batch, not once per result in parallel - #863
Open
m4bard wants to merge 1 commit into
Open
fix(scoring): resolve indexers once per batch, not once per result in parallel#863m4bard wants to merge 1 commit into
m4bard wants to merge 1 commit into
Conversation
… parallel ScoreSearchResults fans out with Task.WhenAll, and each task built a SearchResultScorer over the same scoped IIndexerRepository and queried it for the result's indexer. IIndexerRepository is scoped and the ListenArrDbContext behind it is scoped, so a batch of N results issued N concurrent queries against one context. EF rejects a second operation started on a context while another is in flight. The symptom is not a failed request. The scorer catches the exception and logs at Debug, leaving indexerRetention at 0 and skipping the Usenet detection that sets isNzb. So age and retention checks silently score against the wrong assumptions for whichever results lost the race, and the batch comes back plausibly ordered and quietly wrong. Resolve each distinct indexer once, sequentially, before fanning out, and pass the results into the scorer. Three callers reach this: the quality profile controller, download submission, and the six-hour automatic search sweep. This is also fewer queries rather than merely safer ones. A batch commonly carries many results across a handful of indexers, so it goes from one query per result to one per distinct indexer. IndexerSearchWorkflow already fetches its indexers before its own fan-out, so this makes the two agree. The single-result ScoreSearchResult keeps its old behaviour and still queries, since there is nothing to batch there. Asserting on the EF exception would mean racing it, so the test counts the overlap directly: a stub repository records the highest number of calls in flight at once. It reports 12 without the change and 1 with it, and also pins the query count at one per distinct indexer rather than one per result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ScoreSearchResultsfans out withTask.WhenAlland each task queried the same scopedIIndexerRepository, so a batch of N results issued N concurrent queries against one scopedDbContext. This resolves each distinct indexer once before the fan-out and passes the results in.Full write-up in #862.
Changes
Fixed
QualityProfileService.ScoreSearchResultsresolves the batch's distinct indexers sequentially up front and hands them to the scorer.SearchResultScorertakes an optional pre-resolved lookup and prefers it over querying.The single-result
ScoreSearchResultis unchanged and still queries; there is nothing to batch there, and it stays on the interface as it was.Why it matters more than a caught exception suggests
The scorer already catches the failure and logs at
Debug, so nothing fails. What it loses isindexerRetention, which stays 0, and the branch that setsisNzbfrom the indexer's type. Age and retention checks then score that result against assumptions that do not match its indexer. The batch comes back ordered plausibly and quietly wrong, and at the default log level there is nothing to see.Three callers reach it: the quality profile controller, download submission, and the six-hour automatic search sweep.
Fewer queries, not just safer ones
A batch commonly carries many results across a handful of indexers. This goes from one query per result to one per distinct indexer, so it removes redundant work as well as the overlap.
IndexerSearchWorkflowalready fetches its indexers before its own fan-out rather than inside it, so this makes the two agree rather than introducing an approach.Testing
Asserting on the EF exception would mean racing it, which makes for a flaky test and a weak claim. The new test counts the overlap directly: a stub
IIndexerRepositoryrecords the highest number of calls it ever had in flight at once, with a small delay so the overlap is observable rather than a scheduling accident.It reports 12 concurrent without the change and 1 with it, over twelve results across three indexers. It also pins the call count at 3, one per distinct indexer, so a future change that restores per-result querying fails even if it somehow avoids overlapping.
Full suite: 3,030 passed, 0 failed, 125 skipped, against a 3,029 baseline on
03958c15.Notes
The context's lifetime is the load-bearing fact here, so I verified it rather than inferring it from the registration. Reading the
ServiceDescriptorback out of the built collection givesListenArrDbContext => Scoped, with the repositories scoped as well.AddDbContextFactory(..., ServiceLifetime.Singleton)atPersistenceRegistrationExtensions.cs:25registers the factory as a singleton and the context as scoped.I have not reproduced wrong scores end to end against a running instance, and the issue says so. A collision needs the queries to actually overlap in time, which is machine-dependent. What is deterministic is that the calls are issued concurrently against a context that permits one at a time.