worker: keep directory construction retries on the same lock - #2750
Open
dumbmoron wants to merge 2 commits into
Open
worker: keep directory construction retries on the same lock#2750dumbmoron wants to merge 2 commits into
dumbmoron wants to merge 2 commits into
Conversation
After a directory construction fails, another caller may already be waiting on its lock. Removing that lock from the map lets a new caller create a different lock and construct the same cache path concurrently with the waiting retry. Keep the lock mapped while other callers hold references to it. Remove it only when the map and the finishing caller are its sole owners, with the map locked to prevent new callers from racing the reference check. Add a regression using the existing get_or_create API that verifies a new request waits for the retry's lock and cleanup removes the lock after the final caller finishes.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
When several build actions need the same input directory, the worker uses a shared lock so only one request prepares the cached directory at a time. If that request fails, the worker removes the lock even though another request may still be waiting for it; a new request can then get a separate lock, allowing two requests to prepare the same directory at once. This fix keeps requests sharing the same lock until the last one finishes.
How was this verified?
The regression test sets up the state left by a failed request, with another request still holding the original lock, then asks the cache for the same directory again. Without the fix, the new request goes ahead immediately. With the fix, it waits until the original lock is released, then prepares the directory successfully. The test also checks that the worker removes the lock when it is no longer needed.
Risk
Low. This changes how the worker coordinates requests for the same cached directory. If the lock is removed too early, those requests could still write to the directory at the same time; if it is kept too long, unused locks could accumulate in memory. The test checks both that requests wait and that the lock is eventually removed.
This change is