Repository navigation
fix: [NACT-2932] retry GitLab MR creation and stop blocking on traceback rendering - #650
Open
feiyunzhou wants to merge 1 commit into
Conversation
tech1ndex
self-requested a review
October 6, 2026 11:22
feiyunzhou
force-pushed
the
nact-2932-fix-foxops-mr-creation-race-and-timeout-handling
branch
from
October 8, 2026 12:46
8d59890 to
39ee978
Compare
…ack rendering foxops asks GitLab to open a merge request immediately after pushing the branch. GitLab answers from a branch cache that can lag the push, rejects the request with `Source branch "<x>" does not exist`, and foxops did not retry. The resulting traceback then took ~52s to log, because structlog's ConsoleRenderer falls back to rich's traceback formatter whenever rich is installed, and that formatter renders every frame's local variables. It ran twice per failed request: once in catch_all, once more after Starlette re-raises and uvicorn logs it. Rendering is synchronous on the event loop and happens before the error response is sent, so replies missed the 60s reverse-proxy timeout and callers saw a misleading 504. - retry the merge request POST for up to 10s while GitLab reports the pushed source branch as missing - raise MergeRequestCreationFailedError with GitLab's own message instead of httpx's "Client error '400 Bad Request' for url ..." - render plain tracebacks, without frame locals
feiyunzhou
force-pushed
the
nact-2932-fix-foxops-mr-creation-race-and-timeout-handling
branch
from
October 8, 2026 13:34
39ee978 to
f6c581e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY?
Nightly
nactl updateruns fail intermittently. foxops pushes the branch andasks GitLab to open an MR in the same breath; GitLab answers from a branch cache
that can lag the push and rejects it with
Source branch "<x>" does not exist.foxops did not retry. Logging the resulting error then took ~52s, which pushed
the reply past the 60s reverse-proxy timeout, so the caller saw a misleading 504
instead of the real GitLab error.
WHAT?
pushed source branch as missing. Safe to retry: GitLab only sends this message
when it created nothing.
MergeRequestCreationFailedErrorcarrying GitLab's own message, insteadof httpx's
Client error '400 Bad Request' for url ....ConsoleRenderersilently switches torich's formatter when
richis installed, and that formatter reprs everyframe's local variables.
ADDITIONAL NOTES
The traceback was rendered twice per failed request — once in
catch_all, oncemore after Starlette re-raises and uvicorn logs it. Rendering is synchronous on
the event loop, so it stalled every other in-flight request too, and the first
render happens before the error response is sent.
Not touched: the mergeability-check loop in
merge_request()can still spend upto 25s of the same 60s budget. Separate concern.