Repository navigation
Preserve checkpoint storage failures across Ray transport - #23
Merged
Merged
Conversation
Signed-off-by: Robert Washbourne <rawashbourne@gmail.com>
Signed-off-by: Robert Washbourne <rawashbourne@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When an asynchronous checkpoint failed, its error retained a local Future and Ray replaced the original failure because the Future contains an unpicklable lock. Serialize the same exception type, message, failure details and timeout flag with checkpoint_future=None; keep the original local Future for settlement and retirement.
Validation: 17 allocated runtime291 CPU controls pass, including two real-Ray actor round-trips through tq.save_checkpoint with injected storage replies, local unsettled/wrong-owner refusal, settlement and explicit retirement; pickle/cloudpickle and existing checkpoint receipt controls pass. Ruff and compileall pass. This does not fix slow storage or qualify a production runtime; no snapshot or publication semantics change.
AI-assisted implementation and tests.