Repository navigation
Retry failed experiments and properly handle runs that were submitted multiple times - #83
Merged
Merged
Conversation
A failed experiment was a dead end. generate_scaling_experiments skipped any layout whose branch the manager already held, with no regard for status, and run_experiments submitted only the new ones, so the only way back was delete_experiments - which sweeps the laboratory and removes the branch directory, taking the evidence of the failure with it. Neither upstream package needed persuading. The generator applies a perturbation block to a branch it already has rather than skipping it, the runner pulls those commits into a run directory it has already cloned, and its job manager resubmits whatever is short of the runs asked for while refusing to submit alongside a live job. The manager's own guards were the whole blockage. generate_scaling_experiments(regenerate_failed=True) therefore stops skipping a failed experiment. What it generates is worked out afresh from the arguments of the call rather than replayed from what was generated before, so correcting the walltime, the allocation, the control options or the model's own layout_config_changes and calling it again is what puts the correction on the branch. Only failed experiments are regenerated: a running one would have its branch rewritten underneath a live job, and one that finished has results its branch should go on matching. The experiment already held is kept as it stands, status included - a fresh one would read as new, and the next look at the record of the failed run, still on disk, would put it straight back. run_experiments(retry_failed=True) submits the failed ones alongside the new. One that failed after archiving output is swept first with payu sweep --hard, since the runner counts what is archived against the runs wanted and would otherwise submit nothing while this went on to call the experiment running. A sweep that fails stops the call rather than leaving that claim standing. The same flag is on the Cylc manager, so the two keep one signature, but what it offers a failed suite is the same rose suite-run as any other rather than a restart, which the suite may refuse. Building the perturbation block moves into _perturbation_block, the generation loop having grown past what ruff's complexity limit allows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Payu names each job file after the scheduler job that produced it, under a directory named after the run number. A run that failed archives no output, so the number it was does not move on: submitting it again writes a second file beside the first rather than replacing it, the new job having a new name. A run directory therefore holds a file per attempt, not a file per run. The reader allowed only for the latter. It sorted on the run number alone, so every attempt at one run compared equal and a stable sort left whatever order the directory happened to be read in to decide - which turned out to be the failed attempt in all of two hundred trials. An experiment queued to run again read back as failed, and the core count could be read from the wrong job. Attempts are now ordered by the number their name begins with, which counts up with each one whether it is a scheduler job id or, where there was no scheduler, the time the run started. A name with no number to read falls back to when the file was last written, and sorts behind every name that has one; a file that has gone sorts first, since Payu removes the job file of a queued job it finds has already exited, and so a listing can outlive what it lists. _latest_run_job and profiling_logs share that ordering rather than each globbing for themselves. profiling_logs had the same ambiguity in a different shape: it keys the Payu log by run number, so of two attempts it kept whichever the glob yielded last and dropped the other without saying so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
micaeljtoliveira
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #83 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 16 16
Lines 1276 1319 +43
=========================================
+ Hits 1276 1319 +43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.