Don't reproduce frozen stages with --force - #11084
Open
arose26 wants to merge 1 commit into
Open
Conversation
Forcing reproduction rewrote a frozen stage's dependency hashes in dvc.lock even though its command is never re-run, so the lock stopped recording what the stage was actually built against.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11084 +/- ##
==========================================
+ Coverage 90.68% 90.98% +0.30%
==========================================
Files 504 505 +1
Lines 39795 41149 +1354
Branches 3141 3264 +123
==========================================
+ Hits 36087 37439 +1352
- Misses 3042 3071 +29
+ Partials 666 639 -27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #11004
dvc repro --forcerewrites a frozen stage's dependency hashes indvc.lock, while plaindvc reproleaves them alone. Using the reproducer from the issue:The stage is still
frozen: truethroughout, and its command is never re-run — only the recorded dependency hashes move. That leaves the lock no longer describing what the stage was actually built against, which is the thing freezing is for. Thedvc freezedocs say a frozen stage is considered unchanged, so--forcereproducing it reads as a bug rather than intended behaviour.Cause
Stage.reproduce()short-circuits every skip check whenforceis set:Without
--forcea frozen stage exits through thatelif—changed_deps()returnsFalsefor a frozen stage — so_reproduce_stagenever callsstage.dump()and the lock is untouched. With--forceit falls through toself.run(...).run()still refuses to execute the command (elif not self.frozen and self.cmd), but the stage is re-saved afterwards and the lock is rewritten.Change
forceis neutralised for frozen non-import stages, so they go through the same checks as an unforced run.It deliberately does not
return Noneoutright.changed()ischanged_stage() or changed_deps() or changed_outs(), and only the middle one is short-circuited byfrozen— so a frozen stage whose outputs are missing still reachesrun()and restores/verifies them, exactly as it does today without--force. Import stages are excluded, matching the existingstage.frozen and not stage.is_importcondition in_reproduce_stageand the import handling inrun().Tests
test_repro_frozen_forcefreezes a stage, changes an upstream dependency, runsreproduce(force=True), and asserts both that nothing is reproduced and that the stage'sdvc.lockentry is byte-identical. It fails onmainand passes with this change.tests/func/repro/andtests/unit/stage/pass: 164 passed. Two failures intests/unit/stage/—test_fill_from_lock_use_appropriate_checksumandtest_dump_nondefault_hash— fail identically onmainwithout this change, so they are pre-existing and unrelated.ruff checkandruff format --checkare clean on the changed files (the one remainingtoo-many-positional-argumentsindvc/stage/__init__.pyis pre-existing and untouched).