Repository navigation
Do not keep the original code in the git history of SWE-smith tasks - #398
Merged
Marc-Alexandre Côté (MarcCote) merged 2 commits intoNov 16, 2026
Conversation
SWESmithEnv committed the bug patch on top of the code from before the bug. For bug patches that only modify or delete tracked files, `git checkout HEAD~1 -- .` gave back exactly the code the gold patch produces, and submitting it scored like the gold patch without any debugging. The image's .git also kept the commit objects of the SWE-smith task branches after the remote was removed. After applying the bug patch, replace .git with a new repository whose only commit is the buggy code. It tracks the files the old index tracked after the bug patch, including files the patch adds (`git commit -am` left those untracked).
Contributor
Author
|
@microsoft-github-policy-service agree [company="Holdout"] |
Contributor
Author
|
@Holdout-github-policy-service agree |
Contributor
Author
|
@microsoft-github-policy-service agree company="Holdout" |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the repository-history reset and setup-failure handling changes.
No open issues 🎉
What changed in this PR
Addresses #397 by rebuilding SWE-smith task repositories with a single buggy baseline commit, removing the Git history that allowed agents to recover pre-bug code without debugging.
Changes:
- Refreshes the index and applies bug patches with
--index, ensuring added files are tracked. - Replaces
.gitwhile preserving the tracked-file set, and stops setup on command failures or timeouts. - Adds regression checks for history removal, tracked additions, and gold-patch reversal.
| File | Description |
|---|---|
tests/gym/envs/test_swe_smith.py |
Verifies the single-commit history, blocked parent checkout, and handling of patch-added files. |
debug_gym/gym/envs/swe_smith.py |
Rebuilds Git history around the buggy code and checks setup failures. |
Review effort: Balanced
Marc-Alexandre Côté (MarcCote)
approved these changes
Nov 16, 2026
Marc-Alexandre Côté (MarcCote)
merged commit Nov 16, 2026
80c77c2
into
microsoft:main
11 checks passed
Collaborator
|
Thank you for the fix! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
SWESmithEnv.setup_terminalcommitted the bug patch on top of the code from before the bug. For bug patches that only modify or delete tracked files,git checkout HEAD~1 -- .therefore gave back exactly the code the gold patch produces. (Files a patch adds stayed untracked and are left in place.) In the tasks below, that followed by submit scored the same as the gold patch, without debugging. The image's.gitalso kept the commit objects of the SWE-smith task branches after the remote was removed.After applying the bug patch, the env now replaces
.gitwith a new repository whose only commit is the buggy code:git apply --indexstages the bug patch, so files it adds are tracked. Before,git commit -amleft them untracked. The index is refreshed first (git update-index -q --refresh), because--indexcompares stat info. In the tldextract image the cached stat info is stale (all 28 tracked files are stat-dirty with no content change); before, the bug'sgit commit -amrefreshed it as a side effect.git ls-files -z→git add -f --pathspec-from-file, with literal pathspecs). It does not usegit add -A, which would newly track untracked build files in some images, e.g.records.egg-info/in the kennethreitz/records image.main, and the identity staysdebug-gym/<>. The commit message is a neutralInitial commit; the old message contained the instance id, which names the bug-generation method.raises=Truedoes not cover), raises an error, so the reset stops instead of continuing with a half-built repository.git init -band--pathspec-from-fileneed git 2.28 or later in the image. The six SWE-smith images I checked ship git 2.34.1..gitalso drops any repo-local git config and.git/info/excludefrom the image. Thedebug-gymidentity is set again.The gold patch (
git apply --reverse) andeval()(git checkout -- <test files>) still work on the working tree and index. The Docker runs below confirm the gold path on six tasks.Tests
tests/gym/envs/test_swe_smith.py:test_setup_terminalpreviously asserted thatgit show HEADequals the bug patch. It now checks that the bug patch is applied (git apply --reverse --check), the tree is clean, there is exactly one commit and one commit object, andgit checkout HEAD~1 -- .fails.test_setup_terminal_with_bug_patch_adding_a_file: a bug patch that adds a file leaves that file tracked in the initial commit, and the gold patch removes it.Both fail on
mainand pass with this change. The rest of the file passes before and after. Run with Docker on a fresh clone (Ubuntu 24.04, Python 3.12, Docker 29.1.3):End-to-end through
create_envwithconfigs/swesmith.yamland scripted tool calls (bashgit checkout HEAD~1 -- ., then submit):john-kurkowski__tldextract.3d1bf184.combine_file__1vnuqpt4fatal: invalid reference: HEAD~1oauthlib__oauthlib.1fd52536.combine_file__09vlzwgcfatal: invalid reference: HEAD~1seatgeek__thefuzz.8a05a3ee.combine_file__2uoca06xfatal: invalid reference: HEAD~1iterative__dvc.1d6ea681.pr_7514(patch adds a file)fatal: invalid reference: HEAD~1On this branch,
kennethreitz__records.5941ab27.combine_file__95trbjmz(18/18) andsloria__environs.73c372df.func_pm_op_change__4wqz9rtn(9/9) also keep the gold patch resolved and the shortcut at 0. Every task above has a single commit object after reset, and no-op scores 0 in every run.blackandisort(the CI settings) pass.Fixes #397