Skip to content

Do not keep the original code in the git history of SWE-smith tasks - #398

Merged
Marc-Alexandre Côté (MarcCote) merged 2 commits into
microsoft:mainfrom
jimmy-holdout:fix/swesmith-git-history
Nov 16, 2026
Merged

Marc-Alexandre Côté (MarcCote) merged 2 commits into
microsoft:mainfrom
jimmy-holdout:fix/swesmith-git-history

Conversation

@jimmy-holdout

Copy link
Copy Markdown
Contributor

What

SWESmithEnv.setup_terminal committed the bug patch on top of the code from before the bug. For bug patches that only modify or delete tracked files, git checkout HEAD~1 -- . therefore gave back exactly the code the gold patch produces. (Files a patch adds stayed untracked and are left in place.) In the tasks below, that followed by submit scored the same as the gold patch, without debugging. The image's .git also kept the commit objects of the SWE-smith task branches after the remote was removed.

After applying the bug patch, the env now replaces .git with a new repository whose only commit is the buggy code:

  • git apply --index stages the bug patch, so files it adds are tracked. Before, git commit -am left them untracked. The index is refreshed first (git update-index -q --refresh), because --index compares stat info. In the tldextract image the cached stat info is stale (all 28 tracked files are stat-dirty with no content change); before, the bug's git commit -am refreshed it as a side effect.
  • The new repository tracks the files the old index tracked after the bug patch (git ls-files -z → git add -f --pathspec-from-file, with literal pathspecs). It does not use git add -A, which would newly track untracked build files in some images, e.g. records.egg-info/ in the kennethreitz/records image.
  • The branch stays main, and the identity stays debug-gym / <>. The commit message is a neutral Initial commit; the old message contained the instance id, which names the bug-generation method.
  • A failure, including a command timeout (which raises=True does not cover), raises an error, so the reset stops instead of continuing with a half-built repository.
  • git init -b and --pathspec-from-file need git 2.28 or later in the image. The six SWE-smith images I checked ship git 2.34.1.
  • Replacing .git also drops any repo-local git config and .git/info/exclude from the image. The debug-gym identity is set again.

The gold patch (git apply --reverse) and eval() (git checkout -- <test files>) still work on the working tree and index. The Docker runs below confirm the gold path on six tasks.

Tests

tests/gym/envs/test_swe_smith.py:

  • test_setup_terminal previously asserted that git show HEAD equals the bug patch. It now checks that the bug patch is applied (git apply --reverse --check), the tree is clean, there is exactly one commit and one commit object, and git checkout HEAD~1 -- . fails.
  • New test_setup_terminal_with_bug_patch_adding_a_file: a bug patch that adds a file leaves that file tracked in the initial commit, and the gold patch removes it.

Both fail on main and pass with this change. The rest of the file passes before and after. Run with Docker on a fresh clone (Ubuntu 24.04, Python 3.12, Docker 29.1.3):

main code + these tests:  2 failed, 9 passed  (test_setup_terminal, test_setup_terminal_with_bug_patch_adding_a_file)
this branch:             11 passed
main code + main tests:  10 passed

End-to-end through create_env with configs/swesmith.yaml and scripted tool calls (bash git checkout HEAD~1 -- ., then submit):

Task before after gold patch, after
john-kurkowski__tldextract.3d1bf184.combine_file__1vnuqpt4 39/39, resolved 0/39, fatal: invalid reference: HEAD~1 39/39, resolved
oauthlib__oauthlib.1fd52536.combine_file__09vlzwgc 18/18, resolved 0/18, fatal: invalid reference: HEAD~1 18/18, resolved
seatgeek__thefuzz.8a05a3ee.combine_file__2uoca06x 25/25, resolved 0/25, fatal: invalid reference: HEAD~1 25/25, resolved
iterative__dvc.1d6ea681.pr_7514 (patch adds a file) 7/7, resolved 0/7, fatal: invalid reference: HEAD~1 7/7, resolved

On this branch, kennethreitz__records.5941ab27.combine_file__95trbjmz (18/18) and sloria__environs.73c372df.func_pm_op_change__4wqz9rtn (9/9) also keep the gold patch resolved and the shortcut at 0. Every task above has a single commit object after reset, and no-op scores 0 in every run.

black and isort (the CI settings) pass.

Fixes #397

SWESmithEnv committed the bug patch on top of the code from before the
bug. For bug patches that only modify or delete tracked files,
`git checkout HEAD~1 -- .` gave back exactly the code the gold patch
produces, and submitting it scored like the gold patch without any
debugging. The image's .git also kept the commit objects of the
SWE-smith task branches after the remote was removed.

After applying the bug patch, replace .git with a new repository whose
only commit is the buggy code. It tracks the files the old index tracked
after the bug patch, including files the patch adds (`git commit -am`
left those untracked).
@jimmy-holdout

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree [company="Holdout"]

@jimmy-holdout

Copy link
Copy Markdown
Contributor Author

@Holdout-github-policy-service agree

@jimmy-holdout

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree company="Holdout"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the repository-history reset and setup-failure handling changes.

No open issues 🎉

What changed in this PR

Addresses #397 by rebuilding SWE-smith task repositories with a single buggy baseline commit, removing the Git history that allowed agents to recover pre-bug code without debugging.

Changes:

  • Refreshes the index and applies bug patches with --index, ensuring added files are tracked.
  • Replaces .git while preserving the tracked-file set, and stops setup on command failures or timeouts.
  • Adds regression checks for history removal, tracked additions, and gold-patch reversal.
File Description
tests/​gym/​envs/​test_swe_smith.py Verifies the single-commit history, blocked parent checkout, and handling of patch-added files.
debug_gym/​gym/​envs/​swe_smith.py Rebuilds Git history around the buggy code and checks setup failures.

Review effort: Balanced

@MarcCote
Marc-Alexandre Côté (MarcCote) merged commit 80c77c2 into microsoft:main Nov 16, 2026
11 checks passed
@MarcCote

Copy link
Copy Markdown
Collaborator

Thank you for the fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SWE-smith tasks expose the pre-bug code through git history (git checkout HEAD~1 -- .)

3 participants