Skip to content

fix: make uproot.recreate physically truncate an existing file - #1697

Open
ariostas wants to merge 6 commits into
scikit-hep:mainfrom
ariostas:fix-recreate-truncate
Open

ariostas wants to merge 6 commits into
scikit-hep:mainfrom
ariostas:fix-recreate-truncate

Conversation

@ariostas

@ariostas ariostas commented Aug 5, 2026 •

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

Addresses finding 3 of "PR 1" in #1688. This is a regression, with a clear origin.

Before the fsspec migration, uproot.recreate truncated the path itself:

if isinstance(file_path, str):
    with open(file_path, "w"):
        pass
    sink = uproot.sink.file.FileSink(file_path)

#1016 / #1034 moved that into FileSink.__init__, which only truncates when the file does not already exist:

https://github.com/scikit-hep/uproot5/blob/main/src/uproot/sink/file.py#L52-L58

so recreating over an existing file opens it "r+b" and leaves every byte past the new fEND in place. Recreating a 100 kB path as an empty ROOT file left the physical size at 100 kB with fEND at 1658.

Changes

  • FileSink takes a mode: "create", "recreate", or "update" (default, so existing callers are unchanged), mirroring ROOT's TFile options. The truncate/create decision lives only there.
    • "recreate" truncates an existing path, and now also a file-like object, which previously kept its stale trailing bytes too. truncate is not required of file-like objects, so ones without it are written as before.
      An existing path is truncated through the file opened "r+b", not with fs.touch beforehand. Filesystems that can write whole files but not open them "r+b" (S3, XRootD) still fail as on main, but with the file intact: truncating first would have emptied the file and then failed.
    • "create" creates the file with exclusive mode "xb", so a file that appears between the existence check and the creation is never overwritten (atomic on local filesystems). Filesystems that don't support "xb" fall back to an existence check. A file-like object has no path to check, so it is written into as is (not truncated), as on main.
    • "update" keeps its "r+b" behavior.
  • uproot.create and uproot.recreate share one implementation, and unrecognized options are rejected before the sink is opened. Otherwise uproot.recreate(path, compresion=...) would truncate the existing file and only then raise TypeError. uproot.update validates first as well.
  • Each call resolves the URL once (url_to_fs) instead of up to three times with repeated exists checks.

Out of scope, but noticed along the way: extra keyword arguments meant as fsspec storage_options are never popped from options, so create/recreate/update always raise TypeError for them (e.g. uproot.recreate(p, auto_mkdir=True)). I left that unchanged here; see #1729.

@codecov

codecov Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.21%. Comparing base (4f2f340) to head (df2ee9c).

Files with missing lines Patch % Lines
src/uproot/sink/file.py 83.78% 5 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (85.71%) is below the target coverage (98.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
Files with missing lines Coverage Δ
src/uproot/writing/writable.py 84.83% <100.00%> (+0.31%) ⬆️
src/uproot/sink/file.py 83.18% <83.78%> (-0.91%) ⬇️

... and 1 file with indirect coverage changes

@TaiSakuma TaiSakuma added the type/fix PR title type: fix (set automatically) label Aug 14, 2026
@ariostas
ariostas force-pushed the fix-recreate-truncate branch from a94227a to 918dd9f Compare September 24, 2026 18:39
@ariostas
ariostas marked this pull request as ready for review September 24, 2026 18:41
@ariostas
ariostas requested review from ianna and a lite review from Copilot September 24, 2026 18:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

create can destructively truncate existing file-like objects; restrict truncation to recreate or reject that combination.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity

Open (1)
What changed in this PR

Fixes uproot.recreate so existing files are physically truncated while preserving create/update behavior.

Changes:

  • Adds create, recreate, and update sink modes.
  • Validates options before opening or truncating files.
  • Adds regression coverage for truncation and mode behavior.
File Summary
tests/​test_1697_recreate_truncates.py Tests truncation and mode behavior.
src/​uproot/​writing/​writable.py Shares create/recreate setup and validates options early.
src/​uproot/​sink/​file.py Implements mode-specific creation, truncation, and update behavior; create currently truncates file-like objects.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/uproot/sink/file.py Outdated
@ariostas
ariostas force-pushed the fix-recreate-truncate branch from a7517b1 to 6695e40 Compare September 28, 2026 18:16
Before the fsspec migration (scikit-hep#1016, scikit-hep#1034), uproot.recreate opened the path with
mode "w", truncating it. Afterwards the truncation moved into FileSink, which
only calls _truncate_file when the file does *not* already exist, so recreate
over a pre-existing file opened it "r+b" and left every byte past the new fEND
in place. Recreating a 100 kB path as an empty ROOT file left the physical size
at 100 kB with fEND at 1658.

Truncate the path in recreate, restoring the documented "RECREATE" semantics.
uproot.update is unaffected and keeps its "r+b" behavior, and uproot.create
still raises FileExistsError before reaching this point. Recreating from a
file-like object is unchanged, as it was before the migration too.

Assisted-by: claude-code:claude-opus-5[1m]
FileSink now takes a mode ("create", "recreate", or "update"), so the
decision to truncate lives in one place and also covers file-like
objects. create uses exclusive creation ("xb"), falling back to an
existence check on filesystems that do not support it. Options are
validated before the sink is opened, so a misspelled option no longer
truncates an existing file before raising.

Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Claude Code:claude-opus-5-5
…files

Truncating with fs.touch before opening the file "r+b" emptied an
existing file on filesystems that can write but not open "r+b" (such as
S3 and XRootD), and only then failed. Now the file is opened first and
truncated through the handle, so such filesystems fail with the file
intact, as before.

Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Claude Code:claude-opus-5-5
@ariostas
ariostas force-pushed the fix-recreate-truncate branch from 6695e40 to df2ee9c Compare September 29, 2026 15:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix PR title type: fix (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants