Support larger TRX files

[36000]

I updated the code to handle tractograms with large numbers of vertices, fixing #50

[Copilot]

Pull request overview

This PR updates TRX reading/writing and in-memory handling to support tractograms with large vertex/streamline counts (and 64-bit offsets), addressing crashes when counts/offsets exceed 32-bit limits (issue #50).

Changes:

• Switch many count/shape variables from int to int64_t across TRX loading, saving, resizing, and mmap shapes.
• Introduce _json_int64() and replace several int_value() reads to avoid 32-bit truncation when consuming header counts.
• Update mmap/remap helpers and shard-merge header writing to better accommodate large datasets.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
src/trx.cpp	Widened header parsing/counts and mmap shape handling; updated shard merge header counts to avoid 32-bit truncation.
include/trx/trx.tpp	Propagated 64-bit counts/shapes through TrxFile construction, loading, resizing, saving, and subset operations.
include/trx/trx.h	Added _json_int64() and widened several public/internal count fields and APIs to int64_t.
include/trx/detail/dtype_helpers.h	Updated remap and length-computation helpers to accept 64-bit dimensions/counts.

Suppressed comments (1)

src/trx.cpp:916

• _create_memmap computes filesize via unchecked multiplication and, when the file is missing, calls allocate_file() which writes a std::string(filesize, '\0'). For large TRX datasets this can overflow size_t (producing a too-small mapping) and/or try to allocate enormous RAM just to preallocate the file, defeating the goal of supporting large files. Consider adding overflow/negative-shape checks and preallocating with filesystem resize (or seek+write) instead of materializing the full file in memory.

mio::shared_mmap_sink _create_memmap(std::string filename,
                                     const std::tuple<int64_t, int64_t> &shape,
                                     const std::string &mode,
                                     const std::string &dtype,
                                     long long offset) {
  static_cast<void>(mode);
  const std::size_t filesize = static_cast<std::size_t>(std::get<0>(shape)) *
                               static_cast<std::size_t>(std::get<1>(shape)) *
                               static_cast<std::size_t>(trx::detail::_sizeof_dtype(dtype));

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 90.44586% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.67%. Comparing base (abe6709) to head (ae6e507).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
include/trx/trx.tpp	90.76%	12 Missing ⚠️
src/trx.cpp	80.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #55      +/-   ##
==========================================
+ Coverage   87.85%   88.67%   +0.82%     
==========================================
  Files          15       15              
  Lines        7269     7773     +504     
  Branches      997     1047      +50     
==========================================
+ Hits         6386     6893     +507     
+ Misses        883      880       -3     

Flag	Coverage Δ
linux	87.87% <89.47%> (+0.02%)	⬆️
macos	88.52% <91.02%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[frheault]

@36000 Quite a big PR was merged, it kind of overhauled the core reading/saving. Could you try a large dataset on the main branch and confirm the problem is still there (or if it is gone, or changed place) and then fix conflicts (and hopefully it will all work flawlessly !