Convert to psc output format v1 by JamesMcClung · Pull Request #43 · psc-code/pscpy

JamesMcClung · 2026-09-25T18:57:55Z

In psc-code/psc#407, the output format of field bp data changed significantly. This PR:

updates decode_psc to support only the new output version (v1), and deprecates it
adds a conversion script to convert legacy data to the v1 format (run python3 -m pscpy.convert.legacy_to_v1 -h to see options)
removes old sample data, leaving only data generated by Add psc_pscpy_sample case psc#411

~~It is also a breaking change for downstream users, because the t dimension is now time.~~

This PR was largely done by Claude, but does not yet add a CLAUDE.md file.

v1/ is written by psc with psc_output_version 1.0.0; legacy/ is the same run written by the pre-1.0.0 writer. The runs are deterministic, so the legacy files convert to exactly the v1 files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

psc now writes each component as its own (z, y, x) variable with cell-centered x/y/z coords and a time variable, so decode_psc() only needs to check the version and squeeze a single time step. Legacy files are rejected with a pointer to pscpy.convert. The species_names, length and corner args are gone since the file now carries everything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Converts legacy psc .bp output (all components in one 4-d variable) to psc_output_version 1.0.0, matching what the new psc writer produces: per-component (z, y, x) variables, cell-centered x/y/z coords, and time as a variable. Run as python -m pscpy.convert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

psc computes corner + (i + .5) * dx, which the compiler contracts into a fused multiply-add, so doing it with two roundings in python can be off by one ulp (e.g. y with corner -0.4, length 0.8, 16 cells). math.fma needs Python 3.13, so emulate it with exact Fraction arithmetic. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With nargs="+", --species e i followed by the input files swallowed the files as species names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Each future format update gets its own converter module next to it (e.g. convert/v1_to_v2), run as python -m pscpy.convert.<from>_to_<to>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

It used psc.PSC_OUTPUT_VERSION, which would start stamping legacy->v1 output as v2 once that constant is bumped for the next format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Unused now that each converter owns the version it writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Both are now written by psc's dedicated pscpy_sample script (the legacy/ run with the pre-1.0.0 writer), on a smaller 1x8x4 domain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ounding The new sample's cell widths are exactly representable, so naive and fma-rounded centers agree; keep the values psc wrote for the old sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Step 1 of legacy and v1 gauss (dive, rho) and continuity (d_rho, dt_divj) output from the same pscpy_sample runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Bitwise-matching psc's fma-contracted coordinates isn't something to rely on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Compute cell centers with plain numpy arithmetic instead of emulating psc's fma rounding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Provides the latest_sample_dir fixture, which follows SUPPORTED_MAJOR_VERSION so tests use the newest supported sample output automatically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The new samples are BP5, so request that engine instead of BP4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…samples The old samples' quirks (float32 data, time/step as 1-element arrays, unprefixed ib/im) are now written directly with _write_legacy; the rest of what the old-sample tests checked is covered by test_matches_psc_v1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Old legacy output that no test uses any more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

numpy 2.5 requires Python >=3.12 and its stubs use PEP 695 type statements, which mypy rejects with python_version = 3.10. CI's fresh hook env picked up numpy 2.5.3 and failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

codecov · 2026-09-25T19:14:46Z

Codecov Report

❌ Patch coverage is 93.07692% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/pscpy/convert/legacy_to_v1.py	91.81%	9 Missing ⚠️

Files with missing lines	Coverage Δ
src/pscpy/psc.py	`100.00% <100.00%> (+3.70%)`	⬆️
src/pscpy/convert/legacy_to_v1.py	`91.81% <91.81%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

JamesMcClung · 2026-09-25T19:28:51Z

The codecov failure seems innocuous to me, all the flagged lines are just trivial error cases. @germasch do you want them to be covered?

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Use typing_extensions.deprecated since warnings.deprecated needs Python 3.13. Tests that still exercise decode_psc ignore its DeprecationWarning, which pytest otherwise turns into an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

JamesMcClung and others added 26 commits September 25, 2026 09:38

convert: take --species as a comma-separated list

a9679e9

With nargs="+", --species e i followed by the input files swallowed the files as species names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

+test_convert

bbb91f1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

convert; *: ->convert/legacy_to_v1

f4d3521

Each future format update gets its own converter module next to it (e.g. convert/v1_to_v2), run as python -m pscpy.convert.<from>_to_<to>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

legacy_to_v1; *: write its own fixed OUTPUT_VERSION

33f5feb

It used psc.PSC_OUTPUT_VERSION, which would start stamping legacy->v1 output as v2 once that constant is bumped for the next format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

psc: -PSC_OUTPUT_VERSION

94fc8de

Unused now that each converter owns the version it writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

psc; *: point to pscpy.convert for older output versions

5b06224

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

->test_convert_legacy_to_v1

3a8ccc8

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sample: regenerate legacy and v1 pfd, pfd_moments with pscpy_sample

c510eb5

Both are now written by psc's dedicated pscpy_sample script (the legacy/ run with the pre-1.0.0 writer), on a smaller 1x8x4 domain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_xarray_adios2: update expected grid and time for new sample

68f4391

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_convert_legacy_to_v1: hardcode psc coords in test_cell_centers_r…

6661e2a

…ounding The new sample's cell widths are exactly representable, so naive and fma-rounded centers agree; keep the values psc wrote for the old sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sample: +gauss, continuity

997a2ab

Step 1 of legacy and v1 gauss (dive, rho) and continuity (d_rho, dt_divj) output from the same pscpy_sample runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_convert_legacy_to_v1: test gauss and continuity samples

a6a6882

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_convert_legacy_to_v1: -test_cell_centers_rounding

d7a624f

Bitwise-matching psc's fma-contracted coordinates isn't something to rely on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

legacy_to_v1: -_fma

d7b8e13

Compute cell centers with plain numpy arithmetic instead of emulating psc's fma rounding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_convert_legacy_to_v1: compare coords to psc v1 output with allclose

2aef2e8

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

+conftest

df701b8

Provides the latest_sample_dir fixture, which follows SUPPORTED_MAJOR_VERSION so tests use the newest supported sample output automatically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_xarray_adios2: open samples from latest_sample_dir

066c1c2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_xarray_adios2: use legacy/ samples in test_legacy_rejected

6bcc9b9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_Adios2Store: open latest sample in test_open_with_engine

e93f616

The new samples are BP5, so request that engine instead of BP4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test_adios2py: read latest sample instead of legacy pfd.000000400

8085fb9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sample: -pfd.000000000, pfd.000000400, pfd_moments.000000400

c809576

Old legacy output that no test uses any more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

JamesMcClung mentioned this pull request Sep 25, 2026

Add psc_pscpy_sample case psc-code/psc#411

Merged

.pre-commit-config: pin numpy<2.5 for mypy

e94ffcb

numpy 2.5 requires Python >=3.12 and its stubs use PEP 695 type statements, which mypy rejects with python_version = 3.10. CI's fresh hook env picked up numpy 2.5.3 and failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

JamesMcClung and others added 2 commits October 1, 2026 10:35

psc; *: rename time dim back to t in decode_psc

d310599

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

psc; *: deprecate decode_psc

3e87a7e

Use typing_extensions.deprecated since warnings.deprecated needs Python 3.13. Tests that still exercise decode_psc ignore its DeprecationWarning, which pytest otherwise turns into an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

JamesMcClung merged commit 392ea28 into psc-code:main Oct 1, 2026
3 of 4 checks passed

JamesMcClung deleted the convert-to-v1 branch October 1, 2026 15:40

germasch mentioned this pull request Oct 6, 2026

compat: psc_decode() handle new by-component output #42

Closed

Navigation Menu

Repository navigation

Convert to psc output format v1 - #43

Convert to psc output format v1#43
JamesMcClung merged 29 commits into
psc-code:mainfrom
JamesMcClung:convert-to-v1

JamesMcClung commented Sep 25, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 25, 2026 •

edited

Loading

Uh oh!

JamesMcClung commented Sep 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

JamesMcClung commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JamesMcClung commented Sep 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JamesMcClung commented Sep 25, 2026 •

edited

Loading

codecov Bot commented Sep 25, 2026 •

edited

Loading