Use Parquet as the Default Format for IOH Data Loading

[Dinu23]

Summary

This pull request adds Parquet file support and fixes a metadata attribute naming conflict.

Changes

Parquet file support

When a .parquet file exists at the same path and has the same name as the corresponding .dat file, the system will attempt to read the .parquet file instead.

Metadata attribute conflict fix

Fixed a bug that caused a failure when an experiment attribute had the same name as a standard JSON metadata field.

Conflicting experiment attributes are now renamed by appending the _exp_attr suffix.

[codacy-production]

Not up to standards ⛔

🔴 Issues 2 minor

Alerts:
⚠ 2 issues (≤ 0 issues of at least minor severity)

Results:
2 new issues

Category	Results
BestPractice	2 minor

View in Codacy

🟢 Metrics 10 complexity · 0 duplication

Metric	Results
Complexity	10
Duplication	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[Copilot]

Pull request overview

This PR updates IOHinspector’s dataset loading to prefer Parquet files when available (as a faster/typed alternative to .dat CSV parsing) and resolves a metadata-schema collision by renaming conflicting experiment_attributes keys.

Changes:

• Prefer reading a sibling .parquet file for IOH scenario data loading, with fallback to the original .dat.
• Rename experiment attribute keys that conflict with standard metadata columns by appending _exp_attr.
• Add package __version__ discovery and bump project version to 0.0.8; include additional Parquet example fixtures.

Reviewed changes

Copilot reviewed 15 out of 30 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/iohinspector/data.py	Prefer .parquet scanning for IOH data; rename conflicting experiment attribute keys.
src/iohinspector/init.py	Add __version__ resolution from installed package metadata.
pyproject.toml	Bump package version to 0.0.8.
.gitignore	Normalize aux/ ignore entry and add try.ipynb.
examples/parquet/SO_Data/RS/IOHprofiler_f2_Ellipsoid.json	Add single-objective example metadata for Parquet-backed dataset.
examples/parquet/SO_Data/RS/IOHprofiler_f1_Sphere.json	Add single-objective example metadata for Parquet-backed dataset.
examples/parquet/SO_Data/RS/data_f2_Ellipsoid/IOHprofiler_f2_DIM2.dat	Add SO .dat fixture paired with .parquet example.
examples/parquet/SO_Data/RS/data_f1_Sphere/IOHprofiler_f1_DIM2.dat	Add SO .dat fixture paired with .parquet example.
examples/parquet/SO_Data/HC/IOHprofiler_f2_Ellipsoid.json	Add single-objective example metadata for Parquet-backed dataset.
examples/parquet/SO_Data/HC/IOHprofiler_f1_Sphere.json	Add single-objective example metadata for Parquet-backed dataset.
examples/parquet/SO_Data/HC/data_f2_Ellipsoid/IOHprofiler_f2_DIM2.dat	Add SO .dat fixture paired with .parquet example.
examples/parquet/SO_Data/HC/data_f1_Sphere/IOHprofiler_f1_DIM2.dat	Add SO .dat fixture paired with .parquet example.
examples/parquet/MO_Data/SMSEMOA_10/IOHprofiler_f0_pymoo_ZDT1.json	Add multi-objective example metadata for Parquet-backed dataset.
examples/parquet/MO_Data/SMSEMOA_10-1/IOHprofiler_f1_pymoo_ZDT2.json	Add multi-objective example metadata for Parquet-backed dataset.
examples/parquet/MO_Data/NSGA_10/IOHprofiler_f0_pymoo_ZDT1.json	Add multi-objective example metadata for Parquet-backed dataset.
examples/parquet/MO_Data/NSGA_10-1/IOHprofiler_f1_pymoo_ZDT2.json	Add multi-objective example metadata for Parquet-backed dataset.

Suppressed comments (1)

src/iohinspector/data.py:135

• This change introduces new behaviors that should be covered by tests: (1) prefer the .parquet sibling when it exists, (2) fall back to the .dat file when parquet reading/validation fails, and (3) rename conflicting experiment_attributes keys without breaking metadata construction. There are existing data-loading tests in tests/test_data.py, but none appear to exercise these new paths yet.

        parquet_file = self.data_file.replace(".dat", ".parquet")
        if os.path.isfile(parquet_file):

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 25 changed files in this pull request and generated no new comments.

Suppressed comments (5)

src/iohinspector/data.py:139

• If the parquet exists but is unreadable/corrupt and the .dat fallback is missing, this code raises FileNotFoundError claiming that neither file exists, which is inaccurate and makes debugging harder. Also, the parquet path is derived via str.replace, which can produce the wrong filename, and the "missing required columns" error currently prints the entire required header list rather than the actual missing subset.

        parquet_file = self.data_file.replace(".dat", ".parquet")
        if os.path.isfile(parquet_file):
            try:
                lf = pl.scan_parquet(parquet_file)
                # Fail fast so we can fall back here if the parquet is corrupt or incompatible.

src/iohinspector/data.py:317

• The metadata attribute conflict fix (renaming conflicting experiment_attributes by appending _exp_attr) isn’t exercised by the current test suite. Since this affects the dataset overview schema/column naming, a focused unit test would help prevent future regressions (e.g. ensure a conflicting attribute like suite is renamed and doesn’t break Dataset.overview).

            metadata_col_names = {col_name for col_name, _ in METADATA_SCHEMA}
            for i, (name, value) in enumerate(experiment_attributes):
                if name in metadata_col_names:
                    warnings.warn(
                        f"Experiment attribute '{name}' is already present in the metadata schema. It will be renamed to avoid conflicts."
                    )
                    experiment_attributes[i] = (f"{name}_exp_attr", value)

src/iohinspector/data.py:116

• parquet_path = data["path"].replace(".dat", ".parquet") will replace every occurrence of the substring ".dat" anywhere in the path, not just a file extension (e.g. a directory name containing ".dat"). Using os.path.splitext avoids generating incorrect parquet paths.

This issue also appears on line 135 of the same file.

        data["path"] = os.path.join(dirname, data["path"])
        parquet_path = data["path"].replace(".dat", ".parquet")
        if not os.path.isfile(data["path"]) and not os.path.isfile(parquet_path):

src/iohinspector/data.py:179

• parquet_file = self.data_file.replace(".dat", ".parquet") can generate an incorrect parquet filename if ".dat" appears elsewhere in the path (not just as a suffix). Using os.path.splitext makes this robust.

        if not is_coco:
            parquet_file = self.data_file.replace(".dat", ".parquet")
            if not os.path.isfile(self.data_file) and os.path.isfile(parquet_file):
                return pl.scan_parquet(parquet_file).collect_schema().names()

src/iohinspector/data.py:135

• Parquet-preference behavior (preferring a same-named .parquet over .dat, and the fallback behavior when parquet is corrupt/incompatible) is new but not covered by existing tests. There are extensive data-loading tests under tests/, but none exercise .parquet inputs or the fallback path, so regressions here would be easy to miss.

This issue also appears on line 311 of the same file.

    def scan_ioh(self, header: list[str]):
        parquet_file = self.data_file.replace(".dat", ".parquet")