[Feature]: Introduce Scenario-Based Regression Testing for Configuration Coverage

[harryswift01]

Feature Request

Problem / Motivation

The current regression testing setup only validates a single configuration per system, meaning most of the available CLI/configuration options are not exercised in end-to-end tests.

While atomic unit tests already provide good coverage of individual functionality, there is currently no systematic way to ensure that:

• different configuration modes work correctly together
• key user-facing options continue to behave as expected
• changes do not unintentionally affect less commonly used features

As the number of configurable options grows, this increases the risk of regressions in real-world usage scenarios.

---

Proposed Solution

Introduce a scenario-based regression testing framework that expands coverage across key configuration options in a structured and maintainable way.

1. Scenario-based regression tests

• Replace the current “one config per system” model with:

  • multiple named scenarios per system
  • each scenario defined as a separate YAML config

• Each scenario is executed as an individual pytest test case

Example structure:

tests/regression/
  configs/
    dna/
      default.yaml
      frame_window.yaml
      selection_subset.yaml
      axes_off.yaml
      external_force.yaml
  baselines/
    dna/
      default.json
      frame_window.json
      selection_subset.json
      axes_off.json
      external_force.json

• Each *.yaml → one regression test
• Each scenario has a corresponding baseline JSON

---

2. Automatic test discovery

• Add a small helper (cases.py) to:

  • discover all scenario YAML files
  • generate pytest parameters dynamically

• Each test is identified as:

  system:scenario (e.g. dna:axes_off)

---

3. Focus on key configuration coverage

Define a curated set of scenarios that cover important configuration groups:

• Frame handling (start, end, step)
• Atom selection (selection_string)
• Algorithm toggles (combined_forcetorque, customised_axes)
• Feature flags (water_entropy, grouping, search_type)

Avoid testing all combinations — instead use representative scenarios.

---

4. Hybrid testing approach

Combine two complementary strategies:

A. Baseline-backed regression tests

• For meaningful scenarios (YAML-backed)
• Compare full output against stored JSON baselines

B. Generated “smoke” tests (Python)

• Programmatically generate config variations

• Vaulidate:

  • successful execution
  • valid output structure

• No baseline required

---

5. Optional: invariance (metamorphic) tests

Add lightweight tests for options that should not affect results, e.g.:

• verbose
• outfile

These compare outputs between equivalent configurations rather than against a baseline.

---

Alternatives Considered

• Single baseline per system (current approach)
  Too limited; does not cover configuration space

• Full combinatorial testing of all options
  Not scalable; leads to excessive number of baselines

• Only unit tests
  Does not capture integration issues between configuration options

---

Expected Impact

• Improved confidence in end-to-end behaviour across configuration modes
• Better protection against regressions in less frequently used options
• More granular and debuggable test failures (per scenario)
• Scalable structure for adding new configuration options in the future

Potential trade-offs:

• Slight increase in number of regression tests
• Additional baseline files to maintain (mitigated by keeping scenarios curated)

---

Additional Context

• Existing test infrastructure already supports:

  • config-driven execution
  • baseline comparison
  • slow test handling

• This proposal builds on top of that rather than replacing it

• The approach follows common patterns used in scientific and CLI-heavy Python projects:

  • data-driven tests (YAML)
  • pytest parametrization
  • hybrid baseline + generated test strategies

[skfegan]

This looks like it would be an improvement to the regression testing.

One note: for the numerical parameters, temperature is listed as it is a parameter set by the user. However this must be chosen to match the temperature used in the simulation (either you put the correct temperature or your results are wrong) so it is not really an option the way bin_width is.

[harryswift01]

This looks like it would be an improvement to the regression testing.

One note: for the numerical parameters, temperature is listed as it is a parameter set by the user. However this must be chosen to match the temperature used in the simulation (either you put the correct temperature or your results are wrong) so it is not really an option the way bin_width is.

That’s a great point, thanks for highlighting it. You’re absolutely right that temperature needs to be consistent with the underlying simulation, rather than treated as a freely varying configuration option. I’ll make sure in this scenario the temperature remains constant