Skip to content

Make examples and validation tests tell a missing backend from a failure - #246

Merged
shaia merged 5 commits into
masterfrom
fix/examples-validation-status
Nov 18, 2026
Merged

shaia merged 5 commits into
masterfrom
fix/examples-validation-status

Conversation

@shaia

@shaia shaia commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Summary

This PR is the follow-up from the v0.4.0 release (#244, #245). Examples and validation tests now tell a backend that is missing from the build apart from a real failure. Before this change, a missing backend could make a test fail instead of skip, and a real failure could pass silently as a skip or as exit code 0.

Only examples/, tests/validation/, the examples guide and the CHANGELOG change. lib/ is unchanged.

Commits

Commit Change
8a70882 Six examples report failures and exit nonzero (custom_source_terms, minimal_example, minimal_example_3d, runtime_comparison, solver_selection, steady_flow_multigrid). Each now checks every solver_init, solver_step and run_simulation_step status, skips a solver whose backend the build or machine lacks, and exits 1 on any other failure. Details for two of them below.
4551012 Validation tests skip only a missing backend. The new tests/validation/solver_availability.h creates solvers with cfd_solver_create_checked() and treats only CFD_ERROR_UNSUPPORTED as unavailable. Every harness that may be given an optional backend now uses it. tg_run_simulation() also checks solver_init and solver_step.
a270767 failure_reason() reports a failure with its own message. It now uses the library's stored message only when its status matches the failure; otherwise an earlier, unrelated message could be printed. Found by /code-review; also applied to the same helper in performance_comparison (from #244).

solver_selection changes:

  • It now checks cfd_backend_is_available() before switching to the AVX2 solver, because a failed switch only shows at the first step.
  • It creates the directory for its VTK file. Before, it wrote into a missing output/ and still printed "Output written to".

runtime_comparison is rebuilt on two helpers, create_sim() and time_solver(). Before, it stepped NULL simulations and divided by zero timings; a missing backend now prints n/a.

Validation tests and the behaviour each had before this PR:

Test or harness Before
test_solver_architecture, test_taylor_green_decay, test_natural_convection, test_ghia_projection_gpu, Taylor-Green backend consistency Any NULL from creation was read as "not available" and skipped. That also hid an unregistered name or an allocation failure.
BackwardFacingStepRe400Test Failed on a build without OpenMP, where projection_omp is not registered.
The two cavity-harness sites from #245 Already fixed; they now use the shared helper.
test_quiescent_3d Moved to the shared helper.
poiseuille_3d_reference.h, taylor_green_3d_reference.h, test_poiseuille_flow, test_turbulent_channel, the cavity harness's scalar site Exempt: they only create the scalar projection, and a NULL there already fails.

Testing

Two builds were used: the default options (AVX2 off) and one configured with -DCMAKE_DISABLE_FIND_PACKAGE_OpenMP=ON.

  • Affected validation tests: 20/20 pass on the default build and 21/21 without OpenMP. Without OpenMP, BackwardFacingStepRe400Test now skips.
  • Probe, validation: a misspelled solver in the Taylor-Green consistency test now fails with "not created: Resource not found". Before, it printed SKIPPED.
  • Examples: all six exit 0 on both builds. steady_flow_multigrid ran at n = 33 (see below).
  • Probe, examples: with a negative dt, minimal_example, runtime_comparison and solver_selection exit 1 and name the failing step.
  • Fast suite: ctest -LE "cross-arch|validation" passes 101/101 (Debug).
  • /pr-ready: passed, including /code-review medium; its one finding is fixed in a270767.

The full 129×129 steady_flow_multigrid run timed out at 600 s, so it was checked at n = 33 instead. This laptop is in the vendor "Silent" power profile with the CPU at 2.2 GHz, which made every timing on 10/4 roughly 5–6× slower. Before that the same run took 104 s. The change to that example only adds a status check on its warm-up steps.

Not in this PR (library issues found along the way)

  • simulation_set_solver() ignores solver_init()'s status, so simulation_set_solver_by_name() reports success when the new solver refuses to initialize, after the old solver has already been destroyed.
  • simulation_list_solvers() returns a hardcoded list. It omits RK2 and RK4, and lists the _gpu names even without CUDA.
  • write_vtk_flow_field() and the other VTK writers return void, so a caller cannot detect a failed write.

🤖 Generated with Claude Code

shaia added 3 commits October 5, 2026 10:22
Six examples discarded solver_init, solver_step and run_simulation_step
statuses, so a solver that stopped working still printed results and
exited 0, and the release gate's example run could not see it. Each now
checks every status, skips a solver whose backend the build or machine
lacks, and exits 1 on anything else.

solver_selection checks cfd_backend_is_available() before switching to the
AVX2 solver: simulation_set_solver_by_name() reports success even when the
new solver's init fails, so the failure would only show at the first step.
It also creates the directory for its direct-API VTK file, which it wrote
into a missing output/ while printing "Output written to". runtime_comparison
is rebuilt on create_sim() and time_solver(), which separate a missing
backend from a failure; it used to step NULL simulations and divide by zero
timings, and now prints n/a.

Checked on a default build (AVX2 off) and one without OpenMP: all six exit
0, and with a negative dt injected minimal_example, runtime_comparison and
solver_selection exit 1 with the failing step named.
Several harnesses treated any NULL from cfd_solver_create() as "not
available" and skipped, which also hid an unregistered name or an
allocation failure (test_solver_architecture, test_taylor_green_decay,
test_natural_convection, test_ghia_projection_gpu, and the Taylor-Green
backend consistency test, which skipped on any failure at all), while
BackwardFacingStepRe400Test failed instead of skipping on a build without
OpenMP, where the _omp solvers are not registered and creation reports
CFD_ERROR_NOT_FOUND.

tests/validation/solver_availability.h creates solvers with
cfd_solver_create_checked(), which reports a missing backend as
CFD_ERROR_UNSUPPORTED before the name lookup, and marks only that status
unavailable. Every harness that may be handed an optional backend uses it,
including the two cavity-harness sites #245 fixed inline. tg_run_simulation()
also stops ignoring solver_init() and solver_step() statuses.

Exempt: poiseuille_3d_reference.h, taylor_green_3d_reference.h,
test_poiseuille_flow, test_turbulent_channel and the cavity harness's
scalar-only site only ever create the scalar projection, and a NULL there
already fails.

Checked on a default build and on one without OpenMP: the 20 and 21 affected
validation tests pass, BackwardFacingStepRe400Test now skips without
OpenMP, and a misspelled solver in the Taylor-Green consistency test fails
("not created: Resource not found") where it used to print SKIPPED.
failure_reason() printed cfd_get_last_error() whenever a message was stored,
so a failure that set none was reported with an earlier, unrelated message,
such as a backend skipped further up the run. It now uses the stored message
only when cfd_get_last_status() matches the status being reported.
Found by /code-review on this branch in solver_selection; the same helper in
performance_comparison (from #244) gets the same fix.
Copilot AI balanced review requested due to automatic review settings October 5, 2026 07:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Sticky status and registry handling can still misclassify genuine creation failures as unavailable backends.

Review effort: Balanced
Findings: 3 Medium severity

Open (3)
What changed in this PR

Improves examples and validation tests so unavailable optional backends are skipped while genuine solver failures fail visibly.

Changes:

  • Adds shared backend-aware solver creation for validation tests.
  • Checks solver initialization and execution statuses in examples.
  • Updates documentation and changelog behavior descriptions.
File Description
CHANGELOG.md Records corrected failure handling.
docs/​guides/​examples.md Documents backend checks and failure behavior.
examples/​custom_source_terms.c Propagates init and step failures.
examples/​minimal_example.c Checks simulation steps.
examples/​minimal_example_3d.c Checks 3D simulation steps.
examples/​performance_comparison.c Prevents stale error messages.
examples/​runtime_comparison.c Handles unavailable benchmark backends.
examples/​solver_selection.c Reports solver failures and prepares output paths.
examples/​steady_flow_multigrid.c Checks warm-up steps.
tests/​validation/​lid_driven_cavity_common.h Uses the shared availability helper.
tests/​validation/​solver_availability.h Introduces checked solver creation.
tests/​validation/​taylor_green_reference.h Checks creation, initialization, and steps.
tests/​validation/​test_backward_facing_step.c Skips unavailable OpenMP backend.
tests/​validation/​test_ghia_projection_gpu.c Distinguishes unavailable GPU from failures.
tests/​validation/​test_natural_convection.c Handles creation failures correctly.
tests/​validation/​test_quiescent_3d.c Uses shared availability handling.
tests/​validation/​test_solver_architecture.c Uses checked solver creation.
tests/​validation/​test_taylor_green_decay.c Distinguishes missing backends from errors.
tests/​validation/​test_taylor_green_vortex.c Fails non-availability errors.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread examples/runtime_comparison.c Resolved
Comment thread examples/solver_selection.c Resolved
Comment thread tests/validation/solver_availability.h Resolved
shaia added 2 commits October 5, 2026 21:32
Review finding (#246): cfd_solver_create() reports a factory's NULL as
CFD_ERROR_NOMEM only while the last status is still CFD_SUCCESS, and that
status is sticky. An UNSUPPORTED left over from an earlier solver made a later
allocation failure read as unavailable, so validation_create_solver() skipped
exactly the failure it exists to expose.
Review findings (#246): runtime_comparison and solver_selection read "not in
the registry" as "backend unavailable", so a misspelled or removed solver
printed n/a or SKIPPED and the example still exited 0. solver_selection also
ignored a failed registry allocation, after which every solver read as
skipped and the run reported zero failures.

Both now probe with cfd_solver_create_checked(), which refuses an unavailable
backend with CFD_ERROR_UNSUPPORTED before the name lookup, and skip only that
status; any other creation failure, an unknown name included, is a failure. A
NULL registry in solver_selection is reported and counted as one.
Copilot AI balanced review requested due to automatic review settings October 5, 2026 18:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Unknown optional-backend names can still be skipped, and some updated output and documentation conflate failures with unavailable backends.

Review effort: Balanced
Findings: None

Resolved since last review (3)
Previously missed (4)

In code that hasn't changed since last review

Medium severity Distinguish failed runs from unavailable backends

examples/​runtime_comparison.c:229

A real initialization or step failure also sets ran to 0, so this prints n/a for both missing backends and failed runs. That makes the results table contradict the new distinction even though stderr and the exit code report the failure. Preserve each solver's unavailable flag in benchmark_result and print FAILED when !ran && !unavailable.

Medium severity Preserve registry allocation failure diagnostics

examples/​solver_selection.c:290

If registry allocation fails, cfd_registry_register_defaults(NULL) is a no-op and the subsequent solver creation overwrites the original CFD_ERROR_NOMEM with CFD_ERROR_INVALID. This new diagnostic then reports an invalid argument rather than the allocation failure. Check registry immediately, as run_solver_comparison() already does, before registering or creating a solver.

Medium severity Unknown solver names misclassified as unsupported

tests/​validation/​solver_availability.h:41

cfd_solver_create_checked() infers the backend from name substrings before consulting the registry (lib/src/api/solver_registry.c:276-297, 2110-2122). On a build without OpenMP, for example, an unregistered name such as projection_omp_typo therefore returns CFD_ERROR_UNSUPPORTED and is marked unavailable here, contradicting this helper's contract that unregistered names fail. Validate that type is a recognized optional solver before treating UNSUPPORTED as skippable, or expose registry metadata that can distinguish a known-but-unbuilt solver from an unknown name.

Low severity Handle all solver initialization failures

docs/​guides/​examples.md:243

This example handles only the unsupported case. If initialization returns NULL for CFD_ERROR_NOT_FOUND, CFD_ERROR_NOMEM, or another real failure, the snippet falls through with a NULL simulation instead of reporting the error, which undermines the distinction this section documents. Handle every NULL result and treat only CFD_ERROR_UNSUPPORTED as a skip.

@shaia shaia self-assigned this Nov 18, 2026
@shaia
shaia merged commit 98a95ab into master Nov 18, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants