test: add opt-in MPICH/Hydra multi-node checkpoint-restart

[alan-lira]

Refs #485

Summary

This pull request adds an opt-in multi-node MPICH/Hydra checkpoint-and-restart test for MANA.

The goal is to cover a workflow that is difficult to exercise through a normal single-node CI runner: an MPI application distributed across multiple machines, checkpointed while communicating through Hydra, terminated, and then restarted from the generated checkpoint images.

The test is intentionally opt-in. It exits successfully with a clear SKIP message unless the user supplies a hostfile and the required runtime configuration.

Why this test is useful

Most local tests exercise MANA on a single machine. In a multi-node MPICH/Hydra execution, additional components become relevant:

• Hydra starts remote MPI ranks through SSH;
• all nodes must use compatible MANA and MPICH installations;
• the coordinator must remain reachable throughout the experiment;
• checkpoint images must be complete for every rank;
• the restarted ranks must reconnect and resume collective communication;
• the test must distinguish genuine rollback from a new execution.

The new test automates these steps in one reproducible workflow.

Test workflow

The script performs the following stages:

1. checks SSH access and required paths on every node;
2. compiles the included MPI counter application;
3. runs a native MPICH/Hydra smoke test;
4. starts a persistent coordinator;
5. runs a finite MANA smoke test;
6. launches a longer-running distributed MANA execution;
7. requests a blocking checkpoint;
8. verifies that checkpoint images exist and that no temporary checkpoint files remain;
9. terminates the original execution;
10. restarts the application from the checkpoint images;
11. verifies rollback from the application logs;
12. verifies continued MPI_Allreduce correctness and forward progress;
13. cleans only files and processes owned by the test.

Configuration

The test is controlled through environment variables, including:

• MANA_HOME;
• MANA_TEST_HOSTFILE;
• MANA_TEST_MPICH_HOME;
• MANA_TEST_RANKS;
• MANA_TEST_COORD_PORT;
• MANA_TEST_TIMEOUT;
• MANA_TEST_WORKDIR;
• MANA_TEST_REMOTE_USER.

When MANA_TEST_HOSTFILE is absent, the script prints a skip message and returns success. This keeps the test suitable for repositories whose standard CI environment does not provide multiple SSH-accessible nodes.

Isolation and safety

The test uses a dedicated work directory and an isolated HOME directory for the MANA coordinator status file. It does not overwrite the invoking user's normal .mana.rc.

Remote SSH commands use ssh -n so that reading the hostfile cannot be disrupted by SSH consuming standard input.

Cleanup is limited to the coordinator, MPI process group, and files created by the test. The script does not use broad process-killing commands or delete paths outside the configured test directory.

Validation performed

I validated the test using:

• two physical nodes;
• one MPI rank per node;
• MPICH 4.3.2 with Hydra;
• native distributed MPI execution;
• distributed MANA launch;
• blocking checkpoint;
• checkpoint image validation;
• termination and restart;
• rollback verification;
• continued collective communication;
• forward-progress verification.

The complete test returned success.

The validation environment also required the loader debug-symbol compatibility setup and the separately reported PMI_FD correction in the bundled DMTCP socket handling. Those source changes are intentionally not included in this pull request and are discussed separately in the umbrella issue.

Scope

This pull request adds only the opt-in integration test and its documentation.

It does not modify MANA's coordinator, restart wrapper, loader symbol lookup, or DMTCP submodule.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e41ef984-55df-44df-8e37-89ea047e9cb1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}