docs: add distributed MPICH/Hydra workflow

[alan-lira]

Refs #485

Summary

This pull request adds a practical guide for running MANA across multiple nodes with MPICH and Hydra.

I prepared this guide while converting a working two-node checkpoint-and-restart environment into a repeatable validation workflow. During that process, several small configuration details caused failures that initially appeared to be MPI or MANA problems.

Examples included accidentally selecting Open MPI compiler wrappers, launching remote ranks from a directory that did not exist on every node, allowing SSH to consume the remaining lines of a hostfile loop, placing .mana.rc outside the HOME seen by remote ranks, and restarting from a working directory that was incompatible with the current restart wrapper.

The purpose of this guide is to collect those details in one place and make the complete distributed workflow easier to reproduce, validate, and diagnose.

What the guide covers

The guide explains:

• how to build MANA with explicit MPICH compiler wrappers;
• how to verify that the lower half uses the intended MPI installation;
• why MANA, MPICH, application, checkpoint, and working-directory paths must be consistent across nodes;
• how to validate passwordless SSH and Hydra hostfiles safely;
• why ssh -n is important inside hostfile loops;
• why Hydra should receive an explicit -wdir;
• how to start and inspect the coordinator;
• how to use an isolated HOME and .mana.rc;
• how to run native MPI and finite MANA smoke tests before checkpointing;
• how to request a blocking checkpoint with a timeout;
• how to verify complete checkpoint images and detect incomplete .tmp files;
• how to terminate and restart a distributed execution;
• how to verify rollback, collective correctness, and continued forward progress;
• how loader debug symbols and GNU build-ID paths affect the lower half;
• how the MPICH/Hydra PMI_FD dependency relates to the separate DMTCP proposal;
• how to run the accompanying opt-in multi-node test;
• how to clean up without broad process-killing commands.

Why this documentation is useful

A distributed checkpoint-and-restart failure often occurs in a different layer from the visible symptom.

For example:

• a missing runtime library on one worker can look like an MPI process failure;
• a missing .mana.rc can look like a MANA launch problem;
• a loader debug-symbol lookup failure can abort one lower half while Hydra reports a collective communication failure from another rank;
• a process-manager descriptor handled incorrectly during checkpointing can appear as a checkpoint timeout.

The guide therefore recommends validating the environment incrementally:

1. verify identical paths and runtime dependencies;
2. run native MPICH/Hydra;
3. run a finite MANA application;
4. verify coordinator connectivity;
5. request and validate a checkpoint;
6. terminate and restart;
7. verify rollback and communication after restart.

This ordering made the distributed setup significantly easier to debug.

Relationship to the other proposals

The guide documents the current-main workflow and identifies places where companion proposals simplify it.

In particular:

• the persistent-coordinator proposal removes the need to invoke the underlying coordinator directly;
• the restart-directory proposal removes the current dependency on checkpoint-directory working paths and option ordering;
• the GNU build-ID proposal removes the need for a loader debug-symbol compatibility link;
• the multi-node integration-test proposal automates the workflow described here;
• the separate DMTCP dependency proposal addresses the required PMI_FD handling observed in the validated MPICH/Hydra environment.

Those source changes are intentionally not included in this documentation branch.

Validation

The guide is based on a successful end-to-end validation using:

• two physical nodes;
• MPICH 4.3.2 with Hydra;
• one MPI rank per node;
• native distributed MPI execution;
• distributed MANA launch;
• a persistent coordinator;
• an isolated test home and coordinator status file;
• a blocking checkpoint;
• complete checkpoint-image validation;
• process termination;
• distributed restart;
• rollback verification;
• continued MPI_Allreduce correctness;
• forward-progress verification after restart.

The documentation uses generic paths and hostnames. It does not contain cluster provisioning, private SSH material, scheduler credentials, checkpoint images, or benchmark source code.

Documentation location

I am opening this as a draft because I would appreciate guidance on the preferred long-term location.

The guide currently lives under doc/, with a small link from the main README. I understand that MANA also maintains documentation through the MANA Manual and a separate documentation project.

I am happy to move or divide the material according to the maintainers' preference. Possible options include:

• keeping the operational guide in this repository;
• moving the complete guide to the documentation repository;
• retaining a short README link here while storing the full guide elsewhere;
• folding selected troubleshooting sections into existing manual pages.

Scope

This pull request changes documentation only.

It does not modify coordinator behavior, restart behavior, the lower half, the DMTCP submodule, or the multi-node test implementation.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: bfc70d7c-401e-4edd-91d3-893b35b40bae

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}