coordinator: add persistent operation mode

[alan-lira]

Summary

This pull request adds an optional persistent mode to mana_coordinator.

During multi-node MPICH/Hydra checkpoint-and-restart experiments, I found that the coordinator may temporarily have no connected MANA processes between the termination of the original execution and the start of the restarted execution.

The current wrapper always starts DMTCP's coordinator with --exit-on-last. Consequently, the coordinator can terminate as soon as the final process from the original execution disconnects. The subsequent restart then has no coordinator to reconnect to.

This change introduces:

mana_coordinator --persistent

When this option is used, the wrapper does not pass --exit-on-last to dmtcp_coordinator. Without --persistent, the current behavior remains unchanged.

Refs #485

Motivation

The option is useful when the coordinator is intentionally started independently from the MPI application and is expected to remain available across a checkpoint, process termination, and restart sequence.

This is particularly relevant for distributed MPICH/Hydra executions where:

1. the coordinator is started first;
2. the MPI application connects to it;
3. a blocking checkpoint is requested;
4. the original processes are terminated;
5. the application is restarted from the checkpoint images;
6. the restarted processes reconnect to the same coordinator.

Without persistent operation, the interval between steps 4 and 5 may cause the coordinator to exit.

Behavior

The existing command:

mana_coordinator --port 7780

continues to use --exit-on-last.

The new form:

mana_coordinator --persistent --port 7780

keeps the coordinator alive when the last client disconnects.

The new option does not change the default behavior.

Implementation

The wrapper now:

• recognizes the --persistent option;
• preserves the existing coordinator arguments;
• omits --exit-on-last only when persistent mode is explicitly requested;
• continues to create the normal MANA status file.

Tests

The pull request adds focused wrapper tests covering:

• default mode includes --exit-on-last;
• persistent mode omits --exit-on-last;
• persistent mode preserves other coordinator arguments;
• normal status-file behavior remains available.

The tests use a temporary fake coordinator, so they do not require a running MPI job or a cluster environment.

Validation

I also validated the behavior with a real coordinator process:

• the coordinator was started with --persistent;
• no clients were connected;
• the process remained available after the idle interval;
• dmtcp_command --status could still query it;
• it was subsequently terminated cleanly with dmtcp_command --quit.

Scope

This pull request is intentionally limited to the coordinator lifecycle.

It does not include the restart-directory correction, MPICH/Hydra integration test, loader debug-symbol handling, documentation, or the separate DMTCP PMI_FD proposal described in the umbrella issue.

[coderabbitai]

Warning

Review limit reached

@alan-lira, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 48 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a3391597-bd56-4d4d-bcf5-85d508e40fbf

📥 Commits

Reviewing files that changed from the base of the PR and between f296e33 and c992152.

📒 Files selected for processing (3)

• bin/mana_coordinator
• ci/test-mana-coordinator.py
• ci/unit-test.sh

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}