v1.1.1: air-gapped Slurm support, dual-mode runtime, GRPO multi-env hardening - #5
Merged
Conversation
…ardening - Air-gapped Slurm support: pre-built nemo-skills/nemo-rl/vLLM/vllm-grpo containers with offline env vars (HF_*_OFFLINE, UV_OFFLINE, TIKTOKEN_*); reproducible Dockerfiles + build instructions - Dual-mode runtime: nvflow/lib/runtime.py centralises venv/python resolution so the same code path runs in airgap and dev modes; checkpoint converter, vLLM serving, training, and SDG scripts all use it - sbatch arg propagation: nvflow/lib/sbatch.py + tests plumb extra_sbatch_args through every Slurm submission - GRPO eval per-environment: outputs split by env (equivalence_llm_judge, finance_sec_search); restored sequence_packing / logprob_chunk_size / make_sequence_length_divisible_by for the equivalence_llm_judge training policy - Eval / SDG / SFT fixes: skip tiktoken download when cache env vars pre-configured, eval base/demo config refinements, removed unused nemotron_nano_9b SFT config - CI: lightweight unit-tests workflow on both GitLab and GitHub. tests/test_sbatch.py reads source files via Path(...).read_text() instead of importing modules that transitively need nemo_skills / typer; both runners invoke `uv run --no-sync pytest tests/ -v --tb=short` so the --no-deps install is authoritative - Documentation: INSTALL.md rewrite, dockerfiles/README + docker_instructions, finance troubleshooting guide, cluster-configuration docs Signed-off-by: Pritam Gundecha <pgundecha@nvidia.com>
psgundecha-nv
force-pushed
the
release/v1.1.1
branch
from
May 18, 2026 02:26
8253271 to
3b097b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
NVFlow v1.1.1 introduces first-class air-gapped Slurm support for the finance recipe, refactors the runtime layer so the same code path works in both air-gapped and developer modes, and ships a set of GRPO/eval/SFT/SDG hardening fixes plus a documentation overhaul.
This is a drop-in upgrade from v1.1.0 — no breaking changes for existing developer-mode users.
Highlights
Air-gapped Slurm deployments
Run NVFlow on Slurm clusters with no network access from compute nodes.
dockerfiles/with reproducible images fornemo-skills,nemo-rl,vllm,vllm-grpo, plus end-to-end build/push instructions (dockerfiles/docker_instructions.md).HF_HUB_OFFLINE,HF_DATASETS_OFFLINE,TRANSFORMERS_OFFLINE,UV_OFFLINE,TIKTOKEN_*.cluster_configs/containers.yamlandtemplate-slurm.yamlupdated for offline images.INSTALL.mdand finance recipe docs rewritten to cover both runtime modes; newdocs/recipes/finance/troubleshooting.md.Dual-mode runtime
A single code path now serves both modes — no need to fork configs or scripts when switching between an air-gapped cluster and a connected developer workstation.
nvflow/lib/runtime.pycentralises venv / Python interpreter resolution.scripts/serve_vllm_patched.py), SFT/GRPO training stages, and SDG scripts all transparently fall back touv run --active pythonwhen the pre-baked venv isn't present.Slurm submission ergonomics
nvflow/lib/sbatch.py(+tests/test_sbatch.py) plumbsextra_sbatch_argsthrough every Slurm submission viaget_executor, so cluster-specific flags (e.g.--exclude,--gres-flags,--account) propagate to all stages without per-stage code changes.GRPO / eval / SFT / SDG hardening
step-9-eval/<env>/step-N/…, eliminating cross-environment overwrite and making downstream comparison straightforward.sequence_packing.enabled: true,logprob_chunk_size: 2048, andmake_sequence_length_divisible_by: 16were inadvertently dropped during a prior config refactor and have been restored.TIKTOKEN_ENCODINGS_BASE/TIKTOKEN_CACHE_DIRare pre-configured.workflows/eval/{base,demo}.yaml).sft/nemotron_nano_9b.yaml.Documentation
INSTALL.mdrewritten end-to-end to cover both runtime modes and prerequisites.dockerfiles/README.mdanddockerfiles/docker_instructions.mddocument image build/push lifecycle.docs/recipes/finance/troubleshooting.md(new) collects common Slurm/airgap pitfalls.docs/architecture/ARCHITECTURE.mdanddocs/cluster-configuration.mdupdated for the new runtime layer.Compatibility
HF_*_OFFLINE,UV_OFFLINE, andTIKTOKEN_*env vars are documented but only required for air-gapped deployments.nflowCLI or workflow YAML schemas.Validation
Full demo workflow (baseline-eval → SEC download → SDG → SFT → GRPO) re-run end-to-end on air-gapped containers and compared against the v1.1.0 baseline:
Upgrade notes
No action required for developer-mode users. For air-gapped Slurm deployments:
dockerfiles/README.md).cluster_configs/template-slurm.yamlper the new template (offline env vars + image paths).NRL_FORCE_REBUILD_VENVS=trueand Gym overlay mounts on air-gapped clusters (they're now redundant with baked images).See
INSTALL.mdanddocs/recipes/finance/troubleshooting.mdfor full setup details.Commit
Signed-off-by).