[AI] Add agents harness

.agents/skills/cloudai-config-contribution/SKILL.md

@@ -0,0 +1,10 @@
+---
+name: cloudai-config-contribution
+description: Create, adapt, review, or troubleshoot CloudAI system, test, and test-scenario TOML configurations. Use for CloudAI config-file requests, not for implementing workload Python code.
+---
+
+# CloudAI config contribution
+
+- CloudAI is a public repo thus configs must contain no internal references
+- `conf/experimental` is the place to put configs
+- prefer test-in-scenario. Use `path`-based references in scenario when a single test TOML may serve different scenarios

.agents/skills/cloudai-remote-experiment/SKILL.md

@@ -0,0 +1,53 @@
+---
+name: cloudai-remote-experiment
+description: Run, monitor, stop, and retrieve a CloudAI experiment (test scenario) on a remote Slurm or standalone cluster.
+---
+
+## Run
+
+- Use [cloudai-remote-install](../cloudai-remote-install/SKILL.md) first. Identify
+  this checkout's remote installation and system config; do not use another checkout's deployment.
+  Use its wrapper for all remote commands and transfers, not direct SSH.
+- Run the CloudAI controller as a detached daemon that survives disconnects,
+  with stdin closed and output redirected. Keep separate per-run controller/debug
+  logs in remote home, not Lustre. Record the actual controller PID, start time,
+  deployment, selected configs, exact results directory and eventual exit status.
+- Establish expected progress and a workload-specific stall timeout before launching.
+  Include legitimate silent phases such as initialization; ask if expectations are unclear.
+
+## Monitor and stop
+
+- Stay responsible for monitoring until the run finishes or is explicitly handed back.
+  Check the recorded process with `ps` and read bounded log increments no more than
+  once per minute; back off when unchanged. Sample only a few known artifacts for progress.
+  No recursive scans, whole-log rereads or continuous result syncing on shared storage.
+- Never use `squeue --me` or equivalent user-wide queue queries. Let CloudAI poll
+  Slurm; do not add scheduler polling loops. If diagnosis requires scheduler state,
+  make a targeted query for this run's recorded job IDs only.
+- Track submitted job IDs from this run's own log. Completion metadata can corroborate
+  ownership but may not exist while jobs are running. Never infer ownership from
+  username, job name or checkout alone: a checkout may have several runs.
+- Silence alone is not a stall, and queued jobs are not wasting an allocation.
+  Stop confirmed stuck work after its stall timeout, using repeated evidence of
+  missing expected progress in an active allocation. Ask if the evidence is ambiguous.
+- Before stopping, recheck process identity and job ownership. Send SIGTERM to this
+  run's controller and allow a bounded grace period for finalization. If it does not exit,
+  keeps submitting jobs, or leaves DSE workers running, recheck ownership and send SIGKILL
+  to the surviving controller/workers belonging to this run. Once submissions have stopped,
+  refresh this run's job IDs and `scancel` its remaining exact allocation IDs, including
+  queued jobs. Killing local processes does not release Slurm allocations.
+  Verify termination and allocation release.
+  For standalone runs, stop only this run's process tree. Never use user-wide cancellation
+  or broad process-name matching. Preserve logs explaining why the run was stopped.
+
+## Results
+
+- Confirm completion from logs and job outcomes, not just a vanished PID or zero exit code.
+  After success, failure or cancellation, copy the exact run directory into
+  `results/<cluster-nickname>/<run-directory>/` locally, without overwriting another run.
+- Derive the cluster nickname from the selected config and SSH target, not blindly
+  from the config name: `<cluster>-for-testing` still belongs under `<cluster>`.
+  Ask if ambiguous.
+- Stream an archive through the wrapper's `run` action for retrieval (`copy` uploads only).
+  Copy this run's output, not the shared results tree; preserve remote files.
+  Report the outcome and local results path, including any incomplete retrieval.

.agents/skills/cloudai-remote-install/SKILL.md

@@ -0,0 +1,55 @@
+---
+name: cloudai-remote-install
+description: Install or update a CloudAI checkout on a remote cluster for Slurm or standalone execution. Does not run benchmarks.
+---
+
+## Installation
+
+- One local checkout -> one remote installation with its own uv environment.
+  Use remote home for Python projects and environments, not the shared artifact directory.
+- Deploy the main local checkout to `~/cloudai`. Never delete it or deploy a
+  linked worktree over it. Keep it maintained without overwriting remote edits.
+- Deploy linked worktrees to `~/cloudai-worktrees/<checkout-id>`.
+  Use the first 16 hex characters of SHA-256 of local hostname + NUL + resolved
+  checkout path, not the branch name. Reuse that directory on subsequent deployments.
+- Transfer only files needed for installation and the task, including local
+  changes. Exclude local environments, caches, bytecode and Git metadata.
+  Git-based deployment is also fine.
+- Use [scripts/deploy.py](scripts/deploy.py) for all remote commands and transfers,
+  not direct SSH. It only provides transport; perform the checks and setup described here.
+  Invoke with Python: `deploy.py HOST run 'COMMAND'` or `deploy.py HOST copy DEST SOURCE...`.
+- Reuse remote uv, or install it if missing. Use it to install CloudAI and manage
+  the deployment's own environment; do not reuse another checkout's virtualenv.
+- Do not update an installation while running or queued work still uses it.
+- Preserve remote edits and personal files, including custom TOMLs. Do not prune remote-only files.
+- Verify CLI startup and the selected system config after installation, without running benchmarks.
+
+## Cluster configuration
+
+- Reuse the user's existing system config across installations on the same
+  cluster. Keep shared configs outside deployment directories.
+- If none exists, inspect the cluster and adapt the closest repository example:
+  cluster name, scheduler, partitions, installation/results paths and any required
+  account or other cluster options. Use the current models for valid fields.
+- Different execution backends may need different configs, but configs on
+  the same cluster should share artifact installation and results paths.
+  Use NFS/Lustre for these paths.
+- Look for existing config and storage hints in the user's SSH configuration
+  and cluster setup. If information is not readily available, propose what you
+  can establish and ask the user rather than guessing or searching broadly.
+- Persist `CLOUDAI_SYSTEM_CONFIG` in the appropriate remote shell startup file.
+  Feature-specific configs should be explicit overrides, not replacements
+  for the user's default. Pass the selected config to noninteractive commands
+  too; they may not load the shell startup file.
+
+## Cleanup
+
+- During remote work, check for deployments unused for more than 21 days.
+  Touch `.cloudai-last-used` after installation and whenever using the deployment.
+  Missing markers or unrecorded manual use are not
+  proof of inactivity.
+- Ask before removing each candidate, after checking running and queued jobs.
+  Never remove `~/cloudai`, active or queued-job deployments, or shared configs
+  and artifacts.
+- If cleanup is declined, place `.cloudai-keep` in that
+  deployment and exclude it from future cleanup suggestions.

.agents/skills/cloudai-remote-install/scripts/deploy.py

@@ -0,0 +1,64 @@
+# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import shlex
+import subprocess
+import sys
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(add_help=False)
+    parser.add_argument("host")
+    parser.add_argument("--dry-run", action="store_true")
+    actions = parser.add_subparsers(dest="action", required=True)
+    run = actions.add_parser("run", add_help=False)
+    run.add_argument("command")
+    copy = actions.add_parser("copy", add_help=False)
+    copy.add_argument("destination")
+    copy.add_argument("sources", nargs="+")
+    args = parser.parse_args()
+
+    ssh = ["ssh", "-T", "-o", "RemoteCommand=none", "-o", "BatchMode=yes"]
+    if args.action == "run":
+        command = [*ssh, "--", args.host, args.command]
+    else:
+        command = ["rsync", "-a", "-e", shlex.join(ssh)]
+        for pattern in (
+            ".git",
+            ".venv",
+            "venv",
+            "env",
+            ".env",
+            ".cloudai.toml",
+            ".cloudai-*",
+            ".DS_Store",
+            ".*cache*",
+            "__pycache__",
+            "*.py[cod]",
+            "*.egg-info",
+        ):
+            command.extend(["--exclude", pattern])
+        destination = shlex.quote(args.destination.removeprefix("~/"))
+        command.extend(["--", *args.sources, f"{args.host}:{destination}"])
+
+    print(shlex.join(command), file=sys.stderr, flush=True)
+    if not args.dry_run:
+        raise SystemExit(subprocess.call(command))
+
+
+if __name__ == "__main__":
+    main()

.agents/skills/cloudai-workload-contribution/SKILL.md

@@ -0,0 +1,40 @@
+---
+name: cloudai-workload-contribution
+description: Create, modify, or review CloudAI workload implementations. Use for workload Python code, not for TOML-only configuration changes.
+---
+
+This skill is a set of guidelines when working on workloads implementation
+
+## Implementation guidelines
+
+- User configs is a trusted data. Don't over-validate test definitions
+- The workload implementation is intended to be pass-through, which means that CloudAI:
+  - defines benchmark shape (processes, installables, etc.)
+  - translates TOML test config into workload interface submission so that user can use the workload fully
+  - doesn't (re-)define underlying workload parameters unless required for workload submission and clean code
+
+- CloudAI cannot support every possible cluster-specific hardware/software setup in terms of benchmark
+  startup/finalization. Prefer generic solutions instead of specific technologies support in the workloads
+  implementation. The generic solutions are:
+
+    - container mounts
+    - installables
+    - pre/post-srun scripts (custom per workload; some workloads already support it)
+    - pre/post-tests for heavy lifting hooks that need an srun
+
+- Fetching workloads sources to understand how to use them. Use `results/vendor-src` folder for it. When
+  working under a worktree, re-use this folder from the main repo checkout. Be aware of the checkout version. Have a
+  single checkout for one code source (switch checkouts)
+- Make the most of parent CloudAI command generation classes so that the workload supports all the builtin features,
+  like single-sbatch, sbatch directives, pre/post-test hooks, DSE/CloudAIGym. When it's too complicated to support one
+  - do not (80-20 rule)
+
+## Testing guidelines
+
+- Don't produce too many unit-tests covering a small feature. More tests != better
+- Prefer maintaining end-to-end tests (tests/test_acceptance.py). Cover distinct behavior and compatibility risks;
+  avoid redundant cases rather than limiting the number of tests
+- Existing workload TOML configs from `conf/` must stay supported. One may extend them with new features; add scenario
+  test cases for the new behavior and affected compatibility paths
+- If a workload may result in diverse execution shapes (number of processes and their orchestration), then prepare that
+  many test cases (not scenarios)

.claude/skills

@@ -0,0 +1 @@
+../.agents/skills

.gitignore

@@ -4,13 +4,11 @@ __pycache__/
 *$py.class
 
 # Distribution / packaging
-.Python
 build/
 develop-eggs/
 dist/
 downloads/
 eggs/
-.eggs/
 lib/
 lib64/
 parts/
@@ -19,7 +17,6 @@ var/
 wheels/
 share/python-wheels/
 *.egg-info/
-.installed.cfg
 *.egg
 doc/_build
 
@@ -33,42 +30,23 @@ doc/_build
 pip-log.txt
 pip-delete-this-directory.txt
 
-# Jupyter Notebook
-.ipynb_checkpoints
-
 # IPython
 profile_default/
 ipython_config.py
 
-# pyenv
-.python-version
-
 # PEP 582; used by e.g. github.com/David-OConnor/pyflow
 __pypackages__/
 
 # Environments
-.env
-.venv
 env/
 venv/
 ENV/
 env.bak/
 venv.bak/
 
 # mypy
-.mypy_cache/
-.dmypy.json
 dmypy.json
 
-# pytype static type analyzer
-.pytype/
-
-# pycharm
-.idea/
-
-# VSCode
-.vscode/
-
 # Editors and IDEs
 *.swp
 *.bak
@@ -78,11 +56,6 @@ dmypy.json
 *.sublime-workspace
 
 # OS generated files
-.DS_Store
-.DS_Store?
-._*
-.Spotlight-V100
-.Trashes
 ehthumbs.db
 Thumbs.db
 
@@ -92,4 +65,9 @@ install/
 results/
 .*
 !.pre-commit-config.yaml
-.cloudai.toml
+!.agents/
+.agents/*
+!.agents/skills/
+!.claude/
+.claude/*
+!.claude/skills

AGENTS.md

@@ -0,0 +1,32 @@
+# CloudAI repository guidance
+
+## Implementation conventions
+
+- Avoid over-engineering
+- Prefer Google Python style guide:
+  - Blend into existing code
+  - No asserts in production code
+  - Prefer absolute imports (`import x`).
+    Use `from x import y` when qualified names are too long.
+    Use relative imports when existing code uses it.
+
+- Backwards compatibility is very important. CloudAI may be integrated into other tools. Users maintain their own
+  CloudAI configs that we may never see. Backwards incompatible changes should be avoided unless explicitly asked. In
+  that case the changes must be highlighted.
+- Update affected documentation in `README.md` or `doc/` when public behavior changes.
+- Import public core APIs through `cloudai.core`, respect import-linter boundaries, and use existing lazy-import
+  mechanisms for heavy modules. Follow established workload structure and registration patterns.
+- Follow `CONTRIBUTING.md`, including SPDX headers and mirrored tests for new Python modules.
+
+## Verification
+
+- Start with focused tests: `uv run --locked --extra dev pytest <test-paths>`.
+- Run `uv run --locked --extra dev pre-commit run --files <changed-files>` and review formatter edits.
+
+## Contribution
+
+- Do not commit, push, open or modify pull requests, or run remote jobs unless explicitly asked to do so.
+- Create PRs as drafts (`gh pr create --draft`). Leave marking PRs ready for review to humans unless asked to do so.
+- Follow PR requirements in `CONTRIBUTING.md` and use `.github/PULL_REQUEST_TEMPLATE.md`.
+- Keep public configs reusable (no internal resources). Never commit credentials, private artifacts, or internal
+  hostnames.

CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md