Eval Gemma-4 via SLURM + W&B by aaronsmulktis · Pull Request #66 · facebookresearch/OpenApps

aaronsmulktis · 2026-08-01T04:51:06Z

Add a one-command sbatch path to run the agent eval on the FAIR cluster, co-located with vLLM (in my login node) so the agent reaches Gemma over the internal network (no SSH tunnel), and logs to the Meta-internal W&B in the open_apps_aaronsmulktis project.

scripts/conduct_slurm.sh: sbatch wrapper around conduct.sh. Requests CPU resources (memorization account, h200_memorization_high qos), auto-discovers the node serving google/gemma-4-E2B-it on :8000 (skippable via VLLM_HOST), and runs the worker pool with agent.hostname=<node> use_wandb=True.
config/mode/slurm_cluster.yaml: add slurm_account/slurm_qos, set a real partition default, point logs_dir at /checkpoint/memorization/$USER.
config/config.yaml: W&B project open_apps_aaronsmulktis; add group/job_type.
src/open_apps/launcher.py: descriptive W&B run names, per-run success/ cum_reward summary metrics, and fix a webapp-log PIPE deadlock.
config/agent/gemma-4-e2b-it.yaml: vLLM client config for Gemma-4.
docs/agents.md: "Running Evals on the Cluster" section (two-job model, discovery, VLLM_HOST override, srun --overlap fallback).
scripts/conduct.sh, docs/notes-aaron.md: bring along local working files.

Example command:
AGENTS=gemma-4-e2b-it COUNT=3 sbatch scripts/conduct_slurm.sh

Outputs:
(oa-env) ~/projects/OpenApps$ AGENTS=gemma-4-e2b-it COUNT=3 sbatch scripts/conduct_slurm.sh sbatch: WARNING: Partition is inferred from QOS prefix so --partition flag is ignored sbatch: Job submitted to Partition: h200 Submitted batch job 10249433

Add a one-command sbatch path to run the agent eval on the FAIR cluster, co-located with vLLM so the agent reaches Gemma over the internal network (no SSH tunnel), and logs to the Meta-internal W&B. - scripts/conduct_slurm.sh: sbatch wrapper around conduct.sh. Requests CPU resources (memorization account, h200_memorization_high qos), auto-discovers the node serving google/gemma-4-E2B-it on :8000 (skippable via VLLM_HOST), and runs the worker pool with agent.hostname=<node> use_wandb=True. - config/mode/slurm_cluster.yaml: add slurm_account/slurm_qos, set a real partition default, point logs_dir at /checkpoint/memorization/$USER. - config/config.yaml: W&B project open_apps_aaronsmulktis; add group/job_type. - src/open_apps/launcher.py: descriptive W&B run names, per-run success/ cum_reward summary metrics, and fix a webapp-log PIPE deadlock. - config/agent/gemma-4-e2b-it.yaml: vLLM client config for Gemma-4. - docs/agents.md: "Running Evals on the Cluster" section (two-job model, discovery, VLLM_HOST override, srun --overlap fallback). - scripts/conduct.sh, docs/notes-aaron.md: bring along local working files.

marksibrahim

Thank you Aaron! Glad to see Gemma-4 is running. I left a few minor comments.

For future pull requests, make sure you include the command/s you used to verify the correctness of the code (ideally with the output) so we can trace errors backward in case they come up

marksibrahim · 2026-08-06T21:12:57Z

            time.sleep(4)
            kill_ports(ports=[self.web_app_port])
            print("OpenApps successfully stopped.")
+        webapp_log_fh = getattr(self, "_webapp_log_fh", None)


Can you add a couple of comments on what this block is doing and how it works?

Yep this is part of the runtime cleanup after the process exits, it closes and releases the webapp.log file.

marksibrahim · 2026-08-06T21:15:45Z

+aws_session_token: null
+aws_region: us-west-2
+
+custom_actions:


Since this a computer-use agent purely based on screenshots (no axtree / html) let's indicate this with a postfix like we did for GPT. See https://github.com/facebookresearch/OpenApps/blob/main/config/agent/GPT-5.5-computer-use.yaml

Yes good call, done. Just renamed the file & the model_pretty_name, but happy to keep rolling out a more simpler name everywhere if needed.

…ulktis/gemma-eval

aaronsmulktis · 2026-08-07T01:47:45Z

For future pull requests, make sure you include the command/s you used to verify the correctness of the code (ideally with the output) so we can trace errors backward in case they come up

Yes will do. I documented the commands in agents.md but there's now an example in the PR message (along with some output). Tho I think there could be better logging for this multi-run concurrency script in general.

Copilot

Pull request overview

Adds a SLURM-oriented workflow for running OpenApps agent evals on-cluster alongside a persistent vLLM serve job, with W&B logging improvements and some launcher reliability fixes to support multi-run sweeps.

Changes:

Introduces scripts/conduct.sh (parallel local worker pool) and scripts/conduct_slurm.sh (sbatch wrapper with vLLM node discovery).
Updates launcher behavior for W&B run naming/summary metrics and avoids webapp subprocess PIPE deadlock by logging to file.
Adds/updates cluster and agent configuration plus documentation for the two-job (vLLM + eval) workflow.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/open_apps/launcher.py	W&B run naming + run summaries; fixes webapp subprocess logging/deadlock behavior.
src/open_apps/agent/dummy_agent.py	Import cleanup + formatting; no functional change intended.
scripts/conduct.sh	New parallel worker-pool launcher for multiple runs and per-run logs.
scripts/conduct_slurm.sh	New SLURM wrapper with vLLM host discovery and forwarding to `conduct.sh`.
docs/agents.md	New “Running Evals on the Cluster” section describing the two-job workflow.
config/mode/slurm_cluster.yaml	Updates SLURM-mode defaults (currently placeholders) and log directory settings.
config/config.yaml	W&B defaults updated (currently hardcoded to a user-specific project).
config/agent/gemma-4-computer-use.yaml	New vLLM client agent config for Gemma-4 serving via internal network.

Suppressed comments (3)

scripts/conduct_slurm.sh:96

The default AGENTS value is gemma-4-e2b-it, but there is no config/agent/gemma-4-e2b-it.yaml in this PR (only config/agent/gemma-4-computer-use.yaml). As written, the sbatch wrapper will fail with a missing Hydra config unless the user overrides AGENTS.

AGENTS="${AGENTS:-gemma-4-e2b-it}" \
COUNT="${COUNT:-1}" \

docs/agents.md:176

This pinned-host command uses AGENTS=gemma-4-e2b-it, but this PR adds config/agent/gemma-4-computer-use.yaml (and no gemma-4-e2b-it agent config). Update the example so it runs as-is.

VLLM_HOST=h200-000-026 AGENTS=gemma-4-e2b-it COUNT=1 sbatch scripts/conduct_slurm.sh

docs/agents.md:188

This fallback example uses AGENTS=gemma-4-e2b-it, but the agent config introduced by this PR is gemma-4-computer-use.yaml. Update the example so it matches an existing config name.

  env AGENTS=gemma-4-e2b-it COUNT=1 VLLM_HOST=localhost \

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+# Submit:
+#   AGENTS=gemma-4-e2b-it COUNT=1 sbatch scripts/conduct_slurm.sh
+#   AGENTS="gemma-4-e2b-it" COUNT=20 MAX_PARALLEL=4 sbatch scripts/conduct_slurm.sh


+# smoke test (1 run)
+AGENTS=gemma-4-e2b-it COUNT=1 sbatch scripts/conduct_slurm.sh
+
+# larger sweep
+AGENTS="gemma-4-e2b-it" COUNT=20 MAX_PARALLEL=4 sbatch scripts/conduct_slurm.sh


+# For client_type=vllm the URL is built as http://${hostname}:${port}/v1
+# (base_url is ignored). The vLLM node changes every SLURM allocation, so pass it
+# at launch, e.g.: uv run launch_agent.py agent=gemma-4-e2b-it agent.hostname=h200-000-026
+hostname: null


 project: open_apps

-logs_dir:  /checkpoint/${oc.env:USER}/logs/${project}/${now:%Y-%m-%d_%H-%M-%S}-${oc.env:USER}-${agent.model_name}/${job_id}
+logs_dir:  /example/dir


+  slurm_account: example_replace_me
+  slurm_qos: qos_example_replace_me
+  slurm_partition: partition_example_replace_me


 wandb:
-  project: ${project}
+  # project: ${project}
+  project: open_apps_aaronsmulktis
  entity: ${oc.env:USER}


            kill_ports(ports=[self.web_app_port])
            print("OpenApps successfully stopped.")
+        webapp_log_fh = getattr(self, "_webapp_log_fh", None)
+        if webapp_log_fh is not None:
+            webapp_log_fh.close()


meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 1, 2026

aaronsmulktis added 2 commits August 3, 2026 12:21

[cleanup] remove aarons notes

c83de40

Cleanup slurm config

9a1987d

aaronsmulktis changed the title ~~Run OpenApps evals against cluster-hosted Gemma-4 via SLURM + W&B~~ Eval Gemma-4 via SLURM + W&B Aug 6, 2026

marksibrahim requested changes Aug 6, 2026

View reviewed changes

aaronsmulktis self-assigned this Aug 6, 2026

aaronsmulktis added 2 commits August 6, 2026 17:27

Merge branch 'main' of github.com:aaronsmulktis/OpenApps into aaronsm…

3b0fe51

…ulktis/gemma-eval

[Tweak] rename Gemma 4 agent with computer-use postfix

ed08eaa

aaronsmulktis requested a review from marksibrahim August 6, 2026 22:53

aaronsmulktis requested a lite review from Copilot August 7, 2026 01:48

Copilot started reviewing on behalf of aaronsmulktis August 7, 2026 01:48 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

Navigation Menu

Uh oh!

Eval Gemma-4 via SLURM + W&B - #66

Eval Gemma-4 via SLURM + W&B#66
aaronsmulktis wants to merge 5 commits into
facebookresearch:mainfrom
aaronsmulktis:aaronsmulktis/gemma-eval

aaronsmulktis commented Aug 1, 2026 •

edited

Loading

Uh oh!

marksibrahim left a comment

Uh oh!

Uh oh!

Uh oh!

marksibrahim Aug 6, 2026

Uh oh!

aaronsmulktis Aug 6, 2026

Uh oh!

marksibrahim Aug 6, 2026

Uh oh!

aaronsmulktis Aug 6, 2026

Uh oh!

aaronsmulktis commented Aug 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

aaronsmulktis commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marksibrahim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

marksibrahim Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

aaronsmulktis Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

marksibrahim Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

aaronsmulktis Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

aaronsmulktis commented Aug 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aaronsmulktis commented Aug 1, 2026 •

edited

Loading