build(docker): add the OpenClaw sandbox image - #19
Open
geojaz wants to merge 11 commits into
Open
Conversation
Only gemini_cli was migrated, so a sandboxed matrix could not actually run the other three arms: AgentHarness.run refuses a sandboxed harness that has not been migrated, rather than silently running it on the host. The published antigravity and claude_code runs WERE sandboxed, but by an uncommitted patch and an untracked tree on the bastion -- neither in any PR, neither reviewed, and neither present in this lineage. Each harness now routes its agent turn through run_agent_cmd and declares supports_sandbox. Fixture mounts need no per-harness work: the eval harness already builds them centrally into the spec. openclaw needed more than a flag. Its state dir and config path cross the boundary as env VALUES, and the host spelling means nothing inside the container, so the agent turn now gets container-translated paths while the post-run 'oc sessions' / export-trajectory calls keep the host spelling. Both read the same bytes: the state dir lives under the workspace, which is the bind mount itself. The container also has its own oc on PATH, so the host binary path is not passed through. container_path() is lifted out of SandboxExecutor.map_host_path so the cwd mapping and these value translations cannot drift apart. secret-rotation opts out via a new Task.requires_unsandboxed. The task IS credential work -- it rotates a Secret Manager version through ADC, and ADC is exactly what the boundary strips, so sandboxing it would not harden it, it would make it impossible. The harness skips the sandbox for that task only and logs a warning, so a sandboxed matrix shows which task did not get a boundary. The flag clears config.sandbox rather than just skipping spec completion: the agent's own gate reads that field, and a leftover skeletal spec would refuse the run.
… all Provider deduction keys off the stack DIRECTORY NAME and only exactly 'kind'/'vcluster' deduce. The directory is 'cp-recovery-kind', which does not, so the run died at provisioning with 'stack requires an explicit provider' before the agent ever started. Verified: _select_provider raises for this task and for no other -- every b-00xx task already declares provider: kind, and this was the only ported kind task that did not. Infrastructure selection, not grading: nothing about the prompt, the verification spec or the expected output changes, so the task remains comparable with the rows already scored against it.
…cs-server on kind The stack configured provider "kubernetes" from module.cluster.endpoint, whose value falls back to the vcluster submodule. That submodule's own resources are served by the same provider, so tofu refused the graph with "Cycle: module.cluster.module.vcluster..." before planning anything -- for every infra_provider, which left the task unable to provision at all. Add a vcluster-free managed_endpoint output and read that instead: the ordering edge on the cluster survives, the cycle does not. Also install metrics-server when infra_provider=kind. The HPA objective grades ScalingActive=True, which needs a live metrics pipeline that a stock kind cluster lacks; without it the objective fails for provider reasons rather than for anything the agent did.
The task was pinned to gcp for one reason: the HPA objective grades ScalingActive=True and a stock kind cluster ships no metrics pipeline, so the objective failed for provider reasons rather than for anything the agent did. The stack now installs metrics-server under infra_provider=kind, so that reason is gone and the objective is decided by the agent again. Verified on kind end to end: the cluster provisions, the seeded workload rolls out, and kubectl top pods returns live CPU for the target deployment. Only the infrastructure provider changes. The prompt, expected_output and verification_spec are untouched, so the task grades exactly what it did before. INFRA_PROVIDER=gcp still selects GKE.
It was gcp before: the k-sigs migration (kubernetes-sigs#143) declared provider: gcp explicitly, and that is the definition every published run of this task was graded against. Switching the provider changes what the task measures, so it should not move as a side effect of making kind viable. The metrics-server install and the vcluster cycle fix stay. They were the real bugs: the cycle blocked provisioning on EVERY provider, and without metrics-server the HPA objective failed on kind for provider reasons rather than agent ones. kind is now a genuinely working alternative for local iteration. What still argues for gcp is the load path, not metrics: on GKE the Service is a LoadBalancer the chaos generator reaches directly, while on kind it is a ClusterIP behind the harness port-forward. The planned spike failed to inject in 8 of 8 published runs, so the fewer moving parts in that path the better until a spike demonstrably lands. Either way every runner must use the same provider for this task -- the Service type and load path differ, so a kind arm and a GKE arm are not comparable.
Same defect optimize-scale had: the kubernetes and helm providers were configured from module.cluster.endpoint, whose value falls back to the vcluster submodule, and that submodule's resources are served by those same providers. Tofu rejected the graph with "Cycle: ...module.vcluster..." before planning anything, so the task could not provision on any provider -- it failed on both arms in the 2026-09-08 rerun campaign. Read managed_endpoint instead, forwarded through this stack's local cluster wrapper. tofu plan now succeeds: 18 to add, no cycles.
b-0059 and b-0061 grade NetworkPolicy enforcement, and kindnet does not enforce policy, so both stacks pass disable_default_cni = true to the cluster module. The module never declared that variable: tofu failed at init with "An argument named disable_default_cni is not expected here", so neither task has ever provisioned. Their setup scripts already document the intent -- Calico is expected to be installed by the cluster module before they run. Implement it: swap kindnet for Calico v3.27.3 behind the variable. Two details that are not obvious. Nodes stay NotReady until a CNI is running, so kind_cluster.wait_for_ready has to be off in that case or the apply blocks until timeout on exactly the clusters that need Calico. null_resource.calico does the waiting instead, and duplicate_context now waits on it too. pod_subnet is set to Calico's own default pool (192.168.0.0/16) rather than kind's 10.244.0.0/16; left mismatched, Calico allocates outside the subnet the node spec and kube-proxy expect. The variable defaults to false and the networking block is emitted only when it is true, so every existing kind task plans exactly as before (verified on b-0024: no Calico resource, wait_for_ready still true, same 3 resources). Verified end to end on a real kind cluster: node Ready, calico-node 1/1, podCIDR 192.168.0.0/24, and a default-deny ingress policy actually enforced -- curl returns 200 before the policy and fails after it. Under kindnet the same probe would keep returning 200.
_build_agent_config_snapshot rebuilds AgentConfig field by field, and extra_flags was not among the fields it copied. AGENT_EXTRA_FLAGS parsed correctly into AgentConfig.from_env and was then dropped on the floor before the agent ever saw it, with nothing logged. The visible symptom is agy keeping its 5m default --print-timeout on every benchmark run: a task whose agent turn runs longer dies with "timeout waiting for response", an errored run with a truncated trajectory. b-0060 hit it twice at 313s and 308s while every run that finished did so under 250s. The documented mitigation for that flake is to pass a larger --print-timeout, which was impossible through the harness. Verified: the snapshot now carries (\x27--print-timeout=24h\x27) from the env, and the 249 agent/evalharness unit tests still pass.
The judge and the chaos driver resolve models differently: the judge reaches Vertex, where gemini-3.1-pro exists, while the chaos driver goes through the API key, whose endpoint only publishes gemini-3.1-pro-preview and 404s the plain id. The existing note covers the judge, so a working judge reads as evidence the whole config is fine. It cost a run to find. optimize-scale came back chaos_invalidated with the spike never injected, which looks like a fault-injection problem rather than the driver being unable to call a model at all.
BENCH_SANDBOX_IMAGE needs an image per harness and only the Gemini CLI one existed, so a sandboxed OpenClaw run had nothing to point at and needed an image built by hand off-tree. Built as a sibling of Dockerfile.gemini-cli rather than a shared multi-stage base with per-agent targets, for the reason that file already gives: the CLIs pin their own runtimes, so a shared stage buys little while coupling every agent's rebuild, and a version bump for one agent would invalidate the others' baselines. Two details are load bearing and are commented in the file. npm registers the CLI's bin as `openclaw`, not `oc`, so without an explicit symlink `oc` resolves to nothing inside the image and every sandboxed run fails looking for the binary. And the anthropic-vertex provider is only discovered under the CLI's own stock extensions root, so installing the package alone leaves it unregistered and absent from `oc plugins list`; it is copied into that root and the npm copy removed, so there is one on-disk copy rather than two that can drift. Both steps are asserted at build time so a silent regression fails the build instead of a run. Also bumps kubectl and helm in Dockerfile.gemini-cli to match the versions the new image pins, so the two sandbox images agree on the tools they hand the agent. The Gemini CLI version itself is deliberately untouched: it is the agent under test, not part of the toolbox, and bumping it would invalidate existing baselines.
pradeepvrd
force-pushed
the
feat/sandbox-all-harnesses
branch
from
September 9, 2026 23:27
afd98dd to
d2fc756
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BENCH_SANDBOX_IMAGE needs an image per harness, and only the Gemini CLI one existed. A sandboxed OpenClaw run had nothing to point at, and until now the image had to be built by hand off-tree.
This follows the existing one-image-per-harness structure (a sibling of
Dockerfile.gemini-clirather than a shared multi-stage base with per-agent targets), for the reason that file already gives: the CLIs pin their own runtimes, so a shared stage buys little while coupling every agent's rebuild, and a version bump for one agent would invalidate the others' baselines.Two details in the new Dockerfile are non-obvious and are commented in place:
openclaw, notoc. Without an explicit symlink,ocresolves to nothing inside the image and every sandboxed run fails looking for the binary.oc plugins list. It's copied into that root and the npm copy removed, so there's one on-disk copy rather than two that can drift.Both steps are asserted at build time, so a silent regression fails the build instead of a run.
Also bumps kubectl and helm in
Dockerfile.gemini-clito match the versions the new image pins, so the two sandbox images agree on the tools they hand the agent.GEMINI_CLI_VERSIONis deliberately untouched: it's the agent under test, not part of the toolbox, and bumping it would invalidate existing baselines.Test plan
Noticed while working here, not fixed in this PR
The
bookworm-backportsapt line inDockerfile.gemini-cliappears to be a no-op. The built image ships git 1:2.39.5-0+deb12u3, which is plain bookworm rather than backports, so the comment's stated reason (reftable support, which landed in git 2.45) doesn't hold today. Left alone since it's pre-existing and orthogonal to this change.