Conversation
Only gemini_cli was migrated, so a sandboxed matrix could not actually run the other three arms: AgentHarness.run refuses a sandboxed harness that has not been migrated, rather than silently running it on the host. The published antigravity and claude_code runs WERE sandboxed, but by an uncommitted patch and an untracked tree on the bastion -- neither in any PR, neither reviewed, and neither present in this lineage. Each harness now routes its agent turn through run_agent_cmd and declares supports_sandbox. Fixture mounts need no per-harness work: the eval harness already builds them centrally into the spec. openclaw needed more than a flag. Its state dir and config path cross the boundary as env VALUES, and the host spelling means nothing inside the container, so the agent turn now gets container-translated paths while the post-run 'oc sessions' / export-trajectory calls keep the host spelling. Both read the same bytes: the state dir lives under the workspace, which is the bind mount itself. The container also has its own oc on PATH, so the host binary path is not passed through. container_path() is lifted out of SandboxExecutor.map_host_path so the cwd mapping and these value translations cannot drift apart. secret-rotation opts out via a new Task.requires_unsandboxed. The task IS credential work -- it rotates a Secret Manager version through ADC, and ADC is exactly what the boundary strips, so sandboxing it would not harden it, it would make it impossible. The harness skips the sandbox for that task only and logs a warning, so a sandboxed matrix shows which task did not get a boundary. The flag clears config.sandbox rather than just skipping spec completion: the agent's own gate reads that field, and a leftover skeletal spec would refuse the run.
… all Provider deduction keys off the stack DIRECTORY NAME and only exactly 'kind'/'vcluster' deduce. The directory is 'cp-recovery-kind', which does not, so the run died at provisioning with 'stack requires an explicit provider' before the agent ever started. Verified: _select_provider raises for this task and for no other -- every b-00xx task already declares provider: kind, and this was the only ported kind task that did not. Infrastructure selection, not grading: nothing about the prompt, the verification spec or the expected output changes, so the task remains comparable with the rows already scored against it.
…cs-server on kind The stack configured provider "kubernetes" from module.cluster.endpoint, whose value falls back to the vcluster submodule. That submodule's own resources are served by the same provider, so tofu refused the graph with "Cycle: module.cluster.module.vcluster..." before planning anything -- for every infra_provider, which left the task unable to provision at all. Add a vcluster-free managed_endpoint output and read that instead: the ordering edge on the cluster survives, the cycle does not. Also install metrics-server when infra_provider=kind. The HPA objective grades ScalingActive=True, which needs a live metrics pipeline that a stock kind cluster lacks; without it the objective fails for provider reasons rather than for anything the agent did.
The task was pinned to gcp for one reason: the HPA objective grades ScalingActive=True and a stock kind cluster ships no metrics pipeline, so the objective failed for provider reasons rather than for anything the agent did. The stack now installs metrics-server under infra_provider=kind, so that reason is gone and the objective is decided by the agent again. Verified on kind end to end: the cluster provisions, the seeded workload rolls out, and kubectl top pods returns live CPU for the target deployment. Only the infrastructure provider changes. The prompt, expected_output and verification_spec are untouched, so the task grades exactly what it did before. INFRA_PROVIDER=gcp still selects GKE.
It was gcp before: the k-sigs migration (kubernetes-sigs#143) declared provider: gcp explicitly, and that is the definition every published run of this task was graded against. Switching the provider changes what the task measures, so it should not move as a side effect of making kind viable. The metrics-server install and the vcluster cycle fix stay. They were the real bugs: the cycle blocked provisioning on EVERY provider, and without metrics-server the HPA objective failed on kind for provider reasons rather than agent ones. kind is now a genuinely working alternative for local iteration. What still argues for gcp is the load path, not metrics: on GKE the Service is a LoadBalancer the chaos generator reaches directly, while on kind it is a ClusterIP behind the harness port-forward. The planned spike failed to inject in 8 of 8 published runs, so the fewer moving parts in that path the better until a spike demonstrably lands. Either way every runner must use the same provider for this task -- the Service type and load path differ, so a kind arm and a GKE arm are not comparable.
Same defect optimize-scale had: the kubernetes and helm providers were configured from module.cluster.endpoint, whose value falls back to the vcluster submodule, and that submodule's resources are served by those same providers. Tofu rejected the graph with "Cycle: ...module.vcluster..." before planning anything, so the task could not provision on any provider -- it failed on both arms in the 2026-09-08 rerun campaign. Read managed_endpoint instead, forwarded through this stack's local cluster wrapper. tofu plan now succeeds: 18 to add, no cycles.
b-0059 and b-0061 grade NetworkPolicy enforcement, and kindnet does not enforce policy, so both stacks pass disable_default_cni = true to the cluster module. The module never declared that variable: tofu failed at init with "An argument named disable_default_cni is not expected here", so neither task has ever provisioned. Their setup scripts already document the intent -- Calico is expected to be installed by the cluster module before they run. Implement it: swap kindnet for Calico v3.27.3 behind the variable. Two details that are not obvious. Nodes stay NotReady until a CNI is running, so kind_cluster.wait_for_ready has to be off in that case or the apply blocks until timeout on exactly the clusters that need Calico. null_resource.calico does the waiting instead, and duplicate_context now waits on it too. pod_subnet is set to Calico's own default pool (192.168.0.0/16) rather than kind's 10.244.0.0/16; left mismatched, Calico allocates outside the subnet the node spec and kube-proxy expect. The variable defaults to false and the networking block is emitted only when it is true, so every existing kind task plans exactly as before (verified on b-0024: no Calico resource, wait_for_ready still true, same 3 resources). Verified end to end on a real kind cluster: node Ready, calico-node 1/1, podCIDR 192.168.0.0/24, and a default-deny ingress policy actually enforced -- curl returns 200 before the policy and fails after it. Under kindnet the same probe would keep returning 200.
_build_agent_config_snapshot rebuilds AgentConfig field by field, and extra_flags was not among the fields it copied. AGENT_EXTRA_FLAGS parsed correctly into AgentConfig.from_env and was then dropped on the floor before the agent ever saw it, with nothing logged. The visible symptom is agy keeping its 5m default --print-timeout on every benchmark run: a task whose agent turn runs longer dies with "timeout waiting for response", an errored run with a truncated trajectory. b-0060 hit it twice at 313s and 308s while every run that finished did so under 250s. The documented mitigation for that flake is to pass a larger --print-timeout, which was impossible through the harness. Verified: the snapshot now carries (\x27--print-timeout=24h\x27) from the env, and the 249 agent/evalharness unit tests still pass.
The judge and the chaos driver resolve models differently: the judge reaches Vertex, where gemini-3.1-pro exists, while the chaos driver goes through the API key, whose endpoint only publishes gemini-3.1-pro-preview and 404s the plain id. The existing note covers the judge, so a working judge reads as evidence the whole config is fine. It cost a run to find. optimize-scale came back chaos_invalidated with the spike never injected, which looks like a fault-injection problem rather than the driver being unable to call a model at all.
docker run --user rejects ids above int32 max:
docker: Error response from daemon: uids and gids must be
in range 0-2147483647
wrap_argv passed os.getuid()/os.getgid() unconditionally on Linux.
Identities minted by an external identity provider (Cloud Identity,
Workspace external users, most LDAP setups) routinely exceed that
limit, and on such a host the sandbox cannot start at all: docker
emits create, attach and destroy with no start event, exits 125, and
the agent never runs.
Dropping --user would make it run, but by giving the untrusted agent
root inside the boundary this module exists to enforce. That trades a
loud failure for a silent containment downgrade, which is out of
keeping with the rest of the module: it already refuses to fall back
to an unsandboxed run rather than quietly weaken the boundary. It was
also tried first, and it left files in the operator's home that
teardown could not remove without root.
So remap rather than degrade. When either id is out of range the
container runs as 1000:1000, the unprivileged node user in the
node:22-slim base these images build on, and the agent stays
unprivileged. A throwaway root container from the same image chowns
the workspace and every fixture mount to that id beforehand, and
chowns them back to the real host ids afterwards in a finally, so the
operator can still read and delete the artifacts without root.
Fixtures are covered too, not just the workspace: they are mounted
read-write by design and live outside the workspace, so chowning the
workspace alone would strand them.
The pre-run chown is fatal on failure, since an unprivileged agent
handed a workspace it cannot write would produce a misleading result
rather than an honest refusal. The handback is best-effort and never
masks a real agent result, but logs the exact repair command when it
fails.
The whole path is inert when the caller's ids are already in range,
including a caller running as root, which takes the existing path
unchanged.
pradeepvrd
force-pushed
the
feat/sandbox-all-harnesses
branch
from
September 9, 2026 23:27
afd98dd to
d2fc756
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
docker run --userrejects uids/gids above the int32 max (2147483647):The sandbox unconditionally passed
os.getuid()/os.getgid()on Linux. Identities minted by an external identity provider (Cloud Identity, Workspace external users, most LDAP setups) routinely exceed that limit (3998470835 is an example of an out-of-range id). On such a host the sandbox cannot start at all: docker emits create, attach and destroy with no start event, exits 125, and the agent never runs.Why not just drop
--userDropping
--usermakes the container start, but it hands the untrusted agent root inside the boundary this module exists to enforce. That trades a loud failure for a silent containment downgrade, which doesn't fit the rest of the module: it already refuses to fall back to an unsandboxed run rather than quietly weaken the boundary. This was tried first, and it also left files in the operator's home that teardown could not remove without root.Fix
Remap rather than degrade. When either id is out of range, the container runs as
1000:1000, the unprivilegednodeuser in thenode:22-slimbase these images build on, so the agent stays unprivileged.1000:1000before the run.finally, another throwaway root container chowns everything back to the real host ids so the operator can still read and delete the artifacts without root.Costs
finallyfires, and ownership is left at the remap id rather than handed back.Test plan
uv run pytest tests/unit/agents/test_sandbox.py -q— 76 passed