Repository navigation
feat(sandbox): route the Vertex metadata emulator work into integration - #31
Merged
Merged
Conversation
…erver A sandboxed agent has no ambient cloud identity by construction: the operator's gcloud config is not mounted, CLOUDSDK_CONFIG and GOOGLE_APPLICATION_CREDENTIALS never cross the boundary, and on a cloud VM the link-local metadata endpoint is blocked to containers by the DOCKER-USER rule vm-setup.sh installs. That is the point of the boundary, and it is also why every Application Default Credentials lookup inside the container fails — so today a sandboxed Vertex run cannot authenticate at all. Give the keyless backends a place to fix that: sandbox_credential_env(), keyed off ProviderSpec.backend, so a new backend is one branch here rather than a change at every harness call site. Key-based providers get an empty mapping (their key already crosses by value); a keyless backend with no recipe yet — Bedrock — is refused with an error saying so, rather than running unauthenticated. Vertex's recipe is a metadata-server emulator: a host-side HTTP server speaking the subset of the GCE metadata protocol Google's auth libraries use to obtain a token, backed by an impersonated token for a service account holding roles/aiplatform.user and nothing else. The container reaches it at host.docker.internal, which the executor already --add-host's to the host gateway on every run. Four simpler things were tried first and none of them work: there is no supported way to hand the SDK a bare access token; no ADC file format carries one (authorized_user needs a refresh token, external_account needs a workload-identity pool); GOOGLE_API_KEY + GOOGLE_GENAI_USE_VERTEXAI is Vertex express mode only and a standard project answers "401: API keys are not supported by this API"; and service-account key files are refused by constraints/iam.disableServiceAccountKeyCreation. Containment properties worth stating, since this hands the untrusted side a credential on purpose: * Only the token-related paths are served. Everything else is a 404, so this is an emulator of three endpoints and cannot be used as a proxy onto the host's real metadata server (instance attributes, kube-env, ssh-keys). * Requests must carry Metadata-Flavor: Google, as the real server requires. * The token is minted lazily, so a run that never calls the model never mints a credential, and refilled inside a five-minute margin so a caller is never handed one that expires mid-generation. * The server binds on all interfaces rather than loopback because it has to: on Linux the container reaches the host at the bridge gateway address and a loopback-bound server does not answer there. Exposure is bounded by the host firewall, an ephemeral port, the header check, and by what the token is worth — one project's aiplatform.user, for an hour. Shelling out to gcloud rather than importing google.auth keeps this module's "pulls no provider SDK" property intact and matches how the rest of the benchmark reaches GCP. Per the proposal's review rule the cloud-specific code lives here, in the model-credential layer, and never in agents/sandbox.py.
Call the backend's mint-and-inject recipe from the vertex branch of _build_env, so the container gets GCE_METADATA_HOST / GCE_METADATA_IP / METADATA_SERVER_DETECTION alongside the google-genai routing vars it already gets. Vertex is the matrix's default provider, so this is what makes a sandboxed matrix run possible at all; until now sandboxing only worked with AGENT_PROVIDER=google plus a Gemini API key. Gated on config.sandbox, not on the backend alone. An unsandboxed run is a host process with the operator's own ADC — that is the right credential for it, the recipe is not consulted, and the flag-off path stays byte-for-byte what it was. Two tests pin that: one fails the test if the recipe is called on an unsandboxed run, one if it is called for a key-based provider. The docs section that said a sandboxed Vertex run "cannot authenticate and will fail loudly" is now the general keyless-backend story plus Vertex's recipe, including the two things an operator has to set up first (BENCH_VERTEX_SANDBOX_SA and serviceAccountTokenCreator for the host identity on it). Bedrock still fails loud and the docs say so.
The previous wording called it "an emulator of three endpoints". It serves nine: the residency ping, project/project-id, universe/universe-domain, the service-account listing, and email / scopes / aliases / token / the recursive listing beneath each accepted account segment. Undercounting is the harmless half. The sentence also left the security argument implicit, when the interesting property is what is *absent* -- instance/attributes/ (kube-env on a real GKE node), the instance identity paths, any account other than the impersonated one, and the identity?audience= OIDC-JWT issuer. Those omissions are why a container that probes the emulator learns nothing about its host, which is the claim a reviewer actually needs to check. Name them.
feat(sandbox): give a sandboxed Vertex run its own metadata-server credential
Brings the branch up to date with the trunk so the metadata-emulator work merged as #25 has a clean route back into integration. #25 was opened against this branch about three hours after #15 had already merged it, so GitHub never retargeted it and the credential recipe has been sitting here with no onward PR. One conflict, in the gemini-cli harness, between two additive changes: #24 replaced the inline location fallback with vertex_location(), while #25 added the sandboxed credential recipe to the same block. Kept both. vertex_location() wins for the location specifically because it drops the GCP_LOCATION fallback this branch still had -- that variable is the deployers' cluster zone, not a Vertex region, which is the bug #24 fixed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this exists
The metadata-emulator work merged as #25 never reached
integration, and this PR is the missing link rather than new work.The sequence:
sandbox/02-credentials→integrationsandbox/02-credentials— ~3h after that base mergedsandbox/02-credentials#25 was opened against a branch that had already landed, so GitHub never auto-retargeted it (that only happens for PRs open at the moment the base merges). Merging #25 did exactly what it declared — it just delivered into a branch with no onward route.
sandbox_credential_envis currently absent fromintegrationas a result.What's in it
The five commits
integrationis missing:43b4d4fmint a sandboxed Vertex run its own metadata servera27cdb8give a sandboxed Vertex run the emulator credential (gemini-cli)78f5984describe the emulator's real route table (docs)3a82745,1506b73)Conflict resolution
One conflict, in
devops_bench/agents/cli/gemini_cli/agent.py, between two additive changes to the same block: #24 replaced the inline location fallback withvertex_location(), and #25 added the sandboxed credential recipe. Both kept.vertex_location()wins for the location specifically because it drops theGCP_LOCATIONfallback this branch still carried — that variable is the deployers' cluster zone, not a Vertex region, which is the bug #24 fixed. Taking this branch's side would have silently reintroduced it.docs/components/agents.mdandtests/unit/agents/test_agents_cli_gemini.pyauto-merged.Verification
uv run pytest tests/unit— 1817 passeduv run ruff check/ruff format --check— cleanNote for the stack
#29 is still open against
feat/sandbox-all-harnesses, which has now merged; it likely needs retargeting tointegration. Deliberately untouched here.🤖 Generated with Claude Code