Repository navigation
feat(sandbox): give a sandboxed Vertex run its own metadata-server credential - #25
Merged
isadominguez314 merged 3 commits intoSep 11, 2026
Conversation
…erver A sandboxed agent has no ambient cloud identity by construction: the operator's gcloud config is not mounted, CLOUDSDK_CONFIG and GOOGLE_APPLICATION_CREDENTIALS never cross the boundary, and on a cloud VM the link-local metadata endpoint is blocked to containers by the DOCKER-USER rule vm-setup.sh installs. That is the point of the boundary, and it is also why every Application Default Credentials lookup inside the container fails — so today a sandboxed Vertex run cannot authenticate at all. Give the keyless backends a place to fix that: sandbox_credential_env(), keyed off ProviderSpec.backend, so a new backend is one branch here rather than a change at every harness call site. Key-based providers get an empty mapping (their key already crosses by value); a keyless backend with no recipe yet — Bedrock — is refused with an error saying so, rather than running unauthenticated. Vertex's recipe is a metadata-server emulator: a host-side HTTP server speaking the subset of the GCE metadata protocol Google's auth libraries use to obtain a token, backed by an impersonated token for a service account holding roles/aiplatform.user and nothing else. The container reaches it at host.docker.internal, which the executor already --add-host's to the host gateway on every run. Four simpler things were tried first and none of them work: there is no supported way to hand the SDK a bare access token; no ADC file format carries one (authorized_user needs a refresh token, external_account needs a workload-identity pool); GOOGLE_API_KEY + GOOGLE_GENAI_USE_VERTEXAI is Vertex express mode only and a standard project answers "401: API keys are not supported by this API"; and service-account key files are refused by constraints/iam.disableServiceAccountKeyCreation. Containment properties worth stating, since this hands the untrusted side a credential on purpose: * Only the token-related paths are served. Everything else is a 404, so this is an emulator of three endpoints and cannot be used as a proxy onto the host's real metadata server (instance attributes, kube-env, ssh-keys). * Requests must carry Metadata-Flavor: Google, as the real server requires. * The token is minted lazily, so a run that never calls the model never mints a credential, and refilled inside a five-minute margin so a caller is never handed one that expires mid-generation. * The server binds on all interfaces rather than loopback because it has to: on Linux the container reaches the host at the bridge gateway address and a loopback-bound server does not answer there. Exposure is bounded by the host firewall, an ephemeral port, the header check, and by what the token is worth — one project's aiplatform.user, for an hour. Shelling out to gcloud rather than importing google.auth keeps this module's "pulls no provider SDK" property intact and matches how the rest of the benchmark reaches GCP. Per the proposal's review rule the cloud-specific code lives here, in the model-credential layer, and never in agents/sandbox.py.
Call the backend's mint-and-inject recipe from the vertex branch of _build_env, so the container gets GCE_METADATA_HOST / GCE_METADATA_IP / METADATA_SERVER_DETECTION alongside the google-genai routing vars it already gets. Vertex is the matrix's default provider, so this is what makes a sandboxed matrix run possible at all; until now sandboxing only worked with AGENT_PROVIDER=google plus a Gemini API key. Gated on config.sandbox, not on the backend alone. An unsandboxed run is a host process with the operator's own ADC — that is the right credential for it, the recipe is not consulted, and the flag-off path stays byte-for-byte what it was. Two tests pin that: one fails the test if the recipe is called on an unsandboxed run, one if it is called for a key-based provider. The docs section that said a sandboxed Vertex run "cannot authenticate and will fail loudly" is now the general keyless-backend story plus Vertex's recipe, including the two things an operator has to set up first (BENCH_VERTEX_SANDBOX_SA and serviceAccountTokenCreator for the host identity on it). Bedrock still fails loud and the docs say so.
The previous wording called it "an emulator of three endpoints". It serves nine: the residency ping, project/project-id, universe/universe-domain, the service-account listing, and email / scopes / aliases / token / the recursive listing beneath each accepted account segment. Undercounting is the harmless half. The sentence also left the security argument implicit, when the interesting property is what is *absent* -- instance/attributes/ (kube-env on a real GKE node), the instance identity paths, any account other than the impersonated one, and the identity?audience= OIDC-JWT issuer. Those omissions are why a container that probes the emulator learns nothing about its host, which is the claim a reviewer actually needs to check. Name them.
pradeepvrd
reviewed
Sep 10, 2026
| @@ -48,7 +48,7 @@ | |||
| materialize_skills, | |||
Owner
There was a problem hiding this comment.
Can we also add support for Antigravity. That is the main one we are trying to run right now (gemini CLI is pretty much fully unsupported)
| # Keyless-backend credential recipes for sandboxed runs | ||
| # -------------------------------------------------------------------------- | ||
| # | ||
| # A sandboxed agent has no ambient cloud identity by construction: the |
Owner
There was a problem hiding this comment.
fine for now. But this will be heavily scrutinized for vendor specific comments when sending a PR to k8s-sigs (JFYI)
| remaining = _TOKEN_LIFETIME_SEC | ||
| return self._token, int(remaining) | ||
|
|
||
| def _mint(self) -> str: |
Owner
There was a problem hiding this comment.
I guess if it mints a fresh access token, we need to ensure the service account being impersonated has the permissions we need for the task at hand - correct? How is that done? I can't quite tell from the PR.
isadominguez314
added a commit
that referenced
this pull request
Sep 11, 2026
Brings the branch up to date with the trunk so the metadata-emulator work merged as #25 has a clean route back into integration. #25 was opened against this branch about three hours after #15 had already merged it, so GitHub never retargeted it and the credential recipe has been sitting here with no onward PR. One conflict, in the gemini-cli harness, between two additive changes: #24 replaced the inline location fallback with vertex_location(), while #25 added the sandboxed credential recipe to the same block. Kept both. vertex_location() wins for the location specifically because it drops the GCP_LOCATION fallback this branch still had -- that variable is the deployers' cluster zone, not a Vertex region, which is the bug #24 fixed.
This was referenced Sep 11, 2026
isadominguez314
added a commit
that referenced
this pull request
Sep 11, 2026
The first live sandboxed google-vertex run against the #25 emulator failed twice before it could reach the model, both times on pre-existing openclaw gaps rather than on the credential recipe: ProviderAuthError: No API key found for provider "google-vertex" Vertex AI requires a location. Set GOOGLE_CLOUD_LOCATION or pass location in options. The auth-profile gate is not plugin-specific. oc 2026.9.1-beta.1 consults its per-agent SQLite auth store for *every* provider before any ADC resolution happens, so a keyless google-vertex run aborts exactly as anthropic-vertex does. Generalize the workaround from the plugin to the whole vertex backend: _needs_anthropic_vertex_auth_profile becomes _vertex_auth_profile_provider, returning the oc provider id to seed. The pasted value is a marker, not a credential -- the transport still authenticates through ADC, and a real but wrong key would 401. The location comes from vertex_location() rather than the forwarding block, which only covered a host that had already set one of the two spellings. It applies to keyed runs too: a location is routing, not a credential. Assigning unconditionally does not clobber a forwarded value, since the chain reads GOOGLE_CLOUD_LOCATION first and resolves it back to itself. The auth profile is seeded on the unsandboxed path as well, which writes the marker to the operator's global oc store rather than the per-run state dir. That path fails the same way today, and scripts/bastion/configure-oc.sh already pastes the identical value for the same provider and agent, so on a configured host this rewrites the same row with the same bytes. With both fixes the pinned image returns a completed turn from google-vertex/gemini-3.8-flash through the sandboxed emulator credential.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
sandbox/02-credentials. Scope is deliberately narrow: the Gemini CLI only.The problem
A sandboxed run cannot use Application Default Credentials. The boundary strips
CLOUDSDK_CONFIGandGOOGLE_APPLICATION_CREDENTIALS, and on the bastion the link-local metadata endpoint169.254.169.254is blocked for containers by aDOCKER-USERREJECT rule. ADC is exactly that chain, so a keyless Vertex arm has nothing to authenticate with.A key would be the easy answer, and it isn't available:
constraints/iam.disableServiceAccountKeyCreationis enforced on this org, so there are no SA key files to mount.The approach
Each keyless backend gets a mint-and-inject recipe: mint a narrow, short-lived credential host-side, inject it explicitly. Recipes live in
core/model_providers.py(sandbox_credential_env), keyed off the provider's backend — never in the sandbox module, so cloud-specific code stays out of the boundary code.Vertex's recipe is a small HTTP server on the host speaking the subset of the GCE metadata protocol Google's auth libraries use. The container is pointed at it with
GCE_METADATA_HOST,GCE_METADATA_IPandMETADATA_SERVER_DETECTION=assume-present, reaching it athost.docker.internal(the executor already--add-hosts that to the host gateway on every run).METADATA_SERVER_DETECTION=assume-presentis load-bearing: without it the libraries run a GCE-residency sniff that a container fails.The token is minted by impersonating a service account holding
roles/aiplatform.userand nothing else — no key file, no ambient credential handed over. It is refilled in place behind a lock, and never minted at all if the run does not call the model.What the emulator does not serve
The route table is fixed and closed. 404 for everything outside it, including:
instance/attributes/—kube-envon a real GKE nodeid,zone,hostname,disks,network-interfaces)identity?audience=OIDC-JWT issuerA container that probes it learns nothing about the host it is running on.
Metadata-Flavor: Googleis required, as the real server requires.Live validation (bastion, Linux, docker 29.1.3)
Preconditions confirmed first:
DOCKER-USERREJECT to169.254.169.254present at rule 3, and the SA'saiplatform.userbinding re-verified immediately before the run (the IAM automation strips project-level bindings).http=200,token_len=1024,expires_in=3600,Bearer000— blockedhost.docker.internal, correct SA email200with a real completion'41'The last row is the one that matters: it exercises the Node
gcp-metadata/google-auth-librarypath rather than a hand-rolled request.It also vindicated binding the emulator to
0.0.0.0rather than loopback — the container reaches the host across the bridge gateway, which loopback would not serve on Linux (and would have looked fine on macOS Docker Desktop).Negative control: emulator suppressed → the run fails.
Known gaps
(sa, project)mint cache is thread-safe by construction but has not been hit by parallel runs, which is how the matrix executes.Related
GOOGLE_CLOUD_LOCATION) for a sandboxed run to reach a working endpoint — the two are orthogonal but both required. This PR's validation set the location explicitly.supports_sandbox = True, so shipping it now would mean shipping dead, unvalidated code. Held on branchsandbox/025-claude-code-wiringand re-lands as a small follow-up after feat(sandbox): wire antigravity, claude_code and openclaw onto the sandbox seam #14, validated against the throwaway image.anthropic-bedrockrun is refused with an error saying so. Ambient unsandboxed runs of either are unaffected.