Repository navigation
fix(agents): send agy a model id it accepts - #28
Merged
isadominguez314 merged 1 commit intoSep 11, 2026
Merged
isadominguez314 merged 1 commit into
isadominguez314 merged 1 commit into
Conversation
Every agy run has been dying at startup. The matrix sets one AGENT_MODEL for the whole run -- gemini-3.1-pro-preview, spelled for Vertex, which needs that suffix -- and agy rejects the -preview suffix outright and refuses any selection that does not name a reasoning tier exactly once. _resolve_model_name only stripped the provider prefix, so the Vertex id reached the command line verbatim and agy exited 1 in about two seconds with no conversation DB and no transcript. Not a sandbox problem, though that is where it surfaced: an ambient run fails identically, and has since the harness was written. Normalize inside this harness rather than respelling AGENT_MODEL for the agy arm. That value labels every arm in the matrix and feeds the judge, so bending it to suit one CLI would desynchronize that arm from the runs it is compared against, and Vertex needs exactly the suffix agy refuses. Emit the bare slug plus --effort: it is the one form that cannot collide, and no lookup table means nothing to maintain as the catalogue grows -- agy validates the result and still fails loud on anything unknown. The tier is a scoring variable rather than a formatting detail, since low and high are materially different agents and agy has no untiered form. Default it to high, because the other harnesses run their model with no reasoning throttle and anything lower would hand the agy arm a handicap that reads as a capability gap. AGENT_MODEL_EFFORT overrides it, and an unknown tier is rejected on the spot rather than passed through to fail later as agy's own error mid-arm. settings.json now carries the same resolved spelling as the flag. agy does not validate modelConfigs.defaultModel -- an unknown value there is ignored in silence and the run quietly falls back to another model -- so the flag is the only guard, and it should not be guarding a value that settings contradicts. Drop GEMINI_MODEL from the env overlay. It is a Gemini CLI variable that agy ignores: a garbage value raises no error and a valid one does not change the model the run reports. It looked like a lever and was not. Spellings verified against agy 1.2.0 on the bastion, in the pinned image and under the executor's real invocation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every
agyrun dies at startup withinvalid model selection. This is not sandbox-specific — an ambient, un-sandboxed run fails identically, and has since the harness was written. It stayed latent because agy had no exercised arm to fail in.Independent of #14 and of #27 (the image). Based on
integration.The incident
PR #14 validation run
pr14-antigravityon the bastion, 2026-09-10 ~16:19Z, agy 1.2.0, sandboxed GKE run:Exit 1 in roughly two seconds, no conversation DB, no transcript. That arm was then validated under the display-name spelling
AGENT_MODEL="Gemini 3.1 Pro (Low)"as a workaround, which is why its model label does not match the rest of the matrix — and why that run is evidence for the sandbox seam, not a comparable agent-quality result.Root cause
AGENT_MODELis a single global value shared by every arm and by the judge (scripts/bastion/vm-setup.sh:266=gemini-3.1-pro-preview), spelled for Vertex, which requires the-previewsuffix._resolve_model_nameonly stripped a provider prefix, so the Vertex id reached agy's command line verbatim.What agy 1.2.0 actually accepts
No
-previewsuffix, and a reasoning tier named exactly once — as--effort, a slug suffix, or a display-name parenthetical.gemini-3.1-pro-preview(what the matrix sends)gemini-3.1-pro-preview --effort=low--effort is not supportedgemini-3.1-prorequires --effort (available: low, high)gemini-3.1-pro --effort=lowgemini-3.8-flash --effort=mediumgemini-3.1-pro-lowGemini 3.1 Pro (Low)Gemini 3.1 Pro (Low) --effort=lowThe catalogue is 11 selections:
Gemini 3.8/3.7/3.6 Flash (High|Medium|Low)andGemini 3.1 Pro (High|Low). Everything is Flash except the two Pro entries. Anygemini-2.5-*id is rejected outright.The fix
Normalise inside the antigravity harness. Respelling
AGENT_MODELfor the agy arm instead would desynchronise that arm's label from every other arm in the same matrix, and Vertex needs exactly the suffix agy rejects — so the quirk belongs in the one harness that has it.google/gemini-3.1-pro-preview→--model gemini-3.1-pro --effort high, logged when the rewrite happens.Emitting the bare slug plus
--effort, rather than reassemblinggemini-3.1-pro-low, is the one form that cannot collide. No lookup table, so nothing to maintain as the catalogue grows — agy validates the result and still fails loud on anything it does not know.A tier already spelled into the id is honoured and nothing is added, so the display-name workaround above keeps working.
The tier is a scoring variable
agy has no untiered form, so the harness must supply one when
AGENT_MODELcarries none — which is always, today.lowversushighis a materially different agent, so choosing quietly would make the agy arm a different experiment than the arm it is compared against.Default
high, since the other harnesses run their model with no reasoning throttle and anything lower would hand agy a handicap that reads as a capability gap. Overridable viaAGENT_MODEL_EFFORT, with an unknown tier rejected on the spot rather than passed through to fail later as agy's own error mid-arm — the same shape asbae4568, which rejects an unknownagent_pod_securitylevel at load rather than at use.settings.json now agrees with the flag
modelConfigs.defaultModelcarries the same resolved spelling as--model.Worth stating plainly, because it is a trap: agy does not validate
defaultModel. An unknown value there is ignored in silence and the run falls back to another model — verified, a garbage value runs happily and self-reports as Gemini 3.8 Flash. The--modelflag is the only validation there is, so it should not be guarding a value that settings quietly contradicts, and it should not be dropped as redundant later.GEMINI_MODEL removed
_build_envset it; agy ignores it entirely. Verified: a garbage value raises no error, and a validgemini-3.1-pro-lowdoes not change the model the run reports — identical to the no-env control. Copy-paste from the gemini_cli harness. It looked like a lever and was not.Verification
1744 unit tests pass. New coverage over the resolver table above, including the collision case, the display-name pass-through, and load-time rejection of an unknown
AGENT_MODEL_EFFORT.Every spelling in the table was exercised against agy 1.2.0 on the bastion, in the pinned image from #27, under the executor's real invocation (
--user <uid>:<gid>,HOMEon the workspace mount, metadata endpoint unreachable).Not in this PR
The
_resolve_binaryhost-path leak into container argv — that belongs with the boundary that creates it, and is held in #14.