fix(optimize-scale): make the load spike actually inject (timeout + kind) - #21
Merged
pradeepvrd merged 2 commits intoSep 9, 2026
Merged
Conversation
Every chaos command shared one 40s wall-clock ceiling, so any spike declared for longer than that was SIGKILLed mid-flight. optimize-scale declares 300s deliberately -- the comment in its chaos_spec says the spike must still be running when the agent finishes and verification starts -- so fortio died at 40s with exit -1 and the fault recorded "load did not reach the workload". That reads as an unreachable target, which is why the failure has been attributed to fixtures, firewalls and providers rather than to a timeout. Derive the ceiling for a load spike from the -t duration in its own argv, plus 60s slack, bounded at 900s. Non-load commands keep the flat 40s. An unparsable or absent duration also keeps the flat 40s rather than inventing a budget from a value it misread. Measured on kind: fortio connected and served traffic for the full spike, where previously it was killed at 40s.
…e workload Reverses 923254a on the strength of what the runs since have measured. That commit kept the task on gcp for the load path: a LoadBalancer the chaos generator reaches directly, versus a port-forward on kind. It runs the other way here. On GKE fortio never connects — the LB external IP times out from the runner (dial tcp <ip>:8080: i/o timeout) because the project firewall admits only 22/3389/443 — so the spike cannot inject at all. On kind the port-forward connects and serves traffic. The other argument for gcp was metrics, and that is already retired: the stack installs metrics-server under infra_provider=kind, so the HPA objective is decided by the agent rather than by the provider. The 8-of-8 historic injection failures were neither: they were the flat 40s chaos command ceiling killing this task's declared 300s spike, fixed in the previous commit. Both changes are needed — reachability alone still gets the generator killed, and a longer ceiling alone still cannot reach a GKE LB. Only the infrastructure provider moves. prompt, expected_output and verification_spec are untouched, so the task grades what it always did, and INFRA_PROVIDER=gcp still selects GKE.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the planned load spike on
optimize-scaleactually inject. It never has — not in the 8 published runs, and not in any attempt since. Two independent causes, one commit each; neither fix works alone.1. Every spike longer than 40s was killed mid-run
generate_load.pycapped every chaos command at a flat_COMMAND_TIMEOUT = 40. This task'schaos_specdeclaresduration: "300s"on purpose — its own comment says the spike must still be running when the agent finishes and verification starts. So fortio was SIGKILLed at 40s, exited-1, and the fault recorded:That message reads as unreachable target, which is why this has been attributed to fixtures, firewalls and providers rather than to a timeout. It would have killed the spike on any provider.
A load spike now derives its ceiling from the
-tin its own argv, plus 60s slack, bounded at 900s. Non-load commands keep the flat 40s. An unparsable or absent duration also keeps 40s rather than inventing a budget from a value it misread.Measured on the runner, both directions:
fortio load -t 60s ...rc=-1rc=0,ok=Truesleep 90(not a spike)2. On GKE the load generator cannot reach the workload at all
923254akept the task on gcp for the load path: a LoadBalancer the generator reaches directly, versus a port-forward on kind. The reasoning was sound; the measurement went the other way.On GKE fortio never completes a TCP connection:
The project's firewall rules admit only 22, 3389 and 443, so the LB's external IP is unreachable from the runner. On kind the Service is a
ClusterIPbehind the harness port-forward, and fortio connects and serves the spike — confirmed in a real run before this branch existed.The other historic argument for gcp, metrics, is already retired: the stack installs metrics-server under
infra_provider=kind, so the HPA objective (ScalingActive=True) is decided by the agent rather than the provider.Scope
Only the infrastructure provider moves.
prompt,expected_outputandverification_specare untouched, so the task grades exactly what it graded before, andINFRA_PROVIDER=gcpstill selects GKE — the better load path the day a runner can reach a LoadBalancer.Testing
300s,5m,1h30m,250ms, junk).ruff checkandruff format --checkclean.run_chaos_commandpath.Not fixed here, and still open
A full
optimize-scalerun has not yet producedverification_status: evaluated. After both fixes the next run failed on a third, unrelated cause —429 RESOURCE_EXHAUSTEDon the chaos model endpoint, so no fortio command was issued at all. That is the documented retryable flake, not a defect in this branch, but it means the end-to-end green run is still owed.Stacked on
feat/sandbox-all-harnesses(#14), which carries the metrics-server install and the vcluster provider-cycle fix this task also needs.