Skip to content

fix(optimize-scale): make the load spike actually inject (timeout + kind) - #21

Merged
pradeepvrd merged 2 commits into
feat/sandbox-all-harnessesfrom
fix/optimize-scale-kind-spike
Sep 9, 2026
Merged

fix(optimize-scale): make the load spike actually inject (timeout + kind)#21
pradeepvrd merged 2 commits into
feat/sandbox-all-harnessesfrom
fix/optimize-scale-kind-spike

Conversation

@pradeepvrd

Copy link
Copy Markdown
Owner

Makes the planned load spike on optimize-scale actually inject. It never has — not in the 8 published runs, and not in any attempt since. Two independent causes, one commit each; neither fix works alone.

1. Every spike longer than 40s was killed mid-run

generate_load.py capped every chaos command at a flat _COMMAND_TIMEOUT = 40. This task's chaos_spec declares duration: "300s" on purpose — its own comment says the spike must still be running when the agent finishes and verification starts. So fortio was SIGKILLed at 40s, exited -1, and the fault recorded:

load did not reach the workload: fortio load exited with code 1

That message reads as unreachable target, which is why this has been attributed to fixtures, firewalls and providers rather than to a timeout. It would have killed the spike on any provider.

A load spike now derives its ceiling from the -t in its own argv, plus 60s slack, bounded at 900s. Non-load commands keep the flat 40s. An unparsable or absent duration also keeps 40s rather than inventing a budget from a value it misread.

Measured on the runner, both directions:

command before after
fortio load -t 60s ... killed at 40s, rc=-1 ran 60.1s, rc=0, ok=True
sleep 90 (not a spike) killed at 40s killed at 40.0s — unchanged

2. On GKE the load generator cannot reach the workload at all

923254a kept the task on gcp for the load path: a LoadBalancer the generator reaches directly, versus a port-forward on kind. The reasoning was sound; the measurement went the other way.

On GKE fortio never completes a TCP connection:

dial tcp 136.64.31.76:8080: i/o timeout   (~3s into warmup, generator aborts)

The project's firewall rules admit only 22, 3389 and 443, so the LB's external IP is unreachable from the runner. On kind the Service is a ClusterIP behind the harness port-forward, and fortio connects and serves the spike — confirmed in a real run before this branch existed.

The other historic argument for gcp, metrics, is already retired: the stack installs metrics-server under infra_provider=kind, so the HPA objective (ScalingActive=True) is decided by the agent rather than the provider.

Scope

Only the infrastructure provider moves. prompt, expected_output and verification_spec are untouched, so the task grades exactly what it graded before, and INFRA_PROVIDER=gcp still selects GKE — the better load path the day a runner can reach a LoadBalancer.

Testing

  • 64 chaos unit tests pass, including 6 new ones covering the derived ceiling, the bound, the non-load path, and Go-duration parsing (300s, 5m, 1h30m, 250ms, junk).
  • ruff check and ruff format --check clean.
  • The two measurements in the table above, run against the real run_chaos_command path.

Not fixed here, and still open

A full optimize-scale run has not yet produced verification_status: evaluated. After both fixes the next run failed on a third, unrelated cause — 429 RESOURCE_EXHAUSTED on the chaos model endpoint, so no fortio command was issued at all. That is the documented retryable flake, not a defect in this branch, but it means the end-to-end green run is still owed.

Stacked on feat/sandbox-all-harnesses (#14), which carries the metrics-server install and the vcluster provider-cycle fix this task also needs.

Every chaos command shared one 40s wall-clock ceiling, so any spike declared
for longer than that was SIGKILLed mid-flight. optimize-scale declares 300s
deliberately -- the comment in its chaos_spec says the spike must still be
running when the agent finishes and verification starts -- so fortio died at
40s with exit -1 and the fault recorded "load did not reach the workload".
That reads as an unreachable target, which is why the failure has been
attributed to fixtures, firewalls and providers rather than to a timeout.

Derive the ceiling for a load spike from the -t duration in its own argv,
plus 60s slack, bounded at 900s. Non-load commands keep the flat 40s. An
unparsable or absent duration also keeps the flat 40s rather than inventing a
budget from a value it misread.

Measured on kind: fortio connected and served traffic for the full spike, where
previously it was killed at 40s.
…e workload

Reverses 923254a on the strength of what the runs since have measured.

That commit kept the task on gcp for the load path: a LoadBalancer the chaos
generator reaches directly, versus a port-forward on kind. It runs the other way
here. On GKE fortio never connects — the LB external IP times out from the
runner (dial tcp <ip>:8080: i/o timeout) because the project firewall admits
only 22/3389/443 — so the spike cannot inject at all. On kind the port-forward
connects and serves traffic.

The other argument for gcp was metrics, and that is already retired: the stack
installs metrics-server under infra_provider=kind, so the HPA objective is
decided by the agent rather than by the provider.

The 8-of-8 historic injection failures were neither: they were the flat 40s
chaos command ceiling killing this task's declared 300s spike, fixed in the
previous commit. Both changes are needed — reachability alone still gets the
generator killed, and a longer ceiling alone still cannot reach a GKE LB.

Only the infrastructure provider moves. prompt, expected_output and
verification_spec are untouched, so the task grades what it always did, and
INFRA_PROVIDER=gcp still selects GKE.
@pradeepvrd
pradeepvrd merged commit afd98dd into feat/sandbox-all-harnesses Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant