feat(eval): support explicit execution step limits - #536
Conversation
There was a problem hiding this comment.
🟡 Human review recommended
It combines a small eval-host feature with a large, behavior-changing artifact/CLI/dependency subsystem and includes at least one confirmed correctness issue needing adjustment.
Pull request overview
This PR makes the eval runner’s scheduler step budget configurable via a new max_steps option on runner.run(config), enforced in the eval host loop, and adds comprehensive tests and spec documentation. In addition, it introduces an artifact resource validation/materialization subsystem and integrates it into the CLI install/update/pack/publish flows and dependency lifecycle handling.
Changes:
- Add
max_stepsplumbing from Luarunner.run→RunYield/RunCmd→Host.Run, with host-side enforcement and tests. - Introduce artifact resource registry/validation/materialization and integrate it into CLI workflows and dependency reconciliation.
- Add a new
wippy artifacts materializecommand for validating/materializing a single embedded artifact resource from a WAPP.
File summaries
| File | Description |
|---|---|
| runtime/lua/modules/eval/runner/yields.go | Adds MaxSteps to RunYield and forwards it into evalhost.RunCmd. |
| runtime/lua/modules/eval/runner/spec.md | Documents the new max_steps option and its semantics. |
| runtime/lua/modules/eval/runner/module.go | Parses max_steps from the Lua config and stores it on the yielded command. |
| runtime/lua/modules/eval/runner/max_steps_integration_test.go | End-to-end integration test exercising step budget enforcement through real yields. |
| runtime/lua/evalhost/max_steps_test.go | Host-level tests for default budget and MaxSteps override behavior. |
| runtime/lua/evalhost/host.go | Implements per-run step budget selection (cmd.MaxSteps vs default). |
| runtime/lua/evalhost/command.go | Extends RunCmd with MaxSteps. |
| cmd/wippy/cmd/update.go | Adjusts update flow to always run install and refines stale artifact pruning behavior. |
| cmd/wippy/cmd/update_test.go | Updates/extends tests around vendor artifact pruning behavior. |
| cmd/wippy/cmd/publish.go | Validates artifact resource declarations prior to packing during publish. |
| cmd/wippy/cmd/pack.go | Validates artifact resources during pack. |
| cmd/wippy/cmd/install.go | Adds artifact reconciliation/materialization to install; verifies cached/downloaded WAPP integrity; retains canonical WAPPs when unpacking. |
| cmd/wippy/cmd/artifacts.go | Adds wippy artifacts materialize command for validating/materializing an artifact resource from a WAPP. |
| cmd/wippy/cmd/artifacts_test.go | Tests artifact materialization command behavior and argument parsing. |
| cmd/wippy/cmd/artifact_formats.go | Adds standard artifact registry construction and materialization helpers for CLI. |
| cmd/internal/entries/loader.go | Switches extraction to keep canonical WAPPs when unpacking. |
| cmd/internal/entries/extract.go | Exposes ExtractWappToDirKeepSource wrapper. |
| boot/deps/hub/dependency_handler.go | Integrates artifact effect into dependency reconciliation; adds downloaded artifact verification; adds artifact options to handler. |
| boot/deps/hub/artifact_effect.go | Implements artifact effect creation as part of dependency handler transactions. |
| boot/deps/hub/artifact_effect_test.go | Tests artifact effect behavior for resolved WAPPs, replacements, and empty graphs. |
| boot/deps/artifact/wapp.go | Adds transactional artifact Effect implementation for WAPP and replacement resources. |
| boot/deps/artifact/wapp_test.go | Tests artifact effect lifecycle: rollback, finalize, collision detection, exact/partial reconciliation. |
| boot/deps/artifact/standard/registry.go | Introduces standard registry composition for built-in artifact formats. |
| boot/deps/artifact/resources.go | Adds resource inspection/validation for artifact declarations (collision checks, FS validation). |
| boot/deps/artifact/registry.go | Adds artifact format registry, declaration parsing, and descriptor validation. |
| boot/deps/artifact/registry_test.go | Tests registry registration rules, declaration parsing, and collision/root validation. |
| boot/deps/artifact/nodepackage/format.go | Implements node-package artifact format inspection and safety rules (semver, name validation, lifecycle script bans). |
| boot/deps/artifact/nodepackage/format_test.go | Tests node-package format inspection and rejection cases. |
| boot/deps/artifact/materialize.go | Adds transactional materialization for a single resource, plus shared filesystem safety helpers. |
| boot/deps/artifact/materialize_test.go | Tests materialization behavior and safety checks (path portability, symlink parents, escaping). |
| boot/deps/artifact/filelock.go | Adds cross-process artifact root lock acquisition with retry/cancel behavior. |
| boot/deps/artifact/filelock_windows.go | Windows-specific file lock implementation. |
| boot/deps/artifact/filelock_unix.go | Unix-specific file lock implementation. |
| boot/deps/artifact/filelock_test.go | Tests artifact lock serialization/cancellation behavior. |
| boot/deps/artifact/effect_state_test.go | Tests rollback recovery path for partially-rolled-back effects. |
| boot/deps/artifact/directory.go | Resolves artifact resources from selected local fs.directory entries for replacements. |
| boot/deps/artifact/context.go | Adds context storage/retrieval for the boot-composed artifact registry. |
| boot/deps/artifact/config.go | Adds config hook for artifact materialization root selection. |
| boot/deps/artifact/config_test.go | Tests artifact configured root behavior. |
| boot/components/core/registry.go | Makes core registry depend on artifacts and wires artifact registry/root into dependency handler options. |
| boot/components/core/registry_test.go | Updates tests to load Artifacts component alongside Registry. |
| boot/components/core/core_test.go | Updates core lifecycle loader to include Artifacts. |
| boot/components/core/constants.go | Introduces ArtifactName as a core component constant. |
| boot/components/core/artifact.go | Adds Artifacts boot component that composes and installs the standard artifact registry into context. |
| boot/components/core/artifact_test.go | Tests artifact boot registration and built-in format availability. |
| boot/components/core/all.go | Adds Artifacts() to the core component list. |
Review details
- Files reviewed: 46/46 changed files
- Comments generated: 2
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
| // Step budget override for legitimately long evaluations (bulk imports yield | ||
| // once per batch and outgrow the interactive default). | ||
| maxSteps := 0 | ||
| if v := config.RawGetString("max_steps"); v.Type() == lua.LTNumber || v.Type() == lua.LTInteger { | ||
| maxSteps = int(lua.LVAsNumber(v)) | ||
| } |
| // Run install even for an empty or replacement-only graph so managed | ||
| // artifact roots converge and stale outputs are removed. | ||
| logger.Info("running install to converge modules and artifacts") | ||
| if err := runInstall(cmd, []string{}); err != nil { | ||
| return NewInstallFailedError(err) |
f0ae0a5 to
872341e
Compare
There was a problem hiding this comment.
🟡 Not ready to approve
The change removes the previous default step budget (making MaxSteps==0 unlimited) which contradicts the PR description and has significant behavioral/operational impact for existing callers.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Review details
Comments suppressed due to low confidence (2)
runtime/lua/modules/eval/runner/spec.md:107
- The spec documents the new
limits.max_stepsoption but the error table does not list the existing/observable failure mode when the evaluation exceeds the configured step budget ("eval exceeded maximum step limit"). Adding it makes the contract clearer for callers that set small limits.
| source is required | errors.INVALID | no |
| limits is not a table or contains an unsupported field | errors.INVALID | no |
| limits.max_steps is not a non-negative integer | errors.INVALID | no |
| compilation failed | errors.INTERNAL | no |
| execution failed | errors.INTERNAL | no |
runtime/lua/modules/eval/runner/module.go:415
parseMaxStepsonly acceptslua.LInteger. If callers provide an integrallua.LNumber(e.g.10.0) it will be rejected even though it is semantically an integer step count, and this also diverges from the PR description note about parsing numbers across Lua numeric types. Consider accepting bothLIntegerandLNumberwhile still rejecting negative or fractional values.
if !ok || n < 0 {
parseErr = "limits.max_steps must be a non-negative integer"
return
}
maxSteps = uint64(n)
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
| var stepCount uint64 | ||
| for { | ||
| stepCount++ | ||
| if stepCount > maxEvalSteps { | ||
| if cmd.MaxSteps > 0 && stepCount > cmd.MaxSteps { | ||
| return nil, ErrMaxStepsExceeded |
872341e to
d77bb47
Compare
There was a problem hiding this comment.
🟡 Not ready to approve
New test files are missing the repo’s SPDX header convention (and one test message is misleading), which should be fixed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Review details
Suppressed comments (3)
runtime/lua/evalhost/max_steps_test.go:34
- This failure message is misleading: the run does have a step limit (the host default, DefaultMaxSteps). The assertion is really that the script completes within the default budget.
t.Fatalf("expected completion without a step limit, got %v", err)
runtime/lua/evalhost/max_steps_test.go:3
- New file is missing the standard SPDX license header used by other evalhost files in this directory (e.g., host.go:1, host_test.go:1).
This issue also appears on line 34 of the same file.
package evalhost
import (
runtime/lua/modules/eval/runner/max_steps_integration_test.go:3
- New runner test file is missing the SPDX license header that other files in this package use (e.g., yields.go:1, module_test.go:1, integration_test.go:1).
package runner
import (
- Files reviewed: 12/12 changed files
- Comments generated: 0 new
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Why
Eval execution needs a configurable scheduler-step budget. The historical 10,000-step default must remain safe by default, while callers need explicit per-evaluation overrides, including an unlimited mode.
What
runner.run({ limits = { max_steps = ... } }).lua.eval.max_steps(10000by default;0is unlimited).max_steps = 0: omitted inherits the host default, while explicit zero is unlimited.uint64accounting.Verification
make lintmake testRebased directly onto current
main; no artifact changes.