Skip to content

feat(eval): support explicit execution step limits - #536

Merged
wolfy-j merged 1 commit into
mainfrom
feat/eval-max-steps
Aug 1, 2026
Merged

feat(eval): support explicit execution step limits#536
wolfy-j merged 1 commit into
mainfrom
feat/eval-max-steps

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Why

Eval execution needs a configurable scheduler-step budget. The historical 10,000-step default must remain safe by default, while callers need explicit per-evaluation overrides, including an unlimited mode.

What

  • Add runner.run({ limits = { max_steps = ... } }).
  • Configure the inherited host default with lua.eval.max_steps (10000 by default; 0 is unlimited).
  • Distinguish an omitted limit from explicit max_steps = 0: omitted inherits the host default, while explicit zero is unlimited.
  • Enforce positive limits in the eval host using uint64 accounting.
  • Reject non-table limits, unknown fields, negative values, fractional values, and strings.

Verification

  • make lint
  • make test

Rebased directly onto current main; no artifact changes.

Copilot AI review requested due to automatic review settings July 30, 2026 02:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Human review recommended

It combines a small eval-host feature with a large, behavior-changing artifact/CLI/dependency subsystem and includes at least one confirmed correctness issue needing adjustment.

Pull request overview

This PR makes the eval runner’s scheduler step budget configurable via a new max_steps option on runner.run(config), enforced in the eval host loop, and adds comprehensive tests and spec documentation. In addition, it introduces an artifact resource validation/materialization subsystem and integrates it into the CLI install/update/pack/publish flows and dependency lifecycle handling.

Changes:

  • Add max_steps plumbing from Lua runner.runRunYield/RunCmdHost.Run, with host-side enforcement and tests.
  • Introduce artifact resource registry/validation/materialization and integrate it into CLI workflows and dependency reconciliation.
  • Add a new wippy artifacts materialize command for validating/materializing a single embedded artifact resource from a WAPP.
File summaries
File Description
runtime/lua/modules/eval/runner/yields.go Adds MaxSteps to RunYield and forwards it into evalhost.RunCmd.
runtime/lua/modules/eval/runner/spec.md Documents the new max_steps option and its semantics.
runtime/lua/modules/eval/runner/module.go Parses max_steps from the Lua config and stores it on the yielded command.
runtime/lua/modules/eval/runner/max_steps_integration_test.go End-to-end integration test exercising step budget enforcement through real yields.
runtime/lua/evalhost/max_steps_test.go Host-level tests for default budget and MaxSteps override behavior.
runtime/lua/evalhost/host.go Implements per-run step budget selection (cmd.MaxSteps vs default).
runtime/lua/evalhost/command.go Extends RunCmd with MaxSteps.
cmd/wippy/cmd/update.go Adjusts update flow to always run install and refines stale artifact pruning behavior.
cmd/wippy/cmd/update_test.go Updates/extends tests around vendor artifact pruning behavior.
cmd/wippy/cmd/publish.go Validates artifact resource declarations prior to packing during publish.
cmd/wippy/cmd/pack.go Validates artifact resources during pack.
cmd/wippy/cmd/install.go Adds artifact reconciliation/materialization to install; verifies cached/downloaded WAPP integrity; retains canonical WAPPs when unpacking.
cmd/wippy/cmd/artifacts.go Adds wippy artifacts materialize command for validating/materializing an artifact resource from a WAPP.
cmd/wippy/cmd/artifacts_test.go Tests artifact materialization command behavior and argument parsing.
cmd/wippy/cmd/artifact_formats.go Adds standard artifact registry construction and materialization helpers for CLI.
cmd/internal/entries/loader.go Switches extraction to keep canonical WAPPs when unpacking.
cmd/internal/entries/extract.go Exposes ExtractWappToDirKeepSource wrapper.
boot/deps/hub/dependency_handler.go Integrates artifact effect into dependency reconciliation; adds downloaded artifact verification; adds artifact options to handler.
boot/deps/hub/artifact_effect.go Implements artifact effect creation as part of dependency handler transactions.
boot/deps/hub/artifact_effect_test.go Tests artifact effect behavior for resolved WAPPs, replacements, and empty graphs.
boot/deps/artifact/wapp.go Adds transactional artifact Effect implementation for WAPP and replacement resources.
boot/deps/artifact/wapp_test.go Tests artifact effect lifecycle: rollback, finalize, collision detection, exact/partial reconciliation.
boot/deps/artifact/standard/registry.go Introduces standard registry composition for built-in artifact formats.
boot/deps/artifact/resources.go Adds resource inspection/validation for artifact declarations (collision checks, FS validation).
boot/deps/artifact/registry.go Adds artifact format registry, declaration parsing, and descriptor validation.
boot/deps/artifact/registry_test.go Tests registry registration rules, declaration parsing, and collision/root validation.
boot/deps/artifact/nodepackage/format.go Implements node-package artifact format inspection and safety rules (semver, name validation, lifecycle script bans).
boot/deps/artifact/nodepackage/format_test.go Tests node-package format inspection and rejection cases.
boot/deps/artifact/materialize.go Adds transactional materialization for a single resource, plus shared filesystem safety helpers.
boot/deps/artifact/materialize_test.go Tests materialization behavior and safety checks (path portability, symlink parents, escaping).
boot/deps/artifact/filelock.go Adds cross-process artifact root lock acquisition with retry/cancel behavior.
boot/deps/artifact/filelock_windows.go Windows-specific file lock implementation.
boot/deps/artifact/filelock_unix.go Unix-specific file lock implementation.
boot/deps/artifact/filelock_test.go Tests artifact lock serialization/cancellation behavior.
boot/deps/artifact/effect_state_test.go Tests rollback recovery path for partially-rolled-back effects.
boot/deps/artifact/directory.go Resolves artifact resources from selected local fs.directory entries for replacements.
boot/deps/artifact/context.go Adds context storage/retrieval for the boot-composed artifact registry.
boot/deps/artifact/config.go Adds config hook for artifact materialization root selection.
boot/deps/artifact/config_test.go Tests artifact configured root behavior.
boot/components/core/registry.go Makes core registry depend on artifacts and wires artifact registry/root into dependency handler options.
boot/components/core/registry_test.go Updates tests to load Artifacts component alongside Registry.
boot/components/core/core_test.go Updates core lifecycle loader to include Artifacts.
boot/components/core/constants.go Introduces ArtifactName as a core component constant.
boot/components/core/artifact.go Adds Artifacts boot component that composes and installs the standard artifact registry into context.
boot/components/core/artifact_test.go Tests artifact boot registration and built-in format availability.
boot/components/core/all.go Adds Artifacts() to the core component list.
Review details
  • Files reviewed: 46/46 changed files
  • Comments generated: 2
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment on lines +363 to +368
// Step budget override for legitimately long evaluations (bulk imports yield
// once per batch and outgrow the interactive default).
maxSteps := 0
if v := config.RawGetString("max_steps"); v.Type() == lua.LTNumber || v.Type() == lua.LTInteger {
maxSteps = int(lua.LVAsNumber(v))
}
Comment thread cmd/wippy/cmd/update.go Outdated
Comment on lines +220 to +224
// Run install even for an empty or replacement-only graph so managed
// artifact roots converge and stale outputs are removed.
logger.Info("running install to converge modules and artifacts")
if err := runInstall(cmd, []string{}); err != nil {
return NewInstallFailedError(err)
Copilot AI review requested due to automatic review settings July 30, 2026 03:42
@wolfy-j
wolfy-j force-pushed the feat/eval-max-steps branch from f0ae0a5 to 872341e Compare July 30, 2026 03:42
@wolfy-j wolfy-j changed the title feat(eval): configurable step budget via runner.run max_steps feat(eval): support explicit execution step limits Jul 30, 2026
@wolfy-j
wolfy-j requested a review from skhaz July 30, 2026 03:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The change removes the previous default step budget (making MaxSteps==0 unlimited) which contradicts the PR description and has significant behavioral/operational impact for existing callers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (2)

runtime/lua/modules/eval/runner/spec.md:107

  • The spec documents the new limits.max_steps option but the error table does not list the existing/observable failure mode when the evaluation exceeds the configured step budget ("eval exceeded maximum step limit"). Adding it makes the contract clearer for callers that set small limits.
| source is required | errors.INVALID | no |
| limits is not a table or contains an unsupported field | errors.INVALID | no |
| limits.max_steps is not a non-negative integer | errors.INVALID | no |
| compilation failed | errors.INTERNAL | no |
| execution failed | errors.INTERNAL | no |

runtime/lua/modules/eval/runner/module.go:415

  • parseMaxSteps only accepts lua.LInteger. If callers provide an integral lua.LNumber (e.g. 10.0) it will be rejected even though it is semantically an integer step count, and this also diverges from the PR description note about parsing numbers across Lua numeric types. Consider accepting both LInteger and LNumber while still rejecting negative or fractional values.
		if !ok || n < 0 {
			parseErr = "limits.max_steps must be a non-negative integer"
			return
		}
		maxSteps = uint64(n)
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment on lines +266 to 270
var stepCount uint64
for {
stepCount++
if stepCount > maxEvalSteps {
if cmd.MaxSteps > 0 && stepCount > cmd.MaxSteps {
return nil, ErrMaxStepsExceeded
Copilot AI review requested due to automatic review settings August 1, 2026 20:46
@wolfy-j
wolfy-j force-pushed the feat/eval-max-steps branch from 872341e to d77bb47 Compare August 1, 2026 20:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

New test files are missing the repo’s SPDX header convention (and one test message is misleading), which should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (3)

runtime/lua/evalhost/max_steps_test.go:34

  • This failure message is misleading: the run does have a step limit (the host default, DefaultMaxSteps). The assertion is really that the script completes within the default budget.
			t.Fatalf("expected completion without a step limit, got %v", err)

runtime/lua/evalhost/max_steps_test.go:3

  • New file is missing the standard SPDX license header used by other evalhost files in this directory (e.g., host.go:1, host_test.go:1).

This issue also appears on line 34 of the same file.

package evalhost

import (

runtime/lua/modules/eval/runner/max_steps_integration_test.go:3

  • New runner test file is missing the SPDX license header that other files in this package use (e.g., yields.go:1, module_test.go:1, integration_test.go:1).
package runner

import (
  • Files reviewed: 12/12 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

@wolfy-j
wolfy-j merged commit 79d6140 into main Aug 1, 2026
5 checks passed
@wolfy-j
wolfy-j deleted the feat/eval-max-steps branch August 1, 2026 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants