test: Add §3.4 range-cap and §7.3.1 Step.Name job conformance fixtures - #155
test: Add §3.4 range-cap and §7.3.1 Step.Name job conformance fixtures#155leongdl wants to merge 1 commit into
Conversation
| # Template Schemas 3.4: a task parameter's range may take on at most 1024 | ||
| # values. A user-SUPPLIED RANGE_EXPR value is only expanded at job creation, | ||
| # so the cap must be enforced there: "1-1025" expands to 1025 values and job | ||
| # creation must fail, even though the template itself is valid. |
There was a problem hiding this comment.
This sounds wrong, range expressions are not limited the same as range values. I remember we made sure the range expression wording didn't include that limit, seems like we need to amend that wording to be clear that the range limit does not apply.
| steps: | ||
| - name: EchoStepName | ||
| let: | ||
| - banner = 'step-is-' + Step.Name |
There was a problem hiding this comment.
Interesting, since this let is executed at step scope, and can access Step.Name
There was a problem hiding this comment.
Quorum verdict: NOT-GOOD (5/5) — spec: §3.4.1.1.1 <IntRangeExpr>.
Upstream commit 3069673 added normative text: the number of values an <IntRangeExpr> expands to "is not constrained by this specification", and the §3.4.1.1 1024-element cap "does not apply to this form" — a >1024-value expansion is the motivating use-case (see the new mainline fixture 3.4--wide-int-range-expression.yaml, which accepts 1-5000). This fixture pins rejection of a supplied RANGE_EXPR "1-1025", i.e. the opposite of current spec. A conformant implementation exits 0 and fails the test; a non-conformant one passes for any rejection reason. Recommend: drop, or invert to an acceptance fixture.
There was a problem hiding this comment.
Quorum verdict: GOOD-WITH-NITS (5/5) — spec: §3.4.1.1 <IntRangeList> max 1024 elements (accept side at exactly 1024, via whole-field LIST[INT] resolution).
Nits: (1) header cites "RFC 0006" for typed whole-field resolution — that mechanism is RFC 0005 / Expression Language §1.3.2 (RFC 0006 is the function library); (2) header wording "the range cap is at MOST 1024 values" conflates the capped list form with the now-uncapped expr form (post-3069673) — scope it to the list form; (3) consider whether a resolved list[int] inherits the literal-list cap at all — the spec doesn't state it explicitly; (4) ~3 min runtime on the Python CLI for two substring assertions; the 1024-int one-line literal is hard to review.
There was a problem hiding this comment.
Quorum verdict: GOOD-WITH-NITS (4/5; 1 dissent NOT-GOOD) — spec: §3.4.1.1 list-form 1024-element cap, rejection side at 1025.
Majority reads the resolved-list value as filling the <IntRangeList> form, which remains capped post-3069673, so rejecting 1025 is correct and doesn't over-constrain (the uncapped path is <IntRangeExpr>). Dissent: the spec never explicitly says the literal-form cap applies to a whole-field-resolved list, and §3.4.1.1.1's philosophy says implementations should bound task count with their own limits — worth an explicit spec sentence either way. Nits: same "RFC 0006" misattribution (should be RFC 0005 / EL §1.3.2); header generalizes "a task parameter's range may take on at most 1024 values", which is false for the expr form — reword.
There was a problem hiding this comment.
Quorum verdict: GOOD (3 GOOD / 2 GOOD-WITH-NITS) — spec: §7.3.1 Step.Name available in the Step Template scope incl. stepEnvironments; §3.6.2 StepTemplate.let may reference Step.Name with bindings visible in stepEnvironments.
All assertions carry the exact resolved value (EchoStepName / step-is-EchoStepName), so a wrong or unresolved value can't pass by substring accident. Fills a real runtime gap: existing coverage was static-only or onRun-only. Nit: header cites "RFC 0007 7.3.1" — Step.Name comes from RFC 0005 (the section number is the Template Schemas wiki's); RFC 0007 is parameter types.
|
Quorum review (5 independent agents: spec-literalist, adversarial, test-craft, service-compat, coverage). Per-fixture verdicts posted as file comments. Net: 1 NOT-GOOD, 2 GOOD-WITH-NITS, 1 GOOD. Blocking item: |
Four fixtures pinning create-time and run-time behavior that no existing
fixture covered, updated for the §3.4.1.1.1 spec fix (IntRangeExpr
expansion is uncapped; only the <IntRangeList> form carries the 1024 cap):
- 3.4--range-from-list-param-at-limit.test: exactly 1024 list-resolved
range elements must be accepted (list-form cap, accept side).
- 3.4--range-from-list-param-too-long.invalid.test: a supplied LIST[INT]
of 1025 items through a typed whole-field range "{{Param.Values}}" is
decode-clean but must fail at job creation (list-form cap, reject side).
- 3.4--range-expr-supplied-value-beyond-list-cap.test: a supplied
RANGE_EXPR "1-1025" must be ACCEPTED and run 1025 tasks — IntRangeExpr
expansion is explicitly unconstrained by the spec (3.4.1.1.1). Replaces
the earlier .invalid fixture, which pre-dated the spec fix and asserted
the opposite.
- 7.3.1--step-name-in-step-environment.test: Step.Name resolves in a
step environment's onEnter/onExit, directly and via step-level let.
Review: quorum-review fixes — corrected RFC citations (whole-field
resolution is RFC 0005 / Expression Language 1.3.2, not RFC 0006;
Step.Name is RFC 0005, not RFC 0007) and scoped all cap language to the
list form.
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
c745815 to
4348402
Compare
|
Quorum-review fixes applied and pushed (rebased onto mainline 3069673):
Verified: all 4 fixtures pass against openjd-rs built from upstream/main |
2 participants
What was the problem/requirement? (What/Why)
Review of the RFC 0007/0008 implementation stack (openjd-model-for-python #318/#313, openjd-sessions-for-python #333, openjd-cli #230) found two spec-observable behaviors that no conformance fixture pinned — both had silently diverged between the Python and Rust implementations:
"{{Param.Values}}"over aLIST[*]parameter, or a suppliedRANGE_EXPR— are only expanded at job creation, where openjd-rs enforced the cap but the Python model did not.Step.Nameresolves inside a step environment's actions — the Rust CLI passed this while the Python CLI failed it.What was the solution? (How)
Four fixtures in
conformance-tests/2023-09/EXPR/jobs/:3.4--range-from-list-param-too-long.invalid.test.yaml— suppliedLIST[INT]of 1025 items via a typed whole-field range; decode-clean, must fail at job creation.3.4--range-expr-supplied-value-too-long.invalid.test.yaml— suppliedRANGE_EXPR"1-1025"; decode-clean, must fail at job creation.3.4--range-from-list-param-at-limit.test.yaml— exactly 1024 values accepted (guards the off-by-one an at-the-cap rejection bug would introduce). Note: runs 1024 trivial tasks — seconds on the Rust CLI, ~3 minutes on the Python CLI; the fixture header documents this.7.3.1--step-name-in-step-environment.test.yaml—Step.Nameresolves in a step environment's onEnter/onExit at run time, directly and through a step-levelletbinding. (Negative polarity — Step.Name rejected in job environments — already exists as decode-time.invalidfixtures.)How was this change tested?
All four fixtures validated against both implementations via
run_openjd_cli_tests.py: the openjd-rs release CLI, and the Python CLI running the fixed RFC 0007/0008 branches. Both.invalidtemplates confirmed decode-clean (openjd checkpasses on both CLIs), failing only at job creation.