Skip to content

test: Add 27 EXPR expression-language conformance fixtures - #160

Open
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-expr-lang-gaps
Open

test: Add 27 EXPR expression-language conformance fixtures#160
leongdl wants to merge 1 commit into
OpenJobDescription:mainlinefrom
leongdl:conformance-expr-lang-gaps

Conversation

@leongdl

@leongdl leongdl commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

27 new EXPR fixtures, 1 strengthened, 3 renamed, from a coverage audit of RFC 0005 / the published Expression Language spec. Suite: EXPR 350 → 379 passing, FEATURE_BUNDLE_1 55 → 57, 0 failing.

Areas covered

  • Direct path property access on all four routes — direct {{Task.File.<n>.name}}, embedded with surrounding text, inside a function call, and Env.File direct. This is the exact shape of the validator bug fixed in openjd-rs PR #292; the only pre-existing fixture reached the property through a let binding, the one route that never failed. All four now green on both implementations.
  • unresolved[T] progressive evaluation (zero prior fixtures mentioned it): valid op on an unresolved symbol passes check; type error on an unresolved value rejected at check; and the failing-arm suppression case — x if <unresolved bool> else <type-error arm> must validate, catching eager over-rejection.
  • let scope must-rejects: step-template let referencing Session.WorkingDirectory / Task.Param.* / Task.File.*; environment-script let referencing Task.Param.* — every §3.6.2 must-reject cell was previously empty.
  • Scope negatives inside expression trees (not just bare format strings): in-function-call and in-let variants.
  • Whole-field null-drop: an EXPR-computed null drops an optional scalar field (FEATURE_BUNDLE_1 timeout — audited every @fmtstring annotation; it is the only optional scalar expression field), plus the non-null control.
  • Operator precedence and associativity (zero prior fixtures): -2**2 == -4, 2**3**2 == 512 (right-assoc), a + b * c, not/and/or binding — 11 pins in one fixture.
  • Literal-table rows: \xhh/\uhhhh/\Uhhhhhhhh/\N{name} escapes; uppercase 0X/0O/0B/R prefixes; 00 valid; 1e10 is float (discriminated via §1.3.4 pass-through); three misplaced-underscore rejects.
  • Cross-type errors: 1 < "a", not 1, path(...)[0] rejected.
  • IEEE floats: 0.1 + 0.2 == 0.30000000000000004 asserted (the spec's own worked example); 0.0/0.0 rejected; float("-inf") rejected.
  • Session.HasPathMappingRules true branch via the runner's pathMapping: key — previously only ever asserted false.
  • Boundaries: exactly-50 let bindings, 512-char identifier, leading underscore, forward-reference chain; int64-max accept twins for param default and supplied value; supplied 2^63 reject (verified green on BOTH implementations in a dual run).
  • Strengthened: expr2.1.6--and-or-value-returning gained or-sentinel null discrimination and the false and x case.
  • Renamed: three expr1.3.11--* fixtures → expr1.3.12--* (they test §1.3.12 range field extensions; §1.3.11 is Error Handling).

Companion PR

Five spec-correct fixtures that FAIL today (int64 param-default acceptance, the openjd-rs 2^62 false-reject, and others) are in the expected-failures PR (branch conformance-expr-lang-expected-failures).

@leongdl
leongdl requested a review from a team as a code owner August 2, 2026 02:38
@leongdl

leongdl commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

CI note: the two failing fixtures on the Rust leg (7.3--task-file-property-direct / -embedded) are expected against the RELEASED openjd-rs — they pin the Task.File property-access fix merged in openjd-rs PR #292, which has not shipped in a release yet. They pass against openjd-rs main. Separately, mainline CI is currently red for every PR due to two pre-existing broken fixtures (wrap-env-let-bindings-in-hooks, wrap-failed-enter-still-runs-wrap-exit) unrelated to this change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: §3.6/§3.6.1 (exactly 50 bindings, exactly 512-char identifier, [a-z_] start, forward chains — all mechanically verified). Nit: header references ../proposed/ (the 513 reject twin lives on PR #164's branch, not here) — note the dependency.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: §3.6.2 table (EnvironmentScript.let excludes Task.Param.*). Nit: no accept twin exercises a valid EnvironmentScript.let, so an implementation lacking env-script let entirely also passes; consider one accept fixture.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Session.* out of StepTemplate.let scope even nested in an expression tree). len/string are valid calls, so rejection isolates the buried symbol. Coverage nit: marginal over the bare-reference twin, kept for tree-walk validators.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: §3.6.2 (StepTemplate.let may reference only Param/RawParam/Job.Name/Step.Name/earlier bindings). Minimal single defect.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Task.File.* scoped to StepScript.let only). The "no Task.File support at all" wrong-reason risk is closed by this PR's own 7.3 Task.File accept fixtures.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Task.Param.* unavailable in StepTemplate.let). parameterSpace defines Frame correctly, isolating the scope violation. The four 3.6.2 negatives cover three distinct excluded-symbol families — mainline had zero.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: §7.3.1 (Session.* not in hostRequirements scope), buried in a call. upper/string are valid, so rejection traces to the scoped symbol. Nits: filename says 7.3 while citing §7.3.1 (matches its sibling; repo mixes prefixes); marginal over the existing bare-reference negative.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.1.6 (no underscore adjacent to the decimal point). No alternate tokenization yields a valid parse; accept twin (valid underscores) on mainline.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5, one nit) — spec: EL §1.1.6 (no underscore adjacent to the exponent marker). Nit: 1_e10 may be rejected via the trailing-underscore or symbol-resolution path rather than the named rule — inherent to exit-code testing; also likely shares a lexer rule with the trailing-underscore sibling.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.1.6 (no trailing underscore). Invalid under every tokenization. The three underscore negatives map to three distinct spec clauses — not duplicates.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — one logic check requested. Spec: EL §1.2.1 unresolved-condition rule (failing arm's error suppressed under unresolved[bool]). The assertion direction (must validate) can't over-reject, but one reviewer challenges the premise: §1.2.3's single-scalar-target coercion may make upper(Task.Param.Frame) succeed (int→string in call position), in which case the fixture never actually exercises arm-failure suppression. Please verify; if coercion rescues it, switch the failing arm to something unrescuable (e.g. a subscript on an int).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 + e432ebe asymmetry rule (reject only when no resolved value could succeed): bool+int has no overload and no bool→int coercion exists, so every concrete value fails too — validation-time rejection is correct. Wrong-reason risk closed by the two accept twins in this PR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 / §7.4 (valid ops on unresolved values must type-check). The anti-over-rejection direction — exactly what the unresolved trio needed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — R100 rename tracking the spec renumber (§1.3.11→§1.3.12, Task Parameter Range Field Extensions); no stale in-file refs. Same for the path/string siblings.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5, one nit) — spec: EL §1.3.4, the spec's own 0.0/0.0 example. Nit: a generic div-by-zero guard also rejects it without any NaN check — but such rejection is still conforming (inf-producing ops are errors too), so immaterial.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §2.1.4 (ordering allows only int/float and string/path cross-pairs). Literal operands, no scope confound; no coercion path exists.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §2.1.6 note ("not remains strictly boolean"; no truthiness). Verbatim rule, minimal fixture.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §2.1.7 / RFC 0005 (path supports no subscript; the coercion escape hatch was checked — subscript receivers get an unconstrained target so path→string can't rescue it). Accept twin on mainline (string indexing).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5, coverage nit) — spec: EL §1.3.4 (float('-inf') explicitly an error; distinct spelling from inf). Low marginal value over the existing inf/nan siblings, but completes the family.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — spec: §7.3.1 Env.File scope + EL §2.3.1 path properties; deterministic given the pinned filename:. Nit (adversarial): DIRECT_STEM:envconf is a prefix of the wrong answer envconf.txt, so a stem→name bug passes that line. Add the :END terminator this PR's own 7.3--task-file-property-embedded already uses.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: §7.3.1 Session.HasPathMappingRules is bool under EXPR; true branch. Previously indistinguishable from a constant false — real gap. Both assertions flip completely on the false branch.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — spec: §7.3.1 + EL §2.3.1, whole-field route (the known validator-bug shape). Nit: STEM:config is a prefix of STEM:config.txt — terminate the STEM line so the stem→name confusion can't pass.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.3.2 embedded stringification + §2.3.1. All three assertions carry :END terminators — this is the pattern the two sibling fixtures should copy.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — spec: EL §2.2.1/§2.2.4 (upper/len over path properties). Nits: UPPER:CONFIG is a prefix of UPPER:CONFIG.TXT, and LEN:10 is a prefix of LEN:10x — add terminators or assert LEN_EQ:{{ len(...) == 10 }}. Coverage: the direct/embedded/in-call trio is at diminishing returns; direct is the known-bug shape and clearly justified.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5, one comment fix) — spec: EL §1.1 grammar; every expected value traced through the BNF and every wrong-parse alternative produces a non-prefix-compatible value or a type error. First precedence pins in the suite. Fix the garbled header phrase: "2 ** 512's exponent tree" should read "2 ** (3 ** 2) = 2 ** 9 = 512 (right-assoc), not 64".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.1.5 escape table (\xhh, \uhhhh, \Uhhhhhhhh, \N{name}). Unprocessed escapes yield raw backslash text that never matches the decoded expectations — discriminates both ways.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5, minor nit) — spec: EL §1.1.6 (uppercase prefixes, 00 explicitly valid, 1e10 float pass-through consistent with the suite's established literal-preservation reading). Nit: ZEROZERO:0 is a prefix of a literal pass-through 00; acceptance is the primary claim so low stakes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.3.4 (IEEE 754 doubles; the spec's own 0.1+0.2 example). Service-compat reviewer verified the formatting is safe for BOTH engines: Python repr and Rust {} both emit shortest-round-trip 0.30000000000000004, and BIG_LIT uses literal pass-through, avoiding the 1e+308-vs-expanded divergence.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 (2^63-1 accepted on the parameter-value axis; DEF_IS_MAX:true pins exact equality — float overflow would print 9.223372036854776e+18 and fail). Nit: header references ../proposed/ (twin lives on PR #164's branch).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5, standard .invalid caveat) — spec: EL §1.2.1 (supplied 2^63 rejected at job creation; no lazy path could succeed). On the mainline runner any non-zero exit passes, so the near-identical accept twin (int64-max, same supplied-parameter mechanism) is the guard — keep them together.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — modification strengthens: original AND_NULL couldn't distinguish null from empty string; new AND_NULL_IS_NULL:true pins actual null and AND_FALSE:false can't be satisfied by "" or "hello". No assertion weakened.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD-WITH-NITS — spec: §5 timeout @fmtstring under FB1 + EL §1.3.2; placement in FEATURE_BUNDLE_1/jobs is correct (both extensions declared; FB1 sibling precedent). Nits: taskFailure: {} alone would also pass if the implementation rejected expression-valued timeouts outright — the null-drops-field twin is what rules that out (the pair works jointly; keep together); the 1s-timeout/10s-sleep margin assumes cancellation actually terminates python before the sleep completes — safe with default SIGTERM, but the spec doesn't bound the grace period.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quorum verdict: GOOD (5/5) — spec: EL §1.3.2 (optional field evaluating to null is omitted) + §5 (onRun default is no timeout). Requires success plus DONE_NO_TIMEOUT, so null-coercion to 0/""/None fails either by rejection or by kill. First real timeout-semantics pair in the suite.

@leongdl

leongdl commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Quorum review (5 independent agents). 30 new fixtures + 3 renames + 1 strengthened; per-fixture verdicts posted as file comments. Net: no NOT-GOODs — the strongest PR of the set. Highlights: operator-precedence pins (first in suite), the unresolved-value trio (new semantic area, properly paired), HasPathMappingRules true-branch, first real timeout-enforcement pair, IEEE 0.1+0.2 pin verified safe for both Python repr and Rust shortest-round-trip formatting. Recurring actionable nit: three fixtures assert unterminated substrings that are prefixes of the most plausible wrong answer (stem-vs-name) — copy the :END terminator pattern from 7.3--task-file-property-embedded. Two headers reference a ../proposed/ dir not on this branch (lives in PRs #164/#166). One logic check requested on the unresolved-condition fixture.

…gaps

27 new fixtures, 1 strengthened, 3 renamed: direct path property access on all four routes (pins PR #292), unresolved[T] progressive evaluation incl. the failing-arm suppression case, let-scope must-rejects, in-expression scope negatives, whole-field null-drop (FEATURE_BUNDLE_1 timeout), operator precedence and associativity pins, string escapes and uppercase numeric prefixes, misplaced-underscore rejects, cross-type ordering/membership/not errors, IEEE float precision and NaN-arithmetic, HasPathMappingRules true branch, let boundary edges, int64-max accept twins, and the supplied-value 2^63 reject verified green on both implementations. Suite: 350 -> 379 passing. Spec-correct fixtures that FAIL today are split to conformance-expr-lang-expected-failures.

Review: quorum-review fixes — :END terminators close stem-vs-name and
LEN prefix traps in the Env.File/Task.File property fixtures; added
3.6.2--env-script-let-valid-scope.yaml as the accept twin of the
env-script let-scope negatives; corrected the garbled operator-precedence
comment (2 ** (3 ** 2) = 512); verified and documented that upper(int)
has no matching signature (the unresolved-condition fixture's failing arm
is real); reworded two headers that referenced a ../proposed/ directory
which lives on the expected-failures PR, not this branch.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl force-pushed the conformance-expr-lang-gaps branch from 8208897 to 2e7adba Compare August 15, 2026 19:26
@leongdl

leongdl commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Quorum-review fixes applied and pushed (rebased onto mainline 3069673):

  • :END terminators added to 7.3--env-file-property-direct, 7.3--task-file-property-direct, and 7.3--task-file-property-in-call (stem-vs-name and LEN:10 prefix traps), matching the pattern 7.3--task-file-property-embedded already used.
  • Added 3.6.2--env-script-let-valid-scope.yaml — the accept twin of the env-script let-scope negatives (without it, an implementation lacking env-script let entirely passed every negative for the wrong reason).
  • expr1.2.1--unresolved-condition-failing-arm: the reviewer's coercion concern was checked empirically — upper(int) really has no matching signature (no int→string call-position coercion; verified against openjd-rs), so the failing arm is real; documented in the header.
  • Fixed the garbled operator-precedence comment (2 ** (3 ** 2) = 512).
  • Reworded two headers that referenced a ../proposed/ directory that lives on the expected-failures PR, not this branch.

Verified: 39 affected fixtures pass against openjd-rs upstream/main 8ae4fb9. PR is now 31 new + 1 strengthened + 3 renames.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant