test: Add 27 EXPR expression-language conformance fixtures - #160
Conversation
|
CI note: the two failing fixtures on the Rust leg (7.3--task-file-property-direct / -embedded) are expected against the RELEASED openjd-rs — they pin the Task.File property-access fix merged in openjd-rs PR #292, which has not shipped in a release yet. They pass against openjd-rs main. Separately, mainline CI is currently red for every PR due to two pre-existing broken fixtures (wrap-env-let-bindings-in-hooks, wrap-failed-enter-still-runs-wrap-exit) unrelated to this change. |
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: §3.6/§3.6.1 (exactly 50 bindings, exactly 512-char identifier, [a-z_] start, forward chains — all mechanically verified). Nit: header references ../proposed/ (the 513 reject twin lives on PR #164's branch, not here) — note the dependency.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: §3.6.2 table (EnvironmentScript.let excludes Task.Param.*). Nit: no accept twin exercises a valid EnvironmentScript.let, so an implementation lacking env-script let entirely also passes; consider one accept fixture.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Session.* out of StepTemplate.let scope even nested in an expression tree). len/string are valid calls, so rejection isolates the buried symbol. Coverage nit: marginal over the bare-reference twin, kept for tree-walk validators.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: §3.6.2 (StepTemplate.let may reference only Param/RawParam/Job.Name/Step.Name/earlier bindings). Minimal single defect.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Task.File.* scoped to StepScript.let only). The "no Task.File support at all" wrong-reason risk is closed by this PR's own 7.3 Task.File accept fixtures.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: §3.6.2 (Task.Param.* unavailable in StepTemplate.let). parameterSpace defines Frame correctly, isolating the scope violation. The four 3.6.2 negatives cover three distinct excluded-symbol families — mainline had zero.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: §7.3.1 (Session.* not in hostRequirements scope), buried in a call. upper/string are valid, so rejection traces to the scoped symbol. Nits: filename says 7.3 while citing §7.3.1 (matches its sibling; repo mixes prefixes); marginal over the existing bare-reference negative.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.1.6 (no underscore adjacent to the decimal point). No alternate tokenization yields a valid parse; accept twin (valid underscores) on mainline.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5, one nit) — spec: EL §1.1.6 (no underscore adjacent to the exponent marker). Nit: 1_e10 may be rejected via the trailing-underscore or symbol-resolution path rather than the named rule — inherent to exit-code testing; also likely shares a lexer rule with the trailing-underscore sibling.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.1.6 (no trailing underscore). Invalid under every tokenization. The three underscore negatives map to three distinct spec clauses — not duplicates.
There was a problem hiding this comment.
Quorum verdict: GOOD-WITH-NITS — one logic check requested. Spec: EL §1.2.1 unresolved-condition rule (failing arm's error suppressed under unresolved[bool]). The assertion direction (must validate) can't over-reject, but one reviewer challenges the premise: §1.2.3's single-scalar-target coercion may make upper(Task.Param.Frame) succeed (int→string in call position), in which case the fixture never actually exercises arm-failure suppression. Please verify; if coercion rescues it, switch the failing arm to something unrescuable (e.g. a subscript on an int).
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 + e432ebe asymmetry rule (reject only when no resolved value could succeed): bool+int has no overload and no bool→int coercion exists, so every concrete value fails too — validation-time rejection is correct. Wrong-reason risk closed by the two accept twins in this PR.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 / §7.4 (valid ops on unresolved values must type-check). The anti-over-rejection direction — exactly what the unresolved trio needed.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — R100 rename tracking the spec renumber (§1.3.11→§1.3.12, Task Parameter Range Field Extensions); no stale in-file refs. Same for the path/string siblings.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5, one nit) — spec: EL §1.3.4, the spec's own 0.0/0.0 example. Nit: a generic div-by-zero guard also rejects it without any NaN check — but such rejection is still conforming (inf-producing ops are errors too), so immaterial.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §2.1.4 (ordering allows only int/float and string/path cross-pairs). Literal operands, no scope confound; no coercion path exists.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §2.1.6 note ("not remains strictly boolean"; no truthiness). Verbatim rule, minimal fixture.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §2.1.7 / RFC 0005 (path supports no subscript; the coercion escape hatch was checked — subscript receivers get an unconstrained target so path→string can't rescue it). Accept twin on mainline (string indexing).
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5, coverage nit) — spec: EL §1.3.4 (float('-inf') explicitly an error; distinct spelling from inf). Low marginal value over the existing inf/nan siblings, but completes the family.
There was a problem hiding this comment.
Quorum verdict: GOOD-WITH-NITS — spec: §7.3.1 Env.File scope + EL §2.3.1 path properties; deterministic given the pinned filename:. Nit (adversarial): DIRECT_STEM:envconf is a prefix of the wrong answer envconf.txt, so a stem→name bug passes that line. Add the :END terminator this PR's own 7.3--task-file-property-embedded already uses.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: §7.3.1 Session.HasPathMappingRules is bool under EXPR; true branch. Previously indistinguishable from a constant false — real gap. Both assertions flip completely on the false branch.
There was a problem hiding this comment.
Quorum verdict: GOOD-WITH-NITS — spec: §7.3.1 + EL §2.3.1, whole-field route (the known validator-bug shape). Nit: STEM:config is a prefix of STEM:config.txt — terminate the STEM line so the stem→name confusion can't pass.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.3.2 embedded stringification + §2.3.1. All three assertions carry :END terminators — this is the pattern the two sibling fixtures should copy.
There was a problem hiding this comment.
Quorum verdict: GOOD-WITH-NITS — spec: EL §2.2.1/§2.2.4 (upper/len over path properties). Nits: UPPER:CONFIG is a prefix of UPPER:CONFIG.TXT, and LEN:10 is a prefix of LEN:10x — add terminators or assert LEN_EQ:{{ len(...) == 10 }}. Coverage: the direct/embedded/in-call trio is at diminishing returns; direct is the known-bug shape and clearly justified.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5, one comment fix) — spec: EL §1.1 grammar; every expected value traced through the BNF and every wrong-parse alternative produces a non-prefix-compatible value or a type error. First precedence pins in the suite. Fix the garbled header phrase: "2 ** 512's exponent tree" should read "2 ** (3 ** 2) = 2 ** 9 = 512 (right-assoc), not 64".
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.1.5 escape table (\xhh, \uhhhh, \Uhhhhhhhh, \N{name}). Unprocessed escapes yield raw backslash text that never matches the decoded expectations — discriminates both ways.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5, minor nit) — spec: EL §1.1.6 (uppercase prefixes, 00 explicitly valid, 1e10 float pass-through consistent with the suite's established literal-preservation reading). Nit: ZEROZERO:0 is a prefix of a literal pass-through 00; acceptance is the primary claim so low stakes.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.3.4 (IEEE 754 doubles; the spec's own 0.1+0.2 example). Service-compat reviewer verified the formatting is safe for BOTH engines: Python repr and Rust {} both emit shortest-round-trip 0.30000000000000004, and BIG_LIT uses literal pass-through, avoiding the 1e+308-vs-expanded divergence.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.2.1 (2^63-1 accepted on the parameter-value axis; DEF_IS_MAX:true pins exact equality — float overflow would print 9.223372036854776e+18 and fail). Nit: header references ../proposed/ (twin lives on PR #164's branch).
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5, standard .invalid caveat) — spec: EL §1.2.1 (supplied 2^63 rejected at job creation; no lazy path could succeed). On the mainline runner any non-zero exit passes, so the near-identical accept twin (int64-max, same supplied-parameter mechanism) is the guard — keep them together.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — modification strengthens: original AND_NULL couldn't distinguish null from empty string; new AND_NULL_IS_NULL:true pins actual null and AND_FALSE:false can't be satisfied by "" or "hello". No assertion weakened.
There was a problem hiding this comment.
Quorum verdict: GOOD-WITH-NITS — spec: §5 timeout @fmtstring under FB1 + EL §1.3.2; placement in FEATURE_BUNDLE_1/jobs is correct (both extensions declared; FB1 sibling precedent). Nits: taskFailure: {} alone would also pass if the implementation rejected expression-valued timeouts outright — the null-drops-field twin is what rules that out (the pair works jointly; keep together); the 1s-timeout/10s-sleep margin assumes cancellation actually terminates python before the sleep completes — safe with default SIGTERM, but the spec doesn't bound the grace period.
There was a problem hiding this comment.
Quorum verdict: GOOD (5/5) — spec: EL §1.3.2 (optional field evaluating to null is omitted) + §5 (onRun default is no timeout). Requires success plus DONE_NO_TIMEOUT, so null-coercion to 0/""/None fails either by rejection or by kill. First real timeout-semantics pair in the suite.
|
Quorum review (5 independent agents). 30 new fixtures + 3 renames + 1 strengthened; per-fixture verdicts posted as file comments. Net: no NOT-GOODs — the strongest PR of the set. Highlights: operator-precedence pins (first in suite), the unresolved-value trio (new semantic area, properly paired), HasPathMappingRules true-branch, first real timeout-enforcement pair, IEEE 0.1+0.2 pin verified safe for both Python repr and Rust shortest-round-trip formatting. Recurring actionable nit: three fixtures assert unterminated substrings that are prefixes of the most plausible wrong answer (stem-vs-name) — copy the |
…gaps 27 new fixtures, 1 strengthened, 3 renamed: direct path property access on all four routes (pins PR #292), unresolved[T] progressive evaluation incl. the failing-arm suppression case, let-scope must-rejects, in-expression scope negatives, whole-field null-drop (FEATURE_BUNDLE_1 timeout), operator precedence and associativity pins, string escapes and uppercase numeric prefixes, misplaced-underscore rejects, cross-type ordering/membership/not errors, IEEE float precision and NaN-arithmetic, HasPathMappingRules true branch, let boundary edges, int64-max accept twins, and the supplied-value 2^63 reject verified green on both implementations. Suite: 350 -> 379 passing. Spec-correct fixtures that FAIL today are split to conformance-expr-lang-expected-failures. Review: quorum-review fixes — :END terminators close stem-vs-name and LEN prefix traps in the Env.File/Task.File property fixtures; added 3.6.2--env-script-let-valid-scope.yaml as the accept twin of the env-script let-scope negatives; corrected the garbled operator-precedence comment (2 ** (3 ** 2) = 512); verified and documented that upper(int) has no matching signature (the unresolved-condition fixture's failing arm is real); reworded two headers that referenced a ../proposed/ directory which lives on the expected-failures PR, not this branch. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
8208897 to
2e7adba
Compare
|
Quorum-review fixes applied and pushed (rebased onto mainline 3069673):
Verified: 39 affected fixtures pass against openjd-rs upstream/main |
Summary
27 new EXPR fixtures, 1 strengthened, 3 renamed, from a coverage audit of RFC 0005 / the published Expression Language spec. Suite: EXPR 350 → 379 passing, FEATURE_BUNDLE_1 55 → 57, 0 failing.
Areas covered
{{Task.File.<n>.name}}, embedded with surrounding text, inside a function call, andEnv.Filedirect. This is the exact shape of the validator bug fixed in openjd-rs PR #292; the only pre-existing fixture reached the property through aletbinding, the one route that never failed. All four now green on both implementations.unresolved[T]progressive evaluation (zero prior fixtures mentioned it): valid op on an unresolved symbol passescheck; type error on an unresolved value rejected atcheck; and the failing-arm suppression case —x if <unresolved bool> else <type-error arm>must validate, catching eager over-rejection.letscope must-rejects: step-templateletreferencingSession.WorkingDirectory/Task.Param.*/Task.File.*; environment-scriptletreferencingTask.Param.*— every §3.6.2 must-reject cell was previously empty.letvariants.timeout— audited every@fmtstringannotation; it is the only optional scalar expression field), plus the non-null control.-2**2 == -4,2**3**2 == 512(right-assoc),a + b * c,not/and/orbinding — 11 pins in one fixture.\xhh/\uhhhh/\Uhhhhhhhh/\N{name}escapes; uppercase0X/0O/0B/Rprefixes;00valid;1e10is float (discriminated via §1.3.4 pass-through); three misplaced-underscore rejects.1 < "a",not 1,path(...)[0]rejected.0.1 + 0.2 == 0.30000000000000004asserted (the spec's own worked example);0.0/0.0rejected;float("-inf")rejected.Session.HasPathMappingRulestrue branch via the runner'spathMapping:key — previously only ever assertedfalse.expr2.1.6--and-or-value-returninggainedor-sentinel null discrimination and thefalse and xcase.expr1.3.11--*fixtures →expr1.3.12--*(they test §1.3.12 range field extensions; §1.3.11 is Error Handling).Companion PR
Five spec-correct fixtures that FAIL today (int64 param-default acceptance, the openjd-rs 2^62 false-reject, and others) are in the expected-failures PR (branch
conformance-expr-lang-expected-failures).