Skip to content

Three shipped rulesets are rejected at load and vanish from every denominator #575

Description

@beyondnetPeru

The defect

The corpus loader globs *.rules.json and assumes every match has one shape — a ruleset with
rules[] or principles[]. Three shipped files declare a different schema, so the loader
rejects them, prints a warning, and drops them out of every denominator the run reports.

This is the failure the project exists to stop, occurring in the project's own loader. A skipped
rule is at least counted as skipped. These are not counted at all.

Reproduction — first five lines of a first run, from a clean container

docker run --rm node:20 bash -lc '
  npm install -g @beyondnet/evolith-cli@1.3.0 --silent
  cd /tmp && evolith init --name my-sat --yes >/dev/null 2>&1
  cd /tmp/my-sat && evolith validate --engine opa'
[Nest] WARN Skipping non-standard ruleset .../rulesets/architecture/topology-recommendation.rules.json: Schema validation failed: data must have required property 'rules', data must have required property 'principles', data must match a schema in anyOf
[Nest] WARN Skipping non-standard ruleset .../rulesets/infrastructure/helm-enforcement.rules.json: Schema validation failed: ...
[Nest] WARN Skipping non-standard ruleset .../rulesets/infrastructure/opa-sidecar-bundle.rules.json: Schema validation failed: ...

What is actually wrong with each file — two different causes

file declared $schema top-level keys what it really is
src/rulesets/infrastructure/helm-enforcement.rules.json ../schema/rule-definition.schema.json id, category, title, description, severity, rationale, validation a single rule, never wrapped in { "rules": [ ... ] }
src/rulesets/infrastructure/opa-sidecar-bundle.rules.json ../schema/rule-definition.schema.json same same
src/rulesets/architecture/topology-recommendation.rules.json ../schema/topology-recommendation.schema.json id, version, progressive, dimensions not a ruleset at all — a different artifact type caught by the .rules.json glob

The two INFRA files are each valid against the schema they themselves declare. They are
rule definitions, and the loader only accepts rule sets. The third is a topology
recommendation config whose only crime is its filename.

Note this means the whole of src/rulesets/infrastructure/ is discarded: those two files are
the only *.rules.json in that directory.

The part that matters: they are in no denominator

Measured against the same captured run (INFRA-001 and INFRA-OPA-001 are the two ids):

corpusTotal:    412        <- does not include them
rulesSelected:  412
Rules Total:    159        <- does not include them
Rules Skipped:   26        <- does not include them
occurrences of "INFRA-001"     in 93 lines of output: 0
occurrences of "INFRA-OPA-001" in 93 lines of output: 0

topology-recommendation appears exactly once, and that occurrence is the warning itself.

So the report says 412 corpus / 159 selected as though that were the whole corpus, and two
infrastructure rules — one of them an OPA sidecar bundle policy — are invisible. A run cannot
report them as skipped, because as far as the accounting is concerned they were never there.

Suggested fix, in the order that matters

  1. The accounting first. A file the loader rejects must land in the report as a
    first-class outcome, the same way an undecided rule does. A warning on stderr is not
    accounting — it does not survive --format json, and it does not reach the exit code.
    This is the load-bearing half: it makes any future instance of this visible instead of
    silent.
  2. Then the three files. Wrap the two INFRA rules in a ruleset envelope, and either
    rename topology-recommendation.rules.json out of the glob or teach the loader to
    dispatch on the declared $schema rather than on the file extension.

Doing 2 without 1 fixes three files and leaves the mechanism that hid them intact.

Proving a fix

# no file is dropped at load
docker run --rm node:20 bash -lc '... evolith validate --engine opa' 2>&1 \
  | grep -c "Skipping non-standard ruleset"     # must be 0

# and the two rules are now accounted for, one way or another
... evolith validate --engine opa --format json | grep -c "INFRA-OPA-001"   # must be >= 1

What is not your problem

  • Open Dependabot alerts and the OpenSSF Scorecard workflow are known-red and unrelated.
  • The bilingual guards apply to the 17-document entry surface only; ruleset JSON is not in it,
    so no .es twin is required for this change.
  • If Governance guards goes red on something you did not touch, say so in the PR — it is not
    a required check.

Found while re-verifying the README's clean-room capture for #574, which is where the
[Nest] WARN lines are now acknowledged in prose rather than trimmed out of the screenshot.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions