Skip to content

fix(scorer): report invalid schema patterns - #22

Open
vsolano9 wants to merge 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-8-invalid-schema-pattern
Open

fix(scorer): report invalid schema patterns#22
vsolano9 wants to merge 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-8-invalid-schema-pattern

Conversation

@vsolano9

Copy link
Copy Markdown

What & why

Convert an invalid user-supplied JSON Schema pattern from an uncaught SyntaxError into a normal failing scorer result. The error now includes the exact JSON path, pattern, and runtime compilation message, so one malformed schema cannot abort the entire eval run.

Closes #8

Changes

  • catch regex compilation failures inside validate()
  • append a path-attributed validation error for invalid patterns
  • cover a nested $.account invalid pattern through the real scorer
  • pin the neighboring valid-pattern mismatch message

Validation

  • npm run typecheck
  • npm run lint
  • npm run build
  • npm test — 9 files, 70 tests passed
  • focused invalid/mismatch coverage repeated 20 times
  • built-validator nested-path proof
  • deterministic 7-case example suite passed
  • full typecheck/lint/build/test/example gate passed on Node 20.20.2 and 22.23.2
  • npm pack --dry-run and clean installed-tarball public validateJsonSchema smoke passed
  • production dependency audit reports zero findings
  • git diff --check

Checklist

  • Focused change with a clear description
  • Ran the complete documented local gate
  • No network or API-key test path
  • No dependency, public API, or configuration change
  • Linked issue with Closes #8

Notes

Fresh install reports seven existing development-dependency advisories; production dependencies report zero, and this PR changes neither manifest nor lockfile.

Catch JSON Schema pattern compilation failures and return a path-attributed validation error instead of aborting the evaluation run. Cover nested invalid patterns and preserve valid mismatch behavior.
@github-actions

Copy link
Copy Markdown

Welcome, @vsolano9, and thanks for your first pull request to Evalgate.

A quick look at CONTRIBUTING.md covers how things are set up and run here.

I'll review this shortly. Ask anything in the thread if something is unclear.

@vsolano9

Copy link
Copy Markdown
Author

Postflight note on the red quality-gate context:

  • The quality comparison itself passed: base 94.2% -> head 94.2% (0.0pp) and No regressions beyond tolerance.
  • The action then exited 2 only while publishing its optional PR comment: [evalgate] failed to create comment (403). Fork-triggered pull_request runs receive a read-only token even though the workflow requests pull-requests: write.
  • The separate repository CI completed successfully on Node 20 and 22, including typecheck, lint, build, all 70 tests, and the example-suite smoke.

This PR does not change the action or workflow; the failing context is the existing fork-comment permission path, not a scorer/test regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An invalid pattern in a JSON schema throws and kills the whole run

1 participant