Skip to content

fix(scorer): report invalid schema patterns - #22

Merged
royalpinto007 merged 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-8-invalid-schema-pattern
Sep 5, 2026
Merged

fix(scorer): report invalid schema patterns#22
royalpinto007 merged 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-8-invalid-schema-pattern

Conversation

@vsolano9

Copy link
Copy Markdown
Contributor

What & why

Convert an invalid user-supplied JSON Schema pattern from an uncaught SyntaxError into a normal failing scorer result. The error now includes the exact JSON path, pattern, and runtime compilation message, so one malformed schema cannot abort the entire eval run.

Closes #8

Changes

  • catch regex compilation failures inside validate()
  • append a path-attributed validation error for invalid patterns
  • cover a nested $.account invalid pattern through the real scorer
  • pin the neighboring valid-pattern mismatch message

Validation

  • npm run typecheck
  • npm run lint
  • npm run build
  • npm test — 9 files, 70 tests passed
  • focused invalid/mismatch coverage repeated 20 times
  • built-validator nested-path proof
  • deterministic 7-case example suite passed
  • full typecheck/lint/build/test/example gate passed on Node 20.20.2 and 22.23.2
  • npm pack --dry-run and clean installed-tarball public validateJsonSchema smoke passed
  • production dependency audit reports zero findings
  • git diff --check

Checklist

  • Focused change with a clear description
  • Ran the complete documented local gate
  • No network or API-key test path
  • No dependency, public API, or configuration change
  • Linked issue with Closes #8

Notes

Fresh install reports seven existing development-dependency advisories; production dependencies report zero, and this PR changes neither manifest nor lockfile.

Catch JSON Schema pattern compilation failures and return a path-attributed validation error instead of aborting the evaluation run. Cover nested invalid patterns and preserve valid mismatch behavior.
@github-actions

Copy link
Copy Markdown

Welcome, @vsolano9, and thanks for your first pull request to Evalgate.

A quick look at CONTRIBUTING.md covers how things are set up and run here.

I'll review this shortly. Ask anything in the thread if something is unclear.

@vsolano9

Copy link
Copy Markdown
Contributor Author

Postflight note on the red quality-gate context:

  • The quality comparison itself passed: base 94.2% -> head 94.2% (0.0pp) and No regressions beyond tolerance.
  • The action then exited 2 only while publishing its optional PR comment: [evalgate] failed to create comment (403). Fork-triggered pull_request runs receive a read-only token even though the workflow requests pull-requests: write.
  • The separate repository CI completed successfully on Node 20 and 22, including typecheck, lint, build, all 70 tests, and the example-suite smoke.

This PR does not change the action or workflow; the failing context is the existing fork-comment permission path, not a scorer/test regression.

@royalpinto007

Copy link
Copy Markdown
Collaborator

Thanks. Same quality-gate failure as #23 - please rebase and confirm eval comparison passes.

@royalpinto007 royalpinto007 reopened this Sep 5, 2026
@royalpinto007 royalpinto007 reopened this Sep 5, 2026
@royalpinto007 royalpinto007 reopened this Sep 5, 2026
@royalpinto007 royalpinto007 reopened this Sep 5, 2026
@royalpinto007
royalpinto007 merged commit cc32c2d into AgentPostmortem:main Sep 5, 2026
41 of 46 checks passed
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

Nice work @vsolano9, this is merged.

Good, focused change. Exactly the kind of thing Evalgate needs.

If Evalgate is useful to you, a star goes a long way.

Happy to see more from you whenever you have the time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An invalid pattern in a JSON schema throws and kills the whole run

2 participants