Skip to content

Add configurable validation checkers - #19

Merged
MariaHov merged 2 commits into
mainfrom
feature/configurable-validation-checkers
Sep 17, 2026
Merged

MariaHov merged 2 commits into
mainfrom
feature/configurable-validation-checkers

Conversation

@MariaHov

Copy link
Copy Markdown
Collaborator

Summary

  • Adds config-gated metrics: regex match, valid JSON, and LLM judge
  • Uses a separately configured judge model for LLM scoring to avoid self-evaluation bias
  • Keeps Course 1 defaults unchanged unless allowedMetricIds opts into the new checkers

Test plan

  • With default config, only original Course 1 metrics appear
  • Enable regex-match, valid-json, and llm-judge in session.config.json and confirm they appear in the metric picker
  • Run regex and valid-json evals with known pass/fail cases
  • Run LLM judge and confirm the UI shows the fixed judge model
  • npm test

Made with Cursor

Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 3bc255fc-526c-4d02-af09-427883262fd9


Comment @coderabbitai help to get the list of available commands.

Co-authored-by: Cursor <cursoragent@cursor.com>
@MariaHov
MariaHov merged commit e044536 into main Sep 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant