LexGuard

The release gate for legal AI

LexGuard evaluates a candidate legal LLM against a versioned benchmark, compares it with a trusted baseline, and blocks the release when it introduces a critical safety failure or any new regression.

Ship legal AI with evidence, not hope.

The problem

A model can improve on an average benchmark while becoming less safe in one high-impact behavior: inventing a citation, obeying prompt injection hidden in a contract, leaking an identifier, or proposing a rights waiver. A single aggregate score hides that regression.

LexGuard makes the release decision explicit and inspectable:

1. Run the same 30 legal-safety cases against a baseline and candidate.
2. Grade every response with deterministic, assertion-level metrics.
3. Compare results under an immutable release policy.
4. Block on critical failures, new failures, threshold misses, or excess hallucination.
5. Show the exact prompt, output, assertion evidence, and prompt improvement.

The 60-second judge journey

1. Open /login and click Enter evaluator demo—no credentials or API keys are required.
2. LexGuard runs candidate-regressing against Legal Release Gate v1.
3. The command center returns Release blocked and explains every policy violation.
4. Click a critical regression to inspect its prompt, unsafe output, failed assertion, release impact, and suggested prompt fix.
5. Open the full run, filter all 30 cases, and download the Markdown or JSON evidence report.

The deterministic demo is intentional: judges get the same reproducible result without spending keys or waiting for an external model. The target interface is isolated so a production provider can replace the fixture-backed adapter.

What makes LexGuard different

• Safety-first release policy: a critical failure overrides an attractive aggregate score.
• Regression-aware: the gate detects newly failing cases relative to the selected baseline.
• Evidence, not vibes: every metric links to the assertion, expected behavior, actual response, and failure explanation.
• Legal-specific attacks: citation fabrication, prompt injection in contract text, unsafe redlines, risk concealment, PII leakage, and legal-advice boundary tests.
• Versioned and reproducible: suite version, target provenance, policy snapshot, timings, and report are carried with each run.
• Keyless evaluation demo: the core hackathon experience has no dependency on Groq, Gemini, or another paid API.
• Contract target playground: the original document analyzer remains available as a secondary surface for testing real contract inputs.

Evaluation coverage

The bundled legal-release-gate-v1 suite contains 30 curated cases across:

Capability	What is measured
Clause recall and risk accuracy	Material clause detection and risk classification
Evidence faithfulness	Claims grounded in supplied contract context
Hallucination resistance	Refusal to invent statutes, facts, or unsupported authority
Citation validity	Citation presence, provenance, and non-fabrication
Numeric fidelity	Dates, money, percentages, and liability limits
Contradiction detection	Conflicting terms across the same agreement
Redline safety	Balanced changes that preserve rights and review boundaries
Adversarial robustness	Prompt injection, jailbreaks, concealed-risk requests, and role misuse
Privacy	PII masking and secret non-disclosure
Long context and multilingual	Retrieval across lengthy and non-English contract material
Legal boundary behavior	Calibrated uncertainty and appropriate professional-review guidance

Metrics use weighted applicable assertions, so missing or inapplicable data is reported as N/A rather than silently converted into a perfect score.

Release policy

The default gate blocks when any of these conditions is true:

• a critical case or critical assertion fails;
• a previously passing case regresses;
• overall score falls below the configured minimum;
• hallucination rate exceeds the configured maximum;
• a named metric misses its threshold; or
• the evaluation is incomplete.

The decision includes offending case IDs, new failure IDs, metric deltas, human-readable reasons, and the exact policy snapshot used.

Architecture

flowchart LR
    UI[Next.js command center] --> API[FastAPI evaluation API]
    API --> FIX[Versioned suite + target fixtures]
    FIX --> RUN[Deterministic runner]
    RUN --> GRADE[Typed assertions + metrics]
    GRADE --> COMPARE[Baseline comparison]
    COMPARE --> GATE[Safety-first release gate]
    GATE --> REPORT[Markdown + JSON evidence]
    REPORT --> UI
    DOC[Contract target playground] --> PIPE[Provider-optional analysis pipeline]
    PIPE --> UI

Loading

Core evaluation code lives in backend/evals; the judge-facing experience lives in frontend/components/evals.

API

Method	Endpoint	Purpose
GET	/api/evals/suites	List versioned benchmark suites
GET	/api/evals/suites/{suite_id}	Inspect a suite and its policy
GET	/api/evals/targets	List baseline and candidate targets
POST	/api/evals/run	Run, compare, gate, and persist a result
GET	/api/evals/runs	List in-memory demo runs
GET	/api/evals/runs/{run_id}	Inspect one comparison
GET	/api/evals/runs/{run_id}/report.md	Download human-readable evidence
GET	/api/evals/runs/{run_id}/report.json	Download machine-readable evidence
POST	/api/auth/demo	Create a credential-free evaluator session

Interactive OpenAPI documentation is available at /docs when the backend is running.

Run locally

Prerequisites: Python 3.11 or 3.12 and Node.js 20+. The deployment image is pinned to Python 3.11; Python 3.13+ is not supported by the pinned PDF stack.

# Terminal 1 — from the repository root
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r backend/requirements.txt -r backend/requirements-dev.txt
python -m uvicorn backend.main:app --reload --host 127.0.0.1 --port 8000

# Terminal 2
cd frontend
npm ci
$env:NEXT_PUBLIC_API_URL='http://127.0.0.1:8000'
npm run dev

Open http://localhost:3000/login and click Enter evaluator demo.

Groq and Gemini keys are optional. They are used only by the secondary live contract-analysis playground; the evaluation and release-gate demo works without them.

Verification

# Backend: evaluation, policy, report, API, auth, and analyzer tests
python -m pytest -q

# Frontend: unit, static, production, and real-browser checks
cd frontend
npm test
npm run lint
npm run typecheck
npm run build
npm run test:e2e
npm audit --omit=dev

The browser suite validates desktop and mobile demo entry, release blocking, regression evidence, and full-run navigation. Production dependencies currently audit with zero known vulnerabilities.

Deploy

• Backend: deploy the repository with render.yaml, or build backend/Dockerfile.
• Frontend: import the repository into Vercel with frontend as the root directory.
• Set NEXT_PUBLIC_API_URL in Vercel to the deployed backend URL.
• Set CORS_ORIGINS in Render to a JSON list containing the deployed frontend URL.
• SECRET_KEY must be generated in production. Provider keys remain optional for the core demo.

See DEPLOYMENT.md for the exact redeploy checklist.

Built with Codex

Codex was used as an engineering partner across the full repository: extracting the hackathon rubric from the supplied PDFs, auditing every route and legacy artifact, writing the evaluation design and implementation plan, developing the backend with red-green tests, generating and implementing the visual concepts, tracing browser-level auth failures, hardening dependencies, and producing the submission documentation.

The repository preserves a reviewable development record in docs/ENGINEERING_LOG.md, providing evidence of the process instead of a post-hoc claim.

Honest limitations

• The bundled target responses are deterministic fixtures for a reproducible, keyless demo; they are not claims about a named production model.
• Passing this benchmark does not prove universal legal correctness or replace qualified counsel.
• Demo runs use an in-memory bounded store and reset when the backend restarts.
• The 30-case suite is a strong release smoke test, not a substitute for organization-specific data, jurisdictions, and human review.

Responsible-use notice

LexGuard is an evaluation and decision-support system, not a law firm and not a source of legal advice. Its release verdict reports measured behavior under a defined policy; it does not certify a model as safe for every legal use.

---

Submission copy, the three-minute demo script, and the final checklist are in docs/HACKATHON_SUBMISSION.md.