Add GHCP end-to-end evaluation harness

[George Ng (GeorgeNgMsft)]

Adds a reproducible Copilot end-to-end evaluation in the dedicated copilot-plugin-eval package, comparing TypeAgent routing strategies with native Copilot. This is a simple initial evaluation for ballpark estimates, not a statistically powered benchmark. The latest corpus replaces list-dependent tasks with ordinary file tasks that all candidates can perform.

• Keeps seven candidates and twenty examples in four five-case cohorts. Protocol 5 (common-files-v1) replaces nine list-dependent cases with file inventory, append, backup/overwrite, comparison, conditional edits and clarification tasks. All candidates have twenty applicable cases: 140 executions per repetition, no native N/A slots.
• Uses existing registered PowerShell file actions for TypeAgent and ordinary native edit/create tools for Copilot. Per-case fixture permissions, pre-clarification gates, exact backup prerequisites and independent file-state checks protect unrelated state. Arbitrary shell writes are not autoapproved; GitHub and network remain read-only.
• Creates fresh conversations, bindings and restored fixtures; contract reuse is earned through separate same-binding discovery and preparation is accounted separately. Preserves provenance-checked SDK overflow artifacts and one scripted final-text clarification within the original deadline.
• Pins future outer sessions, contract preparation and nested Copilot reasoning to Luna 5.6 (gpt-5.6-luna). Rejects mismatched ledgers and stale preflights; freezes fresh corpus/model/order specifications rather than relabeling historical runs.
• Records E2E timing, model/tool activity, actual internal failed-translation fallback and interaction evidence. Unknown timing stays unknown. Retains bounded credit admission, uncertain reservations and cumulative accounting; the ceiling is 50,000 cumulative credits, not a fresh allowance. External-provider accounting remains separate and incomplete.
• Moves eval scripts/tests and maintained methodology into ts/packages/copilot-plugin-eval; runtime guards remain at dispatcher enforcement boundaries. Explicitly documents residual host/plugin/environment-isolation risks: a separate directory is not a sandbox.
• Preserves strict no-replay behavior in this base layer. The dependent Refine prospective GHCP evaluation recovery, applicability and ambiguity case #3077 layer integrates positive-evidence safe-read recovery while retaining denial/cancellation/uncertainty precedence.

Historical evidence, not new file-corpus measurements

The completed 140-trial historical pass used protocol 2 at e847c7a00907a1f4e2c6c4426c935b964fd9997c with gpt-5.6-sol and the original list workload. All combinations were attempted; not all succeeded. Original strict successes were 6, 6, 11, 11, 6, 8, 3 /20 for C1-C7. The later protocol-3 native terminal-gate correction was tested offline, not live-measured.

Removing only the blanket continuation penalty produced retrospective content scores of 6, 6, 11, 11, 7, 8, 8 /20. This is not a new execution or certification of recovery safety: some native failure evidence is incomplete, and network final-presentation failures remain. Excluding nine list-dependent native cases gave a historical native 8/11, not a matched-workload ranking. Original success-conditioned timing populations and uncertainty remain separate; these observations establish neither a general winner nor an internal-fallback benefit.

No Luna or common-files-v1 live evaluation has run. New corpus outcomes must not be pooled with old results. Fresh readiness, model-specific accounting bounds and authorization remain prerequisites. The maintained methodology records the approved corpus amendment; historical reports, sanitized exports, original grades, frozen specifications and accounting artifacts remain external and unchanged. Private network evidence, credentials, billing traces and machine-specific paths are not committed.

Latest offline validation

• Dependency-inclusive evaluation-package build passed; final dispatcher build passed.
• 23 eval tests passed, covering all20/all7 scheduling, stale readiness, file/backup oracles, clarification, scoped confirmations, link rejection, Luna admission and relocated entry points.
• 174 relevant dispatcher/structured/permission tests passed for the corpus commit; the final directory-read clarification guard reran all 17 affected policy/artifact tests successfully.
• Final-head formatting and all four ratchets passed against origin/main: no added lint/complexity violations, cycles unchanged at 220, no focused/newly skipped tests.
• Earlier offline artifact validation covered all 140 historical conversations/reviews, passing fixture/clarification/route checks, native post-failure exclusions, network-export sanitization, historical preservation and cumulative accounting. This is historical validation, not a run of the new corpus.
• Published head: f2f045c2066605f69c990f3b99bfd38cdb257bfe. Hosted CI on this new head is not claimed complete. No shared registrations, main checkout, Azure identities or stack metadata were changed.

[Dominic Nguyen (datduyng)]

The grader has too much regex more than what. Iwould like to see but this is a good start. please work on cleaning up