test: proof infrastructure — snapshot corpus, diff manifest tool, benchmarks, CI - #47
Merged
Conversation
Fifteen fixtures under Snapshots/ cover fields, UO types, collections, sorted-set comparers, enums, save flags (small, ulong, multi-enum), versioned migrations, inheritance, structs, generics, readonly fields, timer/dirty-tracking, and field modifiers. Each fixture's generated output is pinned byte for byte (generator version normalized) and must compile against Server stubs aligned with the real interfaces. The compile gate surfaced latent emission defects, kept visible for the correctness sweep: - an instance MarkDirty() method on the serialized class emits ClassName.MarkDirty()() via ToDisplayString plus appended parens - KeyValuePair fields emit a != comparison, a Clear() call, and undeclared key/value locals - struct Serialize/Deserialize are emitted as virtual (CS0106); pinned via the Structs fixture's KnownBroken marker - emitted code references Server-namespace types unqualified (Core, IGenericReader in migration content structs), so it only compiles for classes declared under Server.*; fixtures mirror that Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DiffTool runs the generator over every project in a solution - migration files included, unlike the schema pass - and writes a deterministic manifest of hint names and content hashes (generator version and line endings normalized). Diffing manifests from two generator builds proves a change is output-identical across a real corpus: a 3,786-class corpus hashes in ~14s and two consecutive runs produce byte-identical manifests. SourceCodeAnalysis now accepts .slnx, which newer solutions use; the schema tool previously refused them outright. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Synthetic corpus of 150 versioned classes with migration files. Baseline on the current pipeline: a warm re-run after editing a single file costs more than a cold full run (29.2ms / 14.8MB vs 19.3ms / 10.3MB) because every class regenerates and re-parses its migrations on every compilation, plus incremental bookkeeping on top. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The only workflow was the manual NuGet publish; nothing enforced the test suite on PRs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Autocrlf checkouts rewrite the Expected files with CRLF; comparing normalized text keeps the corpus green on any checkout style. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Groundwork for the upcoming incremental-pipeline and correctness work: before changing the generator, this PR establishes the machinery that proves such changes are output-identical and actually faster. No generator behavior changes (one tool fix aside).
What's included
Snapshots/, 15 fixtures): pins exact generated output byte-for-byte across the feature surface — fields, UO types, collections, sorted-set comparers, enums, save flags (small/ulong/multi-enum), versioned migrations, inheritance, structs, generics, readonly fields, timer + dirty tracking, field modifiers. Output must also compile against Server stubs aligned with the real interfaces.UPDATE_SNAPSHOTS=1regenerates; the generator version stamp and line endings are normalized so version bumps and autocrlf don't invalidate the corpus.SourceCodeAnalysisnow accepts.slnxsolutions; it refused them outright, which broke the schema tool against solutions using the new format.Latent defects surfaced by the compile gate (left for the correctness-sweep PR)
MarkDirty()on the serialized class emitsClassName.MarkDirty()()— fully-qualified display string plus appended parens; never compiles.KeyValuePairdirect fields emit a!=comparison, a.Clear()call, and undeclaredkey/valuelocals.Serialize/Deserializeare emittedvirtual(CS0106) — pinned via the Structs fixture'sKnownBroken.txtmarker.Core,IGenericReaderin migration content structs), so it only compiles for classes declared underServer.*; the fixtures mirror that constraint.Test suite: 79/79 green (64 existing + 15 snapshot fixtures).
🤖 Generated with Claude Code