Skip to content

test: proof infrastructure — snapshot corpus, diff manifest tool, benchmarks, CI - #47

Merged
kamronbatman merged 5 commits into
mainfrom
kb/test_infrastructure
Aug 22, 2026
Merged

test: proof infrastructure — snapshot corpus, diff manifest tool, benchmarks, CI#47
kamronbatman merged 5 commits into
mainfrom
kb/test_infrastructure

Conversation

@kamronbatman

Copy link
Copy Markdown
Member

Summary

Groundwork for the upcoming incremental-pipeline and correctness work: before changing the generator, this PR establishes the machinery that proves such changes are output-identical and actually faster. No generator behavior changes (one tool fix aside).

What's included

  • Snapshot corpus (Snapshots/, 15 fixtures): pins exact generated output byte-for-byte across the feature surface — fields, UO types, collections, sorted-set comparers, enums, save flags (small/ulong/multi-enum), versioned migrations, inheritance, structs, generics, readonly fields, timer + dirty tracking, field modifiers. Output must also compile against Server stubs aligned with the real interfaces. UPDATE_SNAPSHOTS=1 regenerates; the generator version stamp and line endings are normalized so version bumps and autocrlf don't invalidate the corpus.
  • DiffTool: runs the generator over every project in a solution (feeding migration files, unlike the schema pass) and writes a deterministic manifest of hint names + content hashes. Diffing manifests from two generator builds proves output-identical refactors against a real corpus: a 3,786-class solution hashes in ~14s, two consecutive runs are byte-identical, zero generator diagnostics.
  • Benchmarks: cold full run vs warm re-run after a single-file edit over a 150-class corpus. Baseline on the current pipeline: the warm re-run costs more than a cold full run (29.2ms / 14.8MB allocated vs 19.3ms / 10.3MB) — every class regenerates and re-parses its migrations on every compilation, plus incremental bookkeeping on top.
  • CI: a build-and-test workflow on push/PR — the repo previously had only the manual NuGet publish workflow, so nothing enforced the test suite.
  • Fix: SourceCodeAnalysis now accepts .slnx solutions; it refused them outright, which broke the schema tool against solutions using the new format.

Latent defects surfaced by the compile gate (left for the correctness-sweep PR)

  • An instance MarkDirty() on the serialized class emits ClassName.MarkDirty()() — fully-qualified display string plus appended parens; never compiles.
  • KeyValuePair direct fields emit a != comparison, a .Clear() call, and undeclared key/value locals.
  • Struct Serialize/Deserialize are emitted virtual (CS0106) — pinned via the Structs fixture's KnownBroken.txt marker.
  • Emitted code references Server-namespace types unqualified (Core, IGenericReader in migration content structs), so it only compiles for classes declared under Server.*; the fixtures mirror that constraint.

Test suite: 79/79 green (64 existing + 15 snapshot fixtures).

🤖 Generated with Claude Code

kamronbatman and others added 5 commits August 22, 2026 13:54
Fifteen fixtures under Snapshots/ cover fields, UO types, collections,
sorted-set comparers, enums, save flags (small, ulong, multi-enum),
versioned migrations, inheritance, structs, generics, readonly fields,
timer/dirty-tracking, and field modifiers. Each fixture's generated output
is pinned byte for byte (generator version normalized) and must compile
against Server stubs aligned with the real interfaces.

The compile gate surfaced latent emission defects, kept visible for the
correctness sweep:
- an instance MarkDirty() method on the serialized class emits
  ClassName.MarkDirty()() via ToDisplayString plus appended parens
- KeyValuePair fields emit a != comparison, a Clear() call, and undeclared
  key/value locals
- struct Serialize/Deserialize are emitted as virtual (CS0106); pinned via
  the Structs fixture's KnownBroken marker
- emitted code references Server-namespace types unqualified (Core,
  IGenericReader in migration content structs), so it only compiles for
  classes declared under Server.*; fixtures mirror that

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DiffTool runs the generator over every project in a solution - migration
files included, unlike the schema pass - and writes a deterministic
manifest of hint names and content hashes (generator version and line
endings normalized). Diffing manifests from two generator builds proves a
change is output-identical across a real corpus: a 3,786-class corpus
hashes in ~14s and two consecutive runs produce byte-identical manifests.

SourceCodeAnalysis now accepts .slnx, which newer solutions use; the
schema tool previously refused them outright.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Synthetic corpus of 150 versioned classes with migration files. Baseline
on the current pipeline: a warm re-run after editing a single file costs
more than a cold full run (29.2ms / 14.8MB vs 19.3ms / 10.3MB) because
every class regenerates and re-parses its migrations on every compilation,
plus incremental bookkeeping on top.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The only workflow was the manual NuGet publish; nothing enforced the test
suite on PRs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Autocrlf checkouts rewrite the Expected files with CRLF; comparing
normalized text keeps the corpus green on any checkout style.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kamronbatman
kamronbatman merged commit 186eace into main Aug 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant