First-class Microsoft.Testing.Platform (MTP) support for `dotnet test`

[dennisdoomen]

Summary

.NET 10 dropped VSTest support for test projects that use the Microsoft.Testing.Platform (MTP) runner (xunit.v3, and MSTest/NUnit once they support MTP). Opting in via global.json's "test": {"runner": "Microsoft.Testing.Platform"} hits two bugs in Fallout and is missing a feature. Found while migrating a real repo (an Azure Functions service and its build-automation library, both xunit v2 → xunit.v3) on Fallout.Common/Fallout.Build 10.4.0.

1. FalloutBuild.GetBuildAssemblyFile() throws under MTP

src/Fallout.Build/FalloutBuild.Statics.cs checks the entry assembly's name against a hardcoded allowlist (testhost, ReSharperTestRunner) to recognize a test host, and asserts if it doesn't match. This runs in FalloutBuild's static constructor, so it fires the moment anything touches a static member of FalloutBuild — including FalloutBuild.RootDirectory, which some generated settings classes read in a field initializer.

Under classic VSTest, the entry assembly is testhost, which the allowlist covers. Under MTP, the test assembly itself is the entry assembly (MTP self-hosts it), so its name matches nothing on the list and the assert throws TypeInitializationException — from unit tests that never reference FalloutBuild directly, only a settings class that happens to depend on it.

Suggested fix: drop the name check. If no type in the entry assembly derives from FalloutBuild, that already proves it isn't a build assembly. The name check adds nothing and can never list every current and future test host.

2. No typed support for MTP's dotnet test CLI in DotNetTestSettings

MTP's dotnet test is a separate, non-overlapping CLI surface from classic VSTest. DotNetTestSettings only generates the VSTest surface — Loggers (--logger), RunSettings, BlameMode, DataCollector, TestAdapterPath — none of which exist under MTP. MTP instead has its own flags with no VSTest equivalent: --coverage / --coverage-output(-format) (from Microsoft.Testing.Extensions.CodeCoverage), --report-trx / --report-trx-filename (from Microsoft.Testing.Extensions.TrxReport), --minimum-expected-tests, --diagnostic-output-directory, --treenode-filter, --zero-tests-policy.

Suggested fix: a dedicated MTP settings class, generated from a new TestMtp.json (or an "mtp": true variant of the existing Test definition):

{
  "help": "Runs tests using the Microsoft.Testing.Platform (MTP) native runner, opted into via global.json's 'test.runner'. VSTest and MTP are separate, non-overlapping dotnet test CLI surfaces (see https://learn.microsoft.com/dotnet/core/tools/dotnet-test-mtp), so this is a distinct task from Test rather than an extension of it.",
  "postfix": "TestMtp",
  "commonPropertySets": ["restore", "restore-runtime"],
  "definiteArgument": "test",
  "settingsClass": {
    "properties": [
      { "name": "ProjectFile", "type": "string", "format": "--project {value}", "help": "Path to the test project. Mutually exclusive with SolutionFile and TestModules." },
      { "name": "SolutionFile", "type": "string", "format": "--solution {value}", "help": "Path to the test solution. Mutually exclusive with ProjectFile and TestModules." },
      { "name": "TestModules", "type": "string", "format": "--test-modules {value}", "help": "Glob selecting already-built test module assemblies. Mutually exclusive with ProjectFile and SolutionFile." },
      { "name": "Configuration", "type": "string", "format": "--configuration {value}" },
      { "name": "Framework", "type": "string", "format": "--framework {value}" },
      { "name": "Filter", "type": "string", "format": "--filter {value}" },
      { "name": "Verbosity", "type": "DotNetVerbosity", "format": "--verbosity {value}" },
      { "name": "NoBuild", "type": "bool", "format": "--no-build" },
      { "name": "NoRestore", "type": "bool", "format": "--no-restore" },
      { "name": "ResultsDirectory", "type": "string", "format": "--results-directory {value}" },
      { "name": "MinimumExpectedTests", "type": "int?", "format": "--minimum-expected-tests {value}" },
      { "name": "DiagnosticOutputDirectory", "type": "string", "format": "--diagnostic-output-directory {value}" },
      { "name": "TreenodeFilter", "type": "string", "format": "--treenode-filter {value}" },
      { "name": "ZeroTestsPolicy", "type": "ZeroTestsPolicy", "format": "--zero-tests-policy {value}" },
      { "name": "Coverage", "type": "bool", "format": "--coverage", "help": "Requires the Microsoft.Testing.Extensions.CodeCoverage package." },
      { "name": "CoverageOutputFormat", "type": "string", "format": "--coverage-output-format {value}" },
      { "name": "CoverageOutput", "type": "string", "format": "--coverage-output {value}" },
      { "name": "ReportTrx", "type": "bool", "format": "--report-trx", "help": "Requires the Microsoft.Testing.Extensions.TrxReport package." },
      { "name": "ReportTrxFileName", "type": "string", "format": "--report-trx-filename {value}" },
      { "name": "ExtensionArguments", "type": "List<string>", "format": "{value}", "prefix": "--", "position": -1, "help": "Raw arguments forwarded verbatim to registered MTP extensions, placed after a literal '--' as Microsoft's dotnet test docs recommend, to avoid argument-binding ambiguity with recognized options." }
    ]
  }
}

Plus the properties that already exist on both surfaces: Configuration, Framework, Filter, Verbosity, NoBuild, NoRestore, ResultsDirectory.

What this buys consumers

DotNetTestMtp(s => s
    .SetProjectFile(Solution.MyProject_Specs)
    .SetConfiguration(Configuration.Debug)
    .SetFilter("Category!=Integration")
    .EnableReportTrx()
    .SetReportTrxFileName(ArtifactsDirectory / "UnitTestResults.trx")
    .EnableCoverage()
    .SetCoverageOutputFormat("cobertura")
    .SetCoverageOutput(ArtifactsDirectory / "UnitTests.cobertura.xml"));

A typed, IntelliSense-able API matching the Set/Enable/Disable conventions the generator already produces for every other tool — instead of raw-string AddProcessAdditionalArguments("--project", ..., "--coverage", ...).

3. Safer forwarding of extension arguments

Microsoft's docs recommend putting MTP-extension arguments after a literal --, since a recognized option between an unrecognized option and its value can change how the leftover tokens bind. RunSettings already does something similar ("position": -1) but is VSTest-runsettings-specific. The generic ProcessAdditionalArguments escape hatch doesn't guarantee -- placement either, even though it's already appended last in every generated argument list.

"prefix": "--" in the JSON above is a new schema field for exactly this case: when set and the list is non-empty, ToolOptions.GetArgument should emit that one literal token before the per-value tokens, keeping each forwarded token its own argv entry. Two existing list mechanisms come close but aren't right: a Separator-joined list glues all values into one token (wrong — a forwarded path can contain spaces), and a plain repeated-format list (like Loggers) repeats the whole format, literal text included, once per value instead of once total.

4. PublishCodeCoverage silently accepts an unsupported glob (secondary, not MTP-specific)

AzurePipelines.Instance.PublishCodeCoverage(tool, summaryFile, reportDirectory) accepts a glob in summaryFile, but Azure Pipelines' ##vso[codecoverage.publish] command has never supported wildcards. It builds fine and fails only at pipeline run time, with a cryptic "file does not exist" error. A guard clause rejecting */? (or at least an XML-doc callout) would catch this at build time instead.

Notes

• See Support Microsoft.Testing.Platform — mirrors upstream #1584 #19 for the parallel investigation, including confirmation that SDK 10.0.302+ (this repo's pinned version) restores the positional project path under MTP.

[ITaluone]

No positional project path at all. --project, --solution, and --test-modules are mutually exclusive and are the only documented ways to select what to test.

Really? I converted a few of my projects to .NET10 + MTP v2 (xunit.v3 v4.0.0) and it works w/o setting --project or --solution
Since this PR here, I think they backported the "old" aka known behavior with positional arguments: dotnet/sdk#52636 / dotnet/sdk#52449

See my insights here as well: #19

[dennisdoomen]

Let me investigate. I suspect it was indeed caused by an older SDK version.

[dennisdoomen]

Confirmed, thanks for the pointer. dotnet/sdk#52449 restored the positional project path for MTP too, backported to the 10.0.3xx band via dotnet/sdk#52636. This repo already pins 10.0.302 in global.json, so the positional-path breakage isn't an issue here. #19 confirms the same fix independently.

Updated the issue body: dropped the "no positional path" claim from point 2, kept the rest (the MTP-only flags gap and the proposed TestMtp API) as is.

[dennisdoomen]

@ChrisonSimtian let me know what you think about this proposal