Repository navigation
who tests the tester? #47606
Description
Activity
I have some beef with compiletest in general. Instead of having this unstable (as in nightly) library that contains a somewhat random list of test kinds that all behave subtly different, we should probably write a stable extensible library that is distributed via crates.io, developed like any other library (so it has tests) and uses cargo instead of manually calling rustc and hoping for the best.
So, the list of issues off the top of my head:
- nightly only
- rustc submodules need to use the crates.io version, which breaks with rustc changes
- using dependencies in tests frequently causes "multiple matching crates" because of old compilation artifacts in the target direcory
- not extensible with new test kinds (cc @killercup for suggestion tests)
Reacted by Andy Russell and Dylan DPCI'd be happy to work on a library like this if we can have a conversation about what is exactly needed :).
@spastorino I started working on a very simple version of this to test clippy's diagnostics output as well as that its suggestions work as intended (when applied by rustfix). I'll have something up in later today, and will ping you in the clippy PR :)
In the meantime, feel free to say hello in the #clippy channel on irc.mozilla.org :)
I have some beef with compiletest in general.
While I don't disagree, I feel a bit wary here. I'm fine with making a new testing setup, but it feels like the kind of thing that will derail progress for a long time. And I'm not convinced that the compiler's needs (e.g., things like mir-opt tests) will be satisfied by a more generic setup.
Unless you have a clear candidate of what test runner we should adopt, I'd rather see us move towards simplifying and centralizing things gradually and then at some later point perhaps switch. (Centralizing run-pass, compile-fail, and ui seems like an example of something we can do to simplify things now.)
Reacted by Isaac Woods and Mark RousskovTriage; Looks like Pietro has added a bit of testing, but it's "just a start"
One way to test compiletest is to add more UI tests that test compiletests' behavior, instead of rustc's. See for example https://github.com/rust-lang/rust/pull/103298/files#diff-6542cbbb7ad12df6cb2daadaf279589c5c1712906cd14222c03132ad5a49d590. I think using that instead of a new framework has several advantages:
- No need to set up a new testing framework
- Existing contributors are already familiar with UI tests
- These double as "integration tests" since they test exactly the property we're interested in, how compiletest runs a UI test from beginning to end
I think it should be fairly simple to keep adding tests to
src/test/ui/compiletest-selftestas necessary; I don't have a set of initial tests in mind though.I wrote a new ui test framework that miri uses and I'm working on getting ready for clippy to use. It uses itself to test its own output.
It could well be, mainly I went forward with #103266 to avoid the other duplicate work of adding things to https://github.com/Manishearth/compiletest-rs that the in tree one already has. For example the JSON output after every test failure is pretty rough
- moved this to Backlog in compiletest maintenance and improvements
on Jun 16, 2024 - added and removed
on Oct 17, 2024 I'm going to close this issue because it's quite dated, though the "who tests the tester" challenge very much is still a problem -- and will continue to be so due to the nature of the things compiletest has to support. Some remarks:
- We're no longer planning to move compiletest out-of-tree. compiletest's core mission is to support testing of rust-lang/rust things, specifically the compiler, standard library, and rustdoc, and a bit of integration testing with cargo. The upside is because compiletest gets to make simplifying / reasonable assumptions that make sense in the context of rust-lang/rust. The tradeoff is compiletest is no longer a general-purpose tool -- and I would strongly prefer to keep it that way to avoid having to handle additional maintenance burden with dealing with out-of-tree consumers.
- We do have some light self-test coverage now, for better or worse. Directive parsing and such have in-tool integration tests, some compiletest behaviors have integration tests through ui tests. Though overall, a lot more self-test coverage is what I want compiletest to have. However, compiletest is currently structured in a way that makes more comprehensive self-testing quite difficult, and so my plan is to gradually refactor compiletest to make it more testable.
While I don't disagree, I feel a bit wary here. I'm fine with making a new testing setup, but it feels like the kind of thing that will derail progress for a long time. And I'm not convinced that the compiler's needs (e.g., things like mir-opt tests) will be satisfied by a more generic setup.
I am very much in agreement with this assessment, there's a lot of complexity in compiletest, especially in cases where compiletest and the $thing_under_test needs to collaborate to be able to effectively test something.
So, by and large, compiletest is untested, as far as I know. That is -- there are no 'self tests' to make sure that it's acting as it should, invoking revisions the way we expect, and so forth. This could be a bit of a tricky thing to do, but it's definitely worth the effort.
For example, an early version of #47605 had a subtle bug where it appeared to be working, but in fact was not passing the revision info through to the final test. I don't think this would have bothered travis one bit, as that would have happily run the same revision over and over.
cc @spastorino @oli-obk ... we need like a compiletest team, don't why? :)