"./x.py test --stage 0 --doc" does not run doc tests

[RalfJung]

I would expect the following command to run the libcore doc tests:

./x.py test --stage 0 src/libcore --doc

This used to work, but now all it does it build libcore and its test binaries. No tests are actually being run (which I verified by breaking one of them deliberately).

@Mark-Simulacrum any idea what is happening?

[ehuss]

I can confirm, doctests are not being run for std. I think a bigger question is, have they ever worked? Cargo will skip doctests whenever --target is used, which AFAIK is how std is always built/tested.

[RalfJung]

I am pretty sure that have used this command before and got test failures when I had screwed up. If you want, I can go back to older versions and verify.

[ehuss]

I'll try to figure out what's going on.

I tried running with -Zdoctest-xcompile and fortunately nothing looks broken.

It's also very possible I am missing something, but I edited some doctests with errors, and a bare ./x.py test doesn't fail.

[ehuss]

I see what's going on. It's a consequence of rust-lang/cargo#6892. That means doctests haven't been running since 1.39. This was wrong, see below.

I'll try to put together a fix.

[RalfJung]

That means doctests haven't been running since 1.39.

Ouch. Is there a way we can ensure this does not happen again in the future? Looks like our test suite needs a test suite...

[Mark-Simulacrum]

It's pretty hard :/

I guess we can add a dummy doc test that writes some "well known" file and we can assert that the file is present -- perhaps do this per-crate.

[ehuss]

OK, it wasn't as bad as I originally reported. This regressed via rust-lang/cargo#8167 which is only in 1.45 (beta). I have a fix started at rust-lang/cargo#8358. This will require various steps:

1. Fixing cargo.
2. Backport to beta.
3. Update cargo on beta.
4. Wait for new beta release.
5. Update bootstrap on nightly.

[RalfJung]

Well, but it does mean that currently on CI we do not run doctests, right? --stage 1 does not change this.

[ehuss]

Right, I just mean it hasn't been that long (only about a week).