Repository navigation
1.100 beta crater regression: "No space left on device" and massive slowdown during doctests #164145
Description
Activity
- added
on Oct 11, 2026 - added a parent issue
on Oct 11, 2026 Some ongoing discussion at #general > How to triage a massive test-running perf regression?
Bisected with
cargo bisect-rustc --start=1.99.0 --end=beta --timeout 50 -- test:******************************************************************************** Regression in 5392d2f545c6836dc79f209bcc14ac7179dbc4f2 ******************************************************************************** Attempting to search unrolled perf builds Found commits ["0e21efee", "0aebea15", "770a9cd8", "449ef23a", "a95bb512", "5fd80363", "a49eccce", "5ba40ad4", "2983377a", "033e87fb", "3b668784", "16c23a49"] installing 0e21efee1ad09fb750d7925735bc0495cc10e971 rust-std-nightly-x86_64-unknown-linux-gnu: 31.40 MB / 31.40 MB [=============================================================================] 100.00 % 11.56 MB/s testing... RESULT: 0e21efee1ad09fb750d7925735bc0495cc10e971, ===> Successfully compiled uninstalling 0e21efee1ad09fb750d7925735bc0495cc10e971 installing 0aebea155e1762d121a36050809579171759e979 rust-std-nightly-x86_64-unknown-linux-gnu: 31.38 MB / 31.38 MB [=============================================================================] 100.00 % 11.47 MB/s testing... RESULT: 0aebea155e1762d121a36050809579171759e979, ===> Compile error uninstalling 0aebea155e1762d121a36050809579171759e979 Regression in https://github.com/rust-lang/rust/commit/0aebea155e1762d121a36050809579171759e979. Note that if it is a legacy rollup build, it might be available in https://github.com/rust-lang-ci/rust/commit/0aebea155e1762d121a36050809579171759e979. The PR introducing the regression in this rollup is #161868: libtest: never iterate over all tests in `--exact` modeRegressed in #161868. cc @RalfJung (author), @Mark-Simulacrum (reviewer)
- added and removed
on Oct 11, 2026 This is very strange. That PR added one more big allocation in doctests, but that should not make such a huge difference. And how could it possibly cause the file system to fill up...?
You marked this as both "doctests" and "libtest". Since you apparently can reproduce this locally, can you figure out which of them it is?
cargo test --testsonly runs regular tests,cargo test --doconly runs doctests.Also apparently this crate is basically identical to
fixed. Did that one not regress?And if you have the output from running the tests anywhere, that would be useful. How many tests are there?
I can reproduce the time usage slowdown with
cargo +beta test --doc. (Note that I have not yet figured out how to measure the differences in disk usage at any point. The only evidence of disk usage increase was from the "No space left on device" errors on crater.)fixed-1.31.0appeared to test fine on crater. Not sure if the version difference (1.29.1vs1.31.0) matters.The outputs for running the tests can be seen in the "crater run 1/2/3/4" links in the OP issue. If you want the outputs on 1.99, here they are: 1, 2, 3, 4. There are 3763 tests (in the last (and biggest) batch being ran, presumably the doctests batch.)
I can reproduce the time usage slowdown with
fixed-1.29.0. I am unable to reproduce it onfixed-1.30.0orfixed-1.31.0. (Thefixedcrate does not have a1.29.1version. Thefixed_star_frame-1.29.1crate is a lightly modified version offixed-1.29.0.)So, some change between
fixed-1.29.0andfixed-1.30.0got rid of the part that's slow.I guess it's possible that the time usage regression and the disk usage regression are two unrelated things? That seems unlikely though.
So there are 3763 doctests. Not nothing, but also not a gigantic number.
So, some change between `fixed-1.29.0` and `fixed-1.30.0` got rid of the part that's slow.There's only about 10 commits between those versions, so you could bisect the crate itself. The only significant changes seem to be formatting, a format bug fix, switching the internals of serde/arbitrary impls, and a "no mut statics" feature.
Did the edition change? rustdoc behaves very different on newer edition with "merged doctests".
Or maybe my PR somehow broke merged doctests for that crate and we are falling back to individual tests and that's why it got so much slower.@rust-lang/rustdoc is there some way to get rustdoc to print which doctests were merged vs split?
- changed the title
[-]1.100 beta crater regression: "No space left on device"[/-][+]1.100 beta crater regression: "No space left on device" and massive slowdown during doctests[/+]on Oct 11, 2026
This regression was found in the 1.100 beta crater run.
The
fixed_star_frame-1.29.1consistently has its tests pass in crater on stable rust. However, it consistently fails in crater on beta rust.It appears to me that we're doing something much less efficiently on beta than on stable, and using much more disk space.
I tried running the tests on the dev desktop (
dev-desktop-eu-2.infra.rust-lang.org).cargo +stable testused 0m32.266s to complete. However,cargo +beta testused 1m11.525s to complete. That is, running tests on beta is over twice as slow.