Skip to content

x doc std unconditionally recompiles the standard library each time #103785

Closed
#104286
Closed
x doc std unconditionally recompiles the standard library each time#103785
#104286
Assignees
onur-ozkan
Labels
C-bugCategory: This is a bug.E-mediumCall for participation: Medium difficulty. Experience needed to fix: Intermediate.E-mentorCall for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Description

@jyn514
jyn514
opened on Oct 31, 2022

I tried this:

x doc --stage 0 std
x doc --stage 0 std

I expected to see this happen: The second invocation takes essentially no time.

Instead, this happened: The docs were completely rebuilt.

The problem is that we remove the docs between each invocation:

rust/src/bootstrap/doc.rs

Lines 613 to 615 in c2b0cc1

if out_dir.exists() {
builder.remove_dir(&out_dir);
}

@LukeMathWalker this is a regression from #101799 - do you think you'll have time to follow up? I think we can either add a stamp file to avoid rebuilding, or find some other way to avoid removing the whole directory - maybe we can only remove/copy the .json files, rather than everything?

Meta

HEAD is branched from 77e57db.

Activity

  1. jyn514
    added
    T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
    C-bugCategory: This is a bug.
    on Oct 31, 2022

  2. jyn514 commented on Nov 5, 2022

    @jyn514
    jyn514
    on Nov 5, 2022
    MemberAuthor

    Ok, I confirmed that rustdoc only creates JSON files with --output-format json:

    ; RUSTDOCFLAGS='--output-format=json -Zunstable-options' cargo doc
 Documenting inner v0.1.0 (/home/jnelson/rust-lang/test-rust)
    Finished dev [unoptimized + debuginfo] target(s) in 0.50s
; ls target/doc
inner.json

    So it should be enough to only copy .json files when passed --json, and only non-json files when it's not passed.

    We should also add a test that x doc && x doc --json produces the same output in json-doc as x doc --json, to avoid a regression if rustdoc starts emitting JSON files in the default HTML format.

  3. jyn514
    added
    E-mentorCall for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.
    E-mediumCall for participation: Medium difficulty. Experience needed to fix: Intermediate.
    on Nov 5, 2022

  4. LukeMathWalker commented on Nov 6, 2022

    @LukeMathWalker
    LukeMathWalker
    on Nov 6, 2022
    Contributor

    I can't jump on it immediately, but I should be able to follow up in a couple of weeks 👍

  5. onur-ozkan commented on Nov 7, 2022

    @onur-ozkan
    onur-ozkan
    on Nov 7, 2022
    Contributor

    @rustbot claim

  6. rustbot
    assigned
    onur-ozkan
    on Nov 7, 2022

  7. onur-ozkan commented on Nov 8, 2022

    @onur-ozkan
    onur-ozkan
    on Nov 8, 2022
    Contributor

    We should also add a test that x doc && x doc --json produces the same output in json-doc as x doc --json, to avoid a regression if rustdoc starts emitting JSON files in the default HTML format.

    We can do this validation for doc dir, but I am not sure if this is possible for json-doc dir since each time json output changes. The layout of json context is getting change even if you don't change anyhing on the source code.

    I print sha2 sums below to demonstrate the situation.

    1st attempt:

    ~/devspace/personal/rust-dist/library-master  $ ../../rust/x doc --stage 0 std --json && sha256sum build/x86_64-unknown-linux-gnu/json-doc/*
Building rustbuild
    Finished dev [unoptimized] target(s) in 0.06s
Documenting stage0 std (x86_64-unknown-linux-gnu) in JSON format
 Documenting core v0.0.0 (/home/nimda/devspace/personal/rust/library/core)
    Finished release [optimized] target(s) in 7.34s
   Compiling compiler_builtins v0.1.82
 Documenting alloc v0.0.0 (/home/nimda/devspace/personal/rust/library/alloc)
    Finished release [optimized] target(s) in 3.14s
   Compiling compiler_builtins v0.1.82
   Compiling memchr v2.5.0
   Compiling unwind v0.0.0 (/home/nimda/devspace/personal/rust/library/unwind)
   Compiling std v0.0.0 (/home/nimda/devspace/personal/rust/library/std)
    Checking alloc v0.0.0 (/home/nimda/devspace/personal/rust/library/alloc)
    Checking cfg-if v1.0.0
    Checking adler v0.2.3
    Checking rustc-demangle v0.1.21
    Checking rustc-std-workspace-alloc v1.99.0 (/home/nimda/devspace/personal/rust/library/rustc-std-workspace-alloc)
    Checking panic_abort v0.0.0 (/home/nimda/devspace/personal/rust/library/panic_abort)
    Checking panic_unwind v0.0.0 (/home/nimda/devspace/personal/rust/library/panic_unwind)
    Checking gimli v0.25.0
    Checking hashbrown v0.12.3
    Checking miniz_oxide v0.4.0
    Checking std_detect v0.1.5 (/home/nimda/devspace/personal/rust/library/stdarch/crates/std_detect)
    Checking object v0.26.2
    Checking addr2line v0.16.0
 Documenting std v0.0.0 (/home/nimda/devspace/personal/rust/library/std)
    Finished release [optimized] target(s) in 7.13s
Build completed successfully in 0:00:18
b4b090b23a1ca1d7b870fbf2c9d3827ffe9aa7b2fdb17faf34e7a64852c141c9  build/x86_64-unknown-linux-gnu/json-doc/alloc.json
83db7b1380a39c886640e0da0fd2590264cc27ca40ec9bdda8e0b4c02fcc3586  build/x86_64-unknown-linux-gnu/json-doc/core.json
e789adebf4ca15850e9c0b3cd3c9126d48cd7ea9e1ae272339e692423009f353  build/x86_64-unknown-linux-gnu/json-doc/std.json

    2nd attempt right after the first attempt

    ~/devspace/personal/rust-dist/library-master  $ ../../rust/x doc --stage 0 std --json && sha256sum build/x86_64-unknown-linux-gnu/json-doc/*
Building rustbuild
    Finished dev [unoptimized] target(s) in 0.04s
Documenting stage0 std (x86_64-unknown-linux-gnu) in JSON format
 Documenting core v0.0.0 (/home/nimda/devspace/personal/rust/library/core)
    Finished release [optimized] target(s) in 7.20s
   Compiling compiler_builtins v0.1.82
 Documenting alloc v0.0.0 (/home/nimda/devspace/personal/rust/library/alloc)
    Finished release [optimized] target(s) in 3.17s
   Compiling compiler_builtins v0.1.82
   Compiling memchr v2.5.0
   Compiling unwind v0.0.0 (/home/nimda/devspace/personal/rust/library/unwind)
   Compiling std v0.0.0 (/home/nimda/devspace/personal/rust/library/std)
    Checking alloc v0.0.0 (/home/nimda/devspace/personal/rust/library/alloc)
    Checking cfg-if v1.0.0
    Checking adler v0.2.3
    Checking rustc-demangle v0.1.21
    Checking rustc-std-workspace-alloc v1.99.0 (/home/nimda/devspace/personal/rust/library/rustc-std-workspace-alloc)
    Checking panic_abort v0.0.0 (/home/nimda/devspace/personal/rust/library/panic_abort)
    Checking panic_unwind v0.0.0 (/home/nimda/devspace/personal/rust/library/panic_unwind)
    Checking gimli v0.25.0
    Checking std_detect v0.1.5 (/home/nimda/devspace/personal/rust/library/stdarch/crates/std_detect)
    Checking object v0.26.2
    Checking miniz_oxide v0.4.0
    Checking hashbrown v0.12.3
    Checking addr2line v0.16.0
 Documenting std v0.0.0 (/home/nimda/devspace/personal/rust/library/std)
    Finished release [optimized] target(s) in 6.77s
Build completed successfully in 0:00:17
07d31750fe73615c800498041adf20e485b86ae6b15860f9fea6c3b0cedcf884  build/x86_64-unknown-linux-gnu/json-doc/alloc.json
c232267e96f186971622d46a55b0471ccef78dba85fe9ad47a43392629b9c5c1  build/x86_64-unknown-linux-gnu/json-doc/core.json
81b83dd44026f5a3705f48e8be5f0f04ceb7ea14e566bd46939882f11868efc7  build/x86_64-unknown-linux-gnu/json-doc/std.json

    The demonstration is from 391ba78

  8. jyn514 commented on Nov 8, 2022

    @jyn514
    jyn514
    on Nov 8, 2022
    MemberAuthor

    @ozkanonur thanks for investigating; sounds like we should report that as a separate bug in rustdoc. But for now it's ok if the files don't match exactly as long as the file names are the same between runs.

  9. Dylan-DPC
    added a commit that references this issue on Nov 23, 2022

    Rollup merge of rust-lang#104286 - ozkanonur:fix-doc-bootstrap-recomp…

    d3e9191
  10. bors
    closed this as completedin d3e9191on Nov 23, 2022
  11. matthiaskrgr
    added 2 commits that reference this issue on Mar 4, 2023

    Rollup merge of rust-lang#108626 - ozkanonur:consistent-json-docs, r=…

    b850cee

    Rollup merge of rust-lang#108626 - ozkanonur:consistent-json-docs, r=…

    03c1e4d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

C-bugCategory: This is a bug.E-mediumCall for participation: Medium difficulty. Experience needed to fix: Intermediate.E-mentorCall for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions