You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Host --output-format json files in addition to generated HTML #1285
There has been some very brief discussion of docs.rs generating and hosting these files along with the html. But at the moment I think it's quite buggy (e.g. rust-lang/rust#80664 is probably going to fail a large proportion of builds) and the format is unstable. I think it would make sense to wait a little longer till it has stabilized a bit more before we add it here (I know of at least a couple of users that are already building alternative renderers based on the json, which will hopefully drive it towards stabilization).
Reacted by jyn, Alona Enraght-Moony, Denis Cornehl and bbb651 🇮🇱
I don't think we should start hosting this until the JSON backend is stabilized. The output format is likely to change several more times before stabilization.
yeah, also valid point, I only had in mind that we could find more ICEs when running it for docs, but the format changing could be annoying for users of the JSON
we're planning on building the JSON with the same docs.rs metadata (= the same feature set) as the HTML docs. ( this seems to be OK for cargo-semver-checks too @obi1kenobi )
the download will be a single JSON download via CDN, probably gz-compressed.
there are some missing features before we deem the format stable enough: ( coming from @jyn514 )
next set of notes coming from the all-hands / rustweek: we can actually start building & hosting rustdoc json.
about the implementation:
memory usage of the JSON build is higher than the HTML build. We might hit limits for crates that reach that limit.
Files would be compressed on S3, potentially with zstd
download API is a redirect to static.docs.rs for download, similar to the archive download.
format_version could be part of the filename / API.
question: can we keep hosting old format versions? and provide via API?
Normal rebuilds for the latest version of each crate would then also build & upload the rustdoc json. So after a couple of months of this being live, all latest versions have rustdoc json.
There are some pending additions that will come later that will make it easier to link between json docs easier (notably: extend the information we provide in external_crates)
edit: don't forget, we have a separate JSON output per target to store.
rustdoc will clear the /target/doc/ directory before the HTML or JSON builds. So we have to pull out the JSON output before we run the HTML build
we have to store & support multiple targets.
due to the potentially higher memory usage of rustdoc json, the JSON build might fail when HTML doesn't. We want HTML to be published even after a failed JSON build. Do we need to store separate logs for crate authors to see this somewhere? Or is our internal sentry / log good enough, similar to coverage-generate?
any API would probably redirect to default target if no target is given.
structure on S3 bucket is currently:
rustdoc: /rustdoc/{krate}/{version}.zip
sources: /sources/{krate}/{version}.zip
build logs: /build-logs/{build_id}/{target}.txt
and legacy locations before archive-storage
rustdoc: /rustdoc/{krate}/{version}/
sources: /sources/{krate}/{version}/
after each build we store the archive, and run delete_prefix on the legacy locations.
For rustdoc JSON we preferably want a folder structure where we can later s3 ls and return the available format-versions?
This means the best structure is probably a separate root: /rustdoc-json/{krate}/{version}/{target}/{format_version}.json.zst, perhaps duplicating krate & version & target on the filename for a nicer download. s3 ls could use all files from the target folder.
That also means every request to docs.rs for a rustdoc JSON file needs an s3 ls call. The only way to prevent that, so we can statically determine the path, would be if we remove the format version from the path, or additionally maintain a file without the format version that we overwrite each rebuild.
Broadly this all sounds good! Thanks for looking into it and writing it up, I really appreciate it.
If it avoids the s3 ls call, I'd be fine with also having an option to download the "latest" format version available using a non-format-versioned request path. This would likely be useful to cargo-semver-checks too: for example, say we're running a SemVer breakage study of crates.io crates. For each release, we'd look to download the latest rustdoc JSON format version available, together with the same format version for the immediately prior release. So long as both files are the same JSON format version, and the format is reasonably new (so cargo-semver-checks still supports it), everything would work well.
I like the idea of adding the parameters to the filename, for when I inevitably need to download a bunch of these files manually to triage some bug in cargo-semver-checks.
So maybe the paths could normally be something like https://static.docs.rs/rustdoc-json/{krate}/{version}/{target}/{krate}_{version}_v{format_version}.json.zst, and the latest one (if you choose to go for it) could be a redirect from https://static.docs.rs/rustdoc-json/{krate}/{version}/{target}/{krate}_{version}_latest.json.zst to whichever version is latest?
Or if the krate component already uses underscores, then use dashes to disambiguate instead?
All this is super minor though — I think everything you proposed is good and the rest is just some tiny tweaks. Thanks again!
If it avoids the s3 ls call, I'd be fine with also having an option to download the "latest" format version available using a non-format-versioned request path.
the latest one (if you choose to go for it) could be a redirect from https://static.docs.rs/rustdoc-json/{krate}/{version}/{target}/{krate}_{version}_latest.json.zst to whichever version is latest?
note that static.docs.rs is not running any kind of dynamic code (it's basically just a read-only view of docs.rs' s3 bucket). we strongly discourage people from ever using static.docs.rs. instead they should use the download urls we document at https://docs.rs/about/download. so the redirect would be something like https://docs.rs/crate/{krate}/latest/json-docs.
the thing we are discussing here is not the public api, it's the implementation, because the layout of the S3 bucket affects how quickly we can do operations on it.
Activity
syphar commentedon Feb 26, 2021
@deeprobin what kind of information would you need?
docs.rs is just hosting some metadata and is serving the HTML files generated by
rustdoc,deeprobin commentedon Feb 26, 2021
Oh okay. Then it would be better if I open a issue in the rustdoc repository? I thought this was the right repository for this.
Nemo157 commentedon Feb 26, 2021
Rustdoc has an in-progress JSON backend: rust-lang/rust#76578
There has been some very brief discussion of docs.rs generating and hosting these files along with the html. But at the moment I think it's quite buggy (e.g. rust-lang/rust#80664 is probably going to fail a large proportion of builds) and the format is unstable. I think it would make sense to wait a little longer till it has stabilized a bit more before we add it here (I know of at least a couple of users that are already building alternative renderers based on the json, which will hopefully drive it towards stabilization).
[-]Can docs.rs generate .json files?[/-][+]Host `--output-format json` files in addition to generated HTML[/+]DottieDot commentedon Jul 2, 2022
Any progress/updates on this?
syphar commentedon Jul 3, 2022
There are still open ICEs when generating the JSON (see this current discussion on zulip, rust-lang/rust#93518 is one of the fixes.
When this is solved I believe we have to partially finish #795 so we have the additional spare capacity for running the additional builds.
Of course on top of that the actual code change needs to be done.
jyn514 commentedon Jul 3, 2022
I don't think we should start hosting this until the JSON backend is stabilized. The output format is likely to change several more times before stabilization.
syphar commentedon Jul 3, 2022
yeah, also valid point, I only had in mind that we could find more ICEs when running it for docs, but the format changing could be annoying for users of the JSON
syphar commentedon May 14, 2025
there are some missing features before we deem the format stable enough: ( coming from @jyn514 )
Item::attrsto not use debug representation. rust#137645pathsis inconsistent and questionably useful rust#93522, [rustdoc-json] Partially removepathsand introduceexternal_indexrust#103085, Rustdoc-Json: some item IDs are missing inpathsfield. rust#101687, [rustdoc-json] absolute position of item is hard to get rust#93524)syphar commentedon May 17, 2025
next set of notes coming from the all-hands / rustweek: we can actually start building & hosting rustdoc json.
about the implementation:
static.docs.rsfor download, similar to the archive download.format_versioncould be part of the filename / API.Normal rebuilds for the latest version of each crate would then also build & upload the rustdoc json. So after a couple of months of this being live, all latest versions have rustdoc json.
There are some pending additions that will come later that will make it easier to link between json docs easier (notably: extend the information we provide in
external_crates)edit: don't forget, we have a separate JSON output per target to store.
syphar commentedon May 17, 2025
implementation notes from first codebase read:
/target/doc/directory before the HTML or JSON builds. So we have to pull out the JSON output before we run the HTML buildstructure on S3 bucket is currently:
/rustdoc/{krate}/{version}.zip/sources/{krate}/{version}.zip/build-logs/{build_id}/{target}.txtand legacy locations before archive-storage
/rustdoc/{krate}/{version}//sources/{krate}/{version}/after each build we store the archive, and run
delete_prefixon the legacy locations.For rustdoc JSON we preferably want a folder structure where we can later
s3 lsand return the available format-versions?This means the best structure is probably a separate root:
/rustdoc-json/{krate}/{version}/{target}/{format_version}.json.zst, perhaps duplicating krate & version & target on the filename for a nicer download.s3 lscould use all files from the target folder.That also means every request to docs.rs for a rustdoc JSON file needs an
s3 lscall. The only way to prevent that, so we can statically determine the path, would be if we remove the format version from the path, or additionally maintain a file without the format version that we overwrite each rebuild.syphar commentedon May 17, 2025
@obi1kenobi following up on your request to be able to fetch old format versions, if we have them.
With my approach from above you would be able to run parallel requests to:
https://static.docs.rs/rustdoc-json/{krate}/{version}/{target}/{format_version}.json.zstusing your list of supported format versions. You'll get either the file or a 404 then, the 404s also cached when I updated the cloudfront config.
Would that suffice for your use-case?
( perhaps I add the other parameters to the filename additionally to make it self consistent in your download directory)
obi1kenobi commentedon May 20, 2025
Broadly this all sounds good! Thanks for looking into it and writing it up, I really appreciate it.
If it avoids the
s3 lscall, I'd be fine with also having an option to download the "latest" format version available using a non-format-versioned request path. This would likely be useful to cargo-semver-checks too: for example, say we're running a SemVer breakage study of crates.io crates. For each release, we'd look to download the latest rustdoc JSON format version available, together with the same format version for the immediately prior release. So long as both files are the same JSON format version, and the format is reasonably new (so cargo-semver-checks still supports it), everything would work well.I like the idea of adding the parameters to the filename, for when I inevitably need to download a bunch of these files manually to triage some bug in cargo-semver-checks.
So maybe the paths could normally be something like
https://static.docs.rs/rustdoc-json/{krate}/{version}/{target}/{krate}_{version}_v{format_version}.json.zst, and the latest one (if you choose to go for it) could be a redirect fromhttps://static.docs.rs/rustdoc-json/{krate}/{version}/{target}/{krate}_{version}_latest.json.zstto whichever version is latest?Or if the
kratecomponent already uses underscores, then use dashes to disambiguate instead?All this is super minor though — I think everything you proposed is good and the rest is just some tiny tweaks. Thanks again!
jyn514 commentedon May 21, 2025
note that
static.docs.rsis not running any kind of dynamic code (it's basically just a read-only view of docs.rs' s3 bucket). we strongly discourage people from ever using static.docs.rs. instead they should use the download urls we document at https://docs.rs/about/download. so the redirect would be something likehttps://docs.rs/crate/{krate}/latest/json-docs.the thing we are discussing here is not the public api, it's the implementation, because the layout of the S3 bucket affects how quickly we can do operations on it.
obi1kenobi commentedon May 21, 2025
Ah TIL, thanks!
Yes, so long as it's reasonable to download a specific rustdoc JSON format version for a given crate & release, I'm fine with anything.