Repository navigation
--emit=metadata emitting empty .rmeta file #67293
Description
Activity
- changed the title
[-]`--emit metadata` emitting empty .rmeta file[/-][+]`--emit=metadata` emitting empty .rmeta file[/+]on Dec 14, 2019 I would be cautious with this, as it could be a noticeable increase in disk usage for some projects. Rust already uses too much disk space. There should probably be a stronger reason than "it seems like it should".
Emitting an empty .rmeta file makes sense, as the metadata is what is needed to use it with
--extern/extern crate. For executables, cdylibs and staticlibs that is impossible, so no metadata is needed in those cases.Reacted by Victor DingThere should probably be a stronger reason than "it seems like it should".
Definitely. IMO, user's explicit request should be honored; therefore, in this case, the explicit request is
---emit=metadata.On the other side, I agree with @bjorn3 's point that the metadata is not needed for certain crate types; however, under this scenario, I feel rustc should print an error or warning message instead of silently generating an empty file.
Reacted by Tyler Mandry and jynit could be a noticeable increase in disk usage for some projects
Does it mean these projects request
--emit=metadatawith crate types that do not need metadata? If it does, is there any obstacles or restrictions that these project cannot remove--emit=metadatafrom the arguments?Does
cargo checkuse--emit=metadatafor executables?Does it mean these projects request
--emit=metadatawith crate types that do not need metadata?Yes. this is how
cargo checkworks.Why is cargo passing
--emit=metadatafor these crate types? It knows already that it's not passingcrate-type=lib, it can omit themetadataflag in that case.I would be cautious with this, as it could be a noticeable increase in disk usage for some projects. Rust already uses too much disk space. There should probably be a stronger reason than "it seems like it should".
cargo uses too much disk space, because cargo does not have any transitive checksum-based invalidation model employing something like LRU gc, so cargo is for example unable to avoid repeatedly regenerating said
.rmetafile. the proposal in OP is the one single thing i would need from rustc in order to implement the kind of extremely powerful cached and pipelined build process we implemented for scala compiles with pants bazelbuild/rules_rust#428 (comment)Cargo always keeps the
-Cmetadataand-Cextra-filenamearguments in sync, so if the output rmeta can possibly be identical to an existing rmeta (which requires-Cmetadatato be identical due to being part of the ABI), then the output filename will be identical (because the crate name and-Cextra-filenameare identical) and thus the new rmeta file will overwrite the old rmeta file. Checksum-based invalidation can only make builds faster, not cause them to use less disk space.Reacted by Danny McClanahanChecksum-based invalidation can only make builds faster, not cause them to use less disk space.
I used "invalidation" too loosely there, sorry. pants does not compute what I call "negative invalidation" like cargo. Build tasks are pure functions which may be async https://www.pantsbuild.org/stable/docs/writing-plugins/the-rules-api/concepts--this results in what I would call "positive invalidation", which is how we can express recursive task dependencies that still terminate. We also completely virtualize file operations https://www.pantsbuild.org/stable/docs/writing-plugins/the-rules-api/file-system.
Checksum-based invalidation indeed does not improve anything (I believe an old pants contributor proposed exactly this to cargo recently; sorry about that). But checksum-based data stores can be used to meaningfully deduplicate results--not just file content, but also shared directories, e.g. if you use a merkle tree encoding.
More generally, a content-addressed store/CAS allows you to manipulate data in the programming language without reference to the filesystem at all. If you can trust that the checksum can be dereferenced from the CAS, then you can apply all sorts of heuristics and whatnot (e.g. materializing extremely common files as symlinks) instead of having to constantly translate your logic through whatever the filesystem is able to express.
A merkle tree can also be transferred across a network safely, while optimistically checking for the existence of a checksum in any such remote node can still be very quick.
I do have some qualms with the pants fs API, which does not allow the task developer to apply any sort of e.g. string tag to contextualize what this arbitrary merkle node represents to the user.
My initial comment was extremely hastily written--I was frustrated because file-level metadata from @xeno-by (now @burmako)'s https://github.com/twitter/rsc was how @winw and I were able to productionize what Eugene calls "outline compilation" in his great talk here https://www.youtube.com/watch?app=desktop&v=8SnIBkJXD8I. However one thing Olafur and Eugene repeatedly described to me is how Odersky's metadata formats like TASTy (https://docs.scala-lang.org/scala3/guides/tasty-overview.html) were not really designed for efficient querying, and require many recursive dereferencing operations, whereas Eugene's SemanticDB is denormalized beforehand https://scalameta.org/docs/semanticdb/specification.html.
Especially consider the framing at https://scalameta.org/docs/semanticdb/guide.html#semanticdb-based-tools, which notes its independence from compiler internals. I think defining a public protocol in terms of denormalized flat records which rustc (or something else) could compile a
.rmetato could be a way to achieve the success we found at Twitter making use of Eugene's work.Reading about the query-based architecture of the rust compiler just now and it being motivated by incrementality and am very surprised to realize how strong the analogy is to the pants v2 engine (https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation.html#the-try-mark-green-algorithm).
@jyn514 for example the bootstrapping "mini pants" task system I was pondering for a prototype to unify
bootstrapandcargowould have been making extremely similar tradeoffs as described for the rustc incremental compilation system https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html#query-modifiers and since I want it to be cute (unlike pants) I was absolutely going to have an explicitly-invoked content-addressed store in the filesystem.(& btw If anyone knows offhand where I should look to learn more about this incrementality persistence mechanism plz lmk https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html#incrementality-of-on-disk-data-structures)
zip-based multiple crate archive format proposal
But the really important thing I wanted to mention was that reading enough of @bjorn3's posts today caused me to feel like I had an epiphany of sorts but maybe not:
(a) we (often) actually lose information when we generate the system-native archive format, particularly the ability to split back out into unambiguous record boundaries
@bjorn3 #73632 (comment)By the way I forgot if I asked this already, but would a rustc command to bundle multiple rlibs into a single staticlib without requiring any rust source for this staticlib work? So something like
rustc --crate-type staticlib -Zlink-libs liba.rlib libb.rlib -o libcombined.a. This should allow diamond dependencies I think. You could run it just before invoking the system linker.(b) in addition to crate-specific data/config, there is global data/config we need before we can generate an output binary. it seems that we lose the ability to modify that information once we bounce to the system-native format
@bjorn3 #73632 (comment)We already have the staticlib crate type for bundling everything into a single
.afile. Turning individual rlibs into.afiles won't work as it would either duplicate the allocator shim and such between every such.aor or omit them from all and produce unlinkable.afiles. One idea I have is to produce a new crate type which is tostaticlibasdylibis tocdylib. It would act like adylibwith respect to producing the allocator shim and bundling multiple crates, but produce a.afile instead of a.sofile and add the necessary crate metadata to allow consuming it like a regular crate from rustc. See also the end of #111594 (comment)(c) the "crate type" seems ridiculously overloaded and it seems confusing by itself to me that it can represent a single (?) compiled crate or like multiple
@bjorn3 #111594 (comment)A couple of days ago my PR #106560 got merged. By combining
-Z staticlib-allow-rdylib-depswith-Z staticlib-prefer-dynamic, libstd will be dynamically linked and--print native-static-libswill tell you how to dynamically link against libstd. This does not however enable statically linking libstd. I'm not sure what the best way to add that is though.The
staticlibjust likecdylibis not supposed to export any symbols outside of those that are#[no_mangle], which would makelibstd.aimpossible. Currently it does export all symbols as there is no way to indicate that a symbol shouldn't be exported from an archive (forcdylibwe have control over the linker, forstaticlibwe don't). This needs to be fixed to allow multiple ruststaticlib's to be linked into the same executable or dynamic library without getting symbol conflicts. (#104707) Maybe one option would be to have something like arstaticlibcrate type which can bundle multiple crates and can be statically linked (that is it is tostaticlibwhatdylibis tocdylib) and then additionally ship libstd asrstaticlib.& then this part:
Maybe one option would be to have something like a
rstaticlibcrate type which can bundle multiple cratesaffordances of a zip-based CAS archive
I am not an expert in backend codegen, but I have spent immense amounts of time with the zip file format (the
zipcrate will never have a CVE on my watch but we do have SIMD parsing because of me) & with packaging software. In python, we have distributions (which I analogize to the current set of rust crate types), but we also have zipapps, which are themselves actually much less structured than pex files.Also important: PEP 777 is the wheel deal (and Emma Smith who is authoring it is also the author of the Rust PEP and she rocks). See my infodump in the PEP thread.
Some final points:
- both java and python allow loading code from a zip file. for various reasons, this can sometimes but not always be significantly faster than traversing the filesystem, particularly in the case of generated module paths with many many entries.
- it also means you can actually be 100% confident about where your code is being loaded from (death to linear path traversal!!!!)
- so rust already has the crate hash. python's analogy would be the direct url. please consider the following:
- introduce --generate-ipex to (v1) python binary creation to lazy-load requirements pantsbuild/pants#8793 => you can safely load a resource from from a registered external store (analogy to https://rustc-dev-guide.rust-lang.org/query.html#how-queries-interact-with-external-crate-metadata)
- https://pip.pypa.io/en/latest/reference/installation-report/#specification => PEP 751 maliciously erased my years of effort to get consensus on this standard i created so every python user could rely upon a standard to do my
--generate-ipexwork- (which i developed specifically to serve twitter data scientists with slow connections to our US-based data centers)
- so pex actually has a filesystem-based CAS store which it unconditionally extracts to upon first startup. but you can actually use a zip itself as a store without decompression consume packed wheel cache in zipapp creation pex-tool/pex#2175
- so i need to actually implement this, but zstd dicts can be used to get faster-than-light compression ratios by codifying context: https://codeberg.org/cosmicexplorer/corporeal
- will be proposing this as part of a wheel format at some point, but unfortunately the PEP process and reference impl for python packaging are hostile territory atm Plans for
packaging.metadatapypa/packaging#570 (comment) - i also plan to start work on a rust-native zstd impl this week
- will be proposing this as part of a wheel format at some point, but unfortunately the PEP process and reference impl for python packaging are hostile territory atm Plans for
- it's kind of a gimmick but you can always just append to the end of a zip file's local entries (before the central directory) and this makes it incredibly useful for incremental accumulation of structured persistent data
- i realized you can both extract zips in parallel as well as construct them this way via recursively merging without intermediate decompression: https://docs.rs/zip/latest/zip/write/struct.ZipWriter.html#method.merge_archive
- astral blatantly steals credit for this technique i invented too but basically zip archives can be introspected even over a network A
pip resolvecommand to convert to transitive == requirements very fast by scanning wheels for static dependency info (WORKING PROTOTYPE!) pypa/pip#7819- this is the absolutely bulletproof version perform 1-3 HTTP requests for each wheel using fast-deps pypa/pip#12208 but the sketch is
_hacky_extract_sub_reqs() (see https://github.com/cosmicexplorer/pip/blob/a60a3977e929cfaed6d64b0c9e3713d7c502e51e/src/pip/_internal/resolution/legacy/resolver.py#L550-L552) will:
a. send a HEAD request to get the length of the zip file
b. perform several successive GET requests to extract the relative location of the METADATA file
c. extract the DEFLATE-compressed METADATA file and INFLATE it
d. parse all Requires-Dist lines in METADATA for requirement stringsFurthermore there may be performance benefits to using different archive formats
this is rust!!!!! we didn't just stick with null terminated strings because they were "system native"!!!!!at a basic level, differentiating "flexible archive format we can extend for our own purposes" seems SUPER important to differentiate from "system-native archive format we can provide to a C linker".
i unfortunately have absolutely no clue what
arorlddoes to generate a.aarchive (yet), but i am absolutely 10000% sure that generating a zip-basedrstaticlibabsolutely can be pipelined, and that a zip is much more structured than most other types of formats i can nameoh and i'm greedy too this means my rust build system can incorporate a concept of ABI and target compatibility too
Emitting an empty
.rmetafile makes sense, as the metadata is what is needed to use it with--extern/externcrate. For executables, cdylibs and staticlibs that is impossible, so no metadata is needed in those cases.thanks for this explanation of those semantics! do i understand correctly, then, that your proposed new output type
rstaticlibwould be a way to e.g. enforce the emission of a.rmetawhile retaining backwards compat?thx for listening!! i will probably put this up as a pre-RFC on the discourse tomorrow!
- both java and python allow loading code from a zip file. for various reasons, this can sometimes but not always be significantly faster than traversing the filesystem, particularly in the case of generated module paths with many many entries.
At present,
rustc --emit=metadata foo.rsgenerates an empty rmeta file namedlibfoo.rmeta. The logic behind is that rmeta is not needed for certain crate types. However, I feel we should still save it to a file when user explicitly requested it by--emit=metadata, especially the metadata is always available in memory regardless of output crate types.