Repository navigation
Define a Rust ABI #600
Description
Activity
See #1675 for some motivation for this feature (implementing plugins, that is plugins for Rust programs, not for the compiler).
Reacted by Victoria Brekenfeld, Shockingly Good, Stephan Sokolow, Félix, Christian Stadelmann, Koutheir A., Gamadril, René Léveillé, Marcin (Miriam) Mielniczuk, 🦀 Yuriy Larin 🦀 and 5 moreAnother motivation is the ability to ship shared libraries which could be reused by multiple applications on disk (reducing bandwith usage on update, reducing disk usage) and in RAM (through shared .text pages, reducing RAM usage).
Reacted by Stephan Sokolow, Asteria Hoffmeyer, Mikhail Zabaluev, Johannes Löthberg, dengelt, sp3d, Félix, Kornel, Patrick, Nirbheek Chauhan and 47 moreIt would also make Linux distributions more happy as it would allow usage of shared libraries. Which apart from memory reduction in different ways, also make handling of security issues (or otherwise important bugs) simpler. You only have to update the code in a single place and rebuild that, instead of having to fix lots of copies of the code in different versions and rebuild everything.
Reacted by Patrick, Christian Stadelmann, Johannes Löthberg, Nirbheek Chauhan, Anton Rapetov, Stephen M. Coakley, Gabriel de Perthuis, JoeyAcc, Alik Aslanyan, Felix Schütt and 39 moreShared libraries can be cool provided that there is a way to work around similar libraries accomplishing something in different ways such as openssl & libressl. There could also be value with making a way to allow a switch in-between static & dynamic libraries that can be set by the person compiling the crate. (possible flag?)
In my opinion, it's very hard to take Rust seriously as a replacement systems programming language if it's not possible in any reasonably sane manner to build applications linking to libraries with a safe, stable ABI.
There are a lot of good reasons for supporting shared libraries, not the least of which is building fully-featured systems for more resource constrained devices (like your average SBC or mobile device). Not having shared libraries really blows up the disk utilization in places where it's not cheap.
@jpakkane describes this really well in his blog post where he conducts an experiment to prove this problem.
Reacted by Patrick, dengelt, Anton Rapetov, Techcable, Alik Aslanyan, ranger, Koutheir A., joejoepie, SomeAnotherDude, Vanessa McHale and 30 moreIn my opinion, it's very hard to take Rust seriously as a replacement systems programming language if it's not possible in any reasonably sane manner to build applications linking to libraries with a safe, stable ABI.
Note that C++ still doesn't have a stable ABI, and it took C decades to get one.
Reacted by Steve Klabnik, Ignacio Carrera, wangcong, Mazdak Farrokhzad, Scott Olson, james gilles, Ashkan Kiani, Jason Carr, Juan Luis Cano Rodríguez, Adam Gausmann and 7 moreIt would also make Linux distributions more happy as it would allow usage of shared libraries.
There are a lot of good reasons for supporting shared libraries
(two quotes from two different people) Note that rust does support shared libraries. What it doesn't support is mixing them from different compiler toolchains. Some linux distros already do this with Rust; since they have one global
rustcversion, it works just fine.Reacted by Günter Zöchbauer and Sohang ChopraNote that C++ still doesn't have a stable ABI, and it took C decades to get one.
While that is true, implementations (GCC, clang, MSVC at least) have a (somewhat) defined ABI and it only changes every now and then. With Rust there is no defined ABI at all and things might break in incompatible ways with any change in the compiler, a library you're using or your code, and you can't know when this is the case as the ABI is in no way defined (sure, you could look at the compiler code, but that could also change any moment).
What it doesn't support is mixing them from different compiler toolchains. Some linux distros already do this with Rust; since they have one global rustc version, it works just fine.
The problem is not only about compiler versions, but as written above about knowing what you can change in your code without breaking your ABI. And crates generally tracking their ABI in one way or another. Otherwise you can't use shared libraries created from crates in any reasonable way other than recompiling everything (and assuming ABI changed) whenever something changes.
At least on Linux, everyone has pretty much settled on the Itanium C++ ABI. But even with the compiler locked down, it still requires very careful maintenance by a library author who hopes to export a stable ABI. Check out these KDE policies, for instance.
Rust crates would have many of the same challenges in presenting a stable ABI. Plus I think this is actually compounded by not having the separation between headers and source, so it's harder to actually tell what is reachable code. It's much more than just
pub-- any generic code that gets monomorphized in your consumers may have many layers of both public and private calls to make to the original crate's library. And all of that monomorphized code has to remain supported as-is when you're shipping updates to your crate.Reacted by Techcable, Neal Gompa (ニール・ゴンパ), Zverev Konstantin, Franklin Yu, wizzwizz4 and Günter ZöchbauerHad a long term crazy idea to solve this.
Define a stable format for MIR and distribute all Rust binaries/shared libraries in that format. Package managers have a post-install step to translate the MIR into executables or
.sofiles. Only the version of the MIR->binary backend "linker" has to be the same, the version of the source->MIR frontend compiler can differ.Because of monomorphization and unboxed types, you need to still need to relink all the MIR files when a dependency is updated. Similarily, updating the backend compiler requires all the MIR files to be recompiled.
However, assuming we can push more and more optimisation passes into MIR rather than llvm, the time spent in the backend should be reduced to something acceptable.
If you want to push it even further, keep everything in MIR form and use miri (or a JIT version of it) to run them. Frequently used files can be linked and persisted to disk. And we've just reinvented the JVM/CLR/Webassembly.
Reacted by Rahul Butani, Shailesh Iyer, Aleksandr Korzun, Karim Vergnes, Adriaan Jacobs, Danyiel Colin, GrizzlT and CimaxRedix64 remaining items
@burdges The current calling convention is RVO-oriented (except for things like returning the variants of
Resultseparately, I guess).
When we say NRVO, we mean more like "removing copies within a function by writing to the destination directly", which is an optimization not requiring ABI changes (modulo theResultthing which is a rather advanced transformation).
Also, if anyone wants to see my informal take on this: https://twitter.com/eddyb_r/status/1166953126928277505
It's close to "we're still nowhere near ready to make a stable ABI" but also IMO, a lot of the stuff around the idea of a "stable ABI" is not thought through that well, or even outright misguided.
If you start from regarding C as an utter failure on pretty much all fronts other than its popularity, you might find better ways to do things.
But yeah it's far off, likely involving programming languages and tooling very different from what we're used to, especially in the systems programming / low-level areas.
Reacted by tesuji, Diana and Jeff BurdgesSo you would still be restricted to monomorphic declarations? What's the advantage over some sort of interop library using the C ABI that's provided as source?
With an interop library you always need a "serialization" and "deserialization" step to convert your types to some
#[repr(C)]type, a fair bit of unsafe code, and you probably need some kind of procedural macro system to define your interface in a way that is not too painful.If we had a
#[repr(MoreThanC)]or something, that extends#[repr(C)]with support for more types (like Vecs, Strings) then you effectively solve the "plugin system" use-case: plugin interfaces typically involve only simple types anyway because you have to keep them stable, and this would make it faster, safer and simpler than if you had to use an interop library.@Diggsey You could probably prototype parts of the Swift approach (i.e. adding a bit of indirection to hide potential differences, while trying to minimize overhead) outside of the language.
You don't need full "serialization", just enough to provide access without relying on any assumptions.
For example,
proc_macro::bridge::buffercontains minimal (e.g.T: Copy-only) versions of&[T]andVec<T>that can be safely passed between two Rust "worlds" (potentially compiled by incompatible-ABI compilers and using incompatible global allocators), without significant overhead (e.g. when extending theVecyou only need to do a dynamic call once you run out of capacity).We don't want to bake anything like stable ABIs into the standard library because that puts hard limits on what we can do with the implementation, whereas any experiments in the ecosystem could thrive and go through many iterations, with just proc macros and traits.
Since you mention serialization, here's an analogy: a stable ABI being used by the standard library is like a stable serialization framework that's baked into every single type supporting it and you can't version it.
There's only one way data is represented in memory, and Rust is already having trouble taking advantage of that representation, due to the compilation model.
So I'd rather move in the direction of delaying that choice of representation while also giving people who really need it more fine-grained control over custom representations (e.g. bit-packing).Reacted by Pauan, Adam Perry, Yagna Srinath Reddy Battula, Tim Janus and Liam Germaina stable ABI being used by the standard library is like a stable serialization framework that's baked into every single type supporting it and you can't version it.
GCC had to deal with this for C++11, and they ended up forking stuff with ABI tags.
https://developers.redhat.com/blog/2015/02/05/gcc5-and-the-c11-abi/I mention this as a cautionary tale.
Reacted by Pauan, Eduard-Mihai Burtescu, Jeff Burdges, Karl Hedberg, sebbu2 and Liam GermainInteresting, it sounds like "Define a Rust ABI" actually means roughly two-ish things:
If you want broader dynamic linking, then you pass some
dyn(v1) Traitacross the interface. It'd require do a stable trait based interfaces for the data structures, but this bring other benefits too. I'd think this might facilitate doing some large GUI toolkit in Rust for example.If you want to "Rust OS", then you want some
#[repr(redox-v1)]that constrains type parameters, probably disallows type generics, but lifetimes definitely work, and const generics might work, maybe at the cost of making them untestable in where clauses.There is an extended version of this second form where you specify
#[repr(wasm-v1)]to write directly into some form that ensures compatibility even with some virtual machine.I'd think
#[repr(redox-v1)]would reduce the depth of type erasure required to exploitdyn(v1) Trait, so maybe that's logically first.Reacted by Eduard-Mihai BurtescuGiven that Rust already lets you switch between a bunch of different ABIs with
reprandexternas it is, I don't see the harm in simply adding one that freezes the current unstable ABI as a stable one that you would have to explicitly ask for, while the unstable one would remain the default. In the future, if there's a better design, just add that one as a selectable ABI as well. I don't think it's the right course of action to wait until we've figured out the “ABI to end all ABIs” before we offer even a single Rust-specific ABI, especially considering that it looks like work on coming up with that perfect ABI isn't happening right now anyway.Also consider that since most functions are only crate-internal, the issue of making sure the ABI is a good one isn't as pressing as C++, where everything is public by default.
Reacted by explosion-mental and Ondřej Běhal@Serentty Besides other reasons, the current rules can't be frozen because they don't exist.
You can't version implementation-defined behavior without copying the entirety of the implementation, it's not even implementation-specified, and ideally it should be specified at least by an RFC.If an RFC for a specific ABI is presented, with details for all supported targets, that might work out.
But why would you bother with that whenrepr(C)andextern "C"exist?Even if you have such an ABI, it won't be used by libstd types, just like the C one isn't.
(that's one of other reasons I alluded to above, probably the main one)(looks like eddyb and I were writing these replies at the same time)
I don't see the harm in simply adding one that freezes the current unstable ABI as a stable one that you would have to explicitly ask for, while the unstable one would remain the default
This expresses a common misunderstanding that "the current unstable Rust ABI" is something that is already fully implemented in a well-defined, well-understood way that we know how to support to everyone's satisfaction.
A large chunk of the work here is achieving consensus on what a "Rust ABI" is even supposed to be, and that whatever it's supposed to be would even be a desirable thing to stabilize. There are already loads of comments in this thread expressing disagreement over what it is or reasons why it's undesirable to ever stabilize any version of it.
Okay, so if there's no rigid documentation for how the current ABI works, then that is indeed a problem that prevents it from being frozen.
If an RFC for a specific ABI is presented, with details for all supported targets, that might work out.
But why would you bother with that whenrepr(C)andextern "C"exist?Because the C ABI is still quite limited. Not supporting trait objects is a fairly big hurdle, for example.
I just heard from a friend that trait pointers are currently an unstable feature in the C ABI. That actually brings it close to what I would want in a Rust ABI anyway. The only thing really left bothering me is that the standard library can't easily be shipped as separate form the executable, which would be a real bandwidth-saver over CDNs, but because of generics that's probably not entirely possible anyway.
For reference, a blog post by @Gankra: How Swift Achieved Dynamic Linking Where Rust Couldn't
Reacted by Jeff Burdges, Franklin Yu, Serentty, Louis Dureuil, Randy MacLeod, Raniz Daniel Raneland, Wael El Oraiby, Martin Kolman, Alex, Linz and 9 more- Reacted by Yavor Kolev, Tim Janus and Hank BaoReacted by Louis DureuilReacted by DimitrisJim, Asteria Hoffmeyer and Axel Karjalainen
Hi everyone! I'm interested in rust compiler development. Could anyone point me to code or a spec for how the abi works in rust as of today?
Reacted by Chris Nobody, Theo Paris and Forrest HopkinsHi guys, sorry to bother you. May I ask if the latest version of
Rusthas a stableABIimplementationReacted by Theo ParisNope.
And that will probably never be added, so that the language can continue to grow
Reacted by explosion-mental, Viktor Popp and Fernández Owen
Right now, Rust has no defined ABI. That may or may not be something we want eventually.