Repository navigation
Tracking issue for RFC 1861: Extern types #43467
Description
Activity
- changed the title
[-]Tracking issue for RFC 1861: Extern tyupes[/-][+]Tracking issue for RFC 1861: Extern types[/+]on Jul 25, 2017 This is not explicitly mentioned in the RFC, but I'm assuming different instances of
extern typeare actually different types? Meaning this would be illegal:extern { type A; type B; } fn convert_ref(r: &A) -> &B { r }
Relatedly, is deciding whether we want to call it
extern typeorextern structsomething that can still be done as part of the stabilization process, or is theextern typesyntax effectively final as of having accepted the RFC?EDIT: rust-lang/rfcs#2071 is also relevant here w.r.t. the connotations of
type"aliases". In stable Rust atypedeclaration is "effect-free" and just a transparent alias for some existing type. Bothextern typeandtype Foo = impl Barwould change this by making it implicitly generate a new module-scoped existential type or type constructor (nominal type) for it to refer to.Reacted by ૮༼⚆︿⚆༽つ, Guillaume Boutry, Tastaturtaste, Vladislav Sukhmel and Magnus MarklingCan we get a bullet for the panic vs DynSized debate?
Reacted by Gábor Lehel, Ixrec, Peter Atashian, scottmcm, Paul Liétar, Felix S., pravic, Tarek A. and Mavity the MadityI've started working on this, and I have a working simple initial version (no generics, no DynSized).
I've however noticed a slight usability issue. In FFI code, it's frequent for raw pointers to be initialized to null using
std::ptr::null/null_mut. However, the function only accepts sized type arguments, since it would not be able to pick a metadata for the fat pointer.Despite being unsized, extern types are used through thin pointers, so it should be possible to use
std::ptr::null.It is still possible to cast an integer to an extern type pointer, but this is not as nice as just using the function designed for this. Also this can never be done in a generic context.
extern { type foo; } fn null_foo() -> *const foo { 0usize as *const foo }
Really we'd want is a new trait to distinguish types which use thin pointers. It would be implemented automatically for all sized types and extern types. Then the cast above would succeed whenever the type is bounded by this trait. Eg, the function
std::ptr::nullbecomes :fn null<T: ?Sized + Thin>() -> *const T { 0usize as *const T }
However there's a risk of more and more such traits creeping up, such as
DynSized, making it confusing for users. There's also some overlap with the various custom RFCs proposals which allow arbitrary metadata. For instance, instead ofThin,Referent<Meta=()>could be usedReacted by TechcableI think we can add extern types now and live with
str::ptr::nullnot supporting them for a while until we figure out what to do aboutThin/DynSized/Referent<Meta=…>etc.@SimonSapin yeah, it's definitely a minor concern for now.
I do think this problem of not having a trait bound to express "this type may be unsized be must have a thin pointer" might crop up in other places though.
Oh yeah, I agree we should solve that eventually too. I’m only saying we might not need to solve all of it before we ship any of it.
315 remaining items
Since no one seems to have asked this question yet:
The current "idiom" for opaque types is having a
_opaque: [u8; 0]field.
This approach supports havingPhantomDatafields for supporting things like adding lifetime annotations on the type.
I make very heavy use of this in rust-sfml:#[repr(C)] pub struct Shader<'texture> { _opaque: [u8; 0], // Keeps track of the lifetime of the texture the Shader borrows, // but of which Rust has no idea otherwise, since // the field that "borrows" the texture is on the C side _texture: PhantomData<&'texture Texture>, }
How would I go about this using extern types?
If extern types cannot support this, then it's not really a universal replacement for the_opaque: [u8; 0]field "workaround".How would I go about this using extern types?
I would assume by doing something like this:
extern { type ShaderExtTy; } #[repr(transparent)] pub struct Shader<'texture> { inner: ShaderExtTy, _phantom: PhantomData<&'texture Texture>, }
Reacted by Martin Habovštiak@programmerjake
Perhaps... Although it would have to be#[repr(transparent)] pub struct Shader<'texture> { _phantom: std::marker::PhantomData<&'texture Texture>, inner: ShaderExtTy, }
Since only the last field can be unsized.
Reacted by Jacob Lifshay and Martin HabovštiakThe third unresolved question shouldn't be a problem anymore, since LLVM now uses opaque pointer types.
Reacted by Mads MarquartThe third unresolved question shouldn't be a problem anymore, since LLVM now uses opaque pointer types.
it is probably still a problem for rustc_codegen_gcc since libgccjit has separate pointer types for each pointee type.
Hopefully, GCC isn't quite so sensitive about which pointee type needs to be picked. (I forgot why it was important for LLVM for
void*.) But anyway given that the GCC backend is not complete yet, I don't think we should block this RFC on that -- it's "just" one more item on the list of things whose implementation is still incomplete.Reacted by Martin Habovštiak and Andrew Fernandes- added 2 commits that reference this issue
on Aug 1, 2025 I was deeply unfamiliar with both this RFC as well as the
sized_hierarchyone. Having just read both of those, I would like to venture a potential counterexample to the conjecture proposed by @kennytm in #43467 (comment):While it sounds good in theory I doubt in practice if there is any actual scenario where the FFI type specified the exact alignment yet the size and all fields are implementation details. Usually you either have none or all of these 3 details.
I believe
struct direntused byreaddir()andgetdents()is just such an example where alignment is extremely significant and in fact required by POSIX for the separate entrypointposix_getdents():Since the
d_reclenvalue for the last entry in buf includes padding to satisfy alignment requirements, applications can grow the buffer and callposix_getdents()again to append to it without needing to perform an alignment calculation.How Alignment Can be Known without Layout
A few important additional concerns about alignment here:
- The buffer provided to
posix_getdents()is a memory address and length, which in most cases are provided in most cases directly to the operating system via syscall arguments.- See musl's implementation: https://git.musl-libc.org/cgit/musl/commit/src/dirent/posix_getdents.c?id=1b0d48517f816e98f19111df82f32bfc1608ecec.
- The "libc" itself (as in any
extern "C" fns which may be statically or dynamically linked into the executable) are not responsible for nor even aware of the individual field offsets into a record--those are set directly by the OS within a syscall.
- This is starkly in contrast to
fread()/fwrite(), where the libc is allowed to perform its own caching on top of the OS block layer through theFILE*data structure. The libc's internal control over extra opaque allocations necessarily involves knowledge of alignment.- Perhaps more importantly, the libc will typically employ some form of opaque mutual exclusion mechanism to achieve atomicity guarantees for file streams (https://www.gnu.org/software/libc/manual/html_node/Stream_002fDescriptor-Precautions.html).
- glibc's ontology of
unsafekinds therefore identifies internal locking and internal malloc/free/realloc as distinct from POSIX safety concepts https://www.gnu.org/software/libc/manual/html_node/Unsafe-Features.html, and this directly produces the atomicity issues thatstd::fs::read_dir()currently fails to solve for all POSIX platforms, precisely because it's an entirely opaque pointer.
- However,
posix_getdents()works under the completely opposite theory as compared toFILE*, and much closer to the atomicity guarantees available e.g. for pipes andio_uring, in allowing the user to provide a buffer directly to the OS, and which the libc doesn't perform any interpretation of.posix_getdents()also happens to invoke a much stronger atomicity guarantee:
If
posix_getdents()is called concurrently with an operation that adds, deletes, or modifies a directory entry, the results fromposix_getdents()shall reflect either all of the effects of the concurrent operation or none of them.libc and ABI Guarantees
I see no case that you specifically only want to know
align_of::<FILE>() == align_of::<usize>()and nothing else.The C interface describes why:
int posix_getdents(int fd, void *buf, size_t len, int flags)
void*is intentionally used to avoid source-level API breakage, even in the event of the OS deciding to add or modify the interpretation of fields (and therefore introduce ABI breakage that requires a recompile). I see that @kennytm in fact predicts one of the exact issues at play here (thatstruct direntalways ends with a runtime-sized array):The only exception might be dynamic-sized array, like an array tail like
xxxx_t tail[0], or C-style stringconst char*/const wchar_t*, but I think they are a different kind of Custom DST that shouldn't be hacked through extern type.In fact, an earlier revision of POSIX specifies that these are in fact always null-terminated strings (https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/dirent.h.html#tag_14_08_05) and the 2024 update with
posix_dentin fact has thed_namefield at the end of the struct.One important note that the current implementation of
std::fs::read_dir()has a lengthy comment about is the remarkable variance in definitions across platforms for the immediated_namefield--usually this is a C11 conditionally-supported VLA, but it often falls back tod_name: [MaybeUninit<u8>; 0],d_name: [MaybeUninit<u8>; 1], or evend_name: [MaybeUninit<u8>; 256], where256 == NAME_MAX. This allows specific implementations to provide the record length and alignment length statically:Implementations that return fixed-length entries can simply set d_reclen to that length in posix_getdents().
Relationship to
sized_hierarchyIt also allows us to demonstrate the sizing hierarchy (see #144404) at play in terms of nested lifetimes resulting from sequential parsing operations, which aligns precisely with the discussion arising from the
externrefproposal for thesized_hierarchyRFC:- The end user of e.g. a Rust
getdents()wrapper owns the buffer used for all directory entries within agetdents()call, so they have to know the alignment ofposix_dentin order to allocate the buffer correctly (although POSIX also specifies they can just fallback to an alignment of 1 likemalloc()gives you).- It also strikes me that alignment being a necessary prerequisite to dynamically allocate memory isn't yet mentioned in the
core::marker::Alignedtrait RFC RFC:Alignedtrait rfcs#3319. - This would seem to make alignment deeply related to ownership, analogous to the
Valuesupertrait ofPointeeproposed in #144404.
- It also strikes me that alignment being a necessary prerequisite to dynamically allocate memory isn't yet mentioned in the
- Then, it strikes me that
externrefseems remarkably appropriate for modelling the direct kernel-to-user writes performed in thegetdents()syscall, as it's memory the kernel has to take special care around storing and loading, and which creates its own ABI.- Notably, the C header files defined in the kernel for
struct direntlayout are the exact struct definitions that glibc and musl propagate to their users.
- Notably, the C header files defined in the kernel for
- The fact some platforms can make
d_recordlenentirely known at compile time is curious:- They would still need to access a custom vtable describing the layout of a custom DST ending with a C11 VLA (the rest is just padded with nulls up to the next alignment boundary),
- However, splitting up the buffer contents into statically-known records can be done in parallel (as a slice), without the serial data dependency that an
Iteratorwith a DSTs for each directory entry would require.
- In order to actually interpret the
getdents()orposix_dentrecords, we would finally require custom DSTs using vtables with actual info about field offsets. Note that the linked prior art for custom DSTs states verbatim that:
Communicating with C types that contain flexible array members is an important part of this RFC.
Conclusion
I had much more to say here about the lifetime of the
getdents()buffer, but I think this should be sufficient for now. I don't think this necessarily refutes anything (for example, whether extern types should support VLAs as a type of DST), but I wanted to mention at least one case in which alignment (necessary for allocations) is distinguished from size and field layout (particularly in the case of DSTs, VLAs, and especiallystruct posix_dent), and why it might be useful to distinguish these for extern types.Reacted by Biscuit Vicious- The buffer provided to
This is a tracking issue for RFC 1861 "Extern types".
Steps:
Unresolved questions:
Rust does not support types that don't have dynamically computed alignment -- we need the alignment to compute the field offset in structs.
extern typeviolates this basic assumption, causing pain, suffering, and ICEs all over the compiler. What is the principled fix for this?Should we allow generic lifetime and type parameters on extern types?
If so, how do they effect the type in terms of variance?
In std's source, it is mentioned that LLVM expects
i8*for C'svoid*.We'd need to continue to hack this for the two
c_voids in std and libc.But perhaps this should be done across-the-board for all extern types?
Somebody should check what Clang does. Also see "extern type" should use
opaquetype in LLVM #59095.RESOLVED because all pointer types are
ptrnow.How should this interact with unsized arguments? Currently it ICEs: unsized_fn_params should not accept types that don't have a dynamically fixed size (such as
externtypes) #115709How should we model
mem::size_of::<ExternType>()? Tracked in Tracking Issue for Sized Hierarchy #144404.externblocks are skipped in the v0 mangling which meansextern typecan cause symbol collisions: mangling_v0: Skip extern blocks during mangling #92316 (comment)