Investigate issues with thread locals

TLS destructors don't run on 9x
Alignment issues on Win7 and lower: Add initial MinGW support #60 (comment), #[thread_local] alignment broken on Windows, GNU ABI rust-lang/rust#135719 (comment)

Update: While working on Rust9x 1.84 I've been testing this once again - still no dice, even though we now use the native TLS setup 🤔

Could you provide some code for how to check this? I've been poking around the TLS area so maybe I can check it out.

@Jookia

@Jookia https://github.com/rust9x/rust9x-sample/blob/0492688fed6f5468bbd38eb8617d5da892911279/src/main.rs#L205-L231 in the sample should show the issue (drop code not running)

Alright, I tested the destructor issue mentioned earlier on NT4 and 98 with MinGW and has_thread_local set to true and false. On both NT4 reports the drop and 98 doesn't.

I also tested the alignment code shown in the bug on NT4 and 98 with has_thread_local set to true and false. For an alignment of 256, NT4 aligns to 128 bytes and 98 aligns to 12 bytes (???).

Regarding alignment, Windows 7 and Wine are ignoring IMAGE_SCN_ALIGN_16BYTES. There's a fix for Wine soon. But the GNU toolchain also doesn't tell Windows to align things using that flag, and that doesn't seem to be working on my mingw-lite toolchain as of today.

Bugs about the alignment:

rust-lang#138903
rust-lang#135719

GCC's emulated TLS has a bad reputation:

rust-lang#91659
rust-lang#102243
msys2/MINGW-packages#2519

In 2025 the Cygwin port using emulated TLS was merged and works:

rust-lang#141846
rust-lang#141719

The latest attempt to enable thread_local on Windows is from 2025:

rust-lang#141794

The author's opinion is that "enabling this is just the matter of waiting an unspecified amount of time before minimal versions for GCC+Binutils+mingw-w64 can be set high enough." This was written by the same developer that commented in the open bug about alignment being broken, so it seems it's going to be a matter of time. Maybe 2027 is the year of use_thread_local on -pc-windows-gnu!

But evidently not today.

Thanks for the analysis!

and 98 aligns to 12 bytes (???).

IIRC it's just DWORD-aligned on 9x (hence the issue with the hashmap random state).

I wonder if some ungodly hacks are possible to manually align them :)

Oh oops, I was reading the hex value rather than the binary bits. Yes it's aligned to 4 bytes.

Some more research: Windows PEs can have a TLS callback directory: https://learn.microsoft.com/en-us/windows/win32/debug/pe-format#the-tls-directory The kernel calls these when a DLL or process is attached or detached: https://github.com/wine-mirror/wine/blob/master/dlls/ntdll/loader.c#L1647 , https://github.com/mirror/reactos/blob/master/reactos/dll/ntdll/ldr/ldrinit.c#L1111. The NTDLL symbol is LdrpCallTlsInitializers.

You can dump information in this table by putting things in linker sections like ".CRT$XLA" to ".CRT$XLB".

MinGW's CRT handles the C destructor code for us here: https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-crt/crt/tlssup.c .

Rust puts a destructor callback in ".CRT$XLB". This is explained in https://github.com/rust9x/rust/blob/rust9x/library/std/src/sys/thread_local/guard/windows.rs .

So maybe Windows 9x doesn't implement this?

Nynaeve has a good write-up on how Windows TLS stuff works: https://web.archive.org/web/20251109192547/http://www.nynaeve.net/?p=190 (go backwards from here). I highly recommend reading them.

This page has some more information on the state of things: http://www.icodeguru.com/cpp/ImperfectCppPracticalSolutions/ch10lev1sec5.html It has this note:

The second problem is that the Win32 API does not provide any ability to clean up the slot when a thread exits. This means that one has to somehow intercept the thread's exit and clean up the resources associated with the value in that thread's slot. Naturally, for C++ folks, this is a painful loss of the automatic destruction that the language provides for us, and can be next to impossible to work around in some scenarios.

There's a few solutions I see:

Link in a DLL that has DllMain and have it somehow call the TLS functions
Do garbage collection whenever registor_dtor runs
Have threads launched by Rust clean up when they return or call an exit function

It looks like option 1 is what old timey thread libraries (such as very old Boost) used, but isn't that great as you have to ship around an extra DLL.

Maybe 2 and 3 in combination could work? I'm not sure.

From nynaeve:

Update: Phil mentions that this support for TLS callbacks does not work before the Visual Studio 2005 release. Be warned if you are still using an old compiler package.

This KB article in the MSDN 2005 library seems to concur:

Apparently in the pre-2005 C++ reference in the MSDN Library there was this restriction:

To paraphrase page 397 of the "C++ Language Reference," class objects may be instantiated with the thread attribute if the objects do not need to be constructed or destructed.

In the one that I can see in the 2005 library, I cannot find such a restriction anymore. In fact, the [current description] seems to be basically identical to the 2005 one. Note how:

Win32 and the Microsoft C++ compiler now support statically bound (load-time) per-thread data in addition to the existing API implementation.

"now" has been in there since at least 2005, haha

Before Windows Vista, statically declared TLS data objects can be used only in statically loaded image files. This fact makes it unreliable to use static TLS data in a DLL unless you know that the DLL, or anything statically linked with it, will never be loaded dynamically with the LoadLibrary API function. However, starting with Windows Vista, improvements were made to the Windows loader to better support the dynamic loading of DLLs with static TLS. This change means that DLLs with statically declared TLS data objects can now be used more reliably, even if they are loaded dynamically using LoadLibrary. The loader is capable of allocating TLS slots for such DLLs at load time, mitigating the limitations present in earlier versions of Windows.

(ref)

This means rust9x shouldn't use native/static TLS at all, anyway, given that we want to support pre-Vista Windows.

I do want to know if a C++ program, made with VC2005, can successfully call the destructor of a declspec(thread) variable on Win98. If it does work, it's "only" a matter of figuring out how it works, hopefully?

@Jookia

@Jookia,

There's a fix for Wine soon.

Would be good to include links to these things right away. ;) Personally I, for example, hasn't succeed in finding it in Google now.

Link in a DLL that has DllMain and have it somehow call the TLS functions

You meant checking for DLL_THREAD_DETACH here, right?

@Jookia

https://www.betaarchive.com/wiki/index.php/Microsoft_KB_Archive/118816 seems to indicate that support for static TLS existed forever.

I forgot that VC ships with the CRT source code for reference: I can now confirm that VC2005 is the first version with constructor/destructor support via TLS callbacks. Given that previous versions don't seem to set up the TLS directory in code in the CRT:

_CRTALLOC(".rdata$T")
const IMAGE_TLS_DIRECTORY _tls_used =
{
        (ULONG)(ULONG_PTR) &_tls_start, // start of tls data
        (ULONG)(ULONG_PTR) &_tls_end,   // end of tls data
        (ULONG)(ULONG_PTR) &_tls_index, // address of tls_index
        (ULONG)(ULONG_PTR) (&__xl_a+1), // pointer to call back array
        (ULONG) 0,                      // size of tls zero fill
        (ULONG) 0                       // characteristics
};

I can only assume that older VC versions did that internally instead?

Now given that

rust/library/std/src/sys/thread_local/guard/windows.rs

Lines 88 to 90 in be4391a

    
           #[unsafe(link_section = ".CRT$XLB")] 
        
           #[cfg_attr(miri, used)] // Miri only considers explicitly `#[used]` statics for `lookup_link_section` 
        
           pub static CALLBACK: unsafe extern "system" fn(*mut c_void, u32, *mut c_void) = tls_callback;

really only puts the callback in the right range, that doesn't seem related to the CRT itself anymore.

I was almost ready to give up at this point, but I just noticed a tiny difference between VC2005 and modern CRT TLS callback setup:

winnt.h defines the TLS callback signature as follows:

typedef VOID
(NTAPI *PIMAGE_TLS_CALLBACK) (
    PVOID DllHandle,
    DWORD Reason,
    PVOID Reserved
    );

Modern CRT signatures for the init and dtor functions seem to match

void WINAPI __dyn_tls_init(PVOID, DWORD dwReason, LPVOID) noexcept

HOWEVER: The VC2005 versions differ!

BOOL WINAPI __dyn_tls_init(
    HANDLE  hDllHandle,
    DWORD   dwReason,
    LPVOID  lpreserved
    )

And even document the return value:

 * Exit:
 *      Returns TRUE always, though the loader code ignores the result.

Hypothesis: The NT loader ignores the result, but the 9x loader might not! Because the rust tls_callback uses the modern/official signature, EAX/retval might end up 0. Could it be that the 9x loader stops processing the callback array if any of the callbacks returns FALSE/0?

Update: Hm, in a random rust9x_sample build from 1.95 there is only the rust tls_callback registered. Unless 9x tracks that it returned FALSE/0 on dw_reason != c::DLL_THREAD_DETACH && dw_reason != c::DLL_PROCESS_DETACH, that also doesn't seem to be the culprit... Maybe it's time to reverse the 9x loader to find out more 😅

@Jookia I did realize something else while thinking through this: The InitOnce emulation alternative you added might actually not work correctly. The Rust9x Mutex depends on compat::mutex_kind:

rust/library/std/src/sys/pal/windows/compat/checks.rs

Lines 72 to 93 in be4391a

    
           #[inline(always)] 
        
           pub(crate) fn mutex_kind() -> MutexKind { 
        
               unsafe { MUTEX_KIND } 
        
           } 
        
           fn init_mutex_kind_check() { 
        
               let kind = if c::TryAcquireSRWLockExclusive::available().is_some() { 
        
                   MutexKind::SrwLock 
        
               } else if { 
        
                   // Windows 9x exports `TryEnterCriticalSection`, but it returns ERROR_CALL_NOT_IMPLEMENTED. 
        
                   // MSDN specifies that the function is available on NT4 and later. 
        
                   is_windows_nt() && c::TryEnterCriticalSection::available().is_some() 
        
               } { 
        
                   MutexKind::CriticalSection 
        
               } else { 
        
                   MutexKind::Legacy 
        
               }; 
        
               unsafe { 
        
                   MUTEX_KIND = kind; 
        
               } 
        
           }

Which is called from within here:

rust/library/std/src/sys/pal/windows/compat.rs

Lines 77 to 90 in be4391a

    
           #[cfg(target_family = "rust9x")] 
        
           unsafe extern "C" fn init() { 
        
               // In an exe this code is executed before main() so is single threaded. 
        
               // In a DLL the system's loader lock will be held thereby synchronizing 
        
               // access. So the same best practices apply here as they do to running in DllMain: 
        
               // https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices 
        
               // 
        
               // DO NOT do anything interesting or complicated in this function! DO NOT call 
        
               // any Rust functions or CRT functions if those functions touch any global state, 
        
               // because this function runs during global initialization. For example, DO NOT 
        
               // do any dynamic allocation, don't call LoadLibrary, etc. 
        
               checks::init_rust9x_checks(); 
        
           }

Which uses CRT pre-main initialization:

rust/library/std/src/sys/pal/windows/compat.rs

Lines 46 to 49 in be4391a

    
           #[cfg(any(target_vendor = "win7", target_family = "rust9x"))] 
        
           #[used] 
        
           #[unsafe(link_section = ".CRT$XCT")] 
        
           static INIT_TABLE_ENTRY: unsafe extern "C" fn() = init;

Given that this is CRT-based initialization, and TLS callbacks are system/loader-run, I think the TLS callback is executed first? This means that MUTEX_KIND is still set to Legacy, and will then change afterwards, likely causing havoc? Maybe we should instead use a OnceBox directly, regardless of MUTEX_KIND, for this.

@Jookia

@Jookia ah, nvm, it should be fine since rust thread locals that aren't const-initialized are lazy-initialized instead, which would only happen from regular user code, so the CRT-based init would already have been done :)

@Jookia,

There's a fix for Wine soon.

Would be good to include links to these things right away. ;) Personally I, for example, hasn't succeed in finding it in Google now.

Here we go: https://gitlab.winehq.org/wine/wine/-/merge_requests/7251 It seems to have stalled though. Maybe you could nudge them about it?

Link in a DLL that has DllMain and have it somehow call the TLS functions

You meant checking for DLL_THREAD_DETACH here, right?

Yep!

Thanks for the check on the Mutex, I don't fully understand it but looking at what you linked and your explanation it makes sense.

Yeah overall it seems static TLS is a no-go for pre-Vista Windows for DLLs and and it just kind of happens to maybe work for executables. So we need to find a way to disable static TLS on all compilers and run destructors at some other point?

Though we could say 'Rust9x doesn't support making DLLs this is a Windows bug'?

I do think it's worthwhile trying to see if Windows 9x does have SOME functionality for this? It seems kind of weird to include it in NT but not 9x given it's just some callbacks, right? ...Right?

Well, I guess it's time to test to see if Visual Studio will give us our beloved thread-local destructors.

According to MSDN for VS6 the .tls section is supported by Windows NT, with pointers of type PIMAGE_TLS_CALLBACK.

According to Visual Studio, I can't have a destructor or constructor on _declspec(thread) (error C2483) which is apparently fixed in Visual Studio 2015 and later.

So I opened up Visual Studio 6 and dumped in this code:

#include <stdafx.h>
#include <windows.h>

void NTAPI tls_callback(PVOID DllHandle, DWORD dwReason, PVOID unused)
{
    if (dwReason == DLL_THREAD_ATTACH)
    {
        MessageBoxW(0, L"DLL_THREAD_ATTACH", L"DLL_THREAD_ATTACH", 0);
    }
  
    if (dwReason == DLL_PROCESS_ATTACH)
    {
        MessageBoxW(0, L"DLL_PROCESS_ATTACH", L"DLL_PROCESS_ATTACH", 0);
    }
}

#if 1 // NT code
ULONG _tls_index;

static const PIMAGE_TLS_CALLBACK g_tls_cb[] = { tls_callback, 0 };

#pragma const_seg(".rdata$T")
typedef unsigned long *ULONG_PTR;
EXTERN_C const IMAGE_TLS_DIRECTORY _tls_used = {
    NULL,         // Start of TLS data
    NULL,        // End of TLS data
    &_tls_index, // Address of tls_index
    (PIMAGE_TLS_CALLBACK*)&g_tls_cb,    // Pointer to call back array
};
#pragma const_seg()
#pragma comment (linker, "/INCLUDE:__tls_used")

#else // CRT code

#pragma comment (linker, "/INCLUDE:__tls_used")
#pragma comment (linker, "/INCLUDE:_tls_callback_func")
#pragma data_seg(".CRT$XLF")
EXTERN_C PIMAGE_TLS_CALLBACK tls_callback_func = tls_callback;
#pragma data_seg()

#endif
  
DWORD WINAPI ThreadProc(LPVOID lpParam) 
{
    ExitThread(0);
	return 0;
}
  
int main(void)
{
    CreateThread(NULL, 0, &ThreadProc, 0, 0, NULL);
    Sleep(1000);
    return 0;
}

The NT branch works on NT, the CRT branch doesn't seem to work with Visual Studio 6 at all, and I didn't want to spend time trying to set up a platform SDK for Visual Studio 2005 so I haven't tested it there but I presume it works. It just seems to be a wrapper around the NT callbacks anyway, which does the same thing. I did a lot of searching online and grepping symbols and I really don't think 9x will do TLS callbacks.

I'm not sure how much we need this functionality if we control the language runtime like we do with Rust. We control when threads launch and how and when they exit (cleanly) and how. I think we should next investigate cleaning up Rust's TLS storage when the a 'Rust' thread exits.

So we need to find a way to disable static TLS on all compilers and run destructors at some other point?

that's just setting has_thread_local: false for the msvc r9x targets, i think.

Though we could say 'Rust9x doesn't support making DLLs this is a Windows bug'?

At least I'd be more specific and document the exact limitation. DLLs are totally fine, unless you use static TLS, then it's Vista+. But we can just disable static tls support instead and not add extra limitations where we don't have to :)

The NT branch works on NT

there is nothing specifically NT about that branch beyond the NTAPI typedef. Does that example work on 9x?

the CRT branch doesn't seem to work with Visual Studio 6 at all

yup, like i mentioned, only starting with VC2005 does the CRT set up the IMAGE_TLS_DIRECTORY for you. Before that you either have to do it yourself or maybe the compiler or linker can do it magically otherwise.

I did a lot of searching online and grepping symbols and I really don't think 9x will do TLS callbacks.

__declspec(thread) itself is already used in samples in the NT4/Win95 SDK. So at least start/end of tls data in the IMAGE_TLS_DIRECTORY must necessarily be supported to some degree by 9x? Or do the old CRTs manually register the 'static' TLS entries dynamically at CRT init time via TlsAlloc (which definitely is supported)?

We control when threads launch and how and when they exit (cleanly) and how.

we don't, necessarily. you can just as well call CreateThread manually like you do in the C code, and ideally it'll still just work.

I think we should next investigate cleaning up Rust's TLS storage when the a 'Rust' thread exits.

https://github.com/rust-lang/rust/blob/63f05e3635171e7ac3f9ca78bad6c71052cda5a3/library/std/src/thread/lifecycle.rs#L66-L83 (and ideally also in the panic/unwind case)

Disabling static TLS support only works on the Rust side, you can still link in C programs that use static TLS. Same with their own thread management. We can't do much to deal with C programs that expect different behaviour of the OS here (TLS destructors, static TLS in DLLs), the most we can do is make Rust code not fail in these situations.

9x does not call the callback, no. I believe the old CRTs initialize TLS keys at startup without the kernel doing anything.

Disabling static TLS support only works on the Rust side, you can still link in C programs that use static TLS.

right we can't magically make incompatible C code compatible, but we can make rust code compatible by disabling static TLS.

9x does not call the callback, no. I believe the old CRTs initialize TLS keys at startup without the kernel doing anything.

seems like an interesting rabbit hole.

(we could potentially hook/detour CreateThread with a custom thread start function that runs the destructor on cleanup 👺)

(we could potentially hook/detour CreateThread with a custom thread start function that runs the destructor on cleanup 👺)

Better write a full-fledged user VxD then...

Now I suddenly recalled once reading an article about TLS callbacks not being run due to some peculiarity in Windows loader:
https://www.virusbulletin.com/virusbulletin/2010/05/anti-unpacker-tricks-part-eight#id2360912

links mentioned there so you won't need to find them again

https://web.archive.org/web/20060211061933/http://blogs.msdn.com/mgrier/archive/2005/06/24/432455.aspx#433355
https://web.archive.org/web/20161225110326/http://nezumi-lab.org/blog/?p=15
https://web.archive.org/web/20161225102924/http://nezumi-lab.org/blog/?p=43
https://web.archive.org/web/20161225095850/http://nezumi-lab.org/blog/?p=51

This one is about NT, of course, but they shared a lot of code with 9x line back then.

Would a Unicows-style linker thunk work for this? Maybe we could hook ExitThread on 9x to go through TLS destructors.

This might be of interest to mingw-lite too which has its own thunking system, so it would be good to have a way to avoid clobbering with a potential thunk of its own.

Maybe we could hook ExitThread on 9x to go through TLS destructors.

most thread exits don't actually use ExitThread

Ah, so hooking Create thread and maybe ExitThread? ... That sounds a bit tricky to ensure we don't double free thread-local data...

Would we have to iterate through all open PE sections and call their TLS callbacks?

with native TLS disabled, I don't think there is much more to do here beyond nasty hacks on 9x. I verified that TLS callbacks work allthe way down to NT3.1, just 9x doesn't support them, as documented in the Limitations section

Navigation Menu

Repository navigation

Investigate issues with thread locals #12

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	#[unsafe(link_section = ".CRT$XLB")]
	#[cfg_attr(miri, used)] // Miri only considers explicitly `#[used]` statics for `lookup_link_section`
	pub static CALLBACK: unsafe extern "system" fn(mut c_void, u32, mut c_void) = tls_callback;

	#[inline(always)]
	pub(crate) fn mutex_kind() -> MutexKind {
	unsafe { MUTEX_KIND }
	}

	fn init_mutex_kind_check() {
	let kind = if c::TryAcquireSRWLockExclusive::available().is_some() {
	MutexKind::SrwLock
	} else if {
	// Windows 9x exports `TryEnterCriticalSection`, but it returns ERROR_CALL_NOT_IMPLEMENTED.
	// MSDN specifies that the function is available on NT4 and later.
	is_windows_nt() && c::TryEnterCriticalSection::available().is_some()
	} {
	MutexKind::CriticalSection
	} else {
	MutexKind::Legacy
	};

	unsafe {
	MUTEX_KIND = kind;
	}
	}

	#[cfg(target_family = "rust9x")]
	unsafe extern "C" fn init() {
	// In an exe this code is executed before main() so is single threaded.
	// In a DLL the system's loader lock will be held thereby synchronizing
	// access. So the same best practices apply here as they do to running in DllMain:
	// https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices
	//
	// DO NOT do anything interesting or complicated in this function! DO NOT call
	// any Rust functions or CRT functions if those functions touch any global state,
	// because this function runs during global initialization. For example, DO NOT
	// do any dynamic allocation, don't call LoadLibrary, etc.

	checks::init_rust9x_checks();
	}

	#[cfg(any(target_vendor = "win7", target_family = "rust9x"))]
	#[used]
	#[unsafe(link_section = ".CRT$XCT")]
	static INIT_TABLE_ENTRY: unsafe extern "C" fn() = init;

Investigate issues with thread locals #12

Description

Activity

seritools commented on Dec 2, 2024

Jookia commented on Jun 14, 2026

seritools commented on Jun 14, 2026

Jookia commented on Jun 28, 2026

seritools commented on Jun 28, 2026

Jookia commented on Jun 28, 2026

Jookia commented on Jun 28, 2026

seritools commented on Jun 28, 2026

cher-nov commented on Jun 28, 2026

seritools commented on Jun 28, 2026

seritools commented on Jun 28, 2026

Jookia commented on Jun 29, 2026

Jookia commented on Jun 29, 2026

seritools commented on Jun 29, 2026

Jookia commented on Jun 29, 2026

seritools commented on Jun 29, 2026

cher-nov commented on Jun 29, 2026

Jookia commented on Jun 30, 2026

seritools commented on Jun 30, 2026

Jookia commented on Jun 30, 2026

seritools commented on Aug 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions