Skip to content

Android: inproc stress test deadlocks (API 24-27) #1578

Closed
#1579
Closed
Android: inproc stress test deadlocks (API 24-27)#1578
#1579

Description

@jpnurmi
jpnurmi
opened on Mar 14, 2026

Random CI hangs occurred while setting up the Android API 26 CI job in #1574 / #1572. I was able to reproduce the deadlock locally using API 24-27 emulators.

test:

$ ANDROID_API=26 pytest -v -s tests/test_inproc_stress.py
[...]
tests/test_inproc_stress.py::test_inproc_concurrent_crash_repeated[1]
[...]
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /private/var/folders/91/ctzy99m94bb6v2k1_37l5x140000gn/T/pytest-of-jpnurmi/pytest-390/cmake0/inproc_stress_build
[100%] Built target inproc_stress_test

process status:

$ adb shell ps -A | grep inproc
shell        14164 14162   34032   3628 futex_wait_queue_me 70d30bd7f0 S inproc_stress_test

stacktrace:

     * thread #1
       * frame #0: 0x00000070d30bd7ec libc.so`syscall + 28
         frame #1: 0x00000070d3107d1c libc.so`pthread_join + 212
         frame #2: 0x00000070d33b3078 inproc_stress_test`run_concurrent_crash at concurrent_crash.c:114:9
         frame #3: 0x00000070d33b28c4 inproc_stress_test`test_concurrent_crash(database_path="/data/local/tmp/.sentry-native-1") at main.c:236:5
         frame #4: 0x00000070d33b27a0 inproc_stress_test`main(argc=3, argv=0x0000007ff56a74a8) at main.c:475:16
         frame #5: 0x00000070d3141440 libc.so`__libc_init + 92
         frame #6: 0x00000070d33b25ec inproc_stress_test`_start_main + 88
       thread #2
         frame #0: 0x00000070d310a810 libc.so`read + 4
         frame #1: 0x00000070d2f15e48 libsentry.so`handler_thread_main [inlined] read(fd=<unavailable>, buf=0x00000070d29b4444, (null)=1, count=1) at unistd.h:190:12
         frame #2: 0x00000070d2f15e44 libsentry.so`handler_thread_main(UNUSED_data=<unavailable>) at sentry_backend_inproc.c:1214:22
         frame #3: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #4: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #3
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #4
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #5
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #6
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #7
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #8
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #9
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #10
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #11
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #12
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #13
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #14
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #15
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #16
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #17
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #18
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #19
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #20
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #21
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #22
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72

Activity

  1. linear-code commented on Mar 14, 2026

    @linear-code
    linear-codebot
    on Mar 14, 2026 – with Linear Code

    NATIVE-173

  2. jpnurmi commented on Mar 14, 2026

    @jpnurmi
    jpnurmi
    on Mar 14, 2026
    CollaboratorAuthor

    AI-assisted analysis:

    Summary

    When multiple threads crash simultaneously on Android API 24–27, the process hangs indefinitely instead of terminating. The deadlock occurs in bionic's debuggerd_signal_handler, which uses a non-reentrant mutex that cannot handle concurrent entry from multiple threads.

    Root Cause

    After the inproc backend processes a crash, all crashing threads fall through to cleanup: in process_ucontext() (src/backends/sentry_backend_inproc.c), where they call reset_signal_handlers() and invoke_signal_handler(). This causes all threads to enter bionic's debuggerd_signal_handler simultaneously, deadlocking on its internal crash_mutex.

    The deadlock has two contributing factors:

    1. Mutex contention: debuggerd_signal_handler acquires crash_mutex and intentionally never releases it for fatal signals. Other threads block on pthread_mutex_lock forever.

    2. Race condition: The winning thread's debuggerd_signal_handler calls signal(SIGSEGV, SIG_DFL) before re-raising, but a losing thread's reset_signal_handlers() can re-install debuggerd_signal_handler via sigaction() in between, causing the winner to re-enter and deadlock on its own mutex.

    Our SA_NODEFER flag and empty sa_mask exacerbate the issue — debuggerd_signal_handler assumes all signals are blocked during its execution (bionic registers it with sigfillset(&action.sa_mask)), but our handler replaces that mask.

    Fix Proposal

    Only the winning thread (the one that won the CAS IDLE→HANDLING) should call reset_signal_handlers() and invoke_signal_handler(). Losing threads should set signal(signum, SIG_DFL) and return, so their re-fault hits the default handler instead of debuggerd_signal_handler.

    Notes

    It's unclear why the deadlock doesn't reproduce on API 28+. The debuggerd_signal_handler source code between API 27 and 28 shows no obvious fix for this issue — both versions use the same crash_mutex with the same "don't unlock for fatal signals" strategy. strace comparison of successful runs shows identical rt_sigaction sequences, with the only visible difference being one fewer signal(SIGSEGV, SIG_DFL) call on API 28. The race window may simply be narrower on API 28 rather than eliminated.

  3. supervacuus commented on Mar 14, 2026

    @supervacuus
    supervacuus
    on Mar 14, 2026
    Collaborator

    Thanks for raising @jpnurmi. The analysis is extremely long-winded for essentially a clean-up bypass. The interactions vs the debuggerd_signal_handler sound a lot like LSD consumption. But yeah, no reason to re-raise specifically from within our handler (however, it is absolutely possible that those crashes hit again, after the return, which is also why I do not buy that sketched interaction model).

  4. supervacuus
    closed this as completedin #1579on Mar 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions