Fix/1174 coreaudio startstop deadlock

[robsussman]

Fixes #1174

Problem

startStopCallback, the kAudioOutputUnitProperty_IsRunning property listener in pa_mac_core.c, calls AudioUnitGetProperty. CoreAudio can deliver that notification synchronously on the HAL IO thread while holding the HAL IOContext mutex, and AudioUnitGetProperty needs the AudioUnit instance mutex — which a concurrent Pa_StopStream already holds inside AudioOutputUnitStop while it waits for that same IOContext mutex. This AB-BA inversion permanently wedges the thread calling Pa_StopStream. See #1174 for the full analysis, stacks, and a standalone stress repro (typically deadlocks within a few hundred open/start/stop/close cycles of a full-duplex stream on distinct input/output devices).

Fix (two commits)

1. Break the lock cycle. The listener infers the stop transition from stream->state instead of calling back into the AudioUnit: StopStream/AbortStream set STOPPING before stopping the units and StartStream sets ACTIVE before starting them, so the state filters the same transitions the IsRunning query did.

2. Restore the stream-finished-callback delivery the minimal fix dropped (paComplete/paAbort self-stops and spontaneous device stops). The listener now does no work at all: it pokes a per-stream GCD dispatch source (dispatch_source_merge_data is lock- and allocation-free, safe in the locked context the notification is delivered in). The source's handler runs on a per-stream serial queue where no CoreAudio locks are held, so it can safely query IsRunning to distinguish starts from stops, and fires the finished callback for stops that happen while ACTIVE or CALLBACK_STOPPED. Normal StopStream/AbortStream no longer depend on the notification: FinishStoppingStream fires the callback once the units are confirmed stopped — matching how other host APIs (skeleton, ALSA) invoke it from a PortAudio-owned non-realtime context rather than from an OS callback. This also stops user callback code from running inside CoreAudio's locked notification context, which was its own re-entrancy hazard. An atomic once-per-start flag keeps delivery exactly-once under races between spontaneous stops, callback-requested stops, and concurrent Pa_StopStream. CloseStream cancels the source and drains the queue before disposing the units the handler may query.

Behavior preserved from the old code: a spontaneous stop leaves the stream state ACTIVE (the app still calls Pa_StopStream), and in two-unit duplex streams the output unit is the one consulted.

Verification

• The stress repro attached to macos: deadlock stopping audio #1174: unpatched deadlocks within 121–299 cycles (4/4 runs); patched survives 3000 cycles. The repro's CMakeLists can be pointed at any checkout via FETCHCONTENT_SOURCE_DIR_PORTAUDIO to reproduce/verify.
• Verified unplug of USB device fires notification properly

[philburk]

@robsussman - I notice that your test runs against the release version of PortAudio 19.7.0.
We are working on the head of the GitHub repo.
Are you seeing the same hang on the current top of tree?

When I ran my test on an M4 Mac Mini.
I ran 13700 iterations and saw no deadlock.
Maybe that is because my input device was a USB mic.

What device are you running on?
A Macbook built-in Mic may have very different timing.

[robsussman]

@philburk, good point, I just retested against master and it happened there too with both Debug and Release builds, both times in less than 300 iterations.

I'm running Tahoe 26.4 on an M5 Macbook Pro.

I tried changing the system default input and output devices to a USB audio interface (Scarlett 4i4 USB). It took longer to deadlock but still did at iteration 769.

Tested on another machine: MacBook Pro M1 running macOS Ventura 13.7.8 with built in mic and speakers. So far 4,100 iterations and no deadlock. So maybe this requires a newer macOS version to reproduce?

[philburk]

I was able to repro the deadlock on my M4 Mac Mini running Mac OS 26.6.2.
I am using PortAudio top-of-tree.

With the changes in this PR #1175, I cannot reproduce the deadlock. Yay! Thank you so much for the clear repro test program and the fix.

Our next step is to fully review the change to see if there are any unintended changes in behavior, which can cause a regression. I will run our small QA suite and the test programs.

[nat3Github]

just to add a new datapoint, i am on a macbook m1 pro and encountered deadlocks when closing my programm pretty often (especially closing when busy). i traced the hang down to a race in portaudio.

File and function: src/hostapi/coreaudio/pa_mac_core.c, startStopCallback

• Thread 1: Pa_StopStream → FinishStoppingStream → AudioOutputUnitStop. It holds the AudioUnit mutex and waits for the mutex inside HALC_ProxyIOContext::StopIOProc.
• Thread 2: HALC_ProxyIOContext::IOWorkLoop (holding the HAL mutex) → the IsRunning listener startStopCallback → AudioUnitGetProperty, which waits for the AudioUnit mutex.

[philburk]

@nat3Github - Thanks for the report. Do you think you are seeing the same bug? It certainly sounds similar.

Does this PR solve the problem for you? You can checkout the fix by doing:

git checkout master
git fetch origin pull/1175/head:pr_1175
git checkout pr_1175

[philburk]

I am testing the fix with various QA tests:

bin/paqa_errs - PASS

bin/paqa_devs hung
------ Test: OUTPUT, device = 3, #ch = 4, rate = 48000, format = 1, CALLBACK, INTER
------ Test: OUTPUT, device = 3, #ch = 4, rate = 88200, format = 1, CALLBACK, INTER
------ Test: OUTPUT, device = 3, #ch = 4, rate = 96000, format = 1, CALLBACK, INTER
HUNG HERE !!!

reran bin/paqa_devs and it PASSed except for BLOCKING, NON-INT. Known issue. See #77
reran bin/paqa_devs 3 more times and it PASSed. - cannot reproduce the first hang!

bin/paqa_latency - PASS
bin/paqa_latency_monotonic - PASS

The following were run and also sounded fine: PASS
bin/paex_sine
bin/paex_write_sine
bin/paex_record
bin/patest_multi_sine
bin/patest_leftright
bin/patest_stop_playout
bin/patest_wire

I then plugged in a Zoom AMS-22 USB device with hardware loopback support.
make loopback
bin/pa_loopback - PASS

[nat3Github]

Does this PR solve the problem for you?

yes i already applied that patch locally and so far no deadlock. thats a good sign but I cant be 100 p certain since this is a racy problem (it will hapen x times in a 100 under opaque conditions). I will report if I encounter a deadlock with this patch. and yes i think its the same bug.

[RossBencina]

The bug is serious enough that we're going to merge a fix for 19.8. Phil will do some more testing and code review. I need to be confident that the logic that detects when to fire the stream finished callback is equivalent (or better) than the old code, since the condition checks seem to differ between new and old code.

[philburk]

This is a complex bug and PR. So I asked Claude Opus 5.5 to review this PR.

It liked most of the PR but found a potential problem if an app calls Pa_CloseStream() from a PaStreamFinishedCallback. That is illegal in PortAudio. But users sometimes do illegal things and often get away with it. If we make that fatal then we could cause a regression in an app. Note that it cannot be the cause of my one observed hang because I do not use PaStreamFinishedCallback in paqa_devs.c.

@robsussman - do you see any quick fix for the issue that Claude found.

---- review from Opus 5.5 --------
Summary

I reviewed [this PR] (CoreAudio stream-finished-callback restoration). The overall approach — moving AudioUnitGetProperty queries off the locked HAL notification thread and onto a private serial dispatch_source/queue, with an atomic once-per-start flag for exactly-once delivery — is sound and correctly avoids the AB-BA deadlock the prior commit introduced. I traced through:

• The OpenStream error path (goto error → CloseStream) correctly tears down the queue/source even on partial init failure.
• The CloseStream cancel+drain sequence correctly prevents use-after-free of stream by the dispatch handler, and the if (stream->startStopSource) guard in startStopCallback prevents a crash if the HAL fires the property listener in the window between drain and actual AudioComponentInstanceDispose.
• The exactly-once semantics via OSAtomicCompareAndSwap32Barrier hold up under every StartStream/StopStream/spontaneous-stop race I could construct.
• The hardcoded unit/element selection in handleStartStopNotification mirrors a pre-existing limitation (not a regression).

One real bug found and reported via ReportFindings: Pa_CloseStream self-deadlocks (or crashes on modern libdispatch) if called from inside the user's PaStreamFinishedCallback, when that callback is invoked via the new spontaneous-stop/CALLBACK_STOPPED path. In that path the finished callback runs synchronously on stream->startStopQueue (via handleStartStopNotification → fireStreamFinishedOnce). If the callback calls Pa_CloseStream — a natural "stream's done, clean up" pattern that nothing in the docs forbids — Pa_CloseStream ends up calling dispatch_sync_f onto stream->startStopQueue from a frame that's already executing on that exact queue. This is new: the same callback fired via the normal Pa_StopStream/FinishStoppingStream path runs on the caller's own thread and has no such issue, so this hazard is specific to the mechanism this commit adds.

※ recap: Reviewed commit 2dce3bc's CoreAudio stream-finished-callback fix and found one real bug: Pa_CloseStream can deadlock if called from inside the finished callback during a spontaneous-stop. Findings reported; next step is deciding whether to fix it. (disable recaps in /config)

[philburk]

If paqa_devs hangs again I will try to get a stack dump. Claude says this is how to do it:

Two good options on macOS, in order of convenience:

1. sample (fastest, no setup needed)

• find the pid
  ps aux | grep paqa_devs

• capture stacks of every thread, 1 sample is enough for a true hang
  sample 1 -file /tmp/hang.txt
  open /tmp/hang.txt # or just cat it
  sample is built into macOS, needs no debugger attach, and dumps a per-thread backtrace. For a genuine hang (not a transient slow spot) one sample is sufficient since the stack won't be moving.

2. lldb attach (if you want to poke around or set breakpoints)
   lldb -p
   (lldb) thread backtrace all # or: bt all
   Then detach (leaves it hung, resumable) or kill/quit to clean up.

What to look for either way: check whether the main thread is inside dispatch_sync_f called from CloseStream in pa_mac_core.c, and separately whether there's another thread stuck inside AudioUnitGetProperty/handleStartStopNotification — that combination would confirm the drain-blocks-on-stuck-query theory I mentioned. If instead the hang is somewhere else entirely (e.g. in BlockWhileAudioUnitIsRunning's poll loop, or in CoreAudio's HAL framework code with no PortAudio frames at all), that points away from the CloseStream/dispatch-queue interaction.

For readable symbols: make sure the build has debug info (-g, and ideally not a fully stripped Release build) — otherwise sample/lldb will show addresses or only system-framework symbols for the PortAudio frames. A RelWithDebInfo-style CMake build is usually enough.