fix: keep device-captured audio out of sink-fed audio senders - #289
Merged
Merged
Conversation
WebRTC's AudioState starts the factory's AudioDeviceModule recording as soon as any audio send stream starts, and AudioTransportImpl fans every captured frame out to every send stream of that factory. A send stream whose track is backed by a CustomAudioSource already receives its audio through WebRTC's LocalAudioSinkAdapter on the thread that calls pushAudio(), so both feeders enter AudioSendStream::SendAudioData and its RTC_CHECK_RUNS_SERIALIZED aborts the process: Fatal error in: ../../audio/audio_send_stream.cc Check failed: !race_checker.RaceDetected() That is why single-threading pushAudio() never helped: the second feeder is WebRTC's own capture thread. Where it does not abort, microphone frames are interleaved with the pushed audio instead. PeerConnectionFactory now hands WebRTC a ProxyAudioDeviceModule that forwards every call to the wrapped module and gates its capture path. Creating an audio track from a CustomAudioSource switches that path off: InitRecording() and StartRecording() fail so AudioState never opens the device, and recorded frames that still arrive are dropped before they reach WebRTC. Playout is untouched, so received audio still reaches the speakers and AudioTrack sinks. Since a factory can send only one kind of audio input, it now commits to the kind used first and rejects the other with an IllegalStateException that names the conflict. Refs #217, #118, #213
Builds on the previous commit, which gated the device capture path so that audio pushed through a CustomAudioSource is the only thing feeding a sender. The gate now starts closed. A factory can only send device-captured audio through createAudioSource(), so gating until that call keeps captured audio away from senders without having to recognise them one by one. A server that forwards a track it received from one peer to another (issue #118) is fed the same way as a CustomAudioSource, on the thread that decodes the audio, and it is now safe without any change to the application. A factory that does both is still a conflict, so RTCPeerConnection reports a track to its factory before the track becomes a sender, and the factory rejects a pushed-audio track once it sends device-captured audio. A receive-only transceiver never sends its track and is left alone. Changing a transceiver's direction later, or swapping a track through RTCRtpSender.replaceTrack, is not covered. HeadlessAudioDeviceModule loses its capture thread. It pulled the render mix back through AudioTransport and handed it to WebRTC as captured audio, which made a peer connection send the audio it had just received straight back to the remote peer, and fed the send stream a second time alongside the pushed audio. Recording is now a state the module reports and nothing more. That capture thread was also the only thing pulling the audio mixer, which uncovered a second bug: the module hands its AudioTransport to AudioDeviceBuffer, which refuses it once playout has started, and the guide tells applications to start playout before creating a peer connection, so the render thread was pulling from nothing. Received audio then never reached an AudioTrack sink. The render thread now pulls the transport directly, and the integration test covers that ordering. CustomAudioSource.pushAudio() validates its arguments. It accepted any sample width although WebRTC reads the samples as 16-bit, read past the end of an array shorter than the frame count claimed, and passed on chunks too large for WebRTC's audio frame, which aborts the process. Refs #217, #118, #213
The previous commit made the headless module's render thread call AudioTransport::NeedMorePlayData directly, but did so with the module's state mutex held. That serialised every 10 ms mixer pull against every other call on the module, and any future path from inside the pull back into the module would have deadlocked. The render thread now decides under the state lock whether the tick pulls and snapshots the format it needs, then releases the lock before calling the transport. The transport pointer gets its own mutex, which is held for the duration of a pull, so RegisterAudioCallback() still waits for a pull in progress and no call reaches the old transport after it returns. The AudioDeviceBuffer the module carried is gone. Since the render thread pulls the transport directly and no capture thread delivers into it, it only received registrations and format updates that nothing read back. Its removal also drops the webrtc::Environment the module was constructed with, which existed solely to create the buffer.
This was referenced Sep 19, 2026
Pushed audio bypasses the audio processing module WebRTC runs on device capture, and the guide said so without saying what to do instead. It now walks through running each chunk through the standalone AudioProcessing class before pushAudio(): noise suppression and gain control first, then echo cancellation, which needs the application to feed the far-end audio through processReverseStream and set the stream delay, since only device capture gets that reference from WebRTC for free.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #217, fixes #118, fixes #213
The crash
WebRTC's
AudioStatestarts the factory'sAudioDeviceModulerecording as soon as any audio send stream starts, andAudioTransportImplfans every captured frame out to every send stream of that factory. A send stream whose track is backed by aCustomAudioSourcealready receives its audio through WebRTC'sLocalAudioSinkAdapteron the thread that callspushAudio(). Both feeders enterAudioSendStream::SendAudioData, and itsRTC_CHECK_RUNS_SERIALIZEDaborts the process:That is why single-threading
pushAudio()never helped: the second feeder is WebRTC's own capture thread. Where it does not abort, microphone frames are interleaved with the pushed audio instead. Hosts without a microphone never crash, becauseInitRecordingfails, which is why reproduction looked inconsistent. Thesinks_mutex from #285 could not address this; it protects a different list on a different path.The fix
ProxyAudioDeviceModulewraps whatever module the factory is created with and gates its capture path. The gate starts closed:InitRecording()andStartRecording()fail soAudioStatenever opens the device, and recorded frames that still arrive are dropped before they reach WebRTC. Playout is untouched, so received audio still reaches the speakers andAudioTracksinks.createAudioSource(AudioOptions)opens the gate and commits the factory to device-captured audio.createAudioTrackwith aCustomAudioSource, or adding a sink-fed track (custom or forwarded remote) as a sender, commits it to pushed audio. The other kind is then rejected with anIllegalStateExceptionthat names the conflict. Applications needing both create a second factory.RTCRtpSender.replaceTrackandRTCRtpTransceiver.setDirectionare not covered by the guard.HeadlessAudioDeviceModuleloses its capture thread, which pulled the render mix back and handed it to WebRTC as captured audio: an echo to the remote peer and a second feeder on the same send stream. Recording is now a state the module reports and nothing more. This exposed a second bug: the module handed its transport toAudioDeviceBuffer, which refuses it once playout has started, so in the documented ordering received audio never reached a sink. The render thread now pulls the transport directly, without holding the module's state lock across the pull, and the buffer is removed.pushAudio()validates its arguments. It accepted any sample width although WebRTC reads 16-bit, read past the end of an array shorter than the claimed frame count, and passed on chunks too large for WebRTC's audio frame, which also aborts the process.Docs
The custom audio source, headless audio and audio devices guides are updated: the one-input-kind-per-factory rule, 16-bit 10 ms frames from a single thread, the
kDummyAudiolimitations, and a note that pushed audio bypasses the audio processing module.Testing
mvn verifyis green at 158 tests. New tests cover every conflicting order of the latch,pushAudiovalidation, and an extendedHeadlessADMIntegrationTestthat pushes for two seconds while the module's recording is running. The touched test classes also pass under-Pjni-check. The concurrent-feeder abort itself cannot be reproduced in CI without a microphone; on a host with one,WebClientExamplecrashes before this change and must not afterwards.