Skip to content

fix: stop audio transceivers with appropriate stress test - #1105

Open
pblazej wants to merge 11 commits into
mainfrom
blaze/audio-transceiver
Open

pblazej wants to merge 11 commits into
mainfrom
blaze/audio-transceiver

Conversation

@pblazej

@pblazej pblazej commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Resolves #1104

@pblazej

pblazej commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

@davidliu @hiroshihorie I'm trying to create a counter-example for that now (something that crashed WebRTC before).

pblazej and others added 5 commits September 2, 2026 13:27
Unify the two E2E cycle tests into one Scenario axis (audio, video, both)
so every case runs the same code path, and always unpublish via
unpublishAll(): for one track it is the same _unpublish path as
unpublish(publication:), and for .both it races two transceiver stops
against one debounced renegotiation — the shape reported in
webrtc-sdk/webrtc#194 that actually crashed in the field.

Drop the subscriber round-trip. It depended on real playout and on a
pre-existing m-line ordering bug when an audio+video pair is re-published,
neither related to transceiver release; the suite went 58s -> 9s.

Verified as a real detector: with the pre-fix video-only condition
restored, exactly the four audio-involving cases fail with 10 leaked
transceivers, and against a pre-webrtc-sdk#194 binary (137.7151.04) the
deterministic test aborts for both media kinds with
"Check failed: !channel_ / Missing call to ClearChannel?".

Add MicrophoneTransceiverReleaseTests for the real audio device module,
which TestAudioTrack bypasses. Gated on microphone permission so hosts
without it skip cleanly instead of failing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…alone

Two defects in the single PC offer path, both surfaced by stopping audio
transceivers (which makes rejected m-sections common where they were rare).

`SDPMediaSection.isRTP` only checks the protocol field, so a rejected
section (`m=audio 0 ... RTP/...`, which is how a stopped transceiver
serializes) still matched, and `mungeInactiveToRecvOnlyForMedia` rewrote its
`a=inactive` to `a=recvonly` — resurrecting an m-section the client had just
killed. Add `SDPMediaSection.isRejected` (port 0, RFC 3264 8.2) and skip
those sections.

`set(localDescription:munging:)` caught every error and retried with one
fewer munge, all the way down to the unmunged original. Any
setLocalDescription failure — including ones munging did not cause — could
therefore discard the direction rewrite that single PC needs to receive
media, leaving a silently deaf connection. Split the parameter into
required `munging:` and `droppable:`; once the droppable ones are gone the
error propagates. The answer path's stereo and nack preferences are both
droppable; single PC keeps the direction rewrite and drops only stereo.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The field repro looped unbounded; the test fixed it at 10 so CI stays fast.
Read LK_SOAK instead, defaulting to 10, so the same test reproduces the
long-running shape on demand:

    TEST_RUNNER_LK_SOAK=200 xcodebuild test -scheme LiveKit \
        -only-testing LiveKitCoreTests/TransceiverReleaseTests -destination 'platform=macOS'

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop the LK_SOAK knob and just do the long run every time: the field repro
(webrtc-sdk#194) looped unbounded, and 600 cycles across the six cases cost
~22s, so there is no reason to make it opt-in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cycle test created one track per kind and republished those same objects,
which is issue #1104's repro shape, not the mic+camera loop from
webrtc-sdk#194 that appeared to defeat the ClearChannel fix. Those differ:
`unpublishAll()` leaves no publication, so the next setMicrophone/setCamera
takes `set(source:enabled:)`'s create-and-publish branch instead of unmuting,
churning a new media source and track every iteration rather than only a
transceiver.

Create both tracks fresh per cycle. A new buffer track has no dimensions and
`_publish` waits on them before starting the capturer, so feed it and await
the completer first. Same treatment in the ADM test, whose feeder cancel also
moves into a `defer` so a throw cannot leak a 30 fps task.

Verified 800 publishes per run (400 audio, 400 video) across the six cases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pblazej

pblazej commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Stopping audio transceivers is safe because of webrtc-sdk/webrtc#194 ("Ensure ClearChannel is called"), first shipped in 137.7151.05: the crash that forced the original video-only restriction was RTC_CHECK(!channel_) << "Missing call to ClearChannel?" in ~RtpTransceiver — kind-agnostic, not the audio stack — and the report that it still reproduced afterwards was measured against the unpatched 137.7151.04 binary (confirmed by the crash report's image UUID). The new TransceiverReleaseTests.stopDuringNegotiationDoesNotCrash guards it by scripting that exact interleaving, verified to abort on 137.7151.04 and pass on the current pin.

@pblazej
pblazej marked this pull request as ready for review September 14, 2026 08:50

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

pblazej and others added 3 commits September 15, 2026 08:29
The SFU answers a rejected `AddTrack` with a `RequestResponse` that echoes the
request rather than carrying a request id, and the handler only looked at the
id-keyed data-blob completers. The publish therefore sat on its completer for
the full 10s and failed with a bare "Timed out" while the server had already
said why. Correlate on the echoed `cid` and fail that add-track completer.

This is how the new cycle test presents on a current server: it caps a
participant at 20 pending tracks and only clears one when the track's media
arrives, so a loop that unpublishes before any RTP flows (and TestAudioTrack
never produces any) leaks a slot per publish and is rejected with
LIMIT_EXCEEDED. Budget the cycles per scenario to stay under that cap —
`.both` publishes two tracks per cycle, so it gets half as many.

Only reproducible against livekit-server >= 1.13.6; 1.13.5 has no such limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

pblazej and others added 2 commits September 15, 2026 09:35
Completer entries live until disconnect and a resolved AsyncCompleter serves
its cached result to every later wait, so reusing a CID replayed the previous
attempt's outcome: a rejected publish failed the retry instantly while the
request it had just sent still reached the SFU, which could then accept a
track the client had already abandoned. The same held for a cached success,
which returned a stale TrackInfo for a track republished under the same CID.

Rearm the entry as part of getting it, so each attempt waits for its own
response.

Reported by Devin on #1105.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pull the fixes that fell out of testing this change back out so the PR
carries one thing: stopping audio transceivers on unpublish, and the tests
that prove it. Each removed fix is real and gets its own PR from
blaze/sdp-munge-and-addtrack-fixes:

- SDP: skip rejected (port 0) sections in the single PC direction rewrite,
  and never drop a required munge on setLocalDescription failure
- Signal: fail a server-rejected AddTrack immediately with its reason
  instead of waiting out the publish timeout
- Signal: rearm the add-track completer per attempt so a reused CID does
  not replay the previous outcome

The test cycle budget from those investigations stays: it is what keeps the
suite under the SFU's 20 pending-track cap on livekit-server >= 1.13.6.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[iOS] Audio transceivers are never stopped/released on unpublish (Transport.releaseTransceiver only handles .video)

1 participant