WIP: Fix data track saturation shutdown deadlock

[alan-george-lk]

Context

In the C++ SDK, we have the following test: https://github.com/livekit/client-sdk-cpp/blob/e6e191b57e1209f2065068f697214235372fb842/src/tests/integration/test_data_track.cpp#L522

This test floods 256 track publishes. The test can race/deadlock on shutdown, where the shutdown event can be lost in the noise as it's not on a separate path, causing failures on nightly runs: https://github.com/livekit/client-sdk-cpp/actions/runs/35841426216/job/107117150642#step:25:3668

Fix

A quick fix is to increase the buffer size. A longer term fix is an out-of-band shutdown signal/event.

Testing

Extensive tests across nightly test runs (which flagged this) as well as a one-off workflow that just ran this flaky test: https://github.com/livekit/client-sdk-cpp/actions/runs/35954130863

Impact

LLM analysis on the memory footprint:

This is an upper bound, not a fixed-size allocation. Each Room has four event channels that use this constant (local inbound, local outbound, remote inbound, remote outbound). Tokio's bounded channel does not reserve capacity slots when the room connects. It keeps a permit count, then allocates storage in blocks of 32 slots as items are queued and frees those blocks once they drain. An empty channel costs about the same at 512 as at 16.

The numbers below are the cost only if that channel is completely full of the heaviest thing it can hold.

Queue	What one queued item owns	Full at 16	Full at 512
Local outbound	One frame, split into packets (PacketsAvailable)	~3 MiB at the test's 192 KiB frames	~96 MiB
Remote inbound	One received packet (PacketReceived), at most the 16,000-byte MTU	~250 KiB	~8 MiB
Local inbound and remote outbound	A publish or unpublish event, mostly a track handle	tens of KiB	tens of KiB

The disconnect deadlock fills the last row: one small unpublish event per track, on the room's single channel. A few hundred of those stay under a megabyte at 512.

The larger ceiling is the packet queues. Local frame sends use try_send, so a full outbound channel drops the frame instead of blocking. 512 is how many undrained frames can sit there before drops start, instead of 16. Remote inbound can retain 512 packets instead of 16.

Per-track buffers are unchanged. Each published track still queues 16 frames, and each subscription still queues 16 packets. Those are separate from EVENT_BUFFER_COUNT.

[github-actions]

Changeset ✓

This PR includes a changeset covering all affected packages:

Package	Bump
livekit	patch
livekit-capture	patch
livekit-datatrack	patch
livekit-ffi	patch
livekit-uniffi	patch

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🔴 Full output queue stalls room disconnect

During disconnect, Manager can block after emitting more than 512 events. The room's forward task stops reading output, so close waits forever for the blocked manager.

Learn more

Output sends await room-side consumption. On disconnect the room stops forwarding output before joining managers. More than 512 queued events block a manager inside a send, preventing it from reading Shutdown. The same failure applies to the local manager.

Example: A remote update with 513 new tracks fills the output queue after close stops forwarding. The manager blocks sending the last event and room close never finishes.

Recommended fix: Give the manager a separate shutdown signal and make output sends cancellable, or drain output until the manager has exited. Cover full output queues in shutdown tests.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[1egoman]

This generally makes sense to me as a workaround for now. I'm assuming that the increase in memory size from 16 -> 512 is fairly negligible given the platforms this would be running on.

As a FYI @ladvoc and I discussed this exact situation during the initial implementation of the livekit-datatrack crate and opted to hold off at the time and do it as a follow up if there was a good reason. So I think the long term approach has general buy in.