Delay late-arriving video into pacing instead of dropping

CLAUDE.md

@@ -129,6 +129,7 @@ Buffer regulation happens through playback speed only, asymmetric like IRLToolki
 | `timing.rs` | Output-clock arithmetic: next timestamp, lead, expected samples, soft compensation, prime threshold. |
 | `dsp.rs` | Fades, shaped concealment silence, last-sample memory. |
 | `video_time.rs` | Mapping video PTS through the audio playout offset, the fallback anchor and its clamps, the frame-interval EMA. |
+| `video_delay.rs` | The standing video delay: sized from the worst arrival margin (due time minus packet arrival) so every frame is in hand a delivery lead early, raised at once before the play head is anchored and only for a shortfall that recurs across a window after it, capped, never lowered within a connection. |
 | `url_opts.rs` | The demuxer option table (probe sizes, SRT latency, RIST/UDP buffers, `tls_verify=0`), parsing of the user's FFmpeg Options, and `url_awaits_caller`, which decides whether the I/O stall deadline applies before a connection exists. |
 | `stats.rs` | `FIELDS`, `StatsSnapshot`, `proc_declaration()`. |
 | `config.rs` | `HwDecode`, `Watermarks::derive`. |
@@ -170,6 +171,8 @@ Four threads. The C plugin enforced its lock contract by convention and a debug-
 
 Frames are handed to libobs a couple of canvas ticks *before* their due time. libobs is a scheduler too — `ready_async_frame` advances its play head by wall-clock deltas and takes the frame it has just passed — so a frame handed over exactly at its due time is not queued yet when its render tick runs and slips to the next one, which at 30fps on a 60fps canvas is visible judder. The frame keeps its due time as its timestamp, so the lead changes when libobs *receives* it, not when it is shown. The exception is the frame that re-anchors libobs's play head after a start or a clear: `get_closest_frame` displays that one on arrival whatever its timestamp and anchors from it, so a lead there would run the whole connection early. That frame goes at its due time, and the anchor only clears once a frame libobs actually received went out at its real due time. And while an audio stream is present, that frame's due time has to come from the *audio mapping*, not the video-only fallback: the two disagree by ~100 ms (the fallback schedules the first frame one Target Buffer out; the mapping puts it at the first audio chunk, a prime threshold plus a chunk after the first *kept* audio), which way depends on whether the audio warm-up or the first keyframe won, and libobs freezes whichever error the anchor frame carried into the connection's lip sync. So video holds until audio primes (the pump wakes the video thread the moment it publishes the mapping), drops anything the mapping lands more than a canvas tick in the past, and anchors on the first frame that is on time. A stream whose audio never primes is let through on the fallback after `VIDEO_ANCHOR_WAIT_MARGIN_MS` past the expected prime.
 
+The lead only exists for a frame that is in hand a lead before it is due. How early a frame is in hand — its *arrival margin*, due time minus the OBS time its packet reached the video thread (`TimedPacket::received_ns`) — is the sender's to set: video that leaves the encoder later than the audio of the same instant has that much less margin, and once the skew exceeds what Target Buffer plus the audio output lead covers, every frame is late. A late frame goes out on arrival, unpaced, and libobs then drops one whenever two arrive inside a canvas tick, which is what a stream with trailing video looks like: low fps. `irl_core::video_delay` closes that gap with a standing delay on every video due time, sized from the worst margin seen so that frames are a full lead early again. Before the anchor a shortfall raises it on the spot (nothing has been shown; `VideoThread::settle_anchor_candidate` does this while it also drops the stale backlog). After the anchor a raise moves the picture, so it takes `VIDEO_DELAY_MIN_FRAMES` late frames spread across `VIDEO_DELAY_WINDOW_MS`, and the delay is capped at `VIDEO_DELAY_MAX_MS` and never lowered within a connection. It is a fixed, known lip-sync error in return for a smooth picture: `video_delay_ms` reports it, and the warning that sets it says how much more Target Buffer would remove it. A clear and a decoder handover reset it. The video-only fallback schedules its first frame at arrival, so a stream without audio always carries a delay of exactly one lead.
+
 Video decode is on the video thread and not the receiver for two reasons, and the second is the load-bearing one. Decoding eagerly would mean holding the stream's whole latency as decoded frames — 8s of 4K60 is ~6GB — where the same 8s of packets is ~20MB. And the receiver spends a network stall blocked in `av_read_frame`, which is exactly when video must keep draining the buffer it already has, so the thread that decodes cannot be the thread that reads.
 
 Lock order, and the whole of it: **`audio_state` → `audio_buf` → `hot.watermarks`.** `video.q` is never held together with any of them. The audio pump takes `audio_state` exactly once per iteration and passes `&mut AudioState` down, so nothing below it can take it again — parking_lot mutexes are not recursive, and a nested acquire would hang the audio thread and then the video thread behind it.
@@ -226,7 +229,7 @@ Releases are tag driven (`.github/workflows/release.yml`, see `RELEASING.md`). P
 The port is behaviour-identical except for these, which are intentional:
 
 1. The dead `network_buffer_mb` setting is gone. Nothing read it; the transport buffer is `irl_core::consts::NETWORK_BUFFER_MB`.
-2. The `video_decoder_flushes` stat is gone (it was always 0 after the video decoder stopped being flushed). 27 stat fields remain.
+2. The `video_decoder_flushes` stat is gone (it was always 0 after the video decoder stopped being flushed). 27 stat fields remained; deviation 16 adds `video_delay_ms`, making 28.
 3. `irl-stats.lua` finds the source by its plugin id instead of by display name, and takes source names as script properties.
 4. The vestigial `hw_map_ok` flag is not ported.
 5. `w32-pthreads.dll` is no longer shipped on Windows: Rust never calls `pthread_*`, so the librist shim hazard that `include/irl-threading.h` existed for is gone. The installer deletes a stale copy.
@@ -239,7 +242,8 @@ The port is behaviour-identical except for these, which are intentional:
 12. The I/O stall deadline is not armed while a listener URL waits to be called, and once connected it is measured from the last byte that arrived rather than from the start of the call (master `6d09dea`). `InterruptWatch` therefore tracks the `AVFormatContext` so the callback can read `pb->bytes_read`; `FormatContext` clears that pointer on a failed open and in `Drop`, which cannot wait for the watch's own `Drop` because the receiver holds the same `Arc` across connections.
 13. Video is decoded on the video thread, just before each frame is due, and the receiver → video queue carries compressed packets instead of decoded frames. The C decoded eagerly on the receiver thread, which made the Target Buffer cost decoded-frame memory: 1 GiB of pacing budget is 5.7s of 1080p60 but only 1.4s of 4K60 and 0.7s of 4K60 10-bit, and past that frames were emitted early and dropped. Decoded memory is now bounded by `VIDEO_DECODE_LEAD_MS` regardless of the target. `PacingQueue` gained the matching soft/hard bound split: holding the decode lead is normal and must not emit early, while the byte and frame ceilings are memory limits that still do. The stats line reports `pktq=` instead of `pinned_peak=`, since no decoded frame pins a decoder surface any more.
 14. PTS repair treats a gap of at most one time-base tick, in either direction, as the sender being on time rather than as a discontinuity. The C tested only `< 1 ms` and only forwards, which is a threshold a 90 kHz clock cannot express a 44.1 kHz frame against: 1024 samples is 2089.795 ticks and `duration` can carry only 2090, so the stream landed a tick *early* every few frames, fell into the leading-edge rule for backward jumps (which deliberately freezes the baseline), and the next frame was then interpolated onto the frozen baseline. The repaired timeline stayed one frame short from there on. Since the audio→video mapping is derived from those PTS, every 44.1 kHz stream — which is what phone encoders send — carried a standing ~23 ms lip-sync error, and its `norm=` counter climbed at the frame rate. 48 kHz divides 90 kHz exactly and was never affected.
-15. Video does not anchor libobs's play head on a fallback-scheduled frame when an audio stream is present. The C handed over whichever frame came due first, fallback or mapped, and libobs — which anchors to that frame's *arrival* and never moves the anchor — then played the whole connection with the ~100 ms disagreement between the two schedules baked in, in whichever direction the warm-up/keyframe race went. In the port the race was almost always lost the same way (the video thread decodes the keyframe while the receiver is still working through the probe backlog, so the warm-up has drained and the fallback runs ~100 ms early), which surfaced as audio consistently lagging on phone encoders. Video now waits for the mapping, drops frames it lands in the past, and anchors on the first on-time frame (`VideoThread::awaiting_audio_mapping`, `drop_stale_before_anchor`).
+15. Video does not anchor libobs's play head on a fallback-scheduled frame when an audio stream is present. The C handed over whichever frame came due first, fallback or mapped, and libobs — which anchors to that frame's *arrival* and never moves the anchor — then played the whole connection with the ~100 ms disagreement between the two schedules baked in, in whichever direction the warm-up/keyframe race went. In the port the race was almost always lost the same way (the video thread decodes the keyframe while the receiver is still working through the probe backlog, so the warm-up has drained and the fallback runs ~100 ms early), which surfaced as audio consistently lagging on phone encoders. Video now waits for the mapping, drops frames it lands in the past, and anchors on the first on-time frame (`VideoThread::awaiting_audio_mapping`, `settle_anchor_candidate`). Deviation 16 is what keeps this from dropping a stream whose frames are *all* in the past.
+16. Video that arrives too late to be paced is delayed by a measured, standing amount instead of played unpaced. The C mapped each frame through the audio playout and handed over whatever was past due on arrival, so a sender whose video trailed its audio by more than Target Buffer covered — 120 ms by default, against phone hardware encoders that routinely run 100 to 300 ms behind their audio — played the whole connection with zero lead: every frame handed to libobs at or after its due time, and one dropped by `ready_async_frame` whenever two arrived inside a canvas tick. That is the "low fps" such a stream showed, and the delivery lead (master `964f74b`) could not help it, because a lead needs a frame in hand early. After deviation 15 the same stream was dropped forever instead, since no frame was ever on time. The port measures each frame's arrival margin and adds the shortfall to the schedule as `irl_core::video_delay` (see the threading section); `video_delay_ms` reports it and the warning that sets it tells the user how much Target Buffer restores lip sync. A stream without audio is affected too: the video-only fallback scheduled its first frame at arrival, with no lead for any frame that followed it on time, and now carries a delay of one lead.
 
 ## Contributing
 

README.md

@@ -330,6 +330,7 @@ Stats are exposed through OBS's `proc_handler` API under the `get_stats` call, a
 | `video_corrupt_held` | int | HEVC frames held back instead of shown because they were predicted from a missing reference and would have rendered gray; the last good frame stays on screen until the next keyframe |
 | `video_lead_ms` | int | How far ahead of real time the last video frame was timestamped. Tracks the audio buffer; a value climbing well past Target Buffer and staying there means concealment has inflated the A/V mapping |
 | `video_lead_excess` | int | Frames whose lead exceeded what OBS's async queue can absorb. Harmless while the lead is steady; sustained growth is what makes OBS drop queued video |
+| `video_delay_ms` | int | Standing delay added to the video schedule because video reached the plugin too late to be paced against the audio playout (the encoder sends video later than audio by more than Target Buffer covers). Lip sync is off by this much; raising Target Buffer by at least this takes it back to zero |
 | `stream_delay_ms` | int | End-to-end stream delay (SRT latency + decode + buffering) |
 | `low_latency_audio` | bool | Whether OBS async unbuffered low-latency mode is enabled |
 | `reconnect_count` | int | Number of reconnect attempts since the source was created |

crates/irl-core/src/consts.rs

@@ -307,6 +307,23 @@ pub const VIDEO_PACING_MAX_WAIT_MS: u64 = 50;
 /// this is the slack past that before a stream whose audio never arrives is
 /// let through on the fallback anyway.
 pub const VIDEO_ANCHOR_WAIT_MARGIN_MS: i64 = 1000;
+
+/// Ceiling on the standing video delay (`irl_core::video_delay`).
+///
+/// The delay covers a sender whose video reaches the plugin later than the
+/// audio of the same instant by more than Target Buffer absorbs. A whole
+/// second of that is no longer a skew any encoder produces; it is a decoder
+/// or a host that cannot keep up, and past this the frames stay late (shown
+/// on arrival, as before the delay existed) rather than the picture drifting
+/// ever further behind the sound.
+pub const VIDEO_DELAY_MAX_MS: u64 = 1000;
+/// After the play head is anchored, a raise of the video delay moves the
+/// picture, so late frames must recur across this window before one is made.
+pub const VIDEO_DELAY_WINDOW_MS: u64 = 1000;
+/// Late frames within that window, spread over at least half of it, that count
+/// as recurring. A single scheduling hiccup on the host makes a couple of
+/// consecutive frames late; a sender skew makes them late all window long.
+pub const VIDEO_DELAY_MIN_FRAMES: u32 = 3;
 /// How long the last audio playout offset is reused after it goes away.
 pub const VIDEO_OFFSET_HOLD_NS: u64 = 500_000_000;
 /// Video-only fallback: clamp on drift between stream and system clock.
@@ -443,6 +460,9 @@ mod tests {
         assert_eq!(VIDEO_PACING_SLACK_NS, 1_000_000); // IRL_VIDEO_PACING_SLACK_NS
         assert_eq!(VIDEO_PACING_LEAD_TICKS, 2); // IRL_VIDEO_PACING_LEAD_TICKS
         assert_eq!(VIDEO_ANCHOR_WAIT_MARGIN_MS, 1000);
+        assert_eq!(VIDEO_DELAY_MAX_MS, 1000);
+        assert_eq!(VIDEO_DELAY_WINDOW_MS, 1000);
+        assert_eq!(VIDEO_DELAY_MIN_FRAMES, 3);
         assert_eq!(VIDEO_PACING_MAX_LEAD_NS, 50_000_000); // IRL_VIDEO_PACING_MAX_LEAD_NS
         assert_eq!(VIDEO_CANVAS_TICK_DEFAULT_NS, 16_666_667); // IRL_VIDEO_CANVAS_TICK_DEFAULT_NS
         assert_eq!(VIDEO_PACING_MAX_WAIT_MS, 50); // IRL_VIDEO_PACING_MAX_WAIT_MS

crates/irl-core/src/lib.rs

@@ -23,6 +23,7 @@ pub mod speed;
 pub mod stats;
 pub mod timing;
 pub mod url_opts;
+pub mod video_delay;
 pub mod video_time;
 
 pub use audio_buffer::{AudioBuffer, BufferState};
@@ -35,3 +36,4 @@ pub use speed::{
 };
 pub use stats::{StatKind, StatValue, StatsSnapshot};
 pub use url_opts::url_awaits_caller;
+pub use video_delay::{DelayRaise, VideoDelay};

crates/irl-core/src/pacing.rs

@@ -153,6 +153,21 @@ impl<F: PacedFrame> PacingQueue<F> {
         self.entries.front().map(|e| e.due_ns)
     }
 
+    /// The head frame itself, for the caller's per-frame bookkeeping before
+    /// it decides whether to pop it.
+    pub fn head(&self) -> Option<&F> {
+        self.entries.front().map(|e| &e.frame)
+    }
+
+    /// Move every due time later by `delta_ns`: the video delay was raised.
+    /// Unlike [`Self::reschedule`] this needs no mapping, so it also serves a
+    /// queue scheduled on the video-only fallback.
+    pub fn shift(&mut self, delta_ns: u64) {
+        for entry in &mut self.entries {
+            entry.due_ns = entry.due_ns.saturating_add(delta_ns);
+        }
+    }
+
     /// Pop the head.
     pub fn pop(&mut self) -> Option<F> {
         let entry = self.entries.pop_front()?;
@@ -289,6 +304,30 @@ mod tests {
         assert_eq!(q.next_due(), Some(100_000_000));
     }
 
+    #[test]
+    fn shift_moves_every_due_time_and_keeps_the_head() {
+        let mut q = queue();
+        q.push(
+            TestFrame {
+                pts_ns: 0,
+                bytes: 1,
+            },
+            1_000,
+        );
+        q.push(
+            TestFrame {
+                pts_ns: 40,
+                bytes: 1,
+            },
+            1_040,
+        );
+        q.shift(500);
+        assert_eq!(q.next_due(), Some(1_500));
+        assert_eq!(q.head().map(|f| f.pts_ns), Some(0));
+        q.pop();
+        assert_eq!(q.next_due(), Some(1_540));
+    }
+
     #[test]
     fn empty_queue_has_no_verdict() {
         let mut q = queue();