Repository navigation
MiniMax-H3: _decode_video_file turns ±90° display-rotated videos upside down #14985
Description
Describe the bug
_decode_video_file in src/diffusers/modular_pipelines/minimax_h3/references.py turns clips that carry a ±90° display matrix upside down, compared with what ffmpeg's autorotate and every player show. Phone portrait videos are exactly that case (iPhone stores landscape pixels plus a −90° display matrix), so a portrait reference video reaches MiniMax-H3 / Viggle-Animate rotated 180°.
The line in question (v0.41.0; unchanged on main since #14355):
turns = round(rotation / 90.0) % 4
if turns:
frames = np.ascontiguousarray(np.rot90(frames, k=-turns, axes=(1, 2)))With PyAV's frame.rotation, the frame that matches ffmpeg's autorotated output is np.rot90(frames, k=+turns, axes=(1, 2)). 180° clips are unaffected, because both signs give the same result there. Only ±90° is wrong.
We hit this while integrating Viggle-Animate through the MiniMax-H3 modular pipeline. Our decoder had copied this formula, and a test against ffmpeg's own autorotate caught it. Happy to open a PR flipping the sign if that is welcome.
Reproduction
Standalone. It needs av, numpy and an ffmpeg binary, and does not import diffusers. It copies the line above verbatim and compares frame 0 against ffmpeg's autorotated decode:
import subprocess
import sys
import tempfile
from pathlib import Path
import av
import numpy as np
def run(*args: str) -> bytes:
return subprocess.run(["ffmpeg", "-v", "error", "-y", *args], check=True, capture_output=True).stdout
def ffmpeg_upright(path: Path, width: int, height: int) -> np.ndarray:
raw = run("-i", str(path), "-frames:v", "1", "-f", "rawvideo", "-pix_fmt", "rgb24", "-")
return np.frombuffer(raw, np.uint8).reshape(height, width, 3)
def main() -> int:
tmp = Path(tempfile.mkdtemp())
base = tmp / "base.mp4"
# 320x192 landscape, grey, a red 96x64 block in the top-left corner.
run("-f", "lavfi", "-i", "color=c=gray:s=320x192:d=1:r=24", "-vf",
"drawbox=x=0:y=0:w=96:h=64:color=red:t=fill", "-c:v", "libx264", "-pix_fmt", "yuv420p", str(base))
failures = 0
for display_rotation in (90, -90, 180):
tagged = tmp / f"rot{display_rotation}.mp4"
run("-display_rotation", str(display_rotation), "-i", str(base), "-c", "copy", str(tagged))
with av.open(str(tagged)) as container:
frame = next(container.decode(container.streams.video[0]))
rotation = frame.rotation
frames = np.stack([frame.to_ndarray(format="rgb24")])
turns = round(rotation / 90.0) % 4
diffusers = np.rot90(frames, k=-turns, axes=(1, 2))[0]
opposite = np.rot90(frames, k=+turns, axes=(1, 2))[0]
height, width = diffusers.shape[:2]
truth = ffmpeg_upright(tagged, width, height)
err_diffusers = float(np.abs(diffusers.astype(int) - truth.astype(int)).mean())
err_opposite = float(np.abs(opposite.astype(int) - truth.astype(int)).mean())
# Codec noise is ~2; a frame turned the wrong way is an order of magnitude more.
ok = err_diffusers <= err_opposite + 1.0
failures += not ok
print(f"-display_rotation {display_rotation:>4}: frame.rotation={rotation:>6} turns={turns} "
f"mean|diff| vs ffmpeg: k=-turns {err_diffusers:6.2f} · k=+turns {err_opposite:6.2f} "
f"→ {'k=-turns matches ffmpeg' if ok else 'k=-turns is upside down vs ffmpeg'}")
print(f"av {av.__version__}, numpy {np.__version__}, "
+ subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True).stdout.split("\n")[0])
return failures
if __name__ == "__main__":
sys.exit(main())Logs
-display_rotation 90: frame.rotation= 90 turns=1 mean|diff| vs ffmpeg: k=-turns 26.93 · k=+turns 1.87 → k=-turns is upside down vs ffmpeg
-display_rotation -90: frame.rotation= -90 turns=3 mean|diff| vs ffmpeg: k=-turns 26.93 · k=+turns 1.87 → k=-turns is upside down vs ffmpeg
-display_rotation 180: frame.rotation= -180 turns=2 mean|diff| vs ffmpeg: k=-turns 1.87 · k=+turns 1.87 → k=-turns matches ffmpeg
av 15.1.0, numpy 2.5.3, ffmpeg version 8.1.2 Copyright (c) 2000-2026 the FFmpeg developersA mean absolute difference of ~1.9 is codec noise. 26.9 is the frame turned the wrong way.
System info
- diffusers v0.41.0. The line was read from the tagged source, and
mainhas the same line. - PyAV 15.1.0, NumPy 2.5.3, FFmpeg 8.1.2, macOS (arm64).
- Also seen on Linux (PyAV 15.1, diffusers 0.41.0): our decoder with
k=-turnsfailed the same ffmpeg-autorotate comparison, and withk=+turnsreal iPhone portrait clips come out upright on the device.
Activity
yiyixuxu commentedon Oct 8, 2026
@hsbcelebi thanks for reporting the issue!
I saw @guspan-tanadi already opened a PR, let's work together from there?
would you be able to share a before/after example from a real phone portrait clip? (since I'm not an expert on this subject, it would be really helpful to see)