Skip to content

[FEATURE] Add LTX-2.5 distilled T2V/I2V inference - #33

Merged
lzx1413 merged 7 commits into
mainfrom
ltxv2-5
Aug 19, 2026
Merged

[FEATURE] Add LTX-2.5 distilled T2V/I2V inference#33
lzx1413 merged 7 commits into
mainfrom
ltxv2-5

Conversation

@lzx1413

@lzx1413 lzx1413 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Description

Add isolated LTX-2.5 distilled text-to-video and image-to-video inference with synchronized audio generation.

The implementation loads checkpoint components through ModuleManager and composes six stages: text encoding, video conditioning, denoising, latent upsampling, video decoding, and audio decoding. It supports one, two, or four H100 GPUs with Ulysses sequence parallelism, FSDP2 transformer sharding, and selectable dense attention backends.

The default DiffVAE path follows upstream CHUNKED_EAGER decoding with NATTEN cutlass-fna. This PR also adds upstream capture, artifact comparison, checkpoint inspection, and cold/warm benchmark tooling.

Motivation

LTX-2.5 differs substantially from the existing LTX-2.3 implementation in model architecture, checkpoint layout, conditioning, sampling, and audio/video decoding. A separate implementation preserves upstream behavior without coupling the two model families.

The integration also provides reproducible quality and performance gates, framework-native stage composition, and multi-GPU regression coverage.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality not to work as expected)
  • Performance improvement
  • Code refactoring
  • Documentation update
  • Other

Changes Made

  • Added the isolated LTX-2.5 model stack, including the transformer, Gemma text encoder, audio modules, duration head, spatial upsampler, ConvVAE, and NATTEN-backed DiffVAE.
  • Added split-checkpoint loading through ModuleManager and a six-stage distilled T2V/I2V pipeline.
  • Added Ulysses sequence parallelism and FSDP2 execution for 2- and 4-GPU inference, including sequence padding, masking, gathering, and selectable attention implementations.
  • Added standalone T2V and I2V H100 examples, a frozen default I2V reference image, and full two-GPU entries in examples/run_examples.py.
  • Added upstream/TeleFuser capture, artifact comparison, checkpoint inspection, transformer structure comparison, and cold/warm benchmark harnesses.
  • Added CPU-offloaded transformer block streaming with pinned host weights and reusable GPU buffers.
  • Updated English and Chinese model indexes, News entries, example documentation, and the reusable example README template.
  • Standardized file-based I2V entry points on input_image_path.

Testing

  • Focused unit tests pass
  • Manual full-resolution H100 testing performed
  • Benchmarks added and executed

Test commands:

pytest tests/unit/models/ltx25 \
  tests/unit/pipelines/ltx25_distilled \
  tests/unit/validation/test_ltx25_capture_utils.py \
  tests/unit/validation/test_benchmark_ltx25_telefuser.py \
  tests/unit/validation/test_capture_ltx25_telefuser.py \
  tests/unit/validation/test_compare_ltx25_benchmarks.py

pytest tests/unit/test_run_examples.py -q
pytest tests/unit/pipelines/minimax_h3/test_examples.py -q

Verification results:

  • Initial LTX model, pipeline, and validation suites: 61 passed, 1 skipped.
  • Formal single-H100 T2V and I2V comparisons passed against frozen upstream captures.
  • Full 121-frame two-H100 T2V and I2V runner regressions passed, including video and audio comparison.
  • T2V: 61.90 dB PSNR and 0.999685 SSIM.
  • I2V: 62.95 dB PSNR and 0.999671 SSIM.
  • Commit-time Ruff and formatting hooks passed.

Checklist

  • Code follows the project's coding standards
  • Commit-time pre-commit hooks pass
  • Full pre-commit run --all-files executed
  • Full repository test suite (pytest tests/) executed
  • New tests added for new functionality
  • Documentation updated
  • Commit messages are clear and descriptive
  • PR title follows the project convention

Related Issues

N/A

Additional Notes

  • The examples require the official split LTX-2.5 checkpoint layout; consolidated checkpoints are not accepted.
  • Formal DiffVAE parity requires a compatible NATTEN/libnatten installation. The fallback path is not the formal quality or performance baseline.
  • Multi-GPU denoising remains GPU-resident because FSDP2 and transformer CPU offload are mutually exclusive.
  • BF16 sequence-parallel output is not expected to be bitwise identical to single-GPU output because rank-local tensor shapes may select different GEMM kernels.

GPU Architecture Support

  • SM80 (Ampere)
  • SM90 (Hopper H100)
  • SM100+ (Blackwell)

Performance Impact

Matched single-H100 BF16 p50 end-to-end results:

Workload Mode Upstream cold / warm TeleFuser cold / warm
T2V 1536x1024, 121 frames CPU offload 76.78 / 77.29 s 64.44 / 60.09 s
I2V 896x512, 121 frames CPU offload 65.02 / 64.52 s 51.98 / 44.08 s
T2V 1536x1024, 121 frames No offload 58.28 / 55.29 s 46.79 / 46.93 s
I2V 896x512, 121 frames No offload 48.45 / 42.34 s 30.24 / 30.29 s

The no-offload TeleFuser profile reserved 79.14 GB at peak.

Add the isolated LTX-2.5 distilled model, T2V/I2V pipeline, NATTEN-backed DiffVAE decoder, and single-H100 example.\n\nAdd frozen upstream capture, artifact comparison, and cold/warm benchmark harnesses. Default DiffVAE decoding follows upstream CHUNKED_EAGER with cutlass-fna.\n\nVerification: 61 passed, 1 skipped in the LTX model, pipeline, and validation suites; formal T2V/I2V artifact comparisons passed.
Reuse a block-streamed LTX-2.5 transformer for CPU-offload requests while retaining pinned CPU weights and releasing transient GPU buffers between calls.\n\nAdd coverage for streamed-transformer reuse and release.\n\nVerification: ruff format --check; ruff check; pytest tests/unit/pipelines/ltx25_distilled -q (27 passed); formal 5-sample T2V performance comparator passed against pinned upstream.
Document the matched five-sample H100 p50 results for T2V and frozen I2V in CPU-offload and no-offload modes. Record the transformer block-streaming behavior, comparison artifacts, and no-offload memory headroom.\n\nVerification: ruff check and format audits for LTX-2.5 and LTX-2.3 paths; focused LTX-2.5 tests (62 passed, 1 model-root skip); shared flow-matching regression (42 passed); public LTX-2.5 and LTX-2.3 CLI help; matched upstream/TeleFuser five-sample GPU comparison gates for all four documented workloads.
Add a reusable example README template and make it the documented convention for new model-family examples.

Expand the LTX-2.5 distilled guide with the official model source, checkpoint layout, supported features, T2V/I2V commands, CLI options, configuration constraints, and troubleshooting guidance.

Verification:
- git diff --cached --check
- LTX-2.5 example --help
- full LTX-2.5 T2V generation and ffmpeg decode validation
Load all LTX2.5 checkpoint components through ModuleManager and initialize the public pipeline from manager-owned modules.

Split text encoding, video conditioning, denoising, latent upsampling, video decoding, and audio decoding into flat stage modules while preserving the two-phase sampling and output contracts. Update the example, documentation, and focused tests.

Verification: pytest tests/unit/pipelines/ltx25_distilled -q; ruff check; ruff format --check; git diff --check; full H100 T2V output matched all decoded baseline video frames.
- add configurable dense attention and Ulysses sequence parallelism for 1/2/4-GPU execution
- split T2V and I2V examples, add the default I2V reference asset, and register full two-GPU regressions
- consolidate LTX-2.5 documentation and news while standardizing input_image_path across LTX and MiniMax examples

Verification:
- full 121-frame two-GPU T2V and I2V regression comparisons
- pytest tests/unit/test_run_examples.py -q
- pytest tests/unit/pipelines/minimax_h3/test_examples.py -q
- ruff check, ruff format --check, and git diff --check
- package the LTX2.5 pipeline tests so duplicate test_pipeline module names collect independently
- register LTX2.5 and MiniMax Turbo examples in service parity coverage and preserve first_image_path compatibility
- remove the unreferenced transformer structure comparison probe while retaining reproducible artifact and benchmark tools

Verification:
- pytest tests/unit -q --cov=telefuser --cov-report=xml --cov-report=term-missing -m 'not gpu and not distributed and not slow and not quant' (1358 passed, 1 skipped)
- pytest tests/unit/pipelines/ltx25_distilled -q (31 passed)
- pytest tests/unit/service/test_example_service_parity.py -q (55 passed)
- pytest tests/unit/pipelines/minimax_h3/test_examples.py -q (26 passed)
- ruff check, ruff format --check, and git diff --check
@lzx1413
lzx1413 merged commit 117029b into main Aug 19, 2026
5 checks passed
@lzx1413 lzx1413 mentioned this pull request Aug 19, 2026
31 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant