Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution

Official implementation of Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution.

TODO List

• Release the project page
• Release inference code
• Release ReMind-5B checkpoints
• Release ReMind-1.3B checkpoints
• Release training code

ReMind elicits dynamic memory in causal video generators so that hidden world state can continue evolving through camera motion, occlusion, and illumination changes.

🌐 Project page · 📄 arXiv · 🤗 Models · 🤗 Dataset

This repository contains the inference implementation, model-loading utilities, and reproducible I2V/V2V presets for the released ReMind checkpoints. Training code is not included in this public Git repository.

Setup

python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt
pip install -e .

Download the corresponding official Wan checkpoint into checkpoints/; the expected paths are listed in configs/model_5b.yaml and configs/model_1p3b.yaml. ReMind checkpoints and the release dataset are not stored in Git; download them from the Hugging Face repositories linked above.

Inference

Custom inputs and captions: read docs/inference.md for the content-only caption contract, control-specific prompt schedules, bundled camera-trajectory rules, and a custom-preset checklist.

Conditioning mode: the original seven examples below are all I2V—each uses only one input image. Four additional examples are explicitly labeled V2V and condition on a 21-frame video prefix (6 latent frames) without reading any future frames.

python inference.py \
  --preset examples/presets/01_latte_occluder_recovery.yaml \
  --model-folder checkpoints/Wan2.2-TI2V-5B \
  --base-checkpoint checkpoints/ReMind-5B/ReMind-5B.safetensors \
  --ema-checkpoint checkpoints/ReMind-5B/ReMind-5b-dmd-ema.safetensors \
  --output outputs/latte_occluder_recovery.mp4

Inference examples

Preset	Task	Control
01_latte_occluder_recovery.yaml	I2V	latte art evolves behind a cardboard-box occluder
02_pancake_occluder_recovery.yaml	I2V	pancake batter evolves behind a cardboard-box occluder
03_whisking_camera_pan.yaml	I2V	whisking under the selected pair-fixed InSpatio camera trajectory
04_cereal_camera_pan.yaml	I2V	granular pouring under the selected pair-fixed InSpatio camera trajectory
05_dog_clean_i2v.yaml	I2V	clean dog motion
06_dry_ice_clean_i2v.yaml	I2V	clean dry-ice/foam evolution
07_flour_clean_i2v.yaml	I2V	clean granular pouring
08_cake_decorating_occluder_v2v.yaml	V2V	syrup decoration evolves on a frosted slice behind a cardboard-box occluder
09_ice_melting_drink_occluder_v2v.yaml	V2V	a flaming, foaming drink evolves behind a cardboard-box occluder
10_pour_wine_beer_camera_pan_v2v.yaml	V2V	a carbonated drink fills under its pair-fixed InSpatio camera trajectory
11_slime_stretch_camera_pan_v2v.yaml	V2V	viscous yellow slime accumulates under its pair-fixed InSpatio camera trajectory

The V2V set intentionally contains two occluder and two camera-pan cases, with no clean V2V case and no content overlap with the seven I2V presets. Its bundled inputs are the first 21 frames of the corresponding source-reference clips; inference decodes only those prefix frames and pads the remainder internally before causal VAE encoding.

Inspect the resolved four-step prompt/camera schedule without loading weights:

python inference.py --preset examples/presets/01_latte_occluder_recovery.yaml --dry-run

See docs/inference.md for caption and control conventions, and examples/README.md for the exact page-case seeds, occluder schedules, pair-fixed camera trajectories, prompts, and input frames. Official weights are available from the Hugging Face model repository linked above.

Acknowledgements

We thank the authors and maintainers of Wan2.1 and Wan2.2 for the open models and code that serve as the foundation of ReMind. We are grateful to PRoPE for projective camera positional encoding, LongLive for its open DMD and autoregressive video-generation work, Helios for the video generator used in our dynamic-data engine, Depth Anything 3 for geometry preprocessing.

We also thank the teams and data providers behind InSpatio-World, NeoVerse, OpenVid-1M, SpatialVID, DL3DV-10K, Pexels, Kubric, and PhyCo for the research and data resources used in this work.

Licenses

Original ReMind code is licensed under Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0). Wan-derived code remains Apache-2.0 and preserves its upstream notices; prope/camera_rope.py is MIT. Full third-party license copies are bundled in LICENSES/; see NOTICE and docs/THIRD_PARTY_NOTICES.md. Model weights and dataset terms are covered separately: ReMind weights are CC BY-NC 4.0 under docs/WEIGHTS_LICENSE.md, while final dataset terms remain part of release review.

Citation

@article{xu2026teaching,
  title={Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution},
  author={Xu, Tianshuo and Xie, Yichen and Meng, Depu and Peng, Chensheng and Herau, Quentin and Jiang, Bo and Hu, Yihan and Zhan, Wei},
  journal={arXiv preprint arXiv:2605.25333},
  year={2026}
}