From 6b089c2c0499b56fe5ca2f3f2a1efc58edb708ca Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 18 Sep 2026 17:13:55 +0200 Subject: [PATCH 01/34] Bump the actions group with 17 updates (#12) Bumps the actions group with 17 updates: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `6` | `7` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.2` | `7.0.1` | | [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) | `3.12.0` | `4.2.0` | | [docker/login-action](https://github.com/docker/login-action) | `3.7.0` | `4.6.0` | | [docker/build-push-action](https://github.com/docker/build-push-action) | `6.19.2` | `7.3.0` | | [huggingface/doc-builder/.github/workflows/build_main_documentation.yml](https://github.com/huggingface/doc-builder) | `e60a538eea9817ab312196d0d233604b01697265` | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | | [actions/setup-python](https://github.com/actions/setup-python) | `5.6.0` | `7.0.0` | | [huggingface/doc-builder/.github/workflows/build_pr_documentation.yml](https://github.com/huggingface/doc-builder) | `e60a538eea9817ab312196d0d233604b01697265` | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | | [anthropics/claude-code-action](https://github.com/anthropics/claude-code-action) | `1.0.155` | `1.0.193` | | [huggingface/security-workflows/.github/workflows/codeql-reusable.yml](https://github.com/huggingface/security-workflows) | `1.1.0` | `1.2.0` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `4.3.0` | `8.0.1` | | [actions/labeler](https://github.com/actions/labeler) | `5.0.0` | `7.0.0` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.2` | | [huggingface/tailscale-action](https://github.com/huggingface/tailscale-action) | `7d53c9737e53934c30290b5524d1c9b4a7c98c8a` | `07d36e4c3b68707bd3fe070dbb96f03f993a6efd` | | [trufflesecurity/trufflehog](https://github.com/trufflesecurity/trufflehog) | `3.94.2` | `3.97.0` | | [crate-ci/typos](https://github.com/crate-ci/typos) | `1.42.1` | `1.49.0` | | [huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml](https://github.com/huggingface/doc-builder) | `9ad2de8582b56c017cb530c1165116d40433f1c6` | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | Updates `actions/checkout` from 6 to 7 - [Release notes](https://github.com/actions/checkout/releases) - [Commits](https://github.com/actions/checkout/compare/v6...v7) Updates `actions/upload-artifact` from 4.6.2 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4.6.2...v7.0.1) Updates `docker/setup-buildx-action` from 3.12.0 to 4.2.0 - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/8d2750c68a42422c14e847fe6c8ac0403b4cbd6f...bb05f3f5519dd87d3ba754cc423b652a5edd6d2c) Updates `docker/login-action` from 3.7.0 to 4.6.0 - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/c94ce9fb468520275223c153574b00df6fe4bcc9...dbcb813823bdd20940b903addbd779551569679f) Updates `docker/build-push-action` from 6.19.2 to 7.3.0 - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/10e90e3645eae34f1e60eeb005ba3a3d33f178e8...53b7df96c91f9c12dcc8a07bcb9ccacbed38856a) Updates `huggingface/doc-builder/.github/workflows/build_main_documentation.yml` from e60a538eea9817ab312196d0d233604b01697265 to af49b8dc3859c3d80c7220c8c0ba146431132e49 - [Release notes](https://github.com/huggingface/doc-builder/releases) - [Commits](https://github.com/huggingface/doc-builder/compare/e60a538eea9817ab312196d0d233604b01697265...af49b8dc3859c3d80c7220c8c0ba146431132e49) Updates `actions/setup-python` from 5.6.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v5.6.0...v7) Updates `huggingface/doc-builder/.github/workflows/build_pr_documentation.yml` from e60a538eea9817ab312196d0d233604b01697265 to af49b8dc3859c3d80c7220c8c0ba146431132e49 - [Release notes](https://github.com/huggingface/doc-builder/releases) - [Commits](https://github.com/huggingface/doc-builder/compare/e60a538eea9817ab312196d0d233604b01697265...af49b8dc3859c3d80c7220c8c0ba146431132e49) Updates `anthropics/claude-code-action` from 1.0.155 to 1.0.193 - [Release notes](https://github.com/anthropics/claude-code-action/releases) - [Commits](https://github.com/anthropics/claude-code-action/compare/80b31826338489861333dc17217865dfe8085cdc...9d7150bc8a3dae8149739a88019d192b579ad90c) Updates `huggingface/security-workflows/.github/workflows/codeql-reusable.yml` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/huggingface/security-workflows/releases) - [Commits](https://github.com/huggingface/security-workflows/compare/dc6ca34688e6876c2dd18750719b44d177586c17...e0d40aaf3c566a5e820787840453f45f909e337f) Updates `actions/download-artifact` from 4.3.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/v4.3.0...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) Updates `actions/labeler` from 5.0.0 to 7.0.0 - [Release notes](https://github.com/actions/labeler/releases) - [Commits](https://github.com/actions/labeler/compare/8558fd74291d67161a8a78ce36a881fa63b766a9...bf12e9b00b37c5c0ca2b87b79b2daf7891dbda13) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.2 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e...dc37677b2e1c63e2034f94d8a5b11f265b73ba33) Updates `huggingface/tailscale-action` from 7d53c9737e53934c30290b5524d1c9b4a7c98c8a to 07d36e4c3b68707bd3fe070dbb96f03f993a6efd - [Release notes](https://github.com/huggingface/tailscale-action/releases) - [Commits](https://github.com/huggingface/tailscale-action/compare/7d53c9737e53934c30290b5524d1c9b4a7c98c8a...07d36e4c3b68707bd3fe070dbb96f03f993a6efd) Updates `trufflesecurity/trufflehog` from 3.94.2 to 3.97.0 - [Release notes](https://github.com/trufflesecurity/trufflehog/releases) - [Commits](https://github.com/trufflesecurity/trufflehog/compare/6bd2d14f7a4bc1e569fa3550efa7ec632a4fa67b...bcfcf73aaf4759d4dadc2783177c245a02792318) Updates `crate-ci/typos` from 1.42.1 to 1.49.0 - [Release notes](https://github.com/crate-ci/typos/releases) - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md) - [Commits](https://github.com/crate-ci/typos/compare/65120634e79d8374d1aa2f27e54baa0c364fff5a...8a48f81b6c64dcfea44b3633223084c4be58ac5f) Updates `huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml` from 9ad2de8582b56c017cb530c1165116d40433f1c6 to af49b8dc3859c3d80c7220c8c0ba146431132e49 - [Release notes](https://github.com/huggingface/doc-builder/releases) - [Commits](https://github.com/huggingface/doc-builder/compare/9ad2de8582b56c017cb530c1165116d40433f1c6...af49b8dc3859c3d80c7220c8c0ba146431132e49) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: docker/setup-buildx-action dependency-version: 4.2.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: docker/login-action dependency-version: 4.6.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: docker/build-push-action dependency-version: 7.3.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: huggingface/doc-builder/.github/workflows/build_main_documentation.yml dependency-version: af49b8dc3859c3d80c7220c8c0ba146431132e49 dependency-type: direct:production dependency-group: actions - dependency-name: actions/setup-python dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml dependency-version: af49b8dc3859c3d80c7220c8c0ba146431132e49 dependency-type: direct:production dependency-group: actions - dependency-name: anthropics/claude-code-action dependency-version: 1.0.193 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: huggingface/security-workflows/.github/workflows/codeql-reusable.yml dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/labeler dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: huggingface/tailscale-action dependency-version: 07d36e4c3b68707bd3fe070dbb96f03f993a6efd dependency-type: direct:production dependency-group: actions - dependency-name: trufflesecurity/trufflehog dependency-version: 3.97.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: crate-ci/typos dependency-version: 1.49.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml dependency-version: af49b8dc3859c3d80c7220c8c0ba146431132e49 dependency-type: direct:production dependency-group: actions ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/benchmark.yml | 4 +- .github/workflows/build_docker_images.yml | 12 ++--- .github/workflows/build_documentation.yml | 2 +- .github/workflows/build_pr_documentation.yml | 6 +-- .github/workflows/claude_review.yml | 4 +- .github/workflows/codeql.yml | 2 +- .github/workflows/issue_labeler.yml | 2 +- .../workflows/mirror_community_pipeline.yml | 4 +- .github/workflows/nightly_tests.yml | 46 +++++++++---------- .../workflows/notify_slack_about_release.yml | 4 +- .github/workflows/pr_comment_gpu_tests.yml | 4 +- .github/workflows/pr_dependency_test.yml | 4 +- .github/workflows/pr_labeler.yml | 4 +- .github/workflows/pr_link_issue_reminder.yml | 4 +- .github/workflows/pr_modular_tests.yml | 14 +++--- .github/workflows/pr_test_fetcher.yml | 12 ++--- .github/workflows/pr_tests.yml | 20 ++++---- .github/workflows/pr_tests_gpu.yml | 24 +++++----- .../workflows/pr_torch_dependency_test.yml | 4 +- .github/workflows/push_tests.yml | 24 +++++----- .github/workflows/push_tests_fast.yml | 4 +- .github/workflows/push_tests_mps.yml | 4 +- .github/workflows/pypi_publish.yaml | 12 ++--- .github/workflows/release_tests_fast.yml | 28 +++++------ .github/workflows/ssh-pr-runner.yml | 4 +- .github/workflows/ssh-runner.yml | 2 +- .github/workflows/stale.yml | 4 +- .github/workflows/trufflehog.yml | 4 +- .github/workflows/typos.yml | 4 +- .github/workflows/update_metadata.yml | 2 +- .github/workflows/upload_pr_documentation.yml | 2 +- 31 files changed, 135 insertions(+), 135 deletions(-) diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 84ff531a5d11..972f87d3afeb 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -31,7 +31,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -61,7 +61,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: benchmark_test_reports path: benchmarks/${{ env.BASE_PATH }} diff --git a/.github/workflows/build_docker_images.yml b/.github/workflows/build_docker_images.yml index 6de59f569a55..cfe78731a8d4 100644 --- a/.github/workflows/build_docker_images.yml +++ b/.github/workflows/build_docker_images.yml @@ -31,10 +31,10 @@ jobs: pull-requests: read steps: - name: Set up Docker Buildx - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 + uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 - name: Check out code - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Find Changed Dockerfiles id: file_changes @@ -105,16 +105,16 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 + uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 - name: Login to Docker Hub - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3 + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 with: username: ${{ env.REGISTRY }} password: ${{ secrets.DOCKERHUB_TOKEN }} - name: Build and push - uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0 with: no-cache: true context: ./docker/${{ matrix.image-name }} diff --git a/.github/workflows/build_documentation.yml b/.github/workflows/build_documentation.yml index 3bcf7cc77345..0508793ce0f2 100644 --- a/.github/workflows/build_documentation.yml +++ b/.github/workflows/build_documentation.yml @@ -17,7 +17,7 @@ permissions: jobs: build: - uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@e60a538eea9817ab312196d0d233604b01697265 # main + uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@af49b8dc3859c3d80c7220c8c0ba146431132e49 # main with: commit_sha: ${{ github.sha }} install_libgl1: true diff --git a/.github/workflows/build_pr_documentation.yml b/.github/workflows/build_pr_documentation.yml index 5b29114cf251..aacf483f4f64 100644 --- a/.github/workflows/build_pr_documentation.yml +++ b/.github/workflows/build_pr_documentation.yml @@ -20,10 +20,10 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v6 with: python-version: '3.10' @@ -42,7 +42,7 @@ jobs: build: needs: check-links - uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@e60a538eea9817ab312196d0d233604b01697265 # main + uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@af49b8dc3859c3d80c7220c8c0ba146431132e49 # main with: commit_sha: ${{ github.event.pull_request.head.sha }} pr_number: ${{ github.event.number }} diff --git a/.github/workflows/claude_review.yml b/.github/workflows/claude_review.yml index 5baf3a378dbf..515e1034b817 100644 --- a/.github/workflows/claude_review.yml +++ b/.github/workflows/claude_review.yml @@ -34,7 +34,7 @@ jobs: cancel-in-progress: false runs-on: ubuntu-latest steps: - - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd #v6.0.2 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #v7.0.1 with: fetch-depth: 1 @@ -84,7 +84,7 @@ jobs: git clone --local --bare . /tmp/local-origin.git git config url."file:///tmp/local-origin.git".insteadOf "$(git remote get-url origin)" - - uses: anthropics/claude-code-action@80b31826338489861333dc17217865dfe8085cdc # v1.0.155 + - uses: anthropics/claude-code-action@9d7150bc8a3dae8149739a88019d192b579ad90c # v1.0.193 env: CLAUDE_SYSTEM_PROMPT: | You are a strict code reviewer for the diffusers library (huggingface/diffusers). diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 587d168ca35b..7da9ad364c63 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -10,7 +10,7 @@ on: jobs: codeql: name: CodeQL Analysis - uses: huggingface/security-workflows/.github/workflows/codeql-reusable.yml@dc6ca34688e6876c2dd18750719b44d177586c17 # v1 + uses: huggingface/security-workflows/.github/workflows/codeql-reusable.yml@e0d40aaf3c566a5e820787840453f45f909e337f # v1.2.0 permissions: security-events: write packages: read diff --git a/.github/workflows/issue_labeler.yml b/.github/workflows/issue_labeler.yml index 30acf9193df0..683a819edd39 100644 --- a/.github/workflows/issue_labeler.yml +++ b/.github/workflows/issue_labeler.yml @@ -12,7 +12,7 @@ jobs: label: runs-on: ubuntu-latest steps: - - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install dependencies run: pip install huggingface_hub - name: Get labels from LLM diff --git a/.github/workflows/mirror_community_pipeline.yml b/.github/workflows/mirror_community_pipeline.yml index bf7d15309773..635ecef660a8 100644 --- a/.github/workflows/mirror_community_pipeline.yml +++ b/.github/workflows/mirror_community_pipeline.yml @@ -69,13 +69,13 @@ jobs: run: | echo "CHECKOUT_REF: ${{ env.CHECKOUT_REF }}" echo "PATH_IN_REPO: ${{ env.PATH_IN_REPO }}" - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 with: ref: ${{ env.CHECKOUT_REF }} # Setup + install dependencies - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies diff --git a/.github/workflows/nightly_tests.yml b/.github/workflows/nightly_tests.yml index 02a8ae6c1896..fdad94b07a72 100644 --- a/.github/workflows/nightly_tests.yml +++ b/.github/workflows/nightly_tests.yml @@ -36,7 +36,7 @@ jobs: pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }} steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: Install dependencies @@ -52,7 +52,7 @@ jobs: - name: Pipeline Tests Artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: test-pipelines.json path: reports @@ -72,7 +72,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -104,7 +104,7 @@ jobs: cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pipeline_${{ matrix.module }}_test_reports path: reports @@ -126,7 +126,7 @@ jobs: module: [models, schedulers, hooks, lora, others, single_file, examples] steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -174,7 +174,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_${{ matrix.module }}_cuda_test_reports path: reports @@ -191,7 +191,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -217,7 +217,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_compile_test_reports path: reports @@ -234,7 +234,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -269,7 +269,7 @@ jobs: cat reports/tests_big_gpu_torch_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_cuda_big_gpu_test_reports path: reports @@ -286,7 +286,7 @@ jobs: shell: bash steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -326,7 +326,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_minimum_version_cuda_test_reports path: reports @@ -357,7 +357,7 @@ jobs: options: --shm-size "20gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -396,7 +396,7 @@ jobs: cat reports/tests_${{ matrix.config.backend }}_torch_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_cuda_${{ matrix.config.backend }}_reports path: reports @@ -413,7 +413,7 @@ jobs: options: --shm-size "20gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -447,7 +447,7 @@ jobs: cat reports/tests_pipeline_level_quant_torch_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_cuda_pipeline_level_quant_reports path: reports @@ -470,7 +470,7 @@ jobs: image: diffusers/diffusers-pytorch-cpu steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -478,7 +478,7 @@ jobs: run: mkdir -p combined_reports - name: Download all test reports - uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: path: artifacts @@ -504,7 +504,7 @@ jobs: cat $CONSOLIDATED_REPORT_PATH >> $GITHUB_STEP_SUMMARY - name: Upload consolidated report - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: consolidated_test_report path: ${{ env.CONSOLIDATED_REPORT_PATH }} @@ -518,7 +518,7 @@ jobs: # # steps: # - name: Checkout diffusers -# uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 +# uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 # with: # fetch-depth: 2 # @@ -558,7 +558,7 @@ jobs: # # - name: Test suite reports artifacts # if: ${{ always() }} -# uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 +# uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 # with: # name: torch_mps_test_reports # path: reports @@ -574,7 +574,7 @@ jobs: # # steps: # - name: Checkout diffusers -# uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 +# uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 # with: # fetch-depth: 2 # @@ -614,7 +614,7 @@ jobs: # # - name: Test suite reports artifacts # if: ${{ always() }} -# uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 +# uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 # with: # name: torch_mps_test_reports # path: reports diff --git a/.github/workflows/notify_slack_about_release.yml b/.github/workflows/notify_slack_about_release.yml index 586450c600ed..65c0a35d3016 100644 --- a/.github/workflows/notify_slack_about_release.yml +++ b/.github/workflows/notify_slack_about_release.yml @@ -13,10 +13,10 @@ jobs: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Setup Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: '3.10' diff --git a/.github/workflows/pr_comment_gpu_tests.yml b/.github/workflows/pr_comment_gpu_tests.yml index d4b44f41e92c..77a5513981f5 100644 --- a/.github/workflows/pr_comment_gpu_tests.yml +++ b/.github/workflows/pr_comment_gpu_tests.yml @@ -125,7 +125,7 @@ jobs: shell: bash steps: - name: Checkout PR head - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: # Works for forks too — no fork credentials needed. ref: refs/pull/${{ github.event.issue.number }}/head @@ -166,7 +166,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: bot_gpu_test_reports path: reports diff --git a/.github/workflows/pr_dependency_test.yml b/.github/workflows/pr_dependency_test.yml index 1f16729efb17..e4ee309da852 100644 --- a/.github/workflows/pr_dependency_test.yml +++ b/.github/workflows/pr_dependency_test.yml @@ -22,9 +22,9 @@ jobs: check_dependencies: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies diff --git a/.github/workflows/pr_labeler.yml b/.github/workflows/pr_labeler.yml index 190e3ef8b921..85b2d96d3138 100644 --- a/.github/workflows/pr_labeler.yml +++ b/.github/workflows/pr_labeler.yml @@ -12,14 +12,14 @@ jobs: label: runs-on: ubuntu-latest steps: - - uses: actions/labeler@8558fd74291d67161a8a78ce36a881fa63b766a9 # v5 + - uses: actions/labeler@bf12e9b00b37c5c0ca2b87b79b2daf7891dbda13 # v7.0.0 with: sync-labels: true missing-tests: runs-on: ubuntu-latest steps: - - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ github.event.pull_request.base.sha }} - name: Check for missing tests diff --git a/.github/workflows/pr_link_issue_reminder.yml b/.github/workflows/pr_link_issue_reminder.yml index a0221284057e..98057f0c80d1 100644 --- a/.github/workflows/pr_link_issue_reminder.yml +++ b/.github/workflows/pr_link_issue_reminder.yml @@ -26,10 +26,10 @@ jobs: # (Sayak Paul, YiYi Xu, Dhruv Nair, Daniel Gu). SLACK_MENTION_IDS: ${{ secrets.SLACK_PR_LINK_ISSUE_MENTION_IDS }} steps: - - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Setup Python - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v6 with: python-version: "3.10" diff --git a/.github/workflows/pr_modular_tests.yml b/.github/workflows/pr_modular_tests.yml index 8d7b90c863d1..68000b47c626 100644 --- a/.github/workflows/pr_modular_tests.yml +++ b/.github/workflows/pr_modular_tests.yml @@ -44,9 +44,9 @@ jobs: check_code_quality: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -64,9 +64,9 @@ jobs: needs: check_code_quality runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -92,7 +92,7 @@ jobs: container: image: diffusers/diffusers-pytorch-cpu steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Install dependencies run: | printf 'torch==2.10.0\ntorchvision==0.25.0\ntorchaudio==2.10.0\n' > "$UV_OVERRIDE" @@ -122,7 +122,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -151,7 +151,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pr_pytorch_pipelines_torch_cpu_modular_pipelines_test_reports path: reports diff --git a/.github/workflows/pr_test_fetcher.yml b/.github/workflows/pr_test_fetcher.yml index 17789ec8a9cd..26c537154e5e 100644 --- a/.github/workflows/pr_test_fetcher.yml +++ b/.github/workflows/pr_test_fetcher.yml @@ -31,7 +31,7 @@ jobs: test_map: ${{ steps.set_matrix.outputs.test_map }} steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 0 - name: Install dependencies @@ -45,7 +45,7 @@ jobs: run: | python utils/tests_fetcher.py | tee test_preparation.txt - name: Report fetched tests - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: test_fetched path: test_preparation.txt @@ -86,7 +86,7 @@ jobs: shell: bash steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -112,7 +112,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: ${{ matrix.modules }}_test_reports path: reports @@ -141,7 +141,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -167,7 +167,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pr_${{ matrix.config.report }}_test_reports path: reports diff --git a/.github/workflows/pr_tests.yml b/.github/workflows/pr_tests.yml index 4bde696e16ac..0179c68ce29e 100644 --- a/.github/workflows/pr_tests.yml +++ b/.github/workflows/pr_tests.yml @@ -39,9 +39,9 @@ jobs: check_code_quality: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -59,9 +59,9 @@ jobs: needs: check_code_quality runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -119,7 +119,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -163,7 +163,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pr_${{ matrix.config.framework }}_${{ matrix.config.report }}_test_reports path: reports @@ -195,7 +195,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -223,7 +223,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pr_${{ matrix.config.report }}_test_reports path: reports @@ -249,7 +249,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -290,7 +290,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pr_lora_test_reports path: reports diff --git a/.github/workflows/pr_tests_gpu.yml b/.github/workflows/pr_tests_gpu.yml index 24b08d2b5727..1fdedc2ebb73 100644 --- a/.github/workflows/pr_tests_gpu.yml +++ b/.github/workflows/pr_tests_gpu.yml @@ -40,9 +40,9 @@ jobs: check_code_quality: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -60,9 +60,9 @@ jobs: needs: check_code_quality runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -92,7 +92,7 @@ jobs: pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }} steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 - name: Install dependencies @@ -110,7 +110,7 @@ jobs: echo "pipeline_test_matrix=$matrix" >> $GITHUB_OUTPUT - name: Pipeline Tests Artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: test-pipelines.json path: reports @@ -130,7 +130,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -179,7 +179,7 @@ jobs: cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pipeline_${{ matrix.module }}_test_reports path: reports @@ -202,7 +202,7 @@ jobs: module: [models, schedulers, lora, others] steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -248,7 +248,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: torch_cuda_test_reports_${{ matrix.module }} path: reports @@ -264,7 +264,7 @@ jobs: options: --gpus all --shm-size "16gb" --ipc host steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -296,7 +296,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: examples_test_reports path: reports diff --git a/.github/workflows/pr_torch_dependency_test.yml b/.github/workflows/pr_torch_dependency_test.yml index 4b3184ce2c3a..39586dc25028 100644 --- a/.github/workflows/pr_torch_dependency_test.yml +++ b/.github/workflows/pr_torch_dependency_test.yml @@ -22,9 +22,9 @@ jobs: check_torch_dependencies: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: "3.10" - name: Install dependencies diff --git a/.github/workflows/push_tests.yml b/.github/workflows/push_tests.yml index e960c6300a28..71b6224fa146 100644 --- a/.github/workflows/push_tests.yml +++ b/.github/workflows/push_tests.yml @@ -37,7 +37,7 @@ jobs: pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }} steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 - name: Install dependencies @@ -55,7 +55,7 @@ jobs: echo "pipeline_test_matrix=$matrix" >> $GITHUB_OUTPUT - name: Pipeline Tests Artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: test-pipelines.json path: reports @@ -75,7 +75,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -106,7 +106,7 @@ jobs: cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pipeline_${{ matrix.module }}_test_reports path: reports @@ -128,7 +128,7 @@ jobs: module: [models, schedulers, hooks, lora, others, single_file] steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -163,7 +163,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: torch_cuda_test_reports_${{ matrix.module }} path: reports @@ -180,7 +180,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -206,7 +206,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: torch_compile_test_reports path: reports @@ -223,7 +223,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -248,7 +248,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: torch_xformers_test_reports path: reports @@ -264,7 +264,7 @@ jobs: options: --gpus all --shm-size "16gb" --ipc host steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -295,7 +295,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: examples_test_reports path: reports diff --git a/.github/workflows/push_tests_fast.yml b/.github/workflows/push_tests_fast.yml index 372637f53bcb..1408f4aed7a9 100644 --- a/.github/workflows/push_tests_fast.yml +++ b/.github/workflows/push_tests_fast.yml @@ -60,7 +60,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -93,7 +93,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pr_${{ matrix.config.report }}_test_reports path: reports diff --git a/.github/workflows/push_tests_mps.yml b/.github/workflows/push_tests_mps.yml index 0223c5734a8e..adf776ea615d 100644 --- a/.github/workflows/push_tests_mps.yml +++ b/.github/workflows/push_tests_mps.yml @@ -26,7 +26,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -68,7 +68,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pr_torch_mps_test_reports path: reports diff --git a/.github/workflows/pypi_publish.yaml b/.github/workflows/pypi_publish.yaml index c37a6aee339d..0c6f6b73cf49 100644 --- a/.github/workflows/pypi_publish.yaml +++ b/.github/workflows/pypi_publish.yaml @@ -16,10 +16,10 @@ jobs: runs-on: ubuntu-22.04 steps: - name: Checkout repo - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" @@ -49,7 +49,7 @@ jobs: python -c "from diffusers import *" - name: Upload build artifacts - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: python-dist path: dist/ @@ -63,15 +63,15 @@ jobs: id-token: write steps: - - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Download build artifacts - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: python-dist path: dist/ - name: Publish package distributions to TestPyPI - uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # release/v1 + uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1 with: verbose: true diff --git a/.github/workflows/release_tests_fast.yml b/.github/workflows/release_tests_fast.yml index f83f2abda7b8..45b81192775b 100644 --- a/.github/workflows/release_tests_fast.yml +++ b/.github/workflows/release_tests_fast.yml @@ -36,7 +36,7 @@ jobs: pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }} steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 - name: Install dependencies @@ -55,7 +55,7 @@ jobs: echo "pipeline_test_matrix=$matrix" >> $GITHUB_OUTPUT - name: Pipeline Tests Artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: test-pipelines.json path: reports @@ -75,7 +75,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -106,7 +106,7 @@ jobs: cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: pipeline_${{ matrix.module }}_test_reports path: reports @@ -128,7 +128,7 @@ jobs: module: [models, schedulers, lora, others, single_file] steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -163,7 +163,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: torch_cuda_${{ matrix.module }}_test_reports path: reports @@ -180,7 +180,7 @@ jobs: shell: bash steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -220,7 +220,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: torch_minimum_version_cuda_test_reports path: reports @@ -237,7 +237,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -263,7 +263,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: torch_compile_test_reports path: reports @@ -280,7 +280,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -306,7 +306,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: torch_xformers_test_reports path: reports @@ -323,7 +323,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 @@ -356,7 +356,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v6 + uses: actions/upload-artifact@v7.0.1 with: name: examples_test_reports path: reports diff --git a/.github/workflows/ssh-pr-runner.yml b/.github/workflows/ssh-pr-runner.yml index 96ffa3bae762..e2b80fdb6ec9 100644 --- a/.github/workflows/ssh-pr-runner.yml +++ b/.github/workflows/ssh-pr-runner.yml @@ -30,12 +30,12 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: Tailscale # In order to be able to SSH when a test fails - uses: huggingface/tailscale-action@7d53c9737e53934c30290b5524d1c9b4a7c98c8a # main + uses: huggingface/tailscale-action@07d36e4c3b68707bd3fe070dbb96f03f993a6efd # main with: authkey: ${{ secrets.TAILSCALE_SSH_AUTHKEY }} slackChannel: ${{ secrets.SLACK_CIFEEDBACK_CHANNEL }} diff --git a/.github/workflows/ssh-runner.yml b/.github/workflows/ssh-runner.yml index 73465ce85869..dce59583301f 100644 --- a/.github/workflows/ssh-runner.yml +++ b/.github/workflows/ssh-runner.yml @@ -38,7 +38,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 2 diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 76dd48d09931..694e3dabd331 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -15,10 +15,10 @@ jobs: env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Setup Python - uses: actions/setup-python@v6 + uses: actions/setup-python@v7.0.0 with: python-version: 3.10 diff --git a/.github/workflows/trufflehog.yml b/.github/workflows/trufflehog.yml index 8eb35832bdf8..80b83df83693 100644 --- a/.github/workflows/trufflehog.yml +++ b/.github/workflows/trufflehog.yml @@ -11,11 +11,11 @@ jobs: runs-on: ubuntu-22.04 steps: - name: Checkout code - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 - name: Secret Scanning - uses: trufflesecurity/trufflehog@6bd2d14f7a4bc1e569fa3550efa7ec632a4fa67b # main + uses: trufflesecurity/trufflehog@bcfcf73aaf4759d4dadc2783177c245a02792318 # main with: extra_args: --results=verified,unknown diff --git a/.github/workflows/typos.yml b/.github/workflows/typos.yml index 2f99fc73b67c..fecb13a94325 100644 --- a/.github/workflows/typos.yml +++ b/.github/workflows/typos.yml @@ -11,7 +11,7 @@ jobs: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: typos-action - uses: crate-ci/typos@65120634e79d8374d1aa2f27e54baa0c364fff5a # v1.42.1 + uses: crate-ci/typos@8a48f81b6c64dcfea44b3633223084c4be58ac5f # v1.49.0 diff --git a/.github/workflows/update_metadata.yml b/.github/workflows/update_metadata.yml index e5e0984c597a..870714cd5537 100644 --- a/.github/workflows/update_metadata.yml +++ b/.github/workflows/update_metadata.yml @@ -18,7 +18,7 @@ jobs: shell: bash -l {0} steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Setup environment run: | diff --git a/.github/workflows/upload_pr_documentation.yml b/.github/workflows/upload_pr_documentation.yml index a97f2a9e10e6..43fbbc931540 100644 --- a/.github/workflows/upload_pr_documentation.yml +++ b/.github/workflows/upload_pr_documentation.yml @@ -11,7 +11,7 @@ permissions: jobs: build: - uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@9ad2de8582b56c017cb530c1165116d40433f1c6 # main + uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@af49b8dc3859c3d80c7220c8c0ba146431132e49 # main with: package_name: diffusers secrets: From 5fffb5644357b6ea4f37072ee907d8b27f214933 Mon Sep 17 00:00:00 2001 From: "hf-security-analysis[bot]" <265538906+hf-security-analysis[bot]@users.noreply.github.com> Date: Wed, 23 Sep 2026 14:37:47 +0200 Subject: [PATCH 02/34] Scope GITHUB_TOKEN permissions per job (#16) A job with no `permissions:` block inherits whatever the repository hands out. Each block added here grants what that job's own steps need and nothing more. Co-authored-by: hf-security-analysis[bot] <265538906+hf-security-analysis[bot]@users.noreply.github.com> --- .github/workflows/benchmark.yml | 6 ++++-- .github/workflows/build_pr_documentation.yml | 2 ++ .github/workflows/claude_review.yml | 10 ++++++---- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 972f87d3afeb..454cdff5649a 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -5,8 +5,6 @@ on: schedule: - cron: "30 1 1,15 * *" # every 2 weeks on the 1st and the 15th of every month at 1:30 AM -permissions: - contents: read env: DIFFUSERS_IS_CI: yes @@ -16,8 +14,12 @@ env: MKL_NUM_THREADS: 8 BASE_PATH: benchmark_outputs +permissions: {} + jobs: torch_models_cuda_benchmark_tests: + permissions: + contents: read env: SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL_BENCHMARK }} name: Torch Core Models CUDA Benchmarking Tests diff --git a/.github/workflows/build_pr_documentation.yml b/.github/workflows/build_pr_documentation.yml index aacf483f4f64..62eb828b24fc 100644 --- a/.github/workflows/build_pr_documentation.yml +++ b/.github/workflows/build_pr_documentation.yml @@ -16,6 +16,8 @@ permissions: jobs: check-links: + permissions: + contents: read runs-on: ubuntu-latest steps: diff --git a/.github/workflows/claude_review.yml b/.github/workflows/claude_review.yml index 515e1034b817..afabdb49f730 100644 --- a/.github/workflows/claude_review.yml +++ b/.github/workflows/claude_review.yml @@ -6,13 +6,15 @@ on: pull_request_review_comment: types: [created] -permissions: - contents: write - pull-requests: write - issues: read + +permissions: {} jobs: claude-review: + permissions: + contents: write + issues: read + pull-requests: write if: | ( github.event_name == 'issue_comment' && From ab98af5fe396748787dd78272b2b167f79caf894 Mon Sep 17 00:00:00 2001 From: "hf-security-analysis[bot]" <265538906+hf-security-analysis[bot]@users.noreply.github.com> Date: Wed, 23 Sep 2026 15:52:47 +0200 Subject: [PATCH 03/34] Pin GitHub Actions to commit SHAs (#17) A tag is mutable: whoever controls the action's repository can move `v4` to different code, and the next run picks it up with no diff to review. Pinning to the commit SHA freezes the code that runs today. Co-authored-by: hf-security-analysis[bot] <265538906+hf-security-analysis[bot]@users.noreply.github.com> --- .../workflows/mirror_community_pipeline.yml | 4 +-- .../workflows/notify_slack_about_release.yml | 4 +-- .github/workflows/pr_comment_gpu_tests.yml | 4 +-- .github/workflows/pr_dependency_test.yml | 4 +-- .github/workflows/pr_modular_tests.yml | 14 +++++----- .github/workflows/pr_test_fetcher.yml | 12 ++++---- .github/workflows/pr_tests.yml | 20 ++++++------- .github/workflows/pr_tests_gpu.yml | 24 ++++++++-------- .../workflows/pr_torch_dependency_test.yml | 4 +-- .github/workflows/push_tests.yml | 24 ++++++++-------- .github/workflows/push_tests_fast.yml | 4 +-- .github/workflows/push_tests_mps.yml | 4 +-- .github/workflows/release_tests_fast.yml | 28 +++++++++---------- .github/workflows/ssh-runner.yml | 2 +- .github/workflows/stale.yml | 4 +-- .github/workflows/update_metadata.yml | 2 +- 16 files changed, 79 insertions(+), 79 deletions(-) diff --git a/.github/workflows/mirror_community_pipeline.yml b/.github/workflows/mirror_community_pipeline.yml index 635ecef660a8..4fe83810a7de 100644 --- a/.github/workflows/mirror_community_pipeline.yml +++ b/.github/workflows/mirror_community_pipeline.yml @@ -69,13 +69,13 @@ jobs: run: | echo "CHECKOUT_REF: ${{ env.CHECKOUT_REF }}" echo "PATH_IN_REPO: ${{ env.PATH_IN_REPO }}" - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ env.CHECKOUT_REF }} # Setup + install dependencies - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies diff --git a/.github/workflows/notify_slack_about_release.yml b/.github/workflows/notify_slack_about_release.yml index 65c0a35d3016..195cff20eb33 100644 --- a/.github/workflows/notify_slack_about_release.yml +++ b/.github/workflows/notify_slack_about_release.yml @@ -13,10 +13,10 @@ jobs: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Setup Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: '3.10' diff --git a/.github/workflows/pr_comment_gpu_tests.yml b/.github/workflows/pr_comment_gpu_tests.yml index 77a5513981f5..aee84d90af2d 100644 --- a/.github/workflows/pr_comment_gpu_tests.yml +++ b/.github/workflows/pr_comment_gpu_tests.yml @@ -125,7 +125,7 @@ jobs: shell: bash steps: - name: Checkout PR head - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: # Works for forks too — no fork credentials needed. ref: refs/pull/${{ github.event.issue.number }}/head @@ -166,7 +166,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: bot_gpu_test_reports path: reports diff --git a/.github/workflows/pr_dependency_test.yml b/.github/workflows/pr_dependency_test.yml index e4ee309da852..2021a8ecba3a 100644 --- a/.github/workflows/pr_dependency_test.yml +++ b/.github/workflows/pr_dependency_test.yml @@ -22,9 +22,9 @@ jobs: check_dependencies: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies diff --git a/.github/workflows/pr_modular_tests.yml b/.github/workflows/pr_modular_tests.yml index 68000b47c626..ee69fe3139de 100644 --- a/.github/workflows/pr_modular_tests.yml +++ b/.github/workflows/pr_modular_tests.yml @@ -44,9 +44,9 @@ jobs: check_code_quality: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -64,9 +64,9 @@ jobs: needs: check_code_quality runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -92,7 +92,7 @@ jobs: container: image: diffusers/diffusers-pytorch-cpu steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install dependencies run: | printf 'torch==2.10.0\ntorchvision==0.25.0\ntorchaudio==2.10.0\n' > "$UV_OVERRIDE" @@ -122,7 +122,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -151,7 +151,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pr_pytorch_pipelines_torch_cpu_modular_pipelines_test_reports path: reports diff --git a/.github/workflows/pr_test_fetcher.yml b/.github/workflows/pr_test_fetcher.yml index 26c537154e5e..5e2906c758f7 100644 --- a/.github/workflows/pr_test_fetcher.yml +++ b/.github/workflows/pr_test_fetcher.yml @@ -31,7 +31,7 @@ jobs: test_map: ${{ steps.set_matrix.outputs.test_map }} steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 - name: Install dependencies @@ -45,7 +45,7 @@ jobs: run: | python utils/tests_fetcher.py | tee test_preparation.txt - name: Report fetched tests - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: test_fetched path: test_preparation.txt @@ -86,7 +86,7 @@ jobs: shell: bash steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -112,7 +112,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: ${{ matrix.modules }}_test_reports path: reports @@ -141,7 +141,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -167,7 +167,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pr_${{ matrix.config.report }}_test_reports path: reports diff --git a/.github/workflows/pr_tests.yml b/.github/workflows/pr_tests.yml index 0179c68ce29e..0a3a7f64faec 100644 --- a/.github/workflows/pr_tests.yml +++ b/.github/workflows/pr_tests.yml @@ -39,9 +39,9 @@ jobs: check_code_quality: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -59,9 +59,9 @@ jobs: needs: check_code_quality runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -119,7 +119,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -163,7 +163,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pr_${{ matrix.config.framework }}_${{ matrix.config.report }}_test_reports path: reports @@ -195,7 +195,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -223,7 +223,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pr_${{ matrix.config.report }}_test_reports path: reports @@ -249,7 +249,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -290,7 +290,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pr_lora_test_reports path: reports diff --git a/.github/workflows/pr_tests_gpu.yml b/.github/workflows/pr_tests_gpu.yml index 1fdedc2ebb73..aa6aa436d42f 100644 --- a/.github/workflows/pr_tests_gpu.yml +++ b/.github/workflows/pr_tests_gpu.yml @@ -40,9 +40,9 @@ jobs: check_code_quality: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -60,9 +60,9 @@ jobs: needs: check_code_quality runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies @@ -92,7 +92,7 @@ jobs: pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }} steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: Install dependencies @@ -110,7 +110,7 @@ jobs: echo "pipeline_test_matrix=$matrix" >> $GITHUB_OUTPUT - name: Pipeline Tests Artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: test-pipelines.json path: reports @@ -130,7 +130,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -179,7 +179,7 @@ jobs: cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pipeline_${{ matrix.module }}_test_reports path: reports @@ -202,7 +202,7 @@ jobs: module: [models, schedulers, lora, others] steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -248,7 +248,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_cuda_test_reports_${{ matrix.module }} path: reports @@ -264,7 +264,7 @@ jobs: options: --gpus all --shm-size "16gb" --ipc host steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -296,7 +296,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: examples_test_reports path: reports diff --git a/.github/workflows/pr_torch_dependency_test.yml b/.github/workflows/pr_torch_dependency_test.yml index 39586dc25028..219eba348360 100644 --- a/.github/workflows/pr_torch_dependency_test.yml +++ b/.github/workflows/pr_torch_dependency_test.yml @@ -22,9 +22,9 @@ jobs: check_torch_dependencies: runs-on: ubuntu-22.04 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Set up Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.10" - name: Install dependencies diff --git a/.github/workflows/push_tests.yml b/.github/workflows/push_tests.yml index 71b6224fa146..0b6fbe4ef434 100644 --- a/.github/workflows/push_tests.yml +++ b/.github/workflows/push_tests.yml @@ -37,7 +37,7 @@ jobs: pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }} steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: Install dependencies @@ -55,7 +55,7 @@ jobs: echo "pipeline_test_matrix=$matrix" >> $GITHUB_OUTPUT - name: Pipeline Tests Artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: test-pipelines.json path: reports @@ -75,7 +75,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -106,7 +106,7 @@ jobs: cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pipeline_${{ matrix.module }}_test_reports path: reports @@ -128,7 +128,7 @@ jobs: module: [models, schedulers, hooks, lora, others, single_file] steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -163,7 +163,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_cuda_test_reports_${{ matrix.module }} path: reports @@ -180,7 +180,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -206,7 +206,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_compile_test_reports path: reports @@ -223,7 +223,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -248,7 +248,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_xformers_test_reports path: reports @@ -264,7 +264,7 @@ jobs: options: --gpus all --shm-size "16gb" --ipc host steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -295,7 +295,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: examples_test_reports path: reports diff --git a/.github/workflows/push_tests_fast.yml b/.github/workflows/push_tests_fast.yml index 1408f4aed7a9..9c91fcc81702 100644 --- a/.github/workflows/push_tests_fast.yml +++ b/.github/workflows/push_tests_fast.yml @@ -60,7 +60,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -93,7 +93,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pr_${{ matrix.config.report }}_test_reports path: reports diff --git a/.github/workflows/push_tests_mps.yml b/.github/workflows/push_tests_mps.yml index adf776ea615d..410071f53b7d 100644 --- a/.github/workflows/push_tests_mps.yml +++ b/.github/workflows/push_tests_mps.yml @@ -26,7 +26,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -68,7 +68,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pr_torch_mps_test_reports path: reports diff --git a/.github/workflows/release_tests_fast.yml b/.github/workflows/release_tests_fast.yml index 45b81192775b..3bc88eaedd2f 100644 --- a/.github/workflows/release_tests_fast.yml +++ b/.github/workflows/release_tests_fast.yml @@ -36,7 +36,7 @@ jobs: pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }} steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: Install dependencies @@ -55,7 +55,7 @@ jobs: echo "pipeline_test_matrix=$matrix" >> $GITHUB_OUTPUT - name: Pipeline Tests Artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: test-pipelines.json path: reports @@ -75,7 +75,7 @@ jobs: options: --shm-size "16gb" --ipc host --gpus all steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 - name: NVIDIA-SMI @@ -106,7 +106,7 @@ jobs: cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: pipeline_${{ matrix.module }}_test_reports path: reports @@ -128,7 +128,7 @@ jobs: module: [models, schedulers, lora, others, single_file] steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -163,7 +163,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_cuda_${{ matrix.module }}_test_reports path: reports @@ -180,7 +180,7 @@ jobs: shell: bash steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -220,7 +220,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_minimum_version_cuda_test_reports path: reports @@ -237,7 +237,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -263,7 +263,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_compile_test_reports path: reports @@ -280,7 +280,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -306,7 +306,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: torch_xformers_test_reports path: reports @@ -323,7 +323,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 @@ -356,7 +356,7 @@ jobs: - name: Test suite reports artifacts if: ${{ always() }} - uses: actions/upload-artifact@v7.0.1 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: examples_test_reports path: reports diff --git a/.github/workflows/ssh-runner.yml b/.github/workflows/ssh-runner.yml index dce59583301f..6c7a1490fa5c 100644 --- a/.github/workflows/ssh-runner.yml +++ b/.github/workflows/ssh-runner.yml @@ -38,7 +38,7 @@ jobs: steps: - name: Checkout diffusers - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 694e3dabd331..4c7f46722ad9 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -15,10 +15,10 @@ jobs: env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Setup Python - uses: actions/setup-python@v7.0.0 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: 3.10 diff --git a/.github/workflows/update_metadata.yml b/.github/workflows/update_metadata.yml index 870714cd5537..53fa833f907e 100644 --- a/.github/workflows/update_metadata.yml +++ b/.github/workflows/update_metadata.yml @@ -18,7 +18,7 @@ jobs: shell: bash -l {0} steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Setup environment run: | From 0f8a86af43d8a2a857b0521b32246b3c81d9e03e Mon Sep 17 00:00:00 2001 From: Pauline Bailly-Masson <155966238+paulinebm@users.noreply.github.com> Date: Tue, 6 Oct 2026 11:20:34 +0200 Subject: [PATCH 04/34] Bump doc-builder workflow pin (#22) * Bump doc-builder pin to 9fc8a41 in build_documentation.yml Co-Authored-By: Claude Opus 4.8 * Bump doc-builder pin to 9fc8a41 in build_pr_documentation.yml Co-Authored-By: Claude Opus 4.8 * Bump doc-builder pin to 9fc8a41 in upload_pr_documentation.yml Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 --- .github/workflows/build_documentation.yml | 2 +- .github/workflows/build_pr_documentation.yml | 2 +- .github/workflows/upload_pr_documentation.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build_documentation.yml b/.github/workflows/build_documentation.yml index 0508793ce0f2..0b2c2b85f5a4 100644 --- a/.github/workflows/build_documentation.yml +++ b/.github/workflows/build_documentation.yml @@ -17,7 +17,7 @@ permissions: jobs: build: - uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@af49b8dc3859c3d80c7220c8c0ba146431132e49 # main + uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@9fc8a41d7729c424fdb2192f2f85ff63102ec9a3 # main with: commit_sha: ${{ github.sha }} install_libgl1: true diff --git a/.github/workflows/build_pr_documentation.yml b/.github/workflows/build_pr_documentation.yml index 62eb828b24fc..3c9f33f69e21 100644 --- a/.github/workflows/build_pr_documentation.yml +++ b/.github/workflows/build_pr_documentation.yml @@ -44,7 +44,7 @@ jobs: build: needs: check-links - uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@af49b8dc3859c3d80c7220c8c0ba146431132e49 # main + uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@9fc8a41d7729c424fdb2192f2f85ff63102ec9a3 # main with: commit_sha: ${{ github.event.pull_request.head.sha }} pr_number: ${{ github.event.number }} diff --git a/.github/workflows/upload_pr_documentation.yml b/.github/workflows/upload_pr_documentation.yml index 43fbbc931540..8dcddd44fae5 100644 --- a/.github/workflows/upload_pr_documentation.yml +++ b/.github/workflows/upload_pr_documentation.yml @@ -11,7 +11,7 @@ permissions: jobs: build: - uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@af49b8dc3859c3d80c7220c8c0ba146431132e49 # main + uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@9fc8a41d7729c424fdb2192f2f85ff63102ec9a3 # main with: package_name: diffusers secrets: From fc63b642bc6a0c3e1e2fa94b3c00a9cfa631a251 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 9 Oct 2026 08:16:52 +0200 Subject: [PATCH 05/34] Add FLUX.3 image: Flux3Transformer2DModel and modular text-to-image / image-to-image pipeline Adds the FLUX.3 image transformer, the modular pipeline (text-to-image and image-to-image with one prompt per call and one or more reference images), Flux3ImageProcessor, the converter for BFL's model package, tests and docs. Verified against BFL's reference on the real checkpoint (integration-notes/parity_bf16.py, temporary folder). Co-Authored-By: Claude Fable 5.1 --- docs/source/en/_toctree.yml | 4 + .../source/en/api/models/flux3_transformer.md | 34 + docs/source/en/api/pipelines/flux3.md | 107 +++ integration-notes/README.md | 105 +++ integration-notes/flux3_examples.py | 87 +++ integration-notes/make_flux3_modular_repo.py | 89 +++ integration-notes/make_tiny_flux3_repo.py | 113 ++++ integration-notes/parity_bf16.py | 115 ++++ scripts/convert_flux3_to_diffusers.py | 173 +++++ src/diffusers/__init__.py | 6 + src/diffusers/models/__init__.py | 2 + src/diffusers/models/transformers/__init__.py | 1 + .../models/transformers/transformer_flux3.py | 625 ++++++++++++++++++ src/diffusers/modular_pipelines/__init__.py | 5 + .../modular_pipelines/flux3/__init__.py | 47 ++ .../modular_pipelines/flux3/before_denoise.py | 397 +++++++++++ .../modular_pipelines/flux3/decoders.py | 170 +++++ .../modular_pipelines/flux3/denoise.py | 278 ++++++++ .../modular_pipelines/flux3/encoders.py | 328 +++++++++ .../flux3/image_processor.py | 116 ++++ .../modular_pipelines/flux3/inputs.py | 252 +++++++ .../flux3/modular_blocks_flux3.py | 398 +++++++++++ .../flux3/modular_pipeline.py | 48 ++ .../modular_pipelines/modular_pipeline.py | 1 + src/diffusers/utils/dummy_pt_objects.py | 15 + .../dummy_torch_and_transformers_objects.py | 30 + .../test_models_transformer_flux3.py | 127 ++++ tests/modular_pipelines/flux3/__init__.py | 0 .../flux3/test_modular_pipeline_flux3.py | 269 ++++++++ 29 files changed, 3942 insertions(+) create mode 100644 docs/source/en/api/models/flux3_transformer.md create mode 100644 docs/source/en/api/pipelines/flux3.md create mode 100644 integration-notes/README.md create mode 100644 integration-notes/flux3_examples.py create mode 100644 integration-notes/make_flux3_modular_repo.py create mode 100644 integration-notes/make_tiny_flux3_repo.py create mode 100644 integration-notes/parity_bf16.py create mode 100644 scripts/convert_flux3_to_diffusers.py create mode 100644 src/diffusers/models/transformers/transformer_flux3.py create mode 100644 src/diffusers/modular_pipelines/flux3/__init__.py create mode 100644 src/diffusers/modular_pipelines/flux3/before_denoise.py create mode 100644 src/diffusers/modular_pipelines/flux3/decoders.py create mode 100644 src/diffusers/modular_pipelines/flux3/denoise.py create mode 100644 src/diffusers/modular_pipelines/flux3/encoders.py create mode 100644 src/diffusers/modular_pipelines/flux3/image_processor.py create mode 100644 src/diffusers/modular_pipelines/flux3/inputs.py create mode 100644 src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py create mode 100644 src/diffusers/modular_pipelines/flux3/modular_pipeline.py create mode 100644 tests/models/transformers/test_models_transformer_flux3.py create mode 100644 tests/modular_pipelines/flux3/__init__.py create mode 100644 tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index f2801a0cd6a6..09b6032996ef 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -353,6 +353,8 @@ title: ErnieImageTransformer2DModel - local: api/models/flux2_transformer title: Flux2Transformer2DModel + - local: api/models/flux3_transformer + title: Flux3Transformer2DModel - local: api/models/flux_transformer title: FluxTransformer2DModel - local: api/models/glm_image_transformer2d @@ -577,6 +579,8 @@ title: Flux - local: api/pipelines/flux2 title: Flux2 + - local: api/pipelines/flux3 + title: Flux3 - local: api/pipelines/control_flux_inpaint title: FluxControlInpaint - local: api/pipelines/glm_image diff --git a/docs/source/en/api/models/flux3_transformer.md b/docs/source/en/api/models/flux3_transformer.md new file mode 100644 index 000000000000..b923f28268a8 --- /dev/null +++ b/docs/source/en/api/models/flux3_transformer.md @@ -0,0 +1,34 @@ + + +# Flux3Transformer2DModel + +A Transformer model for image-like data from [FLUX.3](https://bfl.ai/), for text-to-image and image-to-image generation. Three token modes share one joint sequence: the noisy image tokens, the VAE-encoded reference image tokens and the text context. Each mode first passes through its own per-mode transformer blocks, then the modes are concatenated for the joint single blocks, and the image tokens pass through the output layers. Timesteps are per token: the image tokens carry the flow timestep, the reference tokens and the text context sit at `0`. + +```python +import torch +from diffusers import Flux3Transformer2DModel + +transformer = Flux3Transformer2DModel.from_pretrained( + "black-forest-labs/FLUX.3-image-small", subfolder="transformer", dtype=torch.bfloat16 +) +``` + +## Flux3Transformer2DModel + +[[autodoc]] Flux3Transformer2DModel + +## Transformer2DModelOutput + +[[autodoc]] models.modeling_outputs.Transformer2DModelOutput diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md new file mode 100644 index 000000000000..254e05b63719 --- /dev/null +++ b/docs/source/en/api/pipelines/flux3.md @@ -0,0 +1,107 @@ + + +# FLUX.3 + +FLUX.3 is an image generation model from Black Forest Labs: a single transformer ([`Flux3Transformer2DModel`]) generates images from a text prompt, alone or together with one or more reference images. Text conditioning comes from Qwen3-VL, and images use the FLUX.2 packed autoencoder ([`AutoencoderKLFlux2`]). + +FLUX.3 ships as a modular pipeline. Prompting matters: the model responds best to full scene captions that describe the subject, composition, lighting, and atmosphere — see the [FLUX.3 prompting guide](https://docs.bfl.ai) for details. Get started with a text-to-image generation: + + +```python +import torch +from diffusers import ModularPipeline + +pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small") +pipe.load_components(dtype=torch.bfloat16) +pipe.to("cuda") + +image = pipe( + prompt=( + "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest " + "clearing at golden hour. The fox is centered in the frame, its thick orange winter coat " + "contrasting with the fresh white snow, breath visible in the cold air. Snow-laden pine " + "trees fill the softly blurred background, and low warm sunlight rakes across the scene " + "from the left, casting long blue shadows." + ), + generator=torch.Generator().manual_seed(0), + output="images", +)[0] +image.save("fox.png") +``` + +`prompt` is the only required input: the pipeline carries the recommended sampling settings (50 steps, guidance scale 3.5, scheduler shift 6.93) and generates 1024 x 1024 images by default. It takes one prompt per call; use `num_images_per_prompt` for several images of the same prompt. + +## Image-to-image + +Pass one or more reference images with `image`. Every reference keeps its own resolution (center-cropped to the 16-pixel grid; references above 16 megapixels are downscaled first), and the generation size defaults to one megapixel at the first reference's aspect ratio. Describe the *target* image as a complete caption of the reference with your edit folded in, rather than a one-line instruction: + +```python +night = pipe( + prompt=( + "The same red fox in the same snow-covered forest clearing, now at night under a full moon: " + "cool blue moonlight, deep shadows, stars visible through the pine branches, the fox's eyes " + "catching the light." + ), + image=image, + generator=torch.Generator().manual_seed(0), + output="images", +)[0] +``` + +With several references, the prompt can refer to them in order: + +```python +combined = pipe( + prompt="The fox from the first image sitting on the wooden bench from the second image, in the same snowy clearing.", + image=[image, bench], + height=768, + width=1360, + output="images", +)[0] +``` + +Pass `height` and/or `width` to override the derived size. + +## Workflows + +The pipeline serves two workflows, `text2image` and `image2image`; the one that runs is picked by the inputs (`image` selects image-to-image). [`~ModularPipeline.from_pretrained`] can also build a pipeline for one workflow with the `workflow` argument, which then only expects and loads the components that workflow needs: + +```python +from diffusers.modular_pipelines import Flux3AutoBlocks + +print(Flux3AutoBlocks().available_workflows) +# ['text2image', 'image2image'] + +t2i = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small", workflow="text2image") +print(t2i.blocks.doc) +``` + +## Memory + +Loading everything in bf16 takes about 20 GB of weights (the transformer is 5.5B parameters, the Qwen3-VL-4B text encoder 4B), with the image VAE in fp32. For GPUs with less memory, enable component offloading: + +```python +pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small") +pipe.load_components(dtype=torch.bfloat16) +pipe.enable_auto_cpu_offload(device="cuda") +``` + +## Flux3ModularPipeline + +[[autodoc]] Flux3ModularPipeline + +## Flux3AutoBlocks + +[[autodoc]] Flux3AutoBlocks diff --git a/integration-notes/README.md b/integration-notes/README.md new file mode 100644 index 000000000000..1c0a71c20aac --- /dev/null +++ b/integration-notes/README.md @@ -0,0 +1,105 @@ +# FLUX.3 image integration notes — TEMPORARY, REMOVE BEFORE MERGE + +> **⚠️ This folder must not be merged.** It holds the parity check against the confidential BFL +> reference implementation (`f3` package), the doc-style examples, and the scripts that build the +> test and release repos. Delete it before the PR is merged. + +## Setup (workspace `/raid/yiyi`) + +- `diffusers-new-model-addition-f3/` — this repo, branch `flux3-image`, with its `.venv` (transformers 5.x). +- `bfl-f3-image/` — the BFL reference on its `f3-image` branch (`f3/image_small/ImageSmallPipeline`). +- `f3-image-small/` — the BFL model package (`hf download herzogenhorn/f3-image-small`). +- `flux3-image-modular/` — the converted diffusers repo (see Conversion). The same content is on the Hub as + `diffusers-internal-dev/f3-small-img` (private), with BFL's knight example image under `examples/`. +- `venv-tf457/` — a venv with transformers 4.57.6 (BFL's training pin) for the reference side. +- `tiny-flux3-image-modular/` — the tiny random repo the tests use (`make_tiny_flux3_repo.py`). Upload as + `hf-internal-testing/tiny-flux3-image-modular` and update `FLUX3_TINY_REPO` in the modular tests. + +Use `CUDA_DEVICE_ORDER=PCI_BUS_ID` so `CUDA_VISIBLE_DEVICES` matches `nvidia-smi` on this machine. + +## Conversion + +```bash +python scripts/convert_flux3_to_diffusers.py \ + --checkpoint_path f3-image-small/model.safetensors --config_path f3-image-small/config.json \ + --output_path flux3-image-diffusers # -> transformer/ +python scripts/convert_flux2_to_diffusers.py --vae \ + --checkpoint_path f3-image-small/ae.safetensors --output_path flux3-image-diffusers # -> vae/ +``` + +`make_flux3_modular_repo.py` does both, adds the text encoder and tokenizer from the package, writes the +`modular_model_index.json`, and can push the result to the Hub (that is how `diffusers-internal-dev/f3-small-img` +was built). `flux3_examples.py` runs the four parity cases as doc examples against that repo, without any +reference code. + +## Parity (`parity_bf16.py`) + +bf16, 50 steps, guidance 3.5, shift 6.93, seed 0, same prompts and the same initial noise on both sides. +The reference runs as shipped (`ImageSmallPipeline.from_package` + `generate`) in the transformers 4.57.6 venv, +diffusers in this repo's venv; the two sides hand over through PNG files: + +```bash +CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 /raid/yiyi/venv-tf457/bin/python integration-notes/parity_bf16.py reference +CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/parity_bf16.py diffusers +``` + +| case | size | mean pixel diff | max | +|---|---|---|---| +| t2i (the fox) | 1024 x 1024 | 0.44/255 | 47/255 | +| i2i (the fox at night, the t2i output as reference) | 1024 x 1024 | 0.31/255 | 9/255 | +| bbox_t2i (BFL's layout-control example: regions in the caption) | 1024 x 1024 | 0.39/255 | 123/255 | +| bbox_i2i (BFL's knight edit with ``; size derived from the 1360 x 768 reference) | 1360 x 768 | 0.26/255 | 26/255 | + +All four pairs are visually identical (`diffusers-scratch/flux3-image/parity-bf16/*_side_by_side.png`, +reference left, diffusers right). The differences are bf16 rounding over 50 steps; in fp32 the two +implementations agree to within 2/255 (checked during development: the text encoder is bit-identical and +the transformer agrees to 4e-5 in fp32). + +## Why the reference runs under transformers 4.57.6 + +Qwen3-VL computes the position id of the padding tokens differently across transformers versions (1 in +4.57, BFL's training pin; 0 in 5.x), and the FLUX.3 transformer attends to the padding tokens. The real +tokens are bit-identical across versions; only the padding rows differ, which moves the final image by about +4/255 mean. `Flux3TextEncoderStep` passes the position ids explicitly, so on transformers 5.x it produces +the 4.57.6 embeddings bit for bit; running the reference in an actual 4.57.6 venv is what makes the table +above a comparison against the training-time behavior. This diffusers cannot be installed next to +transformers 4.57.6 (huggingface_hub pin), hence the two venvs. + +## Speed and memory + +Single A100 80GB, bf16 transformer and text encoder, fp32 VAE, 50 steps, no offloading: + +| workflow | time | peak GPU memory | +|---|---|---| +| t2i 1024² | 30.5 s | 22.7 GiB | +| i2i 1024² (one 1024² reference) | 63.6 s | 22.7 GiB | +| i2i 1360 x 768 (two references) | 105 s | 22.7 GiB | + +## Design notes + +- `Flux3Transformer2DModel` (image + reference modes, no pooled-vector embedder, output head for the image + tokens; forward: `hidden_states`, `encoder_hidden_states`, `timestep (B,)`, `img_ids`, `txt_ids`, + `image_cond_hidden_states`, `image_cond_ids`). The video model, if it comes, is a separate class. +- One prompt per call (as the reference): the text window is the prompt's length rounded up to 80 tokens, + and the transformer attends to the padding, so prompts of different lengths cannot share a batch without + changing their context. `num_images_per_prompt` gives several images. +- References keep their own resolution (`Flux3ImageProcessor`, Flux2's helpers: downscale above 4096² pixels, + center-crop to the 16-pixel grid), ids on the `l` axis `-(1 + index)`, H/W ids scaled by 0.5. The i2i size + defaults to 1 MP at the first reference's aspect ratio; a given `height`/`width` overrides its side (BFL CLI). +- Sampling defaults on the from_config components: guidance 3.5, shift 6.93, 50 steps, for both workflows. + +## Open items + +- Batched prompts: not supported for now. Before adding: measure the quality effect of a shared text window on + shorter prompts, whether batching helps throughput at all at this sequence length, and the semantics of + several prompts together with several reference images. +- Seed layout: BFL draws the noise token-major `(h*w, C)` on the CPU; diffusers draws `(B, C, h, w)`, so the + same seed gives a different image than BFL's CLI (the parity passes the noise explicitly). Two lines in + `Flux3PrepareLatentsStep` would align them. +- VAE dtype: BFL runs the VAE in fp32; `load_components(dtype=torch.bfloat16)` casts it too. Pass + `dtype={"vae": torch.float32, "default": torch.bfloat16}` (as the examples do) or decide the documented default. +- Hub repo layout with BFL (diffusers subfolders in the release repo vs a `-diffusers` repo); the repo ids in + the docs are placeholders. `from_single_file` for the bf16 transformer is a small follow-up. +- fp8 / nvfp4 variants (`variants/fp8r`, `variants/nvfp4`, ModelOpt export layout with `hf_quant_config.json`): + not loadable as-is; follow-up through the ModelOpt or torchao backend with BFL's weights repackaged. +- Merge upstream `main`; `test_compile_works_with_aot` fails for Flux2 too in this venv (torch 2.13). diff --git a/integration-notes/flux3_examples.py b/integration-notes/flux3_examples.py new file mode 100644 index 000000000000..61d6dc3c7669 --- /dev/null +++ b/integration-notes/flux3_examples.py @@ -0,0 +1,87 @@ +"""FLUX.3 image examples, as they would appear in the docs: text-to-image, image-to-image, and BFL's two +layout-control examples (regions in the caption; an edit that refers to the reference as ). + +Standalone: loads the modular pipeline from the Hub repo and writes four PNGs under `out/`. + + CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/flux3_examples.py + F3_REPO=/raid/yiyi/flux3-image-modular ... # a local repo instead of the Hub +""" + +import os + +import torch +from huggingface_hub import hf_hub_download +from PIL import Image + +from diffusers import ModularPipeline + + +REPO = os.environ.get("F3_REPO", "diffusers-internal-dev/f3-small-img") +OUT = os.environ.get("F3_OUT", "out") +os.makedirs(OUT, exist_ok=True) + +pipe = ModularPipeline.from_pretrained(REPO) +pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) # the image VAE runs in fp32 +pipe.to("cuda") + +# text-to-image: 1024 x 1024, 50 steps, guidance 3.5 by default. FLUX.3 responds best to full scene captions. +prompt = ( + "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest clearing at golden hour. " + "The fox is centered in the frame, its thick orange winter coat contrasting with the fresh white snow, breath " + "visible in the cold air. Snow-laden pine trees fill the softly blurred background, and low warm sunlight rakes " + "across the scene from the left, casting long blue shadows." +) +fox = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] +fox.save(os.path.join(OUT, "t2i.png")) + +# image-to-image: the reference keeps its own resolution and the output defaults to its aspect ratio at ~1 MP. +# Describe the target image as a complete caption of the reference with the edit folded in. +prompt = ( + "The same red fox in the same snow-covered forest clearing, now at night under a full moon: cool blue moonlight, " + "deep shadows, stars visible through the pine branches, the fox's eyes catching the light." +) +night = pipe(prompt=prompt, image=fox, generator=torch.Generator().manual_seed(0), output="images")[0] +night.save(os.path.join(OUT, "i2i.png")) + +# layout control: append a JSON list of regions to the caption. `bbox` is [y0, x0, y1, x1] normalized to 0-1000, +# so it is independent of the image size; tag a region in the caption with and describe it in `desc`. +prompt = ( + "Minimalist graphic illustration featuring a black silhouette of a person centered against a " + "solid, vibrant chartreuse background . The figure is captured in a dynamic, mid-stride running " + "pose, facing toward the left side of the frame. " + '[{"id": "background_1", "bbox": [0, 0, 1000, 1000], "desc": "A flat field of neon yellow-green with a subtle ' + 'paper texture and fine, organic grain."}, {"id": "silhouette_1", "bbox": [150, 150, 850, 850], "desc": "A ' + "black silhouette of a person in motion, made of dense stippled ink. The leading edges are solid; the trailing " + 'arm and leg dissolve into coarse square pixels and scattered dots."}]' +) +runner = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] +runner.save(os.path.join(OUT, "bbox_t2i.png")) + +# editing with layout: refer to the n-th reference image as ; each region names its source image +# (`from`), where it is in that image (`src_bbox`) and where it should end up (`tgt_bbox`). Here the knight moves +# up and to the left and everything else is pinned in place. +knight_path = ( + os.path.join(REPO, "examples", "knight.jpg") + if os.path.isdir(REPO) + else hf_hub_download(REPO, "examples/knight.jpg") +) +knight = Image.open(knight_path) +prompt = ( + "In , move the miniature grey amigurumi knight figure upwards and to the left along the " + "yarn cliff . Keep the rest of the image completely unchanged, preserving the blurred yarn backdrop " + ", the large knitted dragon , and the fiery orange thread . " + '[{"id": "knight_1", "from": "ref_image_0", "src_bbox": [500, 150, 850, 350], "tgt_bbox": [194, 55, 544, 255], ' + '"desc": "A miniature amigurumi knight crocheted from thick grey woolen yarn."}, ' + '{"id": "backdrop_1", "from": "ref_image_0", "src_bbox": [0, 0, 1000, 1000], "tgt_bbox": [0, 0, 1000, 1000], ' + '"desc": "A soft, out-of-focus backdrop of cream and indigo knitted yarn."}, ' + '{"id": "cliff_1", "from": "ref_image_0", "src_bbox": [300, 0, 1000, 450], "tgt_bbox": [300, 0, 1000, 450], ' + '"desc": "A steep cliff of stacked yarn skeins in mustard, terracotta, and indigo."}, ' + '{"id": "needle_1", "from": "ref_image_0", "src_bbox": [420, 250, 550, 450], "tgt_bbox": [420, 250, 550, 450], ' + '"desc": "An oversized polished steel tapestry needle."}, ' + '{"id": "dragon_1", "from": "ref_image_0", "src_bbox": [100, 600, 950, 1000], "tgt_bbox": [100, 600, 950, 1000], ' + '"desc": "A large, fluffy knitted dragon in terracotta and mustard yarn."}, ' + '{"id": "fire_1", "from": "ref_image_0", "src_bbox": [350, 400, 650, 650], "tgt_bbox": [350, 400, 650, 650], ' + '"desc": "Curled puffs of fiery orange angora thread suspended in mid-air."}]' +) +moved = pipe(prompt=prompt, image=knight, generator=torch.Generator().manual_seed(0), output="images")[0] +moved.save(os.path.join(OUT, "bbox_i2i.png")) diff --git a/integration-notes/make_flux3_modular_repo.py b/integration-notes/make_flux3_modular_repo.py new file mode 100644 index 000000000000..615d18a6ecf8 --- /dev/null +++ b/integration-notes/make_flux3_modular_repo.py @@ -0,0 +1,89 @@ +"""Build the diffusers modular repo of FLUX.3 image-small from a BFL model package. + +Converts the transformer and the image VAE, copies the text encoder and tokenizer as they are, writes the +`modular_model_index.json`, and (optionally) pushes the result to the Hub. + + .venv/bin/python integration-notes/make_flux3_modular_repo.py \ + --package /raid/yiyi/f3-image-small --output /raid/yiyi/flux3-image-modular-release \ + [--push_to_hub diffusers-internal-dev/f3-image-diffusers] + +Scheduler, guider and image processor are `from_config` components created by the blocks (shift 6.93, guidance +3.5), so they are not stored in the repo. +""" + +import argparse +import os +import shutil +import subprocess +import sys + + +REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) +sys.path.insert(0, os.path.join(REPO, "src")) + +import torch # noqa: E402 +from transformers import AutoProcessor, Qwen3VLForConditionalGeneration # noqa: E402 + +from diffusers import AutoencoderKLFlux2, Flux3Transformer2DModel # noqa: E402 +from diffusers.modular_pipelines import Flux3AutoBlocks # noqa: E402 + + +parser = argparse.ArgumentParser() +parser.add_argument( + "--package", + required=True, + help="the BFL package directory (config.json, model.safetensors, ae.safetensors, text_encoder/, tokenizer/)", +) +parser.add_argument("--output", required=True) +parser.add_argument("--push_to_hub", default=None, help="repo id to push to (private)") +args = parser.parse_args() + +python = sys.executable +subprocess.run( + [ + python, + os.path.join(REPO, "scripts", "convert_flux3_to_diffusers.py"), + "--checkpoint_path", + os.path.join(args.package, "model.safetensors"), + "--config_path", + os.path.join(args.package, "config.json"), + "--output_path", + args.output, + ], + check=True, +) +subprocess.run( + [ + python, + os.path.join(REPO, "scripts", "convert_flux2_to_diffusers.py"), + "--vae", + "--checkpoint_path", + os.path.join(args.package, "ae.safetensors"), + "--output_path", + args.output, + ], + check=True, +) + +pipe = Flux3AutoBlocks().init_pipeline() +pipe.update_components( + transformer=Flux3Transformer2DModel.from_pretrained(args.output, subfolder="transformer", dtype=torch.bfloat16), + vae=AutoencoderKLFlux2.from_pretrained(args.output, subfolder="vae"), # fp32, as shipped + text_encoder=Qwen3VLForConditionalGeneration.from_pretrained( + os.path.join(args.package, "text_encoder"), dtype=torch.bfloat16 + ), + tokenizer=AutoProcessor.from_pretrained(os.path.join(args.package, "tokenizer")), +) +pipe.save_pretrained( + args.output, + overwrite_modular_index=True, + push_to_hub=args.push_to_hub is not None, + repo_id=args.push_to_hub, + private=True, +) +# `AutoProcessor.from_pretrained(..., subfolder="tokenizer")` needs the model config next to the processor files +# under the pinned transformers (see the README); the package already ships it, keep it +shutil.copy( + os.path.join(args.package, "tokenizer", "config.json"), os.path.join(args.output, "tokenizer", "config.json") +) +print(f"saved {args.output}") diff --git a/integration-notes/make_tiny_flux3_repo.py b/integration-notes/make_tiny_flux3_repo.py new file mode 100644 index 000000000000..fb93931db304 --- /dev/null +++ b/integration-notes/make_tiny_flux3_repo.py @@ -0,0 +1,113 @@ +"""Build a tiny local FLUX.3 image modular repo for fast tests. + +Creates randomly-initialized tiny versions of the four loadable components with +self-consistent geometry (the transformer's token width derives from the tiny VAE +config the same way the real repo's does), saves them under ../tiny-flux3-image-modular +(next to this repo) with a handwritten modular_model_index.json, and smoke-tests a +2-step t2i run and a 2-reference i2i run. + +The text encoder + processor are vendored from trl-internal-testing's tiny Qwen3-VL +(hidden 16, 2 layers, real Qwen vocab/processor files); tests pass +text_encoder_out_layers=(1,), so joint_attention_dim = 16 * 1 = 16. + +Upload later as hf-internal-testing/tiny-flux3-image-modular and point the tests there. +""" + +import json +import os +import sys + +import PIL.Image +import torch +from huggingface_hub import snapshot_download + + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src")) + +from diffusers import AutoencoderKLFlux2, Flux3Transformer2DModel, ModularPipeline # noqa: E402 + + +OUT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "tiny-flux3-image-modular")) +TINY_QWEN3VL = "trl-internal-testing/tiny-Qwen3VLForConditionalGeneration" + +torch.manual_seed(0) + +# text encoder + processor: vendor the tiny Qwen3-VL +qwen_dir = snapshot_download(TINY_QWEN3VL) +os.makedirs(OUT, exist_ok=True) +from transformers import AutoProcessor, Qwen3VLForConditionalGeneration # noqa: E402 + + +text_encoder = Qwen3VLForConditionalGeneration.from_pretrained(qwen_dir) +text_encoder.save_pretrained(os.path.join(OUT, "text_encoder")) +tokenizer = AutoProcessor.from_pretrained(qwen_dir) +tokenizer.save_pretrained(os.path.join(OUT, "tokenizer")) +text_hidden = text_encoder.config.text_config.hidden_size + +vae = AutoencoderKLFlux2( + block_out_channels=[32, 64], + down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"], + up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], + layers_per_block=1, + latent_channels=4, + norm_num_groups=16, + patch_size=[2, 2], + sample_size=32, +) +vae.save_pretrained(os.path.join(OUT, "vae")) + +transformer = Flux3Transformer2DModel( + in_channels=vae.config.latent_channels * 4, + cond_in_channels=vae.config.latent_channels * 4, + num_layers=1, + num_single_layers=1, + attention_head_dim=8, + num_attention_heads=2, + joint_attention_dim=text_hidden, # tests pass text_encoder_out_layers=(1,) + timestep_channels=32, + axes_dims_rope=(2, 2, 2, 2), +) +transformer.save_pretrained(os.path.join(OUT, "transformer")) + +index = { + "_class_name": "Flux3ModularPipeline", + "_blocks_class_name": "Flux3AutoBlocks", + "_diffusers_version": "0.40.0.dev0", +} +for name, lib, cls in [ + ("transformer", "diffusers", "Flux3Transformer2DModel"), + ("vae", "diffusers", "AutoencoderKLFlux2"), + ("text_encoder", "transformers", "Qwen3VLForConditionalGeneration"), + ("tokenizer", "transformers", "AutoProcessor"), +]: + index[name] = [ + lib, + cls, + { + "type_hint": [lib, cls], + "pretrained_model_name_or_path": OUT, + "subfolder": name, + "variant": None, + "revision": None, + }, + ] +with open(os.path.join(OUT, "modular_model_index.json"), "w") as f: + json.dump(index, f, indent=2) + f.write("\n") + +print(f"saved tiny repo to {OUT}") + +# smoke tests: 2-step t2i, and i2i with two references of different sizes +pipe = ModularPipeline.from_pretrained(OUT) +pipe.load_components(torch_dtype=torch.float32) +common = { + "num_inference_steps": 2, + "max_sequence_length": 8, + "text_encoder_out_layers": (1,), + "generator": torch.Generator().manual_seed(0), +} +out = pipe(prompt="a tiny fox", height=64, width=64, **common, output="images") +print("t2i smoke test OK:", out[0].size) +references = [PIL.Image.new("RGB", (64, 48), "red"), PIL.Image.new("RGB", (32, 32), "blue")] +out = pipe(prompt="a tiny fox", image=references, height=64, width=64, **common, output="images") +print("i2i smoke test OK:", out[0].size) diff --git a/integration-notes/parity_bf16.py b/integration-notes/parity_bf16.py new file mode 100644 index 000000000000..4af12da89e5d --- /dev/null +++ b/integration-notes/parity_bf16.py @@ -0,0 +1,115 @@ +"""Visual parity, bf16: BFL's ImageSmallPipeline vs the diffusers modular pipeline, same seed, same prompts. + +Two runs, one per environment, handing over through PNG files: + + # 1. reference, in BFL's training environment (transformers 4.57.6) + CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 /raid/yiyi/venv-tf457/bin/python integration-notes/parity_bf16.py reference + # 2. diffusers, in this repo's venv (transformers 5.x) + CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/parity_bf16.py diffusers + +Cases: t2i and i2i (the fox), and BFL's two layout-control examples (bbox regions in the caption; the knight edit +with , which also exercises the i2i size derived from the reference). Writes _reference.png, +_diffusers.png and _side_by_side.png under OUT, and prints the mean pixel difference. +""" + +import os +import sys + +import numpy as np +import PIL.Image +import torch + + +REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) +PACKAGE = "/raid/yiyi/f3-image-small" # the BFL package +MODULAR_REPO = "/raid/yiyi/flux3-image-modular" # the converted diffusers repo +OUT = "/raid/yiyi/diffusers-scratch/flux3-image/parity-bf16" + +SEED = 0 +HEIGHT = WIDTH = 1024 +PROMPT = ( + "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest clearing at golden hour. " + "The fox is centered in the frame, its thick orange winter coat contrasting with the fresh white snow, breath " + "visible in the cold air. Snow-laden pine trees fill the softly blurred background, and low warm sunlight rakes " + "across the scene from the left, casting long blue shadows." +) +EDIT_PROMPT = ( + "The same red fox in the same snow-covered forest clearing, now at night under a full moon: cool blue moonlight, " + "deep shadows, stars visible through the pine branches, the fox's eyes catching the light." +) +# BFL's layout-control examples (examples/generate.py): regions appended to the caption as JSON, references as +KNIGHT = os.path.join(REPO, "..", "bfl-f3-image", "examples", "assets", "knight.jpg") # BFL docs image, 1360 x 768 +BBOX_PROMPT = 'Minimalist graphic illustration featuring a black silhouette of a person centered against a solid, vibrant chartreuse background . The figure is captured in a dynamic, mid-stride running pose, facing toward the left side of the frame. [{"id": "background_1", "bbox": [0, 0, 1000, 1000], "desc": "A flat field of neon yellow-green with a subtle paper texture and fine, organic grain."}, {"id": "silhouette_1", "bbox": [150, 150, 850, 850], "desc": "A black silhouette of a person in motion, made of dense stippled ink. The leading edges are solid; the trailing arm and leg dissolve into coarse square pixels and scattered dots."}]' +KNIGHT_PROMPT = 'In , move the miniature grey amigurumi knight figure upwards and to the left along the yarn cliff . Keep the rest of the image completely unchanged, preserving the blurred yarn backdrop , the large knitted dragon , and the fiery orange thread . [{"id": "knight_1", "from": "ref_image_0", "src_bbox": [500, 150, 850, 350], "tgt_bbox": [194, 55, 544, 255], "desc": "A miniature amigurumi knight crocheted from thick grey woolen yarn."}, {"id": "backdrop_1", "from": "ref_image_0", "src_bbox": [0, 0, 1000, 1000], "tgt_bbox": [0, 0, 1000, 1000], "desc": "A soft, out-of-focus backdrop of cream and indigo knitted yarn."}, {"id": "cliff_1", "from": "ref_image_0", "src_bbox": [300, 0, 1000, 450], "tgt_bbox": [300, 0, 1000, 450], "desc": "A steep cliff of stacked yarn skeins in mustard, terracotta, and indigo."}, {"id": "needle_1", "from": "ref_image_0", "src_bbox": [420, 250, 550, 450], "tgt_bbox": [420, 250, 550, 450], "desc": "An oversized polished steel tapestry needle."}, {"id": "dragon_1", "from": "ref_image_0", "src_bbox": [100, 600, 950, 1000], "tgt_bbox": [100, 600, 950, 1000], "desc": "A large, fluffy knitted dragon in terracotta and mustard yarn."}, {"id": "fire_1", "from": "ref_image_0", "src_bbox": [350, 400, 650, 650], "tgt_bbox": [350, 400, 650, 650], "desc": "Curled puffs of fiery orange angora thread suspended in mid-air."}]' + +# name, prompt, reference image (None, the t2i output, or a file), size (None: the pipeline's default -- 1024 x 1024 +# for t2i, ~1 MP at the reference's aspect ratio for i2i) +CASES = [ + ("t2i", PROMPT, None, (HEIGHT, WIDTH)), + ("i2i", EDIT_PROMPT, "t2i", (HEIGHT, WIDTH)), + ("bbox_t2i", BBOX_PROMPT, None, None), + ("bbox_i2i", KNIGHT_PROMPT, KNIGHT, None), +] + + +def reference(): + sys.path.insert(0, os.path.join(REPO, "..", "bfl-f3-image")) + from f3.core.hub import resolve_package + from f3.image_small.pipeline import ImageSmallPipeline, default_size, prepare_reference, to_pil, to_tensor + + package = resolve_package(PACKAGE, verify=False) + pipe = ImageSmallPipeline.from_package(package, device="cuda") + for name, prompt, image, size in CASES: + references = [] + if image is not None: + path = os.path.join(OUT, "t2i_reference.png") if image == "t2i" else image + references = [prepare_reference(PIL.Image.open(path))] + width, height = (size[1], size[0]) if size else default_size(references, package.config) + # defaults from the package config: 50 steps, guidance 3.5, alpha 6.93 + result = to_pil(pipe.generate(prompt, [to_tensor(r) for r in references], width, height, seed=SEED)) + result.save(os.path.join(OUT, f"{name}_reference.png")) + print(f"{name}: {result.size}") + + +def diffusers(): + sys.path.insert(0, os.path.join(REPO, "src")) + from diffusers import AutoencoderKLFlux2, ModularPipeline + + pipe = ModularPipeline.from_pretrained(MODULAR_REPO) + pipe.load_components(dtype=torch.bfloat16) + pipe.update_components(vae=AutoencoderKLFlux2.from_pretrained(MODULAR_REPO, subfolder="vae", dtype=torch.float32)) + pipe.to("cuda") + + for name, prompt, image, size in CASES: + inputs = {"prompt": prompt} + if image is not None: + path = os.path.join(OUT, "t2i_reference.png") if image == "t2i" else image + inputs["image"] = PIL.Image.open(path) + if size is not None: + inputs["height"], inputs["width"] = size + # the reference draws the noise token-major on the CPU from manual_seed(SEED) and casts to bf16 + # (f3.image_small.pipeline.noise_tokens); same draw here, reshaped to the (1, C, h, w) latents diffusers + # takes. The size is the one the pipeline resolves (the reference's png has it when derived). + height, width = size or PIL.Image.open(os.path.join(OUT, f"{name}_reference.png")).size[::-1] + h, w = height // 16, width // 16 + noise = torch.randn((h * w, 128), generator=torch.Generator().manual_seed(SEED)).to(torch.bfloat16) + inputs["latents"] = noise.T.reshape(1, 128, h, w) + + result = pipe(**inputs, output="images")[0] + result.save(os.path.join(OUT, f"{name}_diffusers.png")) + print(f"{name}: {result.size}") + + for name, _, _, _ in CASES: + a = PIL.Image.open(os.path.join(OUT, f"{name}_reference.png")) + b = PIL.Image.open(os.path.join(OUT, f"{name}_diffusers.png")) + diff = np.abs(np.asarray(a, dtype=np.float32) - np.asarray(b, dtype=np.float32)) + print(f"{name}: mean pixel diff {diff.mean():.2f}/255, max {diff.max():.0f}/255") + side_by_side = PIL.Image.new("RGB", (a.width * 2, a.height)) + side_by_side.paste(a, (0, 0)) + side_by_side.paste(b, (a.width, 0)) + side_by_side.save(os.path.join(OUT, f"{name}_side_by_side.png")) # reference left, diffusers right + + +if __name__ == "__main__": + os.makedirs(OUT, exist_ok=True) + {"reference": reference, "diffusers": diffusers}[sys.argv[1]]() diff --git a/scripts/convert_flux3_to_diffusers.py b/scripts/convert_flux3_to_diffusers.py new file mode 100644 index 000000000000..a93b2f80d0ff --- /dev/null +++ b/scripts/convert_flux3_to_diffusers.py @@ -0,0 +1,173 @@ +import argparse +import json + +import safetensors.torch +import torch +from huggingface_hub import hf_hub_download + +from diffusers import Flux3Transformer2DModel + + +""" +Converts the FLUX.3 image transformer from a BFL model package (`config.json` + `model.safetensors`) to diffusers. + +# from the Hub package +python scripts/convert_flux3_to_diffusers.py \ + --original_state_dict_repo_id \ + --output_path flux3-image-diffusers + +# from local files +python scripts/convert_flux3_to_diffusers.py \ + --checkpoint_path /path/to/model.safetensors --config_path /path/to/config.json \ + --output_path flux3-image-diffusers + +The image VAE (`ae.safetensors`) is the FLUX.2 autoencoder; convert it with the FLUX.2 script: + +python scripts/convert_flux2_to_diffusers.py --vae --checkpoint_path /path/to/ae.safetensors --output_path flux3-image-diffusers + +The text encoder (`text_encoder/`, `tokenizer/`) is Qwen3-VL in transformers format and needs no conversion. +""" + +parser = argparse.ArgumentParser() +parser.add_argument("--original_state_dict_repo_id", default=None, type=str) +parser.add_argument("--dit_filename", default="model.safetensors", type=str) +parser.add_argument("--config_filename", default="config.json", type=str) +parser.add_argument("--checkpoint_path", default=None, type=str) +parser.add_argument( + "--config_path", + default=None, + type=str, + help="the package config.json; the default architecture is used when omitted", +) +parser.add_argument("--dit_dtype", type=str, default="bf16", choices=["bf16", "fp32"]) +parser.add_argument("--output_path", type=str, required=True) + + +def load_original_checkpoint(args): + if args.original_state_dict_repo_id is not None: + ckpt_path = hf_hub_download(repo_id=args.original_state_dict_repo_id, filename=args.dit_filename) + elif args.checkpoint_path is not None: + ckpt_path = args.checkpoint_path + else: + raise ValueError("please provide either `original_state_dict_repo_id` or a local `checkpoint_path`") + + return safetensors.torch.load_file(ckpt_path) + + +def load_package_config(args): + if args.config_path is not None: + config_path = args.config_path + elif args.original_state_dict_repo_id is not None: + config_path = hf_hub_download(repo_id=args.original_state_dict_repo_id, filename=args.config_filename) + else: + return None + with open(config_path) as f: + return json.load(f) + + +def transformer_config_from_package(package_config): + """The diffusers config of the package's `dit` section.""" + dit = package_config["dit"] + if dit.get("vec_in_dim") is not None: + raise ValueError( + "this package has a pooled-vector embedder (`vec_in_dim`), which Flux3Transformer2DModel does not have" + ) + if list(dit["out_modes"]) != ["image"]: + raise ValueError(f"expected an image-only package (`out_modes` == ['image']), got {dit['out_modes']}") + return { + "in_channels": dit["in_channels"]["image"], + "cond_in_channels": dit["in_channels"]["image_cond"], + "num_layers": dit["depth"], + "num_single_layers": dit["depth_single"], + "attention_head_dim": dit["hidden"] // dit["heads"], + "num_attention_heads": dit["heads"], + "joint_attention_dim": dit["in_channels"]["text"], + "mlp_ratio": dit["mlp_ratio"], + "axes_dims_rope": tuple(dit["axes_dim"]), + "rope_theta": dit["theta"], + } + + +def swap_scale_shift(weight): + # original adaLN weights produce [shift, scale]; diffusers Flux3AdaLayerNormOut expects [scale, shift] + shift, scale = weight.chunk(2, dim=0) + return torch.cat([scale, shift], dim=0) + + +# inner-module renames shared by the mode blocks and the joint single blocks +BLOCK_KEY_MAPPING = { + "q_proj.weight": "attn.to_q.weight", + "k_proj.weight": "attn.to_k.weight", + "v_proj.weight": "attn.to_v.weight", + "attn_out.weight": "attn.to_out.0.weight", + "norm.query_norm.scale": "attn.norm_q.weight", + "norm.key_norm.scale": "attn.norm_k.weight", + "mlp_in.weight": "ff.linear_in.weight", + "mlp_out.weight": "ff.linear_out.weight", +} + +# module-prefix renames +PREFIX_MAPPING = { + "emb_in.": "x_embedders.", + "txt_in.": "context_embedder.", + "content_mode_blocks.": "mode_blocks.", + "single_blocks.": "single_transformer_blocks.", +} + +# exact-key renames +KEY_MAPPING = { + "time_in.in_layer.weight": "timestep_embedder.linear_1.weight", + "time_in.out_layer.weight": "timestep_embedder.linear_2.weight", + # final_layer.image.linear -> proj_out; final_layer.image.adaLN_modulation.1 -> norm_out.linear ([shift, scale] -> [scale, shift]) + "final_layer.image.linear.weight": "proj_out.weight", + "final_layer.image.adaLN_modulation.1.weight": "norm_out.linear.weight", +} + + +def convert_flux3_transformer_checkpoint_to_diffusers(original_state_dict): + converted_state_dict = {} + for key, value in original_state_dict.items(): + # training-only REPA projection heads, not part of the inference model + if key.startswith("repa_projectors."): + continue + if key in KEY_MAPPING: + if key.endswith("adaLN_modulation.1.weight"): + value = swap_scale_shift(value) + converted_state_dict[KEY_MAPPING[key]] = value + continue + + new_key = key + for old_prefix, new_prefix in PREFIX_MAPPING.items(): + if new_key.startswith(old_prefix): + new_key = new_prefix + new_key[len(old_prefix) :] + break + new_key = new_key.replace(".lin.weight", ".linear.weight") + for old_suffix, new_suffix in BLOCK_KEY_MAPPING.items(): + if new_key.endswith(old_suffix): + new_key = new_key[: -len(old_suffix)] + new_suffix + break + + converted_state_dict[new_key] = value + return converted_state_dict + + +def main(args): + original_ckpt = load_original_checkpoint(args) + converted_transformer_state_dict = convert_flux3_transformer_checkpoint_to_diffusers(original_ckpt) + + package_config = load_package_config(args) + transformer_config = transformer_config_from_package(package_config) if package_config is not None else {} + transformer = Flux3Transformer2DModel(**transformer_config) + transformer.load_state_dict(converted_transformer_state_dict, strict=True) + + dtype = torch.bfloat16 if args.dit_dtype == "bf16" else torch.float32 + transformer.to(dtype).save_pretrained(f"{args.output_path}/transformer") + + # confirm the checkpoint round-trips + Flux3Transformer2DModel.from_pretrained(f"{args.output_path}/transformer") + print(f"saved transformer to {args.output_path}/transformer") + + +if __name__ == "__main__": + args = parser.parse_args() + main(args) diff --git a/src/diffusers/__init__.py b/src/diffusers/__init__.py index 3ed956c75e49..5668afbc66ba 100644 --- a/src/diffusers/__init__.py +++ b/src/diffusers/__init__.py @@ -285,6 +285,7 @@ "EasyAnimateTransformer3DModel", "ErnieImageTransformer2DModel", "Flux2Transformer2DModel", + "Flux3Transformer2DModel", "FluxControlNetModel", "FluxMultiControlNetModel", "FluxTransformer2DModel", @@ -525,6 +526,8 @@ "Flux2KleinBaseModularPipeline", "Flux2KleinModularPipeline", "Flux2ModularPipeline", + "Flux3AutoBlocks", + "Flux3ModularPipeline", "FluxAutoBlocks", "FluxKontextAutoBlocks", "FluxKontextModularPipeline", @@ -1162,6 +1165,7 @@ EasyAnimateTransformer3DModel, ErnieImageTransformer2DModel, Flux2Transformer2DModel, + Flux3Transformer2DModel, FluxControlNetModel, FluxMultiControlNetModel, FluxTransformer2DModel, @@ -1381,6 +1385,8 @@ Flux2KleinBaseModularPipeline, Flux2KleinModularPipeline, Flux2ModularPipeline, + Flux3AutoBlocks, + Flux3ModularPipeline, FluxAutoBlocks, FluxKontextAutoBlocks, FluxKontextModularPipeline, diff --git a/src/diffusers/models/__init__.py b/src/diffusers/models/__init__.py index 8ba17d896434..b2bd610d37af 100755 --- a/src/diffusers/models/__init__.py +++ b/src/diffusers/models/__init__.py @@ -118,6 +118,7 @@ _import_structure["transformers.transformer_ernie_image"] = ["ErnieImageTransformer2DModel"] _import_structure["transformers.transformer_flux"] = ["FluxTransformer2DModel"] _import_structure["transformers.transformer_flux2"] = ["Flux2Transformer2DModel"] + _import_structure["transformers.transformer_flux3"] = ["Flux3Transformer2DModel"] _import_structure["transformers.transformer_glm_image"] = ["GlmImageTransformer2DModel"] _import_structure["transformers.transformer_helios"] = ["HeliosTransformer3DModel"] _import_structure["transformers.transformer_hidream_image"] = ["HiDreamImageTransformer2DModel"] @@ -255,6 +256,7 @@ EasyAnimateTransformer3DModel, ErnieImageTransformer2DModel, Flux2Transformer2DModel, + Flux3Transformer2DModel, FluxTransformer2DModel, GlmImageTransformer2DModel, HeliosTransformer3DModel, diff --git a/src/diffusers/models/transformers/__init__.py b/src/diffusers/models/transformers/__init__.py index 0e167812ad88..50a5072b4632 100755 --- a/src/diffusers/models/transformers/__init__.py +++ b/src/diffusers/models/transformers/__init__.py @@ -34,6 +34,7 @@ from .transformer_ernie_image import ErnieImageTransformer2DModel from .transformer_flux import FluxTransformer2DModel from .transformer_flux2 import Flux2Transformer2DModel + from .transformer_flux3 import Flux3Transformer2DModel from .transformer_glm_image import GlmImageTransformer2DModel from .transformer_helios import HeliosTransformer3DModel from .transformer_hidream_image import HiDreamImageTransformer2DModel diff --git a/src/diffusers/models/transformers/transformer_flux3.py b/src/diffusers/models/transformers/transformer_flux3.py new file mode 100644 index 000000000000..2f5fbd91956b --- /dev/null +++ b/src/diffusers/models/transformers/transformer_flux3.py @@ -0,0 +1,625 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Any + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from ...configuration_utils import ConfigMixin, register_to_config +from ...loaders import PeftAdapterMixin +from ...utils import apply_lora_scale, logging +from ...utils.torch_utils import maybe_adjust_dtype_for_device +from ..attention import AttentionMixin, AttentionModuleMixin +from ..attention_dispatch import dispatch_attention_fn +from ..embeddings import TimestepEmbedding, Timesteps, apply_rotary_emb, get_1d_rotary_pos_embed +from ..modeling_outputs import Transformer2DModelOutput +from ..modeling_utils import ModelMixin + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +# Copied from diffusers.models.transformers.transformer_flux2.Flux2SwiGLU with Flux2->Flux3 +class Flux3SwiGLU(nn.Module): + """ + Flux 2 uses a SwiGLU-style activation in the transformer feedforward sub-blocks, but with the linear projection + layer fused into the first linear layer of the FF sub-block. Thus, this module has no trainable parameters. + """ + + def __init__(self): + super().__init__() + self.gate_fn = nn.SiLU() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + half = x.shape[-1] // 2 + x = self.gate_fn(x[..., :half]) * x[..., half:] + return x + + +# Copied from diffusers.models.transformers.transformer_flux2.Flux2FeedForward with Flux2->Flux3 +class Flux3FeedForward(nn.Module): + def __init__( + self, + dim: int, + dim_out: int | None = None, + mult: float = 3.0, + inner_dim: int | None = None, + bias: bool = False, + ): + super().__init__() + if inner_dim is None: + inner_dim = int(dim * mult) + dim_out = dim_out or dim + + # Flux3SwiGLU will reduce the dimension by half + self.linear_in = nn.Linear(dim, inner_dim * 2, bias=bias) + self.act_fn = Flux3SwiGLU() + self.linear_out = nn.Linear(inner_dim, dim_out, bias=bias) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = self.linear_in(x) + x = self.act_fn(x) + x = self.linear_out(x) + return x + + +class Flux3Modulation(nn.Module): + """Produces one set of per-token shift/scale/gate modulation parameters from a pre-activated per-token + conditioning embedding of shape `(batch_size, seq_len, dim)`. The per-mode and the joint heads of a token mode + share one activated embedding (activated once by the caller), so this module only holds the projection.""" + + def __init__(self, dim: int, bias: bool = False): + super().__init__() + self.linear = nn.Linear(dim, dim * 3, bias=bias) + + def forward(self, temb: torch.Tensor) -> torch.Tensor: + return self.linear(temb) + + +def _get_projections(attn: "Flux3Attention", hidden_states: torch.Tensor): + return attn.to_q(hidden_states), attn.to_k(hidden_states), attn.to_v(hidden_states) + + +def _get_fused_projections(attn: "Flux3Attention", hidden_states: torch.Tensor): + return attn.to_qkv(hidden_states).chunk(3, dim=-1) + + +def _get_qkv_projections(attn: "Flux3Attention", hidden_states: torch.Tensor): + if attn.fused_projections: + return _get_fused_projections(attn, hidden_states) + return _get_projections(attn, hidden_states) + + +class Flux3AttnProcessor: + _attention_backend = None + _parallel_config = None + + def __call__( + self, + attn: "Flux3Attention", + hidden_states: torch.Tensor, + attention_mask: torch.Tensor | None = None, + image_rotary_emb: tuple[torch.Tensor, torch.Tensor] | None = None, + ) -> torch.Tensor: + query, key, value = _get_qkv_projections(attn, hidden_states) + + query = query.unflatten(-1, (attn.heads, -1)) + key = key.unflatten(-1, (attn.heads, -1)) + value = value.unflatten(-1, (attn.heads, -1)) + + query = attn.norm_q(query) + key = attn.norm_k(key) + + if image_rotary_emb is not None: + query = apply_rotary_emb(query, image_rotary_emb, sequence_dim=1) + key = apply_rotary_emb(key, image_rotary_emb, sequence_dim=1) + + hidden_states = dispatch_attention_fn( + query, + key, + value, + attn_mask=attention_mask, + backend=self._attention_backend, + parallel_config=self._parallel_config, + ) + hidden_states = hidden_states.flatten(2, 3) + hidden_states = hidden_states.to(query.dtype) + + hidden_states = attn.to_out[0](hidden_states) + hidden_states = attn.to_out[1](hidden_states) + + return hidden_states + + +class Flux3Attention(torch.nn.Module, AttentionModuleMixin): + _default_processor_cls = Flux3AttnProcessor + _available_processors = [Flux3AttnProcessor] + + def __init__( + self, + query_dim: int, + heads: int = 8, + dim_head: int = 64, + dropout: float = 0.0, + eps: float = 1e-6, + processor=None, + ): + super().__init__() + + self.head_dim = dim_head + self.inner_dim = dim_head * heads + self.query_dim = query_dim + self.heads = heads + self.dropout = dropout + self.use_bias = False + + self.to_q = torch.nn.Linear(query_dim, self.inner_dim, bias=False) + self.to_k = torch.nn.Linear(query_dim, self.inner_dim, bias=False) + self.to_v = torch.nn.Linear(query_dim, self.inner_dim, bias=False) + + self.norm_q = torch.nn.RMSNorm(dim_head, eps=eps, elementwise_affine=True) + self.norm_k = torch.nn.RMSNorm(dim_head, eps=eps, elementwise_affine=True) + + self.to_out = torch.nn.ModuleList([]) + self.to_out.append(torch.nn.Linear(self.inner_dim, query_dim, bias=False)) + self.to_out.append(torch.nn.Dropout(dropout)) + + if processor is None: + processor = self._default_processor_cls() + self.set_processor(processor) + + def forward( + self, + hidden_states: torch.Tensor, + attention_mask: torch.Tensor | None = None, + image_rotary_emb: tuple[torch.Tensor, torch.Tensor] | None = None, + ) -> torch.Tensor: + return self.processor(self, hidden_states, attention_mask, image_rotary_emb) + + +class Flux3ModeTransformerBlock(nn.Module): + """ + Per-mode transformer block: parallel attention + feedforward computed from the same modulated input, with + self-attention restricted to a single token mode (the image, the reference images, or the text context). + """ + + def __init__( + self, + dim: int, + num_attention_heads: int, + attention_head_dim: int, + mlp_ratio: float = 3.0, + eps: float = 1e-6, + ): + super().__init__() + + self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=eps) + self.attn = Flux3Attention( + query_dim=dim, + heads=num_attention_heads, + dim_head=attention_head_dim, + eps=eps, + ) + self.ff = Flux3FeedForward(dim=dim, dim_out=dim, mult=mlp_ratio) + + def forward( + self, + hidden_states: torch.Tensor, + temb_mod: torch.Tensor, + image_rotary_emb: tuple[torch.Tensor, torch.Tensor], + ) -> torch.Tensor: + # modulation is split here rather than in Flux3Modulation, to avoid passing tuples into checkpoints https://github.com/huggingface/diffusers/issues/12776 + mod_shift, mod_scale, mod_gate = torch.chunk(temb_mod, 3, dim=-1) + + norm_hidden_states = self.norm(hidden_states) + norm_hidden_states = (1 + mod_scale) * norm_hidden_states + mod_shift + + attn_output = self.attn(hidden_states=norm_hidden_states, image_rotary_emb=image_rotary_emb) + ff_output = self.ff(norm_hidden_states) + + return hidden_states + mod_gate * (attn_output + ff_output) + + +class Flux3SingleTransformerBlock(nn.Module): + """ + Joint transformer block: text and content tokens are concatenated into a single sequence for parallel attention + + feedforward, with separate per-token modulation for the text and content tokens. + """ + + def __init__( + self, + dim: int, + num_attention_heads: int, + attention_head_dim: int, + mlp_ratio: float = 3.0, + eps: float = 1e-6, + ): + super().__init__() + + self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=eps) + self.attn = Flux3Attention( + query_dim=dim, + heads=num_attention_heads, + dim_head=attention_head_dim, + eps=eps, + ) + self.ff = Flux3FeedForward(dim=dim, dim_out=dim, mult=mlp_ratio) + + def forward( + self, + hidden_states: torch.Tensor, + encoder_hidden_states: torch.Tensor, + temb_mod: torch.Tensor, + temb_mod_context: torch.Tensor, + image_rotary_emb: tuple[torch.Tensor, torch.Tensor], + ) -> tuple[torch.Tensor, torch.Tensor]: + # modulation is split here rather than in Flux3Modulation, to avoid passing tuples into checkpoints https://github.com/huggingface/diffusers/issues/12776 + mod_shift, mod_scale, mod_gate = torch.chunk(temb_mod, 3, dim=-1) + c_mod_shift, c_mod_scale, c_mod_gate = torch.chunk(temb_mod_context, 3, dim=-1) + + norm_hidden_states = self.norm(hidden_states) + norm_hidden_states = (1 + mod_scale) * norm_hidden_states + mod_shift + + norm_encoder_hidden_states = self.norm(encoder_hidden_states) + norm_encoder_hidden_states = (1 + c_mod_scale) * norm_encoder_hidden_states + c_mod_shift + + combined = torch.cat([norm_encoder_hidden_states, norm_hidden_states], dim=1) + + attn_output = self.attn(hidden_states=combined, image_rotary_emb=image_rotary_emb) + ff_output = self.ff(combined) + output = attn_output + ff_output + + context_output, content_output = torch.split( + output, [encoder_hidden_states.shape[1], hidden_states.shape[1]], dim=1 + ) + encoder_hidden_states = encoder_hidden_states + c_mod_gate * context_output + hidden_states = hidden_states + mod_gate * content_output + + return encoder_hidden_states, hidden_states + + +# Copied from diffusers.models.transformers.transformer_flux2.Flux2PosEmbed with Flux2->Flux3 +class Flux3PosEmbed(nn.Module): + # modified from https://github.com/black-forest-labs/flux/blob/c00d7c60b085fce8058b9df845e036090873f2ce/src/flux/modules/layers.py#L11 + def __init__(self, theta: int, axes_dim: list[int]): + super().__init__() + self.theta = theta + self.axes_dim = axes_dim + + def forward(self, ids: torch.Tensor) -> torch.Tensor: + # Expected ids shape: [S, len(self.axes_dim)] + cos_out = [] + sin_out = [] + pos = ids.float() + freqs_dtype = maybe_adjust_dtype_for_device(torch.float64, ids.device) + # Unlike Flux 1, loop over len(self.axes_dim) rather than ids.shape[-1] + for i in range(len(self.axes_dim)): + cos, sin = get_1d_rotary_pos_embed( + self.axes_dim[i], + pos[..., i], + theta=self.theta, + repeat_interleave_real=True, + use_real=True, + freqs_dtype=freqs_dtype, + ) + cos_out.append(cos) + sin_out.append(sin) + freqs_cos = torch.cat(cos_out, dim=-1).to(ids.device) + freqs_sin = torch.cat(sin_out, dim=-1).to(ids.device) + return freqs_cos, freqs_sin + + +class Flux3AdaLayerNormOut(nn.Module): + """Output adaptive layer norm conditioned on a per-token embedding of shape `(batch_size, seq_len, dim)`.""" + + def __init__(self, dim: int, eps: float = 1e-6, bias: bool = False): + super().__init__() + self.act_fn = nn.SiLU() + self.linear = nn.Linear(dim, dim * 2, bias=bias) + self.norm = nn.LayerNorm(dim, elementwise_affine=False, eps=eps) + + def forward(self, hidden_states: torch.Tensor, temb: torch.Tensor) -> torch.Tensor: + emb = self.linear(self.act_fn(temb)) + scale, shift = torch.chunk(emb, 2, dim=-1) + return self.norm(hidden_states) * (1 + scale) + shift + + +class Flux3Transformer2DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, AttentionMixin): + """ + The Transformer model introduced in FLUX.3 for text-to-image and image-to-image generation. + + Three token modes share one joint sequence: the noisy image tokens (`hidden_states`), the optional VAE-encoded + reference image tokens (`image_cond_hidden_states`) and the text context (`encoder_hidden_states`). Each mode first + passes through its own per-mode transformer blocks (`mode_blocks` for the image modes, `txt_mode_blocks` for the + text context), then the text context and the active image modes are concatenated for the joint single blocks, and + only the image tokens pass through the output layers. The modulation is per mode: the image tokens are modulated by + the flow timestep, the reference image tokens and the text context by timestep `0`. + + Args: + in_channels (`int`, defaults to `128`): + Channel width of the image tokens (the packed VAE latents, `latent_channels * 4`). + cond_in_channels (`int`, defaults to `128`): + Channel width of the reference image tokens. + num_layers (`int`, defaults to `5`): + The number of per-mode transformer blocks. + num_single_layers (`int`, defaults to `28`): + The number of joint single transformer blocks. + attention_head_dim (`int`, defaults to `128`): + The number of dimensions to use for each attention head. + num_attention_heads (`int`, defaults to `24`): + The number of attention heads to use. + joint_attention_dim (`int`, defaults to `20480`): + The embedding/channel dimension of `encoder_hidden_states` (stacked Qwen3-VL hidden states). + timestep_channels (`int`, defaults to `256`): + The number of channels of the sinusoidal timestep embedding. + mlp_ratio (`float`, defaults to `3.0`): + The expansion ratio of the feedforward sub-blocks. + axes_dims_rope (`tuple[int]`, defaults to `(32, 32, 32, 32)`): + The dimensions to use for the rotary positional embeddings, per `(t, h, w, l)` position axis. + rope_theta (`int`, defaults to `10000`): + The base frequency of the rotary positional embeddings. + eps (`float`, defaults to `1e-6`): + The epsilon of the layer norms and the QK norms. + """ + + _supports_gradient_checkpointing = True + _no_split_modules = ["Flux3ModeTransformerBlock", "Flux3SingleTransformerBlock"] + _skip_layerwise_casting_patterns = ["pos_embed", "norm"] + _repeated_blocks = ["Flux3ModeTransformerBlock", "Flux3SingleTransformerBlock"] + + @register_to_config + def __init__( + self, + in_channels: int = 128, + cond_in_channels: int = 128, + num_layers: int = 5, + num_single_layers: int = 28, + attention_head_dim: int = 128, + num_attention_heads: int = 24, + joint_attention_dim: int = 20480, + timestep_channels: int = 256, + mlp_ratio: float = 3.0, + axes_dims_rope: tuple[int, ...] = (32, 32, 32, 32), + rope_theta: int = 10000, + eps: float = 1e-6, + ): + super().__init__() + + self.inner_dim = num_attention_heads * attention_head_dim + in_channels = {"image": in_channels, "image_cond": cond_in_channels} + modes = list(in_channels.keys()) + + # 1. Sinusoidal positional embedding for RoPE on all token modes + self.pos_embed = Flux3PosEmbed(theta=rope_theta, axes_dim=list(axes_dims_rope)) + + # 2. Per-token timestep embedding + self.time_proj = Timesteps(num_channels=timestep_channels, flip_sin_to_cos=True, downscale_freq_shift=0) + self.timestep_embedder = TimestepEmbedding( + in_channels=timestep_channels, time_embed_dim=self.inner_dim, sample_proj_bias=False + ) + + # 3. Modulation: one head per mode (+ text) for the per-mode blocks and for the joint blocks + self.early_stream_modulations = nn.ModuleDict( + {name: Flux3Modulation(self.inner_dim, bias=False) for name in [*modes, "txt"]} + ) + self.single_stream_modulations = nn.ModuleDict( + {name: Flux3Modulation(self.inner_dim, bias=False) for name in [*modes, "txt"]} + ) + + # 4. Input projections + self.x_embedders = nn.ModuleDict( + {name: nn.Linear(channels, self.inner_dim, bias=False) for name, channels in in_channels.items()} + ) + self.context_embedder = nn.Linear(joint_attention_dim, self.inner_dim, bias=False) + + # 5. Per-mode transformer blocks + self.mode_blocks = nn.ModuleDict( + { + name: nn.ModuleList( + [ + Flux3ModeTransformerBlock( + dim=self.inner_dim, + num_attention_heads=num_attention_heads, + attention_head_dim=attention_head_dim, + mlp_ratio=mlp_ratio, + eps=eps, + ) + for _ in range(num_layers) + ] + ) + for name in modes + } + ) + self.txt_mode_blocks = nn.ModuleList( + [ + Flux3ModeTransformerBlock( + dim=self.inner_dim, + num_attention_heads=num_attention_heads, + attention_head_dim=attention_head_dim, + mlp_ratio=mlp_ratio, + eps=eps, + ) + for _ in range(num_layers) + ] + ) + + # 6. Joint single transformer blocks + self.single_transformer_blocks = nn.ModuleList( + [ + Flux3SingleTransformerBlock( + dim=self.inner_dim, + num_attention_heads=num_attention_heads, + attention_head_dim=attention_head_dim, + mlp_ratio=mlp_ratio, + eps=eps, + ) + for _ in range(num_single_layers) + ] + ) + + # 7. Output layers (image tokens only) + self.norm_out = Flux3AdaLayerNormOut(self.inner_dim, eps=eps) + self.proj_out = nn.Linear(self.inner_dim, in_channels["image"], bias=False) + + self.gradient_checkpointing = False + + @apply_lora_scale("attention_kwargs") + def forward( + self, + hidden_states: torch.Tensor, + encoder_hidden_states: torch.Tensor, + timestep: torch.Tensor, + img_ids: torch.Tensor, + txt_ids: torch.Tensor, + image_cond_hidden_states: torch.Tensor | None = None, + image_cond_ids: torch.Tensor | None = None, + attention_kwargs: dict[str, Any] | None = None, + return_dict: bool = True, + ) -> Transformer2DModelOutput | tuple[torch.Tensor]: + """ + The [`Flux3Transformer2DModel`] forward method. + + Args: + hidden_states (`torch.Tensor` of shape `(batch_size, image_seq_len, in_channels)`): + Noisy image latent tokens. + encoder_hidden_states (`torch.Tensor` of shape `(batch_size, text_seq_len, joint_attention_dim)`): + Text context embeddings (stacked Qwen3-VL hidden states). + timestep (`torch.Tensor` of shape `(batch_size,)`): + Flow timestep in `[0, 1]` of the image tokens. The reference image tokens and the text context sit at + timestep `0`. + img_ids (`torch.Tensor` of shape `(image_seq_len, 4)`): + Position ids `(t, h, w, l)` of the image tokens, used for the rotary positional embeddings and shared + by all samples in the batch. + txt_ids (`torch.Tensor` of shape `(text_seq_len, 4)`): + Position ids of the text context tokens. + image_cond_hidden_states (`torch.Tensor` of shape `(batch_size, cond_seq_len, cond_in_channels)`, *optional*): + VAE-encoded reference image tokens (clean conditioning content, for image-to-image generation). + image_cond_ids (`torch.Tensor` of shape `(cond_seq_len, 4)`, *optional*): + Position ids of the reference image tokens. Required with `image_cond_hidden_states`. + attention_kwargs (`dict`, *optional*): + A kwargs dictionary whose `scale` entry, if specified, scales the LoRA layers for this forward pass. + return_dict (`bool`, *optional*, defaults to `True`): + Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain + tuple. + + Returns: + If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a + `tuple` where the first element is the predicted velocity of the image tokens, of the same shape as + `hidden_states`. + """ + if image_cond_hidden_states is not None and image_cond_ids is None: + raise ValueError("`image_cond_hidden_states` requires `image_cond_ids`.") + if image_cond_hidden_states is None and image_cond_ids is not None: + raise ValueError("`image_cond_ids` was passed without `image_cond_hidden_states`.") + + batch_size, image_seq_len = hidden_states.shape[:2] + dtype = hidden_states.dtype + + # 1. Timestep embeddings: the image tokens at `timestep`, the text context and the reference tokens at 0. + # Each is a per-token embedding (batch_size, seq_len, dim) since the modulation is per token. + image_temb = self.time_proj(timestep.to(dtype) * 1000) + image_temb = self.timestep_embedder(image_temb.to(dtype))[:, None, :].expand(batch_size, image_seq_len, -1) + zero_temb = self.time_proj(torch.zeros(batch_size, device=hidden_states.device, dtype=dtype)) + zero_temb = self.timestep_embedder(zero_temb.to(dtype))[:, None, :] + zero_temb_act = F.silu(zero_temb) + + # 2. Text context: the two modulation sets, input projection and RoPE + txt_temb_act = zero_temb_act.expand(batch_size, encoder_hidden_states.shape[1], -1) + txt_early_mod = self.early_stream_modulations["txt"](txt_temb_act) + txt_single_mod = self.single_stream_modulations["txt"](txt_temb_act) + encoder_hidden_states = self.context_embedder(encoder_hidden_states) + txt_rotary_emb = self.pos_embed(txt_ids) + + # 3. Image tokens + image_temb_act = F.silu(image_temb) + image_early_mod = self.early_stream_modulations["image"](image_temb_act) + image_single_mod = self.single_stream_modulations["image"](image_temb_act) + hidden_states = self.x_embedders["image"](hidden_states) + image_rotary_emb = self.pos_embed(img_ids) + + # 4. Reference image tokens (optional) + if image_cond_hidden_states is not None: + cond_temb_act = zero_temb_act.expand(batch_size, image_cond_hidden_states.shape[1], -1) + cond_early_mod = self.early_stream_modulations["image_cond"](cond_temb_act) + cond_single_mod = self.single_stream_modulations["image_cond"](cond_temb_act) + image_cond_hidden_states = self.x_embedders["image_cond"](image_cond_hidden_states) + cond_rotary_emb = self.pos_embed(image_cond_ids) + + # 5. Per-mode transformer blocks (each mode self-attends within itself only) + for block in self.txt_mode_blocks: + if torch.is_grad_enabled() and self.gradient_checkpointing: + encoder_hidden_states = self._gradient_checkpointing_func( + block, encoder_hidden_states, txt_early_mod, txt_rotary_emb + ) + else: + encoder_hidden_states = block(encoder_hidden_states, txt_early_mod, txt_rotary_emb) + + for block in self.mode_blocks["image"]: + if torch.is_grad_enabled() and self.gradient_checkpointing: + hidden_states = self._gradient_checkpointing_func( + block, hidden_states, image_early_mod, image_rotary_emb + ) + else: + hidden_states = block(hidden_states, image_early_mod, image_rotary_emb) + + if image_cond_hidden_states is not None: + for block in self.mode_blocks["image_cond"]: + if torch.is_grad_enabled() and self.gradient_checkpointing: + image_cond_hidden_states = self._gradient_checkpointing_func( + block, image_cond_hidden_states, cond_early_mod, cond_rotary_emb + ) + else: + image_cond_hidden_states = block(image_cond_hidden_states, cond_early_mod, cond_rotary_emb) + + # 6. Joint single transformer blocks over [txt, image, image_cond] + if image_cond_hidden_states is not None: + content_states = torch.cat([hidden_states, image_cond_hidden_states], dim=1) + content_mod = torch.cat([image_single_mod, cond_single_mod], dim=1) + joint_rotary_emb = ( + torch.cat([txt_rotary_emb[0], image_rotary_emb[0], cond_rotary_emb[0]], dim=0), + torch.cat([txt_rotary_emb[1], image_rotary_emb[1], cond_rotary_emb[1]], dim=0), + ) + else: + content_states = hidden_states + content_mod = image_single_mod + joint_rotary_emb = ( + torch.cat([txt_rotary_emb[0], image_rotary_emb[0]], dim=0), + torch.cat([txt_rotary_emb[1], image_rotary_emb[1]], dim=0), + ) + + for block in self.single_transformer_blocks: + if torch.is_grad_enabled() and self.gradient_checkpointing: + encoder_hidden_states, content_states = self._gradient_checkpointing_func( + block, content_states, encoder_hidden_states, content_mod, txt_single_mod, joint_rotary_emb + ) + else: + encoder_hidden_states, content_states = block( + hidden_states=content_states, + encoder_hidden_states=encoder_hidden_states, + temb_mod=content_mod, + temb_mod_context=txt_single_mod, + image_rotary_emb=joint_rotary_emb, + ) + + # 7. Output layers: the image tokens sit first in the content sequence; the reference tokens produce no output + hidden_states = content_states[:, :image_seq_len] + hidden_states = self.norm_out(hidden_states, image_temb) + output = self.proj_out(hidden_states) + + if not return_dict: + return (output,) + + return Transformer2DModelOutput(sample=output) diff --git a/src/diffusers/modular_pipelines/__init__.py b/src/diffusers/modular_pipelines/__init__.py index 8c3f9ccc62c8..b87974c79e8b 100644 --- a/src/diffusers/modular_pipelines/__init__.py +++ b/src/diffusers/modular_pipelines/__init__.py @@ -81,6 +81,10 @@ "Flux2KleinModularPipeline", "Flux2KleinBaseModularPipeline", ] + _import_structure["flux3"] = [ + "Flux3AutoBlocks", + "Flux3ModularPipeline", + ] _import_structure["ideogram4"] = [ "Ideogram4AutoBlocks", "Ideogram4ModularPipeline", @@ -168,6 +172,7 @@ Flux2KleinModularPipeline, Flux2ModularPipeline, ) + from .flux3 import Flux3AutoBlocks, Flux3ModularPipeline from .helios import ( HeliosAutoBlocks, HeliosModularPipeline, diff --git a/src/diffusers/modular_pipelines/flux3/__init__.py b/src/diffusers/modular_pipelines/flux3/__init__.py new file mode 100644 index 000000000000..21d39cc60ce3 --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/__init__.py @@ -0,0 +1,47 @@ +from typing import TYPE_CHECKING + +from ...utils import ( + DIFFUSERS_SLOW_IMPORT, + OptionalDependencyNotAvailable, + _LazyModule, + get_objects_from_module, + is_torch_available, + is_transformers_available, +) + + +_dummy_objects = {} +_import_structure = {} + +try: + if not (is_transformers_available() and is_torch_available()): + raise OptionalDependencyNotAvailable() +except OptionalDependencyNotAvailable: + from ...utils import dummy_torch_and_transformers_objects # noqa F403 + + _dummy_objects.update(get_objects_from_module(dummy_torch_and_transformers_objects)) +else: + _import_structure["modular_blocks_flux3"] = ["Flux3AutoBlocks"] + _import_structure["modular_pipeline"] = ["Flux3ModularPipeline"] + +if TYPE_CHECKING or DIFFUSERS_SLOW_IMPORT: + try: + if not (is_transformers_available() and is_torch_available()): + raise OptionalDependencyNotAvailable() + except OptionalDependencyNotAvailable: + from ...utils.dummy_torch_and_transformers_objects import * # noqa F403 + else: + from .modular_blocks_flux3 import Flux3AutoBlocks + from .modular_pipeline import Flux3ModularPipeline +else: + import sys + + sys.modules[__name__] = _LazyModule( + __name__, + globals()["__file__"], + _import_structure, + module_spec=__spec__, + ) + + for name, value in _dummy_objects.items(): + setattr(sys.modules[__name__], name, value) diff --git a/src/diffusers/modular_pipelines/flux3/before_denoise.py b/src/diffusers/modular_pipelines/flux3/before_denoise.py new file mode 100644 index 000000000000..08cc2195349f --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/before_denoise.py @@ -0,0 +1,397 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import inspect + +import numpy as np +import torch + +from ...configuration_utils import FrozenDict +from ...schedulers import FlowMatchEulerDiscreteScheduler +from ...utils import logging +from ...utils.torch_utils import randn_tensor +from ..modular_pipeline import ModularPipelineBlocks, PipelineState +from ..modular_pipeline_utils import ComponentSpec, InputParam, OutputParam +from .modular_pipeline import Flux3ModularPipeline + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.retrieve_timesteps +def retrieve_timesteps( + scheduler, + num_inference_steps: int | None = None, + device: str | torch.device | None = None, + timesteps: list[int] | None = None, + sigmas: list[float] | None = None, + **kwargs, +): + r""" + Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles + custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. + + Args: + scheduler (`SchedulerMixin`): + The scheduler to get timesteps from. + num_inference_steps (`int`): + The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` + must be `None`. + device (`str` or `torch.device`, *optional*): + The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. + timesteps (`list[int]`, *optional*): + Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, + `num_inference_steps` and `sigmas` must be `None`. + sigmas (`list[float]`, *optional*): + Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, + `num_inference_steps` and `timesteps` must be `None`. + + Returns: + `tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the + second element is the number of inference steps. + """ + if timesteps is not None and sigmas is not None: + raise ValueError("Only one of `timesteps` or `sigmas` can be passed. Please choose one to set custom values") + if timesteps is not None: + accepts_timesteps = "timesteps" in set(inspect.signature(scheduler.set_timesteps).parameters.keys()) + if not accepts_timesteps: + raise ValueError( + f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom" + f" timestep schedules. Please check whether you are using the correct scheduler." + ) + scheduler.set_timesteps(timesteps=timesteps, device=device, **kwargs) + timesteps = scheduler.timesteps + num_inference_steps = len(timesteps) + elif sigmas is not None: + accept_sigmas = "sigmas" in set(inspect.signature(scheduler.set_timesteps).parameters.keys()) + if not accept_sigmas: + raise ValueError( + f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom" + f" sigmas schedules. Please check whether you are using the correct scheduler." + ) + scheduler.set_timesteps(sigmas=sigmas, device=device, **kwargs) + timesteps = scheduler.timesteps + num_inference_steps = len(timesteps) + else: + scheduler.set_timesteps(num_inference_steps, device=device, **kwargs) + timesteps = scheduler.timesteps + return timesteps, num_inference_steps + + +class Flux3SetTimestepsStep(ModularPipelineBlocks): + model_name = "flux3" + + def __init__(self, shift: float = 6.93): + # the constant timestep shift ("alpha") of the FLUX.3 sampling recipe, configured on the scheduler + self.shift = shift + super().__init__() + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ + ComponentSpec( + "scheduler", + FlowMatchEulerDiscreteScheduler, + config=FrozenDict({"shift": self.shift}), + default_creation_method="from_config", + ), + ] + + @property + def description(self) -> str: + return ( + "Step that sets the scheduler's timesteps for FLUX.3 inference (constant shift, configured on the " + "scheduler)." + ) + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template("num_inference_steps"), + InputParam.template("timesteps", note="overrides `num_inference_steps` when provided"), + InputParam.template("sigmas", note="overrides `num_inference_steps` when provided"), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam("timesteps", type_hint=torch.Tensor, description="The timesteps to use for inference"), + OutputParam( + "num_inference_steps", + type_hint=int, + description="The number of denoising steps to perform at inference time", + ), + ] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + device = components._execution_device + + scheduler = components.scheduler + + num_inference_steps = block_state.num_inference_steps + sigmas = block_state.sigmas + timesteps = block_state.timesteps + + if timesteps is None and sigmas is None: + sigmas = np.linspace(1.0, 1 / num_inference_steps, num_inference_steps) + + timesteps, num_inference_steps = retrieve_timesteps( + scheduler, + num_inference_steps, + device, + timesteps=timesteps, + sigmas=sigmas, + ) + block_state.timesteps = timesteps + block_state.num_inference_steps = num_inference_steps + + components.scheduler.set_begin_index(0) + + self.set_block_state(state, block_state) + return components, state + + +def _prepare_image_ids(height: int, width: int, image_rope_scale: float, l_coord: int = 0) -> torch.Tensor: + """4D position ids `(t, h, w, l)` of shape `(height * width, 4)` for an image token grid: `t=0`, the H/W axes + scaled by the FLUX.3 image RoPE scale, and a constant `l` (0 for the generated image, negative for references).""" + t = torch.arange(1) + h = torch.arange(height) + w = torch.arange(width) + l = torch.tensor([l_coord]) + + ids = torch.cartesian_prod(t, h, w, l).to(dtype=torch.float32) + ids[:, 1:3] = ids[:, 1:3] * image_rope_scale + + return ids + + +class Flux3PrepareLatentsStep(ModularPipelineBlocks): + model_name = "flux3" + + def __init__(self, default_height: int = 1024, default_width: int = 1024, image_rope_scale: float = 0.5): + # text-to-image size; image-to-image derives its size from the reference image upstream (its branch declares + # no default, each branch of the auto blocks resolves its own) + self.default_height = default_height + self.default_width = default_width + # the model was trained with the image H/W position ids scaled by this factor + self.image_rope_scale = image_rope_scale + super().__init__() + + @property + def description(self) -> str: + return "Prepare latents step that prepares the initial noise image latents as tokens, with their position ids." + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template("height", default=self.default_height), + InputParam.template("width", default=self.default_width), + InputParam.template("latents"), + InputParam.template("num_images_per_prompt"), + InputParam.template("generator"), + InputParam( + "batch_size", + required=True, + type_hint=int, + description="Number of prompts, the final batch size of model inputs should be `batch_size * num_images_per_prompt`.", + ), + InputParam("dtype", type_hint=torch.dtype, description="The dtype of the model inputs"), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "latents", type_hint=torch.Tensor, description="The initial latents to use for the denoising process" + ), + OutputParam( + "img_ids", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Position IDs (T, H, W, L) for the image latent tokens (for RoPE)", + ), + ] + + @staticmethod + def check_inputs(height: int, width: int, image_scale_factor: int): + if height % image_scale_factor != 0 or width % image_scale_factor != 0: + raise ValueError( + f"`height` and `width` have to be divisible by {image_scale_factor} but are {height} and {width}." + ) + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + device = components._execution_device + + self.check_inputs(block_state.height, block_state.width, components.vae_scale_factor * 2) + batch_size = block_state.batch_size * block_state.num_images_per_prompt + + # image tokens sit on a (height // 16, width // 16) grid with vae_latent_channels * 4 channels + # (the VAE downsamples 8x and its latents are packed 2x2) + latent_height = block_state.height // (components.vae_scale_factor * 2) + latent_width = block_state.width // (components.vae_scale_factor * 2) + + latents = block_state.latents + if latents is None: + shape = (batch_size, components.num_channels_latents, latent_height, latent_width) + if isinstance(block_state.generator, list) and len(block_state.generator) != batch_size: + raise ValueError( + f"You have passed a list of generators of length {len(block_state.generator)}, but requested an" + f" effective batch size of {batch_size}. Make sure the batch size matches the length of the" + " generators." + ) + # draw in float32 then cast: a reduced-precision draw consumes the RNG stream differently, + # which would break seed-compatibility with the reference implementation + latents = randn_tensor(shape, generator=block_state.generator, device=device, dtype=torch.float32) + latents = latents.to(block_state.dtype) + else: + latents = latents.to(device=device, dtype=block_state.dtype) + + block_state.latents = latents.flatten(2).transpose(1, 2) # pack to tokens: (B, seq_len, C) + block_state.img_ids = _prepare_image_ids(latent_height, latent_width, self.image_rope_scale).to(device) + + self.set_block_state(state, block_state) + return components, state + + +class Flux3PrepareImageCondLatentsStep(ModularPipelineBlocks): + model_name = "flux3" + + def __init__(self, image_rope_scale: float = 0.5): + self.image_rope_scale = image_rope_scale + super().__init__() + + @property + def description(self) -> str: + return ( + "Step that packs the reference image latents into one token sequence with their position ids for FLUX.3 " + "image-to-image generation. The references are told apart on the `l` position axis (-1, -2, ...)." + ) + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam( + "image_cond_latents", + required=True, + type_hint=list[torch.Tensor], + description="Normalized packed latents of the reference images, one tensor of shape (B, C, height_i//16, width_i//16) per image", + ), + InputParam("dtype", type_hint=torch.dtype, description="The dtype of the model inputs"), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "image_cond_hidden_states", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Reference image latent tokens, shape (batch_size, cond_seq_len, C)", + ), + OutputParam( + "image_cond_ids", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Position IDs (T, H, W, L) of the reference image tokens, shape (cond_seq_len, 4)", + ), + ] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + + device = components._execution_device + + cond_tokens, cond_ids = [], [] + for index, latents in enumerate(block_state.image_cond_latents): + latent_height, latent_width = latents.shape[-2:] + cond_tokens.append(latents.flatten(2).transpose(1, 2)) # pack to tokens: (B, seq_len, C) + cond_ids.append( + _prepare_image_ids(latent_height, latent_width, self.image_rope_scale, l_coord=-(1 + index)) + ) + + block_state.image_cond_hidden_states = torch.cat(cond_tokens, dim=1).to(device=device, dtype=block_state.dtype) + block_state.image_cond_ids = torch.cat(cond_ids, dim=0).to(device) + + self.set_block_state(state, block_state) + return components, state + + +class Flux3TextRoPEInputsStep(ModularPipelineBlocks): + model_name = "flux3" + + @property + def description(self) -> str: + return ( + "Step that prepares the 4D RoPE position IDs of the text tokens for FLUX.3 denoising. Place it after " + "the text encoder step." + ) + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template( + "prompt_embeds", + description="Text embeddings used to guide the generation. Can be generated from text_encoder step.", + ), + InputParam.template( + "negative_prompt_embeds", + description="Negative text embeddings used to guide the generation. Can be generated from text_encoder step.", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + name="txt_ids", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="4D position IDs (T, H, W, L) for text tokens, used for RoPE calculation.", + ), + OutputParam( + name="negative_txt_ids", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="4D position IDs (T, H, W, L) for negative text tokens, used for RoPE calculation.", + ), + ] + + @staticmethod + def _prepare_text_ids(x: torch.Tensor): + """Prepare 4D position IDs (T, H, W, L) of shape (seq_len, 4) for text tokens; text advances along the L axis.""" + seq_len = x.shape[1] + t = torch.arange(1) + h = torch.arange(1) + w = torch.arange(1) + seq_l = torch.arange(seq_len) + + return torch.cartesian_prod(t, h, w, seq_l).to(dtype=torch.float32) + + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + + device = block_state.prompt_embeds.device + block_state.txt_ids = self._prepare_text_ids(block_state.prompt_embeds).to(device) + + block_state.negative_txt_ids = None + if block_state.negative_prompt_embeds is not None: + block_state.negative_txt_ids = self._prepare_text_ids(block_state.negative_prompt_embeds).to(device) + + self.set_block_state(state, block_state) + return components, state diff --git a/src/diffusers/modular_pipelines/flux3/decoders.py b/src/diffusers/modular_pipelines/flux3/decoders.py new file mode 100644 index 000000000000..32f3d5a72e17 --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/decoders.py @@ -0,0 +1,170 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +from typing import Any, Union + +import numpy as np +import PIL +import torch + +from ...configuration_utils import FrozenDict +from ...models import AutoencoderKLFlux2 +from ...utils import logging +from ..modular_pipeline import ModularPipelineBlocks, PipelineState +from ..modular_pipeline_utils import ComponentSpec, InputParam, OutputParam +from .image_processor import Flux3ImageProcessor +from .modular_pipeline import Flux3ModularPipeline + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +class Flux3UnpackLatentsStep(ModularPipelineBlocks): + model_name = "flux3" + + @property + def description(self) -> str: + return "Step that unpacks the image latent tokens from the denoising step back onto their 2D grid." + + @property + def inputs(self) -> list[tuple[str, Any]]: + return [ + InputParam( + "latents", + required=True, + type_hint=torch.Tensor, + description="The denoised image latent tokens from the denoising step. Shape: (B, seq_len, C)", + ), + # no default: the size is already baked into the latents, every workflow sets these upstream + InputParam( + "height", + type_hint=int, + description="The height in pixels of the generated image; must match the size the latents were prepared at", + ), + InputParam( + "width", + type_hint=int, + description="The width in pixels of the generated image; must match the size the latents were prepared at", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "latents", + type_hint=torch.Tensor, + description="The denoised latents, unpacked to (B, C, height, width).", + ) + ] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + + latent_height = block_state.height // (components.vae_scale_factor * 2) + latent_width = block_state.width // (components.vae_scale_factor * 2) + if latent_height * latent_width != block_state.latents.shape[1]: + raise ValueError( + f"`latents` has {block_state.latents.shape[1]} tokens, which does not match " + f"height x width = {block_state.height} x {block_state.width} ({latent_height * latent_width} tokens)" + ) + + # unpack: (batch_size, latent_height * latent_width, num_channels) -> (batch_size, num_channels, latent_height, latent_width) + batch_size, _, num_channels = block_state.latents.shape + block_state.latents = block_state.latents.view(batch_size, latent_height, latent_width, num_channels).permute( + 0, 3, 1, 2 + ) + + self.set_block_state(state, block_state) + return components, state + + +class Flux3DecodeStep(ModularPipelineBlocks): + model_name = "flux3" + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ + ComponentSpec("vae", AutoencoderKLFlux2), + ComponentSpec( + "image_processor", + Flux3ImageProcessor, + config=FrozenDict({"vae_scale_factor": 16, "vae_latent_channels": 32}), + default_creation_method="from_config", + ), + ] + + @property + def description(self) -> str: + return ( + "Step that decodes the denoised latents into images using the FLUX.3 image VAE (the FLUX.2 packed " + "autoencoder) with batch norm denormalization." + ) + + @property + def inputs(self) -> list[tuple[str, Any]]: + return [ + InputParam.template("output_type"), + InputParam( + "latents", + required=True, + type_hint=torch.Tensor, + description="The denoised latents from the denoising step", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "images", + type_hint=Union[list[PIL.Image.Image], torch.Tensor, np.ndarray], + description="The generated images, can be a list of PIL.Image.Image, torch.Tensor or a numpy array", + ) + ] + + @staticmethod + def _unpatchify_latents(latents): + """Convert patchified latents back to regular format.""" + batch_size, num_channels_latents, height, width = latents.shape + latents = latents.reshape(batch_size, num_channels_latents // (2 * 2), 2, 2, height, width) + latents = latents.permute(0, 1, 4, 2, 5, 3) + latents = latents.reshape(batch_size, num_channels_latents // (2 * 2), height * 2, width * 2) + return latents + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + vae = components.vae + + latents = block_state.latents + + latents_bn_mean = vae.bn.running_mean.view(1, -1, 1, 1).to(latents.device, latents.dtype) + latents_bn_std = torch.sqrt(vae.bn.running_var.view(1, -1, 1, 1) + vae.config.batch_norm_eps).to( + latents.device, latents.dtype + ) + latents = latents * latents_bn_std + latents_bn_mean + + latents = self._unpatchify_latents(latents) + + block_state.images = vae.decode(latents.to(vae.dtype), return_dict=False)[0] + block_state.images = components.image_processor.postprocess( + block_state.images, output_type=block_state.output_type + ) + + self.set_block_state(state, block_state) + return components, state diff --git a/src/diffusers/modular_pipelines/flux3/denoise.py b/src/diffusers/modular_pipelines/flux3/denoise.py new file mode 100644 index 000000000000..52948b0f3bfa --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/denoise.py @@ -0,0 +1,278 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import inspect +from typing import Any + +import torch + +from ...configuration_utils import FrozenDict +from ...guiders import ClassifierFreeGuidance +from ...models import Flux3Transformer2DModel +from ...schedulers import FlowMatchEulerDiscreteScheduler +from ...utils import is_torch_xla_available, logging +from ..modular_pipeline import ( + BlockState, + LoopSequentialPipelineBlocks, + ModularPipelineBlocks, + PipelineState, +) +from ..modular_pipeline_utils import ComponentSpec, InputParam, OutputParam +from .modular_pipeline import Flux3ModularPipeline + + +if is_torch_xla_available(): + import torch_xla.core.xla_model as xm + + XLA_AVAILABLE = True +else: + XLA_AVAILABLE = False + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +class Flux3LoopDenoiser(ModularPipelineBlocks): + model_name = "flux3" + + def __init__(self, guidance_scale: float = 3.5): + self.guidance_scale = guidance_scale + super().__init__() + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ + ComponentSpec("transformer", Flux3Transformer2DModel), + ComponentSpec( + "guider", + ClassifierFreeGuidance, + config=FrozenDict({"guidance_scale": self.guidance_scale}), + default_creation_method="from_config", + ), + ] + + @property + def description(self) -> str: + return ( + "Step within the denoising loop that predicts the velocity of the image tokens. The static conditioning " + "fields (`img_ids`, the reference image tokens and their ids) arrive via `denoiser_input_fields`; " + "everything matching the transformer signature is forwarded. " + "This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` " + "object (e.g. `Flux3DenoiseStep`)" + ) + + @property + def inputs(self) -> list[tuple[str, Any]]: + return [ + InputParam( + "latents", + required=True, + type_hint=torch.Tensor, + description="The image latent tokens to denoise. Shape: (B, seq_len, C)", + ), + InputParam.template("denoiser_input_fields"), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [OutputParam("noise_pred", type_hint=torch.Tensor, description="The predicted velocity for this step")] + + @torch.no_grad() + def __call__( + self, components: Flux3ModularPipeline, block_state: BlockState, i: int, t: torch.Tensor + ) -> PipelineState: + denoiser_input_fields = block_state.denoiser_input_fields + guider_inputs = { + "encoder_hidden_states": ( + denoiser_input_fields.get("prompt_embeds"), + denoiser_input_fields.get("negative_prompt_embeds"), + ), + "txt_ids": ( + denoiser_input_fields.get("txt_ids"), + denoiser_input_fields.get("negative_txt_ids"), + ), + } + + # transformer kwargs = every collected static field matching the forward signature (`img_ids`, and the + # `image_cond_*` fields of image-to-image generation) + transformer_args = set(inspect.signature(components.transformer.forward).parameters.keys()) + transformer_kwargs = { + field_name: field_value + for field_name, field_value in denoiser_input_fields.items() + if field_name in transformer_args and field_name not in guider_inputs + } + + latents = block_state.latents.to(components.transformer.dtype) + # the flow timestep in [0, 1] (scheduler timesteps are sigmas * 1000); the transformer applies it to every + # image token while the reference image tokens and the text context stay at 0 + timestep = (t / 1000).expand(latents.shape[0]).to(latents.dtype) + + components.guider.set_state(step=i, num_inference_steps=block_state.num_inference_steps, timestep=t) + guider_state = components.guider.prepare_inputs(guider_inputs) + + for guider_state_batch in guider_state: + components.guider.prepare_models(components.transformer) + cond_kwargs = {input_name: getattr(guider_state_batch, input_name) for input_name in guider_inputs.keys()} + + guider_state_batch.noise_pred = components.transformer( + hidden_states=latents, + timestep=timestep, + return_dict=False, + **transformer_kwargs, + **cond_kwargs, + )[0] + components.guider.cleanup_models(components.transformer) + + block_state.noise_pred = components.guider(guider_state)[0] + + return components, block_state + + +class Flux3LoopAfterDenoiser(ModularPipelineBlocks): + model_name = "flux3" + + @property + def expected_components(self) -> list[ComponentSpec]: + # config-less: the scheduler config is owned by `Flux3SetTimestepsStep`, whose spec is combined first + return [ComponentSpec("scheduler", FlowMatchEulerDiscreteScheduler, default_creation_method="from_config")] + + @property + def description(self) -> str: + return ( + "Step within the denoising loop that updates the latents after denoising. " + "This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` " + "object (e.g. `Flux3DenoiseLoopWrapper`)" + ) + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [OutputParam("latents", type_hint=torch.Tensor, description="The denoised latents")] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, block_state: BlockState, i: int, t: torch.Tensor): + latents_dtype = block_state.latents.dtype + block_state.latents = components.scheduler.step( + block_state.noise_pred, + t, + block_state.latents, + return_dict=False, + )[0] + + if block_state.latents.dtype != latents_dtype: + if torch.backends.mps.is_available(): + block_state.latents = block_state.latents.to(latents_dtype) + + return components, block_state + + +class Flux3DenoiseLoopWrapper(LoopSequentialPipelineBlocks): + model_name = "flux3" + + @property + def description(self) -> str: + return ( + "Pipeline block that iteratively denoises the latents over `timesteps`. " + "The specific steps within each iteration can be customized with `sub_blocks` attribute" + ) + + @property + def loop_expected_components(self) -> list[ComponentSpec]: + return [ + # config-less: the scheduler config is owned by `Flux3SetTimestepsStep`, whose spec is combined first + ComponentSpec("scheduler", FlowMatchEulerDiscreteScheduler, default_creation_method="from_config"), + ComponentSpec("transformer", Flux3Transformer2DModel), + ] + + @property + def loop_inputs(self) -> list[InputParam]: + return [ + InputParam( + "timesteps", + required=True, + type_hint=torch.Tensor, + description="The timesteps to use for the denoising process.", + ), + InputParam( + "num_inference_steps", + required=True, + type_hint=int, + description="The number of inference steps to use for the denoising process.", + ), + ] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + + block_state.num_warmup_steps = max( + len(block_state.timesteps) - block_state.num_inference_steps * components.scheduler.order, 0 + ) + + with self.progress_bar(total=block_state.num_inference_steps) as progress_bar: + for i, t in enumerate(block_state.timesteps): + components, block_state = self.loop_step(components, block_state, i=i, t=t) + + if i == len(block_state.timesteps) - 1 or ( + (i + 1) > block_state.num_warmup_steps and (i + 1) % components.scheduler.order == 0 + ): + progress_bar.update() + + if XLA_AVAILABLE: + xm.mark_step() + + self.set_block_state(state, block_state) + return components, state + + +# auto_docstring +class Flux3DenoiseStep(Flux3DenoiseLoopWrapper): + """ + Denoise step that iteratively denoises the latents for FLUX.3. + Its loop logic is defined in `Flux3DenoiseLoopWrapper.__call__` method At each iteration, it runs blocks defined + in `sub_blocks` sequentially: + - `Flux3LoopDenoiser` + - `Flux3LoopAfterDenoiser` + This block supports text-to-image and image-to-image generation. + + Components: + transformer (`Flux3Transformer2DModel`) guider (`ClassifierFreeGuidance`) scheduler + (`FlowMatchEulerDiscreteScheduler`) + + Inputs: + timesteps (`Tensor`): + The timesteps to use for the denoising process. + num_inference_steps (`int`): + The number of inference steps to use for the denoising process. + latents (`Tensor`): + The image latent tokens to denoise. Shape: (B, seq_len, C) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + The denoised latents + """ + + block_classes = [Flux3LoopDenoiser(), Flux3LoopAfterDenoiser()] + block_names = ["denoiser", "after_denoiser"] + + @property + def description(self) -> str: + return ( + "Denoise step that iteratively denoises the latents for FLUX.3. \n" + "Its loop logic is defined in `Flux3DenoiseLoopWrapper.__call__` method \n" + "At each iteration, it runs blocks defined in `sub_blocks` sequentially:\n" + " - `Flux3LoopDenoiser`\n" + " - `Flux3LoopAfterDenoiser`\n" + "This block supports text-to-image and image-to-image generation." + ) diff --git a/src/diffusers/modular_pipelines/flux3/encoders.py b/src/diffusers/modular_pipelines/flux3/encoders.py new file mode 100644 index 000000000000..5bd1be7accd8 --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/encoders.py @@ -0,0 +1,328 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math + +import PIL.Image +import torch +from transformers import AutoProcessor, Qwen3VLForConditionalGeneration + +from ...configuration_utils import FrozenDict +from ...models import AutoencoderKLFlux2 +from ...utils import logging +from ..modular_pipeline import ModularPipelineBlocks, PipelineState +from ..modular_pipeline_utils import ComponentSpec, InputParam, OutputParam +from .image_processor import Flux3ImageProcessor +from .modular_pipeline import Flux3ModularPipeline + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +class Flux3ProcessImagesInputStep(ModularPipelineBlocks): + model_name = "flux3" + + @property + def description(self) -> str: + return ( + "Image preprocess step for FLUX.3 image-to-image generation. Each reference image keeps its own " + "resolution: it is converted to RGB, downscaled only when it exceeds 4096 x 4096 pixels, and " + "center-cropped to a multiple of 16 pixels." + ) + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ + ComponentSpec( + "image_processor", + Flux3ImageProcessor, + config=FrozenDict({"vae_scale_factor": 16, "vae_latent_channels": 32}), + default_creation_method="from_config", + ), + ] + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template( + "image", + description="The reference image or images to condition the generation on; each keeps its own " + "resolution (center-cropped to the 16-pixel grid)", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "condition_images", + type_hint=list[torch.Tensor], + description="The preprocessed reference images, one tensor of shape (1, 3, height_i, width_i) in [-1, 1] per image", + ) + ] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState): + block_state = self.get_block_state(state) + + images = block_state.image if isinstance(block_state.image, list) else [block_state.image] + condition_images = [] + for image in images: + if not isinstance(image, PIL.Image.Image): + raise ValueError(f"`image` must be a `PIL.Image.Image` or a list of them, got {type(image)}") + # references keep their own resolution: downscaled only above 4096 x 4096 pixels, then center-cropped to + # the 16-pixel grid (`resize_mode="crop"` is a pure crop for this processor) + image = components.image_processor._resize_if_exceeds_area(image, 4096 * 4096) + condition_images.append(components.image_processor.preprocess(image, resize_mode="crop")) + block_state.condition_images = condition_images + + self.set_block_state(state, block_state) + return components, state + + +class Flux3VaeEncoderStep(ModularPipelineBlocks): + model_name = "flux3" + + @property + def description(self) -> str: + return ( + "VAE Encoder step that encodes the preprocessed reference images into normalized packed latents, one " + "latent tensor per image." + ) + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ComponentSpec("vae", AutoencoderKLFlux2)] + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam( + "condition_images", + required=True, + type_hint=list[torch.Tensor], + description="The preprocessed reference images, one tensor of shape (1, 3, height_i, width_i) in [-1, 1] per image", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "image_cond_latents", + type_hint=list[torch.Tensor], + description="Normalized packed latents of the reference images, one tensor of shape (1, 128, height_i//16, width_i//16) per image", + ) + ] + + @staticmethod + def _patchify_latents(latents): + """Convert latents to patchified format: (B, C, H, W) -> (B, C*4, H//2, W//2).""" + batch_size, num_channels_latents, height, width = latents.shape + latents = latents.view(batch_size, num_channels_latents, height // 2, 2, width // 2, 2) + latents = latents.permute(0, 1, 3, 5, 2, 4) + latents = latents.reshape(batch_size, num_channels_latents * 4, height // 2, width // 2) + return latents + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + + vae = components.vae + device = components._execution_device + + latents_bn_mean = vae.bn.running_mean.view(1, -1, 1, 1).to(device, vae.dtype) + latents_bn_std = torch.sqrt(vae.bn.running_var.view(1, -1, 1, 1) + vae.config.batch_norm_eps).to( + device, vae.dtype + ) + + image_cond_latents = [] + for image in block_state.condition_images: + latents = vae.encode(image.to(device=device, dtype=vae.dtype)).latent_dist.mode() + latents = self._patchify_latents(latents) + image_cond_latents.append((latents - latents_bn_mean) / latents_bn_std) + block_state.image_cond_latents = image_cond_latents + + self.set_block_state(state, block_state) + return components, state + + +class Flux3TextEncoderStep(ModularPipelineBlocks): + model_name = "flux3" + + def __init__(self, pad_multiple: int = 80): + # the text window is right-padded to a multiple of this many tokens (capped at `max_sequence_length`); the + # padding is masked inside Qwen3-VL but not in the transformer, so the window size is part of the model recipe + self.pad_multiple = pad_multiple + super().__init__() + + @property + def description(self) -> str: + return ( + "Text Encoder step that generates text embeddings using Qwen3-VL to guide the generation. Takes one " + f"prompt per call; it goes through the chat template and is right-padded to a multiple of " + f"{self.pad_multiple} tokens." + ) + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ + ComponentSpec("text_encoder", Qwen3VLForConditionalGeneration), + ComponentSpec("tokenizer", AutoProcessor), + ] + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template("prompt", description="The prompt to guide the generation (one prompt per call)"), + # no `default=""`: a materialized default would be indistinguishable from a user-passed value, + # and the guider-disabled warning depends on that distinction (the block falls back to "" itself) + InputParam.template("negative_prompt", description="The prompt not to guide the generation"), + InputParam.template( + "max_sequence_length", + default=8192, + description="Maximum number of prompt tokens; longer prompts are truncated.", + ), + InputParam( + "text_encoder_out_layers", + type_hint=tuple[int], + default=(4, 8, 12, 16, 20, 24, 28, 32), + required=False, + description="Indices of the text-encoder hidden layers whose states are stacked to form the text embeddings", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "prompt_embeds", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Text embeddings (stacked Qwen3-VL hidden states) used to guide the image generation", + ), + OutputParam( + "negative_prompt_embeds", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Negative text embeddings used to guide the image generation", + ), + ] + + @staticmethod + def check_inputs(block_state): + # one prompt per call (the text window is sized per prompt, and the transformer attends to its padding); + # `num_images_per_prompt` generates several images from it + if not isinstance(block_state.prompt, str): + raise ValueError( + f"`prompt` must be a str; batched prompts are not supported, got {type(block_state.prompt).__name__}." + ) + if block_state.negative_prompt is not None and not isinstance(block_state.negative_prompt, str): + raise ValueError( + "`negative_prompt` must be a str or None; batched prompts are not supported, " + f"got {type(block_state.negative_prompt).__name__}." + ) + + @staticmethod + def _get_qwen3_vl_prompt_embeds( + text_encoder: Qwen3VLForConditionalGeneration, + tokenizer: AutoProcessor, + prompt: str, + device: torch.device | None = None, + max_sequence_length: int = 8192, + pad_multiple: int = 80, + hidden_states_layers: tuple[int, ...] = (4, 8, 12, 16, 20, 24, 28, 32), + ): + dtype = text_encoder.dtype + device = text_encoder.device if device is None else device + + text = tokenizer.apply_chat_template( + [{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True + ) + # the window is the prompt's length rounded up to the pad multiple, capped at `max_sequence_length` + real_length = tokenizer.tokenizer(text, return_tensors="pt", truncation=True, max_length=max_sequence_length)[ + "input_ids" + ].shape[1] + padded_length = min(math.ceil(real_length / pad_multiple) * pad_multiple, max_sequence_length) + inputs = tokenizer.tokenizer( + text, + return_tensors="pt", + padding="max_length", + padding_side="right", + truncation=True, + max_length=padded_length, + ) + + # Qwen3-VL computes the position id of the padding tokens differently across transformers versions (1 in + # 4.57, the training environment; 0 in 5.x). FLUX.3 attends to the padding tokens, so the position ids are + # passed explicitly to match the training environment. + input_ids, attention_mask = inputs["input_ids"].to(device), inputs["attention_mask"].to(device) + position_ids = torch.arange(padded_length, device=device)[None].masked_fill(attention_mask == 0, 1) + + output = text_encoder( + input_ids=input_ids, + attention_mask=attention_mask, + position_ids=position_ids, + output_hidden_states=True, + use_cache=False, + ) + + # Only use outputs from intermediate layers and stack them along the channel dimension + out = torch.stack([output.hidden_states[k] for k in hidden_states_layers], dim=1) + out = out.to(dtype=dtype, device=device) + + batch_size, num_channels, seq_len, hidden_dim = out.shape + prompt_embeds = out.permute(0, 2, 1, 3).reshape(batch_size, seq_len, num_channels * hidden_dim) + + return prompt_embeds + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + self.check_inputs(block_state) + + device = components._execution_device + + block_state.prompt_embeds = self._get_qwen3_vl_prompt_embeds( + text_encoder=components.text_encoder, + tokenizer=components.tokenizer, + prompt=block_state.prompt, + device=device, + max_sequence_length=block_state.max_sequence_length, + pad_multiple=self.pad_multiple, + hidden_states_layers=block_state.text_encoder_out_layers, + ) + + # The guider is not a component of this block. Combined with the denoiser, the pipeline's guider decides + # through `requires_unconditional_embeds` whether unconditional embeddings are needed (the negative prompt + # then defaults to ""); with the guider disabled a provided `negative_prompt` is ignored with a warning. + # Standalone (no guider), the negative prompt is encoded only when the caller passes one. + has_guider = getattr(components, "guider", None) is not None + if components.requires_unconditional_embeds or (block_state.negative_prompt is not None and not has_guider): + block_state.negative_prompt_embeds = self._get_qwen3_vl_prompt_embeds( + text_encoder=components.text_encoder, + tokenizer=components.tokenizer, + prompt=block_state.negative_prompt or "", + device=device, + max_sequence_length=block_state.max_sequence_length, + pad_multiple=self.pad_multiple, + hidden_states_layers=block_state.text_encoder_out_layers, + ) + else: + if block_state.negative_prompt is not None: + logger.warning("`negative_prompt` is provided but the guider is disabled, it will be ignored.") + block_state.negative_prompt_embeds = None + + self.set_block_state(state, block_state) + return components, state diff --git a/src/diffusers/modular_pipelines/flux3/image_processor.py b/src/diffusers/modular_pipelines/flux3/image_processor.py new file mode 100644 index 000000000000..08b8df17fdff --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/image_processor.py @@ -0,0 +1,116 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math + +import PIL.Image + +from ...image_processor import VaeImageProcessor + + +class Flux3ImageProcessor(VaeImageProcessor): + r""" + Image processor for FLUX.3: preprocesses the reference images of image-to-image generation and postprocesses the + generated images. + + Args: + do_resize (`bool`, *optional*, defaults to `True`): + Whether to downscale the image's (height, width) dimensions to multiples of `vae_scale_factor`. Can accept + `height` and `width` arguments from [`image_processor.VaeImageProcessor.preprocess`] method. + vae_scale_factor (`int`, *optional*, defaults to `16`): + VAE (spatial) scale factor, including the 2x2 packing of the latents. If `do_resize` is `True`, the image + is automatically resized to multiples of this factor. + vae_latent_channels (`int`, *optional*, defaults to `32`): + VAE latent channels. + do_normalize (`bool`, *optional*, defaults to `True`): + Whether to normalize the image to [-1,1]. + do_binarize (`bool`, *optional*, defaults to `False`): + Whether to binarize the image to 0/1. + do_convert_rgb (`bool`, *optional*, defaults to be `True`): + Whether to convert the images to RGB format. + do_convert_grayscale (`bool`, *optional*, defaults to be `False`): + Whether to convert the images to grayscale format. + """ + + # Copied from diffusers.pipelines.flux2.image_processor.Flux2ImageProcessor.__init__ + def __init__( + self, + do_resize: bool = True, + vae_scale_factor: int = 16, + vae_latent_channels: int = 32, + do_normalize: bool = True, + do_binarize: bool = False, + do_convert_rgb: bool = True, + do_convert_grayscale: bool = False, + ): + super().__init__( + do_resize=do_resize, + vae_scale_factor=vae_scale_factor, + vae_latent_channels=vae_latent_channels, + do_normalize=do_normalize, + do_binarize=do_binarize, + do_convert_rgb=do_convert_rgb, + do_convert_grayscale=do_convert_grayscale, + ) + + @staticmethod + # Copied from diffusers.pipelines.flux2.image_processor.Flux2ImageProcessor._resize_to_target_area + def _resize_to_target_area(image: PIL.Image.Image, target_area: int = 1024 * 1024) -> PIL.Image.Image: + image_width, image_height = image.size + + scale = math.sqrt(target_area / (image_width * image_height)) + width = int(image_width * scale) + height = int(image_height * scale) + + return image.resize((width, height), PIL.Image.Resampling.LANCZOS) + + @staticmethod + # Copied from diffusers.pipelines.flux2.image_processor.Flux2ImageProcessor._resize_if_exceeds_area with Flux2->Flux3 + def _resize_if_exceeds_area(image, target_area=1024 * 1024) -> PIL.Image.Image: + image_width, image_height = image.size + pixel_count = image_width * image_height + if pixel_count <= target_area: + return image + return Flux3ImageProcessor._resize_to_target_area(image, target_area) + + # Copied from diffusers.pipelines.flux2.image_processor.Flux2ImageProcessor._resize_and_crop + def _resize_and_crop( + self, + image: PIL.Image.Image, + width: int, + height: int, + ) -> PIL.Image.Image: + r""" + center crop the image to the specified width and height. + + Args: + image (`PIL.Image.Image`): + The image to resize and crop. + width (`int`): + The width to resize the image to. + height (`int`): + The height to resize the image to. + + Returns: + `PIL.Image.Image`: + The resized and cropped image. + """ + image_width, image_height = image.size + + left = (image_width - width) // 2 + top = (image_height - height) // 2 + right = left + width + bottom = top + height + + return image.crop((left, top, right, bottom)) diff --git a/src/diffusers/modular_pipelines/flux3/inputs.py b/src/diffusers/modular_pipelines/flux3/inputs.py new file mode 100644 index 000000000000..0e08f7173797 --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/inputs.py @@ -0,0 +1,252 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math + +import torch + +from ...models import Flux3Transformer2DModel +from ...utils import logging +from ..modular_pipeline import ModularPipelineBlocks, PipelineState +from ..modular_pipeline_utils import ComponentSpec, InputParam, OutputParam +from .modular_pipeline import Flux3ModularPipeline + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +# Copied from diffusers.modular_pipelines.qwenimage.inputs.repeat_tensor_to_batch_size +def repeat_tensor_to_batch_size( + input_name: str, + input_tensor: torch.Tensor, + batch_size: int, + num_images_per_prompt: int = 1, +) -> torch.Tensor: + """Repeat tensor elements to match the final batch size. + + This function expands a tensor's batch dimension to match the final batch size (batch_size * num_images_per_prompt) + by repeating each element along dimension 0. + + The input tensor must have batch size 1 or batch_size. The function will: + - If batch size is 1: repeat each element (batch_size * num_images_per_prompt) times + - If batch size equals batch_size: repeat each element num_images_per_prompt times + + Args: + input_name (str): Name of the input tensor (used for error messages) + input_tensor (torch.Tensor): The tensor to repeat. Must have batch size 1 or batch_size. + batch_size (int): The base batch size (number of prompts) + num_images_per_prompt (int, optional): Number of images to generate per prompt. Defaults to 1. + + Returns: + torch.Tensor: The repeated tensor with final batch size (batch_size * num_images_per_prompt) + + Raises: + ValueError: If input_tensor is not a torch.Tensor or has invalid batch size + + Examples: + tensor = torch.tensor([[1, 2, 3]]) # shape: [1, 3] repeated = repeat_tensor_to_batch_size("image", tensor, + batch_size=2, num_images_per_prompt=2) repeated # tensor([[1, 2, 3], [1, 2, 3], [1, 2, 3], [1, 2, 3]]) - shape: + [4, 3] + + tensor = torch.tensor([[1, 2, 3], [4, 5, 6]]) # shape: [2, 3] repeated = repeat_tensor_to_batch_size("image", + tensor, batch_size=2, num_images_per_prompt=2) repeated # tensor([[1, 2, 3], [1, 2, 3], [4, 5, 6], [4, 5, 6]]) + - shape: [4, 3] + """ + # make sure input is a tensor + if not isinstance(input_tensor, torch.Tensor): + raise ValueError(f"`{input_name}` must be a tensor") + + # make sure input tensor e.g. image_latents has batch size 1 or batch_size same as prompts + if input_tensor.shape[0] == 1: + repeat_by = batch_size * num_images_per_prompt + elif input_tensor.shape[0] == batch_size: + repeat_by = num_images_per_prompt + else: + raise ValueError(f"`{input_name}` must have batch size 1 or {batch_size}, but got {input_tensor.shape[0]}") + + # expand the tensor to match the batch_size * num_images_per_prompt + input_tensor = input_tensor.repeat_interleave(repeat_by, dim=0) + + return input_tensor + + +class Flux3TextInputStep(ModularPipelineBlocks): + model_name = "flux3" + + @property + def description(self) -> str: + return ( + "This step:\n" + " 1. Determines `batch_size` from `prompt_embeds` and the compute `dtype` from the transformer\n" + " 2. Ensures all text embeddings have consistent batch sizes (batch_size * num_images_per_prompt)" + ) + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ComponentSpec("transformer", Flux3Transformer2DModel)] + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template("num_images_per_prompt"), + InputParam( + "prompt_embeds", + required=True, + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Pre-generated text embeddings. Can be generated from text_encoder step.", + ), + InputParam( + "negative_prompt_embeds", + required=False, + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Pre-generated negative text embeddings. Can be generated from text_encoder step.", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "batch_size", + type_hint=int, + description="Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt", + ), + OutputParam( + "dtype", + type_hint=torch.dtype, + description="Data type of model tensor inputs (the transformer's dtype)", + ), + OutputParam( + "prompt_embeds", + type_hint=torch.Tensor, + kwargs_type="denoiser_input_fields", + description="Text embeddings used to guide the image generation", + ), + OutputParam( + "negative_prompt_embeds", + type_hint=torch.Tensor, + kwargs_type="denoiser_input_fields", + description="Negative text embeddings used to guide the image generation", + ), + ] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + + block_state.batch_size = block_state.prompt_embeds.shape[0] + # the denoiser consumes every tensor this chain prepares, so its dtype is the compute dtype: + # the text encoder may run in a different precision (e.g. a bf16 encoder with an fp32 transformer) + block_state.dtype = components.transformer.dtype + + block_state.prompt_embeds = repeat_tensor_to_batch_size( + "prompt_embeds", block_state.prompt_embeds, block_state.batch_size, block_state.num_images_per_prompt + ).to(block_state.dtype) + + if block_state.negative_prompt_embeds is not None: + block_state.negative_prompt_embeds = repeat_tensor_to_batch_size( + "negative_prompt_embeds", + block_state.negative_prompt_embeds, + block_state.batch_size, + block_state.num_images_per_prompt, + ).to(block_state.dtype) + elif components.requires_unconditional_embeds: + raise ValueError( + "The guider needs unconditional embeddings but `negative_prompt_embeds` is None. Pass `negative_prompt` " + "(an empty string for plain classifier-free guidance) to the text encoder step, or disable the guider." + ) + + self.set_block_state(state, block_state) + return components, state + + +class Flux3ImageCondInputStep(ModularPipelineBlocks): + model_name = "flux3" + + def __init__(self, default_height: int = 1024, default_width: int = 1024): + # the generation size defaults to this area at the first reference image's aspect ratio + self.default_height = default_height + self.default_width = default_width + super().__init__() + + @property + def description(self) -> str: + return ( + "Step that expands the reference image latents to the final batch size and sets the generation " + f"`height`/`width` when not provided: {self.default_height} x {self.default_width} pixels of area at the " + "first reference image's aspect ratio, on the 16-pixel grid (a given side overrides its default). The " + "reference images keep their own resolution, the generation size is independent of them." + ) + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam( + "image_cond_latents", + required=True, + type_hint=list[torch.Tensor], + description="Normalized packed latents of the reference images, one tensor of shape (B, C, height_i//16, width_i//16) per image", + ), + InputParam( + "height", + type_hint=int, + description="The height in pixels of the generated image; defaults to the first reference image's " + "aspect ratio at the default area", + ), + InputParam( + "width", + type_hint=int, + description="The width in pixels of the generated image; defaults to the first reference image's " + "aspect ratio at the default area", + ), + InputParam.template("num_images_per_prompt"), + InputParam("batch_size", required=True, type_hint=int), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "image_cond_latents", + type_hint=list[torch.Tensor], + description="Normalized packed latents of the reference images, expanded to the final batch size", + ), + OutputParam("height", type_hint=int, description="The height in pixels of the generated image"), + OutputParam("width", type_hint=int, description="The width in pixels of the generated image"), + ] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + + # the generation size defaults to the default area at the first reference's aspect ratio, rounded to a + # multiple of 16 pixels (one token); a given `height` or `width` overrides its side + if block_state.height is None or block_state.width is None: + multiple_of = components.vae_scale_factor * 2 + latent_height, latent_width = block_state.image_cond_latents[0].shape[-2:] + scale = math.sqrt(self.default_height * self.default_width / (latent_height * latent_width)) / multiple_of + block_state.height = block_state.height or max(multiple_of, round(latent_height * scale) * multiple_of) + block_state.width = block_state.width or max(multiple_of, round(latent_width * scale) * multiple_of) + + block_state.image_cond_latents = [ + repeat_tensor_to_batch_size( + "image_cond_latents", latents, block_state.batch_size, block_state.num_images_per_prompt + ) + for latents in block_state.image_cond_latents + ] + + self.set_block_state(state, block_state) + return components, state diff --git a/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py b/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py new file mode 100644 index 000000000000..bb049e300c47 --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py @@ -0,0 +1,398 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from ...utils import logging +from ..modular_pipeline import AutoPipelineBlocks, SequentialPipelineBlocks +from ..modular_pipeline_utils import InsertableDict, OutputParam +from .before_denoise import ( + Flux3PrepareImageCondLatentsStep, + Flux3PrepareLatentsStep, + Flux3SetTimestepsStep, + Flux3TextRoPEInputsStep, +) +from .decoders import Flux3DecodeStep, Flux3UnpackLatentsStep +from .denoise import Flux3DenoiseStep +from .encoders import Flux3ProcessImagesInputStep, Flux3TextEncoderStep, Flux3VaeEncoderStep +from .inputs import Flux3ImageCondInputStep, Flux3TextInputStep + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +### +### VAE encoder (reference images) +### + + +# auto_docstring +class Flux3VaeEncoderSequentialStep(SequentialPipelineBlocks): + """ + VAE encoder step that preprocesses and encodes the reference images into their latent representations. + + Components: + image_processor (`Flux3ImageProcessor`) vae (`AutoencoderKLFlux2`) + + Inputs: + image (`Image | list`): + The reference image or images to condition the generation on; each keeps its own resolution + (center-cropped to the 16-pixel grid) + + Outputs: + condition_images (`list`): + The preprocessed reference images, one tensor of shape (1, 3, height_i, width_i) in [-1, 1] per image + image_cond_latents (`list`): + Normalized packed latents of the reference images, one tensor of shape (1, 128, height_i//16, + width_i//16) per image + """ + + model_name = "flux3" + block_classes = [Flux3ProcessImagesInputStep(), Flux3VaeEncoderStep()] + block_names = ["preprocess", "encode"] + + @property + def description(self) -> str: + return "VAE encoder step that preprocesses and encodes the reference images into their latent representations." + + +# auto_docstring +class Flux3AutoVaeEncoderStep(AutoPipelineBlocks): + """ + VAE encoder step that encodes the reference images into their latent representations. + - `Flux3VaeEncoderSequentialStep` is used when `image` is provided (image-to-image). + - If `image` is not provided, step will be skipped. + + Components: + image_processor (`Flux3ImageProcessor`) vae (`AutoencoderKLFlux2`) + + Inputs: + image (`Image | list`, *optional*): + The reference image or images to condition the generation on; each keeps its own resolution + (center-cropped to the 16-pixel grid) + + Outputs: + condition_images (`list`): + The preprocessed reference images, one tensor of shape (1, 3, height_i, width_i) in [-1, 1] per image + image_cond_latents (`list`): + Normalized packed latents of the reference images, one tensor of shape (1, 128, height_i//16, + width_i//16) per image + """ + + model_name = "flux3" + block_classes = [Flux3VaeEncoderSequentialStep] + block_names = ["image_conditioning"] + block_trigger_inputs = ["image"] + + @property + def description(self): + return ( + "VAE encoder step that encodes the reference images into their latent representations.\n" + " - `Flux3VaeEncoderSequentialStep` is used when `image` is provided (image-to-image).\n" + " - If `image` is not provided, step will be skipped." + ) + + +### +### Core denoise +### + +Flux3CoreDenoiseBlocks = InsertableDict( + [ + ("input", Flux3TextInputStep()), + ("prepare_latents", Flux3PrepareLatentsStep()), + ("set_timesteps", Flux3SetTimestepsStep()), + ("prepare_rope_inputs", Flux3TextRoPEInputsStep()), + ("denoise", Flux3DenoiseStep()), + ("after_denoise", Flux3UnpackLatentsStep()), + ] +) + + +# auto_docstring +class Flux3CoreDenoiseStep(SequentialPipelineBlocks): + """ + Core denoise step that performs the denoising process for FLUX.3, for text-to-image generation. + + Components: + transformer (`Flux3Transformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) guider + (`ClassifierFreeGuidance`) + + Inputs: + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + prompt_embeds (`Tensor`): + Pre-generated text embeddings. Can be generated from text_encoder step. + negative_prompt_embeds (`Tensor`, *optional*): + Pre-generated negative text embeddings. Can be generated from text_encoder step. + height (`int`, *optional*, defaults to 1024): + The height in pixels of the generated image. + width (`int`, *optional*, defaults to 1024): + The width in pixels of the generated image. + latents (`Tensor`, *optional*): + Pre-generated noisy latents for image generation. + generator (`Generator`, *optional*): + Torch generator for deterministic generation. + num_inference_steps (`int`, *optional*, defaults to 50): + The number of denoising steps. + timesteps (`Tensor`, *optional*): + Timesteps for the denoising process. (overrides `num_inference_steps` when provided) + sigmas (`list`, *optional*): + Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + Denoised latents. + """ + + model_name = "flux3" + block_classes = Flux3CoreDenoiseBlocks.values() + block_names = Flux3CoreDenoiseBlocks.keys() + + @property + def description(self): + return "Core denoise step that performs the denoising process for FLUX.3, for text-to-image generation." + + @property + def outputs(self): + return [ + OutputParam.template("latents"), + ] + + +Flux3ImageConditionedCoreDenoiseBlocks = InsertableDict( + [ + ("input", Flux3TextInputStep()), + ("cond_input", Flux3ImageCondInputStep()), + ("prepare_latents", Flux3PrepareLatentsStep()), + ("prepare_image_cond_latents", Flux3PrepareImageCondLatentsStep()), + ("set_timesteps", Flux3SetTimestepsStep()), + ("prepare_rope_inputs", Flux3TextRoPEInputsStep()), + ("denoise", Flux3DenoiseStep()), + ("after_denoise", Flux3UnpackLatentsStep()), + ] +) + + +# auto_docstring +class Flux3ImageConditionedCoreDenoiseStep(SequentialPipelineBlocks): + """ + Core denoise step that performs the denoising process for FLUX.3 with reference images (image-to-image). + + Components: + transformer (`Flux3Transformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) guider + (`ClassifierFreeGuidance`) + + Inputs: + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + prompt_embeds (`Tensor`): + Pre-generated text embeddings. Can be generated from text_encoder step. + negative_prompt_embeds (`Tensor`, *optional*): + Pre-generated negative text embeddings. Can be generated from text_encoder step. + image_cond_latents (`list`): + Normalized packed latents of the reference images, one tensor of shape (B, C, height_i//16, width_i//16) + per image + height (`int`, *optional*): + The height in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + width (`int`, *optional*): + The width in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + latents (`Tensor`, *optional*): + Pre-generated noisy latents for image generation. + generator (`Generator`, *optional*): + Torch generator for deterministic generation. + num_inference_steps (`int`, *optional*, defaults to 50): + The number of denoising steps. + timesteps (`Tensor`, *optional*): + Timesteps for the denoising process. (overrides `num_inference_steps` when provided) + sigmas (`list`, *optional*): + Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + Denoised latents. + """ + + model_name = "flux3" + block_classes = Flux3ImageConditionedCoreDenoiseBlocks.values() + block_names = Flux3ImageConditionedCoreDenoiseBlocks.keys() + + @property + def description(self): + return ( + "Core denoise step that performs the denoising process for FLUX.3 with reference images (image-to-image)." + ) + + @property + def outputs(self): + return [ + OutputParam.template("latents"), + ] + + +# auto_docstring +class Flux3AutoCoreDenoiseStep(AutoPipelineBlocks): + """ + Core denoise step that selects the FLUX.3 denoising workflow by the conditioning inputs. + - `Flux3ImageConditionedCoreDenoiseStep` is used when `image_cond_latents` is provided (image-to-image). + - `Flux3CoreDenoiseStep` is used otherwise (text-to-image). + + Components: + transformer (`Flux3Transformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) guider + (`ClassifierFreeGuidance`) + + Inputs: + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + prompt_embeds (`Tensor`): + Pre-generated text embeddings. Can be generated from text_encoder step. + negative_prompt_embeds (`Tensor`, *optional*): + Pre-generated negative text embeddings. Can be generated from text_encoder step. + image_cond_latents (`list`, *optional*): + Normalized packed latents of the reference images, one tensor of shape (B, C, height_i//16, width_i//16) + per image + height (`int`, *optional*, defaults to None or 1024, depending on the workflow): + The height in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + width (`int`, *optional*, defaults to None or 1024, depending on the workflow): + The width in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + latents (`Tensor`): + Pre-generated noisy latents for image generation. + generator (`Generator`, *optional*): + Torch generator for deterministic generation. + num_inference_steps (`int`): + The number of denoising steps. + timesteps (`Tensor`): + Timesteps for the denoising process. (overrides `num_inference_steps` when provided) + sigmas (`list`, *optional*): + Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + Denoised latents. + """ + + model_name = "flux3" + block_classes = [Flux3ImageConditionedCoreDenoiseStep, Flux3CoreDenoiseStep] + block_names = ["image_conditioned", "text2image"] + block_trigger_inputs = ["image_cond_latents", None] + + @property + def description(self): + return ( + "Core denoise step that selects the FLUX.3 denoising workflow by the conditioning inputs.\n" + " - `Flux3ImageConditionedCoreDenoiseStep` is used when `image_cond_latents` is provided (image-to-image).\n" + " - `Flux3CoreDenoiseStep` is used otherwise (text-to-image)." + ) + + +### +### Auto blocks +### + +AUTO_BLOCKS = InsertableDict( + [ + ("text_encoder", Flux3TextEncoderStep()), + ("vae_encoder", Flux3AutoVaeEncoderStep()), + ("denoise", Flux3AutoCoreDenoiseStep()), + ("decode", Flux3DecodeStep()), + ] +) + + +# auto_docstring +class Flux3AutoBlocks(SequentialPipelineBlocks): + """ + Auto blocks that perform text-to-image and image-to-image generation using FLUX.3. + + Supported workflows: + - `text2image`: requires `prompt` + - `image2image`: requires `prompt`, `image` + + Components: + text_encoder (`Qwen3VLForConditionalGeneration`) tokenizer (`AutoProcessor`) image_processor + (`Flux3ImageProcessor`) vae (`AutoencoderKLFlux2`) transformer (`Flux3Transformer2DModel`) scheduler + (`FlowMatchEulerDiscreteScheduler`) guider (`ClassifierFreeGuidance`) + + Inputs: + prompt (`str`): + The prompt to guide the generation (one prompt per call) + negative_prompt (`str`, *optional*): + The prompt not to guide the generation + max_sequence_length (`int`, *optional*, defaults to 8192): + Maximum number of prompt tokens; longer prompts are truncated. + text_encoder_out_layers (`tuple`, *optional*, defaults to (4, 8, 12, 16, 20, 24, 28, 32)): + Indices of the text-encoder hidden layers whose states are stacked to form the text embeddings + image (`Image | list`, *optional*): + The reference image or images to condition the generation on; each keeps its own resolution + (center-cropped to the 16-pixel grid) + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + image_cond_latents (`list`, *optional*): + Normalized packed latents of the reference images, one tensor of shape (B, C, height_i//16, width_i//16) + per image + height (`int`, *optional*, defaults to None or 1024, depending on the workflow): + The height in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + width (`int`, *optional*, defaults to None or 1024, depending on the workflow): + The width in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + latents (`Tensor`): + Pre-generated noisy latents for image generation. + generator (`Generator`, *optional*): + Torch generator for deterministic generation. + num_inference_steps (`int`): + The number of denoising steps. + timesteps (`Tensor`): + Timesteps for the denoising process. (overrides `num_inference_steps` when provided) + sigmas (`list`, *optional*): + Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + output_type (`str`, *optional*, defaults to pil): + Output format: 'pil', 'np', 'pt'. + + Outputs: + images (`list`): + Generated images. + latents (`Tensor`): + Denoised latents. + """ + + model_name = "flux3" + block_classes = AUTO_BLOCKS.values() + block_names = AUTO_BLOCKS.keys() + _workflow_map = { + "text2image": {"prompt": True}, + "image2image": {"prompt": True, "image": True}, + } + + @property + def description(self): + return "Auto blocks that perform text-to-image and image-to-image generation using FLUX.3." + + @property + def outputs(self): + return [ + OutputParam.template("images"), + OutputParam.template("latents"), + ] diff --git a/src/diffusers/modular_pipelines/flux3/modular_pipeline.py b/src/diffusers/modular_pipelines/flux3/modular_pipeline.py new file mode 100644 index 000000000000..574a3ace6f7b --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/modular_pipeline.py @@ -0,0 +1,48 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from ...utils import logging +from ..modular_pipeline import ModularPipeline + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +class Flux3ModularPipeline(ModularPipeline): + """ + A ModularPipeline for FLUX.3. + """ + + default_blocks_name = "Flux3AutoBlocks" + + @property + def vae_scale_factor(self): + vae_scale_factor = 8 + if getattr(self, "vae", None) is not None: + vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) + return vae_scale_factor + + @property + def num_channels_latents(self): + num_channels_latents = 128 + if getattr(self, "transformer", None): + num_channels_latents = self.transformer.config.in_channels + return num_channels_latents + + @property + def requires_unconditional_embeds(self): + if getattr(self, "guider", None) is None: + return False + return self.guider._enabled and self.guider.num_conditions > 1 diff --git a/src/diffusers/modular_pipelines/modular_pipeline.py b/src/diffusers/modular_pipelines/modular_pipeline.py index 69a8f730284f..d3717e878210 100644 --- a/src/diffusers/modular_pipelines/modular_pipeline.py +++ b/src/diffusers/modular_pipelines/modular_pipeline.py @@ -140,6 +140,7 @@ def _helios_pyramid_map_fn(config_dict=None): ("flux-kontext", _create_default_map_fn("FluxKontextModularPipeline")), ("flux2", _create_default_map_fn("Flux2ModularPipeline")), ("flux2-klein", _flux2_klein_map_fn), + ("flux3", _create_default_map_fn("Flux3ModularPipeline")), ("ideogram4", _create_default_map_fn("Ideogram4ModularPipeline")), ("krea2", _krea2_map_fn), ("qwenimage", _create_default_map_fn("QwenImageModularPipeline")), diff --git a/src/diffusers/utils/dummy_pt_objects.py b/src/diffusers/utils/dummy_pt_objects.py index 1598814f835a..714dce00e8a5 100644 --- a/src/diffusers/utils/dummy_pt_objects.py +++ b/src/diffusers/utils/dummy_pt_objects.py @@ -1320,6 +1320,21 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch"]) +class Flux3Transformer2DModel(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + class FluxControlNetModel(metaclass=DummyObject): _backends = ["torch"] diff --git a/src/diffusers/utils/dummy_torch_and_transformers_objects.py b/src/diffusers/utils/dummy_torch_and_transformers_objects.py index 17bca9f23414..ac3676f4448b 100644 --- a/src/diffusers/utils/dummy_torch_and_transformers_objects.py +++ b/src/diffusers/utils/dummy_torch_and_transformers_objects.py @@ -212,6 +212,36 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch", "transformers"]) +class Flux3AutoBlocks(metaclass=DummyObject): + _backends = ["torch", "transformers"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch", "transformers"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch", "transformers"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch", "transformers"]) + + +class Flux3ModularPipeline(metaclass=DummyObject): + _backends = ["torch", "transformers"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch", "transformers"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch", "transformers"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch", "transformers"]) + + class FluxAutoBlocks(metaclass=DummyObject): _backends = ["torch", "transformers"] diff --git a/tests/models/transformers/test_models_transformer_flux3.py b/tests/models/transformers/test_models_transformer_flux3.py new file mode 100644 index 000000000000..3538739f7c38 --- /dev/null +++ b/tests/models/transformers/test_models_transformer_flux3.py @@ -0,0 +1,127 @@ +# coding=utf-8 +# Copyright 2026 HuggingFace Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch + +from diffusers import Flux3Transformer2DModel +from diffusers.utils.torch_utils import randn_tensor + +from ...testing_utils import enable_full_determinism, torch_device +from ..testing_utils import ( + AttentionTesterMixin, + BaseModelTesterConfig, + MemoryTesterMixin, + ModelTesterMixin, + TorchCompileTesterMixin, + TrainingTesterMixin, +) + + +enable_full_determinism() + + +class Flux3TransformerTesterConfig(BaseModelTesterConfig): + @property + def model_class(self): + return Flux3Transformer2DModel + + @property + def output_shape(self) -> tuple[int, ...]: + return (16, 4) + + @property + def input_shape(self) -> tuple[int, ...]: + return (16, 4) + + @property + def main_input_name(self) -> str: + return "hidden_states" + + @property + def uses_custom_attn_processor(self) -> bool: + return True + + @property + def generator(self): + return torch.Generator("cpu").manual_seed(0) + + def get_init_dict(self) -> dict[str, int | list[int]]: + return { + "in_channels": 4, + "cond_in_channels": 4, + "num_layers": 1, + "num_single_layers": 1, + "attention_head_dim": 16, + "num_attention_heads": 2, + "joint_attention_dim": 32, + "timestep_channels": 32, + "axes_dims_rope": [4, 4, 4, 4], + } + + def get_dummy_inputs( + self, height: int = 4, width: int = 4, batch_size: int = 1, num_cond_tokens: int | None = None + ) -> dict[str, torch.Tensor]: + # all per-mode blocks share one code object; equal text/image lengths keep compiled shapes uniform + num_tokens = height * width + generator = self.generator + inputs = { + "hidden_states": randn_tensor((batch_size, num_tokens, 4), generator=generator, device=torch_device), + "encoder_hidden_states": randn_tensor( + (batch_size, num_tokens, 32), generator=generator, device=torch_device + ), + "timestep": torch.full((batch_size,), 0.5, device=torch_device), + "img_ids": torch.zeros(num_tokens, 4, device=torch_device), + "txt_ids": torch.zeros(num_tokens, 4, device=torch_device), + } + if num_cond_tokens is not None: + inputs["image_cond_hidden_states"] = randn_tensor( + (batch_size, num_cond_tokens, 4), generator=generator, device=torch_device + ) + inputs["image_cond_ids"] = torch.zeros(num_cond_tokens, 4, device=torch_device) + return inputs + + +class TestFlux3TransformerModel(Flux3TransformerTesterConfig, ModelTesterMixin): + @torch.no_grad() + def test_image_cond_inputs(self): + torch.manual_seed(0) + model = self.model_class(**self.get_init_dict()).to(torch_device).eval() + + # a distinct reference token count: the output is sliced back to the image tokens only + inputs = self.get_dummy_inputs(num_cond_tokens=10) + output = model(**inputs, return_dict=False)[0] + assert output.shape == inputs["hidden_states"].shape + + # the references change the prediction + text_only = model(**{k: v for k, v in inputs.items() if not k.startswith("image_cond")}, return_dict=False)[0] + assert not torch.allclose(output, text_only) + + +class TestFlux3TransformerMemory(Flux3TransformerTesterConfig, MemoryTesterMixin): + pass + + +class TestFlux3TransformerTorchCompile(Flux3TransformerTesterConfig, TorchCompileTesterMixin): + @property + def different_shapes_for_compilation(self): + return [(4, 4), (4, 8), (8, 8)] + + +class TestFlux3TransformerTraining(Flux3TransformerTesterConfig, TrainingTesterMixin): + pass + + +class TestFlux3TransformerAttention(Flux3TransformerTesterConfig, AttentionTesterMixin): + pass diff --git a/tests/modular_pipelines/flux3/__init__.py b/tests/modular_pipelines/flux3/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py new file mode 100644 index 000000000000..5b370e929314 --- /dev/null +++ b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py @@ -0,0 +1,269 @@ +# coding=utf-8 +# Copyright 2026 HuggingFace Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import numpy as np +import PIL.Image +import pytest +import torch + +from diffusers.modular_pipelines import Flux3AutoBlocks, Flux3ModularPipeline +from diffusers.modular_pipelines.flux3.encoders import Flux3TextEncoderStep +from diffusers.modular_pipelines.flux3.inputs import Flux3ImageCondInputStep +from diffusers.utils import logging + +from ...testing_utils import CaptureLogger, torch_device +from ..testing_utils import ( + BaseModularPipelineTesterConfig, + ModularGuiderTesterMixin, + ModularLoadingTesterMixin, + ModularMemoryTesterMixin, + ModularPipelineTesterMixin, + ModularWorkflowTesterMixin, +) + + +# TODO: local tiny repo built by integration-notes/make_tiny_flux3_repo.py — upload as +# hf-internal-testing/tiny-flux3-image-modular and switch before opening the PR +FLUX3_TINY_REPO = "/raid/yiyi/tiny-flux3-image-modular" + +FLUX3_WORKFLOWS = { + "text2image": [ + ("text_encoder", "Flux3TextEncoderStep"), + ("denoise.input", "Flux3TextInputStep"), + ("denoise.prepare_latents", "Flux3PrepareLatentsStep"), + ("denoise.set_timesteps", "Flux3SetTimestepsStep"), + ("denoise.prepare_rope_inputs", "Flux3TextRoPEInputsStep"), + ("denoise.denoise", "Flux3DenoiseStep"), + ("denoise.after_denoise", "Flux3UnpackLatentsStep"), + ("decode", "Flux3DecodeStep"), + ], + "image2image": [ + ("text_encoder", "Flux3TextEncoderStep"), + ("vae_encoder.preprocess", "Flux3ProcessImagesInputStep"), + ("vae_encoder.encode", "Flux3VaeEncoderStep"), + ("denoise.input", "Flux3TextInputStep"), + ("denoise.cond_input", "Flux3ImageCondInputStep"), + ("denoise.prepare_latents", "Flux3PrepareLatentsStep"), + ("denoise.prepare_image_cond_latents", "Flux3PrepareImageCondLatentsStep"), + ("denoise.set_timesteps", "Flux3SetTimestepsStep"), + ("denoise.prepare_rope_inputs", "Flux3TextRoPEInputsStep"), + ("denoise.denoise", "Flux3DenoiseStep"), + ("denoise.after_denoise", "Flux3UnpackLatentsStep"), + ("decode", "Flux3DecodeStep"), + ], +} + +# `expected_workflow_defaults`: the components, inputs and component configs each workflow's blocks carry +FLUX3_COMPONENTS = { + "text_encoder": "Qwen3VLForConditionalGeneration", + "tokenizer": "AutoProcessor", + "transformer": "Flux3Transformer2DModel", + "scheduler": "FlowMatchEulerDiscreteScheduler", + "guider": "ClassifierFreeGuidance", + "vae": "AutoencoderKLFlux2", + "image_processor": "Flux3ImageProcessor", +} +# the recommended sampling settings the blocks pin on the scheduler / guider they create +FLUX3_SAMPLING = {"scheduler": {"shift": 6.93}, "guider": {"guidance_scale": 3.5}} +# every optional input with its declared default. text2image declares the 1024 x 1024 size; image2image derives the +# size from the first reference image, so its branch declares none +FLUX3_INPUTS = { + "negative_prompt": None, + "max_sequence_length": 8192, + "text_encoder_out_layers": (4, 8, 12, 16, 20, 24, 28, 32), + "num_images_per_prompt": 1, + "latents": None, + "generator": None, + "num_inference_steps": 50, + "timesteps": None, + "sigmas": None, + "output_type": "pil", +} +FLUX3_TEXT2IMAGE_INPUTS = {**FLUX3_INPUTS, "height": 1024, "width": 1024} +FLUX3_IMAGE2IMAGE_INPUTS = {**FLUX3_INPUTS, "height": None, "width": None} +FLUX3_WORKFLOW_DEFAULTS = { + "text2image": { + "components": FLUX3_COMPONENTS, + "required_inputs": ["prompt"], + "inputs": FLUX3_TEXT2IMAGE_INPUTS, + "component_configs": FLUX3_SAMPLING, + }, + "image2image": { + "components": FLUX3_COMPONENTS, + "required_inputs": ["prompt", "image"], + "inputs": FLUX3_IMAGE2IMAGE_INPUTS, + "component_configs": FLUX3_SAMPLING, + }, +} + + +class Flux3ModularPipelineTesterConfig(BaseModularPipelineTesterConfig): + pipeline_class = Flux3ModularPipeline + pipeline_blocks_class = Flux3AutoBlocks + pretrained_model_name_or_path = FLUX3_TINY_REPO + params = frozenset(["prompt", "height", "width"]) + batch_params = frozenset() + expected_workflow_blocks = FLUX3_WORKFLOWS + expected_workflow_defaults = FLUX3_WORKFLOW_DEFAULTS + + @staticmethod + def get_reference_images(sizes=((64, 64),), seed=0): + rng = np.random.RandomState(seed) + return [PIL.Image.fromarray(np.uint8(rng.rand(height, width, 3) * 255)) for width, height in sizes] + + def get_dummy_inputs(self, seed=0): + return { + "prompt": "a tiny fox in the snow", + "generator": self.get_generator(seed), + "num_inference_steps": 2, + "height": 64, + "width": 64, + "max_sequence_length": 8, + "text_encoder_out_layers": (1,), + "output_type": "pt", + } + + +class TestFlux3ModularPipelineFast(Flux3ModularPipelineTesterConfig, ModularPipelineTesterMixin): + @pytest.mark.skip(reason="FLUX.3 takes one prompt per call; batched prompts are not supported.") + def test_inference_batch_consistent(self): + pass + + @pytest.mark.skip(reason="FLUX.3 takes one prompt per call; batched prompts are not supported.") + def test_inference_batch_single_identical(self): + pass + + def test_num_images_per_prompt(self): + # one prompt per call, several images from it + super().test_num_images_per_prompt(batch_sizes=[1]) + + def test_multiple_reference_images(self): + # references keep their own resolution; several of them go into one conditioning sequence + pipe = self.get_pipeline().to(torch_device) + inputs = {**self.get_dummy_inputs(), "num_inference_steps": 1} + references = self.get_reference_images(sizes=((64, 48), (32, 32))) + + images = pipe(**inputs, image=references, output="images") + assert images.shape == (1, 3, 64, 64) + + # only using the first reference changes the result + single = pipe(**inputs, image=references[:1], output="images") + assert not torch.allclose(images, single) + + def test_image_to_image_default_size(self): + # without a size, image-to-image generates at the default area (1024 x 1024 for the real model) at the first + # reference's aspect ratio, rounded to the token grid; a given side overrides its default. The default area is + # shrunk to 64 x 64 for the tiny model. + blocks = self.pipeline_blocks_class().get_workflow("image2image") + blocks.sub_blocks["denoise.cond_input"] = Flux3ImageCondInputStep(default_height=64, default_width=64) + pipe = blocks.init_pipeline(self.pretrained_model_name_or_path) + pipe.load_components(dtype=torch.float32) + pipe.to(torch_device) + inputs = {k: v for k, v in self.get_dummy_inputs().items() if k not in ("height", "width")} + inputs["num_inference_steps"] = 1 + inputs["image"] = self.get_reference_images(sizes=((64, 48),)) # 64 x 48 pixels, 4 pixels per token + + # 64 x 64 of area at the 4:3 aspect ratio: 72 x 56 on the 4-pixel grid + assert pipe(**inputs, output="images").shape[-2:] == (56, 72) + assert pipe(**inputs, height=32, width=32, output="images").shape[-2:] == (32, 32) + + def test_text_window_pads_to_multiple(self): + # the prompt is right-padded to the block's pad multiple (80 for the real model), capped at `max_sequence_length` + text_node = Flux3TextEncoderStep(pad_multiple=20).init_pipeline(self.pretrained_model_name_or_path) + text_node.load_components(dtype=torch.float32) + text_node.to(torch_device) + + short = text_node(prompt="a tiny fox", text_encoder_out_layers=(1,), output="prompt_embeds") + long = text_node(prompt="a tiny fox in the snow " * 10, text_encoder_out_layers=(1,), output="prompt_embeds") + assert short.shape[1] % 20 == 0 and long.shape[1] % 20 == 0 + assert long.shape[1] > short.shape[1] + # the cap wins over the pad multiple + capped = text_node( + prompt="a tiny fox", max_sequence_length=8, text_encoder_out_layers=(1,), output="prompt_embeds" + ) + assert capped.shape[1] == 8 + + # TODO: move to `ModularGuiderTesterMixin` -- guider-disabled negative-prompt handling is generic + def test_guider_disabled_skips_negative_prompt_with_warning(self): + pipe = self.get_pipeline() + inputs = {**self.get_dummy_inputs(), "negative_prompt": "blurry", "num_inference_steps": 1} + + assert pipe(**inputs, output="negative_prompt_embeds") is not None + + pipe.guider.disable() + logger = logging.get_logger("diffusers.modular_pipelines.flux3.encoders") + with CaptureLogger(logger) as cap: + negative_prompt_embeds = pipe(**inputs, output="negative_prompt_embeds") + assert negative_prompt_embeds is None + assert "guider is disabled" in cap.out + + # TODO: generic too (huggingface/diffusers#14469): a standalone text encoder node has no guider, so it + # encodes the negative prompt only when given one, and never a "" placeholder + def test_text_encoder_standalone_encodes_negative_prompt_only_when_given(self): + text_node = ( + self.pipeline_blocks_class().sub_blocks["text_encoder"].init_pipeline(self.pretrained_model_name_or_path) + ) + text_node.load_components(dtype=torch.float32) + text_node.to(torch_device) + assert "guider" not in text_node.component_names + + inputs = { + k: v + for k, v in self.get_dummy_inputs().items() + if k in ("prompt", "max_sequence_length", "text_encoder_out_layers") + } + assert text_node(**inputs, output="negative_prompt_embeds") is None + assert text_node(**inputs, negative_prompt="blurry", output="negative_prompt_embeds") is not None + + def test_text_input_requires_negative_prompt_embeds_with_cfg(self): + # node graph as in the docs: embeddings from a standalone text node (no negative prompt) fed to a + # generation node whose guider needs unconditional embeddings -> a clear error, not a crash in the transformer + blocks = self.pipeline_blocks_class() + text_node = blocks.sub_blocks.pop("text_encoder").init_pipeline(self.pretrained_model_name_or_path) + blocks.sub_blocks.pop("decode") + generate_node = blocks.init_pipeline(self.pretrained_model_name_or_path) + for node in (text_node, generate_node): + node.load_components(dtype=torch.float32) + node.to(torch_device) + + inputs = self.get_dummy_inputs() + text_inputs = {k: inputs[k] for k in ("prompt", "max_sequence_length", "text_encoder_out_layers")} + embeddings = text_node(**text_inputs, output=["prompt_embeds", "negative_prompt_embeds"]) + assert embeddings["negative_prompt_embeds"] is None + generate_inputs = {k: v for k, v in inputs.items() if k not in text_inputs} + with pytest.raises(ValueError, match="unconditional embeddings"): + generate_node(**embeddings, **generate_inputs, output="latents") + + # with the negative prompt encoded (an empty string for plain CFG) the graph runs + embeddings = text_node(**text_inputs, negative_prompt="", output=["prompt_embeds", "negative_prompt_embeds"]) + assert generate_node(**embeddings, **generate_inputs, output="latents") is not None + + +class TestFlux3ModularPipelineGuider(Flux3ModularPipelineTesterConfig, ModularGuiderTesterMixin): + def test_guider_cfg(self): + # the tiny random model's cond/uncond predictions differ only slightly + super().test_guider_cfg(expected_max_diff=1e-3) + + +class TestFlux3ModularPipelineLoading(Flux3ModularPipelineTesterConfig, ModularLoadingTesterMixin): + pass + + +class TestFlux3ModularPipelineMemory(Flux3ModularPipelineTesterConfig, ModularMemoryTesterMixin): + pass + + +class TestFlux3ModularPipelineWorkflow(Flux3ModularPipelineTesterConfig, ModularWorkflowTesterMixin): + pass From 41ff90d4e761e3b648e00a82423c3fb1445fc500 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 9 Oct 2026 11:00:42 +0200 Subject: [PATCH 06/34] integration-notes: ship the knight example image with the doc examples Co-Authored-By: Claude Fable 5.1 --- integration-notes/flux3_examples.py | 11 ++--------- integration-notes/knight.jpg | Bin 0 -> 127590 bytes 2 files changed, 2 insertions(+), 9 deletions(-) create mode 100644 integration-notes/knight.jpg diff --git a/integration-notes/flux3_examples.py b/integration-notes/flux3_examples.py index 61d6dc3c7669..e39e7d7222ba 100644 --- a/integration-notes/flux3_examples.py +++ b/integration-notes/flux3_examples.py @@ -4,16 +4,14 @@ Standalone: loads the modular pipeline from the Hub repo and writes four PNGs under `out/`. CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/flux3_examples.py - F3_REPO=/raid/yiyi/flux3-image-modular ... # a local repo instead of the Hub """ import os import torch -from huggingface_hub import hf_hub_download -from PIL import Image from diffusers import ModularPipeline +from diffusers.utils import load_image REPO = os.environ.get("F3_REPO", "diffusers-internal-dev/f3-small-img") @@ -60,12 +58,7 @@ # editing with layout: refer to the n-th reference image as ; each region names its source image # (`from`), where it is in that image (`src_bbox`) and where it should end up (`tgt_bbox`). Here the knight moves # up and to the left and everything else is pinned in place. -knight_path = ( - os.path.join(REPO, "examples", "knight.jpg") - if os.path.isdir(REPO) - else hf_hub_download(REPO, "examples/knight.jpg") -) -knight = Image.open(knight_path) +knight = load_image(os.path.join(os.path.dirname(__file__), "knight.jpg")) # BFL's docs image, 1360 x 768 prompt = ( "In , move the miniature grey amigurumi knight figure upwards and to the left along the " "yarn cliff . Keep the rest of the image completely unchanged, preserving the blurred yarn backdrop " diff --git a/integration-notes/knight.jpg b/integration-notes/knight.jpg new file mode 100644 index 0000000000000000000000000000000000000000..65436c1e8e3cfde30980007357f276755875a0a8 GIT binary patch literal 127590 zcmb4qby!s26ZZlOOGzx<3oISNf(X*x-6h@KAR!^TOS9Cj(%oGWO047xlEQ+DfXkN_ zK@=2x`MrO<|G)D*=dZbE<~--lo%qb$e{26X0O|lz5)u#zF)0WHA|oRur=-6}NkKu$ z0;Z#;=V0aH(y;#tAs{0oqokl@ zzITrq#?8(R`~RkYBLG@bA^@=o5djZ?kd}ammf#--zy<&i5)lv(0RA5l0SO3+NdTZb zrV0&!kdT0gh=`B~L$o z3gm&&yM!eD!vQG&$EPKt1tPw#&F9f_@2)`FpjlGfAks@{9r74LD+7B5{=DH!agb5r@Ez=y3y z7k+W?jWg+s_B{#E8}3d%?OZl>j3JdIvliC0AjhB{ww^=vWtPs3Jwd~C84u)_FxpgW zPYhw`y!@5S@wHerRYKqTs%p7(!LEpXqLDd6D+y^(Ih~_gFyVpu4z{--yLTVt7VDlI z2Ao76a#BP(Yk(xuVpRpAJ!s*kI6Jj0z7Cp;Gbzjr-d-Q>kwlZYO&Xz~uL8o9naSJP z&H>4M_PTg!L>F;1Zy?QaDZ)0RgF1p($geQI3`Td&Pl^K+;g{~I*!HK@q_7_(3`r7+ z@0Nxhu|T(DN$BX-($vnxwyo*bk-GUj4rqgENN5O0d1Tsse3tn(n^t0SU+N1R`4`~& zl!O?Fe*Rwfz-j`&(K5S!V3AM~lRsYD^&~&+7?#wbFrRy`vXApjTk=(QbY6EJX8InX zLC&Jq!ZV3oO_LwIg3&J#cp8814%9$IOSc<#~kO546d1yH^~8Q?`*uoB*wj{LyI#k%fB%F&LZ zXpk&Y*T=(~{am~qCqdc7PWJ6sVFzA5U1%=om46O&ksoh%q2)W)lPfv{WMTy6XbZS8MCyIsbf1NE_x+7sbscoD zgH#ZksrzW+@?SGWxisA+ABcBfBtNpF>~8Qe+{94UN|}4XUv|6J9@EFO5yr|TQU5Ib zg67sph7v(h49qIIRL)aKQaIxl^C)dBl)z1p8E>0G0~7F3tonJ5K&u(;D@E>|OVq9BP5 z*0-xI5;x842Dteul`D9*Mjk7h^TDIsTI|jmtzy)!3O}R_31^H5O=nndYKk-Jg13#< zc~06jL)d($j|F|t;XGSv6?k1;;c8~SQKVNQQp4mvQ{Y)`T0dU*o2=xs=w_Zb&@2)C zlWk#68XLZDNj&kYjY2VAXAnwMM>|0yDvN~v%o7$?rmZZVW|#sNl;Sz8!3QbZG@@n< z?#I_^1d0gA{hqI6WbzgA&8gG2J2F@(*?ZfaQ11_Lv}6NSe1`B zSV?WFqm?dLbz42MK(!B?crN;Bm+HGnn-9MpnJ7=nC&f4Fq%h}aVj%R8HwuT;VYb z1>|}TWtMweh@vEYBp#P&|m!<{h~#aE30Yoxiez*=x*#2tcmF*TG0vEjKQ;=ttb zAw{=D=LH4S*!@Mph`*g)Mct^H5C|en&>)$tKmc(xlgGpHTyzf=l^FP4*L>>v<85(j ziIq|fR|yiYw0WUzhjSV`%(mzuX9&h|m9&*oShQqlOn9V5n+_}c%oNj<#Y54KsW0-D z<$yP9xUdTIGW>>1*C9fHIptH0J0grBSf4=^Tf9yerK+^lh+MMi;Pm4k)zjuv$x^n@ zLGVs#K}j{q6ii>t7EcaiG+aeCM+30a0M~%ZD*R5zSaXdV+(d*vf z9rRq9PWoJ$LR#k87m+ul-I+hsIGs^p0^w&!Y8rS9_<1xtb4o!mT&GipJ0%4ATnS=F zJBW7-@0FmOv6L*C?Bz3cmU~8oz>~9)hbnm#=pviQtU1?X%rZ6!KI=5;xys_4sVHJQ zkdVxXri?che|1G{KoGb)b5Wg)%*6ozfxr`nu(jwcP5 zOqH}aW!9~8!G|SB!)%*YTS9$Em^2Acs-RD`*fxUA+vrdn zib?juuyo-`M?(aSoDGpP>$x0qJZ$W$%W_y5o+)FX%cd$VxYYo$EvX1$i%LxR{X7gX zLF_Oqm`^jMIHuQ{UPdz*!pFqV;fGZrkbACr@r)r1=U|-;)=S;#K36*)N$MmOp{9|q z*&NL!6h@740&PLw;m7pc$BgYE%mQ$b94+x)HQFo$iu4*pn|Q2pZ*^{jKy~?b%|{X= zpG77spdr&RDi`t_Cy&P=d!>Q=RoEo%1yJoO+m9F zG$kQ1cArPOVbVvh2&&B>i{`FGSuz)r1B-PkP4e!;Wgw~qUZ}2&3HII5TpoM(%T2Xh z1z>lJ`0Pc#M<$GD+vMX#2HX9veQ2|C-9?U>U=yY4_`oPFFhdutf&}K&N7KwiC(@+F z>XH#|v);p)x(-bplWR0MRf9Xj1W01RI-Bl+rMb{E7zpN(Jk^|Aa7NffIiy2L4Gt5g zrdv<5L2F52o!7={I9?~YiBFJa#C!SIu>VrE3VOT*V@uax4LGg)^#GaV zMF<}(J(BkkXY!>d*fKN>vSjG1v1JR{4&KG*4h46Y-=LQG)a7Kdr5??%JK~;qAG?3& zi%|44%d#kB;dBu?thxNSt}^K}_S7(9=n>=wWT|_zrRl_skSWPslVfJzovZdc5N6K# zq7HjhLhdI+Xk)vx{UrEX!K|pLi;Yg8d>jIB@^_M-C6~3O{9)f4NV51?XsEnVtFPo9 zO;o2BxjdCI_Ej+H=$v!0yV|1TRyIG(L_*FPD$zg_Fl&3ht#jW1Lit-@+Y;e6r!?k} z{Ao7;slpC-KLx=xw{>Rp_uvdZac9t`UY?U5m&_!mP&TA9{KcBFkBja()ZAZDcj^Uw zJa=PV>7v?);r>hlOpQ+>cL@FL#rd|yLK$t3QLI{8|M5dN%CA}q>ei2!*e>4WjSWR7 z8LCjKd%+)U>556|akR_cg)lY_3a}e+$4OML@O?Tqwyq;?pRluOLQ#>7fLhm1Md*eNah>UQVX<}q>krw0TIlF zHVm4GjEv`AQ9^F;mfPHyNEbKX=VeaAedc5TzRpm~B5s@^g_OeN2V^h*uuSrT!LFqgx}9t>^| z1mG2o?4*nL3CS2GQs6ZOFpsot+vq|VKz1bEo~`Q#s-%`a*+ZwOl#uM(mi}`Lf~~Uf zli*?)j9-pV~Y;2CRcS#_GPg0^29+@i-aQT7Fe zumrGoAK9J48-C`^V6&xJg5`Vm_l&XQHFA+cCtnQhLU<{Vn5QV6qj4!v4SyDpV+B@4 z3c21u$R{?Vl01+^bnAsKLMU$f*=AOhDio5lG!|2t0s|ECM3Vtsqv3`_dbv;vNYVt5P9bokCHf})VECwbzXy2 zo(YCQt+ZnVkThhLoIvqU)kHKEVB*+pC;_#j>X%O?{W&bQX)yL^9>x*{uAttoWM0xj zdy}7%eTrYhaT@IszH{GhyX>_;JnvBN%#jaJ;tpNN&B9rUinR7p8Utfq*%@UwTjJ)v zQ3uDAki4p0sZp^Z>Q>KcsC`WDJ|`EZYlpJm&vZKa-Z`^=bQJ12@eeRwP-kouGP13Q zo6|iOYWQeg)to#YLRaYUD$vc{7V$_5JQ*He;53 zoz59tjoo;fR);w&KHVGv$ToNIp7ucjhem0o?QH)5csD7%(jH&fN4Ip5Vna)jTC*`_ zTOrTNR~Az2bV(1zhoxfUSp@ADvdYNW!?5#zr6_LrJ7P}B79toE=(3_Qe&%(HtId~k z)(SgAze2oTZ@iz*YBOk(E+?GWW#6H-_qSxDyBc4M_k3!oGWn4O(S7JJuJ9{|(TQ$E z10s4xgTgfFl4yLF!g}Saq}4hpsFtS^As{w-&p<Dx^@T zq|s^Wqwx(;AB;hBzpc8hC zu*bpnaIU?a{63^qF7%NpAe5$JIdY8bg+#M6>YOK^Me^mxeIlDo$d(p)XLGN>kPcRj zG{xl}$)aW%uY~A2Z8}L9KqJTSF4nvB;*D!vDu4QAxmmW+MAQZHk`7l2*7Lp8Bg2C= zk9Y)6fW9YWa69LmRlO4|tv^}>3$H?Hsu|O*v}${QvUE%25sX;5$h5Ws(G+~l+U78W zJTTBtf*bbd2SVy@WEfDesU>dD5mO}X?QdWwrDd_h-7DxAo$j& z?{Ua8M=7^E(}|ZR1U_)w1zxK`Sdy7_T%assIKieY=W{Z50tIYE!Rw+=fLaIU>LJ=9 zg&A7DZhZnilRU~E7Gefitd>YqE=uoUr`vNNk>homX2dx0$T6P^q3$uXLN>>s5SHO= zbzj(t#(yo*PY(SEJS)ZMSsDvD&x7`#k^?KnV~Wxm3OoxuF%-60Kh~I{&lNaD1Gr#{ zLCw8c&rA6Qdv846q0i}gqmo0!(AEmNfbR(|oYdGH1SybJT~tj4sgWyihbicQH@x6! zu^D=3xb+t15AgcaHDdXqG)wwrh{w1L7YH+F>(jve@V~~&9v9O&zV2&^ zHsSn_$3JXC5R0F$eUCZoS(NxrKKfkztS3r2lZ{(gbkPsLE%@zfh;r5V;P$w%Qe`6U zUDlfpQ&0H>5)USG!xx3jf97mMu`f~Hg-!J~Hu+|=-+}Y-c4mwiHS0sp0}izTzaX`q z%_=B*Vq-PvfI%_R{y(2kTeP-v_1FAk+>@X$D?Li(N@dl6TbSY(@2ofPjcTg`(!05! z=h@wRjUShnKY0iA(e;*0^3gv?zD3{m=gf7?;@-`>>}T9w%tCLCO+ZvFu`6 zz)R@7jv~q3hfRzl15TTh7{g{_b(wX7Ws{o6o6`u@gzxO-t1BlFp^;aIu=C=2A*h?t zBY)gEM1aQEMKz7viT0T{pHnqtJKWh|CQnh=fIJff*GfT)Aow~+jQCi-cK*!%nJ&LO z-Eozk2kcVFTV@*4E$~$s4@aWb<-A@Q!wrV;; zF4)(aJE*9QCI!vs3SaB+WS0D64zEcEGRmcD!W!l9EU~1}q$C9+;F*N~VqK8b=t;gO z-aK4J>mFsDq{@e>zgi|f9wW(Gcn)LQHCv4?${BoYcyChQ-KBkd#*X<+DiKp;E(n4h zK&@b?)C~m@nANaNMw_>^jtLa$5jPF%)c3SNZ@JT^}FP03e2S_rq? zK8kq}#-zBNSL!jkO-VT>d%ha-f>WM zynP!`tRdP0xAgAa_wZMF1%ho8*h|brr#-fqy~CzcS*>AM4+1=?2fMzjslgl9QVdFC zJ#)USJq1d44C1m8=t69JAp7FBg1MWjOC|FzkYr@!p>m2fGsh+hsCN+`8t22nv+LG) zK+byP)Nc%{xMk9jdt;$2A+6)zQocj0PdTzzFFnHBJYv@K5XL5w+>=Coi_GSJCtV;P zSx#QRMNZr=>kQ5@wnsgs@C+DmY4KVW+{?!%dnz67cyGVze)q#w&el?dPbZ%0GYGR6 z1NC(=zW9uGqz6IrHa+bfmm3#Xqi&1e{Z!NyZnmOQFz6L&iDZ%;`KrAdeI*xgAn)>g zu=>Hv*&b>);bN)@DSKg`#f_C&0ydsBOY`6@_gpE_1K;pVYn?B)0$jRdJ9ROUvj@Z0 zpmYf(5CS#)`0R>(=TxMAQM|#w_no`x`VdQLuUXYX+c_1UjuZE}s=Utfahz7^ygIST5Mw6l{nXp9aUxk(Fgd~(wzp^vg?6!i2eVw)y@i`xS zx-TTQI{tQ$y!IJp&|;=U1t`^^Q#vl;H9tMK)AfES?ygi_BjjuEhod-GJ{? zD0KSoT&G=-zfat}D>dwgjtfvj39{*|mh+OhqJ||5XgT^x(0ZT6cUnR3+C4h?e&4<* zm|0+vR^(b+gT755iB<11mo$H;Tc9op+E~*0b*p%rbWIz@`?&g70~2G2K%@rgZMCqR zQZ(XZG_~alsq`Ft-*%3B(x;pw1>Ta**41q{x1bVPNMf1|OPxA~rFxAd#!@~HjUACv zsVT;*+~q~s&byWl52ptstwzq&LUjVr5(q0MO_SLtt|2J!L@r@Jh)RhYWK2D0LTTs< zw^yjZRg64Vph=|VA!2-yTgIGxST+`m23TsfWQu2!!_eh_;KheJEas#@TUHJOS9>t! zP?g4=3#y$@AvsPWw5JnLe?@lUT!O9lp9)-5m79_LppJDJ zIvP3pGwQdA}7L@^ymoRV`U7Oey6Xk^P9XKE%3zl>Tlo$7n*>klM;>ANvT%p zDJi5aJkHX{4+M4TI^w4&Wmxl=iX5B+c!zSU%VspbC$D!f=Hw;?sJzDX=DbH3E@pg) zt&I$q*=NBrOQfKx-Ov$cv1G#N7n;WyH4y`~EP+JK*|wH=%l#qEbQ-TdbV>^i0EaDz_fHSJWz$ z(w+8iIkPsg0Ho0J51PMPpH1X-(w4-?pK(8X=zw1{TVYq%jaTZer~W>!57$~4g7ou; zuUz>0cRxY{Y)8*<#Fz3&!y%AYM%}thlfDK#E)Uw@4!#sAJv2aPM*T>!v=X#VQuVM z?Tv3MApnGoooHyBof9=^xH@Bxk?H&zmaYjs(hU1mEHr?air#}hqS_82E z15l$w7A1Rrs638LS$lsa7cH5UwI@^vB)a0CDAIKlCNpEES|}+XGMZ82r*?QVm>nMeH(Mu{&^sHh)x%_xsyHG*F2 zJ1VSSMp$C12>k;#>CTwAI&ZXf>v1?=HX{rC(wx#48zXt4p83H?2Z4+Gg@1hbQrLI< zK~o&<9uJ;mYQ551w$xiblIaru^T{|lHtBC|d7H`VEu|908%M?av-2R>@`b8lv)zWZ zd7w&T8@B53VGGMfL6hBnZC8BkVLR)Dk=*GGQeOR}4t%HDYfEob#=PvQ!4{6{v+v)YW%PrL794Qph42`ENOWJ)UxRfv$T*NQBy zofs-5H_L{fQh#}eEM2fmd)H8Fgj;XYb?CDH%;^bFDf;`FM(DCjx>bb)1D;E>cg zsX0E~pt5g}3b*WF_UL1@<49Gg_6S8TX=@X5in){|mgqbiw(BcKxJtR)#==XkG~S)T zv)K(33+!bR#ZgI=(-mIF8Hy?Ew3*sJY&lVVJ=Pl8HfIdpQ$Jizx2-`AkG210+!Iiq zpMJi*;=%hQhD~o{EqU$6yR{WNY_nP&JRqX(apLa7t*4 z;gRWDXzX7-3*slS&6XVtE6Qo_hn%DKrn%HX&X!t5nUt-lDi%M07Q6& zC;YXIhhCO*q^G05nP0uHH?x%PsmIBP%age@R_RrJ?4XF+*ql?AN?@3)(|gVK+q7Nx zQzGd6AAlyRf{pxT-+c(L*FS)Hz(6W%;fRBvEZ?ht08=L42UfCH1c_IZ-aGFf;)Wt& zcB=-GePMyJq;Y@CK-;Fi9euVP(JbQ+vx}b&Ep!>(qO96^i$3+=8@a9YbTVy@4eOiG z*&M&oh^FhyZ?>)cMMxDr`2pS)x6KvP{4J!c9!uPcrI*}_CZhe2A~}*G#xt4TgCtQu zr>yH%^~b{sBau?2d|Wy1DY=kydd78mCFGVq0hGDbvMGZAg5ddFw#Yc}l`x)fAq4uP ze9}j>iY75c#OoW9H)K>27L)13uz`MQZ7xpP*bK3u_k7&DdpAeaFpBiHSm;qOxVA55 z{=vy#-e!vv3K_|w&1LL~mP>;k!+F+=y2AjQ2GxZr{LyH&4?-s{G@*PT$;LbgrG{ua z%A*b8Za{#6)p1m&SWL(kO>j$M*q8+n+r%$zH{gTatTyY|f;i1^OFet*V*{m>@*qq?Z^6f}*i;~r*`@4^Wv8#@I9P^lh zZW!vxG*(l#1&zm>dTe(;USM#;`YxfGN}WqdY)}&JQ$q$nGUlla9{5a+z}E@{*eCLx z^;S#zizc8<_YlEXiDD4jd%5eCbDPumfs_22$D8wk`SUR6oIQ9=Zk>Xr(~t>tRz&Y% z3=gS^hrV741fyUu{PZOixZ(t?-M~==QL)^H@h8`cxty|kw+v&2h4)Yjv&zwugufplmKB2pOz zHOcA3j#|Cu53%1IzdQfZd8G~c_2N@5t@SZhB0+wN|1%!YEGeO-Yr+jcR!JneRrJWr z2c(igp(ex-T&G5mduqj(9}T>Fm|R{JfB&s3;7BKftd#zeg;{)yvjWi?HA+5S%Rv^% zRnB~)Q3y}EOG%g?iO_p_J6q7G2Sd%PJ7**7Mc26jz>~?z+i(5dd<2F@{<7v-<9U}E z3j-4>LX`Tkk^cZpODL_z5*@7o&8$Bv{?^%-FH6W@et589^P&1{v)pDv-a{eYT3MWU z-la$DMg+!!OwM5|i7TF}uuwh_kXH%<<1#}UG*6y}4;>Tr7r!+-TsaU+h-%yePzL;# z{UJax{kS>W{$aRpK-E(1U%J`i(BrML?y14L&c#R=(;jq(clQRAmEatX`x?gP*S{K> zvE|i_#YXo0-yDKXmAS*hla}yIL0(@=9 z$m1Owt$oW~zDz5H-F|-{jvQ>yxZjmyjVz!0Z2Vx-BDa3Yy1_+IcPv}di$R2jG+r+{SSlElWhnjTQic3AQ7*j!IGLr{H~?_2KOQ_&)cTLsNx>*lNe* z&*6m&*MWPuL<%Zv-u8?GmLsh%^`=hF<(hW|aE&EV-rknMr|3R&*A){_wV1da#NMok z-^b^iV)q98Nc==R=K!FNoZH%xF&;wZuoRrZ--W0@auDjMyktn>sC-eL~hRU+d ztq>Rt`Vfr{{|~ovu*h)02A`%3>7^C18*OL~n4JUVk~e4-*8U7$b!$K3F>a_(j`1;L zb_P^&*_;{}VZ~-fBUg3Z);Jel>R0xDx9ZqX@>D&jEFT1SrjPJu+~zu3TQ!+{PB(XB z_Yx`dzIySpfA#!Y9BaCO`2LyF>c&y`^T_)Qz)k2N z%vRUq5naYt1vUiK6z<5xUY!ghC@OBn;Xsu@YpIJeU{MPS3<66Fw(j`oeTEq6H^-lU+ms1l@ zH~zejZCc#Fe6=a|w8)vQ(~Y8aL>aLyqvTJA!UtP4 zKUzN4Dt&He!6-hP^cVl+c9HKTvT}IS-EDrE4AFE`vuaz(c9CCZ;o@gvlAR+5w|4yK zODkOdBtMzV|68a$m^$N<1$Nw!QToaEC-6;gU$CWXFyPkfm?x{sP0pgDxsJtip5l7j zZQuLAx3plt;Hb@IVns=I+k5|FAZW~{cGr3GbJUjCsZ@=-R7{Pcp}BP-(V7nDQC!?y zkjbh;TO+G;-=po`_I8e`zg4eKA3D`q{E27kA`&VH*wFq5K%)aVUk#_*ABePs9N?HG z<`b3o_C;j)mbh$NmK~Qr>Q^!fRRDf62Q=@TbH>JCYU|9aS1p7V6n)>H^r+0deylR;EvPXqKZQpDkb%BC}uwLU0x+xdpEnwNdk(yk)&q*m>+iCvH2h6)j+T#7}83F7lXm@tQUBy=5TPR<-OD z=#VA7P-n?olsv;cR^m)1e89OTH8JQrFi2Cgw!E4r1PLPp{Uycir)HcuSbK4Ps_EmZ zyWKNzFA9{S8JPfJxEtQNt*Px6@1$@wzC60uThKDguC$XoPqgJJ`^e2Em9}$6fiN^K zd8scybV>MGJ@13&kmGY*W@)aigYfn{v$39A z?j2>@7}Z*yMjP_R&UzEI#uL88zw7@1K6>4ekN;RR6Nx4OPX7V61KSKy?cYHqQcm?HD;4EG;Q zd#8WvbZhmx>4`ji1J0Q~MyxDZ_z$N>>~O;#KKw&AD;|>`eYWx6*uJ#lL!;%d?7v(m z<i#zm* z^dG@Fu#;A6(^2Q1x}`Id>DEo1NZ9N~^iIazK)`H6MEA=Bk>g;VVQr_{%`bzL)Ms6V zL6^5Vk6Y{ph84y9ey5Cn-cYMgJ8W85QZK(BmXsW-mU6-Q)Mk3MV7MX3LNh|eqJiaC zNXpNw{efQe2zB5FC=^7O*HhSJ8gE$LYH5A=nJcJW0;d8(f|O?-J^lngD1B9v{O-wj zEoxpap!(W?qapu+*1|V!X62YEpNIytt>vg=-0#4h-8h!I#U83YR@RTgq>kH$^xk1sI;2nqE5An28muZT67Oxngu49I^YEX#8LDO_Rq)}; zBF&;>1zB*0bIyC<@!eFl_&myNkP~81JWGs)Yv^FymXyKq%GKcjQ|u!lnAO{ASRD~O zwTU+Mn52sN8CWM_L0=v<+if;6Em6+{xjZL>l6g1u?ei{4jE0fcQzKenj(!sL{ckLgBD3a1N9rx!5wVkvj~>d5W+=0xouHA>-e_l z9-5Cj#zjPY2=yOS^vuRYAWd7}Iq9BVF@jAlwM8d+z0u-~ndZS|YMSMs=Fu@Fo*+X= zbRS#a%8uZ6&q>cvOM+1G4Qd3OEN#3l;HOp?TlV3?xOSilw8c1EHB4j+_#H|i4pYLX zOi{>V#*VLc`U(_7P0FjpucRI*9AnEBrcWV4xi->*36KuOVx4<$qS;6Ajs})eHe2eA zBP(;f`Z%BbfKm)UR8t5cYB$;k;T*Y}zuKlAB-jvmeFRB@`2`n>Z?TZECH>CeNBgs- zM9Bhar?3A~H@oTA#D22i%T{6LfSWwsgWGMsijfxcdsF%Lvp9g9iaz7&Mj@Vm;zjkB zj*^>43}?@P#W(oEbR+43fQ;&$+=?r>M*v(=qdV=WM@s>-j(t{1X~#lprAi8J+mVl)*72>gC7bPm zGFfYQ&}|y=x3>1Q6@)Me5R>)Ufc(+pejh{9*@JdDobvriE;{SVOwR%lKlRvAGmEFx zpM6F4Ddf!(H>~IrWB$0to-yq^P%LwI2d?K#a@E~6_PQ5+Na~ClQi{C}nK`5h3)fyx zixMyc|K|OC*07%YG^)3mu0MkSF48046jg7dvIlS`+I#EsR!lBBKxV&Xpmjfo%5;#c zmdAK8m!Zj_LxFCK8Zg;?T@&di*5pPtMHP0LDP?79L+lz`8Vp*VolBp&MarCVym6!OJl;AsNbmy57NNf{YL(E*nyJdU zk*+MZ9Xu28-rDAlf0Qn&l-0yN{H|-~<`(&?$I9gAPT)^+?E5dADrb_t)$3c!PpDmC zR!70>?C{jW?eYH(=wih`L~`=Huzj3*q#r;zzA3vESDf_^a6UBMWgXbscZZ2LIet}> z309p}yj9}%i{7b-crv*7#rRCjW8PCua3fI)8FsA|ArrWM@SCqPh8zHOXgbRn}qD;4P?p5xZ*-)y@R3m$GgeD+hVQD zY5PKsh6x00ImOu@!ibwz9P7LHFV@V*)^>JIsVmy_YvW2*H)jXMqfIY!P6!;W9jX`1 zg@xE*M8$vOJyR_dwO3Z&jDDqHzwPk*2k>rMwCUmh*iIN(fAbV$*i(2Y+&Z!3x*?Ug za~qw}H1Q?i;mXDFqa;^ZvXlFzBv0!kBUbk9R>PYDjmd3DltZ>d2W|_R{ciLlNQH5s z@ZhbRw|g|#oV7U3Cx$9r01otP-kX{)xc=KmuLi8rEh-L|S;xgmLyZ#T;Qs*EL3*n9 zLjYS0ZTF|nf5A(;MHlH8t_MfQmZ_|{U;~_Wk!kPZ9c@4BC!CC^ax!n-qOJ|Yp$*I7 zLE{ehS+yfI{fV;n<`lSAjgkk$|EnMDZvr~U%Irod$;U`lv^i>3wpl6~09X>Jx#(~Q z;UD*dTVYRAN#$VsWk3G7+SaWwiMYP`X3(UV;>PZtah8YC3}<~6WkPE$5Jr2YGNRyi zm}8u<@a{t|av@Ok&EmJN`ff(fu#u5Jnh}m_C5ojbyXG!{7hfCMds1b@ zwWzwTItkikzd~<3*{JVqvyGx}{q?Q4i_rQQ_8^|GbVIE5o0d>gV(8l-&NzeWgQ5p+ zAH|(Gm+Ufb-bhkL6q)Uvr-7n82pqVQ&M$J#?l0$(i&-gFiB5DT20DaX6V0=adWRcb zziM1kt_!oWkQaaGl^vA2_YZ*my!f)Geg+Pnm{^@VQBpb&(_RX<^Z1$U-%gZtYrJZo zo00b4^@E10@)}HMBI%^u?Uxy!l|{dWRYmPDe!E<}M@o>0PrBh-HCDt>uR~Z*yt+v} zyX?i3^5vDs^>*mcz_8@kKJDrypQC^oNwc!ZogB&UydL`+E;<_i!K%k{0ZLVt&Zhf4 zc9Fkae(&D6zC7s_8nS34#U# z!OIAip0Lz8JTvHsSM-{#Ii5d#=|HEEg-LO}{|}}11Kfb=jHEbLa{5SVN#1^WIdi$I zlJRd(y90XOsjc(9`g5Y_&>GcG*Z8~X%(+hCjr}*v8haUB<=64Gd?hM}(qE$hqH=s$ zFi)yB2NTxAUmd&!r+>?XRv)eW159tCAMd36%3o6qG7ETwTj5xDMDt$1v%-`5#e(Vz ziPydxU;Qz_F|_sAHuH3bMmD%=PyfCRoqucHVB$xn_s3Lqml_$DhYwWk>nY@~ys)#6 zRBXyV@Oa+#eTDooIV^E`tX-C4KXd^4WVa^vOBDk@O?m`scSA8{KFUVp zR~YspIR%tHh2YvQQt34zRqSp|Q14^+6cQ;>pe>CtV-DY#ySvH|l?$VZ5C~-geWp&8 zLoWA1CrN{+6oTpf-|t{y!~;hE1&BIlbikfXRT?y3y0yPh&=ghNk&!s2?<65~PZ5rt zCj!EHc`a=-H1f|W?4752Xn#MqBJo=NoynV_>+k6>GVDCw0Mx}%s>}UXK>wbkNwJ39 z`4UOeJ@VZ4$kpNvv`B%J$xZYzS(bBRsPj4Lcj)##^7fx?k{Utbp{*g@S{5KoUoEtu z5UL?aL*^{dr%NKlTq*!w%-YlyJjU2&_m9bKy<2~sY!9ZY#u=P)giVOlG7xr1^k|Ke4Iu&!Tt`N_`Lm(YcC6*&F-i~;hg&p%qj)0 z0sh06e8XGbNow*s*KRx&PHHUpZ*JVjFV#bDIMUBkJM9WayIt zXF8BZko78GRSA!3sW1^VGO#p+s~LMu?R^tpMNE&PV5RM+^`mScnMfN9yXT|u?ha{H z6M*$U+klF8=UYCb5N{x5d_+7miki*P{A_1$=BO-@a!%%i2iwaT1jv)tXw#+!@{q94 z(+acR!H|Z6Asy+QAO*Gd{EbBBt>lpGTxgh1L4TCsY?p5r(V+LW*X!-@;fDs@`)}#G z!XA>7eV6B|#!F94Tn|eSN^U*hqCGFW!hT-%Ege9n{ivfjRr2E0y9jZ6vwox9^xz@A zz%p-{SIPw{UaH>r$;6xYu%0HnGC#WK=fzv!LDBa#1sZ;u5qPxN|7w+YJAGvIVqso? z;Xt%IaV|b3dv>P}yM6hSzsj;@=+XYg&p#3@CqC4%^5l&w797=O=!0LjT02p-uSYLF zo|k1W@--e%R4g$rm0LjzH)AHTHcRFac281lGk4y|VkcH?uJIvuMJ@gJoWB19AfCN! zv9?(J~e083S!xV6@ukm9bXA- z^mC{i<+rtj)>`!4o8lqxUp#2Gav2L71m)fuyh*S0xv(_5?p61t?3fI7+mA$`Dg(DA zGjHYkmVNT7_dZL%eEYb($s>dLJZWrabO%w_xDKD9u*V#QP zCE!s_Qd7H7pg#9)&f?xPHAwBql!L*%!#9Z_`0?)Rj899*b(}I}pb`jo4>D z*~_?5J!N^@u+OB|6u6;#6q0vsaB$t5+7zp`)6v@#_-5yJTl=m#a?s$Y>OYhfUqga% zRq*QNNon&AT>j)}w{p*vXJ)>eJ?`P{HqjMAj(l0Fy4EQN_`Sj+|X0UoBH2U|b zvdsS9K&{4!ajK;`OsfR3eR-%db{J+Bf~ zT-M^F7X}dvU6d>FE8RLQfFd;ZE>C%v9|zNtbaC%XsF6#ryhzZTfD_!1sL zG0I!Kr}#$M3FcE*^Vg77Um9o0=(wLeSA}E#^^3V=Sj5y&4_6X)ddup=)_;w%SN8E) zIsjLS*!AXSPzO!$I~{}UAFQgi6fimk;wQEeK-sHi9BWSYD~Ah@tzKD8@_I-9{o(@r z(!+ERXlFIS(8h=Zxvb44d@hOad(hP+^?61+C~Gl z&0KnCC!olqn~XSrubD@jWp#h;QX(r3hlFM`7^kl}m$Z#59;1;JEe=s3-ER0W*14jV zK7-!Mm z6ginZI=*oiEi*9GT=B8kU*pfhjiq)O{rSc9k-co6TC2<6@#n_xbmkqXJfRJjLUjH| z)9UsAOlVSfH|o2OroTt?5?%bkipivWJP60(->oFq1N3%ErLs1&Q$*4x1&mfL zo2CZd8nUIf2UCCKf@yNotbVga9c#^#Cp}44T>oP1& zPMzOVTesoQE?8p+e^Af(bma*NpZX~CWx&Ht& zosu)#bE|bqD^{+9xBx*U@<{GyD^9Ue_!*s718LM zsJKS66#~xocK-mLW^^rVzN|fF%|g|K`Et0x{!=c_v~)G>rre5f4~Vx>^z>E+ zrJC?ZCd}j?v6xl0cQM=*`N>s$mTr7aOh(%po~X}qD=4gOCyv0#PNo#$h`6x)*x-Gp zK+)XIxst~Zk%7lO@f>|8q-bZsEZbR!Y)az~k>u2yEn6$0+iBd$8#xi@>rPHna^!n~ z;Dh8r3Ozc)tMEV|sNno!3fmUCNv41Z$la0zJRtHprt3oMQ)#ZO4h@GV@6Tw?s_QiX zmb(ZBLkGw+hSchdZefkga1Z)KEp_T`Rw&1S00Ycs18$qvTWcCx{p?#VJGOaq|?e(v|E3^^$6%>8aD~Rlr{44h&UU zDzkf$j^jLLFJ_@?rB#k!k`Q-*0reZyF&S=1`1{QT^>>?^!k}($Nik}w^xr#6F~=G0 z^^K;1wvm?al218}>3U6$-qD3WnTV8@uAyqFHraqTGY}_Pq^#*^6@$snVwxVDbo~DS zANq)oGvtmtPfr&fyRKR1^#l*+YVP^Oj2R0riQUDNsq~i#@gzo3Q7hbJ3;Jw48DO{& zmRxK-uqEhEQL&C7l=RNrnPpgOD%r=007W52ApnpDC4iD$dr2|`lO&AKW=3RY1!B^L zr5U9QLugIY+vO*u5XX}jv$=+20R7B}XS|TJz(T6aKs;huc!VkRmD`CC708U%{CA0n zRBUkq&`01ys{;_O$aS(iRMtcr9%VQEWEbXJ4>qvW<;C75j6U52w1m_Z|CFPZ(#if;{uxob^Zrn=llw&QBtZwim z!apWdE?|`%;S^5h-%-65-}`?m?PD<5Sfcsc!mctNj%EpNHTGmDj9=w?Ti(GuyG$it%utc zep(*Va~3`1p2WJ0%4S_;gbOYg={$olp<*6I`a=OBM80LTkR*X4GC_fOj7phe&}6|O zgcA%x9z>&YB&H;0L1{*DBC|9LH1j+eaRl*c;VGQUtdXiuX!U2b&Z)3OaUX#ZOfljw zVTIk9Mj47HV9sDl#%x|-iok`oTqLkxq_`4D3kVi@M7R+ESe!*H7@91(irZ1MuNCXCe=A#4Rdv*}z~jRlX%u zhIEy_RyfaGl{+V4imjIemELX53O6s+t+1J!kTwzqe>w2YI4w|~&z!z&+4}Y9Y!F5!z zHKSv&-g0v}+V7^;w^LSwLU#Uj8RfXo(qM{p9dA!<#^GAz<=>Fs?JB69LtLuN%(ad{ z?96_C(whFUUXOO0N_QfFFtTe}YSbv{+C@sM-p2!R?SbBA^`^g5c?R8-fHPzAp<-2a zTdp8s6KHf{b>q1N>p^g3@x zNt`E;GJjc<)w<<+8kUOCZ*Bsw6F%;>rs;GCxx?yQ?&S8C{-W0Vp=))WMBD3X+*YTR zS0oelnUsFi{u^6i0hD8o=QA^?b(?y=pe{otf6^@9E2W zDpW`$ZakmTX6Sn6qMOZDIR#D*PGeVffcjDCC>5`@c<(V&Osd|x1!l=CtZ|s=ZPe;? z?W=Qo2dT1!pcs5soAcd8Ob2Pf$>^uM%v)wZgV%76oI+<2P0UbUs_e#=Lv zVdHQ(=1lg<)Mz?|JH0DKx=@)cxKcConQMi5UXnd$#jWnGvRjOHKSPMWQ0f|uKB20# zuuB}EXC1NQiI>+k<$Lpa6YC#r`FP{ zQtIs|2XhbM2lz&34S!R#yY33_xG7M$1HUt*>H6EO)2j_a+>6Eu%OLwlOKyXv>r+jB zqOCjS1$&&g0Ixie&p9%-b&X5Cj7^nQVtPwWe$sLMPl?Qb)hV^^8k*QRBWn)a9D>~B z<2|O9m#OD9tvj+Yk_S2C1B{R73l`cQyHORKT$)ly~vqXE7<wP^`{sLF(lm5w?bB&Vdv_>j&Tu)&jDp(Y#qUcJi)*tS8xOdQ`H$)JTK#>s zsdhEv+n(o>9cIcYR8`RRE*rS|}^D^xHqt_`mdg{YFLXL6YkNwR)oBKDrU) z{u@=1&Hz1$j~xi8*lDlRVtwkaN|E4!2Wd`q4Le;~)X>dt;gE+T`9So$mbGBbV`PHc zn+GG~jwLA9GCEQc#V9{JuX%fF2U0BZ{B0jUNHp9_sRTW00qbd&?K`y! zx1lfUD`y{Gaq;42(|c*i2-X`2Ck@6vpEEnCYHYU8LYMT}LCUDk?2H~e<}9Dpbt@Ne zXInPsBh$wxo_Oz`a~RY060{X6pkq1QIsX8jCKpZAEh^f3Y8M*Z9IzlSy9Yl^{my@S;HjaOFa!oa15)M7~k=jq3ppQmrdI-q@u z4p(z_A9yaMU7ng1P0pZ<1Hf;u_nF#mt6^tipp{}7fCOjeV=On^bGJ^Gy-=+cSRJ78 zp2c|vTS2ciXm72^49$U)f&ddLt!;Jy%79?!8OSH;0=E)LBOqmn&-Iw`X4AUeOZ5je zsQ}n82#pJNN`)YzkMY>+u8;!ji6RlcVOaYK|JTw*M@ zYP9M#OhN#ZE))*L`OMDhn&qvxRB#xaj{yB;=+&B5q+iH?H`M)RHl6m$qYObjrWH&EaV#1H#_0_F!ETf-%(hl{fh|1G23iSy(&LyA(8==< zx~@!Oy1Q{JQsy*^j7?}IYryRFsNeYU0WD|G7R=J zD5o%jDMFc&5PVDGND>N#o-)UXCu~V1w5c+Mqyq|8l`_%{P!4;+XP8GE$^ir-^N7kX z5EZXz$_xmF@qs)$I!VnmWk{~T!BOTz} zaAFzuuNjIz20v=|h}bY>g3mCk%orTQRymR}TE-P{Agdjr#{wZ8rAd;;1)64&1M@5! zF>UfipIIFkEhtV*9`bz4E=iJH30GfiC|Oq?*$tF#Ae_TI1IUEUDVZ(` zy`w4>TLv>*c8P-xTk#ZiPZH27(&3^I5rrZHE~l8j>`RB1 z(Ea5Zawmb89EgJLUVKOuJnABp7m{4{tCb$p#nICHJ^j9e_QIx2Tp_ zTE^O0z06lQ{RB3Y4#&yujwAXlCZ^rDBIdo>1)Bkk1vD$%Y$Q4;EsSgp@%9r^uTv@k zO~paLZWMSJg}pb7?WJm4R*WMk4tt>ZnD*+ar$cV6G4(1er;fz<{!y)c=B33TltOZT zOyzo|?Ajqh+E&M6GRkn@6U&!%@4{*_pQ1yq$)Z^5r)E3`nz&?}Ot!+Iw zOTM+Q)rDaxh$=%~U=Oa(-eVT1WciDAri{JtW^6+M@`lF-D zEyUPLuYLs8*U+23%Tx+vpF_A(Mh}zQGeY*OPEaQ=pP9of zbzShwanQYPgb=}l@~k_FwbMTbye;76eage;VbEALVwgrhCqI(M@(UYTm}ZhXZK<{;`YoXKq>BYHkaT`S>s=sW#6^ zn!Ew+ED0dZzzOKoU5!FH40{IcpRdH(>AfzkjJ7Ior(*o*>ay;Yb>nZy@Gj(>h=NZAHOQr#x~SRiR@!jgyA$#M0Jw$yPPFu^tP2A% z3{N;8@XT8J#k-H;$XdwS%KUBm5h-q`(_3t|kWe<~LU=jjgW_5{bvn+YOlB%+kvzk z9mBRW_lUln-|aU})l;des|y5XEl?*pQ`r4H<_AwxQB!JIF4A~hjNo}=@isqcwPuRJ zbq>o`0RXnZIUM;pGn4#xukCQbLS*D)kAaUfwjQ27PODp2)RWR0hSOFb}^DeG4~%4TD5EkqDeu6v4F_O_Y(BnwpUda+M=aq9H<~<{Nis;dv5eianyAx z>(pMm4Co$w)3oYPV=Mz>#(d5S%YPB54y#%m;XoM9IL;?~(d@HX(b<~?+N>)B zoDngUGO_xy?XW!ssAH@Kvih+n(d+F?uz@>kQad_{fzydvDae z_2VT+Bur`9t!-A8Yq7NL1UCmK;}H#ANo8KLp{rml*!yId%@=Ikcb{Pkkt1CH0Ek`X z#t#g~Bz^eiFl5NDP_>r2+nLL-aC!004t$xjps9ZVE3)=3;E=7@H*=9W%~2QXN-Lgl zH)Dc(WKB&C?2NUsR$=)^#At-?>L#sVsno4$`?crWj{JW)rA^H09JX=5Zzt$t`lPc^ z1F9(6r5ipt#LUy{1$2trmKX$qf%k==q>5@BRqWL6Dw0i+FO8q+ZTl- zag68u&3!LY9+1U^0>0%u{{Z-t)NZd+r}iicqX&)%oUgVk8ogc_KqXWR6$gPF)oI$* zrG;+l3CnHqh`&tIR$SQpuJ2;!AWnwlCY2@AU>t2+lQVRf&#G!LmO8P{LB}H>;u}WT z8}a#uaCpGX4^kC+Wzw@_dBGzwO7#TSw4JyZ1HM0$wbs+8{AFY}r&$W1?ZEm>twyHX z9Pd{EdF=X9Q$m zcgJxYqf4crS+<8v_Bs5&dD3xO&CW)4lcj~EhQMr&F_Sc@(u#_!X_t0#a0uh$h|YJ6 za0i{d6ZbHo_HEh+2PC&7pQ)X(!bs04mDP@dR;9Y(E4*srwxXPUJt zPpm&l(qPY>QvCwGg_V>+wuqqwFp}5Om{WK4giyC*ZKEG~UO2qzVWTZ?BtjmXr1t_6 z{T}P4da%R=Goc!0dbvEn*`{j8BQrM$*PEX- zIsOy;pv@L%@|n;o7S23Ml|V6pn0!ahFxHF$Y#qBq*(xr0ojYkLx}pL1{6ycQcVI;4 zmKHTLj2jnkA21BL#N0h8*TG>tdVO_}Sq^bC<682J9fn}1NLY{=AS@h58DjPhAc>yxLrG!Gs0L)GbfY&i z}zlVzJr*1U-HvjZ|?NQ^ZvcAvfYE#_3XExBQtqhyc98 zppr7L3a=Jk@POhXnhlu``CyggvkXK5Z)lx^lN_|>D*eomAz>78JOr>2L!sgn9L3kP zDxM^g#s#~>k(pANf+|dm!!ms(0(0600>t;|x6 za!d;eCG#UWG9j|TmBe6(GO_^(MUjbMBvK)(Uebt}Jj1BKlC(iBnORmbOYV`FBJwxG zXhvuzG7G_$%o2BVBs>{1yQGaRpICHoL_u4aba^A7gDnUd7-FD!0&>km*{C+3ZAw7ro_XMW&bzOIjzI;hb>Icb81p$Nvi!YO zA^_gy;~|Xm-*0vvl+E1Wgehy>RXeKf_!}@j-`ueoS*xy zNPWEE{C`s!qd}(AXv4A~34j>+lS#Jqs)~V1_@ikJ+wTR?HEVFS-2hTO&GU~F7pUtF zrvCssbntfvE`fe*_2%`rhzt&0B>w1Yz5ASTP6#oF*MdadgUai;b(z97K z?#bsQ%)5Gyqj#uXQCRx}{XF2nXu687t?kraSbkl@uqVXUhT9*s0dJ{2B0ydUZxfAu zHJ4wp&Pe`~cdKfDB7j35KO$K)si@QbAmB0WpEG#$=bwEtm!wr~AyZ*&O=f>AL5i%sa|gQq`%kgJcur_LzVgj3uT#z({gSz|d=vb9oh5V!aEk9yZz)T>5PJ2}r|#5w5| zChI~BKYR%)MS7d1k3%lag9MSoe~w{v`me`WNnMCI!kiDJ9=k>{Wt9-bCJk-mk25Nr zW%b;YvhAxlR@<}_Ei-ccYHM^Fm(BRA77WWup z+FV+MHqxYmd=VmrTN5VDtIh@q$@@S#SHc3>9m{x^3vS=bkUaREo?{JVWM^SGJ%QqO zpR%{T9X(U(PCoOBsM&Smpv2p6wszVIjWnrRN|jBg9%nrLHH;gmDvE96+HzWQl{2C- zTFV~Wf$k?k)Ld1hHI`g4AbY&ddr@NP>O4SxUfQ4XCo!s%ym_1#?(mMeGh+aHO z=@?s9^1(qPl45$Ss11_2CkM~oQ`)|n+P8GK6*hN}eWQt?Q!OP(W!sRXj&Ywc7aQGj z%1Ph_F^U`wE2vCkoS&rK^u`yqoz}9@+3tst;DS5DX*4w5R}~UU{$ROSj;mLV6?{`*O-^=Aw${lUC6{!gNkPrEWvsNyz zaeRUDdqmOf4$WBujz%XfCeD*bO^91jkFM_#ze`CF9OoYqjb-AEHMa}wc69}C zz1z>)Zew6~8f$c>q@-O-F~Py)!o3Ftt3$Ww4kyhA?G=<#IUccaKt;d<$elM${iidHCs}Ov=W&+w=?!n`sVYGE#IDBMyJ^SFwOWL~PObwnKahstev-wmTmtN) z98JpSmO2cBZl8tCW09>&0(b%vySKGT`$IKBj*Ii?|EZ@oqn8t39 z#tfNeFl59C47e~V+{P=dtT-Myl4$8QLX_UX?D3dy`p$;bY^N*Z5e3xKt3ZcL=m+r@ zz!Ht3&>_mOQbtJ+ne?>sZCiSmPW9P?lr82jtfN(^r*@9^u}^FmL(uwmsnX)l$Bg%v z{{X7dp)FN)>?kszj`3!)6Q%VVY{ksu=qA>mN@%GW?sK=ej*r?3j+t)!Kc~2I&D$^u z)U;Z&U}hVL@gm#P>Sze7>Wq=(#dVtI(^?k00SDq^UDLl6Ug~u_bK95)>Ag!$R$cvX zw{Z;}*GpEvQ|WH?0s3Z9`r?-&?4cd!Ds9!%U%t(x9z#cJ>O)ae2d=vvM{(v}FmAdA z{iD;0{iq1}Gu8E7J*}4;bLMlpjcqTdQD%^V+c=8Xr+YN!{WbiL;V{%&uF;;F(6TqT zzC<75s@b`0IGEiqYVNCcp_JF;2*uP?8+A;1INP2ka*anpMy{_!IqZ8u*<#qteK0dI zHJY_PooGUY1Np<~`kK_E)gX@?%=mA%rdsIwP?oV689lgyZkB3o!E=uie-T(;TBDKh zBKnT4btoHd;C?1NHhkB8Fge@pKNySt9ZYVHsYU1uE*SUvMjE>9P1W#ba^h-c$!dxp zq>M+3cLie3fHxD3$4-)hL<`niu#RCBd5gV(1A#I~*qcra7CDEolG%dCGbI>WVV$ucgh|mcR-99M))Ap#LHw|)7!HHx>z=4 zuqHDjWR)<0W+JE-OtvSnB(7lDp)mtt#0rdgl_pfQsd*=xnJXY1%Lx%w@F78hQQ8?q zK#C>6#HvdZh=-uelG3}Ow5b`M2`4h^Bp%7_klSamBoIkpF)uxdC2=eyfjuPgIFQdO z#tGUQh(cpS?Eh&&P8OJ*s2%9-rudLbB=QOvGol=CDq4n!~z zDbI*dWRiG+bAb$1nMqa19pdkerzZyxssvtpbP+N^ffF*M$(3U*Boiw=JOq-<-W5-2 zMOals#w3JO%war0q8#!e&`Aoi##B#cXgo>S>M_rJ{6(>m3gl$xvEoL+LHCcK0Gz<9 zSqq$uXgw8esE})4b7W-L*!%eZ z0C}3;KF8|%p;H2I`;QyHgpoPjeV>LR+SDDVHMt%Bgm{~_fvwV-SuH}KirH{@`NYf9 z8Y))xPqML=XYRy*&zZw4bza^)rJXsWP3y03Qlx5%^R)1}`9I1Ns^0$qgiSqpW*h|` zVS2sQYV5P{kM1LrUt$l^VpuF%Yp-*$MI-_|^Yw$H9JAZ52vxJ6chYI=39ClBa#$(k zZh`6Uwd-G1U?srqpAb6h)LwmJPT}ohd(A;kXnJiIRIRD762RG9Bx<#) znaL%iF|6(zr%KmqfVKB3jocZ%dR3+fNytzR3G*&I+!Dr3$I zJ6W^d{3soS%GL_^*x}fm<8CuFwC-)MD1?unqzBMH;&Fy6;ybbLCr{t`Edi_ULtKseLmp z!0Ol`LGAX4)2#?S8&QyCU}G?vUaC@}rcyd3np3>~pE*Swo!oZG1>odA}^W*QL zJu9cJdn~(#Za~E7j-Z4a4hdgy_%Q)RdxER7HajP7n2{n<`9QRn2q*!JnH^fTN72{S`QTJYRt=brO6IzyYbrD~Z}$G0Ca z7PWi5<8UB?qEKYGO}QZUhe}zj&kOhS2Crqc8Y-Q4RgiY%XOqu}m1M1M+s;&t!x;Jh z0BOF=v?r+JetW($7ykeeGkT3l>9w`iicbWL#$B{2kG7i(6>b5#T%+8 zkUU0uC7!mb)Cxsy$v#x}{!?44Y3rv=UKiMG!ynVX^p)$Qzp(1nt5c840LVD+EomBx z>>&H`oDVT8be30ASeE--XSW~Mv?iG0YxiKD#%8xNQqW2v#t7hKOuC#UJWWR|y_r`P)Hi@v<{q1tz~F^Y2Pe!z?`?J2 z!x5BVKKcIuGRbTH9hf27PBD|*Ok84bY4qWTv=VXH98J1(%cslRX6o^rr$40SG#b{W z7`U;Qfw?$kV0b^?Z`am+H4Rm*kUus(!#`P_X}I6e>gzR9LslDUCvfl2h#sR#>O)wV zM!{cb;0ewCD%}-T+g?pxpysSwn)1Q4jm zc>PD#0rghhdeBR3UI`6@x8v(Ne!0`M`l=4M3Rye+u>CwvLsg|Tlc)?*w_SosOb^CE16o-QG+Nr5p`p;HdkH`VtbiY*BeKt zWi@U2Rly%v*|@~&?cV;%bz5t^QDvD{9@0+}v)Ej*E5~5o1o>}8eVW9uAKlfAx5+#K zK9>5Hvb9`|pmqZjIQ!FCyvw(-2kil49OUb|Mbrd@O=~|cII&w4ce!ofdy>Clha+J4CyUzm{`(kD4 zKV@sSwKd;M)M_7Ibb5U=hX-i{WSQ6MI&PlrC5UwGWwj{6Y8avY2OZDJ zoR+Jp^(`LiRW8tMAqq8M20ZW>WcHmg=9yhaa^tc$snTld*ok#z`?7}MezS9?>P3~6 zvtLH=aB$hg;Pl;JR`oJps$cO<^)-5T zwM3ud&od(b0NIK(&Ti3MNI%1ziSvrOrQV3mtjauQYUn>}Uq+!_otGVn?Gd+3+RAIM z@cN1fLBlg2>oV`s+of+?NpAW{$pGhwTKcY+sjf=y1M_FRjX=9fqRlX^^v}Qm=Md7A zuCMlj>UO0~u@46mU#9D;brxz#1pZ!TAkbH-BUI^%m)z)nVk_1X?Q6H>$F&p;Vi#Aa z)a$zFy0tbvHa?Qxd*?HotEuWLu+VoG9m$&YYK3ZRCnS#(oY(rFi)U8@919_h+l)u>078(eM@BxxDMm%K4kjajRSI=uaO(jYxOiRZI9{a;v;U`SE|DcDLY&a zRzK3Z%2`${d>NCX>s>O#jZJC}G0btO>AIEQr*T<5xs{tPio4V?AnpguivqI}2L@tj zI=-5}+}d-4jLlPEW%cVJt0TncmmIOYa5XvDrG89Cw%nFowk)66qIBK91dz=Yl8Ec17s zStY=PQ9ObaFCC&DRqYBp$O8aL%u?CJt2Yu0K{G>TCFOU)GjlvU!48L*Wr44}I1zzt z$jtPN#Ux9KK`tbdD8^WvK{1k5+Yx&j62|5$V{-zAY)MlDPcrGu6*QfMxs+4Dgd;Od zgk*=okirgDlS^+9o~%eY62bH48(KQAnSIeUa|*8{BM}0|SeZsVO$As=>@UC>GR3fE zSdqsgh!hPw7TR91_hBKl?4u#;Z=$rOGH151?50TtFbO^LDq(jr03j9TAaR z0kM&c=Mj%r3hIry-0`?$6E_{Y;mP9U>gnf8rTjs$i`Y5ZGslqvq*yE!)s;b?Vgvm? zCq=K)*K*jP8rYsl9254O&ZqW&P+Qcg+tUE!Z(-(fxY=I3c_vYtsthhhubVp$8;96M zZt3@3lX@SnJ55a{jda^>b}M3@bCMhRa|^2KXqD}{zixjr@C>~)<<~*!ExAI@Jov;t zH&oOr*ue-L`*9pwDY=P7!so|1{Ut2v)oMxn;CIF`iLLFOGpxA!#G3&az{&9dTFp+Y zZV9XqP9}=!D0>yXFvUhW>$gQb<8Zv9Ff}4&&x|;xFW_$TDjZTyC44{T9pK%?>)&`!8 z`b$l!bJ*pM0GiuAH*09Rb!lPT#0MD8a~jjNb?O$o2nQuvQtNeEYKyoEKAvVsyMCq1 z0=OH9Zs0@PT9D?yK*H>4^RNy;`uK>8^)*_x6ji8pE5Mum8qh(p)SdY}uLJdq^=&nF zXxn0(u6Z)pzpK@#TPfY2t^teCO`}pyFwDdZck!R;G1~X8y_Ea6ah3l7pIG*a##*rc z-J>L!Yerc>*3s!{*4nt-aLY5**O2Pmj9^Wt*{!Gdc_i3zk^Nw=(XDo&10{g~f2awm zSjub-2EDY{s=|FN<$3(GSUTB4+gKUS09F~8cWK-l0hK&tj#e?FyftFC;g^j3%FXt1 zy6MrV)SC8HbGSXjCQMSzF0!jlo}Sbr2b>&z2h3G9R?r6LB?p}Nmu{izw#w9u_al=s zYL;E9_12}VwWMcn(jJkkW*}BbBL{HAc8e5NNIN$S7blQE-aypVU7*s0Dv#&&CR;j1 z9VlE4nvfMi{G+$@?jzeZK zUcDNkk6N~StNTcb(^7y`J5^$Fj^bMWoo$sDONQ`$p?C-HIlBd|)$a1}FRsJtj36SVf4j&|Rtx*`f0tbPGIN&Ck- zuAnzh3D^QyfHTZmfUBZgO0lT6+^@ipFI%svACNPU4{=HgFJnlovpRUkiAy&w zsnI|+eq|%;9kWD~pt6tw#~}G1?*@o2)GB4T931}u&-i3nrnaL}+lk%&uaof%l#_E7 zI7|XgBYLDqshAJqVt$nuk;#*f^*Ii9f zKf7;nC%$}QBUYxwcG|^&JOko77V5O9r_^R&Qn05ba0W(xvgxNZSlhRi10!nYd#Bpg zko2zbPq_v+FV_d0eP*8IxvbONn{aSEOkZu!K=kgRrS<(g!PBD+tGNkQ`{S{iI^98S zI7Fw^s;rMwJC5I{!1_!#(hBXcBG!n`G3wys=j=YyVzjeNp*J%lpl@2UxmfvLJouTh zr0DM#Qrc%Ais;d|p9ZC#<7b??En{7Iv#0J3g@f_A^Ur|!slqC*x*pcV7 zaja)I91CCn0LTpX5*zbp`$m5aLlku^wVvU*jCDSrSUbt|v6gM!xro1P#62}txGiUljA!dUlcYYjnSXW*Z5+-A_Ii|kQjiN{2Z{6*(?<2Ry(q0CIAE1Ur=sedLkAi@r(89p^~nRAc<0SjHPqU`^W&AYFE* zTk{^mLWdH;mmEO=AB5z z7^AM~bvId1uC%dxDNqaZjEK%k&B7#p%IJUoO>0i&3++*kCP!8Mh}Ph|TA^UZYlV~uW-wW+k)iisJ^pob zSmhpR!IbtvXss0Kg@+z+W@X(o)VV9I2(%xL@vT_YEzD`syOKc#Iqp2m>h=1qH|~@s z@xd$3Pu_OJDPtK;T^`cts?pTGyAQ5Ud48kN^qRF@PLN@M_b+_TX5DV4<{N_-Q`_q= zqUze6HQZRh{28Vj6m(kOZb3!!x^=p32TG$zzM_ZY$nQBt{iL$Ii)&X=0jT|^YZ(fF zfF8nTwIiYGT_>Y!5VZi+cYJo77Ozo%Pu8_4?odvW?zLD8cF~Ub&_h);&7_ z@#bb_*|C9MzUEeBV~qUIp#IHuO7$89t=q;qpE}iP-$_vbocYXi_HU{kHlz&USj|y7 zud6z(3%;^Q_=Q7Wsd40Zne}R0RZ2ByBNIDO)c*hrtQiCx%v^WvxpdLhRuQ%cgN#i* zKB~T=zxiS*sGzG6$LTzI;bnR;%VjesIFspPyc?WOZ3(_q*dR;dP`oyDVAv%KK<|^B z#R)8fCQt};Tu3ab^p@E!qls{`MVTdiIMGSqxRTnHp%9~qAYe}(1d^=S5EfZ4V`3>| zz@?rEpE3z1Qb}y)6$fZF5sH+VD@3hfz>tgD7jF|lBg7jtGYAI~N^g}XGsnE}6D32* zmPO`UZ_*JGI$K_);v9CAL4hxrg<4Wr%E>Gxi$U&?UP)r`PV>xQ%n}{AfM`-6+kqkx zmP5>j?;sfim7WANQn&3IR>+D~hG3XR2+X4LC>bn_390EjY`)i`3 zQob?VCp*68Rnh%6usKRGk&a3Fec~m9V>N;L_lL83Ls(t1-MeBB7$y*}Cjq{HNIuhY zIoq_$D3Geo!Bt5=07DAJV{x%J_S@WvB?_NBb0Zu{BAYt%mc&3icH#g+RCf5uB2KWkciK*yF$$kV9j05&le} z{J5Bl^Cc5j`+qzMY=6kF*d2%IRKs}sM<-~HxIm_EL%IV^qBCDlak2A zyP{J@Y}24t{)S;+q$^-;2p{cBUc^{jYcO883auw+A z(^$4v{@nK*lO3;7Yn&m;{{YD3%^gQd6=llo`y7wu5x-2Rq>f1MGjeWYzT%eGsCDHk zA|wu?vQcowakm4ojwOX~tUkt3*PmuM?aygUqPW!uP+^e3wg({3KBG}+-&J5K^PJ=K zo@_IF)CN9Q6+I*v3|Hzz*ONCARa)e_+T>vW066_lykiV8#t7s2%38IR-mI+Sxd)R5 z5~jm21#$Bots1iGRAeP&li0T7{?k7F3u;NOwc1$a-1joq>1!4Gl=1d5G`jM<5Wx7_ zGc~q!Jv}n|G|q`?va#gu#$uY^TDtZ&3lqjNKm5o40P5bB?d`QchXcG^>Q?^%b_)*~ z?Iq&L>DmSt5L<931o)a-o}R5o?(#PO02Y5Rl60Qq>97U4#&S8HvF)aoQs9+6$7tKO zWY?lrTsnLDO}B2`qXFOMEBK>!v21V3I2`zquDCsApCqZq0Pz7=gw zyJ%f)Xsr9VJ;cw^SgUuqI;$xF{P)N8kQE;EOP`Q(K|CLb`qN2psxqhx570~0Q@RUX zW)w0O-c)no!J!jbXjVs3=NaU8{bL@nB)Q2@ z2M58GR2Q-`E=qo3o=AmEIuzOfJ<0I}5}JFhxtwMDa~c(zR5sPHXSi-h(r0Qdw@L@1 zNf_|mzR|57y4(GZs#p(q%#YZXI$W%b03c)%49qAIy}yv-7?0}94b}xY+He$1svX_Y z+>8wIpLg<(F`IRC)2jDDTV6u~cB_1d>N&!kkO#*xN?NT(oYv9Ij{W0w2# z#L0!K)Ovvndxmk0{KD3Pm48guscES4|4-UDP|1BGY>z+K9fejQ_w9^1Cmq{F^OBDvbv+{vev8) zGEO`nzsyd$RBsyzX22lvkCQZ0M%1;IT!6UXDH;7h5;VOz+&h{ZYH$j+cb}$XGK~8# zq^`|Ar7bKm-AE(+PJ{ueq^)Y_VBRosf$@{@V3VTiHxpXtAbwB|59{R33c4>){65F7 z&;#X($FvjJ8H<#)xpcs_+fvPcq(EE;Daaom&k@ZMDYm^no$arA_WHnMf59EberB3} z$?Ekr`ip84$TbEYeh2>mm@R+o{;Nx}o9?p8&Q9bFw0G`E`*0(;HeSjeyoifuo}liYttayagVxYL1}w@ETkZ8h&fn0SpB%;M)C%;+&;i!I52!K*eBkylYEb%@ zPNMxYSyf0t86!Nm{^u{M(fT*0>j6Xm0BjNbGq=;&w&}vQ(Pn#m=i*`Yof>I&olR~= z;uTPh*!_G>W0%vOc>T41WOR#Gu&wr@HU}BZ?^2x*Wp7_9PBISye42|r5?Z@Avp28+ zIUsiWM^mfy0HaD!gkkQk&IGKp(>;B@ms;rQV!`VxlY)7~sOo(YSu7WZU@$oDXU!W^ z)oDO5)rRMHs17qLUcW_alXJQ{H@T6xV-NB=#q@hM8Xa?PT}i+^`|-@3PL(V`Mgl6} zF!vUpDz94AHJ**`LlJ|X)1v8Lw2e_p?69#TlP@a;#~RzVmQSpuTUGbJDxL#kF{~T^EP`=H#@fzZtjQD^~h`3uTwYuwq42kXR-R`3Xj;z>9*#o+NxU+ zD8}R4vB5Zx3#)Y>zhxikAdl$?zNPwbYSgB4k&}tf%6>Jg7t{3JPM=HZwxnF(VOa(+ zGZNP6$x`WG0?cPQ0juk7M# zpDv210^ytZag)X|HLB^lI(M~{Et?5DtmLtTEn6rFd9C?4JdS2hQ`7Yz#4TxVoD7T% zocx?}VSnh(&g(k$T2v<#4WsgJ#{kbjFuxz9O}}h(l(t!cs186mI`@VzE>LzZVq-*tB)i>8oHI+N|piT#TugKzO{M(Zsx6ml-Ju_a)b~+Mrt&9f# zMmkpF3+c52&N;8o=uh^aHoDrhh|gu4D|78%LjgMiRQ_y#xSM(oxp}BEA+#XF8lNiO z3FcDYvEx3Sa^4iG+Pg^~#l&BzblP>*a2IkPX(zFO(RC_y^@Cu`FJRx9a~eS7aaUVL zJA;Eaj(YUP!}Y^;PLudVt@QwAw|7?0wLu>zyy8zwx~svnQ%SAAHQ6M8FD8Aq!A7p8 zru7c3#@Al+js2zRnmsTHW|0j_Hmi*8PyVKRczOPwZcZ-TA6r`}w#TJeRd5?TmKfo` zUh@yvX8q$ZIh{vQ>FC`&S)kNC!on&!8216&o_QZL4LVIaOF#brWvTR!{{Y)TmY+im z_Z~d4yF%THkE%UF!+;^{l=v{8)J40lAP0p_L z-KR^+`^UuMItAX9n=Ax2drs>^QtEWhbqIC?#$*@hwyAaC&CNhYsO=btBGRN9lmG@J zzMyJAoH>0van@Ce!Mm6y1~qcw;9PM$j^$r4M8jAhu?4gOVUc*u8mT9U-AUR>z7Gskx@0cvfEhH?m zsUSkCW;#}=sgsD+X@DhCymN@kE(AhQU`xpDAh!|Pc!&oaOP7OZ%*qUjAWvpoNW2tF ziRtXbEg+d0m1D${#pQR(lLxv8ZPFzJW)O2Ej7Td81cvPt3kVkN4$vkD6D#Hr^DDMV zWH>RJ+kq7V9Zn>TD`ZPzF0Tk4YhiLF2$#&RWP>6~uI%y#K4s+(5KvWJi@0H!_9Mg| z0~5Wl-ZSP|eLzEiK+m@xPsH*9T-d=$#zOx9lQV9clVUAssmreX5(5Jam{r(OUd(c= zGqikt;%Js!SZ6KVl4Eq}mFr?$Zv$}T^(hL8!aQ;!<;=os0@;j5b{{ZR;o&fk5l{`+aTM^Zwog@hfD4OK(A|3ezJ3z;X#3{{ZSh4P*P5vmd5o4X02|nv0Y( z5=c?FlM0%4v52i1R1SScobGZu&O1veRR4lxZ|{l zg-oG^GKNVi*%)SQ56gq*dn0xV;Q7gvlKCnaeQ@adg@hJ*{t3vp} z@L-|{EA4I!8xjohVpx_rG8pkDk(`Wub1n<*!6Ytn24zziB$2@F^DDbBTU_9bUS%HkyHb zEL>xcq}%i>ELpkwM>?XZ>7i*E7?`+tyJO39eA`mrgi&fXU0Wk}aAh?5!Ki(<60Sx; zKg1yV+B#Z8kEobs-7N|mjK*QO-O9&G)Q2SToO8?#bdcK)Ta4$tLr<}_BjA<9x(-`*4axHa7Z2S4|w0=Y1CuW ze6sf#?H@yD^!HM<0!ju0gj4=tbPYb3BR};zumg-XMguQ6`be~2e&brIf-nhh6MJJ$e(-H| z*fQjhA!*c_*A56jB4`emsI4B~Nr)$$9{&IaE34A`0M6F?W@=c*tn97&gym~ecMueh z5^w4HgjKWxM`I`{QK;FBa9D2RKcwH&-A&Z(EsP&&{{U2N*P%Ckv2;&su|=ysNdvTH zwShD0DZ&0BFJh*e=5Pka?j&7hEa|j{4eDSK>AY+yyf*~nw5pf2wHmti;J9J%4?nC%nv|=WtYWTnlim{kiK%Nw;Hz0V z3xk+t(YBBi?c=nkLe18AFQl;KF-PTaGvB;(Ox9SqaB-Xg=2uZz^#ZTlpQI&aHh`HY z9N=^MO5K?>i}Ze;qb+G#bCLk<^TglLDp0TKAYQ~U^YJj+mn>P%8Ti1-z#b!2>s+Cx zz5*{JJ?3uOl18qlQ&m(1HWRZX{RGC*+AUVgW%5b+Bj#Omr~%*SIOp9K!_}y4%lx!k)w zrJ+*d!vHev4UCgIZ9NtGWo=ae_|7H;PL^AgU>F=OW~~-mT4vf>(4z#8cN5Q+R)hZl z^pwgyY6%B$930{&Z7Zg<2Otb%@`L<`#H)3R+RPx>G2mlAq!Zs}tN~ImE68F!vp!VD z-%sn>HBgEbP!DSg`yZ%>)%9v0Z>hS(&I9L<`Tb`nUY6v$)C!CS3167{Xie(D?XV1c zYbQAW05dx+dUe)4;Z<3(2{`aaeE$GRxo(f~H5%{LS6eU}pCqvu>}Nlvpe>A+lyxcy zDtX{{`p25bR^O)~{2qD7#AVVuul=9WY7o|{3zQjZz+czv-^Arkm2|GbD;)bX)#GCqR z>$I%sgrXcPHy@@yOvc-#>U1?0x`w6EPb6*!IRNLxzL~3~3bR*h9!}xgGp5hD<9d%s zWEJ{Wl(QtDC{=Nuc|E3h)a&W7scfOov74QPpFb15)oCi#FQ&j)nLfaI#O8Xwgi&F3 ztv?PihhASlIQaTbxaHHHcx$#AYc&9~?iO)@V8vdyo7=vKY#K0!VN##byGa0aC1jyd{% zu>SyybhT8K*?<5Z1kO`)rm?!Op>#cEmg!w?+8KERx9K#OPS@X)&U<~PC+y3MD6`aJ z%YmPy>DyY(QS_d*7~6#$%;D0{wuer$4rmoBfH}mbwxn0GhI}V|&|Nj?t+$fS-GI*- zhU*TtzSIEaa$CgX%^Tx4sMhKzt(!u{nDWZUn1y|JRZZdbMN_!RT!Hl;S&^sfhb&UU z_!&6yG=tAU2=Om3)vNT#(RBu>BDV~164ojWs(IV;X4m6=1r{+jQ~=1y+@D{Ze`uz& zq;&qFa@}5~tFcKWqVdntSN@x&5nurM;v{oO&^GB6>BZK>pl40yq0OU-Adgxo*hSGAobPN zzP)0C6#j3dz1=nH71-`Ku=gw|>H6JHnbcHDu;5JrQp(1{ z=5-6yKM$rsP~`cAQ1C5Ylnte9Q|HWortP`Nayv{f`i+*Ipz|N-*8cgG&$!N6&Bt6l zwlx|hW+gjBt*lqU8JZ1g18Vn!Z8esx@?UPzk!cOa0E=uHnm1_O2_!&zKS`1jAmaiy ziGVnjO1Tl0TnMBp;7Kyrm4QTtl1?QU%%`_vRi041scgkWyYUrvg=3f^AVQW(`Q}_# zlNd_)<}g%OUVKBXVp$ge2tWXc+M#hgc$Ov!jEYM^v++F8W=ZCA23|`JB{rFqAUTCW z1dA9&nomlxh1&&U@gT10Qe{cRuIOyM#t$+{lqZ-GQ1cG4bc`oxh2>cBEe7T*VYz{p zbIcZbf?Nq9WmLi|aS*8!-JuK|j@}K4b$rTVdlCYR2=gQ{o&+)lBa!AtHpwr#c!n#G zry1{#B_*2Yd~bGaa58%d9if=vLHQyvr>J^lxKO9uNo;ZB@8(jQ$ZRqzXN-}^5UBK9 zRz_0Z6KGR|@y=&FoiKwYkcx9v%fGUdz~{W(r%9*L(^xJ0@8Wadp7@JtRM3m+VB7&< zxa>U4-lePcHEpIj3P5f=ne+buqni4Ue+LZrOlx&D-FTv!0bbb8=K7c$D}04^C^3+u z$nquh!(!N2fpwkS4$!yIE3eeFGO!-pH$PK6@@jckrrZ9pYV_!+LRqzciJ0;~-UV$2 z%vN1g;1CzT{P~Mt)srfSH?wd~BG>f|O(zX#aC<8#kIneMjktawFexo z(g6Pea}(9oW|L4|0V)3g^ERF%PPMZOPT=RpGJp31Q`*%cHcu>Ak)McVW%RM!x~jd$ zP=(a{kGF5q2HJ`uvcDsYXEM4?MaNR0s9yn;dH(=-&FB{0p;_+6_GWMJKGV~~mgjFj zHX6Ma^xEXyi|!rOLC@Y()3Fb^3D47saZ}{l3xE&i23sKpJ!WFVa0{M3`JTCx$>J?5 zU23+JQv6{_;14W++FHx()#cdX2Vyw;_LEJh>=wmEKnY`x{C_D>U7EA+sQFDZvNpF6f8@Si2aXYG*hfF`93xyac2$0RHp7KGRan zdV=F^-3>I)Uf0?PJY<{!{Nw<%VST%?o!!Ep^Cj+U3pw4F_Kc1cK^9V@oHxN3AI@r9 zS+Yg$KoB@3ox6-;2$DU!aD2;;RG}MNJ;C^bC!XZ|=jLu#JfdW9dypr;=Xn{LVPK{4 zx4+kkR;|MkfydTdl4Vs{fn$=Q8$K!oW+MY5&LmpppYq@Vfyaqzi<=lB!6Ubm zB`LDexPqz<2{PW6l%z8fK>z_J=WKw?NgqBTUA-hFNhdx}BOgeqQpY2a^p@JQPca5m zNnx`al!AU_eo_es2a%Zsxde;=1feqdR$Ksco-rcC2%~ZLd`T)LgsB)f%#kFS;5_(} ztfQ06HEHWFw;+5;DyMl0qb6%rq;sJ@ra0zXr*!^9k$^dr>RS4#qS}(op4i7}%R2ON zV|Mw+?FO8FJ&SS&iObtsCZ&n^IGs0JxYX?Sa!ll(Qf<_G223#Co{Xq|+Ig`<9{~21 zI<$0jIRKr=3G99#^#z=h;vs@t81p?CV=V3F&plI*sBOlkPjVmP9sZNUSzu9H@h#{(0x^&zHA)pk zcZk+U-&=3tmGVN9$IdbRXSBUd#y5~vN4q>8_{>dcB@3|_9m#-oVlX=ynT|K6yt19T zLMheL?P5ssr;!b$vGn{7apyCZR}rTP;~DcOQKzoFvU?bJz~W=mds+HP)H1LZe?|*ZZh4AMuSaLCAI;UINaTaA&#?7 zr0vJIvVq+E{bqA(FhX(%`j7iS>91j^Urw?ZoR)E)kW_`eH(6fWEd|01Z5TX7t^ls> zC}6qE0W|Ko*;!^=cAn=P7@G~(4^L{A7(8Piq-7Qt_3TVDbDzpQm@Peejb5QjSFybD z#M#r)>C}~2h)6!x`1}6=Sn7IcH&l!1s45SZ@heuD8YmsHngI6WjEd zc4+l_TEM}vGl0V*`N|E>zU~U}RFjkT%;#I%q9Uhc6;=WQNg&{J<|_ABr*c-rXJM1I zw`1=STbKZ~stkC*=lR7o7TVlc09>3f;!4v4R%_L-sO_|$2h828T8!5^f4-lt_}(@+?13^Ot5uWsF|({&mRS5zFIkOx0FoVLHFCW&oIwE+mm z#l}k?zxW;Imrm5Crr%K5cPPmp5dBL}_{x1n%K$LkkU&3a(+vAwoK;m>T@7!!t~RiB z=PU7_zxbBDk?HjH3PA&$d6(4a^%&b)Y8OhUZ@#_Mwi%(p_LP~MuIfcsUs+CC=cT{+w~$+aytyHtfJM!><&e&@_xA1tAU?Z?b}S)si& zaymAi zi}VUA0axH2@!d9r(8a+N)Ob1WXE|oJ-HT{6-lqbVBRC73M|!S`sufh01decfM|yZV zT7mTez&Ivh_1#m~X?&7?2Z_TwsManr+f{cpWB@*qj-S-2VhXvg?<&WG7_*=%)NG-_ z1Gop7&h;%V`fwdraR)zHj}^DjwEE$z)0!^ESYR;tnH@t#)N~BiijCS;?&x{M!swlJ z(!nnLjmIOAJ8cfSyI0!G=tc??kVG|M-rTv=4Psa~>^SaMkJejBuGHx*Rz}?UPs)^rMnz{ zkFNT5U2dY=9cclGOasZl{U+{G1mldWqsDp+h_zS2SLehQZ$MgfhXMK0FM(V zuXOYmuB6!NjJYLwI35q*iE`C8oAFqB-3HCA$T;&JxmR7&fl9+!w;YZ?S-w2{bJL!9 z%O5rBhTS(vy0=%|W0OkmCu?^EW9ObCnl{u~qj%PUN~-QgW!lVVQ`S0m%l$?560zEq zBL#r)PwUT#&EKfg>Mskby1QE6Z|<10*MZoX_PF!heBANXWUac3Zr&Ug`ps=8S=Q6K zwkwQrGVIcoR=R_Ih&c|m_8(bY8MIyv+m#1>!xPai+-ps>OR3MQ*0n~PR!Qu8M>O`edg{h=p7DKnYqWIQgDZh33#FBlIo9L3qzr<2$js5H4h{s( z(&}m&D$5+4M*jfn3F-Sg&n8~piRn#MMY0UddUiK6L-D(~amUOBz_RWGB4m{c1i0c> zS`{G5875T7s>TK1#3hU(uFy#>BX(fPk_-y+F_R-Q!1EDSMHz=t1f~>437G}u34&&l z6`ogiV7E&H1eq9V8o|kcFe@qM5ljk*MI1y4ndxD-Z=A~o z2j(e2CBl!1aGu@6wD1y2fr&qSNyM2XJ$M_HgD=y>uH4Ix##-Lc+@w?A!u_@&09W+jpF)VsM~+B+%vT1(O-!dBo$Jwhe1pGR~)0RhfW#Q6FD0Kads$3MZSl~+|>rRrG^ z^!;X5zgjJERV#tmf$=M@)|Ao;l)ebyXRv|k=~_Cypp%^E91pzbyKcUnW-}S(P1V!M zuYS}(i{sDSz#SDR%)qw%5)M1!KkF9L?#OV5Aqel!oJu{4KnAM7&ORqxc;$1;lE!7U z1>6-khCiGxKa^R{qfPt3&eV?U-yd8}mPPeIbt*?-M-c_C!p^3)Vx0SY31yC}SY_oq zTS2Y$N$e{eh4JSGd$e9F<~t$@@dCbpk_B3-?j=`oUdV?oGQ`4oa}jKG4m**m_ks+%Un&Ir~nW zd0AXJG3hM9Qq4I)N6HQd>lVL4;=rn;h5!{_1~J+eiXOr%+Uk6{{*y|iN-G;qY(4%K zcD&v(%5Zry#+lNxJw4tq2atcsj_LYyf*RJrE1jpD$g^Bj>_df224w)`pNNa=Z6uP+ zz~^Y)-yhm?%ld8f>yPtE?AquW?3TM~j>v*HUrR+I1KuA8ujvoo7*IDm#<`oa4XT%N~5C-BwT_OUSz7V{Jbth>ZuJ?0oux?5aXU=+dU^80$Jk<~7(8vy=Fbw_8ojIlIV1w) z0sjEF#Z>AazV%Oyz5f76!f4rB_k}6|WrKiE^5$#gW8>LfDpp3~g_wOZ40};sL(*HV z_6&aP9A*yd@QqCj%n9YN1_#nBR+N`#wOkx~a(}6u{J!mV#}2h1h?+9zHWufAzA^Lj z2W6D80=gcuGD$p_&!5x$$SYNO76Q%VXy^IK6*Zpa53?8m_7eFg>G<;e0fj1som#~n z)yD(xDPje3g4M{O4vh` zqzZt~Dl)|V;$^q6T$050B5B>FTH9ez&IdR=5BGpty%pT5eMSd7c3=I?PD%8|-DwZG)^H-+y%dW9laay0}1EbQm zrsNwy8Sp07r(Se5jXf)q*y3j_wHxE(-7%@uSEbUjD`oM4iNu=^<1VuQ0LV#?U8{X3 zP&R5oIUTdwa@||2`uDO@FbVNE?{8)^#gY3*c-3kdDqQ2@WAy%9{8Q{%U+yrK3>T!b}0iJvGE~>F}sL%u%flrN{}7c4uiiqntCliq{iDxP0Cd<=RN-bq5iWO>8vhIFHW0K;dd(z`Nw#x z2cdx7y?M%>;7MBSDy4I|y`!Jy+BP~n`Xnln$MW;eerG3UVPg7~TFY%Y#ym(XW#9JB z;&KZPa3pmk{4S-DFn+@a^^f$rg*2-_C6Km$MAz2Y-+E^@g8JMehUD?*iFQl$H+n7` z40t%tGiuv5+q-BeKq?PnVzxNHSye=mLlSuULuO)JRdtl^Ye=m20Adj|!=`Fc>;b_C z^fOWHMW)Jj1tjgjJ^l<0tSq3`$ASF;7=f@B*H^hX0E3OEjE^zB6?OVvqX1ZtM?W(^ zrs-*|8t2JjpK%HSKYs|xlxVcR&#nz!qo%0Tz+VU-vMF-+}7 zg1W3enHzri0%edZs|+wn@AR5Bix!t-yCm_S6F$_#U@jXO3!H*Jk(5`o*PS)1X(t~O zLY+&nR>2^i5+_Xz$1Q+-E_uZ0I%b-n=XeC;EJul)nv~~Dc1yjZWo$A}z@1i~TU%3V z=L`P;*CD@2$bKPxHrOzo%x6!oHK{9tpz)tGIQwXAea~%|dVAGz!e=1-;N#=_PH4ZO zs%u`06|hKs!`wXM?G0nKS2dy8v&dzzVtT-AMZXq+cm~3_&yWYi^v0r$)#^@F!nhnW z6M`UaICAZsrv#W`=}~2&TD?Q5_JlcAKVP5i_mM)I3f4i(c|C}WNubkpY=@L#iI>zm z8L1ljoY~LiC&Xdg8`)@J4sulZ=4w%@TS2YGY*&Ii;#<=;Y@bc)FQTn$Enz*tUlXj* z_5DV(QB4|h&Krl^Is48NswS^Ra<~N4-T~tu6MvxVG}>BhzfV~V5M|ip^D&(3Y&*SA zP_BzBK){jT?KytA)s6a=#x{=;El$1fxgEWuwn^_ZHE(9)V*0b zSw=RuV-jsz>J-(hpqBXpFr7l?8@B?#BgA2^r%_l@fa8#H-gI+wI{x)ebo3WV!38w@YuVqctAoe}-;!o}`RZ zw(xuWiO!v0rNL$y;I4RpTa7GUrraC04UH#NoVQbQ$261XJGgqTy_(HFuEIgKx(8(JN#a8F zpHa5jL2PZV;b#guO2kJbll_0xU|`lz@awgMj=x$`g6uJrYHY2SCXuup9N0KppS{++tE&8fR+ z1P(^;krQ@m=uJzu;eoublh}W0n>#;6ziKq}QnlT6mwEwD6b@I9&Gp3K`j<_(Et-ug zi-n35IA9nr$ruD5top*8F1tyx%F&m+`R7>FUDCRnY3xlsbvENu1qakpMpXF+865o2 zqx{^K^zn1!uH3CZi_$Wy>I*0*HH&?t=wlj&sqG9)g(ptk!(4N6{AY=QUE|aVol|7kYtf71f0s2m181ERcykk%nAZbVON0> zQlcDj2CFZeNhGvd*U2!B*owPzBC8n#yrDeF1_(eiJ+@h8qN=LA!zh&lm_?9W4kaJ7 zzHtX(;wY89;@IXhZzRCum=~2{0Fa^(2`7M;0zebNhnxZOViypMfPhmx0i68DH2oh_ zW}ESPma9Xe^QhIoroX0nWA+m}Q~OiVbax#mM9^E-_5HnT)TJQ&iufdd_8>=Jqko1d z(rPt1^eKA)8DgV_`<}z~gLEkDjf|4Vr4d{euCI z+lD)-;qXGgN{CtnlGhUQ4!pM z-yOe{&Al~@>;k@m0#%0YR1@A>()4N9(@;FI<8pim!qBXDJ$HK#AdbiRLLCsU!DBO6 z0`BEV$KG+dM{IGR3ZBgcmoVq!{LIC;Y2Dvw$rvlUAO2?JqpCHG{-JE`HTJ~C)>Ovt za~D&?5zpFu?reM9eJ|(Btx{IHSix<4HzY7Q<2-^PYH}2e=ef_BoE666;D%F-{Epx4 z9noD_ASncaf=`k3_?%tT$~PkHpS5TS6t)sT+v_af6I^{{Zte60c8o zEGppRA+kT0hzs=B<9*h8pBu17f0VMD?i@1JDik(IaKw>@4WH8yojF>YV*0FdHxe)~ zJH@Nh-fk;Zdl+y{dFSsmsZUCcnt=Bf$i{Pz`SUq?b+%MnYHlvfHo4?Kk?=9vJdBlq z&BbAIPJc0(=9*deI3ZV&*q^b2)C+Fiw$p-g0bG+cE&3Xl+f_9nQggQ;WNkgBalN%w zwPdoceeLv@(}SebSlWt)<`}?k-Tweg#I)Nl)UZ2|S0Je8yw10KJxS4;)hHGL?@R% zRirGt4Vm0{$rxYfDao%xEA)Dsu8}=QO--A1|u|SD51C6 zwcK_K*h{&!vi1eo+%^wjCjMPHd~mm?O%<@O0N24_y`sxo*?t4DUz}$@tYMG2s;`wR zmTwW14YX7jDhIiVobM&Ixs1jXT`g&6@A4FWRmcR7tTPQiq$}Dzyo~XWq#;V^Pg>2M ztDkQENFQmRb;xy$Jtixh06)Au>Bk+M6;jB=XMQn&*Z}=vu9c_*c`gopq~j1*y_MP% za^#Ga&Px5_;?{dlD&P5M9LvtN;L$eMUYlrl!S`-(Fh1WiIBRUO<~ehY%m;X2s8lVN z0;}84KLQ2VZDkoCjsQH(OsKLW*4Tr#3C35sh8C9~x|6(Z8;&yo6@X?4g^B!*I{^j; zJ*rvyb%-(RzPpEUMJ?P!=5R&w12qwUzXM5i2fn zjOPPCS&F*7Mzc}sEtRb`kWGWM&CBt*j{Tc*+I<+)>Cr*PdGR@m>9Ff7vd2E+cm{NK z^$j>?*chqvj^ZQew@|gLq$*V1$Q^{SYPs5|)ao^C#Wt=;1onnwPJ)K0Ida3?N0TS2 zI?}~Z=L4LXiY&3Jw|+3Zg2U!?j8t2TPqYqumVvuVS<@r(2^{_7Nz?7Web?`X24&d{{Y-BT?vn4=(;$d6LN+*&O9t7 z$x}OSWmGG)eW5iisJvNd7_Yw8m(pofH1)lym%v~M-)6qao^%= zt-EyosAXEz6P`)>%**v*e{>ac+~kb$^C|d{X2VYd8yeBx&6H1B1W1uasZRQW%Dfj1blw!V(vQ%UJrP$L7r zFn?K}s-#n>TPd)Qc33gb5PCaKpWFF{Fxcb?ZBOs&e@q6!!By@ie$UePTtkvi8037y z)Js>iGW)qBJj~k#)>KrS?c}y{N5ITI9<5Z7a8QJK;KfQUnysu9F!V%olXAHe8I0{BAft3GxrbcTJ-dA!6A)jG0Y@nu`8V zVVbsbcY8w-wNJ!$6l*qu6aLbV^y4NaVw*yePjW~C7RtgSFgR@E8-x`~#=>`;lMVf` zkS(MyQnG?E{0F!bSEg3tvDu$E+I)|Q*Wl3UP$_N1XFc)sh*xo`*$4~B=UQv%u zg^_I26X~*T=O$xcY8xyWP^^8q@*{NXPNIQP2kL%dG@1spF2isn@=kkBQs}cOsr2l? zXLAoHw13rGs4B!E100?N_M=P~6#y1dj1e*t??)l`kMxEa_WuBxuOB70#*;#}>DwxI zDuOV6^16S-Qs%b=DEzFWf%C*aM!9I;zXWa0W}Es?SqgAK+yUc1GoPlSD_ZKLmMYdb z+6f)Ld&1Y#v|ufkFyN8_Kh(^zYgd1&=-9?91CR_j)dwJu4m=Us8#*&?mX)L0fD9JI z1$Vl+eY+J0CmqaYonD-s_Acbgj>wr5Ja>A`4xsT|Sgn zIrTUs9zgjYF>P+Xnzq!Na(m~*K>@8Pg0m*;_+(mg<=uiKMR5Gmw6; z`l`JL7pf@v81Vy8dRcH)oM6}oB>kcKp1D~A3d0U^2OrjT({W0XSZ{ka2R;6gwbZR+ zV!7Hrdzq1K!n>6MG4_lR&IBbwtNhuKz}wndJ8Js1+T4W%@tk>qK(*@`hHyQt#0t~^ zEG*<_hdAvRuTkr+{l^rX_r^>#kLm5JQKl(-9#41N@`I{Xs?9@V#9oB$bJ5xv% zn$AZdmULI#*L42?R=>=i;2-X>>OmuCh zI;&e%O`pEq?Las=9DP2&F^b6A_1Ngb;m+0Dwa1b_NRLK#8awJ!mD!QF^50*)k?7H{ ztuAr5I43YIUY4y!qRQJI;7G?~IltO&nO*vpTXiXX4J}PeEsJaB&qYZnKHc#- zy=#iCdTOw`DIhj6nYnq@rk2Okxpfw0?X8NG7{&q=?H<2SZ6EhUZ(huUGaTKBIt*H<4iGOb%% zcmtKpXIpDBfx!Dr-r76U-n+>=Fi)Akpwg9n6mS;1i(CbTKOy=Ei(_f`rpn)^C zwA#7IT!@RYnk56*n2o3bGG>=BOxcEUftbEz9x!YnWk{0Qb`qdNFqFikNkPmyh!q^e zq*laHEF=yDoZ?AoR*||%r}3`l5C z5KLr&vAe}w^ABO<#hB#6l|0KVj7bthDeVdj0>Ug+6nB>ri!LG)2WM$>1m}q&)(K2P zyrQXRR8U!%;Z;USKLATf;2{xNm?9}=V}Pm&B>Vs-#*g-D`zvep)(tHc^JYlOkNk=D zr|NU`nO#fv{{W)&Rc$Zq?wo721e7{~yVED&zvr{;yH+xOvv1>CpV{V3J(|9$s;j5z zEBxxXSlp*Pw0R6HR60Lg$q_?gVxPhS0{ z)Fz{<>g`lh`?}NxHIMv2_S8Py{bs(0rD?i-TPdv+c>s6&p6B(KmcP`@cmBle({+#9 zJxAjD?zQm8BD&n<3E@<*BT zd4Hz;K8F|bv@h0EPh=+p>T~zZTh?1titQi~&IfTZFV+6}{W^`7fJWwGKctN|px0?G z;MPcOx$I|}a%H{k>&t>`(_q(Sb`YFl2?_^^nbhg6wEIP8dHmu;`F~HDpQB4{Ur!~AA7}?4e9h&xbmV%BJwiR8e|dRdw>+_*OVOhl57}3{lg~dN z5HD_$oTXJ30CVvc)$88E61=Ma0Hk?yn{W^3s>h_;-N8E}5Tr_yg5-)!ZNT{kUcyn1IF zJDhQkpu|PoF${LF&MGv@LWt(5ev+B%a(#WvszklW0EE{oyt(rovf*W8A87`j~vQ{Bc`QR8(%r$1S^p ze$jnJ+mqg{&unp+XBE>?XUTRP0@*)#m3XrIij@R`?NR1(xmxRnFdM37>xl#^+Uhgg2ONJWxEnzrlky1fGpB8~Db(s3u(Snody(ctHTW#f zJ-=SlB;8$VF}PXuVSylDcW}x<@J42CCP>-Sx2PK1ocdtow}!`InW6ZkdL7el(Zb}; zU|Fbdb*s5YY@7iuEPGv0y=$M~2ZSGZ%c#a!%UV{OExb5vs*Gfg^5}>aGWm1M1@3;4 zdRnjs((h(JHV9Q8Kz#jT_gePa60x@@yw=5o+I{&zTRoHG=wMCSgg8Y|o<=3qEjSz; zf-s}M+{=tI5Tt~taNuJn=4xYWIb|S1=R&5;F7Clb1R^l}#WuD;4Bdu*NJgT!1T~Bf zPX&DaA%$YwDy)TkjEdLBC#McQJlW?%^sQ@f6^oP49RC0)$)dCy?^frYdz?n<2%2|% zf_t)q+GOely0$l*gQ#uSmBSih?c*qx2gNJ_MQu#(-T>$ zTJ=0!3UV+($@+Gg?XRZoPqeMp074@j%DD1IIhfV1y+*@pwHDV{sR*bN>spKvj`{jc zu9anl((74qlEnu&nEGu6KvhLmn{vDITOeo8GiUyq^fA9al_!m!Tb};_K`m%ovi(Cw zy2boRu&~J>G2u_tMf4py)1+arl*@Z>ze-`GU6nZMJGJ%OyN)JAXC@ zVf@6Jk&V_Ks+s*O0cAL6X3z4NDb_U<(#pDr!6z%XjD04DQKwedEjvI~;DWsUNr*O! z@j)qF+tiZHo&Xc3GFEb9YOl9dZpsy#Ja_J9o|j2wbq{vtJ~m_mBi%cvRJz1m?Esfy z_7d8lR~4%ez7!7s0BP5APF?IfcWLzPP5VZ9_Y!}c%WEq(8mPd|Kmg`HuUjewEU*_& zHx5BQL7CQ4Yqr3oI|kdDl;gK}!AM!5b}n)T20;0gRI;j#s63DW813B6dUQoXwU{r1 zv~YPoPy0sg($zwW*yYe?k>vB4)>kV#bK_5C@9a>yjSP@<;=;km~d$K~NL$qnwx>RT`VYML?>1kaAD^eWO*X zZx~ScNYvF*vmaWfL2&yWio;193XbX^sKATP)O{V_9@)MV8a3sKU?8Lz?eCp*;Z z{5Md*T$blJNTte~{aW#i<0rA1 zl{0Nzb~!A5W%UocN`;4G?m6rxCmU@usnhM)!9V(g%u)J|uYW8Hl~3{8iQh31~;cU9x^^h)5I;BGirY8WwbXWqkoR$^_f+E z^9-$r19seG%^f%>uKRpo4E8XyP0q>nDwQ=EyHe#F+NcHq`<#D?Z4FwIO4C&&H*yYf z`sO#OWw%O2^@l;#0_?}Oeg zX1d!o4eDm9z~^xO&{k@zM+5`u_vg>nWbEAzTH84#h&aH&67B%@tCh!cPk4&l+r6NL zx1Ko6wYpTdYl3@_%6o`uZB0cd)>`mcUywkNrexCU(b+>NJ)eMn47GZDO1dkZpn;Yg z0r3ZDRj1wVZ2NtjefgG{v9gP0w1NiWe0)MZe!X9Zx4_3D4eUB=(b0of+|3rfEI?fY zDeXBqIrUf2TB=DIJRh08qtvUQ77VHGBD$I@)YezNJAB0YZH-0VfyXC;VM|wE)9QLl z*X|wi`d6@lQONG!R1ajkMk3{ zx6^Cx6r8SkALc33Z7hKA!;bOYOW97PSmSMrfI;&zM$}bSvqzg41K@$4W^S3i9FPX& z_MCYWLc7?e>c$6^B4yudQPi@-i#z?{VU^MJt#PT>2Q@zg8MD+h^jXuJRi!NIxyp`8 zgX11&Bd1#{)NKda!*KYWH%rwQQl|>+o~3{SP=ZPS08u7dJ9J$o(@mm(P?8p&82Aw`q-|sj zq_zOvo_7K>oc{nZEcPy&OR1rVWH_sDgU|f_amvwKqWWN~X1uO@9DKs4FHOB)PhA?u zeV>&!>aQk^@UMndi5VB{YkqaMR!y5IVeJ%Ew;bRN)8Cy+c$9dWZMx%Vc(D#73{!2YwW>QrfUw%TvosI(|` zTE8#{{{RvH05LgDOKIAsqfoZdZASq?-;9y{d-wWJug8|-=kiX-ddpWi<_+w9Lk#x# zmGlv=ZKRY3f??XHTx}DpnB*;VsI2>snAV!RF115*i5gl#BY4lm*V5phO;-~+T{Vo` zN2X}2R~9}nCqY+Gr%rIZiO*G4r`ic05|&?OCuY)Tn=5|aoMe-yMbwRYtv{-r=2njF z#G`i*nk${qmm5(0`HC|l>d9Xd4K@XSqCr(-Q)V;fQjJa<8RA&7wfQqGY13$RTdTty z!dirriIt*naKW1<-`#OH7|Ic_afkxAjK4g<E0fHP)ZsZIco&W=rBq1rT+l&1NM9NbheM8 z)Oxp3Y^WM$J2UXs<8%FxD$PuO>o+F8i}r5&Rj$@q_*$#{<0Ep9{E0hL^*Q=Xex>_; z&^ik?i}rg*{{U6jkW{s7eJ}q2bN*)^LA7Tmuhf3f{?}To*7b!n;O4%aq)@1za6PN{ zJCE9G=sFb|TEZI96U}Ml{ITxethYXvuiGzFzqDVq_N#S`VQo5oO&wTDD*Xl*Kg|7R z_fP1WE}nyGE7|^{KcM=31|y&V08kVTzzH)qzioQ-u{>jpeIU%RQpW=s`kYT~QMfTA zly!XDg`IGjz^?B?z|> zyq-`n8TKfsF1P<3CAiU)#5EdkOrwM#t+(2>Y}DE>Sg-_w;1|?A6*iu^%>U*$1VuZ$dZ&L z&I`Wu@(T7by|a%@Te@{tUAm?O9!jwyKc&zIHF|`A+!iM!;%qLR8iDDm)}x#%{{VTN zs>KVIx3}dPobA)LHqkD=trZ@XLj|~5mfc5v$8Wfv6yvb@ncYLEhG0kV<|?|7KPsG+ z#t$EP&i3nuI`-1HR>AV@K8n6#mg{ztw+*gu58Sz8-&p+PQnAvUon?OVY zvT@%fqz;K|9n{-mll(iN`fjUjL-%99aAwcqYxK7>FbsQvlaDi6j9t%6YS*c0tYr4HkDjuz#p409*?mm%+yGY7et8^Xr4(Igk5o+}Ij09CV&)#Ou`ZFngt*F*- zR`gl6`6HZuvb0`KcPnkjEJic>OL_#R!oUIB?H6To@9jGmlg@r|4;vShr!?B@Hw*V4 zIlv#{EcWFYa!6CoIObCAd2+(4&IkZ)=ZGTMU~X@3eW3H19WGt$U=(fo=fB*_KAnK! z*V;LJc!W@F&!&JXjPi4d=e1Det6KTVIsDU1+FVc&?>QqpgLn|9! zb*@!0j0HFY=?Pe;0CHD3!5A5rj@OpGms{*&s#d}Q#sTglD|^>qViYdl;T^w8WlDWN zcnVvQft-G@wR%+K`XN9b)_JUW?PD%T@pdXz6#C3Nk-!oFEW>bR#s=KyKT`?n22HyF z+;}A9aXj^Azink-9C5}xb35b1KHM_q{6^a>7!8o1mLKU7`;>yC{^2=RQGiFE@he>z ztw2@;@^F7J{{VtA*`<_Rr7|}x-S+Y*3cjOm2f^nbK{RTu1r}MzY;l3W zF$&7=SFwjb!ZVOQk>yvjHqls0*F3r4{*zKOeV4zeRtVw%Hl0B?fkFNv7=en1z>muJ_T{@iaZqB~|M^(9G2U;qnT&t@C?)lmW zU@_eH=6j^{%{%L>wQGF6ZM$jb>oVx8R5dQzyHMnUcD6wivFPJ;cezM)or`v=KbLt-?4!{YXdfocHD!R}a6;5z7v=01D z-Cm}xR)Gp^sZ|I8hDOuhiOOB5MWWNDeVYp-B92=gVBa##8}?_!&LS z$JBK)TmV7_alq$~_k@hv9i1UB)Eh2xNdS6l#~)a(ruuY*)|6p8U>;BV!&%kuyHIIe zIRVZU0ObA5@zOA@?{0t456PV)7NbL?vs}nfttt))E1o~ZwfYoh^wF%b&wZ!3{{Rv% z*Zv<%RaIpOO`zkyF)KQql~?U2EPc#zoXtVAH|Pa*v)FJyg9ZadqCT~DxK*m7l203t zfHM`RsJT1YhVVV0^MU^Gho~D6mM+RU&x8K*8#%7l@XX3cQT!)^^qEIib{j0gY-f|e z`k%MNjYXDf89loY{UlmQq^MF(bD0gQYx)1*1L+aI+ zYdVtOVJ(8oO43gyn9Kg~v$yd9-UiDGqar5k_#6lzNK=d{KN;is$r`f&yNtFzxQH99 z4a<^87(OBW4c0RNcAmlcIFQ;^MHy08=q0pXyRP)vkUOjXQQr_JPpY?WScA`Fi17W{ zTE_0g2i1}z)vh-=1vHQz1J`AWM+^SpCJRFWQ@*#^GMi3G54;hwS zF=ydh_3$PX9~=@Cq2F#nuF;=kCOHL%6PViD8!Q}y$W!q< z+q8>iK5%;xIPGNzPpD-?&pU%>yz9qx(JNTV1ON;7$?f%zy-M^Az-=dK80|AR=WU9( zQ}xd<<5m$Ak%C9haXTHtx}87aZ7PlcB;*f?p{A{N){GgBJMkC2>%GBH#J2~|Cyk9& z#0AIzWpgTHQEJhFf`O0DB6^OWRhyc%*ht3{$*E&YZDEY^36o>kD_k*UX61)GOSo)= z^mQ_7KxP34<|pEAQ(`M3Iv!6XOtDEJnJXL)3_hOcPfvIR5r91A6ywqUtFiRT!Np~V z#yK74HFkwN$-&119}{a+Q)(#zIAQL=`1?dxrtj6Dz!f0c*z#o+LiHfaw$%;bmBvJ4 z@Y-8b)~)zqgOR|1w@FTyQq}HK7-Y74$~|qTO6?d6xTqiI6+>gE-|H)&HY%>%1CS=B zi>Gf^tk|N*;1bx{B04Qak5V8Epd9{_qpd>H0IjO}&f5yrJEOBvCepUR`K`k9-n)Ud#7D%j&HszJxVg#wneALSOft@>sxFx#?Aa=-eQs>e=7MPy)tKQQoRvR|v&VrlCNa+`_9Jjc3b zr52hMk1do3DB2Rsnoh1ibs!(_a1!A8dFMvtZ;06kSfM7e)HW{=d-FWqRd>J1}7ev z2lW2{n4UF?&|?fsu*mWUnf18jk>Jm|r{U?hEQ`0!+~*Si0JyD}1Z13jqLf}myLlt{ zvGEA?RjROhD9BHoaW`|3j~iNgZKduX>pMyHp z)#?!B1##vJSZUUcwpD_B;%0LGuaEJCMT9WHn|gOIUokf6AccBcCTh{5wI~??&e&%w zmUviMr8dSbpsE-31_8ksJW3kbS0sP#b(e2Ff$ift1MlVuqqZ_xQmZN67u&RUA6TKO z$l*?V^ws`;xA$16>J9NnE*>-ND?(FArwdgSdgP9x6DFu zDtO1#;%Vr5Cby_Eghs6!ki|)6A1b~_)Zl$!`mgOr?9c3LnnRwU)d2qhv8HhLXa4{z zKP_YEy!~dTKdjup3~0Y+b(*#Y&}dv|XQP`%U{{YSq;*eHMo~uczr#Zh!rs{{YjHX17cIoYVC5 z!qzGTVU-&$-~FG@*l{+g)77P~Wk?ixK1ul}xt1pow`NTQl~+;(trZk_0$ePx8H5SM zq1ZT{A{4GABo)TdkLN3ie)%Ru3IWI8G1?8Q!`k5+lY`3v*nY4z@q|Tk&%ki5Ff!AzmZ_>{T%Dd9=`=KyzNZ0gKAHi!LC84z7^RErX_wGaEDe_< zbCH3CKyi^J9T2sYX=_jhOECi@?Kyo$ zt9p_RhSB+P%7NxAy0!`e-sBoSqN*GTp zv~1P5ODhb7al0e=Ls}Ogfa*^tJP*=U9IBGa#hbGMm}R(FrCPQuquj@n`U%TT`6GSK zrC1JsgmyoqYLG+ZuKP#j2eN2~#20WkA z0dBE!6*2Gx_byKahCGsCrmZ(-pne=JYq1H&NaVo^%Wih}sO^^ENv@DcByRh7jT8%o z4jTiwJWiPL?Z+-j0vdZ0+Q*D!ZZg{ss&WGoFnGuImjr}h?AZIoH5#|9ah$gbKEE?v zVkYtx?Ml&!yCXf9%o6Kqxp9WtKZuC2OE2~Y_J0{CAJ!0b*LB()l^upXzsO2l@1tfP zW>;B5+P>lx9F_TpH|6lG7@j~L1jbdqjMTU+KZytX!zbdI3Z21m*m0k~%+~Y1*s}@S zPbP<5*K8zZi1!iy0Jv(c4z@Qu@Tg8sK1^3hK*04?nCAsl!&6(nI+p?T+$moQ#EB zf+uXTk~wkPZ(+8Ukd1Dj*yTpmMB@TrYkE=EuGOrnk0rCPeCA&LZRwTTA&RSknBn{nOb;BoOXO%F}vrYvsZx#v7~l-Bi2b()4$FJ-Uo!sLUVzzFwB zU3Q+wWYdFUk8?W#{{Z4@Nd?M_M?hM1^y)QxnTYn*SSpXMJ|Z^0lC4=fRjlLlDxWc} zdTSb^sj~DoA8)vp10CiqJtoagsY2dzNd?tFAEetBgVZ{ciyV4mVeyQDOgBu^->b5; zI62NnVa~j1meE3l0Aj|+bMO!IkoK$TYtTNWx0k-f zAhU8tWEgyB8Ihss9f?-@*h($p3h$?yV~B|=NX!jn^CCO zGi53YGBM6r{pB>b*M8FI1&Vpc$Ic^kbj21Z+8aC!cm7fnr&U$r)JCmP4(L0W;)TPQ{#`O&qWHgEcM3YuIz}+zCq zNI)J7@I(XZl74_k$Y_O1i&$G2Fb5g(A=Dc2y0yp7BPUEncv{;P1!Kksm~Ak&i)>xBI9&D|<~lUWQ!F_g zW<0~w_N7PeJiy(3XxLN4#(0cVZA!-u3#j(w9{&J;fj4_*O*<^7uU_RR86P5hsOV|E zf>uJ{3={q0`fXKulnd&k*rT1NI5S4NmiqRn#fm@S`%V^Vpmbk}sQ|S_iHsHV0jTJ! zb2FEGf(Qe-A9JDIbS15Tk`{Zg2+vG5bJdJS7wyfJ-PYg zGo04dTUTH!jC^>+&_}dXEC6ycF)6ItbmfN&jDzzTbxV%a>GFuwIof@sV``E=%oS7u zq&NoyjK(cj)7bU?WMW;3?Tlt1mxo=sBLgEoXR2=cQ%5MMA|?-hWjdbaztsG@FbN)W zFQ#C2zVB*|-zHmArnb~K4|1^N8Bc9rM3&;zuqaCXvCOqyKrY(U9!A#W_Jz?|zQV0o z8OJc%&Y4;Fl-N&|;7xV*%-Omv8nTq9wL24t4Q`z2wFnUHRXG4368d{rN(CbfSIF=n zEYw)~hjd48cMJr>ZCw_jbz9IJagp?duHd1RV|y=i^ojKPEEhOqx6eI^gw{25T2ibQ z@-yZQY%AzWEZ=c$-TsO`vIe4?cDpg!q>N**@dV@6BsP_S`Z>qyOtz0g>Aiyxt>l>~ zaV}-f#>;spl4I2CTCE_l{{T5S1IZp{Gm9C*rJv8VyXz4Urb?=DljrL*WolU2PN`dx z4pjU%69-ls@vfIxre?sksMI(3;2_*OB#@)QhRzStWq&p8{xT?Ns)Tk z4a_@Z6J~&Y2B?Jus3fmB_=syLSM;qp0Z1f|A~mF93laH%7#|{5eW=mV-|E)YA8N4t z+44WIg}&jXt^Gm0z}vKcBQtcCH9Cjdz^;EN?+?^A)q;$~u-&&A=4LdXOC+2d!NwqrHzREA5M$qie+kG`X&JWiU zw-!C^jO#b3ToTJQIv_1u2IpnNHD~HPedMON-K5g3fR--gzIo>nOZt1!cdw+?X$52g zJv~Us{{S(Du^lQ^UTRWod|)#k_!HNcC79{vnD%<0yNzFmP*3rXGjFHqG_?${3dM3W z%*WBu_>cv8MaMr`y`|8dFjuUHz{3vn&zB{2m`1jrL(@e|iV?{fIT6+Uo~Gg&g;e?} z#v)WzZKZt`0C^aMr>I+{yjNniXFMG7IbKYiW+nB%vVBYQqHp;h5q%#{YK5t&tgE2z ze9pV4_1hNlTjZ~t$NGksO|PM>uL%9{Jk5E^X5;7SuheNS^vZ6p#}jiyudeEdva&em zh%H@hrJ!%q?*p?h@XW^QO>KX;8(a=R9p-DcF}{{*sAhGrus_2HHQHi(natO9tmyYE z1#U_C1H9?_cSY1|^wj!`T^e9ZmO^)6_~*PP7c0@ZV}#y9P}Q)o))`0{PzMe4ni?G@ zvc=p{)RBT1WM?y5OVM;%ErVjU&4wy+aq*7=HK^)VXuXAMr3nD28DM_6o~$@5_VQ=_ z68a#$66*jrx2IS<;Q-Iy<~vtL>8t{a8M!AV`INg+uD{=1U{a})7;bO2I}g{HnbbPJ zps3ZQZ|aPMY-BbLJ8(h4`ibb6$n#w7tyk#UCbgGyjyV{>{XUQyj)&5CQ*+e7ch7&< z%nwX%r%z6<%WA6o$^tgNKm*1M&)0s`YHD_;QBzg7-YePpvB?CViLFL=N7R1Hs*?3; z1*qOPW0{t{A5%jUZkgSC242AZr$^H+U3Q$_spwK0J4wiFc^?8f{{T`7tOeS82nUdH zkB%mMVcUl~J1fDuw?I#Fh6tfm+(`W=&}pKreM;nPW#EN5S(|S|(HqJu+N{Tw2Zb=O z?Z=fNK_i3O4U`cy`i_Fl9TOfHvruLH&BSyVsyWQQaXJR* zHB?P)btS;uS0w)cF*5c2J@iaj-Ax`G9OvmRA`jXxXJ}VtJcGy91)x*{NR8;aZmVC; zp1<9LhPlqvQ}Lglpvn48eO(;~_Df1NeRo+}u^C0NH#7eL{yKY~yYcjllv^47Kcnk9 zfks61>SKXxf&+i`7|VWu!}VUJ`!V|vrTVU?@f!8zbjSl-PmgaaSo@OS7@T)o{{Z-x z*1FKPpV|7GrD$AjePyki>>vI6kFs|8nH@9sTFYPY7VpcPwfB#EXW)3x&vWxNxf$-7 z>tDA20JOighRrUJ9+A@A;p(A?cmDwVmVRP>oS*z>x>xL-R4Tiu1pBu-{{T>@`?h|< zbXs1YrqQbUN$vw1m<*4%f%-<iBQ$OdSg=sM;0*5lCoWmjw+-!EFFIziD?1pkA%}U*>&aV~d0ZL2)N9QvVvu)x zfgjdk+WxgbGX`Z`bHOL;J}y$@lkV|ZaV>AVN`=#LU|9bEIiA=j({4Y~W4oTU2mxXI z;2jl{^5lO|Wa+Uqx~HouP!;^76NvKaYE!&cGQ+VXfiZIH?#N=Z$zv#OnZtJnkt$sV zmef$%j4lx8YWah|SogWf2LmVlqS{?WDM9s-inm}62i{aVE}+CF(3O*C<~)pe5v@Lw zvr9VwfMc-d6Sg?p)6JH1*dL7DUK-3W;{aqr3s6^r%5%wtx`p)u<7oc?Nci!YDyXCK zWjg_k;Qak#W%XH>UqwjWj&K0L&l!dlS%S^RJCJ#jR9BYvorjRzC9=L3a?L0y^u0OZ!(Ws?{)+iD!r!JFtiL)zYo zU8;NdO4#H0v5d>5cG%cr034hMaYv-(!BdRmB%J5Dmlf|!)&UB6B;qrVV)94{rBJZ} zRflXgNc%->UsA=v?s(i|c9g20Rs8mubTyZC0q;9koP)E?stHDcs%(;}9f(Frx$FK$`J$^QW}hrJ}`mqYOF5Nd{>>>v>dr z7tkY;566jyS$3gd57_&N`1(gp%Vmrfu%*Wb8OO(IjPdE$i#=Y*^^L82r3lv6N9M@m z^z$+LuA0p*ol=&u!znF-Kr=&7x9PPOtMAxIVOZ>W=lV?R_14*`7=5*&;IaIoaeHND zB?kRC-nK4%`NriRGe)j}^|D!AgVH22?KmX;V6__aT_xJ1wX^)bapq;))UGU+Rc^&m zf_(n~DJ{CnhKokltF^a)2zMvigWwM)DOQuF-DTQ}^oWCL0D@VY>OLnqcTuraR)FpP zB73>>D`u|d1*=D_f#tFM=Az!4-iD7x71V8F4o+~x^#1^OEmv5tPCXi1(o`Fc2cN(6 zm=|i8X*XKKHv5b32RP3hd6`z~*?|QIr{#&7n6Yw=oBD~>F=xoBcvLOk8jWY+bg!|C zSmS6QbB{Qj+UnenG{XSkF>XJ!Fa1H(GLRJs8GQErBjV4TY~QR+GMlYtd$H{YfQ0=# znUQ}}3<@KKBH4t*{Pxv(9@F_u*3tCf_5wh_6XX$ z01yrWfG}ef=?6GP0FF53GYed=E|~Omm3nuzeEm#ydO%X3pxNW=G-{;QFndco6QYv)h@XYtQDCWCOIkVDqf^(bwN+C$zHBTPp&f9AkkT#wDvzTR4}DGdjq- zrUynf_xPOdt4P#4Z%pES2DJw`K`{EjO1EJVv)~>?;hq|sb;CWm4@$Sse4hq-{VTN~ ztDK+B_Kew3VY;z`JFHE^0kb48#N{h=DwQiI-dvyImM9w5F22*){1^t&65)U>eyxcu zu1YqCOb{_Ot@;aHLa52)5sXa69-gCMPIJ#QTD@IbgbRWj{{SvCGqq|Imp~Z*06DBJ z-l(T^I^V;%HF7dHo>Z7xK?NF1 zx*O5ocA+ztz+eV*@etGB{{YEq4TWq2D#Tz$^!l)Ysx9nc=fobrOG~>y10a&ZacLTg zD%5F;T&uYG5qf$E^u>k93^9i}++bC;49l^D_$*9+Ytg37+0Sjw&*&qup$3hUPKvt= zR=Ff&v{O@Vbjx0I=Q8I|6oTY%Ffd|0O4Cl7%ZO|dg#*lS-KmMKaH7!IEr6qszd8LT zL9LrCgLi4AeXMlZWl_72O^;*eKcvqWGA3OM1K^x-J-GT~Av6r(O7XWo4|%ec6Qc?U zIPdlo15Q~k*^*FXjyvXUC7zWa0uJnsC6w*c3)3uL>1r8OMn1DjolTBGbT9G=&VNWN zw2A2@z<}e7nR+#1r6hz>JTS(8S({A%0J99$`n#J+fsS0J0WhD6r4)y)g13CGB&N7N zIdD&44r1DyQpVZN7bl-59ii;)YBiO$yQ^F%>`!<#yEZwpqDJF@qA#F!uWLSbkIYE! zVK2XKrI!S%p2AvNXwh=GRSFS+z*cVX?NixX0vU0}GckV|RbYmx_KtbtY17mbQ%G%& zwfJ^9-Sm}}t;es^K9v^-aBfR4Vtxe6y_-_)wY{6Yp^svDn%3NPI%8VoxdgETjQD`j z^vhbzx`ZU;Z6cD1Uc z_Tw|0n|V>L)f+WIv%Lm$=MbeUwy;ybaAtagr>|+oN%JE|sWoa@ahlKco0#;?qfqUo zyJd}pW5M{E>ZxsDD~9@ZF$U*y_mb%P^xlg!xp1s!XHgGu^5Z7dV2CH_Sh8(sqPGL2cD7unAiBCVQ%g{Ih z5cb{2(jQde&1lW5nz4m{*T>zt)^ zrww5zZy5q}(|c&?ttgvR`v(jUW0B0hp5VKcEAxSZdmiFs*36!ol6h^!5i~TRkP^cj zpVUJ~Z4G6~_yMO&RO*>a1#ja20AnyZo3)`ekz>wHDT97~9x= z<*R3|`xv;nNFMIk82a{|@#;9`cUih#m#VtVl|&u0j&nEk-3qGf#f`_D#D5y>!Dwz_ zE4NVPh1P`%Gta~_%GEKAy6n?wy^6WVnRM#9MzAV?55&W>R)V6Y(#n5`MArq2r>Mih z#K&#TK7mHEn$y@y*nAk*;%U|>uKhZ`e2Mew?bVAEm1pP2dC}-~H>k-!m$^OXJHNK& zZ@!kQ(A`z8$i6X>Bl>+(sx7HzNgRVSsnz5tdhkf$$0O`!hy6UTR#L2cjiuD$uWyTETwO_NE-2ib_Ss!Q_Ae>;%oO{aX zT>N`xe$Vt;e*)b@s#2D-5NoTD028Lu>k{u(H5CfzK%rFt9DOk`I=4&en%ziU7N)9n z8e}Zqp+6kw7@0cF9>3L19*6to@+HFcx zcA09^x`I`fr3%1(LFYeQ#5%k6onKE!r=Yh+mHz+;W;uV;ALeGRr&H9@r0P12GN3-! zR4l;uBakuhb2v?B?H5sP70*K0?@`LSrz5!M`g1jUQ}vAxLM7gk_U7ABo1C7*Ne8)VvYcJ)GJ^G@GIPN3ChPFsnhO7T9tQ7?*By z!T9YQN;Q|#x`tJ#t8W|s05g08@i=;VF1tp$jYV4b+}`!R;ecBn;08P#@?mt{TUn#h zzNf1718INzN~kjD{DBKjou0j_*J;^E&Gip!0!C*iVia--2ki->zQIMhyPffFLK1Rt zWo<{mVlpw3M>v_KK(fJSmJ$nUVssrNrdMqXZN*(dB>+AClPgO>vei3x0NBPjiT=`c z2GMH23r?WYD?)hr{{VQzcc}i+-m7IjB9rS-&FWT(ky?)=e3)$y?PiavY0+$J=?a^i zwgAu6c$}TRb5)?xl=L%pqS0;IjDOt&>+v~WrPXZJ+j}M3i`Y)Wa1Y!5d`$;`U+pDb zJ6*kA%hvk0RBj-#1&?vt;{r8D?T&*{ci}a9K~Z%%-Zra`1QFQJob>8Cr%>00(`nw& z6e9Xe|&@Ca%@})qX#B zSmg8M{{T6|y*+>Szp0&HTGd;tupi1>*;mseg&#ru<<#~&zMYp_{{Z-@ZN;>|vN}Ia zYxaNCH7Ewb{{Vu1RQ_4|PE$+JHGN-H+WmdDYVWg_)ar>N^aqyxc=}F{r~c3BG%AM0 zTirh|7)ARm)rHEnB!o1bBTJ(p_WHMft@+2;b`jEZJkq4h7q?3YG2RtKQtBZ9kfVq| zB%V>sFoMq6?^6O>S{axD7q-KWB^u}#vocRJNG4F6$jp*YZv4*)1kVE`HG5R$q1MI7 zBD1+W5PU^b8RwY5agbCV-t7MX&d|zjtZtsQQpMUBmceWeF**H3PBj<-H`;Vji-1_RULzGso&`NF@K12(;hB#GTcp&(U>JxHs zfylrowjv*gxU(>oPPXKxTbj+q2vNrDafy1XYpe#)p;sK^6Hb$J zpH9DGcnjij%P-e_anmnt&gy+TJtgCBJi|Vmux6=F0Lk3FzLRii(xhX1iwcbLdra+5 zQl^$7ylx*)GagaBT3IVp=EVv=Fiv4=a^$6XHSW*Ge(^@?7r7)5GI6;4LRHn6k4=>} zj^`5cR|}?+)%(L)T)91m_LgE0HsTM{JP3fYw>#CCljFHER<^J(vv5(!9y2xNYoCmr zhp5f=Wih?Dz{v_BsnI5xcR2x%>SJijT}r5%6nAnv!MX)4xm!K(K0foe9vc4uCmu}q z{wjMFHwHP*I1-}3!)R_ho+aAU+7)Y6Z)jHd(zo?5?&@ z-b*kipNUWl(8%@zU$*JW;PmnGeOCdf7BebX`6S?E{KV{`*5aFk5;rbR82u+juj%ZT z7Hp^lgYB8Oty;Wx^#^nECpU5yX>DX{P_l(w2hS!YeKnxk;j+Uei)7+x7qd>Aag-+^ zU~JiKN;Wd>2e|Py1!UIjjH_C1tdD3TAfNY|I(ikW+~^2#+rbe!l_<5+%G_kG~3rNXhf?VStX}egFCq|4FD+NcNh^Drhn{`Q??Z5+)W`?TsS_;tnh~)ejt#vs* z^R6*gS$hPkEtDfYh(M*geDP^@F!dZIx^rK<&7kneB_Dq$ob(Hi70= zodv5cdhJXA#~hes(>o2El>>(V0BK4PUv?K8OJo5zYtfq>7ij%tI)rq+Ij(TKLo&H(mTv9lF-YDT!`zrC(cJ*j0jSB6uNOzw&O-Y__r z{Y-Ab%=7YN;_Zf8_T#N6TaWvb#4H`{N#)i z<}s$#+rFT@k(>rT1lCw}xTSPdEqX0>^TIH|GeNqIFJ9LPX#=xx!|Y~H<7?E`fyUM= z?K$%hI@(ilYQX0Kk7k(gtu=447Q<-31#&}p6A!3reNy$Quz`XYWDh*?4XV_;)*S-a zq#s}y@1LLZ5qoR76_xV99?g#7 z67y$mz+;awzJ@;ZKDx(E7-ivH!0FKHS8tK+ACHLH=f-o5T#6W7Utym}Vk?te3 zHmo48l6X*xGo1LAt?IVS+f%Ur09NudDDSzf3;|Za_VMSwBicei!jH7o z)Dw1^(`iC05lO5v{{X8bkB)w1TUGwj5~NPpJmMj$*4(VcY$HEeI(>ek&rfh-tZ=y) z24>4+dz;f+Pjb{18)oc#wvRs(Q%c{zQmhH%x&HGJbq=G7fX9#{`g?O_Y`~}Z`pjLr zMK#)kQL&!ZQtoq+-YVMS#MV)j1^^`I`c2#Fqo`Eq?YXnKAA>V4=!><0PR7M zELLBpZlhYxjzOnwA z<%sbYe5#fwEKV~9+M$xQdz;C`_W*m`Zu2@!Sh*R~>&gLDjDF%c>Fd>mBX@Dk=H1D1 zetxmN9<9}a6Z*?8KDtvr`)%nh!P#&lk0M>H72Nd_Z}u{I;Br$MnJ&N%vrTLBC>eLxcnKJji+s0JF{7kJ)rlR zja7P_7pr!}BlBaKk)o}ek%m#+1JClC79eWt9HH72WZ)Ax?Jc1ze!IqV^Ht4&DRP&XbXi&B6tnN$*k+&;a}@|vv3QmSaGpUM>D zk(lKg9X*H?>U;jP$)M9MzN3Ig#(Pbz9-B(t`+0J~7^UvQIoWbg}(WKMUp|qC>$jTc#iSv);G4(p0qttqZ8eKC`&wYU~xdO6J zb3KnQr(f1Jhf>Q**{HwOf6A8Ag#-=`Gc0O{SVQu`b!&{{YuL z_~vx_ai-|q7+HG_X0FxVp~8Xz)3vsJXv)QkJ+YXC(Gn zC3+9+{{U6jjUKmKXQ8IZv5Ruse0zJ0_=eW?U%>TknqVCT#W3FD%iQFAV~I|s*KX6@ zW{oUh!IrxAsXLbkkO#+^?LSRgwZmCrHO`Dl-hHZ1AfE&00tKnmdd-8SBkCT+s~I(x z9Fy_!pP7wC_iHtkU8t_Nr#VGqz#mWRGy2a`s4EKqabxW`Yz4<-{N{&4YAv$58J#zy z7FI%Vd>_03-9RG0&4+KH=ql3cdV5v4BdE%IvPaTm-BVRz$r#QuPC1Ko>(fbgm^+3j z#eGyf=Q$V{=g;$-{a2^kRif+=gl_kJW^6399AXg^vjqbr@G&F^dMUE|R=QE%!`L9_ z@AMy8X(RD=Xsz|#R0PkoFLRD3FZ*58x^9(e)oIB(t85l}kH{opJeM)URD#tN3c!LWR^1O$z*~{%_9F-bb+jE_~&?_wzIQr%~#9ZD6*Iy2=~_mK&LrXFd5KXTR2QeOsyBqoUtVvifui zwz`kvKmH7Jr*+%($@t5hrL3O7!-~`RMh6iNr`16f+{IY(a6^3J zVzmh3MrG^`TOW7Rg9J>;Niamq6Dye~frqce8m>%X6ktIJqF9+x+Etn<=4cmlJmv|E z60GJ|ah@_HR?PNgWX4doXR{+S(F4>08=Hma7$YRl1}oV25P!fvoJkBNX(7uwql8ov}%v~w8IO7@ z$8&cbgz~1tje}|U;|Jz-$A?kNE`Pvlpf!z*q5;S!^JC^5R8)GH0B0oU8J?=L`<$pC z@woYgBR(?U(saWOIc1*eh!Ku>&PX74h8RjpfrcDrQltpJr)ECc=OI*|(1_ciJEiRe z9tL?aP!$==k^$s%*hU$)jGxTI5o?rKst?TZWp`ggj!g^2r`oI&KG8JSS@>-~?#38` zH<_Bj>Bkr)V6VN3uG4}DoUd-_)@t8S-jbf*nD^!l5dx6I)OHx>#H~S3=Hor1HICC* z;kP{bJXHJ=sP`O$;}Vv133!dK{<4=!`B0StRg-ap4l%enqSuAtg|)&xE9 zVfHVjx!m9#-sb$r=prA7v3S%gA6O&NgyX2eP-t!x3(`N$I^*ODY;XQ`EL;9ciNnvtdLo`=Odo-Fr<5d zB6{%S)5ponwM#dXkUI$janCYIQ~V zSk^EI@O#bRNdS?Mf03530^wMYJm6+x;O~zujw4Y`gOga!c^?oq(Or9q$sK^i>$Mss zu?y`RyLRz2ZRxLJBDLHzkEopNvb#*>j$v!9-MvL16T;((Cqr8^+5qJ60pLyBI$?6y zT=Bv9fE_CLa8Ysxe}gabYjKC@>ODdO`$@>>v}F;&+E}?A$@!a$(HM}=jOQ54JzX}v zJ=tU#=YifG(Q}cIx_8)0qLYqP4nz%|ChJ^aF#Gd5+NmtBu>nGI3kcfj(AR47)aUOy9UhcyE)O``8yLiE)3X~vQ=gIkV-(Zrh75NxIVNsm<)0?w z)gVRM03;r8WVpfl!*+tRoKBi>*=7VDJ5M8Ek&g0SLidOhJYy0|3`*uzjFZKn%TE(1 zObX$lRieA0QK7PE86;*2Oe(}ZwpE9*zR-Y4#I@2bQN?SB;xRN3qnz?$pOVd|@{`0b zs8vW1D;xwQn2x%!W}gBdc_W!{7tgb?+ndih zhNjRK#&Mm)7?p1t_NZEwwj1ByB4<4T@XAV@cEr;FTS}u0vka~dIg1KNcT!6p;eI13 zu&ykqlW^Uhaw8kb)9nPABRs8$RZpC8DNc&_Vx6nbGoJHX0qJWmb1>r^XW~sReO6WC z#XJ%GQL8Ha6^mN{2^bz4kCm;x(s`^ENLXbhoUD=gQ5qils8nSVM zILs{_GO8T4p=^H+Te15_X`8!r4O&}T#Iyiek77A9cC8w^GipxeU(7ow5ELqUq;cu0 z`;L1E5w2?oq{@9YCmT=rnYV6|NM*EZYkFx#9Ci!;06(nH>Ka9@4?mP;aoZ+7{Yfd& zimF}N<&H79{{XOw4C}24a|H+IoS&>Q+PY_<)U|BDHV(>0}cWwy585kRi{$ljzWIL{~TH-80q;FSyT zAlpy80;&(rdx`YE7sdG)RkH3m8+hcHwU22yGorVpf`wmp1_+RyDwO^eB=RLVsPl{j z(^%jh;P>OSGW6>dscd_XV8bKtI&D6ht=mIjV2z{>-$^H=>B$1Jr01Q{_~`Rz)1bzT z@3j=za03uNV&kRXM(49&0OQ-vX4>g2-Rb)__6wYhLV6;)_PGG51xg+cJP{v;ACZyx zMs(nSN0%5f$4{&ED=rDEw#7c&O$S|4qi)zso!dy1O+Y9?LZP@A9L(MbXDc?{E~$x8 zRz`A%xgYN?y+6~m2VzH(6yqOxyITgWQq)oqV3)TE5p_;jcKzTz@NM+98ILIfDyfNx5wjw(OAZ}Fyl0Lt9`Kj$sPU>r`aQw+`S%1cSVj?k!yYg5!2bg1<&xQyYf_&u?St<$>wj6k`@#TdwGg>Ig`Eh}X>B%B*DoDU{m z^r*UA%2LJv?;HjL`c5`C?b7QyZTgbBjRk~r!w&O*+Y z)Ot3zb!1znVtq=7=2aef9})U>T8$MV)&(B6*|qrNzGmBN{{R`J^(w;_8Ff9x-Z_0% zZY1jBqY^tu|~+Q<&>{(t;VX3c4? zi$>BbbCPq6&wmg4yu6m$-m0A&>%SRQXqDAgYfvz)p6fG@;dMP-(z<;$rza>d+q04S z%;VK9(KY&2uq3XD3gu>pH51fv>mIPDiSk$Mxo9 z>AF^@U8&Rc9Ya`}3p<6zTjKzqFqL%eX17CTvi8YjWK=!W;~6=C>U~F7MwYtks^3O{ zp5RIw`HeL)zU&ZDjBsMZZGuo|o&?F<1pKL^D1eRoxL zRnU#YazQ}Bey53_tkR10z`d(u0$4iZk@g*;1l3on*4(t{g^g(EaXgdvn{@SSx<5t~ zF&~IIX54vZJQMaErmYH^FWJrPfLs%M0KBRC{fCdN#Od8XMXI&6H`Xq1Oc9g8WBvTV znCWg?*4Jun1w|zn9f4!X{$JiT)w;gzDJZ2WUMo+Q$8vndZ0Uf8y{-QM<5TY1I7K7R z_n3OUX|d}%QX5aVwm>R;AFg0d>1%B=2f2l{H+61R$I$pZ_Kex9(W57(TqX z#LTn)tE|#3i&;wR83=jF@@G4y)z-~@nw&cyb$f{egU&v((zW%C7f~0bl>o>Z8O|z=&EV8|8 zBO(GS1ODg8&w(?PGQNYYrl8BH(~xMmZ%lR>@j3lh?VTEQ>U|o!PKQ>es?$>0ColQz z{{Z%aHjf!6zA@EX=nV#zsu$B#7zjPXA36D$TF==om)CV-?OiS`MH#l2*xN_X&OXyJ zmn+nIU+lht0@2s>r%#MJXv1H1$K;X#`R;h}G9Opdw70MQH$YO?!2C|EmtsCZ0|0)L zU#;l+Uq`O(K8-9u7t?2E4%s|oAPLOqG;gcFeQwWaZ(@_n5$0?+`cA0c`joQm!5@oQ z_g{?o1jAXX*L3|g)wb%aF3-7!310<#5s!)5{?2p?^gTMK>MWHtnPl7to^VbwKUk+# z(P(2OF&FiKyvGq5yWzM{$mDeqw&2-=nsaa;nYgn+|wY zC%4zMXQ+P6*V3r5#;ocvyFjcQ?>&Y*0sUrmZGM)Nr75psjn+Fp4t~F{5~$l~G<0du zHAc(ZD^^YpetyH|YV=C)U3Q+T?q(~71UDlc&)Rb}S#z>l$JDWb+@-%D&uP{44Kp1= zYE5g~86!W+bFIkgI)nP^&$&**2OaWzk@Vtc7L4Du5UT{pJOtzQiI^1UGU_E15@FS3 z9#M#O?F1#mNiPP{@~tEpgLp$I(m;_QiDF>Mfk~1jVHPkX31G6yuz?~r5G8Uj2tC1) z89^i<`edFZ*(FGU+6gRjNhh~5>IwUKl>$tX=ZWtEO0hs#08TN;%%}v9@eLikMofZs zf=78d`5ei}#~`1il(aUjl<%m3eB-lyY+_wNUc)i~=awJuCpkT_i1_r+mudo3 z8tCEAU(m$N*AIg8dt%E2A}uzJfqN^%Z( z60};15CB2PFp;ZzYg1NKyJG{(9`NBpp;F)s=bsQEhyyFeJHQ>pZfxM3V>7(7A5Sfo z`#HX};2(~2OpV1 z$jm~U$r}jde8&{;K)Z%<{HF1{XDOxA3UUM{3aR{{d}o+El{h3G$1s>t+@3~8C!-u? zk1lywsqdaS-~la?dyWT*;0E5qC(Q8gI!t78xmuPRK--)X=1`nu4${w(6z8$+D~Yx$ zv&56%5_!fvPh;D=ILC;yw}-bR@O%kLCzFzMDqYEK?qvsY z@H=KVfs=uO;7B4`Gb&9~@T{P&4rEn8_b@;S=jbCWicWFNiJum{=r3Ozc*x{W3YC}= zxnMiO5$?>8WufI#iN`&p>>zsp!0{wWmW8z{4+Mb<3>Y%N#tO8f%gV5bEW?>m2;-SZ zq>~1Dgd;Kv$pl2B3`3RrN)?$q0uc;?ze(UAwmO`cWie&q6tNodMaNgd99AUA?kU<`2`)3E@zf#w$O9d^X)dKca5 z0l}GTwe-zXk8nIrzTLOd%IXREGoROV??HE}oEhZH9ao}V#~H2D+8B!XSi@8eD%&y$ z<0d_NErGJ1mLpyJ9jha^AQJ?PB@CPZPk|KH`^XE(K4-0SW~gmB!SG}w6fCxZv*{G2 zd&Oa#9($SXPM()d#S|Y05&Cw8;1USIA4ok;jTi>6vvZEomc?V9^w-uX4$wyg_Kmvw zbq)Y0;K+I!n?o?b`odak+6Lj1$m27%_NPx)qOg`fe0P;Mk*_tJfZTw4edW~eL63}m z;+mzk7`1Dyd>{KvH9{xx!VRXoLdaVtw|w*4wJMHICS0j(ou`9PhwSXO<)UK`py(A+Uq^ za68V6Mu?OFN^aUY+B>(w%(dEedcS$4UE~m0_>RSw>6xSI2zJ+IOmwxYQX6(RYy%wj?;YvX zz51=@vOpVHwsJlU8wwGMrk{6_?*9PP+tSo@S~IC|MtK0AnObp{du7w9w7Djr0USn= zUdC2hut#&^FqctnWqlE}daeLS+H>X>9hrWN>KxUod^eIg#&h03r0MSU zgnPkp$jIgXpIPedx|;t053UF^`9c06^9xgHls$Fc$iVIZHb!Nmu-8e{^fsSGP1SN* zwm8q&{KqS_Bh;=esZYCWpg0G|$o(O!)!AYI?u4Q4kIDTb15@f$1FEsC?JU^H{YPn& zC}~%xacY>F*$D%1CqM7^m#I?p*{RWwO#2&eb^id!e8y_&wb`ebT_aVdc3_0tX&|<8 zrF?z4>}53?-By;N>fw?7gVXnHIsVa&KU&?WVYwJD2EJP$dC!CQ#Kq!`&RRyz1@7;s z)M@Ju_z80Ij64&L$Lsa&GZM$irQA5lEB^r4c%7K8jcU6g*|~AKd*kgfdWMf&8pOYL zr%vABjCcda4{(3yGt0~Tv)p<(`Tm&xwy#g9^z`gSY;dC|Zcm@>AFEM)B-(!|J6XO@ zk&hB@(-){>>g#IjS1G(PRq%fQ08c*?F*T1xmQ{U$)Sf{nk@}7&gOiIaUaWJLI)t*G zp;4qb+ugTw&&-O5(rarY6;ykvB!UN!XFB$cIt#0~J0QrxJAm`Y{{Wc&ldaLzSJJ+) z4{%gq50VV$B{xfTr}dXoRrhIuBCt5gGnah}vQ*hV@>d*@z{&1sQPxFNX%r)|A@bY5 zhd&3*=Pv3#h5BO3EvzZ_vG$+&#QGdQwDI!$W9oWsn!Qdb+$&o)wWrV6O&vH8Hr@BF4WRn_bH9+*=V&AE;A7Bg=9R9Hg z=-QoMPtbbLLYl}RvHFwx%|4%?udC|m)#|J(q(;Ls{QLv`;$wESCYr3*P-0ji$Zsb< z1O4MV?Q3+Z6G})6Rz0dQl1VvKhMTGL)ec*om0 zKl98tP3by&sIu;(?j&)x%DtI~9>I91vZ@xjSqd(!p9Qu99 z`TKn)PKlDWuvuy{E)*7A={EF=^){0G z3BU&&XNV4^`$8;DZiiSaMyr)jz=j;)Olh~ze`)TgCWIr{OB+cfl27%7{g3^d)m_x9 z)~Qvk7b@+N9lnAu)O6KdYS}=sJ$x^wSTJVKIP;IVou}ij=-n4jrs>OIYO5`6YMX%1 zcF*#}+>EQK(sj%F(4>(?TXAKcrsK5v1Lx~A`nOQj^tOLQjaJ?^)TCu`Nysc74MAnwk_^#J>xWQu9^wOOeNWyJ-fDqQV?>RM{` zb?mOb7!l76pN>0mGxa)Gt5By)5|L;R`s$6Bx7&uAfAr zArgXj92d_A={mY8T^65e9nJ?WxD z9%Wgx^O@xDAZDS=Fc5WihaI9eqj4Y}Wr*ffNeCn!(pR*H?IasPw7VW<8b~t+(8NKJ zB&zp@K?J?wl1OC`fbj$jkkSGXDpq@ZjnPQN_9u%-%o2Zu%I14ArYT^}@^ST&4u8D1 z6eqcf_yA)ctbu4=$Uf(jGq&R3U=r9E>^ztv*(8upz|4|EF=i(wRvJ4*_i5?j!OtJ8 zeKWP!`JVzu-g_c@Bkq61j9phFB;kHwtz~-7dBMOQWReR3jy^Lg=r)s(4tVBy`6k|; zEVWp+(TiA>$pa(l6}t5!Boan3^qRWbPSW5IKfJ}gaDMaW<&TYhJTvlxSnF<>0H8P0 zEWab3aXnLHw?5JGWY#dpXe9SAY~6QU!vuh;KZ-IKs6OM!=i+0VkAhF!#;9f>_P`k= z%3XFW+ekXKK%VAK?5Nr~92kWcpHz{8F~`JZEGM6?aXRCk+_J+-Wngy^#L&{Pdh$Ty zfN>LY#d+Wu#+!k(9v8A9iP4o)DzE!(%uuv7#cr{|gD4l)i1o;(Z# z!J1~Mc$Jqt%4ZP>G6#7i&j|&TLa6NqrW76`7Xm?^6U4}b!Ucpgca$PQK(LA@fP_&Z z@`yny1h|4uWD_ScgvmT)3F+x19@3e_xr2d=gDwmRJi-GaCS>LhGugy|>nubPIE4fe z5CaZi$mSWxFl0p{dU{qt=9DwU%9&`rBNN$?kp#5xG9^nuiJ1qPP13`eBw|<&WrEL3 zNo}BlPng);C<>z^CI=H z5gEkEPUHZQ-*EFP?_*cBtM`0~tEp0hmE1g$;vr>@M#_+2X5#4P>=YBq> zH>JF;#daru7?t!dd?*KMjIrz`?4g;{+(F|YO)aNerZy;G4sn@{a#>Z5xb~B{XPi!< zmtrarfKCj<>Lb#*gAzd;=5Gu1DS8+>6$5C_LCn*uN~V&QjGTMEvmtO(au65cX2E0m zhCpi1f0@y&Z%qE3RUW;Svw(BLx87*oWmFKgZETNrN_E zT`^Nk<+h!tl3?{(+v)V$wE$y?#NX1cw@gbi2LyMAuTbo9VvPBiH1(_Q0eRd!Lw+Mu zyMN*A-S&!Z*eS4J)FADx5k=Z&EPE9OC#KbE3WRMtdyjG@t<&6ulv{E15F1?eVuNjp z`5tk|n)*FJy*DMQ=Na0^na#L1)HgdTkDkUYP4`!0HVET~5l^8eopt&nU5MEo)Q;1X zx20=zTJkDZbFQ>fKVNskO37rPBOKyopp#}7gODNUHOVg(k`FQv`_1WBv*f1_WEtf(XB zI8oTb!-#Z#)KZ$CO5Iot7iQZNE8o8pS54Hew@j^1qhJg}5c~o2gN`O2Uj2`^Ru@^* z&15p*fO5pnZC|Qaajw=iS}+)}@A^VReZHk%QLO&}{aB*FKP{Cu2>K9WTHO|tP`;*! zIMgu8Y;aje$JBpm%;@!-wEqBw*4`>n`ADaLezQkUKwnH%t4OX2S2m+nqPkqXNhaO>J+g|RQsMYGM_JpBG+T-$oK1ZJZK4xv2B{o)<)oRv( zmpcK+&m)X|%<03optjXQHD|b7lb^H(C}(?>8Czx@7lt_QGvaxP&BAY6lSgNJ=o-f z=RR}f%neSfE3~-7`@N)3pPm|e@yl_w>o&RBN_)xS0py9x^$OmrE!(w`dBbs!v~+7f z4On8q%J3L>QTB-Ht>1=H*mdVI;s$i;~;1dk` zy0$3_c^r&nLEEfW(&D|s%;$Ibp2wDonmV+#ja7-g*4(;|;bS_R^fy#$sa;EVC071zfxCqt(=x8;&Zn> zn(}`T+kHRvH#qShEl6Z~Y%gFLKLEmX{T7x>kPsI=rqw!us4HD{jg=P0?n|Gq^5S#atv}&Rn4g(exU98(gb)%n}@cfBnDjI9{=+R?+?>>K4ncC)u|+8SZn*IEw4K zg7sZ6Q1)dxM`OcqcmhBBH~TYxQEG43T%$$5^N>#OxAMTuw6@m$hUr?%SnAfA?NeW| z>=+5pGo<~dx2h(B+9;@Ow+mLr1OeEu#J^Lq`UZ(V09NfKb^EJ04~+70^qK3{->YCSbkcMzelG1!4J>glbcV8n)`xhwwwij&9tPRBt=)9Dnl<=@@gkPPQ* zaLu09@J2Yn?G|G$?X=^A_An&Om%Ers$1p>@Jz_&$M?e6)_JmO`E5Yq97nr3nJ%~-S#UO*sg3{$op;IKL z>Fv@PNR^Qg8E{k_k)MbmJ)B7hU5PnAr-|UC!R8SmrM0#ngC}8yOzj+*IEnzF9OH=$ zM}-{1xWIujNbxPZj9`87%&ugDJjkw&2XsJ?J5Dz1~~7`w&RSEf!m3T zj@!pA0c{Eh=W!kvw8YhEPpEx`fAKyiL7$EcrP^V71%0?b#CQUEd3d+m*Tu`H8F*}C zXvB=!QVqBxf=Zn6Ivcujs38vJ`pnz3EkSTAQ=Se+XO`oqw;bgr=Xv1|&`O&y011Kj zmVGlXyBLJ@n;}P@!edU8D4jF5RAT_2nED(e27D9a61q(hDY?7-W5-C#mjfdm=JCSD zGvt&y*LsCsezQiJ$Fv3jcFBn8s5{Rc&StW)QNSJWK^esKamw;?UHp#@k+3aFLQ6xSkNyM&YvN(Lnf>Q`PNFk3hnU+1E z<+398g%Ncy>SULmo}Q9;nNnoWVnrB|1dt`bkO+ttjTM4)g&=1VL&$N6AYhDdGuz;S zp3IQS{KgUS5D$oCw}}%+m?I;2$9V3W?Y|6zb(uB*HCOpABOo8yZ$r4{6&n zme#3*FL4{MQu>AW5`L2x2T75AZj04j)sPr-IeXh{J9?O5+;*MXDvFN-2ZK1?v7=Sg zGbulDJelLF?~Uo1WesIusa4&}e2BdyR)iEeEsUNpAuJYF!NCQ{7y#l6RL*H3lPpF_ zUeF;~bwP@NNe9P~EvKz)zPoX_jPQ8DnTFd?vRJAg!82c_t%~ZJh1$)Y0qqbDuoB>@ z>?S_AGwKyNJoX-D&FFWWaykB!6Ip8uzT-cv);G0%JaqM~C6HjSz%l*7P%41g85_PL zp{g|n$j6)+YFSwj5s*n<_~$tjq^lZa)Dur$7juAE@!DqT5}W@3cafABJ}`bJv42{Z zzyJg+K=Cm&p@!=Dz&Xh0GjXL)O0-seAjD^aKhMnemaj(bma7W^8%8^yGjg3suAp<0 zJK?dIEmhRqi|QdOobLyb_K%V9`(|}R?@k5)EOY$jbh?UcWfi+|&eP!iB{j6)bv7+P zwmX7E2TiMDGwg5to8##41Lma)e&>=th6y#=O95(;SB(3czuPfq zs=Ar4C3ZJE0|SqyW~SNPJi02c_pxKN{{RP{)Ihe1yB5HTp|A#J^!n!9t+93v0Pp;S zz3**~4O{lFV#hy!w2q$A=+3Wlpdo?yhtzac$0PWhfohm!lK*+{%G4(x7uVSLY z37>PP=Z?|Q8pPO-k>)nN2S-@V#2kG1nR-sHbjz_=mdBGlcU7|ln+GyYsycs8(KA}8 zq`+C$1$Sgtm>w}O$5m!R4o+f~v)yqdkiSVMSl_f&5I=B$k03_0{{XafpJP&(`%XnD zt*qtS9})WWwFDUi{;|+?Eq6=SFTR9Ae25#mA5XTw+&YCl`I(ENV_xxRKX@NcP^Uht z544che$1^rRE!jyhMv>@%IoUX3v0PZ@!}kH?M)SgAP}RumAm$kqN|HCDbQ?j`1L-g41H>MS_P)2;tPj*QYRxxI>Qy^eXQu8s#~Ff3Xu5W# z0McIA&Q(ixj8oCNf)nVKWj)tC8IC$t{Wj$^Zeia!%w>OQZqnPb*4p0($d-(!>pHHV ztFwJi>AK_(h%TX{UZ?2YQYP%@cH9*|20x?yq^`}WWqY>_5=1*G_t~c^^!H)pOtoB& zr%Cunv+)*IUB4&?ID@XU8b{P)7-T$b#i^bY0c?6_O65p_={CQ$z@`r$bGImwO1o2e<_G zoaa!|Dr$0pN)^L13}g=@oM#h%Q&y^~LsI6|UU2y5?mj=-XX|LL%9}o-SB~X#-@qR| z=MG*i&Yf`N)pD(2Q?YjXW41qtMAWf!Kn1=r9J@xeSoCi^ukhoK=6U1dDSfY|uh3Gs zC*Og|`V;UzJ7+V&WtHpR;C8K|!RZ_la8LQ3zOweRwZH+`_r$rlT|fzd+0P#_X0)}K zCk^Iz!!5@Q@zozgLtM>LFv=X4J)@e9V^>Pjr*^`4Q{2VU-@dG6^y_gL9Nj=P1H>ayzfR!ok$Hb1UaXy=an#*GzL4w$e z=+Y9wYl06acZq8CI$JfKucEsEBLGa+ybAFCD8a{`p)YSGfZLNE@cqi!^Zh*Z{@4)dzf^lN&Sj@lL7ZgmF)dr18c5_)Isww=HE9{kofHDkxxblov58g;UgsINN)!LdxN?4mIsedSPc|UxA zc)ePyIu*3gTh7%uZSXA4xxth-jY9d0`4B=0RI3m}Hn$^BrRm>NxNyp(fzN-bn|2jbwR-`9J*U#0 zE}2(?#9+2yoyf^OzK6W$w!H+$lPQ^2Y?HY&zz97WT-VOxlyhXQ*rd*lY@NOkWo zo*;piCn5<2pqR_F5t2kA_=A^Ih@nKVGJwx+kXlZ6@WtIkb z$QNXuw1AD@A8te< zfP@xd^&Rswbz8>WsOgmjLF2%SH7H3XatQ4lXDz~=tMnY1M$XJBB5}NXcbuAMogg0I zbF_9MS*oa^463B|$0jxD?)?0pnc+xLj&|pdyvM}ckw z*vvo#N^Oi3`bun)LWn>l1;o6NY#^9M7(p8Zyn`@ zpIEHAnLue=pD>YhI55GYE6gzf{o&ZaHzp}j3W8J!MI1>dw5c-22=|g$7kgfz z6%=-rX<^z`q?FDK_U> z7}L^e*|e>^WS(a))lhq?Ck#7%XH}`x`Wi(G*olL7nYxXQg#fQ76UU!*>cwkAirL>Jd>HPw^bhYz?Vh#jtqqpQOn2g}4Kj2L@?Xs8c{)#PC-ncM~sA z$4hXtoP3DEQr#h%vKG7n+<_9hoAl*U%!Nb22an}8snvr>FOmSxK+H>1V(P5)t0b{K z%;_?$4!WOF4Jm~zb7K?WOoZBPv1%1(nuw$mE|hPN3^UEn_&7JK0(T)2VB1$|G=h z1MWXa)uMufpRO4Ag49y2NUY$wJB~cS>G6AwI8ys~-Hc*7=}x+7RadP*^wg*v;Q5*g zUW5f#mL2o-g3}a1w3F&5@b=;ydz@Ow&*l0~SvunxHnsYz8nIHK!6)Qn=4a}3wpOZr zK*Z$zP-i#`A(BI*&t0fs)kAf2jWeX^$r6E-keUZ?!@FVRZdA z%g7?kXX0)x8U+Ni0~If@6I+E@_VoPvKk0SplLfcG&R|ZLb~lUC;e9LQ2sWloz_cDTF`o62B zUTb_4GcVD;#UYPqo~uCHLxbCr1g)vJrt2FQEpBCx?EU7D~b`puXe~5XUFx2WQ#@0K2^Gie4 zEl1kSMq{wiw@-GHsIKH?rG5eth)q}8e9{{RX2gc^p>R)WW4 zPlMkOF}T&O`dg!wr>e)|`!INz?Yi~9-+X3yL5NXc! z9+jhf?WV=gAg>@9lK%j++SI?+ro}{LC3D=Ie{9a~s?}~N&WMAuk~#e%`tG4ZQZZ5b zXUEz)X1IQxsnpY`TJ<)gInGrFVdt6Ebn8~@k?L#Z?$pM#Z(kmScS~?YFyX`9G}+A0!FAswn!QSUZ61-$>P+FQ-{R4MQvACDh=7^EWv^Sh#*URUsEUvd6rjWJYbL zb9c{-Ms@mHbmaPWI}Gp)%{8+Ea7HJ2+m$jkY>pU_v<_nS>n+qU!S;_5Dbl^IgOUCw z-%aVXBCFm&?}47vOo!6?Et>XIVA_qIbHD?+GpH5x?ww+3LJ|BcwK*m^`#VKtj_eAp zM_Spu@&59fy*iOjpo6&zs)BRQd%~8DdQG}#)M`RgQUmGLfwX6~eoR5SkNrbZl<7b! zpy7rHKR*%9w2NgMYki2SRe?M!A0}ebx_uUzEDLAuhq-^$-#=*?8@AV|(R8vjx7f?H zoxwm2^OKTdU$dIT)alx-LE1SWk-_FErRti_s<(=l36D*3J8_%{eK%2VkEF7Sl+do{ z35@Q~oS&RY7X6{u`h7;Rr)`yn8vC?s`u_ zT@Hu z>{P3+2iu91^zArV+5ujnHV`@U0{tIPjGRLrdG>PQzfL#Oe+p09R4E6MJ0HMll{WiK z%GGFs*jz6Yq~^x-=Xh*oqB5u?aV#25u5*tNY}`(pa`d@*<$gq5vCNR%i6g4~!j42$ z4r2;cY`KKP13uiB+E-_CVD`ZQJDm+D##0oB;ZJ{44l9hp5A_7 zgmc6RGFkH}n5BTsHE1&go#P4GL2$BXrAtXIx=^&CXh)5(A1cCa< zA&KmAev+Yz_Vea=+hOB*!I?XWAynbBpQQ0CkISFDq>v@0w49#!{REGsf!;~v=d{1>K#2(+s_rtV;c1{pbkPH`DQffOb%AgqSZ5H=Vh@7^x) zhvo!{5V6devoZn%xiBQpzI#lqNQ;v4>Iq-ln_Fpg;T@;g$=}l0 zu~y8C62)P|aAEXCN9L15S8aekNid62jY&{I0%HC;<1ewb&uY@Fpq0mYiC0mvH5--i z#Pc`rwQQ|w9ghUe9d4UVr_=!KPDdaakG)26FVwka%61OYH_(^poY4ZH)7 z=Q_@pUNjZ^M>#zAm>Se3+Zp|*`_6aUjbv*mKu;gL6DLtcmVQ!IaD4WUz56j(R@!;b z%v)CMLi4qJrhCLP3aGe-e39qiMomJ^D5!$R!TLfjm?n6Cv+Jei=)rkgbP zY+c(4NaIQI~?~AM%FeJQdqxnI!3Xl8isv8rGbERxOfvUOum)=lVy1Qsx7)ky1mtr$jHx_3pCcHNKqK~^*E2|izpik?gN!8jQyvz z7dG4!V_(OC9j!2|cX72&;yz4&rB(v^Wp>@%=k@;pc&*-@kNdH-IocVx!TF5o8Fc>u z_g^dy2?IacH_+E@TIe*RmczS{aC`wJTh-KVLUcLMh4ayP26XgTE7`{Z+59vd2bLJ*0WBlQB&<+8|0qke`tlehP0i; zs9+8cAFQT3^t|=$3DlG@-_o1kN9rfHl03}@cu<#(+pE(>m&Qb!P~Da7V*dbG-%qif zTANQe<1(^T=06Cc(yTtxmg$;N+M*dXCWx7>e-daq~IZrnXj{c^0nlaod>b zwQu(*!w5IfE~1^u+(g3Jra59`+UdEY*FWUvB5&yWh-wa>QbAMX&Nhuht$>a{S+z!_ zT(~?mU^lJMdMDP?H?!#t()x>6UD&SiAEh5)}nK)+WNG2r~j>Kb*^YLKFa56=-h zbl2&$y-!TnDJgYhhF>CL*G*ft>NU0wbFhqNMvtl;F*UM&r?`%F4J!2(r&UlTe88-& z(?*iPRflsAV8(Wzh??26U#4KniCX$ZN@)Rmm#EpCl2?=z3!%+D(kgEDMeK>^>$Ip%m)vP(56s zA~;+0Jw#KQmiX%R!`hN)9o5( zQk5KT3P>%S8P7S>c-=Ycz@R8>n+YHff#NG^U_sA$rLJGz4Yd-Xm~Z&W@iXihTu%;D zRqDp>HFPb$%F7U6x6WhTO82VQsdzZsjLve6MUi0T^0 zvnP;S07e%_W%h)@G(YpS);X*Sy9+E>TxJL$sq_fP28z09^( zfQPYcZ~I0TfHk8{sRp{2n*%o=&-9IHbbUc^)}5%`!dgG9qjbBice=Or&Q4W4f#WB< z(9@u?>(uG@Pynk=HiDzD$2geXQ&iP;S}JRCS~|I5*pHbM>aL-#t=G}G9&2HYjCPdJ z>NM8AqMNHFp93s=DDWn0QL)xE=cre6Q+ffn+}Y;<;PK?lI+e7xdJa^TPi}rb@l99b zI%ctG&idPm+?;|wB=t}piuIZPAPD+m8zX4-H)w9KetO!^Q*y8z{K{S`)iJkGsv-XX zGOc8jpB<({>#JJNJ6VtA!5AMCMRnD^NGRLuW0&2?A5WQWK=dfyuG?=S!|uuMWBQsZ zwzp=^7qKTLzPbMZ5ofp6K##RHo-jNA0KA<(qO)OCR%It~4d)&}nXN|0Md@0NZR=K* z23vxBPo-&kCYz#Z_g7Or+j+f*pE&;jo+k^U>(_Mqsq~)}yYf=LLGk|pIG;t+YIUPY zW%XDsp;sF*+v)Q$l-z1{Rnu!LYOO{9x54o#_$@W=9W%LxK_4+4W|5l9u1c9(D#1en z7wCWI5;|SB`nHP!L0LEr-#-K9c(caK+t-d$O>}#7?gLccuZe9%*4tYUxxn!v-k{dj z3uAwb`I*P4*-M6C0+ZZMIY(@6*RIAM=g1M0i|gQV2HL>&sdWVa&LL@3wwoJD*&t_Y zN^W{LSE|gvrPvs^kF?FVPi1)oPMW<+x@~13_U1vVyALWeJK@i+tA<<>_Z*X>FC<7_ z;&fV`m0gNT{{Sh2Z6=#and!$llC%*FJO2uv9 zrb$f8sK}SZfks?G75YZqU`qWZVhuzx4kcA#R7ND?LE;dOARNgpx>y*#?q!3CAxVQX z+sDkIX(hpuTZlc6Dj^7v1d!Es5vA-R^^9UL zJ4q#=nh?q~kR(bKh9X8JfeFuOMOkpaNI;xP0z?g?h|`F*f*_-g@}Xgvca_@$pb|v5 z5<*K!<>bSt;s`D!!IBnHr?+5AlbIv|%pwlbgS;Z1A{WGiJ)R{Qq*6?-WeJ`J2`vXK zWk-1!G9N|JvNC=nhM!u%>P}cd>1p2Eh#N^`H{l&?1du-S zA5)@dz0G)R38vjhJwaW2`HA1EL|M%)BHLHW}0|)$xpbn=%6HWz%m0SWF z#$xnnF;f*8axt0Fbqf(RX;}Q52OY*flLaqoDm5bxPD2sQ+^SONWohZ|{oolqc9}Q3 z*RD3!8zc?NM+RtGyH`2M@_uFX+IF{jae@#w4+eC?>b2~(dZfxBU8TlG3}H07jYUps zWo|%R9!w6qOWYvO@h;KKV?nAVcD542#N_e2iMy|+<@zq&H&T@p)K*cMvK*Y_=^R== z_{&hqZM>1@HZEOK?R8WC0Esi!ROwo>+Nc0IPbL>5qJqPvsCZyV?e6MYgWua$sQs%=*X*uA%k+G zxSIMZ1=R)K4S*bDhW>M&SGQ?3G`-EH)T2;pRb2MQb2(nG(rwc(sbB9aJe&*_KUucy z^=g)q95EcQ10elClKPU^^{N(DeXa68Nn@49>|7;cQ>v}FIad0^SEwrdG|SX!$!*BM zN-d9Hw9mJ_KRu@H)K#mmo*HG@qrXVomALUdnwLy-92$hRaMIc>UbH%eCk9he3#^Ss ziC>yY4IJtHAaTU086`U!Z$ zvNYWEI<`qs;%jO3V75WXF*};EL0zI{rnAwv9mIAWHmgpBJ!-gbGd`QZemFViGo-y= zP86TkE2^y!?_S5eLqV8oG}t_u&)$72TBjeZ(9`O6wll={Rno4ZQrgG|JpCbiAoTY1 z(UFgF!GqOQY)tLr=2)`Sfd_fxZsiJlOX$XJ#k9Ao4o7(9dek;4P*unEgVyXeDaUb% znQI!gQybCKtzA#5fP?p%DX0tr+B1&Ea=I;G$R9shsadeTGH`ygQtdSLE*mx>Qr=`MZ3H!;h z&6h2XcpMTB^@d)IE3IAX=Wqa=@Wc#{q|C6^wScF0p8br})gl@l2su;ZzD&w>MP^W^ z1Hm)J%Qaq{FG1}hvQJ~$Qk$7Sgs>hVz-B^63OK}C;%YqbXE}Rrwe;7Fk+dE?-%&Jc zyH}`PsH6Zqk-(g%;u%07;Eyo1>#hfI1MBsjW1VhUsMb?yKbQlJyRv_j)2pv)Ytwp4 z1srz>GxyBolT&-J+Sp;{dZf1Sabq5Qo+28Jso$()>Jka}I~wCnkxZS2Lu*uexFEvFsn}Do0XQ& zf-y8SkEZl1+~Ae=;w97)y+*AQPI8p;%sY3GBwpjIu_}LNcqH}{bu09`cc?;TIaf?$ z?Zg@BNV+$+mI1N2ao^+^uTNX`ZYf3vklY;q07x~q$4Rv6?OQJc(iuCJ%n_eB{RHgv zoo3#S_0;I?X6>LaBE@dlIryAa`YSXfl|n1D6T84a`;6%Nh2FI;MU7nmGBU30d}0f4 zvuL$+(M1|}(4S4z7PR0H$GmdxpVMB`Vz5i|%LCk>r1{35_JeIgzh7$v4eA^-7Vbt5 z`JG2d>fIXi__fwub?uF)P6EC@hk^d^vR`d^S?Q_&0748#qRmRK!wr`63|Pd^O8G$ztVGd{?xxw)w18G->0FsCAQX~Tj)kP z6Elw=!}4YN>zd!$`#}QA(MCZiNx}Q}mAj^+(4$6EXc)qtd-u%eHJ`Psrmyi8(3RC@ z1L?x$Tj%NeiAp-&mbOFFw=Hjr8oMpwN06ld0BB{!jp?b$l)KwkQ)S(XkQ0tXoh{pm z4Z)hWdVNEuD}c#iKzYH)`s1|a@4nq7!95m5GJww$B(XS>44{`lNs<{mLWx?_ zv@l!MGba);1gjYe2{QY030g?4T5$=C31t`pd5#1?W2bd+d8Ew#DO93RlWn@T_ zX;ENZD*$;jHJs##N|=Iv($7gNybdOPS8eVIrsRObBjO8Jp;uOF%P2nvH={+*xPgzD z&b)=I*nfs_3T)AYbqR_r?%rcy8?@!a_-&YKd06H z8&jsKFuNBdP8K?64E3c-*fJm`OvBXZUdmawg(RF7@BF7rtI{1^ps>P^cg)STmc_L( zhwh;H6E0QH>Gb}R?JNPtcr!a&U4{PucVS?D4rZmbHFa-x9^=69AvBa}>aW!#5>8y> zx6)~sH%6+~wdfTCfx$iIpGoyAz3Ox!PavMzj%!m&z*qnTb}Qay>H3E4Szk{9NFc9| zyivZUc9&2N0OLMBlb+PA)YfY8!3X`PQrcmtxe~+9PIL70GkTt#EUcpVIB8>h6^U1@ z>Z(-xjtdVHN|Z|N2RJM?{oriIlBLh)Mgty5`NXxh5`l2cJRbZ`yEv(1o$gsms-3w$ zC27)DShBFl8;M-z2 z$6$Nn39DwLdV?PA*~fUQXrJDNhdh9M$6Hr_NYra8w2IdNMe5wWh+q$c3HW6X=rw3m zoE~wHu6#;9lvi-9+CnKA&m>N=Pp2zx+I=dL+lzp3ayaoBXBF5{7YaN4XZ%6v+}%k< zKJ^?QoOp`rmB!};><8VCtml;7b(dJ}&{R_8^;m+eyEx=X)0b-vHaez1hqv;X{U+`7 z`fY-Z-*1E9;vZc_x`riK23!&Kq zi+{XAo}enTYtqWcgSh$o%}qYMu*H=tW5Or*GA?Ufqk1w|EWR z^$v3s*7Vx7DVmNm@MqoA^OCunE&+}VZ>I}g!5xfKQ=zh(w&3SK6Kh8G=_mt&*qNLA zJsqVhp?5eNz998|L5hV@h@HBC=OE?*?b>PDwdyItkPPJzJvMLn zn)+>DY{pTbF5RZL-5#${E~5euS`yH^haOB_lS#8o zxj#`Twg*rE!wH?84Q*k8C%=iCuTBG={$`y{r`p1L{U~rpaSPdEQE65j&WlB>E}L`6 zm^bMfmpJVSrM;@N8NnW9E{mz6)wOoMIr72+?wlDJ>QYGV1X8VD?&k;OM|Aqj6~ICW z@(-D<>}gFEtV=HP89CeLZ0`I9Jv)@;;J9>1TT^-C0ANV3>9c z*VeR!C-{n)8j4@i7Hz{a@7g@)%mrZA`BUAD4q;N)w$vr9I2ge_&-@b@?cdz(VV*?V za`#uTjn*J6HxC3%i|H?ofmJb*#Nk2vndRl3y%=7UR`!vc6WR||UO+waGYgv1y`t1& z*gONr^oDJwj5Y{A6VK|pdt$IB7#T9TE7-#nIUsnIwJ@O0^10YHmes%?#q%^Y)~$QC zuu}g3GZvxm;LUmp+sCv-47!D??S!9C$j5nmY&NM)Ff2$Xjz=FN5FI)IzL;DPq(=V$ z6Q|NpPMuUK?f~uc_kk^^ZBSbD>8B#r8@;^o52WeZ*6Leqdekp~ag*>qu-ZPMQ&NRk z8%Wx~_aov7dXZLo*>`~5{E~Q*wC=A@R+CV+>KKqvsZzKEeI?S}GNp}M1{ml603dt+ z0AUx_E~izbHM2NY7y{JSXTIxm~})bIDlH=|MEh zt3Vfc-GRX$NL_o;R*_P~uA^5fyKp~3Ag}7$Yt<>MA*|yAoxf~BY{sKq>d=~c!xgXs zNy$0#e(;?$Q&l>1Q#e-u?PKO=X|#=^S}wTd&OrwwXT;I!Dwb&y(5Bm3y}+pF81o{I zoV99oM$)YHSNuxA0sj90Y1rtr{W_6a(zKkRE&MDA>`y%ToHm~0q0!Qnwix3q4+G?w z-CnKwqUn1CDl@d_apxFm#YQiQc0&v&ThLmXp_e@Vn@tW74p*2VR5gjVdt zusP??>6xq6ts|%CnSd-U+Q4zfJLBWD&ev)c^lhj`1%U64cYR=LwCWuYsI&rq{#AgS zaoZv@(fW>}okfjBfo&=j`!Klp6C+K1I*lKuu2#Z5#GGK_6271kfKU%r7Y*bHr(x4S zX!UvyuT^jA6n4W54iD5$zpYWGZ4uSiSTADmFmsXQW5oH6g>2MaRrT2EYJjZ6jr`~Q z&i??U>yEQYewS2SZGEMQ$;WTfa^%KX>&KPHCqBm4E&)(#P&Na}A2ID7ukgWJ9y8uw zU7=H9vbJrkLf5$E*pB$ei3y=&U{)L+)8}Q5tLbBw<4JXktE;y$7Z_p;cTb@-jE^$b zX*nc}&qoG*YP`Joceu^Ox~Z3H_OTe2RmM1;t-On(@Hm&-E{njIE5txplbPcxV3*x2 z3>bw`#8bp(2E_9i#H%hukrr7|-Z1F`%Y!75E*c9sk{U>G(sw{+SQw#mDDpFj8O#t& zGbxC0xiZ49q>w5&mjWDB^p+KUB#RLS!aQpFK)qi`BD2O6)%2bfpNSv}6C_ekc-mE* zaSkc^Nf7itl06@!(e!?jtyj`XpXq%ky(94*^vA^arSy^|=^uzhOh$b%@gSfPNK8x0 zXA{NYQpQM?23<^$ia3%)@}4A^pcrysu@1wNDTyQjB#9zP?Ia71APD9#{h%AXLb#sW zBpGO+_WdWZ^n}Fm5FrsDfiA*O$vgzQiBM(4lKH@ccH&u}$bwULOt6qvR2)eRrWH(G zOe&txMgh!*?JBZb1yW~&VnLPjB4HaQGMvOv@*~yC4XjoU#pAR*s|<*ybFaDxq;c1t z&?PzTGyzoR948aXy!M4v;z>=ARwQgF_=9@$nIa6l&jpN{x$QmaeISd0Ji~}%SrVmW zOE5W+w8l2)#PCQUl_kYz9iak4D&xdaai?R)j0)wTV@iHx0bRy<`A`~x_>Hf}qTz|e#oowzfe>VZ*)6}Jw;KhiF= z`kH%r2jq5%bvoX?LghvQ?>XOYRx(Y%DfNg09Fy4oksI|mAR_{Yf;-}AYjmpCIb~g= zwq`b`rdmU~7Qycr#WkhV`m~F0Y$5&*+(^}WR8$hQ3yye;J8G7-qufXt3)_h%twZIk zLJUI5sA8{TI4rpPkEB_(7Jel9o%|n}Dos`ZwUDRtePc&VmZ)W72TFVvYGn`;@fEX4>D=vQp3rgw z4#sO6rgFyLM|wtut3|c!RYe4bBN>Y8oe!!i8&vYS@^jzmH7nF0p_R^kpfAv3V{6Ki z6%^&N&%4Au(|X5CbkZnx-nnlLnWy-g_JH;P;A7p!XF2Jc6WR#K$Oa9AP%ua<=X7ox zqa5_W1p^kEHug&omt>uwjy7>Z!F@?L&ywdiyFkas1YiltBk=icK)|?aVfqPGE z%8M~FPgLnr;wgTh9Y+RL+?3)5x*gkb?E-MvT`*zv?5+ZXf+E$Z9@1qjCa2t|5>48| zvTS3Ytgq!89C6^pscWr4?L7I!(iX0ESULHc1xqb10XakBO`1_*^(r{bHM(HRNQ$f3 zLv3z843ew}uaSlxA$0nc9@dn04opVv^*AH3k}<4dyAdR)yY2&l8%k`{?c4|v=cPp< zRA2}uY1+2{GnS88nnJ$}K4x>P9ZI%C4EUNhvGj0x$dWTgs9RA}orBC`iVIeEV2mJ8fNC1O z-VQj8C7zJnM{)UnVyahr6=t`dbI04?~q~Dd!l+%(>Foqa;PowSn>s=`P&$rkEC* z)Zh?)Om?je+O$=62pf`d(0B0G2vrDhk)M*wft~m_eeu8OJr(T!SeLTH`85RzC`+X!;8dlfn z-}jCwVb79J#v|HppP|tz8i-4)L5FY3=gxkxc6A@c=)-2FU@%ycq<<-$N@}%wimCNg zPX7SUo)6MWv~|5pN9cia5Urlg*#7{}5Pc_Dn%7Fn457+{{{V;_MRh$e{yEoPYkJi| zuVKI=oXcxdPo=iIX>-Oo9)9pgx`vvqbq!5y5H@f;AFuR<>6#Ov>2;QJUAe$ADX?hh z^oTZC!#N$nJ;&M~Q>gU1U5?=Hv}{m0<2|Nr(^3P}^_Ocs4LWRLNFcK?$tV8+#L8Vm z@AqL*MsUYFqS|q78j;Reu17e}C!gsrtV@C$>nCEJp?mikG9ha(8b)8kVV=hvfPvPF&e-(Z}#pg6e{-uGpf`rHmht>wg9^~5_9trTiOd1ZFHr8 zC_TZO10NvGo}2qb)%9Q$C`Q7$9PeJ;{LUXj+M?{=ky!R)gE#BCgw&AiLlz9Bd-K5% zu}b?Mm!atOD%zVh1S=iELlQ=N&mY<4rtBq~keuyi$s%(-E2>qY&{b`^s+`sjsH_=F zf4Ti8wu|8!U*?!UZn3BmsWpA((p#=1^iqMEAp6`p@oi*ON;2hklZ=k@&CAE*+=g@2*%sCKjyu>KHn92-@ zBwk}Eyh~vlWEF)AJ+~erin2hxe0Gl&5Ke+gp38{BRs?5G@;O%w5#U0LPh3%fHMS zf!K=*z95nmNo-X(6WgT;#Gz?y2+Sp5%SsoL7__N)xw2PziW!?QXPHg;mIop!7?x2a z33XWqu{=b$S!_ucW%@ytjTOwM!E*-K%T@jl_FG#P$j|-i4`hC+{V#xB@r=*VFaAYG`1xg zC^sBIViCJRn1_Jo6-N?^IfYRa_9X3=0$2!b1?hbx`hRFy=%UYG)0n#_G4>gPHRSRn ziUcyjN5o^&6UMOw1~KAD62xLelh~4~3C1UcxEPV0jH;-cpqTDCGVIw+xRu9g!z;S< z9OiYbRU9&%hv#(-mtpyJ{<_oHv;D zmfd3)DtUvd*R36J86UVmGn3P=4ni)Nsm zO95MM;Aj0YTW!@?aDf$j62@-o^dK_@UugWt%t7?E<7$n{c_5zB4As+@VV8Uk{7c=6 z6LK>H_k?D#6_&BJbL32oMf!^^7^sj>#|CP4DVNqg^peSe{;WvT({8S`wC&rD$|aXd z$yWVVRG!Eq-87L?tzZM>6XHEGLaKn)vA`;$l44y>pGEg$lh}#1uc4_^aTrGKGu~rs zS;FH37~*Ri+eN4roz__AJ9tn$LKegRQmRKIGHSGM9^9WC$g!5ZhQKYHnb&)2EVuq1 z@T7j|_8{;KZr@JeLn|@y#7@eMm7=KJQ=hnzMyM4la`K~sJczZu42bop55OD^#~2YV zsKhXC!MpuOgZ<&|qbiOT$5Y9~Quk+L$X}9UJ&e1G=v!m93C?iEp>0aupcnzT_xLbN zQ&e)^MsxECsIdT07u-9X@IGemwiI0-rr9+yK5%`rO*fsPNl<-=2al5}L2_R!khsnV z2R|lj^!gIj7P1kzVRJdwmcAyNrje*k z!MprLEK8^ajFA$rSz-`D#CU2=$<7bVcFvPlsFm-`J2Kn?k&U_Uebj?336s}15x`fY+nXNWc~(sHqD zlFHAtjj{*MsaM(r7N=ANSW(G|8#a%V;7Y09yPLdHuemKoUeipsuoLHrlI&Sbe@$np zrd9qWnb}6BuTI4)@e1v|7dhRJ61tO>Mt_`3)~9d}+rg1BqjD8@ax;S~aw2j5rp|o8 zBeixK&PLPyAhk9#U%6ZV031eDOsPb_YZV2^W6XBaR=W1FT<186Q%Jh>egIRrgD&=x z*1K8ksd&Kr#GfCjmRV{EI}1?F>G&)?yNST&W;&AKa9H=pXu+|W@6y9Fxu<=~7Mt@Vt|r)4a0BlFKV@p;h&&)OK04smGkkrm^TA zg8;0jAok49)6(yA5o;=f0U%_}I&>>-i(#>1;#%n^NeHYO~ha!8x6O?6+Ffo}emA9g&cs1-!z=E05}WJG6aH+SC;x$SQnu z+D%NlSFYBq7Z@sk%I|yTwg~U?+p>L?dq;R$X z0qzv}!G$j0pxvm~QkGU3MxBc?SLBo6PS;ld0EpA2f~yF}xROC17!&3lCe3b_YgcZ- zeWyMKCvQ!+U(=z12qokr46+I;^2KN|PhxH9SOBRdG5*@>-ox~ku5fq@sh&?72&o{T@EA64bg zhWms{K{CfZJ~NDoMnGq$9HpC&K3Li8y73N&ID;#LETe$MwYShEGrQNO-4wp$9*9u(UL5q7=}|NlO3gTGf3;q zGNxkno?~^HC%rK$)ri0n+X3z)pg`0!+DE1IkD7!VpdKZn&9=ODi?HTBd_~S6kR^eE z-HBk*BMeJmK?5zj0!ebB0@Rl_N|pr6Xq|{N09HEc34ml7l2JzxgNa;E6D6mZ!IM1g z1Y4Ocub4J6cS;!q6Dkb0+{+5gQi2f#y3~Fl1!P1j77z&`ev{Hm0WKC*Wx5ThEB>r3(Y~g!K5*vdtlde4)OG^xqGBviTyXl-wY$v11 zoZnMoxyQ`&<$HSg=S>Tqh0=^pQ(bvm2(0ln5}PfjC5+A26{Sk|nRT&|_@}DT&Pm|& zI^L1g{fnl*o-?0`$=_{hVvqqi^Wqw-O{i@~%LCjV#x^b#^K)+o}8>JcCZ@3Vn`!`W?9tKUw0Ytp9VW+DOh4Z zDDDp;WNPbHB<(mqln&Ag>viU}190EqcZyTC#ocABPIK)riPMeN2O)5L_lAC&)SRLB zd`(S#GP2o2E_VWRj!sM-n^?7V8kJACklROW`HiJ?3e{Z#wmsiSH5zIIdP+0$WCiMo zZgoNAgZGQ;saq*+%Z@vIL#wUhKMHU@vKr7SDoJiVyT7!~`!02=YEW31VmqmVFZhrg zf&n-Rz4J6~sXcYcAy0gG65@*V%*qqr<%!Jg=QOlRg|d^D;DeApAhnHAtXp0;gU%yb zgmTX< z^M&lnnCoi-w zdTWr3Ve>=r)>G82?nCYnBId25zx4e@RtMT}OunZ>yp4;8#!P-#JSY|GcVILlvK$pW z#aB*J&8#_ywoxG8)h@wR=9ZH#WR!N1zf3;9?Dicn5(Qdl|+Wv$sAV?9peM z^Y$_FS_-?_8w_}iJ%Q`v1jL@JP@U*;8>vduky$y9kkD4?Zd6w&cE#+w+psP^xtm&N zYY+t)tV`GLF&O&`Hg4-ki&ptoJbm+&Ci;l(|bo4&qw}wTB^7frI42Re2SP z+#LMO{bi+!5;=uJN+`*|?Jbk6H*RwHBzX`*#j${6!sD4$-D6<4^D@G%sIE`9<1vV1 zVxkba!=Co5FT!92^bo+3q`H zG`cPJD|d2E0U5~nfczDH+>_g!4o}iMPMvGW1nfBAaXi^%_G5<0S)#c$C5rGQE|JJ~ zurMQ&nYfI_ftoD<*bF*BB`C~}C~lWt9fW9~kawmHilO!)k7LJvx&7MFKdY%%OH^Ba9; zmcjUaK9ybAkQj}}@5}`YqSRN7tgFs&@HPG$OKR-g$WI^`?Tt3B*W07hucQ%nDN$KH zfgQ}v((5c`+gL5~+qg2e+M1`y2$e58N6+2@-Evf(ofs{g4@cb&`*K*1nKY}YBFb}$vG$!{NF{{RHX z)@p2ipXzDvuiC)A@6CbQAx@y$8C>HhBu_4RT%OK6CN2F&xv$e&X(ST3-GR!>be&`A z6*AZcWa;$*5vp^Yr-Sr?t%W+F?;vL)dF*$U+iErB)+|6tZ!mOqioXyQ+XKuMTB`5R zY*c3Ys>~n!rE@sFI^LCmxFJ#9jxuKUi>NWE zEr;x9emwHU$CHm7q31)MB>*79qlX!i%RTtr&Buc*b#+x=%OuDgNn^xWF5(Xm>@z90 zK`1*$QB0*to?|GHPb!>C?K0p2ECi5YWrD(l%pk1%Nfc^{97-BmJ?YO1rJ=G(Sq&LM zjAj$j0!A?{Ar$t7Q2?h9s?kdk7qoyOo%o4eXMC7iOI*X4jhvW!UnU0-L1US962WEF zWuvqOki!BUSLrJCoJeeoxy-0@3uR;yV78pYMSRK=Er~`EN|T5;X>ufNCQ8sxCRfuf z8m^tq+BaU9W3+=gX)=8S@imuDvfqau1XCna^p{%9(mHRXp`#um3N3$VK%97lHRHta zA4woxAc(<@mCuMKtjPhKOKy@-$Ouf5HyP~0EhP7r^s(YdE@y!-zMf~jE+mxSEE-g% z4>B<#F6kSj1t+w)t&fNWH&0~3dPm|&tLY+zCKMT6pAy)_l4KQ=oJ$U65PN0SaV@q; z5(xrH6Vf+GgzFcvF&iX6*ajqu;|38M5~On_d%&1e+7uW7WQLL+S7e1@2nfVLM!7Ojy?O5NV zduu8gPoEQ}E_r%;OQ{4R)Wp7XiPmT=I+eq&2wY}-?u|>0K`i@#=L#`8&Y!7$)IR;G zjF3)YmOpM*oz_w^SF-$k#AwmAwpgvSARgJbelgO{%Sc3C3Cy@)UA-@?zGr*|Zw>Fd4@@N*gsE zo*Jge?TBk-I)pnK-z9K9rXZS^byW?tws0}tYwQ=)Djc@n{{SW=-5{-u&WCGo2=Ohm>lFB9>PP}itwk2mrkWvV6f+v z8NnhHTfM+`*bkFD_ZsgmeOxY{l5{KdKQ+E#Y;D~H^lOV&RTN$a(R3s2y~?N zGL?u=`brRYApTQFQK+u7#9UuyrFO3}`7%BbpZzmRrzJ=7o^39h{USV?ER4Gx{ot(8 zDSymm=U0p(`YkiJzNAjA~=+rmnhiy1Pk<7g4xW_eT?B1H^+*AU7l8gH# zSZ&!ep)a(Nw96o78RSh)T&LLh@vd!uWOSjki57mz(BK2&blKWv($ILser(6Zi%vOM5r%=_a9QKkl+3hc%HSyx+TlQkJMmQkEt^Jm5xKco!i&pjam)OJu ziK+b=!FaK*65rX2O&ddgu`O3X)`~@SC$xE=Ng86{=32gwjb5sY+vVmzWoyh8<5QB zLq*|ScQ~CtP||BOnt~ZJ2b@6aGzqBFDgGncda=)ay!Q^+I@(!CTz2GQMHgdeJ-p46 z(=M}00bs>O&_L`WW2VJhDrDo$)b*K_llK33_NIRXuW0X*10EvH5E4OI_ zIqX3>2kY8-a(m`qh@)7g!4`zw&RpObu9rOsjH+^Zai8Uhk!Ma8>@|SA9zS3G%azn< zMRn>>n(@M~K734A=`#IqSFWz~!}0J>82iEM^n0vkRly^IzG9kN#hI)jn09zX*l=b!hEC#hCI2-Ww&;Pcyz_xnv$Uj)UcD)|rtbL)j zj!5|?Xw$b>P*H(lT#x-i^tLCir5M&QM+~6yKGQQ>(~Z@)qH^Tt7{NdLhSXK)*}(x^ z;2qe(`$N}e(&*K)n!wySJ^adFwU#@MX%fV8zTiBZV;^`ksIeDnC>S^-Kvx z7v3|HfbdMn)>pRHM@^ud&CJ{~;$?c)xWm2__#Pmws=|x67%k>$eiF2>Z3jKTBZC6# zlZ~XX&Ss`;-ZCv#Jm=(zWJdAG`^T1(jz9zEQq4I$r+Inpj{>Hlk5#9YHMh(54SF(j34nC>3tKY)WwAfWr_Zxaz3~-FZL^~9XqvG0Ce{I2xFPq z{xhT|{{V8csl&4ZKS7bq;hq<6*kwM$zKS&(6Grd|<9Iw~a=Psfr$mg}f`BUFRenCe zPQyji`c}VC^wCMJ`v$OJMf#qU3oz9S8yihz@QS^~FFMM}_?0a!b@ z_ygzfF&%fNM@|-9TQO2V87(%bu9W@YscpBzIrfjVs?&a^v`UexsQ&=luN)}<04M5Z zhL5IeQpLW)`!Rx~V1Fr_7n~N~#p-nGi>R=X{*$#w@{^_06I!iBp6}ykW66zctGNnS zT5Ydr&IUstyihfiCdz=O^W9q}8mDlj-9q=f(A0ZPXN++)^aC}7^-fPCF&lJ-^bdW> z>Yw5Wc@nxGS4(LZdw~3hwnS{`%Iq{3fXcgreMHh_oYzfdM-BDuI+bx?26o}gU00u< z9o?29KoQ?a5!y)8XT3QuUrq?61tY2l-?LbEQYej~z(VNekS!XJq& zxDnq>O1(o7469eXT&c{^x~}{{`eVp}E-GU^=otNIyhD0QW2DAv7#`EYq9(zmObgHf z&LE1^pm>KINGXoexyh0hOe&bTg;OLU<_)ntP9WIDA%_x$W#!cg@gNS_1p$m2jG;n1 z$pUHwO08V>j1fEpwv}3eg!G0b)JZWw8icQ=A&jY#0jkf5PeU=a88FJ35Jl5Y0*)p9 zFr&0S+z2*r5ibZ%(9G}L@Y^_CKGN13+FJU40wTPOlC&eB>;q841luYr}&b19&?FoGZBdVT0t3&Sj@;mjAzWovqo;!fc|HjlJ32nSjhuDU0LwKr=P3U zvjm(?W$4tRl2bF5>wPDOilQ`6aD91-=L{{UL9y<4A8AEaMNMZb(FOy93X zPR?H^?=&lyuS;cgkb|^A>$EE?H#waqy=$mpJI;GmPNO8?fPRy4iqEf7bdJo3q>+we zmS1L``rge#Z(s2PvI;*!J zxZ*W+VVsrt=5gI6wJl_T@yvT&TCOtSoa3?LW~J>~K3E)% zFlL!r{Y8`ydf*=>2{i!ZvHZgl9Iflv<-Cbj(+#{S9C;E;)n!DDt+a#l%y8Pr8(FsF zpLR1auG8uR16d2tiC~(Iq*(jcJQ60ndf3}l)wLDwZMG5Ie15)SqRWj|)NuQ8f^nQh z>UW~mWnwZiN75BTi@iaBI~*L#Xt~j|I)bAtc`zEFXe?2NLC!}rLG>oKU|3*|KHg?_ zsnv-pNm4oDZyVVrZ5BCJ2PEV{GPq)QHs1VrhH3?Yh|dQrj`B9Wc@|}GFjd#To^jfG zAlZ!-TgtZJH+=WU&L*4Ct+`JsIUYOA+G@35A&`PQsp4tXs#93@NnX)g^w!s2K#XbnlGq0#I4u?r1bj+lwC1e2(EF6;A4{wR;g)RK zqo&U57qg}>bOLK}Xb;9zUvFr_A1ZS)^_o_xOF?M0OpS&<*(T|nb1tlk}X9tM*FT&e4*Y$i(e*TGB?+aAt!9um&Hfop9#M7d-Hl z^6f^B%>#rQa@c~qL#NbGfB4}X_Y>(ljSUxqu7YH0e`b;FFGKV2C#Md5`*FNle1~6M zntPg(nQ%R7&U^NsQtKbG_T58=LBAXkIE3gep|PyGz3e$qM-$tJEVHMZ zE;}hsp4zv#Hp?(ifJifUsbQz5L|04 zSG?;k`mAeAlcEn|xlhfWag6@3H|Vw$!Q7#CmCs?|&05l%aMf4=9k(9wuR^nvWq=$7 z+EjD;!!9Xfc=FiGuKxhTtx8ii5(6n6hy2Gey=@k1Xs^*54=CHQ+rM+#I2F+dTI&FC z2wp_K9^E*vZbzNNAKG!e7CYnp61ihj>20lmDpZ^tclsZRxR$GF>U`sKV2t)K@1{eQ zcLSV-2ehfsyH?1!!RJ3&%PuUPapuR<)wiwtw;Y^;f3&8ugtojpG4|n=G1w0#U6t2P zpw{b-8;|6Fyf3Mf>IEjUF!=7XmaML_%H}Lk&+bP7I4C>goxWIL0zk07#Q9gIEWo77b^xL(h4{$$htk+(VV zGU=mJrmR>AP^Tmj!RNQ%{pPA_mFnB6t<@M~c*E&~892}qJiIx8VX58R> z81A0HnXyP$1D~wwi%rzKb%9DEt4(t4$%PzI*ZX$LlLuW|Yhyv7Rp7h7wm z)2OHn=aOcVUqyPev;kPbVtL8L3%jqV^!HS{h8lfBFSV5LAhNxsp(?6-=BSC z^26yqT$;jdpzZ^1K?gV>cAGjKHJv)cu1qSEk+r!|F_fBb8)bL8og%Bg>eaA#M)CB| zgA=QwjZyfyl(c)TWeb6xmxdo=^;pqISAorCp$uEyG6mLy<{4_oh)@T_l46i57?b#X&nw{b2rXPh0M%^g5`^L~>=kry(N#;6l3=mQOs>M}%j zrQ@_1(~x^ZC@LcYN(^>@u>>;`Wigp;7AZWaa~l-{;t?p%i6ZGqI1uAXaU8C&pifxJ zq9}`Lxa|aV=i)JDpbqdR;LzMU1Jl~j%h*pIWz=?-#+7c+N}HNBw<1|90OlzbRg4oZ z*d3+M0xffWVQP`)VFp?Ot>#nJIqx&Jv^!Wq;!SoE+`yu20H_*iQ?i=R%H5Q z$6+Amc%ULkLcQe00O|x@z&m0#;sB;kXd=FtJ%sn+{*kd*9w&|jw3+Qu2eiF(Zt)xl z3wDJ{z>+F7z<819fO|(91GEc%X&_BNh$f(u90+B$Ni+Wd4=QQLiKVg((~~66^!E6c z+R|qG(1D>^*y2%y*|;BCoSqd9zE%8EnCJrn3lSysq#h$ z>|@DqqWz|RyMC&|cjFw+7rON0G(z64^b(AF_JG(d(>>PrBQqzdSm#uWUsMa$(G+w zwSXY`^E7lXV-(wJC4lfj?mjU!X58g>S{+vUg0~=!{{SK#sMs37MnK?v&RavOxoT&& zm%unf&JXD~>D9H%=@3E2K{L&sI%-|Bb(@KBK;&`6$kcnr+$cEYf!=f%Q|q0i5C?OP z(=P2G77$PLn3YyFX?1sR&j;c^Ua$hH;Nv(Uj}mNTq?K;cgVT0mhj!7Kz3e@y)uu}T z0*{U&ln5!WP>q`vLmcfr<29rr*Za`Jk&H`Nqtq;;1-p-! zCAc6etZ)w*Ip#|{DF-=H2>Z;IX0$bA^MUc=TBGV$x$sZGJ>wT3y0S%N1NF?!^lVEC z$nmK5~HTD{wU9;K;X(fwjknocE*OML&q)sV=JXZ9l4n&sX87> zmR_tF;FB$#G;Pz?z=jcHqO1KP5Amb@P_^r8bAow?m0D2r`dhloCrzVp_J~t@NnJt{ zjK|821$Rr+9Qz{Q^sPUInt@ipWm`;tgv)7(Z%3@20Gax(n$#}j@j6!1+&dXRg%&0s z%z924dWE`sFmgV#Z$s8Y@X`B!XQ;LER7IQiX;qJ=aDI@~rc>+G>nq>-$0^pfm;BXon!WPo}inQ0CD0)FY0oHVSb6qM1PkhX(nf)AUi6Ah>C)c{iNz?T$F$|!bPn~N&XYJA(Dys&j zCnMs0E-q=5{{Xj(lZ#iTRr-UbwEAQU&miNpgx6f;vJglhapq`O(H-=RE-DBdn2ir! zk&2~W;lN@uJ(%)ur<&q&rg{`vslIOrxd<08C&Eam*7}WoOuOgk9wfK zxJI*SaC|GQHBXo%9w>B!Ch*AwSh>%=W^$?-YkcO+wNa^-N5=8Pr_8zGj_@mjN=n6 zmD1{ig*!k8+QT20?-}%x>VO+VcpdXIO53AhZB~O~)_qi73*h+uBN}~T-T*etnJQA=&C z?FG9`N^I$n)K{?iR{-D`y^2L8Hsus#ePdULs^zOKw1Ub{Z{_~-q5Y|D7t<%WKjL(i z!s1T5ML-KYjIY--+J##!R&vUx+qif?Slj6q+i=|BPq|J8AcmE>t(yu+BaF>T&BoSN zuCSyoILV&Uplg6^?ecz;OI4z>+KCmo$2lW0IyA<;M+6QJEx;q@cE>4W&z=@mjXEm- z02%6U+KiEx)Yi9zAPujC=Z~?OjccYo+mwu`&ob4}bt*l<2PBi?ZH+YV*Kbo;QG(bC z268dpw-Xm}r_hFvQ=}oNxvi;=ah`v#5r@=O z%{scWt~dd>Kd780llG@v_XRdH41cSzoi>ZFY4v}*vZNXAbGM(Lq$OHs>vX?@-F}IW zqFXLYXCG6ZKoRW^P*>BMY<Ax5e)VL9h#8@#h4{3VTGjk8=tk@#QnIWN`p}T=8>3txQt|XQQ zQ)WC*6(hD_l4N$3EW&$u^Bt`Ujm%{TiK}y%VCNE)Q!L;J7PBS{?rC78MLoDrD~#eV zV~|V;xPy~3Y13U8ZI)sT&aYH9!hR<`qqU)&^8(yvNO0a|m;xlA1+yuQ&=!p58Af4NphqGFo%nWXVWePliLA9K9i?0J?<6ApM3+;?ctNPh?;jzdoj?(GQd$sMx)!?k5OFQ72USGE8psi}uWXA7iEU_+ z9LWg1Bnv+J^uT*rI=qc8>wrZE@4v=vYJk!JwM+#A2MrN3sncq# z8wqnWbQdK{W3~Z3rlUG5q3P9VN!$T%CTHs{s?A~A2Hz7$U93Q@wD+9t&Wgl?pS0X^ zvPC+JR(~iN`a}Y-U!*0>805=_U4nFuSJ`zqY=90q=d^E1?2B%x(J(!u+CMjlwZvd! zw6j^dl&yWL4sZ{1^EAb2YIT=tZA3Q`E36h^c>7ImmH4J9bxgvv<7*860QsEddcR7w z42r=|pE2RqF1#%$O(?&VD{=n-xs0aNuGuwP!!!GiI9}09LfVW&0NwKyqW=I@7a1T9 z7{WC+Dtd(RINU61WePPOpapDL9+(ho86b>@D(!`O*jgbRNkaVR z#~y#2-016LXf!P8UDs_uL1i01r`By%QrdxV!=KE2z-bP4sr>mP5Sp!O(9MC!9_Btj zPcrg%#~CBWoG~f_sXs9KYE5ZwqXZ@f0}mOQ)N0o1?%S5Islg{F>nt_IqN|1|M=SCA zLv1@$)G+1{YI1PHh>aS$%E*0Pd#91e?X^{C$P+g01NwoI9d(u95YdYDb&zskiY|`g zo#%#@^f{W9-h>PZcCZnExr%50@GcHucUZ5HHIl9U)*WB;Fe52V#>a7GMM=&NcIfPYAbQ0d8fmsJ^yXtf5w2+!*_5w|ENYaF~;vRL`YR;OD= zpjYVFDf$V|>NClA-CyGLWHji^Tl6X=%KEmZJL;T*k~1=ZWpr*y>bcCkA~CdwDJTIag43 zAw6be^4v`gE~d8jyX}7j_L$n88AaZ>%KSmosyPbh+;|3VR%@oz>1ncq>ENE_gj-FW z7=Kf9@x~exMXD~FPqc{Hs=E*Zn=Au@e8fSk+pDY;d@$swz8Nu754yh&xqsO8FVhEQfjcJYq1C!IOM^m ztg7)*xGs{XU}KXkZZh6~T@gD%7g)QgS#T zaz5V^Dz!`OW2Y{zdK7CevFwC9qT zSYo}2vE5nypqJ5U`8A7HIn)+J2!#|$T&RAo~ZU&rz*$= zkBzQRIUXiIPS;(ev>WUsgMzv39qBdtrkht&n-^`UIb=QNby~4a{<>5>z!FKF4v$@> z>PYn10_T87%**PUC+XGt0~S`}*nTnp0Ombq4)kO3wc3exm4S8TmoryK)JCO3zYIJX zz%9?G)nF2*Jmz$I4SL#r+&_acH;rz|lx|+^wh;!`e8X1Wr5oG=K>l+u+HG=A%=#R8 z`A3I5bwsySwl1APOYQ#vF@0MRkW_=<%-UR=PUAaFWpcMpnyptNKSrQ2?=fgx)(k3N zv9X@fykpJO>FxlH(a%*%%ndfJc8tf9Qlxj8xY&0o;^Q(yW-1srm{d51h|q0`V%$Kx zVg)2-R8mRiLccKT{7kK&Q;!nCC5zxen+S*<6ibk%B-)UkWd^iMXGSfo9K_zEo*{n} z45M7I_LNG9Fr{z=3DhW*ZAd&uVzw`=5wRre79!-a%vHf2X66HJgw&~S^6#nvIGBlH zL*5GgcF}_$0p!rsYcO~-6Iaw*1Ot!O3hhyI#$whFQBG$}agm!#S*dGZA2U~{=*5e; z%vN2F4BF7^R)~NFyU3%hi)an~>8JacN`+qde!||z;qWo%~FsXQdGoPyF4atm+Uct=Ig+5Wg zTdy$QvVl4E>SN1Gop>YK(-vsP<7z-3`-$DH2RO-@jjSv&x_KLan0qFlQ<08hH{G5y zFLMQ6e9ZfGHLsbhc=j%GJldUy22^z|yl%`#Q>muRk~IOa6A?i@@i-yUQ1-Y_}la?6iR{*S?%b@i}V(?fXEr1OWlqD z#9ePeZW+b|KjLc~BIh$6PM;iDx4+-|NYq(PE0AKA>qWK-f{chAOM(C^oDY~>o1Ahg z-9gR1!|jgOt+CVc6NsjnGnVwmKCqT*zYGXvIX@8ctz0EK$R37w_?+#B8iLs+N0~PD zEkXU=dFQlM)G9TO{_Nz4RJ?Y)M>FcnVW63ZPW`vp!C;c1B~{J-=xU_azXC1dhH6RBo0S0 zJvy@%7$ff=9l7~A>$+ZQI{{Cr8JRZe{WmD%pPBSSr#_ZK4m|u!`?^JMrDjD<$DT~a zTynKDaOFnbrAu&+?iYtt%z2P!gcxBBhqnpm***XB#YanM(*gV|l`oev*A6 zyK+YF^qxd$*Sf*RbK~pVAI?Cedb@C0h~opXltCt)LIl_z!h2^Me)CeLO4$GztvnBy ztyEZ>@s$K%0sa2b{UsaS8I-1Y+Hi7XT`=~&V%0n81eEt2L9Ukj?)0v0xd4&}kCQa* z3q>g5HmUA2-bEMHsdB8_H;iXI!RU}JA4*D8BBOHH&UpqpKMkaBU{|#E^9)+cD?Xk! zt^puf3U=LVQd&|XbsP&d??!<)C?#^0gf7Sv1 zDF>`583tM#ZB;_40x}k{Ib+torVh@w*v8}h;iI=@SEIW-gezT2oM0c;aI4lVKg55W z^L1+h$zv@oyN|}x@g;S16NVFo{{XDFmN4{nP{bX^8e4Z8w5|cjklIRLmE%4{<*U~# z56|8|MzpR7c0h7QhJ|lnd8h(zM@+J@L^n6s7^7NRqe96ighc4gAvqp z{fE`B`b_%L%A1Bbnwou9_1fLU4wR%9tfhi;{b!}B+3KqDIWwlQkPt@`KdC8wIV0j~ zGg#UUXsm5uawkcqy)CR2Gl@X;>WFF5boK;s6w-Gj;LPr;qFsKG(2sZ-257oqXD}Hj zyw*8*-N9zx7wXz3)9Ng)Na!ceEqf6QD>&~0=e#I z*Wu;fFPq87<3p!bT^6D_`7$+Hbh9ORKOYd*Ri|qb+8bzZ1DsD-$zImn+3m#i%eN&o z8m6h(Kj&OdlS`?t#R+0=&mhiM)mkpX4{`Fs6*bq@@7!r)-6 z(bMan{@->UT1A@T+sh~)Bur(;GIqz7;}~sUrD>^7RYo^5=eK$1^qF63wm2n+AFnfD z*vnXTDTCht8G4qTVa2tEQ|FE+48Nc8-hWU1Iq(Ta+J%in+rP*?xQyxbiE=X7-r{zV z{-2H}#houqqZRRygNc;X8ns4)O6?;%)w|9a@Z4UGK53Bju}fO3>l;YHJ`Q||nRI4Ra`M_b_C)JSz2l|;8X^7?O=bT<0HN8m={ld19}XfCA|Ltc|U?( zq^)X;oD!oaGN(aqop4m4d^bFd{7cp?4f~`!dk{uf^5->hPtUZD`J^s`obivc4aJa(9^!^QOchOL2M z2Gwof&JgW8ap~H-KxJcr`o(_>T9va**DrQmn_7VKZ?fj;Ad-_#D^%U=`$XD<7{&RaltkLR4ewJvt z#&GS$gW!Y2g`19OC98O%)xg0Fg#KVb&If`be}h+cT{ew1C;(<^Z})TZ8$UC->K!ht zk(#$!>~Yv*lm7tY=45KWXDr&6X}8mTc@62D7XzKc2N>{V<#Dl)JzWW{E~4REj%vM1 zu=&Tw=^eYL*Qlke)3ZAU0>o@?U-y>K>o$7&JxztGO!{Kb9a#1p9^a%Er&Vr`SCrja z-aB{Xf93J@gBl8|FaH3kyVwau&g_Vey#nf&W2i6ZJ*Krzb=q`m3mX(<5OKI424{6@ z%YkB60-OW1=byX+^{?SE?InoGBmQ@rT`#C8zh#HoeV}K7`c6n{+O?))H8IZn1H|EAW7|NZj zdHvvAHjalFz(1t=SXYNQ=F-BH;6yIevcG&zk5kiv44{}A-4o>C zcAchjxn+%7j-h}((@RUQ+;O`TnoTw8_}hrm8)}bLnC$A-0E2fhpY;V_Gml=P>bT}f z*0KI0GjWaiS6%-AR8#y=i>Ft|lBuEiWTAjVX2IZFDAr*MCq z)YJ7Xz+}!>9Xd+aCRL9SVR>Uw%p0&*<{+Bkw;7bPQds9Rq{W+)O|>YaF$?vp z%ud};7|hLQce4P;T3c!UD=bLN^EhFg!P7%JgA4d(`Ih2~rjR}|BbDd|$?Y{x6lQ3O z*=q)7)20@yrTGG8ojtWSCrO;g}te<*Ef%OzLXaj?b*OtF_Fnboe2g*n92G7OzG1-sQRZ3kbB0oy0If4PG?x>SenmK z2?heH9l7rt>KCr~Z6RA10%@HAL2T_krkzzbGbW7zHtxrHx|%c;fsSW8lx|~LNQ?|> zirt8ejW7qaHA#m635>FuZyavLriMky5z3ct8c(db6@L^hiC&xsW5y?klRdrsTzdBR z;u}nbnQ|s4Q0v*d(_HrN4X^6Hx{4GH#$^8h(X}_~0~-d?aB(=jGf;wCsMK{(DkXz_ z&BwOwFbJ!bV}Ud@{X0q1YThda$CE0yP;V?i$Qc|d-Z5j&F&9i7wltE#AmY>;sw`q;6Tc#)~;wr||drl#^H zKD4bmp@o^R`spO;$>>?HZdv*&xkim+d_#@VsXI4R=RbSY7^9450f*d zogRbJvp`%v)*a?EFx2%0OI#JtJk2KQ_FG9cI8{C~In8HG)M+)Lg{6jh=fq@ZZEmU4 zsJ^Yba06rbe9YTLnjIk11O>_A=E7>U+HRVUkZe7VgOsM~H>VrhY@%T;%wZ*B6fN(+ER+6h^I>F z>83^FnWL*Bs?~y$JLFAB##g5JV$@{!#8Rt$D+u{9eIC79unB^4)?CK0Ngf9z%{jUX z{WlrhKUgb}SDdEgZb24?BPKotuFwDw@hIX=$Wp@{171MFxeLm9_(nlq4OuDMJrC|KO_>0k{WDa)* zd*{xmmHtcbIVcJ9Bp&yqaLU%SsZfDlahZ(_Lj07 zqOn&BN|G_p%+Rrk04F4lO!HEyU#iQu1Cie|p5BRunRZuez^bo-i?1hBykEH|0}6=Ll7Uc^bU!ks@$_UCcr^W01()fcs^L)(e1d)-0`-m$nY zFb-ntr%|T@wA#dbnWUXH-CSUC(U|FZ8f|i@h~L&g7zPvYdH(?M6}O}{ry?>7l`Usfdw5cF7WFk2 zKL%)5(A^GsKdf@C7SGIs=@GItX)J!^%UjcI!8n^Ws;6NauhU_k(%P;gQBIMtjmdus-=4K5UF zD65<509STLaASQxQ;b1%oik*S_ljsYs)A?V_vE19`NvL`_E0jJn z+GtgH)K&$UR`7mh^&KruSbc@ahMv~I1%Z5K9TAFNB%cvmu zJ>aW%MqF9#ZxDA|ZC7wfjG(|vH0`>)i2WT!(*O(~If(lER5NodEv2x|MBJ>`K~Gdi zBYJXs;6@VNHmGbz+n<@slqBY>NYB<Y@1`USPH+md0i&VIv-4^*UPh8dJ4K>iHS?mta^btweoe zYo=f~?IZ6qmOfKWaZ-l1Q?;>rbxM&f|s)MG!adrVtUO-icA8Bis{ zgI;d}u1cbV@#h{V8(pOOWu~d?eammFRxm<<7!Rob05eBR)DFc^1F0i!NdEvaHtF|S zxvXmJL+w^v1`|HLx=ll&uF8;Se&g}xT9oeM>Kmy783XJDbGTzaUeR4vy^dK|Y5|W? zMoh}nbuYvpLdx=($OLEgnp$lzxxv%f>83r($AkJAi!80orH3@VNiMBT#f{1VJaLR; zF}o|(AWdGGmgFFZTmnb=2mWSdYdUVDPFB^yR9qLq3IXjH>ALmPDh*o$O`xve-t2h9 zj@$hk_JYCb>BUQXzAGx<#qr;?+Sd&&+CchudW}o8AYw7kf76-BUr5tlbyWjqo&DQ^ z$o`#-)9IRryEU^r`?oH61bv}Z?OUy@R2IIXzKzN~x{#a^{{WB~{&7uKjkMu6y5CKC zH99L0+3$`qo;ybR%~dN6O+~D2N9I51A3-uYcCOt%v+(=PYWW)gz+vOJ_nC3ZYMIft z2K8b?1Dy8}O?$qdP}tlma0zAc^f9e%O+qb{mNk>kd;Nc`TUDqVKY4~Ya_=AAUjAsz`-+W{Y91N!K~+REs@+#eFp2m{{YH#1JZgGQ0I>L;%jXoF5E+- zWcd@N={maYuhMSwjH4^+E0PKcqtZ1dyOgmTkAP-9U7Or?hjnd%+E>(J&SizGZ6UFq zJH;)$*Wa1$;?Hj%CK?^CW5J$U5xa?%ZoPMRho@Aej_&N)Gv9g-7F%U0TOK; zjQA#Ma^pwGV=K^N_)biItp%=tm75wqoT-Ol9DAs!V~kE2{{T$J>DL||xodiDf&Tz3 zr{Ff6fG0;`OM3tyn@UQ`e@wo`;k#lh`W!Hg(Q+{akLxXVi; z^=#e(DweWflPYwqPcupGsZ#*Vo3*3VAagZ;q+07!+DSyAngG?!YNc`!C! zmgG%i{XMJ0(+g9UKWGZrFBqGTrAr6aBX3Q8+zIT*l4Ux1a^taGJtEVd;%w>buA~t? zCY9F!2+OHKlP@J8)@oRAvWux<+`*$M;s*W0@{S-)ooa!S#t7?>lkPJQP;Dm}m(ysq zHv5Kg3v{P#Ek?aUcbOLI!@dFHJ7-M2m*Qqy(y!g-IOi(3){dQPIM4QuhSQ?<#O8`> zX*+r15gO-TaX6hAZaw;Q?cxQJm;AVyV@}R+40lhYw~k@E2$bm&ZO9zQofK3?CF#-- z0qqX-uG~gZW1802S$^EWr3}gK98KGF2+UZ-?b}G&JWY8GIju)dweou5vhRn#=YNY3?*-X=43+ES7Rdv+5qRjWFX0~{dW0r3{p3TUMP zBX^US>20lVTfM>s;c@wM8`5j){9+ACFX=eywKiR;70?MEGepsKNNuo{W7x|>ZtmTB zT-#_>1Hhc`Q|f0=U<9cnz?|snI*Ofw-N(;t#O=AS%QKQk%rtiDY5k*W^z5LL$Fz#I zS8Ellb_5RjoVVg{J*wheHN*@yN643rUQyB6Nxf7C+ssp2qQS$R*z9ISol?hdX*)pT zT=h440yU&fNX^Ot)Ps7etV0ZA=4D%o$sirSCTu0tvU|BdGL|hbTLu74$151;R4@VY z1loYu#}aR>GT;u=Ik#^<>C+r#a?2|#Krn97x`k?h1ow*8%%>p6thsGXr@zc~p}vWd zOYsh%kpe~} zKw8De9R1_S%xu}SIrHC-yf%}ix~sLJ85m$5@r{e7J6FeX@j2zoHtUBC*I=gL&FVFb zgYt&tO53M5Rz*c?Zs(lw^pY8^@ONh%@DJ8fv0mfI4l{-Moc>2y!wa^VnGdW%9hSr7 z!cuCbHojT2k&ZB7tW43kh0b_JiX&rK6~+{fPIC;ix2Cb;lJ+ZV)tR;&p5VnChB;BS zyaOyhbz75Pw4Z4nU`?*uw^hF*f@U{ewGGzcJG$>9If8vtRHU~V?0dl3X|AM8Ya58n z9d4H+DO{?Ik)HBOjbEt5=W}@x%_dC?aX!*JRPAY zv>tn(5gksvdPI|gWiGeYMMZ8}d;X7Qg-hX(^PhOUj=YP z;U0GlTe2J;BOa1o zyyHJ;ZBCSJ%88zCdRr1nvqQ1BAdYFW2GjlE>(mEcWQ%}22_Y`ht)s+D^rbz3{!x*r z=4ffIRs%BH(+KG7p?hRcU^3v&nX0bQ#wK2)N)VIt@g(K-+I@#|hz&lg91}P0rEEe7 zoc^ywYqN4hR`#CNxiC*LKB1%-Er=eSaYhQ==B3!SnI;8#bYp`BZk?%p%bY_NHDAwc z%@wV0r|TjHi&De@PaGnRUd7K7MkXlb=YNakjlU2dYZ)K#2jVcB1##Xyc} zb%m21{$kpu#iYh69SBXoX{%MuSsdnPQrv?WzR%rcL?89NKxTJTqu0|mJBS@FsIks_ zaU5$E*BIc<;dOS-Q*ranZC;JLN^WmFL9Uj&TR6ZzCht?N66Ob9mB|nO;xJH3UXsrbi18};7DEhTy?Kx_x{Vw(P?Lef&QO9+O=y6T02 zYEVzfma|v|c?UW01m!8#n|moAa~yqkDcGm=k#1MjmmoTgqG-{5HUiby0W+7;x>yIe z2J$9{;k5vVM;J3LMyhHprswsSdJzGxrkzk*6HJT`aKb_h!iHgxJ|){&z-)CJ#LdQ5 zvC6_VSEZqMrkxhW;Hd8cMS9W|Q)4OMnV_PbU|@K(aTQ*lYE)59!JbU=^6}sPJ)B&U z{{ZPCx|*1Jm7%XYMQXgXExjabbmd(gLj(EOF{>?oI{yIMEQ7lNnU{Z8Yu$}bq{{Rui z)?268c5hHJ&WHyeq~~>8(@#ybzE-o0k%qzXHFS0*^wn~{-%(F<*pHv2E+$s_v+-JR zDqgaN9`+1D zaq%%=%6*A%P1T(tTXgoQ10ZeWACb)EdX-K1raqA?@DF*rr0QDAN~Mp$T%xx)9!Na@ z04RNZboCu2jY5QWt%g;uuM>8(ZCShLeydrFYUKR;i;^q19Yx|34Iq^#7LSM-Pncgv6c zPFJ^SgQQJ5e)Tq^SKL?*LH#?-ylT)8gI$sTdzzkx9H{bQ+V7>lX)s^EwIEaw=( z;%(@9-5UDMY}pC`HM3_wWAiO&+#{a4fa@4GxjhtKNn_hRhu`TYZaPk^fb5I;k8>~6LW=|L$v;Rtm01A? zF2lgdl+keP1%nE4?%?KTvhMU6g*5@=v5nuQTQb0m8Hdvusv9gE@JBOtgd+~w0vK{! zXv-bOHQF#N#zY!^={?h+yG>2sml>H_q(xwa;CxSZ9J0LGap(nbOe(uCV*$AXcK`@O zVBmN@XL)wvOnxw^!Dut6+;RysGRp2}_Ke~^r_tI|-az`z=xoAjK4E=DU^#*O3w*@2 z72iW-5=X>z&e~qUbHs{BRu|-H>RU0+QRr*vD4JlfsAsVmKC`X z<1$G<_Qdr4Hk$j99T%+? z?E)RcwXS{M(ZtBr0y{J->Pd1(If`6uZbZe$ZA`QpUiF-VE_#y>{{S#cRgB!MAkwHg zX`6#nBJZr!T22l>S$cH0TltJNwo^rX49Zp1f_5^IS|aqz3C80ty*bhGxWKT``p0X&VBrM*w$-xU%QK z97YXqVB@r6ihVxfduDP+o5t@z(V%RI(`g3WVp+9oS;uMXHA(5@HZwV2=-1P8h@pR_)crHbi)uQ)K?+YZhJ3dtTzKSY->Inv0Ezml=Qx>xtiHl?8m*N- zFwEyXy|-i$wADD}N8Sm#eQY=bA~JP!mB2Wj8j8o6*X7NN$Bkv%qiW7nLsMffaUW}I zaf8fUpe%Owo$=?Fs^ymx$6K*^6E^;)cX^Iqq-Pn-%x;+M=L59#O!0BJ*t&HBsqlm4#d%@bH0(RBu{Yu8{VkxTW%5${y8P9LC@W&gz z-7q_Fe(|jqvwAZ*BkMGE9Wxm%h>oL0>DQIud5&C|*d{j`($zTp0Z3i?80xL6L6`@Wu8vztCXX9sv=tnPg=L1y5dXFb#F77MPkfSd?@S5b3g zY1Mda!|AlPE|?$~^6X)zy!6HDx{yG^u5sMXW%V{Cxjtu0))JP=j!5oiJsJaH0da;r zVroI>Z8Ou2wvEoXP&~)9{Wi^2=IekJcm@+psYaan$IQ03tgBb)(xC=9+b31)O*V@1 z$UU)LcTyUg#bAEzekN7jLmY0;+GgFVIRQ^;Xj>z1S$!|3+qVtiiG(#}Utr;7EYuRc z^8l5z^W@D;e_6{v#@3rvmBSo}$+o6eIPoyh*+=FW2h5e$6Sxz`K4aoM{{T&TWLV6k z6o%C3_Np4mClO69qiMr-GfZkwDO_ZjX>FL7GBy}bHQwMe-d9?nF$zn=EA02fX*WKrB1< ziyKDV&20@vwpI!c(9G53XA^FM%TnxG>!^>+`8)nf!kJn+O0`;xNv@(sG8IRUpS0+- z$huuPR;gOi$A8QJ0J&`zoomt4q^j&agL92J{J@UPf5(rc+OI)fP7om2n8+q_%DZ8P z`(@}B#%8r?AwW1TKdjr+>FP8KEn-G^$?joVqQ=Hb_Z8h(zB4z#9wT^OG-MRdZ^+q^kT(w$Rj89oYq*lo2nRfzh#Kp>nbBec!xJw!6vykPkf(JOwg(S{6ugFWD@)mU;^ z9FEX>1#NS(B6|?vnYF3XrjJSj@O*t@HbUKPtx~#=YK$0elVuvx;h3l#7zMQ}sl89i z!;vqh71BkBeW$tlMJ=r9f}c?C-(c)>+)NvNYFMyR7ImWZDZH+7F@rKSNmcdO!~h3u zL2Zv!_Lui?B-v`UxL!xh(5FRPEhwiuk0v#&>2=XibzkK)C_N`jyHFj@6?rpPTdfE} zDC4li#;;aULaLme^6O!`(AFFx9GuBDlTlj?=M&6|$KrB&ZGEcPCTh{tdje+*j-GwA z%~(0_EmfNi4{1tuO5?njT%3p_M5C?@!q@53cIlzKh^DVDq{&u5v(bL=>8>MzkrBS8 z^$Gcn(WT9Kh&NWfgLj)6PNuvAIlJoghGO;W4OV&L8e4a%s9OeN`i(C9k=kKs`jGzs zRwmw`R<1cYhMH$MdRr2sS@n-=D_aziIlWfuSePe6ONSW-Fpg1%4Cgc&(~vVXTT$mE z$!*csc2H!?Si92`BT1mpENf?^Cmp_Dr_OGRBurlH@pxC8mE{{>02%CCe znYKt2G~7me$0;Q7m>p));IDW(Y$2kVopzz>oE$`Sb}fifzzzqP{{UI{+3_-}BD7+B z@jkBw-ZAs}XZZoErm&>1aR#JYPSS9GVf6r|M3KbI)4nZZiS3@AYwR?sjqIum9L|$T zqNh(-vc((zBAPK?oehKoV{$VOWr2TralkPM)>xqlMG$%|WpoC2{QeBj>RRtmZ9~V* z&s|cCFL`XYpQtszoBPbWt{ETaTZ6x`Ot(F0hd(Kyf@!E6$0Na$+Xe>IMXR2Lq6%F{5NpA&MeF0W$Q zUY){$mR3{9`_5}k)HQ=EW&Y+Jw|`NN(^pB<-%Y67se2iQF=1?D?q?g0oh95xvrlam zn{T5P=iQKSTgEf=$b(DYO-dB6kzg&Ja6Z$dx{9@xmVMr#P87cn=Z_vH9rO({F}AA+ zem0|%pV#JM#(gsmx~g>xWrYY0hh?O0IR5~iJ|M1pE3J#0du43SYjK=#-K8!~K9DwP zVQ6d$gBLvZ9sK=%BL1-yS?Ukm_j*Nv&l&Cq-)_eXMPw)}e0CqH!G~AWX;!i=8CCYK{twiQ{{Vl9 zzX+yv**8WihRw2U2(fZt$o!QuevXf-{$H?P;uPDn-z&ma!xlR1j*vc>fkNiN%t z)yk9l_A&a-q0MWh)7oy^@51o8`6qxej{HnqTbXUvXtldlDLQt-uGTiN%O4)o+B9p^ z^;!lBk@1iD&)#!dQ>{{!>IF8S$p)v%Qhuw)_?y)mY=2V6q?<4>2t1D5jNr`J*7gtK zbg0a_iIP2sBr5@roJRiud8nx?)TgIvsZiB|LHXyu z>L9fJHt<{3QRy=6SL6ad`I)YHPR7uL-?vS6LG8R_KTOTl*IaAZ+%VvS;&OFWWe;U9 zo=!t}np%BbX?O1|F2HT$wB?tz9af&I)k9g_%3q~h4t8!IF)I3o^%WIvM-7e)DP3NH zMVq&wl(s93rLWPod1F{3?EsoZEA9E_YNt?J*KWa+-crmW`*hzQw#~ioF4*4-AXVFOQv&;+)P|@y|=jx zjV`5Gm?$IYdx@Q{w^3iQQ|+E6lC7+F6OW*Px7Kf9UHS6L8z0Ue9X@f+)zlepDRPw{ z;%;efOY0;K0MB^ceJyNk+ek0reINn-L<4BGUb8CHs;D_sJ4QZ`@TO-2d%~|1BU!CJ zIGR>;-BpT3bhhF7Looejd+0R^=hdOt?!13F%i^uhNg0i0_UGbSwRAh5wA!mmDopLi z-Y(Ro?T`ujncEII9P-Z46s@Tn(xNAL?wh5f3?R#V_>KPn(JfrFSh)I5zA=lHLavtD zihu;mze*{$eSM?9gk)yg$C2hxX)VW+K4x;lQ;v318cTN83v(M9#PUC^&-^B$r`l^E z9y11XcWv6POP<1JVPjV*TIA`SJ;MaVuF|1mYkZH0nbT?)aCYPm5kJJ=Q{$dxao4rK z#*=CID?`RFDy3RoKAAUKK|kIjexpuoJ7sn1Yy|~} zFR4d^@Ufd2sSXt?}+?cA10Z}y1Itt zIhRW_*c|;Pn>L#IvN`#Lqf1II;Jk@^HXo6izM=4clp@t(A-z1!Mz(~@S&n>xDfpV+ zuocEWeq%W77dc9T#PXProkrS5AcLZ=9%pN=0g47AGdBwq$}6pJRY}i36C%G@Y5?z; z=4>Nq7zg;6;#JkuEWh}Tp$qA(v>RlUb8$S~E7ODmhy-?3~a_;j#T5+m^BAhd(|bo8 zaTN+hjb=E<1`7ME(}FC)I0O$ds~f5%`sZ|N#!gR&m|Gv)+C>_aa!PVff24p9s7Nr= ztjl6*G}YHm0W42)Vm7ZB6GE!()T8X|JGX&7cTca@L^o_KqY4nWf49umF`TlDrA(fC zMk}a3cKDkbUWc0MwLNQ1g|?Naa9nxDet)!ezh-Z&hbG6~J?!8hAKaP08poV+*Mby` z=MZ#MEnETa^EUOpI;s$RwHuiGYc(j(#!hBus)*masG$S7eCO-h7Fax`dX(GeIF~l< zZOh&_qNVjWZbd4EOBOx5c=?O&^w`%;XD9Qs4sq}YfekgV%bN~NTTp-@A~$tHqDpRc zBR#u)BD*XW;>BfbXK5VGNX}UlM(+ttC)ZJG9jE%vm!^KrzN)`bt11cOGf}^0{ivk- zSGdg6m2!0SzgW1fj^Ch$Iu)V!79@_rmxu}p^lEK=V$`^5l{Y%L$WpQZdB^}d9A zSyvl-{NUmj(Z+V@dQOV1J~b??9dh|6^Aj``eL!t4RR=7fa^GksO_~}uk9IpBA9ycW z%?(IkSblG;^5uJb@SQAfsa4qJE9W>oh>aF{*KAt=)0HPDy9kd_b-}8qB(^dmdMzl? z(x&cQkQ97Ov>*K?i=9HL6$6};6xODhn<1Dg4oEzcGw3xFCN>o~Kr87ONgJ?U>J~Z9o~^#P|jyVWy~hu5KWl`#~IH zYEyMDPRlR{(qd{f29xUUImc+K^%v8y4n2|88+Mtg(6&y}G0e`@`e8qIKUjr5R9#q5 z;18s@zuB}Lj}Xz=nglPY4s0O@gEqR3y1Mjb0Nls4=Ct=J+RX2?4#O~7&aF}thspUd z(DD-7O(9=6E!;r9vX;rk24D;>7 zNK0C#!H%Y{+ukE$@8m`IG8vlFWp507<{w~nnDtwpb2MnU;By2W+CjfF!)F{UxInx88zFTx9IklQoxR8_M2CLIha~X3Z7%5 zavX`zDK^$M(bcLB0AmNJ(yhzKnT(pUrfKQ5%EJWCZEUyTn!5>6E?tpu!?e=emI%G{ zZd`~EblT0fd`(L3^#H`??WwAKiJ_p>oPbOzZlEA?W@l4P$V_#0DnOBPi%^V7IaMS+ zCWS7*cae6JWrTqSjkTjf+P7eY=4R7*E}LPig0m}IT5J| zc@O+vXVd}%&ZM!uMW{i-=0*6ZL+}+IeO;xA@zB4B68G7-%`A=l8HH-c;b-4qG zM%1050@wJVR$k&7POF zPZB*A+qxz-A?yUrI(43+1PVCw5J~Wd1q?o(+T86tO6!TSae*^;nz_s=(`xF%0>Jxz z34y0mkOa>f$FU0J;x(roqbO@s>PHGQFIJ(ex156%sa$)3hx|fhdW(be5a<_LQj8on zV^j}SHvkv)n6;Yhr-lTGMAo%gQcgT#T03+VP~eAKFb95Sg6cxyeMQHa%Ga;BXVcE# zv?Us`>H}*K(bC(ow#yQw^xb*yBArVqDR~8cla#-#vOdtq1L9h~uFUyi4H;csx@xUO z^cXLmVxRP?fg<}6^Uo7Do~YY1x44(7n^ZOn3X?hBJY6RyMLNGtbi3CT??5{V%x2w2 zb~ZM)ap7loK(?hRbBh!EM72E)EG=amExa%jhbJ8O9r(F#s^lmdnso*2T-Thj1M;7w z)TdirwW>7?{PJ6z!<{CTZdCQ1OA@7`OA9Kpw6dd+7+?@b)_C*A?d`)ov@{)5 zBep~B2b3+_>L%8V>2<}+UIA`HH|G9xi*xut-K}i)JAgUwHR#pnc&}~;BxAJYW325x zW2KEQp3AGHnL}=7&vWyE^_co@pQu#zThwb`l918)8N z@(+)gdvxD{)oM3B`o;%rjGw6Oqp%Ev^amf(YCl5iPhM*D%?*!j=r9I3IWT|zoz`2n zt(4VM4c4s(8;8d4j(@B-D_%4=rJjO=Tc`9$!q!yON9ad?ytbQ1sHuPW&sG7;`c@Ff z>@mpuWZ?mEeXVo3U@Nc={xjfDRndB#8jA0-yz`yNz=8d7gEMs0m8YZE>S<-7 z!WOg2Iotyrp8o&_YirjTSGYOg%+9Yy$5C@w)#j1`2RSnrU8}SBo4nAkQU3tP3hDqo;(bF% z%*xfEVcQ0MALi!zuNOZZ`ecoFq(LnJ{ZW#_QMfXioeh+^^>L3OPcRMBnY62BN}S?- zDU`9|@v*0;>d5%q0VC!x>X(eD%NdoW(VDOrf)9DOr+O(W2RNCg7G8+fux+@S74*6( zMcRVLBP5=~^oO%ikP2}aT|z5kX#>eK&y}^?8H(C%ElhIN;Qs*QH8j*EZMkuXmY)9r zQ*+izf!vW9s?%bG85#48P8nRc)2=fec5Rs1g##Zksu0!Qyq|(N654GgLZx;&f3IN; zw(YsW3O=(F8}`++=rq!L)_b#)rrSrEwBbCtSZ=i~V z_huR5Ls*F0tzZr~22$RsN_(A%TwrbSFDVwSIGS33<(a_jS`D=6wBnJs&QE!lZ&+*C z06@p^lZ?YxtTqB0Bj^c-oIVulfa*d3gm!?jFR)<_6X16+e~8-TD8TW|Kdg&R`v~Xq zAn`5AwMjOb>q{IUKUK~m$EVqoifal6Nl}TYHI<)p0toDLi7Ts3xLz3d%(aT_wv|HE zN~Qq$^9IUU?9Nnp$M%ldw`nyTABc_W<@oFeIO8TDqiLTc?jU@zz?81K4?Qnpl>;2&XWOGy$Qzl5nDX5C$Ee}6oPP|1TQlWUV>#Xrh$z6;P~%YEj{T6- zC78+F0QsFIdL%AYRF=j@c;*yowg4<4jx&-wOU556%G1$V2JN{x;C#ib>MK@qgKzO* zO-r<`dKLA3H6sL~%*%A9%JvHgOyd~?Kc|_ze0?#MtnE&%^-8a%a6!-U5>_i(YA)Pf zyMP0^5xVQIr!|({yK}i4PXM17`hV_H-mcYxhRQ{!mao4b0(vpU%PUu_Nv>17XjS{h zQ(JDCb~Z~7(kgXYg{6qvm63|t&k;)%olCnb?L3t^ApG$+;VT|iG~V{NAhy$-Dv$35 z+O$Ln5v;Ht#N+KSwOv#T82WFxbB6DjG1SGtvM;4p0CJ-wvFtxtxyJ{cZdI)hUgsvX z&fF3XPty`ExeEK-fsuiXpU`=kOQ*Y5ifAUCxfuLe| z#d+4=qV#(4ZKYHLj4pq)O1(`NF3?>;G{s&Ic@h^D_%t^U?YfTWfOf>7U|0e$kq=-q5y{Tnw`}Khx%9H(Chnfp=hM{^9L0 zwf%n4sAd{_DNr-E#Uys`{{XHg=M?*5^53Uix{W0V+QQPnjjLZAcgZnrL-fTgtyN3> zVAh!N=#y8UXiRO+lfq@wLVZ;#SnTh{e9qE~ZzY{QpU#~ug3{{WoLe0Y8jJe?id zEgz=0R^=5>8(ew%cZgrnskj?XrrQT6+@LY~4F3Rmm8R8Lwr2ZR)KwS^`8gx$7;d22 zO8_moa6RXYgX9koyjYpZlP2DqsY{ljrLL~QAsAM)-oG}CJMt{sk>h@YaG_7jcw5~JX;N$yZa=cN~9!lYkoK&TT8hT8o;m#|TsY+Jxo635QB&3{_KIjn ztyF?l@<1K{nOeTNS}O{b6-mx{1P>tm#dY05^t+i-wW|!fdu2!?^*#F;mtoW1E6NX1 zfZRDOe#f|-bsTqEW2osaHL(9=)eB} zjc`+pFlOL==B9?1WcM%#L03D5GC!Zp##l<;IoRjZLfV9TF0A^9ygjz$AGdGwHFUaa z7fRZVMwKqGagEqr+!9!ixZ-R-PSQ|S?zmc`z&r?NIx7@=mu@kYJbjGk@j85te^GF= zQ}{*O=?>@+6;uBJi67tl!}O2Ya(aW^th5X_f!ut}+BEufKEYG6kIi54O$zP_P&mOn z509kfmO5`6wJuueK~+oaQH2A!neTm+S{C}wBkgX?T*`6>w?8op&Dwy(s)OW_%)>@s zSwo*l5P?Sr<|C@HYX-+|Ob*tq52S;CV9a|&?5k6Cij9o7c0GWQ+TOCgfL5i0@X9>R zZ3QTyvv9(w+`#)s)9W$@m4&3OOcNn9j`^J*Of~48sr22BHg2 zH;F)vbFO_g1Cfl*XIazjurSB8_MK~K)}*bRdx?o_?W|Z70l@J!D>gZW(%qx0b8;uE z*AjUoM|GDVT!Wc_63d;9YH^=4Qf-~-wwJC6BX)kWmeHoNqU^YAWM%_TMd|@7+Bo!C zs2etg2f31FY0}HC+MJo8_-iO*y3YKZ_?n$NK~)*CumJwB9Y~{A#~6mpja)1JVz*LM zk|uVaseo`y)TZ^lah%U4toG(z9D{=_)rT1nuTc1d2OL78UQvFnL`XktU{(q zs1qG1~3SA|YmvY0e=1G20VikUGVVoQBNblQtEupQN>l0>;%q@dJ8+ zE$uR`$vjA|_d|kAZE6hD*6Btwh@*O4X~@aQnswY9aAzsGs%%XiF0VM}nVhDY+h0*i zx@R2B!i`fBGas()!}{VDQ%`RKX5B*v-m!{_RT+VXbsr)VN2n-9 z58eQh%LbNgd`UV${;gvthz4Q~1g@F)RymcjvJPVr+FW2ob&r-ZsdmB4UsHB+({eL! zPH(P~P~hTn*H&4>3BA>IYh$tlGJPD@O3$amEXCpEyQS!`*5jGIMWmo0@j23r90D+W zO?@SXs>bs@8*&d=8MPaa<%Xz`ow6YHu3n&2d4j6T6$G|4o7qyp2N{CCu*tb$+Ow$u z5zow{)99gGlk?8paA$f4o6eo%xJ5naRv!O`x&H z^ME0|_9lU&Cm%Ps0K6IekH8 z{o?Wr^<wj$#{HzLG1;Rc7gGH3>=qWn(cDta7Dt3;3f$xl(AvG} zi_yyg+)Q5EhqXE+ZR*vj-TYXXnx2@{Qc)R|m!2l(mr`|_d}oy$LD+%QHn9O)kWBFN zO;@{zEwQWV>wc3UDY3^7*@6E6GQ*@>r%KhXN38)agyEY%6JPN4QOX6w9E_gQ29n!v zTR^VmJTN{4^W^nn&eHm9Hw830aq1TQ-*`Yi~>A=?ltK!D@F7u8TRA9 z=w*{lr?c9dy@Y^=gE{Td%wMM0*J^$lKs>sh`pEGBpRL`v=QW#Z`3E%8)dykH0 zzu?^}4m9PqjCSYknT*$4qIZA5veQ;i3`pny0K~V_nD*-J&<%3E4xXK$6S;W94#du0 z#n)GL)uq?H`v`D(#yt1pZS?-Kx}&G6l10E_!#Cz04q^K4fa(=23dNhcDZni;D|Rt- zI<~0oVxF!)Fj98^04P^&ey!{PP~kx>>;QrCOrs4&9-TpJs>S?@*TGdv;1Ks}sK9M_WKa8n$A+0ERDm#yn^983%MzJ2CDhZ4d+fQ%j1$8zntI@IW za#f$*_KGn~3n@kn$At<&Bm5ZNpLMoZxoJUA$?|{v!JM&`pQ!*Atxd04<^44Puh9I& zt*WY3omK8oc&Ug7K5&1;=&sXIaa(IHn@>3YAbar_Z9RILuB!`b0nT`Dxs#omU{zqY z^XTp$D90pyezCr$wpFJZjXhzu3^8s<`v^O9hevjD_SuHn*Z@D&eCO%<%`_tYDvRp< z;2sZo&cU{3ms&MCRb~fz@Ohdvvs+fyVsY~_zv;G9d1`C9t32&rA9<&xs`X+`Z?Fz> zJ56JCcCwqAElun-Xa4|>9fy@U{*zVurjKDuE4eoe9M-?>2WncCf{v+?k%7)9cPdORTBKDB{GS}m zqWWDOtj%i>e&>uv30*GLtdd8VajHwXQWdd*^_e>DMT1fhg(tx$j7(*gnv-H{AAh-s z9Py5FW;M@qI4O~X-!k@IS&2eO&x45zO^$aE%6tRNt(Es?z|o*q-LB=Qr5u>GMslKx6)`=btOPz4;dIfCOcZMgixyIu*P#a?yJBZ98XUl70JsywCnX+x~Wd!3}gj>{{X4fq!(STM;YKwVvTxTr%jGMO2&6F+y}_` znb-A8TG3Nfd;VoSFW>Dw9}IH&Yv@#pwG}IlTNuGSK)?7R73?HoNlJ<2kVFiz14}aEi%Z|$Fj(alCSsO_#`Nko=F)KmFccPcwOKTXyFm!TiAY{U?r&txx^b-?Z+? z9FMf`$H`-!SZ+C*wVIaI78j~jf=g#PTxY*=+Euk>EA@R=zOjrnoX z+S>?y{y^*xfsWkFD>NFqZdPP3TBi@{g;U3Td{1UM$C~4!%ha^@AtfA|j`7@q_xeQb z>h)>c)1b4Xdy20-{Z1nKifdi=Ie${V_UH3wIQIVlv;uUibyjVyl?nXWIKrRe`(v5X zXBe@$PQ67Tnc00(Rat;ien1|4O&WTzusTqu)K6i4NIc`bosHqE( zIBa~8j{auF-A3wFK`Ia&F?YdZjFLW)*ggLMS9PMpVxYkG?qRr`@;L4K&uRL;k?m1x zDz{^hTff&h{bqGqq02qmZJo*r2=SbT$0T`z{APg|wwkbAM*jd)aREsAA0Klwv!_xn zqMu@7z=I%?4%73Gq^*ml)m4h^Otu|>Q;hlgLly^2B!rS))@!54o2dBAS~(? zQW=gIg~}<&-^c{_$H_62g6(5!!ctAkTasIl?s@O@lY6G3%Ie!vC>Y$TT=(Wu{-b82 zPiUfr^zH*G#!fr-$A9q_({z1xwwmZyCek+XlB#pdO5F`?OS(-OlyOcZB#tw0N7d&NMXQ2h^q^%HuZ*=;U#_^GMV z43ds}{GNOG?=f?csf&Qp{vkseaIWX`5&4(zDb(`pp5=Y|JyBRq`G{{a60 z)4eN9rK?uCQ1Mc+!25kfR?eNK(rs$cmnV(G@@MbJ&&)nZ*T-4CJErT6C9F$RPFFjD z&T>zWryNGtNv6|V>R-CoJdvHi9}`BMmm>uNz^%0kwtt}gd(8b+8uHszS7Mgh6k`MK zBj*Q=A5E&#xhkLwSMw%vTmF1vGpAuqTY@TJ5xvevPu_F4bj@;HuU%%xCm=2rYadgd z`2O&C z_5SkIY3XY1TAf3`v@OR0LEwOYUgj-coHdNCVNlrx1AcF$kos+1mb<>Cwv6FH?Ee5v zc8yuARjmN7s>c}&IC+rz9bIlIuDzfg#1oUxl785lYtj0a7e{WMrBbDW?i^(JA|6LV z*RL4~{{U_4=Z|nB{{Y0@(`%{LQY=R(+fTC@6OPpBF3^+~hpqPEO9O`kfWt7_UZgbj z0Pe+Klk}CwQ!S3A!quFz7Vg2DApQP1j0Rij^#dpXVS(pq`^-u;HC;yGdg0CqIphJv zV^4mR0a2_gF~;I}=dfTuzuscwBU38XrD~xRFjBUXc=CS!B3g>+^%K&moRgJMbC1?I zntH3#Kdo{;dxaZ+PJ6}ew!J<*%Aj^mPDj`4Io&tk*dnqo31Yt9%27}D5xVr^tL%Ev zlz+}k;|J_d^ouQ`+bi^4s;P!ZL7%>S!}{rvH!Gs{AH+XjnT+pFm`=jeUZYiTDj#U} zf^afs7gb$d%*k~`B8=g?0r#30vL>;Y^DV;)-Hv`H2D?{XmpvCXn-JkL#(7?in9$~m zoV~k==RYudix`6Y@S~iA4yr8CJE_iB0DOth-DQH*Ur+#B7@Ct~%S{@Fs^OGhXUlmJ zr0H+HJwgB{fi)B^n?X>mK>%fmi&NBnQ)Tb@K)@3&lGy^iY0-Z0a(fKS`#OTN=Q)pc zB}*#YSPXMHr>q;WNdU=tR`c~gRe)CK7!yOKRb64Nyz}B<=<7nr2Z-hOSOyocnXTh? z(ds(UQ!dk$1BuRSHFw$>f&0b+U3+_u5Pe}R(}bak@*^p=f-gD4oaQrC$hg2Cq(@sp zs?7M7rI&68dA)Syw$(cDppBwkb+x~@w8M5e81D$WtT+WoBfK=!u9a_8uQ|`_Gxb)R af!shI1WC0|cATw5gA@&M?%9Tp-~ZW0!i`)2 literal 0 HcmV?d00001 From c8941202f2c39e80fae8db0e2e63c10495997b5c Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 9 Oct 2026 11:14:12 +0200 Subject: [PATCH 07/34] integration-notes: drop the tokenizer config workaround (not needed with transformers 5.x) Co-Authored-By: Claude Fable 5.1 --- integration-notes/make_flux3_modular_repo.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/integration-notes/make_flux3_modular_repo.py b/integration-notes/make_flux3_modular_repo.py index 615d18a6ecf8..2e9422e33849 100644 --- a/integration-notes/make_flux3_modular_repo.py +++ b/integration-notes/make_flux3_modular_repo.py @@ -13,7 +13,6 @@ import argparse import os -import shutil import subprocess import sys @@ -81,9 +80,4 @@ repo_id=args.push_to_hub, private=True, ) -# `AutoProcessor.from_pretrained(..., subfolder="tokenizer")` needs the model config next to the processor files -# under the pinned transformers (see the README); the package already ships it, keep it -shutil.copy( - os.path.join(args.package, "tokenizer", "config.json"), os.path.join(args.output, "tokenizer", "config.json") -) print(f"saved {args.output}") From 0a774136f798ec676d991c9bae8a5215bb299f48 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Fri, 9 Oct 2026 20:39:57 +0200 Subject: [PATCH 08/34] Address review: rename the mode channel dict and the final norm, name the reference workflow image_conditioned, document the guidance scale Co-Authored-By: Claude Fable 5.1 --- docs/source/en/api/pipelines/flux3.md | 14 ++++++++++++-- scripts/convert_flux3_to_diffusers.py | 2 +- .../models/transformers/transformer_flux3.py | 12 ++++++------ .../flux3/modular_blocks_flux3.py | 4 ++-- .../flux3/test_modular_pipeline_flux3.py | 12 ++++++------ 5 files changed, 27 insertions(+), 17 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index 254e05b63719..edf88afeab38 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -74,15 +74,25 @@ combined = pipe( Pass `height` and/or `width` to override the derived size. +## Guidance scale + +The pipeline applies classifier-free guidance through its `guider` component and does not take a `guidance_scale` argument at call time. The default guider is [`ClassifierFreeGuidance`] with `guidance_scale=3.5`, the recommended setting for both workflows. To change it, create an updated copy of the guider: + +```python +pipe.guider = pipe.guider.new(guidance_scale=5.0) +``` + +See the [guiders guide](../../using-diffusers/guiders) to switch to a different guidance method. + ## Workflows -The pipeline serves two workflows, `text2image` and `image2image`; the one that runs is picked by the inputs (`image` selects image-to-image). [`~ModularPipeline.from_pretrained`] can also build a pipeline for one workflow with the `workflow` argument, which then only expects and loads the components that workflow needs: +The pipeline serves two workflows, `text2image` and `image_conditioned`; the one that runs is picked by the inputs (`image` selects image-to-image). [`~ModularPipeline.from_pretrained`] can also build a pipeline for one workflow with the `workflow` argument, which then only expects and loads the components that workflow needs: ```python from diffusers.modular_pipelines import Flux3AutoBlocks print(Flux3AutoBlocks().available_workflows) -# ['text2image', 'image2image'] +# ["text2image", "image_conditioned"] t2i = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small", workflow="text2image") print(t2i.blocks.doc) diff --git a/scripts/convert_flux3_to_diffusers.py b/scripts/convert_flux3_to_diffusers.py index a93b2f80d0ff..1ebacf613eb4 100644 --- a/scripts/convert_flux3_to_diffusers.py +++ b/scripts/convert_flux3_to_diffusers.py @@ -89,7 +89,7 @@ def transformer_config_from_package(package_config): def swap_scale_shift(weight): - # original adaLN weights produce [shift, scale]; diffusers Flux3AdaLayerNormOut expects [scale, shift] + # original adaLN weights produce [shift, scale]; diffusers Flux3AdaLayerNorm expects [scale, shift] shift, scale = weight.chunk(2, dim=0) return torch.cat([scale, shift], dim=0) diff --git a/src/diffusers/models/transformers/transformer_flux3.py b/src/diffusers/models/transformers/transformer_flux3.py index 2f5fbd91956b..6bd81f21aee5 100644 --- a/src/diffusers/models/transformers/transformer_flux3.py +++ b/src/diffusers/models/transformers/transformer_flux3.py @@ -322,7 +322,7 @@ def forward(self, ids: torch.Tensor) -> torch.Tensor: return freqs_cos, freqs_sin -class Flux3AdaLayerNormOut(nn.Module): +class Flux3AdaLayerNorm(nn.Module): """Output adaptive layer norm conditioned on a per-token embedding of shape `(batch_size, seq_len, dim)`.""" def __init__(self, dim: int, eps: float = 1e-6, bias: bool = False): @@ -399,8 +399,8 @@ def __init__( super().__init__() self.inner_dim = num_attention_heads * attention_head_dim - in_channels = {"image": in_channels, "image_cond": cond_in_channels} - modes = list(in_channels.keys()) + in_channels_per_mode = {"image": in_channels, "image_cond": cond_in_channels} + modes = list(in_channels_per_mode.keys()) # 1. Sinusoidal positional embedding for RoPE on all token modes self.pos_embed = Flux3PosEmbed(theta=rope_theta, axes_dim=list(axes_dims_rope)) @@ -421,7 +421,7 @@ def __init__( # 4. Input projections self.x_embedders = nn.ModuleDict( - {name: nn.Linear(channels, self.inner_dim, bias=False) for name, channels in in_channels.items()} + {name: nn.Linear(channels, self.inner_dim, bias=False) for name, channels in in_channels_per_mode.items()} ) self.context_embedder = nn.Linear(joint_attention_dim, self.inner_dim, bias=False) @@ -471,8 +471,8 @@ def __init__( ) # 7. Output layers (image tokens only) - self.norm_out = Flux3AdaLayerNormOut(self.inner_dim, eps=eps) - self.proj_out = nn.Linear(self.inner_dim, in_channels["image"], bias=False) + self.norm_out = Flux3AdaLayerNorm(self.inner_dim, eps=eps) + self.proj_out = nn.Linear(self.inner_dim, in_channels, bias=False) self.gradient_checkpointing = False diff --git a/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py b/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py index bb049e300c47..2ef1b48c3596 100644 --- a/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py +++ b/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py @@ -326,7 +326,7 @@ class Flux3AutoBlocks(SequentialPipelineBlocks): Supported workflows: - `text2image`: requires `prompt` - - `image2image`: requires `prompt`, `image` + - `image_conditioned`: requires `prompt`, `image` Components: text_encoder (`Qwen3VLForConditionalGeneration`) tokenizer (`AutoProcessor`) image_processor @@ -383,7 +383,7 @@ class Flux3AutoBlocks(SequentialPipelineBlocks): block_names = AUTO_BLOCKS.keys() _workflow_map = { "text2image": {"prompt": True}, - "image2image": {"prompt": True, "image": True}, + "image_conditioned": {"prompt": True, "image": True}, } @property diff --git a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py index 5b370e929314..9f348a8a8fbf 100644 --- a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py +++ b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py @@ -49,7 +49,7 @@ ("denoise.after_denoise", "Flux3UnpackLatentsStep"), ("decode", "Flux3DecodeStep"), ], - "image2image": [ + "image_conditioned": [ ("text_encoder", "Flux3TextEncoderStep"), ("vae_encoder.preprocess", "Flux3ProcessImagesInputStep"), ("vae_encoder.encode", "Flux3VaeEncoderStep"), @@ -77,7 +77,7 @@ } # the recommended sampling settings the blocks pin on the scheduler / guider they create FLUX3_SAMPLING = {"scheduler": {"shift": 6.93}, "guider": {"guidance_scale": 3.5}} -# every optional input with its declared default. text2image declares the 1024 x 1024 size; image2image derives the +# every optional input with its declared default. text2image declares the 1024 x 1024 size; image_conditioned derives the # size from the first reference image, so its branch declares none FLUX3_INPUTS = { "negative_prompt": None, @@ -92,7 +92,7 @@ "output_type": "pil", } FLUX3_TEXT2IMAGE_INPUTS = {**FLUX3_INPUTS, "height": 1024, "width": 1024} -FLUX3_IMAGE2IMAGE_INPUTS = {**FLUX3_INPUTS, "height": None, "width": None} +FLUX3_IMAGE_CONDITIONED_INPUTS = {**FLUX3_INPUTS, "height": None, "width": None} FLUX3_WORKFLOW_DEFAULTS = { "text2image": { "components": FLUX3_COMPONENTS, @@ -100,10 +100,10 @@ "inputs": FLUX3_TEXT2IMAGE_INPUTS, "component_configs": FLUX3_SAMPLING, }, - "image2image": { + "image_conditioned": { "components": FLUX3_COMPONENTS, "required_inputs": ["prompt", "image"], - "inputs": FLUX3_IMAGE2IMAGE_INPUTS, + "inputs": FLUX3_IMAGE_CONDITIONED_INPUTS, "component_configs": FLUX3_SAMPLING, }, } @@ -166,7 +166,7 @@ def test_image_to_image_default_size(self): # without a size, image-to-image generates at the default area (1024 x 1024 for the real model) at the first # reference's aspect ratio, rounded to the token grid; a given side overrides its default. The default area is # shrunk to 64 x 64 for the tiny model. - blocks = self.pipeline_blocks_class().get_workflow("image2image") + blocks = self.pipeline_blocks_class().get_workflow("image_conditioned") blocks.sub_blocks["denoise.cond_input"] = Flux3ImageCondInputStep(default_height=64, default_width=64) pipe = blocks.init_pipeline(self.pretrained_model_name_or_path) pipe.load_components(dtype=torch.float32) From be156325d19857ecbc349eb317b48a2bce24a559 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Tue, 13 Oct 2026 00:58:16 +0200 Subject: [PATCH 09/34] Add the step-distilled FLUX.3 image model (flow map) - transformer: `flow_map` config and `target_timestep_embedder`; forward takes `target_timestep` (the timestep each step jumps to), added to the modulation of every mode and the final layer - convert script: map `ref_time_in.*` and read `flow_map` from the package config - modular: dedicated `Flux3Distilled*` blocks (text encoder, text input, text RoPE inputs, denoiser) with no guider and no negative prompt, 16 steps by default; `Flux3DistilledAutoBlocks` and `Flux3DistilledModularPipeline` - tests for the flow-map transformer path and the distilled modular pipeline - docs: step-distilled section and autodoc entries Co-Authored-By: Claude Fable 5.1 --- docs/source/en/api/pipelines/flux3.md | 23 ++ integration-notes/README.md | 29 ++ integration-notes/make_flux3_modular_repo.py | 26 +- integration-notes/make_tiny_flux3_repo.py | 20 +- integration-notes/parity_bf16.py | 15 +- scripts/convert_flux3_to_diffusers.py | 4 + src/diffusers/__init__.py | 4 + .../models/transformers/transformer_flux3.py | 32 +- src/diffusers/modular_pipelines/__init__.py | 9 +- .../modular_pipelines/flux3/__init__.py | 6 +- .../modular_pipelines/flux3/before_denoise.py | 45 ++- .../modular_pipelines/flux3/denoise.py | 140 ++++++++ .../modular_pipelines/flux3/encoders.py | 67 ++++ .../modular_pipelines/flux3/inputs.py | 67 ++++ .../flux3/modular_blocks_flux3_distilled.py | 333 ++++++++++++++++++ .../flux3/modular_pipeline.py | 8 + .../modular_pipelines/modular_pipeline.py | 1 + .../dummy_torch_and_transformers_objects.py | 30 ++ .../test_models_transformer_flux3.py | 23 ++ .../flux3/test_modular_pipeline_flux3.py | 100 +++++- 20 files changed, 957 insertions(+), 25 deletions(-) create mode 100644 src/diffusers/modular_pipelines/flux3/modular_blocks_flux3_distilled.py diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index edf88afeab38..ed55f07ea74c 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -84,6 +84,21 @@ pipe.guider = pipe.guider.new(guidance_scale=5.0) See the [guiders guide](../../using-diffusers/guiders) to switch to a different guidance method. +## Step-distilled model + +The step-distilled checkpoint generates in 16 steps without classifier-free guidance. Its transformer is a flow map: at every step it is told the current timestep and the one the step jumps to. It loads like the base model (the repo's `modular_model_index.json` resolves to [`Flux3DistilledModularPipeline`] and [`Flux3DistilledAutoBlocks`]); the pipeline has no `guider` and does not take `negative_prompt`: + + +```python +pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small-distilled") +pipe.load_components(dtype=torch.bfloat16) +pipe.to("cuda") + +image = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] +``` + +It serves the same two workflows as the base model. `num_inference_steps` defaults to 16; the model is trained for 4 to 16 steps. + ## Workflows The pipeline serves two workflows, `text2image` and `image_conditioned`; the one that runs is picked by the inputs (`image` selects image-to-image). [`~ModularPipeline.from_pretrained`] can also build a pipeline for one workflow with the `workflow` argument, which then only expects and loads the components that workflow needs: @@ -115,3 +130,11 @@ pipe.enable_auto_cpu_offload(device="cuda") ## Flux3AutoBlocks [[autodoc]] Flux3AutoBlocks + +## Flux3DistilledAutoBlocks + +[[autodoc]] Flux3DistilledAutoBlocks + +## Flux3DistilledModularPipeline + +[[autodoc]] Flux3DistilledModularPipeline diff --git a/integration-notes/README.md b/integration-notes/README.md index 1c0a71c20aac..e7be21305901 100644 --- a/integration-notes/README.md +++ b/integration-notes/README.md @@ -32,6 +32,14 @@ python scripts/convert_flux2_to_diffusers.py --vae \ was built). `flux3_examples.py` runs the four parity cases as doc examples against that repo, without any reference code. +`--variant sd` builds the step-distilled repo from `variants/sd/` of the package (`flow_map: true` in its config; +the converter maps `ref_time_in.*` to `target_timestep_embedder.*`), with `Flux3DistilledAutoBlocks` in the index: + +```bash +.venv/bin/python integration-notes/make_flux3_modular_repo.py \ + --package /raid/yiyi/f3-image-small --variant sd --output /raid/yiyi/flux3-image-distilled-modular +``` + ## Parity (`parity_bf16.py`) bf16, 50 steps, guidance 3.5, shift 6.93, seed 0, same prompts and the same initial noise on both sides. @@ -55,6 +63,27 @@ reference left, diffusers right). The differences are bf16 rounding over 50 step implementations agree to within 2/255 (checked during development: the text encoder is bit-identical and the transformer agrees to 4e-5 in fp32). +### Step-distilled variant (`F3_VARIANT=sd`) + +Same four cases with the `sd` package variant: 16 steps, no guidance, BFL's flow-map branch +(`/raid/yiyi/bfl-f3-flowmap`, their PR #5) against `flux3-image-distilled-modular`: + +```bash +F3_VARIANT=sd CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 /raid/yiyi/venv-tf457/bin/python integration-notes/parity_bf16.py reference +F3_VARIANT=sd CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/parity_bf16.py diffusers +``` + +| case | size | mean pixel diff | max | pixels off by > 30 | +|---|---|---|---|---| +| t2i | 1024 x 1024 | 1.74/255 | 221/255 | 1.8 % | +| i2i | 1024 x 1024 | 0.21/255 | 12/255 | 0 % | +| bbox_t2i | 1024 x 1024 | 0.46/255 | 247/255 | 0.2 % | +| bbox_i2i | 1360 x 768 | 0.31/255 | 89/255 | 0.03 % | + +Visually identical (`parity-bf16-sd/*_side_by_side.png`). The t2i case drifts more than with the base model +(1.8 % of pixels differ by more than 30, fine fur texture only): 16 large Euler steps amplify bf16 rounding more +than 50 small ones, and the attention kernels differ (cuDNN in the reference, PyTorch's default here). + ## Why the reference runs under transformers 4.57.6 Qwen3-VL computes the position id of the padding tokens differently across transformers versions (1 in diff --git a/integration-notes/make_flux3_modular_repo.py b/integration-notes/make_flux3_modular_repo.py index 2e9422e33849..d228701b53a0 100644 --- a/integration-notes/make_flux3_modular_repo.py +++ b/integration-notes/make_flux3_modular_repo.py @@ -5,13 +5,17 @@ .venv/bin/python integration-notes/make_flux3_modular_repo.py \ --package /raid/yiyi/f3-image-small --output /raid/yiyi/flux3-image-modular-release \ - [--push_to_hub diffusers-internal-dev/f3-image-diffusers] + [--variant sd] [--push_to_hub diffusers-internal-dev/f3-image-diffusers] + +`--variant` takes the DiT weights and config from `variants//` of the package (the shared encoders stay at the +root); a flow-map config (the step-distilled `sd` variant) gets `Flux3DistilledAutoBlocks`. Scheduler, guider and image processor are `from_config` components created by the blocks (shift 6.93, guidance 3.5), so they are not stored in the repo. """ import argparse +import json import os import subprocess import sys @@ -24,28 +28,36 @@ from transformers import AutoProcessor, Qwen3VLForConditionalGeneration # noqa: E402 from diffusers import AutoencoderKLFlux2, Flux3Transformer2DModel # noqa: E402 -from diffusers.modular_pipelines import Flux3AutoBlocks # noqa: E402 +from diffusers.modular_pipelines import Flux3AutoBlocks, Flux3DistilledAutoBlocks # noqa: E402 parser = argparse.ArgumentParser() parser.add_argument( "--package", required=True, - help="the BFL package directory (config.json, model.safetensors, ae.safetensors, text_encoder/, tokenizer/)", + help="the BFL package directory (config.json, model.safetensors, ae.safetensors, text_encoder/, variants/)", ) +parser.add_argument("--variant", default=None, help="package variant (variants//), e.g. sd") parser.add_argument("--output", required=True) parser.add_argument("--push_to_hub", default=None, help="repo id to push to (private)") args = parser.parse_args() python = sys.executable +dit_dir = os.path.join(args.package, "variants", args.variant) if args.variant else args.package +with open(os.path.join(dit_dir, "config.json")) as f: + package_config = json.load(f) +blocks_class = Flux3DistilledAutoBlocks if package_config["dit"].get("flow_map") else Flux3AutoBlocks +# newer packages keep the tokenizer files in text_encoder/ +tokenizer_dir = os.path.join(args.package, package_config["text_encoder"].get("tokenizer", "tokenizer")) + subprocess.run( [ python, os.path.join(REPO, "scripts", "convert_flux3_to_diffusers.py"), "--checkpoint_path", - os.path.join(args.package, "model.safetensors"), + os.path.join(dit_dir, "model.safetensors"), "--config_path", - os.path.join(args.package, "config.json"), + os.path.join(dit_dir, "config.json"), "--output_path", args.output, ], @@ -64,14 +76,14 @@ check=True, ) -pipe = Flux3AutoBlocks().init_pipeline() +pipe = blocks_class().init_pipeline() pipe.update_components( transformer=Flux3Transformer2DModel.from_pretrained(args.output, subfolder="transformer", dtype=torch.bfloat16), vae=AutoencoderKLFlux2.from_pretrained(args.output, subfolder="vae"), # fp32, as shipped text_encoder=Qwen3VLForConditionalGeneration.from_pretrained( os.path.join(args.package, "text_encoder"), dtype=torch.bfloat16 ), - tokenizer=AutoProcessor.from_pretrained(os.path.join(args.package, "tokenizer")), + tokenizer=AutoProcessor.from_pretrained(tokenizer_dir), ) pipe.save_pretrained( args.output, diff --git a/integration-notes/make_tiny_flux3_repo.py b/integration-notes/make_tiny_flux3_repo.py index fb93931db304..4f149a332002 100644 --- a/integration-notes/make_tiny_flux3_repo.py +++ b/integration-notes/make_tiny_flux3_repo.py @@ -10,7 +10,10 @@ (hidden 16, 2 layers, real Qwen vocab/processor files); tests pass text_encoder_out_layers=(1,), so joint_attention_dim = 16 * 1 = 16. -Upload later as hf-internal-testing/tiny-flux3-image-modular and point the tests there. +With `--flow_map`, builds the step-distilled counterpart instead (a `flow_map` transformer and +`Flux3DistilledAutoBlocks`) under ../tiny-flux3-image-distilled-modular. + +Upload later as hf-internal-testing/tiny-flux3-image-modular (and -distilled-) and point the tests there. """ import json @@ -27,7 +30,15 @@ from diffusers import AutoencoderKLFlux2, Flux3Transformer2DModel, ModularPipeline # noqa: E402 -OUT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "tiny-flux3-image-modular")) +FLOW_MAP = "--flow_map" in sys.argv +OUT = os.path.abspath( + os.path.join( + os.path.dirname(__file__), + "..", + "..", + "tiny-flux3-image-distilled-modular" if FLOW_MAP else "tiny-flux3-image-modular", + ) +) TINY_QWEN3VL = "trl-internal-testing/tiny-Qwen3VLForConditionalGeneration" torch.manual_seed(0) @@ -66,12 +77,13 @@ joint_attention_dim=text_hidden, # tests pass text_encoder_out_layers=(1,) timestep_channels=32, axes_dims_rope=(2, 2, 2, 2), + flow_map=FLOW_MAP, ) transformer.save_pretrained(os.path.join(OUT, "transformer")) index = { - "_class_name": "Flux3ModularPipeline", - "_blocks_class_name": "Flux3AutoBlocks", + "_class_name": "Flux3DistilledModularPipeline" if FLOW_MAP else "Flux3ModularPipeline", + "_blocks_class_name": "Flux3DistilledAutoBlocks" if FLOW_MAP else "Flux3AutoBlocks", "_diffusers_version": "0.40.0.dev0", } for name, lib, cls in [ diff --git a/integration-notes/parity_bf16.py b/integration-notes/parity_bf16.py index 4af12da89e5d..32f100c2cd45 100644 --- a/integration-notes/parity_bf16.py +++ b/integration-notes/parity_bf16.py @@ -10,6 +10,9 @@ Cases: t2i and i2i (the fox), and BFL's two layout-control examples (bbox regions in the caption; the knight edit with , which also exercises the i2i size derived from the reference). Writes _reference.png, _diffusers.png and _side_by_side.png under OUT, and prints the mean pixel difference. + +`F3_VARIANT=sd` runs the step-distilled variant instead (16 steps, no CFG): BFL's flow-map branch against the converted +`flux3-image-distilled-modular` repo, under OUT-sd. """ import os @@ -21,9 +24,11 @@ REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) +VARIANT = os.environ.get("F3_VARIANT") # None (bf16 base) or "sd" (step-distilled) PACKAGE = "/raid/yiyi/f3-image-small" # the BFL package -MODULAR_REPO = "/raid/yiyi/flux3-image-modular" # the converted diffusers repo -OUT = "/raid/yiyi/diffusers-scratch/flux3-image/parity-bf16" +BFL_CODE = os.path.join(REPO, "..", "bfl-f3-flowmap" if VARIANT else "bfl-f3-image") # the reference implementation +MODULAR_REPO = "/raid/yiyi/flux3-image-distilled-modular" if VARIANT == "sd" else "/raid/yiyi/flux3-image-modular" +OUT = "/raid/yiyi/diffusers-scratch/flux3-image/parity-bf16" + (f"-{VARIANT}" if VARIANT else "") SEED = 0 HEIGHT = WIDTH = 1024 @@ -53,11 +58,11 @@ def reference(): - sys.path.insert(0, os.path.join(REPO, "..", "bfl-f3-image")) + sys.path.insert(0, BFL_CODE) from f3.core.hub import resolve_package from f3.image_small.pipeline import ImageSmallPipeline, default_size, prepare_reference, to_pil, to_tensor - package = resolve_package(PACKAGE, verify=False) + package = resolve_package(PACKAGE, variant=VARIANT, verify=False) pipe = ImageSmallPipeline.from_package(package, device="cuda") for name, prompt, image, size in CASES: references = [] @@ -65,7 +70,7 @@ def reference(): path = os.path.join(OUT, "t2i_reference.png") if image == "t2i" else image references = [prepare_reference(PIL.Image.open(path))] width, height = (size[1], size[0]) if size else default_size(references, package.config) - # defaults from the package config: 50 steps, guidance 3.5, alpha 6.93 + # defaults from the package config: 50 steps, guidance 3.5, alpha 6.93 (sd: 16 steps, no guidance) result = to_pil(pipe.generate(prompt, [to_tensor(r) for r in references], width, height, seed=SEED)) result.save(os.path.join(OUT, f"{name}_reference.png")) print(f"{name}: {result.size}") diff --git a/scripts/convert_flux3_to_diffusers.py b/scripts/convert_flux3_to_diffusers.py index 1ebacf613eb4..6aadc298d250 100644 --- a/scripts/convert_flux3_to_diffusers.py +++ b/scripts/convert_flux3_to_diffusers.py @@ -85,6 +85,7 @@ def transformer_config_from_package(package_config): "mlp_ratio": dit["mlp_ratio"], "axes_dims_rope": tuple(dit["axes_dim"]), "rope_theta": dit["theta"], + "flow_map": dit.get("flow_map", False), } @@ -118,6 +119,9 @@ def swap_scale_shift(weight): KEY_MAPPING = { "time_in.in_layer.weight": "timestep_embedder.linear_1.weight", "time_in.out_layer.weight": "timestep_embedder.linear_2.weight", + # flow map (step-distilled packages): the embedder of the timestep each step jumps to + "ref_time_in.in_layer.weight": "target_timestep_embedder.linear_1.weight", + "ref_time_in.out_layer.weight": "target_timestep_embedder.linear_2.weight", # final_layer.image.linear -> proj_out; final_layer.image.adaLN_modulation.1 -> norm_out.linear ([shift, scale] -> [scale, shift]) "final_layer.image.linear.weight": "proj_out.weight", "final_layer.image.adaLN_modulation.1.weight": "norm_out.linear.weight", diff --git a/src/diffusers/__init__.py b/src/diffusers/__init__.py index 5668afbc66ba..35ee907960c4 100644 --- a/src/diffusers/__init__.py +++ b/src/diffusers/__init__.py @@ -527,6 +527,8 @@ "Flux2KleinModularPipeline", "Flux2ModularPipeline", "Flux3AutoBlocks", + "Flux3DistilledAutoBlocks", + "Flux3DistilledModularPipeline", "Flux3ModularPipeline", "FluxAutoBlocks", "FluxKontextAutoBlocks", @@ -1386,6 +1388,8 @@ Flux2KleinModularPipeline, Flux2ModularPipeline, Flux3AutoBlocks, + Flux3DistilledAutoBlocks, + Flux3DistilledModularPipeline, Flux3ModularPipeline, FluxAutoBlocks, FluxKontextAutoBlocks, diff --git a/src/diffusers/models/transformers/transformer_flux3.py b/src/diffusers/models/transformers/transformer_flux3.py index 6bd81f21aee5..202c29fe13e2 100644 --- a/src/diffusers/models/transformers/transformer_flux3.py +++ b/src/diffusers/models/transformers/transformer_flux3.py @@ -346,7 +346,9 @@ class Flux3Transformer2DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, Attenti passes through its own per-mode transformer blocks (`mode_blocks` for the image modes, `txt_mode_blocks` for the text context), then the text context and the active image modes are concatenated for the joint single blocks, and only the image tokens pass through the output layers. The modulation is per mode: the image tokens are modulated by - the flow timestep, the reference image tokens and the text context by timestep `0`. + the flow timestep, the reference image tokens and the text context by timestep `0`. A flow-map checkpoint + (`flow_map=True`, the step-distilled models) additionally embeds the timestep each denoising step jumps to and adds + it to every mode's modulation. Args: in_channels (`int`, defaults to `128`): @@ -373,6 +375,9 @@ class Flux3Transformer2DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, Attenti The base frequency of the rotary positional embeddings. eps (`float`, defaults to `1e-6`): The epsilon of the layer norms and the QK norms. + flow_map (`bool`, defaults to `False`): + Whether the model is a flow map (step-distilled): adds the `target_timestep_embedder` and requires + `target_timestep` in the forward pass. """ _supports_gradient_checkpointing = True @@ -395,6 +400,7 @@ def __init__( axes_dims_rope: tuple[int, ...] = (32, 32, 32, 32), rope_theta: int = 10000, eps: float = 1e-6, + flow_map: bool = False, ): super().__init__() @@ -410,6 +416,12 @@ def __init__( self.timestep_embedder = TimestepEmbedding( in_channels=timestep_channels, time_embed_dim=self.inner_dim, sample_proj_bias=False ) + # flow map (step-distilled checkpoints): a second embedder for the timestep each step jumps to + self.target_timestep_embedder = ( + TimestepEmbedding(in_channels=timestep_channels, time_embed_dim=self.inner_dim, sample_proj_bias=False) + if flow_map + else None + ) # 3. Modulation: one head per mode (+ text) for the per-mode blocks and for the joint blocks self.early_stream_modulations = nn.ModuleDict( @@ -486,6 +498,7 @@ def forward( txt_ids: torch.Tensor, image_cond_hidden_states: torch.Tensor | None = None, image_cond_ids: torch.Tensor | None = None, + target_timestep: torch.Tensor | None = None, attention_kwargs: dict[str, Any] | None = None, return_dict: bool = True, ) -> Transformer2DModelOutput | tuple[torch.Tensor]: @@ -509,6 +522,9 @@ def forward( VAE-encoded reference image tokens (clean conditioning content, for image-to-image generation). image_cond_ids (`torch.Tensor` of shape `(cond_seq_len, 4)`, *optional*): Position ids of the reference image tokens. Required with `image_cond_hidden_states`. + target_timestep (`torch.Tensor` of shape `(batch_size,)`, *optional*): + Flow timestep in `[0, 1]` that the current denoising step jumps to. Required by, and only by, a flow-map + model (`flow_map=True`). attention_kwargs (`dict`, *optional*): A kwargs dictionary whose `scale` entry, if specified, scales the LoRA layers for this forward pass. return_dict (`bool`, *optional*, defaults to `True`): @@ -524,6 +540,10 @@ def forward( raise ValueError("`image_cond_hidden_states` requires `image_cond_ids`.") if image_cond_hidden_states is None and image_cond_ids is not None: raise ValueError("`image_cond_ids` was passed without `image_cond_hidden_states`.") + if (target_timestep is not None) != bool(self.config.flow_map): + raise ValueError( + "`target_timestep` is required by, and only by, a flow-map transformer (`flow_map=True`)." + ) batch_size, image_seq_len = hidden_states.shape[:2] dtype = hidden_states.dtype @@ -531,9 +551,17 @@ def forward( # 1. Timestep embeddings: the image tokens at `timestep`, the text context and the reference tokens at 0. # Each is a per-token embedding (batch_size, seq_len, dim) since the modulation is per token. image_temb = self.time_proj(timestep.to(dtype) * 1000) - image_temb = self.timestep_embedder(image_temb.to(dtype))[:, None, :].expand(batch_size, image_seq_len, -1) + image_temb = self.timestep_embedder(image_temb.to(dtype))[:, None, :] zero_temb = self.time_proj(torch.zeros(batch_size, device=hidden_states.device, dtype=dtype)) zero_temb = self.timestep_embedder(zero_temb.to(dtype))[:, None, :] + if target_timestep is not None: + # flow map: the embedding of the timestep this step jumps to is added to every mode's embedding, so it + # reaches the per-mode blocks, the joint blocks and the output layer + target_temb = self.time_proj(target_timestep.to(dtype) * 1000) + target_temb = self.target_timestep_embedder(target_temb.to(dtype))[:, None, :] + image_temb = image_temb + target_temb + zero_temb = zero_temb + target_temb + image_temb = image_temb.expand(batch_size, image_seq_len, -1) zero_temb_act = F.silu(zero_temb) # 2. Text context: the two modulation sets, input projection and RoPE diff --git a/src/diffusers/modular_pipelines/__init__.py b/src/diffusers/modular_pipelines/__init__.py index b87974c79e8b..4a7dc0a7ba02 100644 --- a/src/diffusers/modular_pipelines/__init__.py +++ b/src/diffusers/modular_pipelines/__init__.py @@ -83,6 +83,8 @@ ] _import_structure["flux3"] = [ "Flux3AutoBlocks", + "Flux3DistilledAutoBlocks", + "Flux3DistilledModularPipeline", "Flux3ModularPipeline", ] _import_structure["ideogram4"] = [ @@ -172,7 +174,12 @@ Flux2KleinModularPipeline, Flux2ModularPipeline, ) - from .flux3 import Flux3AutoBlocks, Flux3ModularPipeline + from .flux3 import ( + Flux3AutoBlocks, + Flux3DistilledAutoBlocks, + Flux3DistilledModularPipeline, + Flux3ModularPipeline, + ) from .helios import ( HeliosAutoBlocks, HeliosModularPipeline, diff --git a/src/diffusers/modular_pipelines/flux3/__init__.py b/src/diffusers/modular_pipelines/flux3/__init__.py index 21d39cc60ce3..2eeae562077b 100644 --- a/src/diffusers/modular_pipelines/flux3/__init__.py +++ b/src/diffusers/modular_pipelines/flux3/__init__.py @@ -22,7 +22,8 @@ _dummy_objects.update(get_objects_from_module(dummy_torch_and_transformers_objects)) else: _import_structure["modular_blocks_flux3"] = ["Flux3AutoBlocks"] - _import_structure["modular_pipeline"] = ["Flux3ModularPipeline"] + _import_structure["modular_blocks_flux3_distilled"] = ["Flux3DistilledAutoBlocks"] + _import_structure["modular_pipeline"] = ["Flux3DistilledModularPipeline", "Flux3ModularPipeline"] if TYPE_CHECKING or DIFFUSERS_SLOW_IMPORT: try: @@ -32,7 +33,8 @@ from ...utils.dummy_torch_and_transformers_objects import * # noqa F403 else: from .modular_blocks_flux3 import Flux3AutoBlocks - from .modular_pipeline import Flux3ModularPipeline + from .modular_blocks_flux3_distilled import Flux3DistilledAutoBlocks + from .modular_pipeline import Flux3DistilledModularPipeline, Flux3ModularPipeline else: import sys diff --git a/src/diffusers/modular_pipelines/flux3/before_denoise.py b/src/diffusers/modular_pipelines/flux3/before_denoise.py index 08cc2195349f..bf6428dc58f4 100644 --- a/src/diffusers/modular_pipelines/flux3/before_denoise.py +++ b/src/diffusers/modular_pipelines/flux3/before_denoise.py @@ -92,9 +92,11 @@ def retrieve_timesteps( class Flux3SetTimestepsStep(ModularPipelineBlocks): model_name = "flux3" - def __init__(self, shift: float = 6.93): - # the constant timestep shift ("alpha") of the FLUX.3 sampling recipe, configured on the scheduler + def __init__(self, shift: float = 6.93, num_inference_steps: int = 50): + # the constant timestep shift ("alpha") of the FLUX.3 sampling recipe, configured on the scheduler, and the + # default number of steps (50; 16 for the step-distilled checkpoints) self.shift = shift + self.num_inference_steps = num_inference_steps super().__init__() @property @@ -118,7 +120,7 @@ def description(self) -> str: @property def inputs(self) -> list[InputParam]: return [ - InputParam.template("num_inference_steps"), + InputParam.template("num_inference_steps", default=self.num_inference_steps), InputParam.template("timesteps", note="overrides `num_inference_steps` when provided"), InputParam.template("sigmas", note="overrides `num_inference_steps` when provided"), ] @@ -395,3 +397,40 @@ def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> Pi self.set_block_state(state, block_state) return components, state + + +class Flux3DistilledTextRoPEInputsStep(Flux3TextRoPEInputsStep): + model_name = "flux3-distilled" + + @property + def description(self) -> str: + return ( + "Step that prepares the 4D RoPE position IDs of the text tokens for the step-distilled FLUX.3 model (no " + "negative prompt). Place it after the text encoder step." + ) + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template( + "prompt_embeds", + description="Text embeddings used to guide the generation. Can be generated from text_encoder step.", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + name="txt_ids", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="4D position IDs (T, H, W, L) for text tokens, used for RoPE calculation.", + ), + ] + + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + block_state.txt_ids = self._prepare_text_ids(block_state.prompt_embeds).to(block_state.prompt_embeds.device) + self.set_block_state(state, block_state) + return components, state diff --git a/src/diffusers/modular_pipelines/flux3/denoise.py b/src/diffusers/modular_pipelines/flux3/denoise.py index 52948b0f3bfa..6e8487a5f7a7 100644 --- a/src/diffusers/modular_pipelines/flux3/denoise.py +++ b/src/diffusers/modular_pipelines/flux3/denoise.py @@ -138,6 +138,76 @@ def __call__( return components, block_state +class Flux3DistilledLoopDenoiser(ModularPipelineBlocks): + model_name = "flux3-distilled" + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ComponentSpec("transformer", Flux3Transformer2DModel)] + + @property + def description(self) -> str: + return ( + "Step within the denoising loop that predicts the velocity of the image tokens with a step-distilled " + "(flow-map) FLUX.3 transformer: no classifier-free guidance, and each step also passes the transformer the " + "timestep it jumps to. The static conditioning fields (`prompt_embeds`, `txt_ids`, `img_ids`, the " + "reference image tokens and their ids) arrive via `denoiser_input_fields`. " + "This block should be used to compose the `sub_blocks` attribute of a `LoopSequentialPipelineBlocks` " + "object (e.g. `Flux3DistilledDenoiseStep`)" + ) + + @property + def inputs(self) -> list[tuple[str, Any]]: + return [ + InputParam( + "latents", + required=True, + type_hint=torch.Tensor, + description="The image latent tokens to denoise. Shape: (B, seq_len, C)", + ), + InputParam.template("denoiser_input_fields"), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [OutputParam("noise_pred", type_hint=torch.Tensor, description="The predicted velocity for this step")] + + @torch.no_grad() + def __call__( + self, components: Flux3ModularPipeline, block_state: BlockState, i: int, t: torch.Tensor + ) -> PipelineState: + denoiser_input_fields = block_state.denoiser_input_fields + + # transformer kwargs = every collected static field matching the forward signature (`img_ids`, `txt_ids`, and + # the `image_cond_*` fields of image-to-image generation) + transformer_args = set(inspect.signature(components.transformer.forward).parameters.keys()) + transformer_kwargs = { + field_name: field_value + for field_name, field_value in denoiser_input_fields.items() + if field_name in transformer_args + } + + latents = block_state.latents.to(components.transformer.dtype) + batch_size = latents.shape[0] + # the flow timestep in [0, 1] (scheduler timesteps are sigmas * 1000), and the one this step jumps to: the next + # scheduler timestep, 0 after the last + timesteps = block_state.timesteps + next_t = timesteps[i + 1] if i + 1 < len(timesteps) else torch.zeros_like(t) + timestep = (t / 1000).expand(batch_size).to(latents.dtype) + target_timestep = (next_t / 1000).expand(batch_size).to(latents.dtype) + + block_state.noise_pred = components.transformer( + hidden_states=latents, + encoder_hidden_states=denoiser_input_fields["prompt_embeds"], + timestep=timestep, + target_timestep=target_timestep, + return_dict=False, + **transformer_kwargs, + )[0] + + return components, block_state + + class Flux3LoopAfterDenoiser(ModularPipelineBlocks): model_name = "flux3" @@ -276,3 +346,73 @@ def description(self) -> str: " - `Flux3LoopAfterDenoiser`\n" "This block supports text-to-image and image-to-image generation." ) + + +# auto_docstring +class Flux3DistilledDenoiseStep(Flux3DenoiseLoopWrapper): + """ + Denoise step that iteratively denoises the latents for the step-distilled (flow-map) FLUX.3 model. + Its loop logic is defined in `Flux3DenoiseLoopWrapper.__call__` method At each iteration, it runs blocks defined + in `sub_blocks` sequentially: + - `Flux3DistilledLoopDenoiser` + - `Flux3LoopAfterDenoiser` + This block supports text-to-image and image-to-image generation, without classifier-free guidance. + + Components: + transformer (`Flux3Transformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) + + Inputs: + timesteps (`Tensor`): + The timesteps to use for the denoising process. + num_inference_steps (`int`): + The number of inference steps to use for the denoising process. + latents (`Tensor`): + The image latent tokens to denoise. Shape: (B, seq_len, C) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + The denoised latents + """ + + model_name = "flux3-distilled" + """ + Denoise step that iteratively denoises the latents for the step-distilled (flow-map) FLUX.3 model. + Its loop logic is defined in `Flux3DenoiseLoopWrapper.__call__` method At each iteration, it runs blocks defined + in `sub_blocks` sequentially: + - `Flux3DistilledLoopDenoiser` + - `Flux3LoopAfterDenoiser` + This block supports text-to-image and image-to-image generation, without classifier-free guidance. + + Components: + transformer (`Flux3Transformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) + + Inputs: + timesteps (`Tensor`): + The timesteps to use for the denoising process. + num_inference_steps (`int`): + The number of inference steps to use for the denoising process. + latents (`Tensor`): + The image latent tokens to denoise. Shape: (B, seq_len, C) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + The denoised latents + """ + + block_classes = [Flux3DistilledLoopDenoiser(), Flux3LoopAfterDenoiser()] + block_names = ["denoiser", "after_denoiser"] + + @property + def description(self) -> str: + return ( + "Denoise step that iteratively denoises the latents for the step-distilled (flow-map) FLUX.3 model. \n" + "Its loop logic is defined in `Flux3DenoiseLoopWrapper.__call__` method \n" + "At each iteration, it runs blocks defined in `sub_blocks` sequentially:\n" + " - `Flux3DistilledLoopDenoiser`\n" + " - `Flux3LoopAfterDenoiser`\n" + "This block supports text-to-image and image-to-image generation, without classifier-free guidance." + ) diff --git a/src/diffusers/modular_pipelines/flux3/encoders.py b/src/diffusers/modular_pipelines/flux3/encoders.py index 5bd1be7accd8..b336233254ca 100644 --- a/src/diffusers/modular_pipelines/flux3/encoders.py +++ b/src/diffusers/modular_pipelines/flux3/encoders.py @@ -326,3 +326,70 @@ def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> Pi self.set_block_state(state, block_state) return components, state + + +class Flux3DistilledTextEncoderStep(Flux3TextEncoderStep): + model_name = "flux3-distilled" + + @property + def description(self) -> str: + return ( + "Text Encoder step that generates text embeddings using Qwen3-VL to guide the generation with the " + "step-distilled FLUX.3 model, which samples without classifier-free guidance (no negative prompt). Takes " + f"one prompt per call; it goes through the chat template and is right-padded to a multiple of " + f"{self.pad_multiple} tokens." + ) + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template("prompt", description="The prompt to guide the generation (one prompt per call)"), + InputParam.template( + "max_sequence_length", + default=8192, + description="Maximum number of prompt tokens; longer prompts are truncated.", + ), + InputParam( + "text_encoder_out_layers", + type_hint=tuple[int], + default=(4, 8, 12, 16, 20, 24, 28, 32), + required=False, + description="Indices of the text-encoder hidden layers whose states are stacked to form the text embeddings", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "prompt_embeds", + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Text embeddings (stacked Qwen3-VL hidden states) used to guide the image generation", + ), + ] + + @staticmethod + def check_inputs(block_state): + if not isinstance(block_state.prompt, str): + raise ValueError( + f"`prompt` must be a str; batched prompts are not supported, got {type(block_state.prompt).__name__}." + ) + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + self.check_inputs(block_state) + + block_state.prompt_embeds = self._get_qwen3_vl_prompt_embeds( + text_encoder=components.text_encoder, + tokenizer=components.tokenizer, + prompt=block_state.prompt, + device=components._execution_device, + max_sequence_length=block_state.max_sequence_length, + pad_multiple=self.pad_multiple, + hidden_states_layers=block_state.text_encoder_out_layers, + ) + + self.set_block_state(state, block_state) + return components, state diff --git a/src/diffusers/modular_pipelines/flux3/inputs.py b/src/diffusers/modular_pipelines/flux3/inputs.py index 0e08f7173797..f4ab66553a6e 100644 --- a/src/diffusers/modular_pipelines/flux3/inputs.py +++ b/src/diffusers/modular_pipelines/flux3/inputs.py @@ -173,6 +173,73 @@ def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> Pi return components, state +class Flux3DistilledTextInputStep(ModularPipelineBlocks): + model_name = "flux3-distilled" + + @property + def description(self) -> str: + return ( + "This step:\n" + " 1. Determines `batch_size` from `prompt_embeds` and the compute `dtype` from the transformer\n" + " 2. Repeats the text embeddings to the batch size (batch_size * num_images_per_prompt)\n" + "For the step-distilled FLUX.3 model, which samples without classifier-free guidance (no negative " + "embeddings)." + ) + + @property + def expected_components(self) -> list[ComponentSpec]: + return [ComponentSpec("transformer", Flux3Transformer2DModel)] + + @property + def inputs(self) -> list[InputParam]: + return [ + InputParam.template("num_images_per_prompt"), + InputParam( + "prompt_embeds", + required=True, + kwargs_type="denoiser_input_fields", + type_hint=torch.Tensor, + description="Pre-generated text embeddings. Can be generated from text_encoder step.", + ), + ] + + @property + def intermediate_outputs(self) -> list[OutputParam]: + return [ + OutputParam( + "batch_size", + type_hint=int, + description="Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt", + ), + OutputParam( + "dtype", + type_hint=torch.dtype, + description="Data type of model tensor inputs (the transformer's dtype)", + ), + OutputParam( + "prompt_embeds", + type_hint=torch.Tensor, + kwargs_type="denoiser_input_fields", + description="Text embeddings used to guide the image generation", + ), + ] + + @torch.no_grad() + def __call__(self, components: Flux3ModularPipeline, state: PipelineState) -> PipelineState: + block_state = self.get_block_state(state) + + block_state.batch_size = block_state.prompt_embeds.shape[0] + # the denoiser consumes every tensor this chain prepares, so its dtype is the compute dtype + block_state.dtype = components.transformer.dtype + + block_state.prompt_embeds = repeat_tensor_to_batch_size( + "prompt_embeds", block_state.prompt_embeds, block_state.batch_size, block_state.num_images_per_prompt + ).to(block_state.dtype) + + self.set_block_state(state, block_state) + return components, state + + class Flux3ImageCondInputStep(ModularPipelineBlocks): model_name = "flux3" diff --git a/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3_distilled.py b/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3_distilled.py new file mode 100644 index 000000000000..aa08bf6ae2a6 --- /dev/null +++ b/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3_distilled.py @@ -0,0 +1,333 @@ +# Copyright 2026 Black Forest Labs and The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from ...utils import logging +from ..modular_pipeline import AutoPipelineBlocks, SequentialPipelineBlocks +from ..modular_pipeline_utils import InsertableDict, OutputParam +from .before_denoise import ( + Flux3DistilledTextRoPEInputsStep, + Flux3PrepareImageCondLatentsStep, + Flux3PrepareLatentsStep, + Flux3SetTimestepsStep, +) +from .decoders import Flux3DecodeStep, Flux3UnpackLatentsStep +from .denoise import Flux3DistilledDenoiseStep +from .encoders import Flux3DistilledTextEncoderStep +from .inputs import Flux3DistilledTextInputStep, Flux3ImageCondInputStep +from .modular_blocks_flux3 import Flux3AutoVaeEncoderStep + + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + + +# The step-distilled (flow-map) FLUX.3 checkpoints: 16 steps and no classifier-free guidance. The text encoder, +# text input, text RoPE and denoise steps have distilled versions without the negative prompt; the denoise loop +# passes the transformer the timestep each step jumps to and runs a single pass. Everything else is shared with the +# base model. + +### +### Core denoise +### + +Flux3DistilledCoreDenoiseBlocks = InsertableDict( + [ + ("input", Flux3DistilledTextInputStep()), + ("prepare_latents", Flux3PrepareLatentsStep()), + ("set_timesteps", Flux3SetTimestepsStep(num_inference_steps=16)), + ("prepare_rope_inputs", Flux3DistilledTextRoPEInputsStep()), + ("denoise", Flux3DistilledDenoiseStep()), + ("after_denoise", Flux3UnpackLatentsStep()), + ] +) + + +# auto_docstring +class Flux3DistilledCoreDenoiseStep(SequentialPipelineBlocks): + """ + Core denoise step that performs the denoising process for the step-distilled FLUX.3 model, for text-to-image + generation. + + Components: + transformer (`Flux3Transformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) + + Inputs: + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + prompt_embeds (`Tensor`): + Pre-generated text embeddings. Can be generated from text_encoder step. + height (`int`, *optional*, defaults to 1024): + The height in pixels of the generated image. + width (`int`, *optional*, defaults to 1024): + The width in pixels of the generated image. + latents (`Tensor`, *optional*): + Pre-generated noisy latents for image generation. + generator (`Generator`, *optional*): + Torch generator for deterministic generation. + num_inference_steps (`int`, *optional*, defaults to 16): + The number of denoising steps. + timesteps (`Tensor`, *optional*): + Timesteps for the denoising process. (overrides `num_inference_steps` when provided) + sigmas (`list`, *optional*): + Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + Denoised latents. + """ + + model_name = "flux3-distilled" + block_classes = Flux3DistilledCoreDenoiseBlocks.values() + block_names = Flux3DistilledCoreDenoiseBlocks.keys() + + @property + def description(self): + return ( + "Core denoise step that performs the denoising process for the step-distilled FLUX.3 model, for " + "text-to-image generation." + ) + + @property + def outputs(self): + return [ + OutputParam.template("latents"), + ] + + +Flux3DistilledImageConditionedCoreDenoiseBlocks = InsertableDict( + [ + ("input", Flux3DistilledTextInputStep()), + ("cond_input", Flux3ImageCondInputStep()), + ("prepare_latents", Flux3PrepareLatentsStep()), + ("prepare_image_cond_latents", Flux3PrepareImageCondLatentsStep()), + ("set_timesteps", Flux3SetTimestepsStep(num_inference_steps=16)), + ("prepare_rope_inputs", Flux3DistilledTextRoPEInputsStep()), + ("denoise", Flux3DistilledDenoiseStep()), + ("after_denoise", Flux3UnpackLatentsStep()), + ] +) + + +# auto_docstring +class Flux3DistilledImageConditionedCoreDenoiseStep(SequentialPipelineBlocks): + """ + Core denoise step that performs the denoising process for the step-distilled FLUX.3 model with reference images + (image-to-image). + + Components: + transformer (`Flux3Transformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) + + Inputs: + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + prompt_embeds (`Tensor`): + Pre-generated text embeddings. Can be generated from text_encoder step. + image_cond_latents (`list`): + Normalized packed latents of the reference images, one tensor of shape (B, C, height_i//16, width_i//16) + per image + height (`int`, *optional*): + The height in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + width (`int`, *optional*): + The width in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + latents (`Tensor`, *optional*): + Pre-generated noisy latents for image generation. + generator (`Generator`, *optional*): + Torch generator for deterministic generation. + num_inference_steps (`int`, *optional*, defaults to 16): + The number of denoising steps. + timesteps (`Tensor`, *optional*): + Timesteps for the denoising process. (overrides `num_inference_steps` when provided) + sigmas (`list`, *optional*): + Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + Denoised latents. + """ + + model_name = "flux3-distilled" + block_classes = Flux3DistilledImageConditionedCoreDenoiseBlocks.values() + block_names = Flux3DistilledImageConditionedCoreDenoiseBlocks.keys() + + @property + def description(self): + return ( + "Core denoise step that performs the denoising process for the step-distilled FLUX.3 model with reference " + "images (image-to-image)." + ) + + @property + def outputs(self): + return [ + OutputParam.template("latents"), + ] + + +# auto_docstring +class Flux3DistilledAutoCoreDenoiseStep(AutoPipelineBlocks): + """ + Core denoise step that selects the step-distilled FLUX.3 denoising workflow by the conditioning inputs. + - `Flux3DistilledImageConditionedCoreDenoiseStep` is used when `image_cond_latents` is provided + (image-to-image). + - `Flux3DistilledCoreDenoiseStep` is used otherwise (text-to-image). + + Components: + transformer (`Flux3Transformer2DModel`) scheduler (`FlowMatchEulerDiscreteScheduler`) + + Inputs: + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + prompt_embeds (`Tensor`): + Pre-generated text embeddings. Can be generated from text_encoder step. + image_cond_latents (`list`, *optional*): + Normalized packed latents of the reference images, one tensor of shape (B, C, height_i//16, width_i//16) + per image + height (`int`, *optional*, defaults to None or 1024, depending on the workflow): + The height in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + width (`int`, *optional*, defaults to None or 1024, depending on the workflow): + The width in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + latents (`Tensor`): + Pre-generated noisy latents for image generation. + generator (`Generator`, *optional*): + Torch generator for deterministic generation. + num_inference_steps (`int`): + The number of denoising steps. + timesteps (`Tensor`): + Timesteps for the denoising process. (overrides `num_inference_steps` when provided) + sigmas (`list`, *optional*): + Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + + Outputs: + latents (`Tensor`): + Denoised latents. + """ + + model_name = "flux3-distilled" + block_classes = [Flux3DistilledImageConditionedCoreDenoiseStep, Flux3DistilledCoreDenoiseStep] + block_names = ["image_conditioned", "text2image"] + block_trigger_inputs = ["image_cond_latents", None] + + @property + def description(self): + return ( + "Core denoise step that selects the step-distilled FLUX.3 denoising workflow by the conditioning inputs.\n" + " - `Flux3DistilledImageConditionedCoreDenoiseStep` is used when `image_cond_latents` is provided " + "(image-to-image).\n" + " - `Flux3DistilledCoreDenoiseStep` is used otherwise (text-to-image)." + ) + + +### +### Auto blocks +### + +DISTILLED_AUTO_BLOCKS = InsertableDict( + [ + ("text_encoder", Flux3DistilledTextEncoderStep()), + ("vae_encoder", Flux3AutoVaeEncoderStep()), + ("denoise", Flux3DistilledAutoCoreDenoiseStep()), + ("decode", Flux3DecodeStep()), + ] +) + + +# auto_docstring +class Flux3DistilledAutoBlocks(SequentialPipelineBlocks): + """ + Auto blocks that perform text-to-image and image-to-image generation using the step-distilled FLUX.3 model (16 + steps, no classifier-free guidance). + + Supported workflows: + - `text2image`: requires `prompt` + - `image_conditioned`: requires `prompt`, `image` + + Components: + text_encoder (`Qwen3VLForConditionalGeneration`) tokenizer (`AutoProcessor`) image_processor + (`Flux3ImageProcessor`) vae (`AutoencoderKLFlux2`) transformer (`Flux3Transformer2DModel`) scheduler + (`FlowMatchEulerDiscreteScheduler`) + + Inputs: + prompt (`str`): + The prompt to guide the generation (one prompt per call) + max_sequence_length (`int`, *optional*, defaults to 8192): + Maximum number of prompt tokens; longer prompts are truncated. + text_encoder_out_layers (`tuple`, *optional*, defaults to (4, 8, 12, 16, 20, 24, 28, 32)): + Indices of the text-encoder hidden layers whose states are stacked to form the text embeddings + image (`Image | list`, *optional*): + The reference image or images to condition the generation on; each keeps its own resolution + (center-cropped to the 16-pixel grid) + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + image_cond_latents (`list`, *optional*): + Normalized packed latents of the reference images, one tensor of shape (B, C, height_i//16, width_i//16) + per image + height (`int`, *optional*, defaults to None or 1024, depending on the workflow): + The height in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + width (`int`, *optional*, defaults to None or 1024, depending on the workflow): + The width in pixels of the generated image; defaults to the first reference image's aspect ratio at the + default area + latents (`Tensor`): + Pre-generated noisy latents for image generation. + generator (`Generator`, *optional*): + Torch generator for deterministic generation. + num_inference_steps (`int`): + The number of denoising steps. + timesteps (`Tensor`): + Timesteps for the denoising process. (overrides `num_inference_steps` when provided) + sigmas (`list`, *optional*): + Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) + **denoiser_input_fields (`None`, *optional*): + conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + output_type (`str`, *optional*, defaults to pil): + Output format: 'pil', 'np', 'pt'. + + Outputs: + images (`list`): + Generated images. + latents (`Tensor`): + Denoised latents. + """ + + model_name = "flux3-distilled" + block_classes = DISTILLED_AUTO_BLOCKS.values() + block_names = DISTILLED_AUTO_BLOCKS.keys() + _workflow_map = { + "text2image": {"prompt": True}, + "image_conditioned": {"prompt": True, "image": True}, + } + + @property + def description(self): + return ( + "Auto blocks that perform text-to-image and image-to-image generation using the step-distilled FLUX.3 " + "model (16 steps, no classifier-free guidance)." + ) + + @property + def outputs(self): + return [ + OutputParam.template("images"), + OutputParam.template("latents"), + ] diff --git a/src/diffusers/modular_pipelines/flux3/modular_pipeline.py b/src/diffusers/modular_pipelines/flux3/modular_pipeline.py index 574a3ace6f7b..c91777ff723c 100644 --- a/src/diffusers/modular_pipelines/flux3/modular_pipeline.py +++ b/src/diffusers/modular_pipelines/flux3/modular_pipeline.py @@ -46,3 +46,11 @@ def requires_unconditional_embeds(self): if getattr(self, "guider", None) is None: return False return self.guider._enabled and self.guider.num_conditions > 1 + + +class Flux3DistilledModularPipeline(Flux3ModularPipeline): + """ + A ModularPipeline for the step-distilled FLUX.3 model (16 steps, no classifier-free guidance). + """ + + default_blocks_name = "Flux3DistilledAutoBlocks" diff --git a/src/diffusers/modular_pipelines/modular_pipeline.py b/src/diffusers/modular_pipelines/modular_pipeline.py index d3717e878210..3c9593e22312 100644 --- a/src/diffusers/modular_pipelines/modular_pipeline.py +++ b/src/diffusers/modular_pipelines/modular_pipeline.py @@ -141,6 +141,7 @@ def _helios_pyramid_map_fn(config_dict=None): ("flux2", _create_default_map_fn("Flux2ModularPipeline")), ("flux2-klein", _flux2_klein_map_fn), ("flux3", _create_default_map_fn("Flux3ModularPipeline")), + ("flux3-distilled", _create_default_map_fn("Flux3DistilledModularPipeline")), ("ideogram4", _create_default_map_fn("Ideogram4ModularPipeline")), ("krea2", _krea2_map_fn), ("qwenimage", _create_default_map_fn("QwenImageModularPipeline")), diff --git a/src/diffusers/utils/dummy_torch_and_transformers_objects.py b/src/diffusers/utils/dummy_torch_and_transformers_objects.py index ac3676f4448b..870b63fd9628 100644 --- a/src/diffusers/utils/dummy_torch_and_transformers_objects.py +++ b/src/diffusers/utils/dummy_torch_and_transformers_objects.py @@ -227,6 +227,36 @@ def from_pretrained(cls, *args, **kwargs): requires_backends(cls, ["torch", "transformers"]) +class Flux3DistilledAutoBlocks(metaclass=DummyObject): + _backends = ["torch", "transformers"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch", "transformers"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch", "transformers"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch", "transformers"]) + + +class Flux3DistilledModularPipeline(metaclass=DummyObject): + _backends = ["torch", "transformers"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch", "transformers"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch", "transformers"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch", "transformers"]) + + class Flux3ModularPipeline(metaclass=DummyObject): _backends = ["torch", "transformers"] diff --git a/tests/models/transformers/test_models_transformer_flux3.py b/tests/models/transformers/test_models_transformer_flux3.py index 3538739f7c38..6832a516827b 100644 --- a/tests/models/transformers/test_models_transformer_flux3.py +++ b/tests/models/transformers/test_models_transformer_flux3.py @@ -13,6 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. +import pytest import torch from diffusers import Flux3Transformer2DModel @@ -108,6 +109,28 @@ def test_image_cond_inputs(self): text_only = model(**{k: v for k, v in inputs.items() if not k.startswith("image_cond")}, return_dict=False)[0] assert not torch.allclose(output, text_only) + @torch.no_grad() + def test_flow_map(self): + torch.manual_seed(0) + model = self.model_class(**self.get_init_dict(), flow_map=True).to(torch_device).eval() + inputs = self.get_dummy_inputs() + target_timestep = torch.full_like(inputs["timestep"], 0.25) + + output = model(**inputs, target_timestep=target_timestep, return_dict=False)[0] + assert output.shape == inputs["hidden_states"].shape + + # the target timestep changes the prediction + other = model(**inputs, target_timestep=torch.zeros_like(target_timestep), return_dict=False)[0] + assert not torch.allclose(output, other) + + # a flow map requires the target timestep; a plain model refuses it + with pytest.raises(ValueError, match="target_timestep"): + model(**inputs, return_dict=False) + plain = self.model_class(**self.get_init_dict()).to(torch_device).eval() + assert plain.target_timestep_embedder is None + with pytest.raises(ValueError, match="target_timestep"): + plain(**inputs, target_timestep=target_timestep, return_dict=False) + class TestFlux3TransformerMemory(Flux3TransformerTesterConfig, MemoryTesterMixin): pass diff --git a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py index 9f348a8a8fbf..f870c4ef6486 100644 --- a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py +++ b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py @@ -18,7 +18,12 @@ import pytest import torch -from diffusers.modular_pipelines import Flux3AutoBlocks, Flux3ModularPipeline +from diffusers.modular_pipelines import ( + Flux3AutoBlocks, + Flux3DistilledAutoBlocks, + Flux3DistilledModularPipeline, + Flux3ModularPipeline, +) from diffusers.modular_pipelines.flux3.encoders import Flux3TextEncoderStep from diffusers.modular_pipelines.flux3.inputs import Flux3ImageCondInputStep from diffusers.utils import logging @@ -37,6 +42,7 @@ # TODO: local tiny repo built by integration-notes/make_tiny_flux3_repo.py — upload as # hf-internal-testing/tiny-flux3-image-modular and switch before opening the PR FLUX3_TINY_REPO = "/raid/yiyi/tiny-flux3-image-modular" +FLUX3_DISTILLED_TINY_REPO = "/raid/yiyi/tiny-flux3-image-distilled-modular" FLUX3_WORKFLOWS = { "text2image": [ @@ -108,6 +114,54 @@ }, } +# the step-distilled model: the same graph with distilled text encoder / text input / text RoPE / denoise steps (no +# guider, no negative prompt, 16 steps by default) +FLUX3_DISTILLED_WORKFLOWS = { + "text2image": [ + ("text_encoder", "Flux3DistilledTextEncoderStep"), + ("denoise.input", "Flux3DistilledTextInputStep"), + ("denoise.prepare_latents", "Flux3PrepareLatentsStep"), + ("denoise.set_timesteps", "Flux3SetTimestepsStep"), + ("denoise.prepare_rope_inputs", "Flux3DistilledTextRoPEInputsStep"), + ("denoise.denoise", "Flux3DistilledDenoiseStep"), + ("denoise.after_denoise", "Flux3UnpackLatentsStep"), + ("decode", "Flux3DecodeStep"), + ], + "image_conditioned": [ + ("text_encoder", "Flux3DistilledTextEncoderStep"), + ("vae_encoder.preprocess", "Flux3ProcessImagesInputStep"), + ("vae_encoder.encode", "Flux3VaeEncoderStep"), + ("denoise.input", "Flux3DistilledTextInputStep"), + ("denoise.cond_input", "Flux3ImageCondInputStep"), + ("denoise.prepare_latents", "Flux3PrepareLatentsStep"), + ("denoise.prepare_image_cond_latents", "Flux3PrepareImageCondLatentsStep"), + ("denoise.set_timesteps", "Flux3SetTimestepsStep"), + ("denoise.prepare_rope_inputs", "Flux3DistilledTextRoPEInputsStep"), + ("denoise.denoise", "Flux3DistilledDenoiseStep"), + ("denoise.after_denoise", "Flux3UnpackLatentsStep"), + ("decode", "Flux3DecodeStep"), + ], +} +FLUX3_DISTILLED_COMPONENTS = {name: cls for name, cls in FLUX3_COMPONENTS.items() if name != "guider"} +FLUX3_DISTILLED_SAMPLING = {"scheduler": {"shift": 6.93}} +# no negative prompt, 16 steps by default +FLUX3_DISTILLED_INPUTS = {name: value for name, value in FLUX3_INPUTS.items() if name != "negative_prompt"} +FLUX3_DISTILLED_INPUTS["num_inference_steps"] = 16 +FLUX3_DISTILLED_WORKFLOW_DEFAULTS = { + "text2image": { + "components": FLUX3_DISTILLED_COMPONENTS, + "required_inputs": ["prompt"], + "inputs": {**FLUX3_DISTILLED_INPUTS, "height": 1024, "width": 1024}, + "component_configs": FLUX3_DISTILLED_SAMPLING, + }, + "image_conditioned": { + "components": FLUX3_DISTILLED_COMPONENTS, + "required_inputs": ["prompt", "image"], + "inputs": {**FLUX3_DISTILLED_INPUTS, "height": None, "width": None}, + "component_configs": FLUX3_DISTILLED_SAMPLING, + }, +} + class Flux3ModularPipelineTesterConfig(BaseModularPipelineTesterConfig): pipeline_class = Flux3ModularPipeline @@ -267,3 +321,47 @@ class TestFlux3ModularPipelineMemory(Flux3ModularPipelineTesterConfig, ModularMe class TestFlux3ModularPipelineWorkflow(Flux3ModularPipelineTesterConfig, ModularWorkflowTesterMixin): pass + + +class Flux3DistilledModularPipelineTesterConfig(Flux3ModularPipelineTesterConfig): + pipeline_class = Flux3DistilledModularPipeline + pipeline_blocks_class = Flux3DistilledAutoBlocks + pretrained_model_name_or_path = FLUX3_DISTILLED_TINY_REPO + expected_workflow_blocks = FLUX3_DISTILLED_WORKFLOWS + expected_workflow_defaults = FLUX3_DISTILLED_WORKFLOW_DEFAULTS + + +class TestFlux3DistilledModularPipelineFast(Flux3DistilledModularPipelineTesterConfig, ModularPipelineTesterMixin): + @pytest.mark.skip(reason="FLUX.3 takes one prompt per call; batched prompts are not supported.") + def test_inference_batch_consistent(self): + pass + + @pytest.mark.skip(reason="FLUX.3 takes one prompt per call; batched prompts are not supported.") + def test_inference_batch_single_identical(self): + pass + + def test_num_images_per_prompt(self): + super().test_num_images_per_prompt(batch_sizes=[1]) + + def test_no_guider_no_negative_prompt(self): + # the step-distilled model samples without classifier-free guidance: no guider, and the block set neither + # takes a negative prompt nor produces negative embeddings + pipe = self.get_pipeline() + assert "guider" not in pipe.component_names + blocks = self.pipeline_blocks_class() + assert {"negative_prompt", "negative_prompt_embeds"}.isdisjoint({p.name for p in blocks.inputs}) + assert {"negative_prompt_embeds", "negative_txt_ids"}.isdisjoint({p.name for p in blocks.intermediate_outputs}) + + def test_image_to_image(self): + pipe = self.get_pipeline() + inputs = {**self.get_dummy_inputs(), "image": self.get_reference_images(sizes=((64, 48), (32, 32)))} + images = pipe(**inputs, output="images") + assert images.shape == (1, 3, 64, 64) + + +class TestFlux3DistilledModularPipelineLoading(Flux3DistilledModularPipelineTesterConfig, ModularLoadingTesterMixin): + pass + + +class TestFlux3DistilledModularPipelineWorkflow(Flux3DistilledModularPipelineTesterConfig, ModularWorkflowTesterMixin): + pass From ef53f7b52f17cb3f1f2e29c0f3f1945949bcbaa8 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Tue, 13 Oct 2026 01:30:45 +0200 Subject: [PATCH 10/34] Add the step-distilled examples script Same four cases as flux3_examples.py, run against the distilled Hub repo. Co-Authored-By: Claude Fable 5.1 --- integration-notes/flux3_distilled_examples.py | 82 +++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 integration-notes/flux3_distilled_examples.py diff --git a/integration-notes/flux3_distilled_examples.py b/integration-notes/flux3_distilled_examples.py new file mode 100644 index 000000000000..c1546ac55f04 --- /dev/null +++ b/integration-notes/flux3_distilled_examples.py @@ -0,0 +1,82 @@ +"""FLUX.3 image step-distilled examples: the same four cases as `flux3_examples.py` (text-to-image, image-to-image, +and BFL's two layout-control examples) run with the step-distilled model. + +Only the repo changes. The distilled pipeline has no guider and takes no negative prompt, and runs 16 steps by default. + +Standalone: loads the modular pipeline from the Hub repo and writes four PNGs under `out/`. + + CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/flux3_distilled_examples.py +""" + +import os + +import torch + +from diffusers import ModularPipeline +from diffusers.utils import load_image + + +REPO = os.environ.get("F3_REPO", "diffusers-internal-dev/f3-small-image-distilled") +OUT = os.environ.get("F3_OUT", "out") +os.makedirs(OUT, exist_ok=True) + +pipe = ModularPipeline.from_pretrained(REPO) +pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) # the image VAE runs in fp32 +pipe.to("cuda") + +# text-to-image: 1024 x 1024, 16 steps by default, no guidance. FLUX.3 responds best to full scene captions. +prompt = ( + "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest clearing at golden hour. " + "The fox is centered in the frame, its thick orange winter coat contrasting with the fresh white snow, breath " + "visible in the cold air. Snow-laden pine trees fill the softly blurred background, and low warm sunlight rakes " + "across the scene from the left, casting long blue shadows." +) +fox = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] +fox.save(os.path.join(OUT, "t2i.png")) + +# image-to-image: the reference keeps its own resolution and the output defaults to its aspect ratio at ~1 MP. +# Describe the target image as a complete caption of the reference with the edit folded in. +prompt = ( + "The same red fox in the same snow-covered forest clearing, now at night under a full moon: cool blue moonlight, " + "deep shadows, stars visible through the pine branches, the fox's eyes catching the light." +) +night = pipe(prompt=prompt, image=fox, generator=torch.Generator().manual_seed(0), output="images")[0] +night.save(os.path.join(OUT, "i2i.png")) + +# layout control: append a JSON list of regions to the caption. `bbox` is [y0, x0, y1, x1] normalized to 0-1000, +# so it is independent of the image size; tag a region in the caption with and describe it in `desc`. +prompt = ( + "Minimalist graphic illustration featuring a black silhouette of a person centered against a " + "solid, vibrant chartreuse background . The figure is captured in a dynamic, mid-stride running " + "pose, facing toward the left side of the frame. " + '[{"id": "background_1", "bbox": [0, 0, 1000, 1000], "desc": "A flat field of neon yellow-green with a subtle ' + 'paper texture and fine, organic grain."}, {"id": "silhouette_1", "bbox": [150, 150, 850, 850], "desc": "A ' + "black silhouette of a person in motion, made of dense stippled ink. The leading edges are solid; the trailing " + 'arm and leg dissolve into coarse square pixels and scattered dots."}]' +) +runner = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] +runner.save(os.path.join(OUT, "bbox_t2i.png")) + +# editing with layout: refer to the n-th reference image as ; each region names its source image +# (`from`), where it is in that image (`src_bbox`) and where it should end up (`tgt_bbox`). Here the knight moves +# up and to the left and everything else is pinned in place. +knight = load_image(os.path.join(os.path.dirname(__file__), "knight.jpg")) # BFL's docs image, 1360 x 768 +prompt = ( + "In , move the miniature grey amigurumi knight figure upwards and to the left along the " + "yarn cliff . Keep the rest of the image completely unchanged, preserving the blurred yarn backdrop " + ", the large knitted dragon , and the fiery orange thread . " + '[{"id": "knight_1", "from": "ref_image_0", "src_bbox": [500, 150, 850, 350], "tgt_bbox": [194, 55, 544, 255], ' + '"desc": "A miniature amigurumi knight crocheted from thick grey woolen yarn."}, ' + '{"id": "backdrop_1", "from": "ref_image_0", "src_bbox": [0, 0, 1000, 1000], "tgt_bbox": [0, 0, 1000, 1000], ' + '"desc": "A soft, out-of-focus backdrop of cream and indigo knitted yarn."}, ' + '{"id": "cliff_1", "from": "ref_image_0", "src_bbox": [300, 0, 1000, 450], "tgt_bbox": [300, 0, 1000, 450], ' + '"desc": "A steep cliff of stacked yarn skeins in mustard, terracotta, and indigo."}, ' + '{"id": "needle_1", "from": "ref_image_0", "src_bbox": [420, 250, 550, 450], "tgt_bbox": [420, 250, 550, 450], ' + '"desc": "An oversized polished steel tapestry needle."}, ' + '{"id": "dragon_1", "from": "ref_image_0", "src_bbox": [100, 600, 950, 1000], "tgt_bbox": [100, 600, 950, 1000], ' + '"desc": "A large, fluffy knitted dragon in terracotta and mustard yarn."}, ' + '{"id": "fire_1", "from": "ref_image_0", "src_bbox": [350, 400, 650, 650], "tgt_bbox": [350, 400, 650, 650], ' + '"desc": "Curled puffs of fiery orange angora thread suspended in mid-air."}]' +) +moved = pipe(prompt=prompt, image=knight, generator=torch.Generator().manual_seed(0), output="images")[0] +moved.save(os.path.join(OUT, "bbox_i2i.png")) From 9f6e4c48a6bfadd2d4543cf3816242cae3cae2fd Mon Sep 17 00:00:00 2001 From: Sayak Paul Date: Tue, 13 Oct 2026 21:40:02 +0530 Subject: [PATCH 11/34] feat: add a single file support in Flux3 image. (#32) --- docs/source/en/api/pipelines/flux3.md | 23 +++++++ scripts/convert_flux3_to_diffusers.py | 64 +---------------- src/diffusers/loaders/single_file_model.py | 5 ++ src/diffusers/loaders/single_file_utils.py | 68 +++++++++++++++++++ .../models/transformers/transformer_flux3.py | 4 +- .../test_models_transformer_flux3.py | 19 ++++++ 6 files changed, 118 insertions(+), 65 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index edf88afeab38..4636b4539d3c 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -108,6 +108,29 @@ pipe.load_components(dtype=torch.bfloat16) pipe.enable_auto_cpu_offload(device="cuda") ``` +## Single-file loading + +[`Flux3Transformer2DModel`] loads the transformer straight from a Black Forest Labs model package with [`~loaders.FromOriginalModelMixin.from_single_file`]. The package `config.json` is not read; pass the diffusers config with `config` and `subfolder`. + +```python +import torch +from diffusers import Flux3Transformer2DModel, ModularPipeline + +transformer = Flux3Transformer2DModel.from_single_file( + "https://huggingface.co/black-forest-labs/FLUX.3-image-small/blob/main/model.safetensors", + config="black-forest-labs/FLUX.3-image-small", + subfolder="transformer", + dtype=torch.bfloat16, +) + +pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small") +pipe.load_components(names=["text_encoder", "tokenizer", "vae"], dtype=torch.bfloat16) +pipe.update_components(transformer=transformer) +pipe.to("cuda") +``` + +Load the other components by name first: [`~ModularPipeline.load_components`] without `names` would also load the transformer from the pipeline repo. + ## Flux3ModularPipeline [[autodoc]] Flux3ModularPipeline diff --git a/scripts/convert_flux3_to_diffusers.py b/scripts/convert_flux3_to_diffusers.py index 1ebacf613eb4..9581854376f7 100644 --- a/scripts/convert_flux3_to_diffusers.py +++ b/scripts/convert_flux3_to_diffusers.py @@ -6,6 +6,7 @@ from huggingface_hub import hf_hub_download from diffusers import Flux3Transformer2DModel +from diffusers.loaders.single_file_utils import convert_flux3_transformer_checkpoint_to_diffusers """ @@ -88,69 +89,6 @@ def transformer_config_from_package(package_config): } -def swap_scale_shift(weight): - # original adaLN weights produce [shift, scale]; diffusers Flux3AdaLayerNorm expects [scale, shift] - shift, scale = weight.chunk(2, dim=0) - return torch.cat([scale, shift], dim=0) - - -# inner-module renames shared by the mode blocks and the joint single blocks -BLOCK_KEY_MAPPING = { - "q_proj.weight": "attn.to_q.weight", - "k_proj.weight": "attn.to_k.weight", - "v_proj.weight": "attn.to_v.weight", - "attn_out.weight": "attn.to_out.0.weight", - "norm.query_norm.scale": "attn.norm_q.weight", - "norm.key_norm.scale": "attn.norm_k.weight", - "mlp_in.weight": "ff.linear_in.weight", - "mlp_out.weight": "ff.linear_out.weight", -} - -# module-prefix renames -PREFIX_MAPPING = { - "emb_in.": "x_embedders.", - "txt_in.": "context_embedder.", - "content_mode_blocks.": "mode_blocks.", - "single_blocks.": "single_transformer_blocks.", -} - -# exact-key renames -KEY_MAPPING = { - "time_in.in_layer.weight": "timestep_embedder.linear_1.weight", - "time_in.out_layer.weight": "timestep_embedder.linear_2.weight", - # final_layer.image.linear -> proj_out; final_layer.image.adaLN_modulation.1 -> norm_out.linear ([shift, scale] -> [scale, shift]) - "final_layer.image.linear.weight": "proj_out.weight", - "final_layer.image.adaLN_modulation.1.weight": "norm_out.linear.weight", -} - - -def convert_flux3_transformer_checkpoint_to_diffusers(original_state_dict): - converted_state_dict = {} - for key, value in original_state_dict.items(): - # training-only REPA projection heads, not part of the inference model - if key.startswith("repa_projectors."): - continue - if key in KEY_MAPPING: - if key.endswith("adaLN_modulation.1.weight"): - value = swap_scale_shift(value) - converted_state_dict[KEY_MAPPING[key]] = value - continue - - new_key = key - for old_prefix, new_prefix in PREFIX_MAPPING.items(): - if new_key.startswith(old_prefix): - new_key = new_prefix + new_key[len(old_prefix) :] - break - new_key = new_key.replace(".lin.weight", ".linear.weight") - for old_suffix, new_suffix in BLOCK_KEY_MAPPING.items(): - if new_key.endswith(old_suffix): - new_key = new_key[: -len(old_suffix)] + new_suffix - break - - converted_state_dict[new_key] = value - return converted_state_dict - - def main(args): original_ckpt = load_original_checkpoint(args) converted_transformer_state_dict = convert_flux3_transformer_checkpoint_to_diffusers(original_ckpt) diff --git a/src/diffusers/loaders/single_file_model.py b/src/diffusers/loaders/single_file_model.py index a07657159d36..44aa403b1e77 100644 --- a/src/diffusers/loaders/single_file_model.py +++ b/src/diffusers/loaders/single_file_model.py @@ -39,6 +39,7 @@ convert_cosmos_transformer_checkpoint_to_diffusers, convert_ernie_image_transformer_checkpoint_to_diffusers, convert_flux2_transformer_checkpoint_to_diffusers, + convert_flux3_transformer_checkpoint_to_diffusers, convert_flux_transformer_checkpoint_to_diffusers, convert_hidream_transformer_to_diffusers, convert_hunyuan_video_transformer_to_diffusers, @@ -197,6 +198,10 @@ "checkpoint_mapping_fn": convert_flux2_transformer_checkpoint_to_diffusers, "default_subfolder": "transformer", }, + "Flux3Transformer2DModel": { + "checkpoint_mapping_fn": convert_flux3_transformer_checkpoint_to_diffusers, + "default_subfolder": "transformer", + }, "ZImageTransformer2DModel": { "checkpoint_mapping_fn": convert_z_image_transformer_checkpoint_to_diffusers, "default_subfolder": "transformer", diff --git a/src/diffusers/loaders/single_file_utils.py b/src/diffusers/loaders/single_file_utils.py index ac707947d491..00fe9a38afa7 100644 --- a/src/diffusers/loaders/single_file_utils.py +++ b/src/diffusers/loaders/single_file_utils.py @@ -149,6 +149,10 @@ "net.pos_embedder.dim_spatial_range", ], "flux2": ["model.diffusion_model.single_stream_modulation.lin.weight", "single_stream_modulation.lin.weight"], + "flux3": [ + "model.diffusion_model.early_stream_modulations.image.lin.weight", + "early_stream_modulations.image.lin.weight", + ], "chroma": [ "model.diffusion_model.distilled_guidance_layer.in_proj.bias", "distilled_guidance_layer.in_proj.bias", @@ -208,6 +212,7 @@ "flux-depth": {"pretrained_model_name_or_path": "black-forest-labs/FLUX.1-Depth-dev"}, "flux-schnell": {"pretrained_model_name_or_path": "black-forest-labs/FLUX.1-schnell"}, "flux-2-dev": {"pretrained_model_name_or_path": "black-forest-labs/FLUX.2-dev"}, + "flux-3-image": {"pretrained_model_name_or_path": "diffusers-internal-dev/f3-small-img"}, "chroma": {"pretrained_model_name_or_path": "lodestones/Chroma1-HD"}, "ltx-video": {"pretrained_model_name_or_path": "diffusers/LTX-Video-0.9.0"}, "ltx-video-0.9.1": {"pretrained_model_name_or_path": "diffusers/LTX-Video-0.9.1"}, @@ -688,6 +693,9 @@ def infer_diffusers_model_type(checkpoint): elif any(key in checkpoint for key in CHECKPOINT_KEY_NAMES["flux2"]): model_type = "flux-2-dev" + elif any(key in checkpoint for key in CHECKPOINT_KEY_NAMES["flux3"]): + model_type = "flux-3-image" + elif any(key in checkpoint for key in CHECKPOINT_KEY_NAMES["chroma"]): model_type = "chroma" @@ -3985,6 +3993,66 @@ def update_state_dict(state_dict: dict[str, object], old_key: str, new_key: str) return converted_state_dict +def convert_flux3_transformer_checkpoint_to_diffusers(checkpoint, **kwargs): + # inner-module renames shared by the per-mode blocks and the joint single blocks + FLUX3_TRANSFORMER_BLOCK_KEY_MAP = { + "q_proj.weight": "attn.to_q.weight", + "k_proj.weight": "attn.to_k.weight", + "v_proj.weight": "attn.to_v.weight", + "attn_out.weight": "attn.to_out.0.weight", + "norm.query_norm.scale": "attn.norm_q.weight", + "norm.key_norm.scale": "attn.norm_k.weight", + "mlp_in.weight": "ff.linear_in.weight", + "mlp_out.weight": "ff.linear_out.weight", + } + + # module-prefix renames + FLUX3_TRANSFORMER_PREFIX_MAP = { + "emb_in.": "x_embedders.", + "txt_in.": "context_embedder.", + "content_mode_blocks.": "mode_blocks.", + "single_blocks.": "single_transformer_blocks.", + } + + # exact-key renames + FLUX3_TRANSFORMER_KEYS_RENAME_DICT = { + "time_in.in_layer.weight": "timestep_embedder.linear_1.weight", + "time_in.out_layer.weight": "timestep_embedder.linear_2.weight", + "final_layer.image.linear.weight": "proj_out.weight", + # original adaLN weights produce [shift, scale]; Flux3AdaLayerNorm expects [scale, shift] + "final_layer.image.adaLN_modulation.1.weight": "norm_out.linear.weight", + } + + converted_state_dict = {} + for key in list(checkpoint.keys()): + # training-only REPA projection heads, not part of the inference model + if key.startswith("repa_projectors."): + checkpoint.pop(key) + continue + + value = checkpoint.pop(key) + if key in FLUX3_TRANSFORMER_KEYS_RENAME_DICT: + if key.endswith("adaLN_modulation.1.weight"): + value = swap_scale_shift(value, dim=0) + converted_state_dict[FLUX3_TRANSFORMER_KEYS_RENAME_DICT[key]] = value + continue + + new_key = key + for old_prefix, new_prefix in FLUX3_TRANSFORMER_PREFIX_MAP.items(): + if new_key.startswith(old_prefix): + new_key = new_prefix + new_key[len(old_prefix) :] + break + new_key = new_key.replace(".lin.weight", ".linear.weight") + for old_suffix, new_suffix in FLUX3_TRANSFORMER_BLOCK_KEY_MAP.items(): + if new_key.endswith(old_suffix): + new_key = new_key[: -len(old_suffix)] + new_suffix + break + + converted_state_dict[new_key] = value + + return converted_state_dict + + def convert_z_image_transformer_checkpoint_to_diffusers(checkpoint, **kwargs): Z_IMAGE_KEYS_RENAME_DICT = { "final_layer.": "all_final_layer.2-1.", diff --git a/src/diffusers/models/transformers/transformer_flux3.py b/src/diffusers/models/transformers/transformer_flux3.py index 6bd81f21aee5..0c4a95bb0f88 100644 --- a/src/diffusers/models/transformers/transformer_flux3.py +++ b/src/diffusers/models/transformers/transformer_flux3.py @@ -19,7 +19,7 @@ import torch.nn.functional as F from ...configuration_utils import ConfigMixin, register_to_config -from ...loaders import PeftAdapterMixin +from ...loaders import FromOriginalModelMixin, PeftAdapterMixin from ...utils import apply_lora_scale, logging from ...utils.torch_utils import maybe_adjust_dtype_for_device from ..attention import AttentionMixin, AttentionModuleMixin @@ -337,7 +337,7 @@ def forward(self, hidden_states: torch.Tensor, temb: torch.Tensor) -> torch.Tens return self.norm(hidden_states) * (1 + scale) + shift -class Flux3Transformer2DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, AttentionMixin): +class Flux3Transformer2DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, FromOriginalModelMixin, AttentionMixin): """ The Transformer model introduced in FLUX.3 for text-to-image and image-to-image generation. diff --git a/tests/models/transformers/test_models_transformer_flux3.py b/tests/models/transformers/test_models_transformer_flux3.py index 3538739f7c38..aeefa8b6b5e3 100644 --- a/tests/models/transformers/test_models_transformer_flux3.py +++ b/tests/models/transformers/test_models_transformer_flux3.py @@ -24,6 +24,7 @@ BaseModelTesterConfig, MemoryTesterMixin, ModelTesterMixin, + SingleFileTesterMixin, TorchCompileTesterMixin, TrainingTesterMixin, ) @@ -125,3 +126,21 @@ class TestFlux3TransformerTraining(Flux3TransformerTesterConfig, TrainingTesterM class TestFlux3TransformerAttention(Flux3TransformerTesterConfig, AttentionTesterMixin): pass + + +class TestFlux3SingleFile(Flux3TransformerTesterConfig, SingleFileTesterMixin): + @property + def ckpt_path(self): + return "https://huggingface.co/herzogenhorn/f3-image-small/blob/main/model.safetensors" + + @property + def pretrained_model_name_or_path(self): + return "diffusers-internal-dev/f3-small-img" + + @property + def pretrained_model_kwargs(self): + return {"subfolder": "transformer"} + + @property + def torch_dtype(self): + return torch.bfloat16 From 6a17c323b5d2ff0ab9401de4f7c5ad70992ba097 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Tue, 13 Oct 2026 18:45:44 +0200 Subject: [PATCH 12/34] Warn about the denoiser input fields that are not passed to the transformer Co-Authored-By: Claude Fable 5.1 --- .../modular_pipelines/flux3/denoise.py | 45 ++++++++++--------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/src/diffusers/modular_pipelines/flux3/denoise.py b/src/diffusers/modular_pipelines/flux3/denoise.py index 6e8487a5f7a7..6ae2efd0f025 100644 --- a/src/diffusers/modular_pipelines/flux3/denoise.py +++ b/src/diffusers/modular_pipelines/flux3/denoise.py @@ -91,26 +91,27 @@ def intermediate_outputs(self) -> list[OutputParam]: def __call__( self, components: Flux3ModularPipeline, block_state: BlockState, i: int, t: torch.Tensor ) -> PipelineState: - denoiser_input_fields = block_state.denoiser_input_fields + denoiser_input_fields = dict(block_state.denoiser_input_fields) guider_inputs = { "encoder_hidden_states": ( - denoiser_input_fields.get("prompt_embeds"), - denoiser_input_fields.get("negative_prompt_embeds"), + denoiser_input_fields.pop("prompt_embeds", None), + denoiser_input_fields.pop("negative_prompt_embeds", None), ), "txt_ids": ( - denoiser_input_fields.get("txt_ids"), - denoiser_input_fields.get("negative_txt_ids"), + denoiser_input_fields.pop("txt_ids", None), + denoiser_input_fields.pop("negative_txt_ids", None), ), } - # transformer kwargs = every collected static field matching the forward signature (`img_ids`, and the - # `image_cond_*` fields of image-to-image generation) + # the remaining collected fields that match the forward signature go straight to the transformer (`img_ids`, + # and the `image_cond_*` fields of image-to-image generation) transformer_args = set(inspect.signature(components.transformer.forward).parameters.keys()) - transformer_kwargs = { - field_name: field_value - for field_name, field_value in denoiser_input_fields.items() - if field_name in transformer_args and field_name not in guider_inputs - } + transformer_kwargs = {} + for field_name in list(denoiser_input_fields): + if field_name in transformer_args: + transformer_kwargs[field_name] = denoiser_input_fields.pop(field_name) + if denoiser_input_fields and i == 0: + logger.warning(f"`denoiser_input_fields` not passed to the transformer: {sorted(denoiser_input_fields)}") latents = block_state.latents.to(components.transformer.dtype) # the flow timestep in [0, 1] (scheduler timesteps are sigmas * 1000); the transformer applies it to every @@ -176,16 +177,18 @@ def intermediate_outputs(self) -> list[OutputParam]: def __call__( self, components: Flux3ModularPipeline, block_state: BlockState, i: int, t: torch.Tensor ) -> PipelineState: - denoiser_input_fields = block_state.denoiser_input_fields + denoiser_input_fields = dict(block_state.denoiser_input_fields) + prompt_embeds = denoiser_input_fields.pop("prompt_embeds") - # transformer kwargs = every collected static field matching the forward signature (`img_ids`, `txt_ids`, and - # the `image_cond_*` fields of image-to-image generation) + # the remaining collected fields that match the forward signature go straight to the transformer (`img_ids`, + # `txt_ids`, and the `image_cond_*` fields of image-to-image generation) transformer_args = set(inspect.signature(components.transformer.forward).parameters.keys()) - transformer_kwargs = { - field_name: field_value - for field_name, field_value in denoiser_input_fields.items() - if field_name in transformer_args - } + transformer_kwargs = {} + for field_name in list(denoiser_input_fields): + if field_name in transformer_args: + transformer_kwargs[field_name] = denoiser_input_fields.pop(field_name) + if denoiser_input_fields and i == 0: + logger.warning(f"`denoiser_input_fields` not passed to the transformer: {sorted(denoiser_input_fields)}") latents = block_state.latents.to(components.transformer.dtype) batch_size = latents.shape[0] @@ -198,7 +201,7 @@ def __call__( block_state.noise_pred = components.transformer( hidden_states=latents, - encoder_hidden_states=denoiser_input_fields["prompt_embeds"], + encoder_hidden_states=prompt_embeds, timestep=timestep, target_timestep=target_timestep, return_dict=False, From a88c2099bee30edea2fa1ff733a2b15187b9e4ec Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Tue, 13 Oct 2026 19:03:02 +0200 Subject: [PATCH 13/34] Document reproducible generation with user-provided latents Co-Authored-By: Claude Fable 5.1 --- docs/source/en/api/pipelines/flux3.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index be7a6c2582ff..4c337a3ac435 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -99,6 +99,21 @@ image = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output=" It serves the same two workflows as the base model. `num_inference_steps` defaults to 16; the model is trained for 4 to 16 steps. +## Reproducible generation + +A `generator` fixes the noise the pipeline draws (see [Reproducibility](../../using-diffusers/reusing_seeds)). To reuse noise drawn elsewhere, pass it as `latents` with shape `(batch_size, 128, height // 16, width // 16)`. For example, the [BFL implementation](https://github.com/black-forest-labs/TODO) draws its noise as image tokens on the CPU and casts it to bf16; this reproduces its sample for a given seed: + + +```python +height, width = 1024, 1024 +h, w = height // 16, width // 16 + +noise = torch.randn((h * w, 128), generator=torch.Generator().manual_seed(0)).to(torch.bfloat16) +latents = noise.T.reshape(1, 128, h, w) + +image = pipe(prompt=prompt, latents=latents, height=height, width=width, output="images")[0] +``` + ## Workflows The pipeline serves two workflows, `text2image` and `image_conditioned`; the one that runs is picked by the inputs (`image` selects image-to-image). [`~ModularPipeline.from_pretrained`] can also build a pipeline for one workflow with the `workflow` argument, which then only expects and loads the components that workflow needs: From 26a32da76d237e06a597c0bc941d7c29d89774fd Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Wed, 14 Oct 2026 00:04:21 +0200 Subject: [PATCH 14/34] Run the QK norm and RoPE in fp32 with a single cast back, matching the BFL implementation Co-Authored-By: Claude Fable 5.1 --- src/diffusers/models/transformers/transformer_flux3.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/diffusers/models/transformers/transformer_flux3.py b/src/diffusers/models/transformers/transformer_flux3.py index c3b3d14392cb..5ab90e3b11ba 100644 --- a/src/diffusers/models/transformers/transformer_flux3.py +++ b/src/diffusers/models/transformers/transformer_flux3.py @@ -120,12 +120,13 @@ def __call__( key = key.unflatten(-1, (attn.heads, -1)) value = value.unflatten(-1, (attn.heads, -1)) - query = attn.norm_q(query) - key = attn.norm_k(key) + query = attn.norm_q(query.float()) + key = attn.norm_k(key.float()) if image_rotary_emb is not None: query = apply_rotary_emb(query, image_rotary_emb, sequence_dim=1) key = apply_rotary_emb(key, image_rotary_emb, sequence_dim=1) + query, key = query.to(value.dtype), key.to(value.dtype) hidden_states = dispatch_attention_fn( query, @@ -382,6 +383,8 @@ class Flux3Transformer2DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, FromOri _supports_gradient_checkpointing = True _no_split_modules = ["Flux3ModeTransformerBlock", "Flux3SingleTransformerBlock"] + # QK norm and RoPE run in fp32 with a single cast back, matching the BFL implementation + _keep_in_fp32_modules = ["norm_q", "norm_k"] _skip_layerwise_casting_patterns = ["pos_embed", "norm"] _repeated_blocks = ["Flux3ModeTransformerBlock", "Flux3SingleTransformerBlock"] From 6348cc0fc1f6752a4b380f51f78d9ba4f1f7c0a3 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Wed, 14 Oct 2026 00:18:24 +0200 Subject: [PATCH 15/34] parity script: run against the Hub repos and print the imported diffusers source Co-Authored-By: Claude Fable 5.1 --- integration-notes/parity_bf16.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/integration-notes/parity_bf16.py b/integration-notes/parity_bf16.py index 32f100c2cd45..ed4543023901 100644 --- a/integration-notes/parity_bf16.py +++ b/integration-notes/parity_bf16.py @@ -12,7 +12,7 @@ _diffusers.png and _side_by_side.png under OUT, and prints the mean pixel difference. `F3_VARIANT=sd` runs the step-distilled variant instead (16 steps, no CFG): BFL's flow-map branch against the converted -`flux3-image-distilled-modular` repo, under OUT-sd. +distilled Hub repo, under OUT-sd. """ import os @@ -27,7 +27,9 @@ VARIANT = os.environ.get("F3_VARIANT") # None (bf16 base) or "sd" (step-distilled) PACKAGE = "/raid/yiyi/f3-image-small" # the BFL package BFL_CODE = os.path.join(REPO, "..", "bfl-f3-flowmap" if VARIANT else "bfl-f3-image") # the reference implementation -MODULAR_REPO = "/raid/yiyi/flux3-image-distilled-modular" if VARIANT == "sd" else "/raid/yiyi/flux3-image-modular" +MODULAR_REPO = ( + "diffusers-internal-dev/f3-small-image-distilled" if VARIANT == "sd" else "diffusers-internal-dev/f3-small-img" +) OUT = "/raid/yiyi/diffusers-scratch/flux3-image/parity-bf16" + (f"-{VARIANT}" if VARIANT else "") SEED = 0 @@ -79,6 +81,9 @@ def reference(): def diffusers(): sys.path.insert(0, os.path.join(REPO, "src")) from diffusers import AutoencoderKLFlux2, ModularPipeline + from diffusers.models.transformers import transformer_flux3 + + print(f"diffusers source: {transformer_flux3.__file__}") pipe = ModularPipeline.from_pretrained(MODULAR_REPO) pipe.load_components(dtype=torch.bfloat16) From 9d6eccf478298944907f3c7ec1df8bb323140581 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Wed, 14 Oct 2026 00:18:24 +0200 Subject: [PATCH 16/34] Wrap a forward docstring line for the doc-builder style check Co-Authored-By: Claude Fable 5.1 --- src/diffusers/models/transformers/transformer_flux3.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/diffusers/models/transformers/transformer_flux3.py b/src/diffusers/models/transformers/transformer_flux3.py index 5ab90e3b11ba..2c5fe7a40b5d 100644 --- a/src/diffusers/models/transformers/transformer_flux3.py +++ b/src/diffusers/models/transformers/transformer_flux3.py @@ -526,8 +526,8 @@ def forward( image_cond_ids (`torch.Tensor` of shape `(cond_seq_len, 4)`, *optional*): Position ids of the reference image tokens. Required with `image_cond_hidden_states`. target_timestep (`torch.Tensor` of shape `(batch_size,)`, *optional*): - Flow timestep in `[0, 1]` that the current denoising step jumps to. Required by, and only by, a flow-map - model (`flow_map=True`). + Flow timestep in `[0, 1]` that the current denoising step jumps to. Required by, and only by, a + flow-map model (`flow_map=True`). attention_kwargs (`dict`, *optional*): A kwargs dictionary whose `scale` entry, if specified, scales the LoRA layers for this forward pass. return_dict (`bool`, *optional*, defaults to `True`): From 4655c6c383de1b823f69885c1753d9ec20ba7abe Mon Sep 17 00:00:00 2001 From: Sayak Paul Date: Wed, 14 Oct 2026 10:35:40 +0530 Subject: [PATCH 17/34] support lora loading (including ostris) in flux3 image pipeline. (#25) --- src/diffusers/loaders/__init__.py | 2 + .../loaders/lora_conversion_utils.py | 20 ++ src/diffusers/loaders/lora_pipeline.py | 183 ++++++++++++++++++ .../modular_pipelines/flux3/denoise.py | 1 + .../flux3/modular_pipeline.py | 3 +- .../flux3/test_modular_pipeline_flux3.py | 6 +- 6 files changed, 213 insertions(+), 2 deletions(-) diff --git a/src/diffusers/loaders/__init__.py b/src/diffusers/loaders/__init__.py index 828744386453..28b760b4bebe 100644 --- a/src/diffusers/loaders/__init__.py +++ b/src/diffusers/loaders/__init__.py @@ -88,6 +88,7 @@ def text_encoder_attn_modules(text_encoder): "Krea2LoraLoaderMixin", "ZImageLoraLoaderMixin", "Flux2LoraLoaderMixin", + "Flux3LoraLoaderMixin", "Ideogram4LoraLoaderMixin", "ErnieImageLoraLoaderMixin", "CosmosLoraLoaderMixin", @@ -129,6 +130,7 @@ def text_encoder_attn_modules(text_encoder): CosmosLoraLoaderMixin, ErnieImageLoraLoaderMixin, Flux2LoraLoaderMixin, + Flux3LoraLoaderMixin, FluxLoraLoaderMixin, HeliosLoraLoaderMixin, HiDreamImageLoraLoaderMixin, diff --git a/src/diffusers/loaders/lora_conversion_utils.py b/src/diffusers/loaders/lora_conversion_utils.py index 1b7bcc795d6b..525cb37b9627 100644 --- a/src/diffusers/loaders/lora_conversion_utils.py +++ b/src/diffusers/loaders/lora_conversion_utils.py @@ -2551,6 +2551,26 @@ def _convert_non_diffusers_flux2_lora_to_diffusers(state_dict): return converted_state_dict +def _convert_non_diffusers_flux3_lora_to_diffusers(state_dict): + block_mapping = { + ".q_proj.": ".attn.to_q.", + ".k_proj.": ".attn.to_k.", + ".v_proj.": ".attn.to_v.", + ".attn_out.": ".attn.to_out.0.", + ".mlp_in.": ".ff.linear_in.", + ".mlp_out.": ".ff.linear_out.", + } + converted_state_dict = {} + for key, value in state_dict.items(): + key = key.removeprefix("diffusion_model.") + key = key.replace("content_mode_blocks.", "mode_blocks.") + key = key.replace("single_blocks.", "single_transformer_blocks.") + for original_name, diffusers_name in block_mapping.items(): + key = key.replace(original_name, diffusers_name) + converted_state_dict[f"transformer.{key}"] = value + return converted_state_dict + + def _convert_kohya_flux2_lora_to_diffusers(state_dict): def _convert_to_ai_toolkit(sds_sd, ait_sd, sds_key, ait_key): if sds_key + ".lora_down.weight" not in sds_sd: diff --git a/src/diffusers/loaders/lora_pipeline.py b/src/diffusers/loaders/lora_pipeline.py index 739ff9d2b3b1..7ec3a7d32767 100644 --- a/src/diffusers/loaders/lora_pipeline.py +++ b/src/diffusers/loaders/lora_pipeline.py @@ -48,6 +48,7 @@ _convert_non_diffusers_ace_step_lora_to_diffusers, _convert_non_diffusers_anima_lora_to_diffusers, _convert_non_diffusers_flux2_lora_to_diffusers, + _convert_non_diffusers_flux3_lora_to_diffusers, _convert_non_diffusers_hidream_lora_to_diffusers, _convert_non_diffusers_ideogram4_lora_to_diffusers, _convert_non_diffusers_krea2_lora_to_diffusers, @@ -5963,6 +5964,188 @@ def unfuse_lora(self, components: list[str] = ["transformer"], **kwargs): super().unfuse_lora(components=components, **kwargs) +class Flux3LoraLoaderMixin(LoraBaseMixin): + _lora_loadable_modules = ["transformer"] + transformer_name = TRANSFORMER_NAME + + @classmethod + @validate_hf_hub_args + def lora_state_dict( + cls, + pretrained_model_name_or_path_or_dict: str | dict[str, torch.Tensor], + **kwargs, + ): + cache_dir = kwargs.pop("cache_dir", None) + force_download = kwargs.pop("force_download", False) + proxies = kwargs.pop("proxies", None) + local_files_only = kwargs.pop("local_files_only", None) + token = kwargs.pop("token", None) + revision = kwargs.pop("revision", None) + subfolder = kwargs.pop("subfolder", None) + weight_name = kwargs.pop("weight_name", None) + use_safetensors = kwargs.pop("use_safetensors", None) + return_lora_metadata = kwargs.pop("return_lora_metadata", False) + + allow_pickle = False + if use_safetensors is None: + use_safetensors = True + allow_pickle = True + + user_agent = {"file_type": "attn_procs_weights", "framework": "pytorch"} + + state_dict, metadata = _fetch_state_dict( + pretrained_model_name_or_path_or_dict=pretrained_model_name_or_path_or_dict, + weight_name=weight_name, + use_safetensors=use_safetensors, + local_files_only=local_files_only, + cache_dir=cache_dir, + force_download=force_download, + proxies=proxies, + token=token, + revision=revision, + subfolder=subfolder, + user_agent=user_agent, + allow_pickle=allow_pickle, + ) + + is_dora_scale_present = any("dora_scale" in k for k in state_dict) + if is_dora_scale_present: + warn_msg = "It seems like you are using a DoRA checkpoint that is not compatible in Diffusers at the moment. So, we are going to filter out the keys associated to 'dora_scale` from the state dict. If you think this is a mistake please open an issue https://github.com/huggingface/diffusers/issues/new." + logger.warning(warn_msg) + state_dict = {k: v for k, v in state_dict.items() if "dora_scale" not in k} + + if any(k.startswith("diffusion_model.") for k in state_dict): + state_dict = _convert_non_diffusers_flux3_lora_to_diffusers(state_dict) + + out = (state_dict, metadata) if return_lora_metadata else state_dict + return out + + @require_peft_backend + # Copied from diffusers.loaders.lora_pipeline.CogVideoXLoraLoaderMixin.load_lora_weights + def load_lora_weights( + self, + pretrained_model_name_or_path_or_dict: str | dict[str, torch.Tensor], + adapter_name: str | None = None, + hotswap: bool = False, + **kwargs, + ): + """ + See [`~loaders.StableDiffusionLoraLoaderMixin.load_lora_weights`] for more details. + """ + low_cpu_mem_usage = kwargs.pop("low_cpu_mem_usage", _LOW_CPU_MEM_USAGE_DEFAULT_LORA) + + # if a dict is passed, copy it instead of modifying it inplace + if isinstance(pretrained_model_name_or_path_or_dict, dict): + pretrained_model_name_or_path_or_dict = pretrained_model_name_or_path_or_dict.copy() + + # First, ensure that the checkpoint is a compatible one and can be successfully loaded. + kwargs["return_lora_metadata"] = True + state_dict, metadata = self.lora_state_dict(pretrained_model_name_or_path_or_dict, **kwargs) + + is_correct_format = all("lora" in key for key in state_dict.keys()) + if not is_correct_format: + raise ValueError("Invalid LoRA checkpoint. Make sure all LoRA param names contain `'lora'` substring.") + + self.load_lora_into_transformer( + state_dict, + transformer=getattr(self, self.transformer_name) if not hasattr(self, "transformer") else self.transformer, + adapter_name=adapter_name, + metadata=metadata, + _pipeline=self, + low_cpu_mem_usage=low_cpu_mem_usage, + hotswap=hotswap, + ) + + @classmethod + # Copied from diffusers.loaders.lora_pipeline.SD3LoraLoaderMixin.load_lora_into_transformer + def load_lora_into_transformer( + cls, + state_dict, + transformer, + adapter_name=None, + _pipeline=None, + low_cpu_mem_usage=False, + hotswap: bool = False, + metadata=None, + ): + """ + See [`~loaders.StableDiffusionLoraLoaderMixin.load_lora_into_unet`] for more details. + """ + # Load the layers corresponding to transformer. + logger.info(f"Loading {cls.transformer_name}.") + transformer.load_lora_adapter( + state_dict, + network_alphas=None, + adapter_name=adapter_name, + metadata=metadata, + _pipeline=_pipeline, + low_cpu_mem_usage=low_cpu_mem_usage, + hotswap=hotswap, + ) + + @classmethod + # Copied from diffusers.loaders.lora_pipeline.CogVideoXLoraLoaderMixin.save_lora_weights + def save_lora_weights( + cls, + save_directory: str | os.PathLike, + transformer_lora_layers: dict[str, torch.nn.Module | torch.Tensor] = None, + is_main_process: bool = True, + weight_name: str = None, + save_function: Callable = None, + safe_serialization: bool = True, + transformer_lora_adapter_metadata: dict | None = None, + ): + r""" + See [`~loaders.StableDiffusionLoraLoaderMixin.save_lora_weights`] for more information. + """ + lora_layers = {} + lora_metadata = {} + + if transformer_lora_layers: + lora_layers[cls.transformer_name] = transformer_lora_layers + lora_metadata[cls.transformer_name] = transformer_lora_adapter_metadata + + if not lora_layers: + raise ValueError("You must pass at least one of `transformer_lora_layers` or `text_encoder_lora_layers`.") + + cls._save_lora_weights( + save_directory=save_directory, + lora_layers=lora_layers, + lora_metadata=lora_metadata, + is_main_process=is_main_process, + weight_name=weight_name, + save_function=save_function, + safe_serialization=safe_serialization, + ) + + # Copied from diffusers.loaders.lora_pipeline.CogVideoXLoraLoaderMixin.fuse_lora + def fuse_lora( + self, + components: list[str] = ["transformer"], + lora_scale: float = 1.0, + safe_fusing: bool = False, + adapter_names: list[str] | None = None, + **kwargs, + ): + r""" + See [`~loaders.StableDiffusionLoraLoaderMixin.fuse_lora`] for more details. + """ + super().fuse_lora( + components=components, + lora_scale=lora_scale, + safe_fusing=safe_fusing, + adapter_names=adapter_names, + **kwargs, + ) + + # Copied from diffusers.loaders.lora_pipeline.CogVideoXLoraLoaderMixin.unfuse_lora + def unfuse_lora(self, components: list[str] = ["transformer"], **kwargs): + r""" + See [`~loaders.StableDiffusionLoraLoaderMixin.unfuse_lora`] for more details. + """ + super().unfuse_lora(components=components, **kwargs) + + class Ideogram4LoraLoaderMixin(LoraBaseMixin): r""" Load LoRA layers into [`Ideogram4Transformer2DModel`]. Specific to [`Ideogram4Pipeline`]. diff --git a/src/diffusers/modular_pipelines/flux3/denoise.py b/src/diffusers/modular_pipelines/flux3/denoise.py index 6ae2efd0f025..d31310d664b5 100644 --- a/src/diffusers/modular_pipelines/flux3/denoise.py +++ b/src/diffusers/modular_pipelines/flux3/denoise.py @@ -81,6 +81,7 @@ def inputs(self) -> list[tuple[str, Any]]: description="The image latent tokens to denoise. Shape: (B, seq_len, C)", ), InputParam.template("denoiser_input_fields"), + InputParam.template("attention_kwargs", kwargs_type="denoiser_input_fields"), ] @property diff --git a/src/diffusers/modular_pipelines/flux3/modular_pipeline.py b/src/diffusers/modular_pipelines/flux3/modular_pipeline.py index c91777ff723c..fbd29f24ace0 100644 --- a/src/diffusers/modular_pipelines/flux3/modular_pipeline.py +++ b/src/diffusers/modular_pipelines/flux3/modular_pipeline.py @@ -13,6 +13,7 @@ # limitations under the License. +from ...loaders import Flux3LoraLoaderMixin from ...utils import logging from ..modular_pipeline import ModularPipeline @@ -20,7 +21,7 @@ logger = logging.get_logger(__name__) # pylint: disable=invalid-name -class Flux3ModularPipeline(ModularPipeline): +class Flux3ModularPipeline(ModularPipeline, Flux3LoraLoaderMixin): """ A ModularPipeline for FLUX.3. """ diff --git a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py index f870c4ef6486..22febdd6872a 100644 --- a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py +++ b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py @@ -33,6 +33,7 @@ BaseModularPipelineTesterConfig, ModularGuiderTesterMixin, ModularLoadingTesterMixin, + ModularLoraTesterMixin, ModularMemoryTesterMixin, ModularPipelineTesterMixin, ModularWorkflowTesterMixin, @@ -91,6 +92,7 @@ "text_encoder_out_layers": (4, 8, 12, 16, 20, 24, 28, 32), "num_images_per_prompt": 1, "latents": None, + "attention_kwargs": None, "generator": None, "num_inference_steps": 50, "timesteps": None, @@ -322,7 +324,6 @@ class TestFlux3ModularPipelineMemory(Flux3ModularPipelineTesterConfig, ModularMe class TestFlux3ModularPipelineWorkflow(Flux3ModularPipelineTesterConfig, ModularWorkflowTesterMixin): pass - class Flux3DistilledModularPipelineTesterConfig(Flux3ModularPipelineTesterConfig): pipeline_class = Flux3DistilledModularPipeline pipeline_blocks_class = Flux3DistilledAutoBlocks @@ -365,3 +366,6 @@ class TestFlux3DistilledModularPipelineLoading(Flux3DistilledModularPipelineTest class TestFlux3DistilledModularPipelineWorkflow(Flux3DistilledModularPipelineTesterConfig, ModularWorkflowTesterMixin): pass + +class TestFlux3ModularPipelineLoRA(Flux3ModularPipelineTesterConfig, ModularLoraTesterMixin): + pass \ No newline at end of file From f97b9200f6f06c1cfb5e27eba52e4d686fbb46a8 Mon Sep 17 00:00:00 2001 From: Sayak Paul Date: Wed, 14 Oct 2026 11:02:42 +0530 Subject: [PATCH 18/34] feat: support loading and performing inference with single-file formatted FP8 and NVFP4 checkpoints (#24) * feat: support loading siungle file format checkpoints including quants. * remove updates.patch * remove fluff. * feat: add a single file support in Flux3 image. * bfl prefix * version guard. * prpoagate changes. * add tests * doc mods and remove integration notes * remove script for quant --------- Co-authored-by: Claude Fable 5.1 --- docs/source/en/_toctree.yml | 2 + docs/source/en/api/pipelines/flux3.md | 28 ++ docs/source/en/api/quantization.md | 4 + docs/source/en/quantization/bfl.md | 63 ++++ docs/source/en/quantization/overview.md | 1 + integration-notes/README.md | 134 -------- integration-notes/flux3_distilled_examples.py | 82 ----- integration-notes/flux3_examples.py | 80 ----- integration-notes/knight.jpg | Bin 127590 -> 0 bytes integration-notes/make_flux3_modular_repo.py | 95 ------ integration-notes/make_tiny_flux3_repo.py | 125 ------- integration-notes/parity_bf16.py | 125 ------- src/diffusers/__init__.py | 20 ++ src/diffusers/loaders/single_file_model.py | 3 + src/diffusers/loaders/single_file_utils.py | 8 +- src/diffusers/quantizers/auto.py | 4 + src/diffusers/quantizers/bfl/__init__.py | 1 + src/diffusers/quantizers/bfl/bfl_quantizer.py | 152 +++++++++ src/diffusers/quantizers/bfl/utils.py | 310 ++++++++++++++++++ .../quantizers/quantization_config.py | 25 ++ src/diffusers/utils/__init__.py | 1 + src/diffusers/utils/dummy_bfl_objects.py | 17 + src/diffusers/utils/import_utils.py | 5 + tests/conftest.py | 1 + tests/models/testing_utils/__init__.py | 6 + tests/models/testing_utils/quantization.py | 287 ++++++++++++++++ .../test_models_transformer_flux3.py | 31 ++ tests/testing_utils.py | 8 + 28 files changed, 976 insertions(+), 642 deletions(-) create mode 100644 docs/source/en/quantization/bfl.md delete mode 100644 integration-notes/README.md delete mode 100644 integration-notes/flux3_distilled_examples.py delete mode 100644 integration-notes/flux3_examples.py delete mode 100644 integration-notes/knight.jpg delete mode 100644 integration-notes/make_flux3_modular_repo.py delete mode 100644 integration-notes/make_tiny_flux3_repo.py delete mode 100644 integration-notes/parity_bf16.py create mode 100644 src/diffusers/quantizers/bfl/__init__.py create mode 100644 src/diffusers/quantizers/bfl/bfl_quantizer.py create mode 100644 src/diffusers/quantizers/bfl/utils.py create mode 100644 src/diffusers/utils/dummy_bfl_objects.py diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index 09b6032996ef..bb1ff8f71225 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -95,6 +95,8 @@ title: Getting started - local: quantization/bitsandbytes title: bitsandbytes + - local: quantization/bfl + title: BFL (fp8r / nvfp4) - local: quantization/gguf title: gguf - local: quantization/nunchaku diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index 4c337a3ac435..6bd1e3295a40 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -161,6 +161,34 @@ pipe.to("cuda") Load the other components by name first: [`~ModularPipeline.load_components`] without `names` would also load the transformer from the pipeline repo. +Loading the FP8r and NVFP4 single-file checkpoints are also supported. Both load through the same call with a [`BFLQuantizationConfig`]. See the [BFL quantization guide](../../quantization/bfl) for the kernel requirements of each scheme. + +```python +from diffusers import BFLQuantizationConfig + +transformer = Flux3Transformer2DModel.from_single_file( + "https://huggingface.co/black-forest-labs/FLUX.3-image-small/blob/main/variants/nvfp4/model.safetensors", + config="black-forest-labs/FLUX.3-image-small", + subfolder="transformer", + quantization_config=BFLQuantizationConfig(compute_dtype=torch.bfloat16), + dtype=torch.bfloat16, +) +``` + +When loading the step-distilled checkpoints, we need their +configs (`flow_map=True`): + + +```python +transformer = Flux3Transformer2DModel.from_single_file( + "https://huggingface.co/black-forest-labs/FLUX.3-image-small/blob/main/variants/sd-nvfp4/model.safetensors", + config="black-forest-labs/FLUX.3-image-small-distilled", + subfolder="transformer", + quantization_config=BFLQuantizationConfig(compute_dtype=torch.bfloat16), + dtype=torch.bfloat16, +) +``` + ## Flux3ModularPipeline [[autodoc]] Flux3ModularPipeline diff --git a/docs/source/en/api/quantization.md b/docs/source/en/api/quantization.md index eab5fc663b82..415762fb1d7b 100644 --- a/docs/source/en/api/quantization.md +++ b/docs/source/en/api/quantization.md @@ -22,6 +22,10 @@ Quantization techniques reduce memory and computational costs by representing we [[autodoc]] quantizers.PipelineQuantizationConfig +## BFLQuantizationConfig + +[[autodoc]] quantizers.quantization_config.BFLQuantizationConfig + ## BitsAndBytesConfig [[autodoc]] quantizers.quantization_config.BitsAndBytesConfig diff --git a/docs/source/en/quantization/bfl.md b/docs/source/en/quantization/bfl.md new file mode 100644 index 000000000000..6bc90ab0506b --- /dev/null +++ b/docs/source/en/quantization/bfl.md @@ -0,0 +1,63 @@ + + +# BFL (fp8r / nvfp4) + +[Black Forest Labs](https://huggingface.co/black-forest-labs) ships quantized variants of its [Flux3](../api/pipelines/flux3) transformers in two schemes, stored as single-file checkpoints next to the original bf16 weights: + +- `fp8r`: rowwise FP8. Each quantized linear stores an E4M3 `weight` and an fp32 per-row `weight_scale`. Activations are quantized to E4M3 per row at runtime. +- `nvfp4`: NVFP4 in the [ModelOpt](https://github.com/NVIDIA/Model-Optimizer) layout. Each quantized linear stores a packed E2M1 `weight` (two values per byte), E4M3 block scales of 16 (`weight_scale`), a global fp32 `weight_scale_2`, and a calibrated fp32 activation scale `input_scale`. + +Diffusers loads both through [`~FromSingleFileMixin.from_single_file`] with a [`BFLQuantizationConfig`]. The scheme is read from the checkpoint, so the same config works for either file. Only loading prequantized checkpoints is supported; pipeline-level loading is not. + +The example below loads a `fp8r` checkpoint of [FLUX.2 klein 4B](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B). Pass `config` so that the model config is read from the Diffusers repository. + +```python +import torch + +from diffusers import BFLQuantizationConfig, Flux2KleinPipeline, Flux2Transformer2DModel + +ckpt_path = "https://huggingface.co/sayakpaul/FLUX.2-klein-4B-fp8r-nvfp4/blob/main/flux-2-klein-4b-fp8r.safetensors" +transformer = Flux2Transformer2DModel.from_single_file( + ckpt_path, + config="black-forest-labs/FLUX.2-klein-4B", + subfolder="transformer", + quantization_config=BFLQuantizationConfig(compute_dtype=torch.bfloat16), + dtype=torch.bfloat16, +) +pipe = Flux2KleinPipeline.from_pretrained( + "black-forest-labs/FLUX.2-klein-4B", + transformer=transformer, + dtype=torch.bfloat16, +).to("cuda") +prompt = "A cat holding a sign that says hello world" +image = pipe(prompt=prompt, num_inference_steps=4, guidance_scale=1.0, generator=torch.manual_seed(0)).images[0] +image.save("flux2-klein-fp8r.png") +``` + +Replace the filename with `flux-2-klein-4b-nvfp4.safetensors` to load the `nvfp4` variant. + +## Kernels + +The quantized weights stay in their low-precision storage dtype. Which GEMM runs depends on the device: + +| Scheme | Fused kernel | Requirements | +|---|---|---| +| `fp8r` | `torch._scaled_mm` with rowwise scales | CUDA GPU with compute capability 8.9 or higher | +| `nvfp4` | `flashinfer.mm_fp4` (CUTLASS) with `flashinfer.nvfp4_quantize` for the activations | Blackwell GPU (compute capability 10.0 or higher), a CUDA toolkit with `nvcc`, and `flashinfer-python` with `flashinfer-cubin` at the same version | + +Install the two FlashInfer packages at the same version, for example `pip install flashinfer-python==0.6.13 flashinfer-cubin==0.6.13`. The `flashinfer-cubin` release can lag behind `flashinfer-python`, and FlashInfer refuses to import when the versions differ. + +FlashInfer compiles its NVFP4 kernels on first use with the CUDA toolkit of the environment and picks the target architecture from the visible GPU (or from `FLASHINFER_CUDA_ARCH_LIST` when set). Consumer and workstation Blackwell GPUs (compute capability 12.0) need a CUDA toolkit of 12.9 or newer. With an older toolkit FlashInfer finds no eligible architecture and the first NVFP4 call fails with `FlashInfer requires GPUs with sm75 or higher`. + +On any other device the weights are dequantized to `compute_dtype` in each forward pass and the layer runs as a regular linear. In that case the activations are not quantized, so outputs differ slightly from the fused kernels. diff --git a/docs/source/en/quantization/overview.md b/docs/source/en/quantization/overview.md index 9ff0de9a4785..06a6b55ce60f 100644 --- a/docs/source/en/quantization/overview.md +++ b/docs/source/en/quantization/overview.md @@ -146,6 +146,7 @@ The following backends support loading prequantized checkpoints out of the box. | [bitsandbytes](./bitsandbytes) | Config is saved in `config.json`; no extra arguments needed. Supports quantizing on the fly as well as loading prequantized checkpoints. | | [torchao](./torchao) | Same as above. | | [GGUF](./gguf) | Uses `from_single_file` with Model classes; pipeline-level loading is not supported. | +| [BFL](./bfl) | Loads Black Forest Labs `fp8r` and `nvfp4` single-file checkpoints with `from_single_file`; pipeline-level loading is not supported. | | [AutoRound](./autoround) | Only loading is supported; quantize first with the AutoRound CLI or Python API. | | [Nunchaku Lite](./nunchaku) | Config is saved in `config.json`; requires the `kernels` package. Only loading is supported. | | [ModelOpt](./modelopt) | Only loading prequantized models is supported. | diff --git a/integration-notes/README.md b/integration-notes/README.md deleted file mode 100644 index e7be21305901..000000000000 --- a/integration-notes/README.md +++ /dev/null @@ -1,134 +0,0 @@ -# FLUX.3 image integration notes — TEMPORARY, REMOVE BEFORE MERGE - -> **⚠️ This folder must not be merged.** It holds the parity check against the confidential BFL -> reference implementation (`f3` package), the doc-style examples, and the scripts that build the -> test and release repos. Delete it before the PR is merged. - -## Setup (workspace `/raid/yiyi`) - -- `diffusers-new-model-addition-f3/` — this repo, branch `flux3-image`, with its `.venv` (transformers 5.x). -- `bfl-f3-image/` — the BFL reference on its `f3-image` branch (`f3/image_small/ImageSmallPipeline`). -- `f3-image-small/` — the BFL model package (`hf download herzogenhorn/f3-image-small`). -- `flux3-image-modular/` — the converted diffusers repo (see Conversion). The same content is on the Hub as - `diffusers-internal-dev/f3-small-img` (private), with BFL's knight example image under `examples/`. -- `venv-tf457/` — a venv with transformers 4.57.6 (BFL's training pin) for the reference side. -- `tiny-flux3-image-modular/` — the tiny random repo the tests use (`make_tiny_flux3_repo.py`). Upload as - `hf-internal-testing/tiny-flux3-image-modular` and update `FLUX3_TINY_REPO` in the modular tests. - -Use `CUDA_DEVICE_ORDER=PCI_BUS_ID` so `CUDA_VISIBLE_DEVICES` matches `nvidia-smi` on this machine. - -## Conversion - -```bash -python scripts/convert_flux3_to_diffusers.py \ - --checkpoint_path f3-image-small/model.safetensors --config_path f3-image-small/config.json \ - --output_path flux3-image-diffusers # -> transformer/ -python scripts/convert_flux2_to_diffusers.py --vae \ - --checkpoint_path f3-image-small/ae.safetensors --output_path flux3-image-diffusers # -> vae/ -``` - -`make_flux3_modular_repo.py` does both, adds the text encoder and tokenizer from the package, writes the -`modular_model_index.json`, and can push the result to the Hub (that is how `diffusers-internal-dev/f3-small-img` -was built). `flux3_examples.py` runs the four parity cases as doc examples against that repo, without any -reference code. - -`--variant sd` builds the step-distilled repo from `variants/sd/` of the package (`flow_map: true` in its config; -the converter maps `ref_time_in.*` to `target_timestep_embedder.*`), with `Flux3DistilledAutoBlocks` in the index: - -```bash -.venv/bin/python integration-notes/make_flux3_modular_repo.py \ - --package /raid/yiyi/f3-image-small --variant sd --output /raid/yiyi/flux3-image-distilled-modular -``` - -## Parity (`parity_bf16.py`) - -bf16, 50 steps, guidance 3.5, shift 6.93, seed 0, same prompts and the same initial noise on both sides. -The reference runs as shipped (`ImageSmallPipeline.from_package` + `generate`) in the transformers 4.57.6 venv, -diffusers in this repo's venv; the two sides hand over through PNG files: - -```bash -CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 /raid/yiyi/venv-tf457/bin/python integration-notes/parity_bf16.py reference -CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/parity_bf16.py diffusers -``` - -| case | size | mean pixel diff | max | -|---|---|---|---| -| t2i (the fox) | 1024 x 1024 | 0.44/255 | 47/255 | -| i2i (the fox at night, the t2i output as reference) | 1024 x 1024 | 0.31/255 | 9/255 | -| bbox_t2i (BFL's layout-control example: regions in the caption) | 1024 x 1024 | 0.39/255 | 123/255 | -| bbox_i2i (BFL's knight edit with ``; size derived from the 1360 x 768 reference) | 1360 x 768 | 0.26/255 | 26/255 | - -All four pairs are visually identical (`diffusers-scratch/flux3-image/parity-bf16/*_side_by_side.png`, -reference left, diffusers right). The differences are bf16 rounding over 50 steps; in fp32 the two -implementations agree to within 2/255 (checked during development: the text encoder is bit-identical and -the transformer agrees to 4e-5 in fp32). - -### Step-distilled variant (`F3_VARIANT=sd`) - -Same four cases with the `sd` package variant: 16 steps, no guidance, BFL's flow-map branch -(`/raid/yiyi/bfl-f3-flowmap`, their PR #5) against `flux3-image-distilled-modular`: - -```bash -F3_VARIANT=sd CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 /raid/yiyi/venv-tf457/bin/python integration-notes/parity_bf16.py reference -F3_VARIANT=sd CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/parity_bf16.py diffusers -``` - -| case | size | mean pixel diff | max | pixels off by > 30 | -|---|---|---|---|---| -| t2i | 1024 x 1024 | 1.74/255 | 221/255 | 1.8 % | -| i2i | 1024 x 1024 | 0.21/255 | 12/255 | 0 % | -| bbox_t2i | 1024 x 1024 | 0.46/255 | 247/255 | 0.2 % | -| bbox_i2i | 1360 x 768 | 0.31/255 | 89/255 | 0.03 % | - -Visually identical (`parity-bf16-sd/*_side_by_side.png`). The t2i case drifts more than with the base model -(1.8 % of pixels differ by more than 30, fine fur texture only): 16 large Euler steps amplify bf16 rounding more -than 50 small ones, and the attention kernels differ (cuDNN in the reference, PyTorch's default here). - -## Why the reference runs under transformers 4.57.6 - -Qwen3-VL computes the position id of the padding tokens differently across transformers versions (1 in -4.57, BFL's training pin; 0 in 5.x), and the FLUX.3 transformer attends to the padding tokens. The real -tokens are bit-identical across versions; only the padding rows differ, which moves the final image by about -4/255 mean. `Flux3TextEncoderStep` passes the position ids explicitly, so on transformers 5.x it produces -the 4.57.6 embeddings bit for bit; running the reference in an actual 4.57.6 venv is what makes the table -above a comparison against the training-time behavior. This diffusers cannot be installed next to -transformers 4.57.6 (huggingface_hub pin), hence the two venvs. - -## Speed and memory - -Single A100 80GB, bf16 transformer and text encoder, fp32 VAE, 50 steps, no offloading: - -| workflow | time | peak GPU memory | -|---|---|---| -| t2i 1024² | 30.5 s | 22.7 GiB | -| i2i 1024² (one 1024² reference) | 63.6 s | 22.7 GiB | -| i2i 1360 x 768 (two references) | 105 s | 22.7 GiB | - -## Design notes - -- `Flux3Transformer2DModel` (image + reference modes, no pooled-vector embedder, output head for the image - tokens; forward: `hidden_states`, `encoder_hidden_states`, `timestep (B,)`, `img_ids`, `txt_ids`, - `image_cond_hidden_states`, `image_cond_ids`). The video model, if it comes, is a separate class. -- One prompt per call (as the reference): the text window is the prompt's length rounded up to 80 tokens, - and the transformer attends to the padding, so prompts of different lengths cannot share a batch without - changing their context. `num_images_per_prompt` gives several images. -- References keep their own resolution (`Flux3ImageProcessor`, Flux2's helpers: downscale above 4096² pixels, - center-crop to the 16-pixel grid), ids on the `l` axis `-(1 + index)`, H/W ids scaled by 0.5. The i2i size - defaults to 1 MP at the first reference's aspect ratio; a given `height`/`width` overrides its side (BFL CLI). -- Sampling defaults on the from_config components: guidance 3.5, shift 6.93, 50 steps, for both workflows. - -## Open items - -- Batched prompts: not supported for now. Before adding: measure the quality effect of a shared text window on - shorter prompts, whether batching helps throughput at all at this sequence length, and the semantics of - several prompts together with several reference images. -- Seed layout: BFL draws the noise token-major `(h*w, C)` on the CPU; diffusers draws `(B, C, h, w)`, so the - same seed gives a different image than BFL's CLI (the parity passes the noise explicitly). Two lines in - `Flux3PrepareLatentsStep` would align them. -- VAE dtype: BFL runs the VAE in fp32; `load_components(dtype=torch.bfloat16)` casts it too. Pass - `dtype={"vae": torch.float32, "default": torch.bfloat16}` (as the examples do) or decide the documented default. -- Hub repo layout with BFL (diffusers subfolders in the release repo vs a `-diffusers` repo); the repo ids in - the docs are placeholders. `from_single_file` for the bf16 transformer is a small follow-up. -- fp8 / nvfp4 variants (`variants/fp8r`, `variants/nvfp4`, ModelOpt export layout with `hf_quant_config.json`): - not loadable as-is; follow-up through the ModelOpt or torchao backend with BFL's weights repackaged. -- Merge upstream `main`; `test_compile_works_with_aot` fails for Flux2 too in this venv (torch 2.13). diff --git a/integration-notes/flux3_distilled_examples.py b/integration-notes/flux3_distilled_examples.py deleted file mode 100644 index c1546ac55f04..000000000000 --- a/integration-notes/flux3_distilled_examples.py +++ /dev/null @@ -1,82 +0,0 @@ -"""FLUX.3 image step-distilled examples: the same four cases as `flux3_examples.py` (text-to-image, image-to-image, -and BFL's two layout-control examples) run with the step-distilled model. - -Only the repo changes. The distilled pipeline has no guider and takes no negative prompt, and runs 16 steps by default. - -Standalone: loads the modular pipeline from the Hub repo and writes four PNGs under `out/`. - - CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/flux3_distilled_examples.py -""" - -import os - -import torch - -from diffusers import ModularPipeline -from diffusers.utils import load_image - - -REPO = os.environ.get("F3_REPO", "diffusers-internal-dev/f3-small-image-distilled") -OUT = os.environ.get("F3_OUT", "out") -os.makedirs(OUT, exist_ok=True) - -pipe = ModularPipeline.from_pretrained(REPO) -pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) # the image VAE runs in fp32 -pipe.to("cuda") - -# text-to-image: 1024 x 1024, 16 steps by default, no guidance. FLUX.3 responds best to full scene captions. -prompt = ( - "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest clearing at golden hour. " - "The fox is centered in the frame, its thick orange winter coat contrasting with the fresh white snow, breath " - "visible in the cold air. Snow-laden pine trees fill the softly blurred background, and low warm sunlight rakes " - "across the scene from the left, casting long blue shadows." -) -fox = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] -fox.save(os.path.join(OUT, "t2i.png")) - -# image-to-image: the reference keeps its own resolution and the output defaults to its aspect ratio at ~1 MP. -# Describe the target image as a complete caption of the reference with the edit folded in. -prompt = ( - "The same red fox in the same snow-covered forest clearing, now at night under a full moon: cool blue moonlight, " - "deep shadows, stars visible through the pine branches, the fox's eyes catching the light." -) -night = pipe(prompt=prompt, image=fox, generator=torch.Generator().manual_seed(0), output="images")[0] -night.save(os.path.join(OUT, "i2i.png")) - -# layout control: append a JSON list of regions to the caption. `bbox` is [y0, x0, y1, x1] normalized to 0-1000, -# so it is independent of the image size; tag a region in the caption with and describe it in `desc`. -prompt = ( - "Minimalist graphic illustration featuring a black silhouette of a person centered against a " - "solid, vibrant chartreuse background . The figure is captured in a dynamic, mid-stride running " - "pose, facing toward the left side of the frame. " - '[{"id": "background_1", "bbox": [0, 0, 1000, 1000], "desc": "A flat field of neon yellow-green with a subtle ' - 'paper texture and fine, organic grain."}, {"id": "silhouette_1", "bbox": [150, 150, 850, 850], "desc": "A ' - "black silhouette of a person in motion, made of dense stippled ink. The leading edges are solid; the trailing " - 'arm and leg dissolve into coarse square pixels and scattered dots."}]' -) -runner = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] -runner.save(os.path.join(OUT, "bbox_t2i.png")) - -# editing with layout: refer to the n-th reference image as ; each region names its source image -# (`from`), where it is in that image (`src_bbox`) and where it should end up (`tgt_bbox`). Here the knight moves -# up and to the left and everything else is pinned in place. -knight = load_image(os.path.join(os.path.dirname(__file__), "knight.jpg")) # BFL's docs image, 1360 x 768 -prompt = ( - "In , move the miniature grey amigurumi knight figure upwards and to the left along the " - "yarn cliff . Keep the rest of the image completely unchanged, preserving the blurred yarn backdrop " - ", the large knitted dragon , and the fiery orange thread . " - '[{"id": "knight_1", "from": "ref_image_0", "src_bbox": [500, 150, 850, 350], "tgt_bbox": [194, 55, 544, 255], ' - '"desc": "A miniature amigurumi knight crocheted from thick grey woolen yarn."}, ' - '{"id": "backdrop_1", "from": "ref_image_0", "src_bbox": [0, 0, 1000, 1000], "tgt_bbox": [0, 0, 1000, 1000], ' - '"desc": "A soft, out-of-focus backdrop of cream and indigo knitted yarn."}, ' - '{"id": "cliff_1", "from": "ref_image_0", "src_bbox": [300, 0, 1000, 450], "tgt_bbox": [300, 0, 1000, 450], ' - '"desc": "A steep cliff of stacked yarn skeins in mustard, terracotta, and indigo."}, ' - '{"id": "needle_1", "from": "ref_image_0", "src_bbox": [420, 250, 550, 450], "tgt_bbox": [420, 250, 550, 450], ' - '"desc": "An oversized polished steel tapestry needle."}, ' - '{"id": "dragon_1", "from": "ref_image_0", "src_bbox": [100, 600, 950, 1000], "tgt_bbox": [100, 600, 950, 1000], ' - '"desc": "A large, fluffy knitted dragon in terracotta and mustard yarn."}, ' - '{"id": "fire_1", "from": "ref_image_0", "src_bbox": [350, 400, 650, 650], "tgt_bbox": [350, 400, 650, 650], ' - '"desc": "Curled puffs of fiery orange angora thread suspended in mid-air."}]' -) -moved = pipe(prompt=prompt, image=knight, generator=torch.Generator().manual_seed(0), output="images")[0] -moved.save(os.path.join(OUT, "bbox_i2i.png")) diff --git a/integration-notes/flux3_examples.py b/integration-notes/flux3_examples.py deleted file mode 100644 index e39e7d7222ba..000000000000 --- a/integration-notes/flux3_examples.py +++ /dev/null @@ -1,80 +0,0 @@ -"""FLUX.3 image examples, as they would appear in the docs: text-to-image, image-to-image, and BFL's two -layout-control examples (regions in the caption; an edit that refers to the reference as ). - -Standalone: loads the modular pipeline from the Hub repo and writes four PNGs under `out/`. - - CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/flux3_examples.py -""" - -import os - -import torch - -from diffusers import ModularPipeline -from diffusers.utils import load_image - - -REPO = os.environ.get("F3_REPO", "diffusers-internal-dev/f3-small-img") -OUT = os.environ.get("F3_OUT", "out") -os.makedirs(OUT, exist_ok=True) - -pipe = ModularPipeline.from_pretrained(REPO) -pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) # the image VAE runs in fp32 -pipe.to("cuda") - -# text-to-image: 1024 x 1024, 50 steps, guidance 3.5 by default. FLUX.3 responds best to full scene captions. -prompt = ( - "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest clearing at golden hour. " - "The fox is centered in the frame, its thick orange winter coat contrasting with the fresh white snow, breath " - "visible in the cold air. Snow-laden pine trees fill the softly blurred background, and low warm sunlight rakes " - "across the scene from the left, casting long blue shadows." -) -fox = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] -fox.save(os.path.join(OUT, "t2i.png")) - -# image-to-image: the reference keeps its own resolution and the output defaults to its aspect ratio at ~1 MP. -# Describe the target image as a complete caption of the reference with the edit folded in. -prompt = ( - "The same red fox in the same snow-covered forest clearing, now at night under a full moon: cool blue moonlight, " - "deep shadows, stars visible through the pine branches, the fox's eyes catching the light." -) -night = pipe(prompt=prompt, image=fox, generator=torch.Generator().manual_seed(0), output="images")[0] -night.save(os.path.join(OUT, "i2i.png")) - -# layout control: append a JSON list of regions to the caption. `bbox` is [y0, x0, y1, x1] normalized to 0-1000, -# so it is independent of the image size; tag a region in the caption with and describe it in `desc`. -prompt = ( - "Minimalist graphic illustration featuring a black silhouette of a person centered against a " - "solid, vibrant chartreuse background . The figure is captured in a dynamic, mid-stride running " - "pose, facing toward the left side of the frame. " - '[{"id": "background_1", "bbox": [0, 0, 1000, 1000], "desc": "A flat field of neon yellow-green with a subtle ' - 'paper texture and fine, organic grain."}, {"id": "silhouette_1", "bbox": [150, 150, 850, 850], "desc": "A ' - "black silhouette of a person in motion, made of dense stippled ink. The leading edges are solid; the trailing " - 'arm and leg dissolve into coarse square pixels and scattered dots."}]' -) -runner = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] -runner.save(os.path.join(OUT, "bbox_t2i.png")) - -# editing with layout: refer to the n-th reference image as ; each region names its source image -# (`from`), where it is in that image (`src_bbox`) and where it should end up (`tgt_bbox`). Here the knight moves -# up and to the left and everything else is pinned in place. -knight = load_image(os.path.join(os.path.dirname(__file__), "knight.jpg")) # BFL's docs image, 1360 x 768 -prompt = ( - "In , move the miniature grey amigurumi knight figure upwards and to the left along the " - "yarn cliff . Keep the rest of the image completely unchanged, preserving the blurred yarn backdrop " - ", the large knitted dragon , and the fiery orange thread . " - '[{"id": "knight_1", "from": "ref_image_0", "src_bbox": [500, 150, 850, 350], "tgt_bbox": [194, 55, 544, 255], ' - '"desc": "A miniature amigurumi knight crocheted from thick grey woolen yarn."}, ' - '{"id": "backdrop_1", "from": "ref_image_0", "src_bbox": [0, 0, 1000, 1000], "tgt_bbox": [0, 0, 1000, 1000], ' - '"desc": "A soft, out-of-focus backdrop of cream and indigo knitted yarn."}, ' - '{"id": "cliff_1", "from": "ref_image_0", "src_bbox": [300, 0, 1000, 450], "tgt_bbox": [300, 0, 1000, 450], ' - '"desc": "A steep cliff of stacked yarn skeins in mustard, terracotta, and indigo."}, ' - '{"id": "needle_1", "from": "ref_image_0", "src_bbox": [420, 250, 550, 450], "tgt_bbox": [420, 250, 550, 450], ' - '"desc": "An oversized polished steel tapestry needle."}, ' - '{"id": "dragon_1", "from": "ref_image_0", "src_bbox": [100, 600, 950, 1000], "tgt_bbox": [100, 600, 950, 1000], ' - '"desc": "A large, fluffy knitted dragon in terracotta and mustard yarn."}, ' - '{"id": "fire_1", "from": "ref_image_0", "src_bbox": [350, 400, 650, 650], "tgt_bbox": [350, 400, 650, 650], ' - '"desc": "Curled puffs of fiery orange angora thread suspended in mid-air."}]' -) -moved = pipe(prompt=prompt, image=knight, generator=torch.Generator().manual_seed(0), output="images")[0] -moved.save(os.path.join(OUT, "bbox_i2i.png")) diff --git a/integration-notes/knight.jpg b/integration-notes/knight.jpg deleted file mode 100644 index 65436c1e8e3cfde30980007357f276755875a0a8..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 127590 zcmb4qby!s26ZZlOOGzx<3oISNf(X*x-6h@KAR!^TOS9Cj(%oGWO047xlEQ+DfXkN_ zK@=2x`MrO<|G)D*=dZbE<~--lo%qb$e{26X0O|lz5)u#zF)0WHA|oRur=-6}NkKu$ z0;Z#;=V0aH(y;#tAs{0oqokl@ zzITrq#?8(R`~RkYBLG@bA^@=o5djZ?kd}ammf#--zy<&i5)lv(0RA5l0SO3+NdTZb zrV0&!kdT0gh=`B~L$o z3gm&&yM!eD!vQG&$EPKt1tPw#&F9f_@2)`FpjlGfAks@{9r74LD+7B5{=DH!agb5r@Ez=y3y z7k+W?jWg+s_B{#E8}3d%?OZl>j3JdIvliC0AjhB{ww^=vWtPs3Jwd~C84u)_FxpgW zPYhw`y!@5S@wHerRYKqTs%p7(!LEpXqLDd6D+y^(Ih~_gFyVpu4z{--yLTVt7VDlI z2Ao76a#BP(Yk(xuVpRpAJ!s*kI6Jj0z7Cp;Gbzjr-d-Q>kwlZYO&Xz~uL8o9naSJP z&H>4M_PTg!L>F;1Zy?QaDZ)0RgF1p($geQI3`Td&Pl^K+;g{~I*!HK@q_7_(3`r7+ z@0Nxhu|T(DN$BX-($vnxwyo*bk-GUj4rqgENN5O0d1Tsse3tn(n^t0SU+N1R`4`~& zl!O?Fe*Rwfz-j`&(K5S!V3AM~lRsYD^&~&+7?#wbFrRy`vXApjTk=(QbY6EJX8InX zLC&Jq!ZV3oO_LwIg3&J#cp8814%9$IOSc<#~kO546d1yH^~8Q?`*uoB*wj{LyI#k%fB%F&LZ zXpk&Y*T=(~{am~qCqdc7PWJ6sVFzA5U1%=om46O&ksoh%q2)W)lPfv{WMTy6XbZS8MCyIsbf1NE_x+7sbscoD zgH#ZksrzW+@?SGWxisA+ABcBfBtNpF>~8Qe+{94UN|}4XUv|6J9@EFO5yr|TQU5Ib zg67sph7v(h49qIIRL)aKQaIxl^C)dBl)z1p8E>0G0~7F3tonJ5K&u(;D@E>|OVq9BP5 z*0-xI5;x842Dteul`D9*Mjk7h^TDIsTI|jmtzy)!3O}R_31^H5O=nndYKk-Jg13#< zc~06jL)d($j|F|t;XGSv6?k1;;c8~SQKVNQQp4mvQ{Y)`T0dU*o2=xs=w_Zb&@2)C zlWk#68XLZDNj&kYjY2VAXAnwMM>|0yDvN~v%o7$?rmZZVW|#sNl;Sz8!3QbZG@@n< z?#I_^1d0gA{hqI6WbzgA&8gG2J2F@(*?ZfaQ11_Lv}6NSe1`B zSV?WFqm?dLbz42MK(!B?crN;Bm+HGnn-9MpnJ7=nC&f4Fq%h}aVj%R8HwuT;VYb z1>|}TWtMweh@vEYBp#P&|m!<{h~#aE30Yoxiez*=x*#2tcmF*TG0vEjKQ;=ttb zAw{=D=LH4S*!@Mph`*g)Mct^H5C|en&>)$tKmc(xlgGpHTyzf=l^FP4*L>>v<85(j ziIq|fR|yiYw0WUzhjSV`%(mzuX9&h|m9&*oShQqlOn9V5n+_}c%oNj<#Y54KsW0-D z<$yP9xUdTIGW>>1*C9fHIptH0J0grBSf4=^Tf9yerK+^lh+MMi;Pm4k)zjuv$x^n@ zLGVs#K}j{q6ii>t7EcaiG+aeCM+30a0M~%ZD*R5zSaXdV+(d*vf z9rRq9PWoJ$LR#k87m+ul-I+hsIGs^p0^w&!Y8rS9_<1xtb4o!mT&GipJ0%4ATnS=F zJBW7-@0FmOv6L*C?Bz3cmU~8oz>~9)hbnm#=pviQtU1?X%rZ6!KI=5;xys_4sVHJQ zkdVxXri?che|1G{KoGb)b5Wg)%*6ozfxr`nu(jwcP5 zOqH}aW!9~8!G|SB!)%*YTS9$Em^2Acs-RD`*fxUA+vrdn zib?juuyo-`M?(aSoDGpP>$x0qJZ$W$%W_y5o+)FX%cd$VxYYo$EvX1$i%LxR{X7gX zLF_Oqm`^jMIHuQ{UPdz*!pFqV;fGZrkbACr@r)r1=U|-;)=S;#K36*)N$MmOp{9|q z*&NL!6h@740&PLw;m7pc$BgYE%mQ$b94+x)HQFo$iu4*pn|Q2pZ*^{jKy~?b%|{X= zpG77spdr&RDi`t_Cy&P=d!>Q=RoEo%1yJoO+m9F zG$kQ1cArPOVbVvh2&&B>i{`FGSuz)r1B-PkP4e!;Wgw~qUZ}2&3HII5TpoM(%T2Xh z1z>lJ`0Pc#M<$GD+vMX#2HX9veQ2|C-9?U>U=yY4_`oPFFhdutf&}K&N7KwiC(@+F z>XH#|v);p)x(-bplWR0MRf9Xj1W01RI-Bl+rMb{E7zpN(Jk^|Aa7NffIiy2L4Gt5g zrdv<5L2F52o!7={I9?~YiBFJa#C!SIu>VrE3VOT*V@uax4LGg)^#GaV zMF<}(J(BkkXY!>d*fKN>vSjG1v1JR{4&KG*4h46Y-=LQG)a7Kdr5??%JK~;qAG?3& zi%|44%d#kB;dBu?thxNSt}^K}_S7(9=n>=wWT|_zrRl_skSWPslVfJzovZdc5N6K# zq7HjhLhdI+Xk)vx{UrEX!K|pLi;Yg8d>jIB@^_M-C6~3O{9)f4NV51?XsEnVtFPo9 zO;o2BxjdCI_Ej+H=$v!0yV|1TRyIG(L_*FPD$zg_Fl&3ht#jW1Lit-@+Y;e6r!?k} z{Ao7;slpC-KLx=xw{>Rp_uvdZac9t`UY?U5m&_!mP&TA9{KcBFkBja()ZAZDcj^Uw zJa=PV>7v?);r>hlOpQ+>cL@FL#rd|yLK$t3QLI{8|M5dN%CA}q>ei2!*e>4WjSWR7 z8LCjKd%+)U>556|akR_cg)lY_3a}e+$4OML@O?Tqwyq;?pRluOLQ#>7fLhm1Md*eNah>UQVX<}q>krw0TIlF zHVm4GjEv`AQ9^F;mfPHyNEbKX=VeaAedc5TzRpm~B5s@^g_OeN2V^h*uuSrT!LFqgx}9t>^| z1mG2o?4*nL3CS2GQs6ZOFpsot+vq|VKz1bEo~`Q#s-%`a*+ZwOl#uM(mi}`Lf~~Uf zli*?)j9-pV~Y;2CRcS#_GPg0^29+@i-aQT7Fe zumrGoAK9J48-C`^V6&xJg5`Vm_l&XQHFA+cCtnQhLU<{Vn5QV6qj4!v4SyDpV+B@4 z3c21u$R{?Vl01+^bnAsKLMU$f*=AOhDio5lG!|2t0s|ECM3Vtsqv3`_dbv;vNYVt5P9bokCHf})VECwbzXy2 zo(YCQt+ZnVkThhLoIvqU)kHKEVB*+pC;_#j>X%O?{W&bQX)yL^9>x*{uAttoWM0xj zdy}7%eTrYhaT@IszH{GhyX>_;JnvBN%#jaJ;tpNN&B9rUinR7p8Utfq*%@UwTjJ)v zQ3uDAki4p0sZp^Z>Q>KcsC`WDJ|`EZYlpJm&vZKa-Z`^=bQJ12@eeRwP-kouGP13Q zo6|iOYWQeg)to#YLRaYUD$vc{7V$_5JQ*He;53 zoz59tjoo;fR);w&KHVGv$ToNIp7ucjhem0o?QH)5csD7%(jH&fN4Ip5Vna)jTC*`_ zTOrTNR~Az2bV(1zhoxfUSp@ADvdYNW!?5#zr6_LrJ7P}B79toE=(3_Qe&%(HtId~k z)(SgAze2oTZ@iz*YBOk(E+?GWW#6H-_qSxDyBc4M_k3!oGWn4O(S7JJuJ9{|(TQ$E z10s4xgTgfFl4yLF!g}Saq}4hpsFtS^As{w-&p<Dx^@T zq|s^Wqwx(;AB;hBzpc8hC zu*bpnaIU?a{63^qF7%NpAe5$JIdY8bg+#M6>YOK^Me^mxeIlDo$d(p)XLGN>kPcRj zG{xl}$)aW%uY~A2Z8}L9KqJTSF4nvB;*D!vDu4QAxmmW+MAQZHk`7l2*7Lp8Bg2C= zk9Y)6fW9YWa69LmRlO4|tv^}>3$H?Hsu|O*v}${QvUE%25sX;5$h5Ws(G+~l+U78W zJTTBtf*bbd2SVy@WEfDesU>dD5mO}X?QdWwrDd_h-7DxAo$j& z?{Ua8M=7^E(}|ZR1U_)w1zxK`Sdy7_T%assIKieY=W{Z50tIYE!Rw+=fLaIU>LJ=9 zg&A7DZhZnilRU~E7Gefitd>YqE=uoUr`vNNk>homX2dx0$T6P^q3$uXLN>>s5SHO= zbzj(t#(yo*PY(SEJS)ZMSsDvD&x7`#k^?KnV~Wxm3OoxuF%-60Kh~I{&lNaD1Gr#{ zLCw8c&rA6Qdv846q0i}gqmo0!(AEmNfbR(|oYdGH1SybJT~tj4sgWyihbicQH@x6! zu^D=3xb+t15AgcaHDdXqG)wwrh{w1L7YH+F>(jve@V~~&9v9O&zV2&^ zHsSn_$3JXC5R0F$eUCZoS(NxrKKfkztS3r2lZ{(gbkPsLE%@zfh;r5V;P$w%Qe`6U zUDlfpQ&0H>5)USG!xx3jf97mMu`f~Hg-!J~Hu+|=-+}Y-c4mwiHS0sp0}izTzaX`q z%_=B*Vq-PvfI%_R{y(2kTeP-v_1FAk+>@X$D?Li(N@dl6TbSY(@2ofPjcTg`(!05! z=h@wRjUShnKY0iA(e;*0^3gv?zD3{m=gf7?;@-`>>}T9w%tCLCO+ZvFu`6 zz)R@7jv~q3hfRzl15TTh7{g{_b(wX7Ws{o6o6`u@gzxO-t1BlFp^;aIu=C=2A*h?t zBY)gEM1aQEMKz7viT0T{pHnqtJKWh|CQnh=fIJff*GfT)Aow~+jQCi-cK*!%nJ&LO z-Eozk2kcVFTV@*4E$~$s4@aWb<-A@Q!wrV;; zF4)(aJE*9QCI!vs3SaB+WS0D64zEcEGRmcD!W!l9EU~1}q$C9+;F*N~VqK8b=t;gO z-aK4J>mFsDq{@e>zgi|f9wW(Gcn)LQHCv4?${BoYcyChQ-KBkd#*X<+DiKp;E(n4h zK&@b?)C~m@nANaNMw_>^jtLa$5jPF%)c3SNZ@JT^}FP03e2S_rq? zK8kq}#-zBNSL!jkO-VT>d%ha-f>WM zynP!`tRdP0xAgAa_wZMF1%ho8*h|brr#-fqy~CzcS*>AM4+1=?2fMzjslgl9QVdFC zJ#)USJq1d44C1m8=t69JAp7FBg1MWjOC|FzkYr@!p>m2fGsh+hsCN+`8t22nv+LG) zK+byP)Nc%{xMk9jdt;$2A+6)zQocj0PdTzzFFnHBJYv@K5XL5w+>=Coi_GSJCtV;P zSx#QRMNZr=>kQ5@wnsgs@C+DmY4KVW+{?!%dnz67cyGVze)q#w&el?dPbZ%0GYGR6 z1NC(=zW9uGqz6IrHa+bfmm3#Xqi&1e{Z!NyZnmOQFz6L&iDZ%;`KrAdeI*xgAn)>g zu=>Hv*&b>);bN)@DSKg`#f_C&0ydsBOY`6@_gpE_1K;pVYn?B)0$jRdJ9ROUvj@Z0 zpmYf(5CS#)`0R>(=TxMAQM|#w_no`x`VdQLuUXYX+c_1UjuZE}s=Utfahz7^ygIST5Mw6l{nXp9aUxk(Fgd~(wzp^vg?6!i2eVw)y@i`xS zx-TTQI{tQ$y!IJp&|;=U1t`^^Q#vl;H9tMK)AfES?ygi_BjjuEhod-GJ{? zD0KSoT&G=-zfat}D>dwgjtfvj39{*|mh+OhqJ||5XgT^x(0ZT6cUnR3+C4h?e&4<* zm|0+vR^(b+gT755iB<11mo$H;Tc9op+E~*0b*p%rbWIz@`?&g70~2G2K%@rgZMCqR zQZ(XZG_~alsq`Ft-*%3B(x;pw1>Ta**41q{x1bVPNMf1|OPxA~rFxAd#!@~HjUACv zsVT;*+~q~s&byWl52ptstwzq&LUjVr5(q0MO_SLtt|2J!L@r@Jh)RhYWK2D0LTTs< zw^yjZRg64Vph=|VA!2-yTgIGxST+`m23TsfWQu2!!_eh_;KheJEas#@TUHJOS9>t! zP?g4=3#y$@AvsPWw5JnLe?@lUT!O9lp9)-5m79_LppJDJ zIvP3pGwQdA}7L@^ymoRV`U7Oey6Xk^P9XKE%3zl>Tlo$7n*>klM;>ANvT%p zDJi5aJkHX{4+M4TI^w4&Wmxl=iX5B+c!zSU%VspbC$D!f=Hw;?sJzDX=DbH3E@pg) zt&I$q*=NBrOQfKx-Ov$cv1G#N7n;WyH4y`~EP+JK*|wH=%l#qEbQ-TdbV>^i0EaDz_fHSJWz$ z(w+8iIkPsg0Ho0J51PMPpH1X-(w4-?pK(8X=zw1{TVYq%jaTZer~W>!57$~4g7ou; zuUz>0cRxY{Y)8*<#Fz3&!y%AYM%}thlfDK#E)Uw@4!#sAJv2aPM*T>!v=X#VQuVM z?Tv3MApnGoooHyBof9=^xH@Bxk?H&zmaYjs(hU1mEHr?air#}hqS_82E z15l$w7A1Rrs638LS$lsa7cH5UwI@^vB)a0CDAIKlCNpEES|}+XGMZ82r*?QVm>nMeH(Mu{&^sHh)x%_xsyHG*F2 zJ1VSSMp$C12>k;#>CTwAI&ZXf>v1?=HX{rC(wx#48zXt4p83H?2Z4+Gg@1hbQrLI< zK~o&<9uJ;mYQ551w$xiblIaru^T{|lHtBC|d7H`VEu|908%M?av-2R>@`b8lv)zWZ zd7w&T8@B53VGGMfL6hBnZC8BkVLR)Dk=*GGQeOR}4t%HDYfEob#=PvQ!4{6{v+v)YW%PrL794Qph42`ENOWJ)UxRfv$T*NQBy zofs-5H_L{fQh#}eEM2fmd)H8Fgj;XYb?CDH%;^bFDf;`FM(DCjx>bb)1D;E>cg zsX0E~pt5g}3b*WF_UL1@<49Gg_6S8TX=@X5in){|mgqbiw(BcKxJtR)#==XkG~S)T zv)K(33+!bR#ZgI=(-mIF8Hy?Ew3*sJY&lVVJ=Pl8HfIdpQ$Jizx2-`AkG210+!Iiq zpMJi*;=%hQhD~o{EqU$6yR{WNY_nP&JRqX(apLa7t*4 z;gRWDXzX7-3*slS&6XVtE6Qo_hn%DKrn%HX&X!t5nUt-lDi%M07Q6& zC;YXIhhCO*q^G05nP0uHH?x%PsmIBP%age@R_RrJ?4XF+*ql?AN?@3)(|gVK+q7Nx zQzGd6AAlyRf{pxT-+c(L*FS)Hz(6W%;fRBvEZ?ht08=L42UfCH1c_IZ-aGFf;)Wt& zcB=-GePMyJq;Y@CK-;Fi9euVP(JbQ+vx}b&Ep!>(qO96^i$3+=8@a9YbTVy@4eOiG z*&M&oh^FhyZ?>)cMMxDr`2pS)x6KvP{4J!c9!uPcrI*}_CZhe2A~}*G#xt4TgCtQu zr>yH%^~b{sBau?2d|Wy1DY=kydd78mCFGVq0hGDbvMGZAg5ddFw#Yc}l`x)fAq4uP ze9}j>iY75c#OoW9H)K>27L)13uz`MQZ7xpP*bK3u_k7&DdpAeaFpBiHSm;qOxVA55 z{=vy#-e!vv3K_|w&1LL~mP>;k!+F+=y2AjQ2GxZr{LyH&4?-s{G@*PT$;LbgrG{ua z%A*b8Za{#6)p1m&SWL(kO>j$M*q8+n+r%$zH{gTatTyY|f;i1^OFet*V*{m>@*qq?Z^6f}*i;~r*`@4^Wv8#@I9P^lh zZW!vxG*(l#1&zm>dTe(;USM#;`YxfGN}WqdY)}&JQ$q$nGUlla9{5a+z}E@{*eCLx z^;S#zizc8<_YlEXiDD4jd%5eCbDPumfs_22$D8wk`SUR6oIQ9=Zk>Xr(~t>tRz&Y% z3=gS^hrV741fyUu{PZOixZ(t?-M~==QL)^H@h8`cxty|kw+v&2h4)Yjv&zwugufplmKB2pOz zHOcA3j#|Cu53%1IzdQfZd8G~c_2N@5t@SZhB0+wN|1%!YEGeO-Yr+jcR!JneRrJWr z2c(igp(ex-T&G5mduqj(9}T>Fm|R{JfB&s3;7BKftd#zeg;{)yvjWi?HA+5S%Rv^% zRnB~)Q3y}EOG%g?iO_p_J6q7G2Sd%PJ7**7Mc26jz>~?z+i(5dd<2F@{<7v-<9U}E z3j-4>LX`Tkk^cZpODL_z5*@7o&8$Bv{?^%-FH6W@et589^P&1{v)pDv-a{eYT3MWU z-la$DMg+!!OwM5|i7TF}uuwh_kXH%<<1#}UG*6y}4;>Tr7r!+-TsaU+h-%yePzL;# z{UJax{kS>W{$aRpK-E(1U%J`i(BrML?y14L&c#R=(;jq(clQRAmEatX`x?gP*S{K> zvE|i_#YXo0-yDKXmAS*hla}yIL0(@=9 z$m1Owt$oW~zDz5H-F|-{jvQ>yxZjmyjVz!0Z2Vx-BDa3Yy1_+IcPv}di$R2jG+r+{SSlElWhnjTQic3AQ7*j!IGLr{H~?_2KOQ_&)cTLsNx>*lNe* z&*6m&*MWPuL<%Zv-u8?GmLsh%^`=hF<(hW|aE&EV-rknMr|3R&*A){_wV1da#NMok z-^b^iV)q98Nc==R=K!FNoZH%xF&;wZuoRrZ--W0@auDjMyktn>sC-eL~hRU+d ztq>Rt`Vfr{{|~ovu*h)02A`%3>7^C18*OL~n4JUVk~e4-*8U7$b!$K3F>a_(j`1;L zb_P^&*_;{}VZ~-fBUg3Z);Jel>R0xDx9ZqX@>D&jEFT1SrjPJu+~zu3TQ!+{PB(XB z_Yx`dzIySpfA#!Y9BaCO`2LyF>c&y`^T_)Qz)k2N z%vRUq5naYt1vUiK6z<5xUY!ghC@OBn;Xsu@YpIJeU{MPS3<66Fw(j`oeTEq6H^-lU+ms1l@ zH~zejZCc#Fe6=a|w8)vQ(~Y8aL>aLyqvTJA!UtP4 zKUzN4Dt&He!6-hP^cVl+c9HKTvT}IS-EDrE4AFE`vuaz(c9CCZ;o@gvlAR+5w|4yK zODkOdBtMzV|68a$m^$N<1$Nw!QToaEC-6;gU$CWXFyPkfm?x{sP0pgDxsJtip5l7j zZQuLAx3plt;Hb@IVns=I+k5|FAZW~{cGr3GbJUjCsZ@=-R7{Pcp}BP-(V7nDQC!?y zkjbh;TO+G;-=po`_I8e`zg4eKA3D`q{E27kA`&VH*wFq5K%)aVUk#_*ABePs9N?HG z<`b3o_C;j)mbh$NmK~Qr>Q^!fRRDf62Q=@TbH>JCYU|9aS1p7V6n)>H^r+0deylR;EvPXqKZQpDkb%BC}uwLU0x+xdpEnwNdk(yk)&q*m>+iCvH2h6)j+T#7}83F7lXm@tQUBy=5TPR<-OD z=#VA7P-n?olsv;cR^m)1e89OTH8JQrFi2Cgw!E4r1PLPp{Uycir)HcuSbK4Ps_EmZ zyWKNzFA9{S8JPfJxEtQNt*Px6@1$@wzC60uThKDguC$XoPqgJJ`^e2Em9}$6fiN^K zd8scybV>MGJ@13&kmGY*W@)aigYfn{v$39A z?j2>@7}Z*yMjP_R&UzEI#uL88zw7@1K6>4ekN;RR6Nx4OPX7V61KSKy?cYHqQcm?HD;4EG;Q zd#8WvbZhmx>4`ji1J0Q~MyxDZ_z$N>>~O;#KKw&AD;|>`eYWx6*uJ#lL!;%d?7v(m z<i#zm* z^dG@Fu#;A6(^2Q1x}`Id>DEo1NZ9N~^iIazK)`H6MEA=Bk>g;VVQr_{%`bzL)Ms6V zL6^5Vk6Y{ph84y9ey5Cn-cYMgJ8W85QZK(BmXsW-mU6-Q)Mk3MV7MX3LNh|eqJiaC zNXpNw{efQe2zB5FC=^7O*HhSJ8gE$LYH5A=nJcJW0;d8(f|O?-J^lngD1B9v{O-wj zEoxpap!(W?qapu+*1|V!X62YEpNIytt>vg=-0#4h-8h!I#U83YR@RTgq>kH$^xk1sI;2nqE5An28muZT67Oxngu49I^YEX#8LDO_Rq)}; zBF&;>1zB*0bIyC<@!eFl_&myNkP~81JWGs)Yv^FymXyKq%GKcjQ|u!lnAO{ASRD~O zwTU+Mn52sN8CWM_L0=v<+if;6Em6+{xjZL>l6g1u?ei{4jE0fcQzKenj(!sL{ckLgBD3a1N9rx!5wVkvj~>d5W+=0xouHA>-e_l z9-5Cj#zjPY2=yOS^vuRYAWd7}Iq9BVF@jAlwM8d+z0u-~ndZS|YMSMs=Fu@Fo*+X= zbRS#a%8uZ6&q>cvOM+1G4Qd3OEN#3l;HOp?TlV3?xOSilw8c1EHB4j+_#H|i4pYLX zOi{>V#*VLc`U(_7P0FjpucRI*9AnEBrcWV4xi->*36KuOVx4<$qS;6Ajs})eHe2eA zBP(;f`Z%BbfKm)UR8t5cYB$;k;T*Y}zuKlAB-jvmeFRB@`2`n>Z?TZECH>CeNBgs- zM9Bhar?3A~H@oTA#D22i%T{6LfSWwsgWGMsijfxcdsF%Lvp9g9iaz7&Mj@Vm;zjkB zj*^>43}?@P#W(oEbR+43fQ;&$+=?r>M*v(=qdV=WM@s>-j(t{1X~#lprAi8J+mVl)*72>gC7bPm zGFfYQ&}|y=x3>1Q6@)Me5R>)Ufc(+pejh{9*@JdDobvriE;{SVOwR%lKlRvAGmEFx zpM6F4Ddf!(H>~IrWB$0to-yq^P%LwI2d?K#a@E~6_PQ5+Na~ClQi{C}nK`5h3)fyx zixMyc|K|OC*07%YG^)3mu0MkSF48046jg7dvIlS`+I#EsR!lBBKxV&Xpmjfo%5;#c zmdAK8m!Zj_LxFCK8Zg;?T@&di*5pPtMHP0LDP?79L+lz`8Vp*VolBp&MarCVym6!OJl;AsNbmy57NNf{YL(E*nyJdU zk*+MZ9Xu28-rDAlf0Qn&l-0yN{H|-~<`(&?$I9gAPT)^+?E5dADrb_t)$3c!PpDmC zR!70>?C{jW?eYH(=wih`L~`=Huzj3*q#r;zzA3vESDf_^a6UBMWgXbscZZ2LIet}> z309p}yj9}%i{7b-crv*7#rRCjW8PCua3fI)8FsA|ArrWM@SCqPh8zHOXgbRn}qD;4P?p5xZ*-)y@R3m$GgeD+hVQD zY5PKsh6x00ImOu@!ibwz9P7LHFV@V*)^>JIsVmy_YvW2*H)jXMqfIY!P6!;W9jX`1 zg@xE*M8$vOJyR_dwO3Z&jDDqHzwPk*2k>rMwCUmh*iIN(fAbV$*i(2Y+&Z!3x*?Ug za~qw}H1Q?i;mXDFqa;^ZvXlFzBv0!kBUbk9R>PYDjmd3DltZ>d2W|_R{ciLlNQH5s z@ZhbRw|g|#oV7U3Cx$9r01otP-kX{)xc=KmuLi8rEh-L|S;xgmLyZ#T;Qs*EL3*n9 zLjYS0ZTF|nf5A(;MHlH8t_MfQmZ_|{U;~_Wk!kPZ9c@4BC!CC^ax!n-qOJ|Yp$*I7 zLE{ehS+yfI{fV;n<`lSAjgkk$|EnMDZvr~U%Irod$;U`lv^i>3wpl6~09X>Jx#(~Q z;UD*dTVYRAN#$VsWk3G7+SaWwiMYP`X3(UV;>PZtah8YC3}<~6WkPE$5Jr2YGNRyi zm}8u<@a{t|av@Ok&EmJN`ff(fu#u5Jnh}m_C5ojbyXG!{7hfCMds1b@ zwWzwTItkikzd~<3*{JVqvyGx}{q?Q4i_rQQ_8^|GbVIE5o0d>gV(8l-&NzeWgQ5p+ zAH|(Gm+Ufb-bhkL6q)Uvr-7n82pqVQ&M$J#?l0$(i&-gFiB5DT20DaX6V0=adWRcb zziM1kt_!oWkQaaGl^vA2_YZ*my!f)Geg+Pnm{^@VQBpb&(_RX<^Z1$U-%gZtYrJZo zo00b4^@E10@)}HMBI%^u?Uxy!l|{dWRYmPDe!E<}M@o>0PrBh-HCDt>uR~Z*yt+v} zyX?i3^5vDs^>*mcz_8@kKJDrypQC^oNwc!ZogB&UydL`+E;<_i!K%k{0ZLVt&Zhf4 zc9Fkae(&D6zC7s_8nS34#U# z!OIAip0Lz8JTvHsSM-{#Ii5d#=|HEEg-LO}{|}}11Kfb=jHEbLa{5SVN#1^WIdi$I zlJRd(y90XOsjc(9`g5Y_&>GcG*Z8~X%(+hCjr}*v8haUB<=64Gd?hM}(qE$hqH=s$ zFi)yB2NTxAUmd&!r+>?XRv)eW159tCAMd36%3o6qG7ETwTj5xDMDt$1v%-`5#e(Vz ziPydxU;Qz_F|_sAHuH3bMmD%=PyfCRoqucHVB$xn_s3Lqml_$DhYwWk>nY@~ys)#6 zRBXyV@Oa+#eTDooIV^E`tX-C4KXd^4WVa^vOBDk@O?m`scSA8{KFUVp zR~YspIR%tHh2YvQQt34zRqSp|Q14^+6cQ;>pe>CtV-DY#ySvH|l?$VZ5C~-geWp&8 zLoWA1CrN{+6oTpf-|t{y!~;hE1&BIlbikfXRT?y3y0yPh&=ghNk&!s2?<65~PZ5rt zCj!EHc`a=-H1f|W?4752Xn#MqBJo=NoynV_>+k6>GVDCw0Mx}%s>}UXK>wbkNwJ39 z`4UOeJ@VZ4$kpNvv`B%J$xZYzS(bBRsPj4Lcj)##^7fx?k{Utbp{*g@S{5KoUoEtu z5UL?aL*^{dr%NKlTq*!w%-YlyJjU2&_m9bKy<2~sY!9ZY#u=P)giVOlG7xr1^k|Ke4Iu&!Tt`N_`Lm(YcC6*&F-i~;hg&p%qj)0 z0sh06e8XGbNow*s*KRx&PHHUpZ*JVjFV#bDIMUBkJM9WayIt zXF8BZko78GRSA!3sW1^VGO#p+s~LMu?R^tpMNE&PV5RM+^`mScnMfN9yXT|u?ha{H z6M*$U+klF8=UYCb5N{x5d_+7miki*P{A_1$=BO-@a!%%i2iwaT1jv)tXw#+!@{q94 z(+acR!H|Z6Asy+QAO*Gd{EbBBt>lpGTxgh1L4TCsY?p5r(V+LW*X!-@;fDs@`)}#G z!XA>7eV6B|#!F94Tn|eSN^U*hqCGFW!hT-%Ege9n{ivfjRr2E0y9jZ6vwox9^xz@A zz%p-{SIPw{UaH>r$;6xYu%0HnGC#WK=fzv!LDBa#1sZ;u5qPxN|7w+YJAGvIVqso? z;Xt%IaV|b3dv>P}yM6hSzsj;@=+XYg&p#3@CqC4%^5l&w797=O=!0LjT02p-uSYLF zo|k1W@--e%R4g$rm0LjzH)AHTHcRFac281lGk4y|VkcH?uJIvuMJ@gJoWB19AfCN! zv9?(J~e083S!xV6@ukm9bXA- z^mC{i<+rtj)>`!4o8lqxUp#2Gav2L71m)fuyh*S0xv(_5?p61t?3fI7+mA$`Dg(DA zGjHYkmVNT7_dZL%eEYb($s>dLJZWrabO%w_xDKD9u*V#QP zCE!s_Qd7H7pg#9)&f?xPHAwBql!L*%!#9Z_`0?)Rj899*b(}I}pb`jo4>D z*~_?5J!N^@u+OB|6u6;#6q0vsaB$t5+7zp`)6v@#_-5yJTl=m#a?s$Y>OYhfUqga% zRq*QNNon&AT>j)}w{p*vXJ)>eJ?`P{HqjMAj(l0Fy4EQN_`Sj+|X0UoBH2U|b zvdsS9K&{4!ajK;`OsfR3eR-%db{J+Bf~ zT-M^F7X}dvU6d>FE8RLQfFd;ZE>C%v9|zNtbaC%XsF6#ryhzZTfD_!1sL zG0I!Kr}#$M3FcE*^Vg77Um9o0=(wLeSA}E#^^3V=Sj5y&4_6X)ddup=)_;w%SN8E) zIsjLS*!AXSPzO!$I~{}UAFQgi6fimk;wQEeK-sHi9BWSYD~Ah@tzKD8@_I-9{o(@r z(!+ERXlFIS(8h=Zxvb44d@hOad(hP+^?61+C~Gl z&0KnCC!olqn~XSrubD@jWp#h;QX(r3hlFM`7^kl}m$Z#59;1;JEe=s3-ER0W*14jV zK7-!Mm z6ginZI=*oiEi*9GT=B8kU*pfhjiq)O{rSc9k-co6TC2<6@#n_xbmkqXJfRJjLUjH| z)9UsAOlVSfH|o2OroTt?5?%bkipivWJP60(->oFq1N3%ErLs1&Q$*4x1&mfL zo2CZd8nUIf2UCCKf@yNotbVga9c#^#Cp}44T>oP1& zPMzOVTesoQE?8p+e^Af(bma*NpZX~CWx&Ht& zosu)#bE|bqD^{+9xBx*U@<{GyD^9Ue_!*s718LM zsJKS66#~xocK-mLW^^rVzN|fF%|g|K`Et0x{!=c_v~)G>rre5f4~Vx>^z>E+ zrJC?ZCd}j?v6xl0cQM=*`N>s$mTr7aOh(%po~X}qD=4gOCyv0#PNo#$h`6x)*x-Gp zK+)XIxst~Zk%7lO@f>|8q-bZsEZbR!Y)az~k>u2yEn6$0+iBd$8#xi@>rPHna^!n~ z;Dh8r3Ozc)tMEV|sNno!3fmUCNv41Z$la0zJRtHprt3oMQ)#ZO4h@GV@6Tw?s_QiX zmb(ZBLkGw+hSchdZefkga1Z)KEp_T`Rw&1S00Ycs18$qvTWcCx{p?#VJGOaq|?e(v|E3^^$6%>8aD~Rlr{44h&UU zDzkf$j^jLLFJ_@?rB#k!k`Q-*0reZyF&S=1`1{QT^>>?^!k}($Nik}w^xr#6F~=G0 z^^K;1wvm?al218}>3U6$-qD3WnTV8@uAyqFHraqTGY}_Pq^#*^6@$snVwxVDbo~DS zANq)oGvtmtPfr&fyRKR1^#l*+YVP^Oj2R0riQUDNsq~i#@gzo3Q7hbJ3;Jw48DO{& zmRxK-uqEhEQL&C7l=RNrnPpgOD%r=007W52ApnpDC4iD$dr2|`lO&AKW=3RY1!B^L zr5U9QLugIY+vO*u5XX}jv$=+20R7B}XS|TJz(T6aKs;huc!VkRmD`CC708U%{CA0n zRBUkq&`01ys{;_O$aS(iRMtcr9%VQEWEbXJ4>qvW<;C75j6U52w1m_Z|CFPZ(#if;{uxob^Zrn=llw&QBtZwim z!apWdE?|`%;S^5h-%-65-}`?m?PD<5Sfcsc!mctNj%EpNHTGmDj9=w?Ti(GuyG$it%utc zep(*Va~3`1p2WJ0%4S_;gbOYg={$olp<*6I`a=OBM80LTkR*X4GC_fOj7phe&}6|O zgcA%x9z>&YB&H;0L1{*DBC|9LH1j+eaRl*c;VGQUtdXiuX!U2b&Z)3OaUX#ZOfljw zVTIk9Mj47HV9sDl#%x|-iok`oTqLkxq_`4D3kVi@M7R+ESe!*H7@91(irZ1MuNCXCe=A#4Rdv*}z~jRlX%u zhIEy_RyfaGl{+V4imjIemELX53O6s+t+1J!kTwzqe>w2YI4w|~&z!z&+4}Y9Y!F5!z zHKSv&-g0v}+V7^;w^LSwLU#Uj8RfXo(qM{p9dA!<#^GAz<=>Fs?JB69LtLuN%(ad{ z?96_C(whFUUXOO0N_QfFFtTe}YSbv{+C@sM-p2!R?SbBA^`^g5c?R8-fHPzAp<-2a zTdp8s6KHf{b>q1N>p^g3@x zNt`E;GJjc<)w<<+8kUOCZ*Bsw6F%;>rs;GCxx?yQ?&S8C{-W0Vp=))WMBD3X+*YTR zS0oelnUsFi{u^6i0hD8o=QA^?b(?y=pe{otf6^@9E2W zDpW`$ZakmTX6Sn6qMOZDIR#D*PGeVffcjDCC>5`@c<(V&Osd|x1!l=CtZ|s=ZPe;? z?W=Qo2dT1!pcs5soAcd8Ob2Pf$>^uM%v)wZgV%76oI+<2P0UbUs_e#=Lv zVdHQ(=1lg<)Mz?|JH0DKx=@)cxKcConQMi5UXnd$#jWnGvRjOHKSPMWQ0f|uKB20# zuuB}EXC1NQiI>+k<$Lpa6YC#r`FP{ zQtIs|2XhbM2lz&34S!R#yY33_xG7M$1HUt*>H6EO)2j_a+>6Eu%OLwlOKyXv>r+jB zqOCjS1$&&g0Ixie&p9%-b&X5Cj7^nQVtPwWe$sLMPl?Qb)hV^^8k*QRBWn)a9D>~B z<2|O9m#OD9tvj+Yk_S2C1B{R73l`cQyHORKT$)ly~vqXE7<wP^`{sLF(lm5w?bB&Vdv_>j&Tu)&jDp(Y#qUcJi)*tS8xOdQ`H$)JTK#>s zsdhEv+n(o>9cIcYR8`RRE*rS|}^D^xHqt_`mdg{YFLXL6YkNwR)oBKDrU) z{u@=1&Hz1$j~xi8*lDlRVtwkaN|E4!2Wd`q4Le;~)X>dt;gE+T`9So$mbGBbV`PHc zn+GG~jwLA9GCEQc#V9{JuX%fF2U0BZ{B0jUNHp9_sRTW00qbd&?K`y! zx1lfUD`y{Gaq;42(|c*i2-X`2Ck@6vpEEnCYHYU8LYMT}LCUDk?2H~e<}9Dpbt@Ne zXInPsBh$wxo_Oz`a~RY060{X6pkq1QIsX8jCKpZAEh^f3Y8M*Z9IzlSy9Yl^{my@S;HjaOFa!oa15)M7~k=jq3ppQmrdI-q@u z4p(z_A9yaMU7ng1P0pZ<1Hf;u_nF#mt6^tipp{}7fCOjeV=On^bGJ^Gy-=+cSRJ78 zp2c|vTS2ciXm72^49$U)f&ddLt!;Jy%79?!8OSH;0=E)LBOqmn&-Iw`X4AUeOZ5je zsQ}n82#pJNN`)YzkMY>+u8;!ji6RlcVOaYK|JTw*M@ zYP9M#OhN#ZE))*L`OMDhn&qvxRB#xaj{yB;=+&B5q+iH?H`M)RHl6m$qYObjrWH&EaV#1H#_0_F!ETf-%(hl{fh|1G23iSy(&LyA(8==< zx~@!Oy1Q{JQsy*^j7?}IYryRFsNeYU0WD|G7R=J zD5o%jDMFc&5PVDGND>N#o-)UXCu~V1w5c+Mqyq|8l`_%{P!4;+XP8GE$^ir-^N7kX z5EZXz$_xmF@qs)$I!VnmWk{~T!BOTz} zaAFzuuNjIz20v=|h}bY>g3mCk%orTQRymR}TE-P{Agdjr#{wZ8rAd;;1)64&1M@5! zF>UfipIIFkEhtV*9`bz4E=iJH30GfiC|Oq?*$tF#Ae_TI1IUEUDVZ(` zy`w4>TLv>*c8P-xTk#ZiPZH27(&3^I5rrZHE~l8j>`RB1 z(Ea5Zawmb89EgJLUVKOuJnABp7m{4{tCb$p#nICHJ^j9e_QIx2Tp_ zTE^O0z06lQ{RB3Y4#&yujwAXlCZ^rDBIdo>1)Bkk1vD$%Y$Q4;EsSgp@%9r^uTv@k zO~paLZWMSJg}pb7?WJm4R*WMk4tt>ZnD*+ar$cV6G4(1er;fz<{!y)c=B33TltOZT zOyzo|?Ajqh+E&M6GRkn@6U&!%@4{*_pQ1yq$)Z^5r)E3`nz&?}Ot!+Iw zOTM+Q)rDaxh$=%~U=Oa(-eVT1WciDAri{JtW^6+M@`lF-D zEyUPLuYLs8*U+23%Tx+vpF_A(Mh}zQGeY*OPEaQ=pP9of zbzShwanQYPgb=}l@~k_FwbMTbye;76eage;VbEALVwgrhCqI(M@(UYTm}ZhXZK<{;`YoXKq>BYHkaT`S>s=sW#6^ zn!Ew+ED0dZzzOKoU5!FH40{IcpRdH(>AfzkjJ7Ior(*o*>ay;Yb>nZy@Gj(>h=NZAHOQr#x~SRiR@!jgyA$#M0Jw$yPPFu^tP2A% z3{N;8@XT8J#k-H;$XdwS%KUBm5h-q`(_3t|kWe<~LU=jjgW_5{bvn+YOlB%+kvzk z9mBRW_lUln-|aU})l;des|y5XEl?*pQ`r4H<_AwxQB!JIF4A~hjNo}=@isqcwPuRJ zbq>o`0RXnZIUM;pGn4#xukCQbLS*D)kAaUfwjQ27PODp2)RWR0hSOFb}^DeG4~%4TD5EkqDeu6v4F_O_Y(BnwpUda+M=aq9H<~<{Nis;dv5eianyAx z>(pMm4Co$w)3oYPV=Mz>#(d5S%YPB54y#%m;XoM9IL;?~(d@HX(b<~?+N>)B zoDngUGO_xy?XW!ssAH@Kvih+n(d+F?uz@>kQad_{fzydvDae z_2VT+Bur`9t!-A8Yq7NL1UCmK;}H#ANo8KLp{rml*!yId%@=Ikcb{Pkkt1CH0Ek`X z#t#g~Bz^eiFl5NDP_>r2+nLL-aC!004t$xjps9ZVE3)=3;E=7@H*=9W%~2QXN-Lgl zH)Dc(WKB&C?2NUsR$=)^#At-?>L#sVsno4$`?crWj{JW)rA^H09JX=5Zzt$t`lPc^ z1F9(6r5ipt#LUy{1$2trmKX$qf%k==q>5@BRqWL6Dw0i+FO8q+ZTl- zag68u&3!LY9+1U^0>0%u{{Z-t)NZd+r}iicqX&)%oUgVk8ogc_KqXWR6$gPF)oI$* zrG;+l3CnHqh`&tIR$SQpuJ2;!AWnwlCY2@AU>t2+lQVRf&#G!LmO8P{LB}H>;u}WT z8}a#uaCpGX4^kC+Wzw@_dBGzwO7#TSw4JyZ1HM0$wbs+8{AFY}r&$W1?ZEm>twyHX z9Pd{EdF=X9Q$m zcgJxYqf4crS+<8v_Bs5&dD3xO&CW)4lcj~EhQMr&F_Sc@(u#_!X_t0#a0uh$h|YJ6 za0i{d6ZbHo_HEh+2PC&7pQ)X(!bs04mDP@dR;9Y(E4*srwxXPUJt zPpm&l(qPY>QvCwGg_V>+wuqqwFp}5Om{WK4giyC*ZKEG~UO2qzVWTZ?BtjmXr1t_6 z{T}P4da%R=Goc!0dbvEn*`{j8BQrM$*PEX- zIsOy;pv@L%@|n;o7S23Ml|V6pn0!ahFxHF$Y#qBq*(xr0ojYkLx}pL1{6ycQcVI;4 zmKHTLj2jnkA21BL#N0h8*TG>tdVO_}Sq^bC<682J9fn}1NLY{=AS@h58DjPhAc>yxLrG!Gs0L)GbfY&i z}zlVzJr*1U-HvjZ|?NQ^ZvcAvfYE#_3XExBQtqhyc98 zppr7L3a=Jk@POhXnhlu``CyggvkXK5Z)lx^lN_|>D*eomAz>78JOr>2L!sgn9L3kP zDxM^g#s#~>k(pANf+|dm!!ms(0(0600>t;|x6 za!d;eCG#UWG9j|TmBe6(GO_^(MUjbMBvK)(Uebt}Jj1BKlC(iBnORmbOYV`FBJwxG zXhvuzG7G_$%o2BVBs>{1yQGaRpICHoL_u4aba^A7gDnUd7-FD!0&>km*{C+3ZAw7ro_XMW&bzOIjzI;hb>Icb81p$Nvi!YO zA^_gy;~|Xm-*0vvl+E1Wgehy>RXeKf_!}@j-`ueoS*xy zNPWEE{C`s!qd}(AXv4A~34j>+lS#Jqs)~V1_@ikJ+wTR?HEVFS-2hTO&GU~F7pUtF zrvCssbntfvE`fe*_2%`rhzt&0B>w1Yz5ASTP6#oF*MdadgUai;b(z97K z?#bsQ%)5Gyqj#uXQCRx}{XF2nXu687t?kraSbkl@uqVXUhT9*s0dJ{2B0ydUZxfAu zHJ4wp&Pe`~cdKfDB7j35KO$K)si@QbAmB0WpEG#$=bwEtm!wr~AyZ*&O=f>AL5i%sa|gQq`%kgJcur_LzVgj3uT#z({gSz|d=vb9oh5V!aEk9yZz)T>5PJ2}r|#5w5| zChI~BKYR%)MS7d1k3%lag9MSoe~w{v`me`WNnMCI!kiDJ9=k>{Wt9-bCJk-mk25Nr zW%b;YvhAxlR@<}_Ei-ccYHM^Fm(BRA77WWup z+FV+MHqxYmd=VmrTN5VDtIh@q$@@S#SHc3>9m{x^3vS=bkUaREo?{JVWM^SGJ%QqO zpR%{T9X(U(PCoOBsM&Smpv2p6wszVIjWnrRN|jBg9%nrLHH;gmDvE96+HzWQl{2C- zTFV~Wf$k?k)Ld1hHI`g4AbY&ddr@NP>O4SxUfQ4XCo!s%ym_1#?(mMeGh+aHO z=@?s9^1(qPl45$Ss11_2CkM~oQ`)|n+P8GK6*hN}eWQt?Q!OP(W!sRXj&Ywc7aQGj z%1Ph_F^U`wE2vCkoS&rK^u`yqoz}9@+3tst;DS5DX*4w5R}~UU{$ROSj;mLV6?{`*O-^=Aw${lUC6{!gNkPrEWvsNyz zaeRUDdqmOf4$WBujz%XfCeD*bO^91jkFM_#ze`CF9OoYqjb-AEHMa}wc69}C zz1z>)Zew6~8f$c>q@-O-F~Py)!o3Ftt3$Ww4kyhA?G=<#IUccaKt;d<$elM${iidHCs}Ov=W&+w=?!n`sVYGE#IDBMyJ^SFwOWL~PObwnKahstev-wmTmtN) z98JpSmO2cBZl8tCW09>&0(b%vySKGT`$IKBj*Ii?|EZ@oqn8t39 z#tfNeFl59C47e~V+{P=dtT-Myl4$8QLX_UX?D3dy`p$;bY^N*Z5e3xKt3ZcL=m+r@ zz!Ht3&>_mOQbtJ+ne?>sZCiSmPW9P?lr82jtfN(^r*@9^u}^FmL(uwmsnX)l$Bg%v z{{X7dp)FN)>?kszj`3!)6Q%VVY{ksu=qA>mN@%GW?sK=ej*r?3j+t)!Kc~2I&D$^u z)U;Z&U}hVL@gm#P>Sze7>Wq=(#dVtI(^?k00SDq^UDLl6Ug~u_bK95)>Ag!$R$cvX zw{Z;}*GpEvQ|WH?0s3Z9`r?-&?4cd!Ds9!%U%t(x9z#cJ>O)ae2d=vvM{(v}FmAdA z{iD;0{iq1}Gu8E7J*}4;bLMlpjcqTdQD%^V+c=8Xr+YN!{WbiL;V{%&uF;;F(6TqT zzC<75s@b`0IGEiqYVNCcp_JF;2*uP?8+A;1INP2ka*anpMy{_!IqZ8u*<#qteK0dI zHJY_PooGUY1Np<~`kK_E)gX@?%=mA%rdsIwP?oV689lgyZkB3o!E=uie-T(;TBDKh zBKnT4btoHd;C?1NHhkB8Fge@pKNySt9ZYVHsYU1uE*SUvMjE>9P1W#ba^h-c$!dxp zq>M+3cLie3fHxD3$4-)hL<`niu#RCBd5gV(1A#I~*qcra7CDEolG%dCGbI>WVV$ucgh|mcR-99M))Ap#LHw|)7!HHx>z=4 zuqHDjWR)<0W+JE-OtvSnB(7lDp)mtt#0rdgl_pfQsd*=xnJXY1%Lx%w@F78hQQ8?q zK#C>6#HvdZh=-uelG3}Ow5b`M2`4h^Bp%7_klSamBoIkpF)uxdC2=eyfjuPgIFQdO z#tGUQh(cpS?Eh&&P8OJ*s2%9-rudLbB=QOvGol=CDq4n!~z zDbI*dWRiG+bAb$1nMqa19pdkerzZyxssvtpbP+N^ffF*M$(3U*Boiw=JOq-<-W5-2 zMOals#w3JO%war0q8#!e&`Aoi##B#cXgo>S>M_rJ{6(>m3gl$xvEoL+LHCcK0Gz<9 zSqq$uXgw8esE})4b7W-L*!%eZ z0C}3;KF8|%p;H2I`;QyHgpoPjeV>LR+SDDVHMt%Bgm{~_fvwV-SuH}KirH{@`NYf9 z8Y))xPqML=XYRy*&zZw4bza^)rJXsWP3y03Qlx5%^R)1}`9I1Ns^0$qgiSqpW*h|` zVS2sQYV5P{kM1LrUt$l^VpuF%Yp-*$MI-_|^Yw$H9JAZ52vxJ6chYI=39ClBa#$(k zZh`6Uwd-G1U?srqpAb6h)LwmJPT}ohd(A;kXnJiIRIRD762RG9Bx<#) znaL%iF|6(zr%KmqfVKB3jocZ%dR3+fNytzR3G*&I+!Dr3$I zJ6W^d{3soS%GL_^*x}fm<8CuFwC-)MD1?unqzBMH;&Fy6;ybbLCr{t`Edi_ULtKseLmp z!0Ol`LGAX4)2#?S8&QyCU}G?vUaC@}rcyd3np3>~pE*Swo!oZG1>odA}^W*QL zJu9cJdn~(#Za~E7j-Z4a4hdgy_%Q)RdxER7HajP7n2{n<`9QRn2q*!JnH^fTN72{S`QTJYRt=brO6IzyYbrD~Z}$G0Ca z7PWi5<8UB?qEKYGO}QZUhe}zj&kOhS2Crqc8Y-Q4RgiY%XOqu}m1M1M+s;&t!x;Jh z0BOF=v?r+JetW($7ykeeGkT3l>9w`iicbWL#$B{2kG7i(6>b5#T%+8 zkUU0uC7!mb)Cxsy$v#x}{!?44Y3rv=UKiMG!ynVX^p)$Qzp(1nt5c840LVD+EomBx z>>&H`oDVT8be30ASeE--XSW~Mv?iG0YxiKD#%8xNQqW2v#t7hKOuC#UJWWR|y_r`P)Hi@v<{q1tz~F^Y2Pe!z?`?J2 z!x5BVKKcIuGRbTH9hf27PBD|*Ok84bY4qWTv=VXH98J1(%cslRX6o^rr$40SG#b{W z7`U;Qfw?$kV0b^?Z`am+H4Rm*kUus(!#`P_X}I6e>gzR9LslDUCvfl2h#sR#>O)wV zM!{cb;0ewCD%}-T+g?pxpysSwn)1Q4jm zc>PD#0rghhdeBR3UI`6@x8v(Ne!0`M`l=4M3Rye+u>CwvLsg|Tlc)?*w_SosOb^CE16o-QG+Nr5p`p;HdkH`VtbiY*BeKt zWi@U2Rly%v*|@~&?cV;%bz5t^QDvD{9@0+}v)Ej*E5~5o1o>}8eVW9uAKlfAx5+#K zK9>5Hvb9`|pmqZjIQ!FCyvw(-2kil49OUb|Mbrd@O=~|cII&w4ce!ofdy>Clha+J4CyUzm{`(kD4 zKV@sSwKd;M)M_7Ibb5U=hX-i{WSQ6MI&PlrC5UwGWwj{6Y8avY2OZDJ zoR+Jp^(`LiRW8tMAqq8M20ZW>WcHmg=9yhaa^tc$snTld*ok#z`?7}MezS9?>P3~6 zvtLH=aB$hg;Pl;JR`oJps$cO<^)-5T zwM3ud&od(b0NIK(&Ti3MNI%1ziSvrOrQV3mtjauQYUn>}Uq+!_otGVn?Gd+3+RAIM z@cN1fLBlg2>oV`s+of+?NpAW{$pGhwTKcY+sjf=y1M_FRjX=9fqRlX^^v}Qm=Md7A zuCMlj>UO0~u@46mU#9D;brxz#1pZ!TAkbH-BUI^%m)z)nVk_1X?Q6H>$F&p;Vi#Aa z)a$zFy0tbvHa?Qxd*?HotEuWLu+VoG9m$&YYK3ZRCnS#(oY(rFi)U8@919_h+l)u>078(eM@BxxDMm%K4kjajRSI=uaO(jYxOiRZI9{a;v;U`SE|DcDLY&a zRzK3Z%2`${d>NCX>s>O#jZJC}G0btO>AIEQr*T<5xs{tPio4V?AnpguivqI}2L@tj zI=-5}+}d-4jLlPEW%cVJt0TncmmIOYa5XvDrG89Cw%nFowk)66qIBK91dz=Yl8Ec17s zStY=PQ9ObaFCC&DRqYBp$O8aL%u?CJt2Yu0K{G>TCFOU)GjlvU!48L*Wr44}I1zzt z$jtPN#Ux9KK`tbdD8^WvK{1k5+Yx&j62|5$V{-zAY)MlDPcrGu6*QfMxs+4Dgd;Od zgk*=okirgDlS^+9o~%eY62bH48(KQAnSIeUa|*8{BM}0|SeZsVO$As=>@UC>GR3fE zSdqsgh!hPw7TR91_hBKl?4u#;Z=$rOGH151?50TtFbO^LDq(jr03j9TAaR z0kM&c=Mj%r3hIry-0`?$6E_{Y;mP9U>gnf8rTjs$i`Y5ZGslqvq*yE!)s;b?Vgvm? zCq=K)*K*jP8rYsl9254O&ZqW&P+Qcg+tUE!Z(-(fxY=I3c_vYtsthhhubVp$8;96M zZt3@3lX@SnJ55a{jda^>b}M3@bCMhRa|^2KXqD}{zixjr@C>~)<<~*!ExAI@Jov;t zH&oOr*ue-L`*9pwDY=P7!so|1{Ut2v)oMxn;CIF`iLLFOGpxA!#G3&az{&9dTFp+Y zZV9XqP9}=!D0>yXFvUhW>$gQb<8Zv9Ff}4&&x|;xFW_$TDjZTyC44{T9pK%?>)&`!8 z`b$l!bJ*pM0GiuAH*09Rb!lPT#0MD8a~jjNb?O$o2nQuvQtNeEYKyoEKAvVsyMCq1 z0=OH9Zs0@PT9D?yK*H>4^RNy;`uK>8^)*_x6ji8pE5Mum8qh(p)SdY}uLJdq^=&nF zXxn0(u6Z)pzpK@#TPfY2t^teCO`}pyFwDdZck!R;G1~X8y_Ea6ah3l7pIG*a##*rc z-J>L!Yerc>*3s!{*4nt-aLY5**O2Pmj9^Wt*{!Gdc_i3zk^Nw=(XDo&10{g~f2awm zSjub-2EDY{s=|FN<$3(GSUTB4+gKUS09F~8cWK-l0hK&tj#e?FyftFC;g^j3%FXt1 zy6MrV)SC8HbGSXjCQMSzF0!jlo}Sbr2b>&z2h3G9R?r6LB?p}Nmu{izw#w9u_al=s zYL;E9_12}VwWMcn(jJkkW*}BbBL{HAc8e5NNIN$S7blQE-aypVU7*s0Dv#&&CR;j1 z9VlE4nvfMi{G+$@?jzeZK zUcDNkk6N~StNTcb(^7y`J5^$Fj^bMWoo$sDONQ`$p?C-HIlBd|)$a1}FRsJtj36SVf4j&|Rtx*`f0tbPGIN&Ck- zuAnzh3D^QyfHTZmfUBZgO0lT6+^@ipFI%svACNPU4{=HgFJnlovpRUkiAy&w zsnI|+eq|%;9kWD~pt6tw#~}G1?*@o2)GB4T931}u&-i3nrnaL}+lk%&uaof%l#_E7 zI7|XgBYLDqshAJqVt$nuk;#*f^*Ii9f zKf7;nC%$}QBUYxwcG|^&JOko77V5O9r_^R&Qn05ba0W(xvgxNZSlhRi10!nYd#Bpg zko2zbPq_v+FV_d0eP*8IxvbONn{aSEOkZu!K=kgRrS<(g!PBD+tGNkQ`{S{iI^98S zI7Fw^s;rMwJC5I{!1_!#(hBXcBG!n`G3wys=j=YyVzjeNp*J%lpl@2UxmfvLJouTh zr0DM#Qrc%Ais;d|p9ZC#<7b??En{7Iv#0J3g@f_A^Ur|!slqC*x*pcV7 zaja)I91CCn0LTpX5*zbp`$m5aLlku^wVvU*jCDSrSUbt|v6gM!xro1P#62}txGiUljA!dUlcYYjnSXW*Z5+-A_Ii|kQjiN{2Z{6*(?<2Ry(q0CIAE1Ur=sedLkAi@r(89p^~nRAc<0SjHPqU`^W&AYFE* zTk{^mLWdH;mmEO=AB5z z7^AM~bvId1uC%dxDNqaZjEK%k&B7#p%IJUoO>0i&3++*kCP!8Mh}Ph|TA^UZYlV~uW-wW+k)iisJ^pob zSmhpR!IbtvXss0Kg@+z+W@X(o)VV9I2(%xL@vT_YEzD`syOKc#Iqp2m>h=1qH|~@s z@xd$3Pu_OJDPtK;T^`cts?pTGyAQ5Ud48kN^qRF@PLN@M_b+_TX5DV4<{N_-Q`_q= zqUze6HQZRh{28Vj6m(kOZb3!!x^=p32TG$zzM_ZY$nQBt{iL$Ii)&X=0jT|^YZ(fF zfF8nTwIiYGT_>Y!5VZi+cYJo77Ozo%Pu8_4?odvW?zLD8cF~Ub&_h);&7_ z@#bb_*|C9MzUEeBV~qUIp#IHuO7$89t=q;qpE}iP-$_vbocYXi_HU{kHlz&USj|y7 zud6z(3%;^Q_=Q7Wsd40Zne}R0RZ2ByBNIDO)c*hrtQiCx%v^WvxpdLhRuQ%cgN#i* zKB~T=zxiS*sGzG6$LTzI;bnR;%VjesIFspPyc?WOZ3(_q*dR;dP`oyDVAv%KK<|^B z#R)8fCQt};Tu3ab^p@E!qls{`MVTdiIMGSqxRTnHp%9~qAYe}(1d^=S5EfZ4V`3>| zz@?rEpE3z1Qb}y)6$fZF5sH+VD@3hfz>tgD7jF|lBg7jtGYAI~N^g}XGsnE}6D32* zmPO`UZ_*JGI$K_);v9CAL4hxrg<4Wr%E>Gxi$U&?UP)r`PV>xQ%n}{AfM`-6+kqkx zmP5>j?;sfim7WANQn&3IR>+D~hG3XR2+X4LC>bn_390EjY`)i`3 zQob?VCp*68Rnh%6usKRGk&a3Fec~m9V>N;L_lL83Ls(t1-MeBB7$y*}Cjq{HNIuhY zIoq_$D3Geo!Bt5=07DAJV{x%J_S@WvB?_NBb0Zu{BAYt%mc&3icH#g+RCf5uB2KWkciK*yF$$kV9j05&le} z{J5Bl^Cc5j`+qzMY=6kF*d2%IRKs}sM<-~HxIm_EL%IV^qBCDlak2A zyP{J@Y}24t{)S;+q$^-;2p{cBUc^{jYcO883auw+A z(^$4v{@nK*lO3;7Yn&m;{{YD3%^gQd6=llo`y7wu5x-2Rq>f1MGjeWYzT%eGsCDHk zA|wu?vQcowakm4ojwOX~tUkt3*PmuM?aygUqPW!uP+^e3wg({3KBG}+-&J5K^PJ=K zo@_IF)CN9Q6+I*v3|Hzz*ONCARa)e_+T>vW066_lykiV8#t7s2%38IR-mI+Sxd)R5 z5~jm21#$Bots1iGRAeP&li0T7{?k7F3u;NOwc1$a-1joq>1!4Gl=1d5G`jM<5Wx7_ zGc~q!Jv}n|G|q`?va#gu#$uY^TDtZ&3lqjNKm5o40P5bB?d`QchXcG^>Q?^%b_)*~ z?Iq&L>DmSt5L<931o)a-o}R5o?(#PO02Y5Rl60Qq>97U4#&S8HvF)aoQs9+6$7tKO zWY?lrTsnLDO}B2`qXFOMEBK>!v21V3I2`zquDCsApCqZq0Pz7=gw zyJ%f)Xsr9VJ;cw^SgUuqI;$xF{P)N8kQE;EOP`Q(K|CLb`qN2psxqhx570~0Q@RUX zW)w0O-c)no!J!jbXjVs3=NaU8{bL@nB)Q2@ z2M58GR2Q-`E=qo3o=AmEIuzOfJ<0I}5}JFhxtwMDa~c(zR5sPHXSi-h(r0Qdw@L@1 zNf_|mzR|57y4(GZs#p(q%#YZXI$W%b03c)%49qAIy}yv-7?0}94b}xY+He$1svX_Y z+>8wIpLg<(F`IRC)2jDDTV6u~cB_1d>N&!kkO#*xN?NT(oYv9Ij{W0w2# z#L0!K)Ovvndxmk0{KD3Pm48guscES4|4-UDP|1BGY>z+K9fejQ_w9^1Cmq{F^OBDvbv+{vev8) zGEO`nzsyd$RBsyzX22lvkCQZ0M%1;IT!6UXDH;7h5;VOz+&h{ZYH$j+cb}$XGK~8# zq^`|Ar7bKm-AE(+PJ{ueq^)Y_VBRosf$@{@V3VTiHxpXtAbwB|59{R33c4>){65F7 z&;#X($FvjJ8H<#)xpcs_+fvPcq(EE;Daaom&k@ZMDYm^no$arA_WHnMf59EberB3} z$?Ekr`ip84$TbEYeh2>mm@R+o{;Nx}o9?p8&Q9bFw0G`E`*0(;HeSjeyoifuo}liYttayagVxYL1}w@ETkZ8h&fn0SpB%;M)C%;+&;i!I52!K*eBkylYEb%@ zPNMxYSyf0t86!Nm{^u{M(fT*0>j6Xm0BjNbGq=;&w&}vQ(Pn#m=i*`Yof>I&olR~= z;uTPh*!_G>W0%vOc>T41WOR#Gu&wr@HU}BZ?^2x*Wp7_9PBISye42|r5?Z@Avp28+ zIUsiWM^mfy0HaD!gkkQk&IGKp(>;B@ms;rQV!`VxlY)7~sOo(YSu7WZU@$oDXU!W^ z)oDO5)rRMHs17qLUcW_alXJQ{H@T6xV-NB=#q@hM8Xa?PT}i+^`|-@3PL(V`Mgl6} zF!vUpDz94AHJ**`LlJ|X)1v8Lw2e_p?69#TlP@a;#~RzVmQSpuTUGbJDxL#kF{~T^EP`=H#@fzZtjQD^~h`3uTwYuwq42kXR-R`3Xj;z>9*#o+NxU+ zD8}R4vB5Zx3#)Y>zhxikAdl$?zNPwbYSgB4k&}tf%6>Jg7t{3JPM=HZwxnF(VOa(+ zGZNP6$x`WG0?cPQ0juk7M# zpDv210^ytZag)X|HLB^lI(M~{Et?5DtmLtTEn6rFd9C?4JdS2hQ`7Yz#4TxVoD7T% zocx?}VSnh(&g(k$T2v<#4WsgJ#{kbjFuxz9O}}h(l(t!cs186mI`@VzE>LzZVq-*tB)i>8oHI+N|piT#TugKzO{M(Zsx6ml-Ju_a)b~+Mrt&9f# zMmkpF3+c52&N;8o=uh^aHoDrhh|gu4D|78%LjgMiRQ_y#xSM(oxp}BEA+#XF8lNiO z3FcDYvEx3Sa^4iG+Pg^~#l&BzblP>*a2IkPX(zFO(RC_y^@Cu`FJRx9a~eS7aaUVL zJA;Eaj(YUP!}Y^;PLudVt@QwAw|7?0wLu>zyy8zwx~svnQ%SAAHQ6M8FD8Aq!A7p8 zru7c3#@Al+js2zRnmsTHW|0j_Hmi*8PyVKRczOPwZcZ-TA6r`}w#TJeRd5?TmKfo` zUh@yvX8q$ZIh{vQ>FC`&S)kNC!on&!8216&o_QZL4LVIaOF#brWvTR!{{Y)TmY+im z_Z~d4yF%THkE%UF!+;^{l=v{8)J40lAP0p_L z-KR^+`^UuMItAX9n=Ax2drs>^QtEWhbqIC?#$*@hwyAaC&CNhYsO=btBGRN9lmG@J zzMyJAoH>0van@Ce!Mm6y1~qcw;9PM$j^$r4M8jAhu?4gOVUc*u8mT9U-AUR>z7Gskx@0cvfEhH?m zsUSkCW;#}=sgsD+X@DhCymN@kE(AhQU`xpDAh!|Pc!&oaOP7OZ%*qUjAWvpoNW2tF ziRtXbEg+d0m1D${#pQR(lLxv8ZPFzJW)O2Ej7Td81cvPt3kVkN4$vkD6D#Hr^DDMV zWH>RJ+kq7V9Zn>TD`ZPzF0Tk4YhiLF2$#&RWP>6~uI%y#K4s+(5KvWJi@0H!_9Mg| z0~5Wl-ZSP|eLzEiK+m@xPsH*9T-d=$#zOx9lQV9clVUAssmreX5(5Jam{r(OUd(c= zGqikt;%Js!SZ6KVl4Eq}mFr?$Zv$}T^(hL8!aQ;!<;=os0@;j5b{{ZR;o&fk5l{`+aTM^Zwog@hfD4OK(A|3ezJ3z;X#3{{ZSh4P*P5vmd5o4X02|nv0Y( z5=c?FlM0%4v52i1R1SScobGZu&O1veRR4lxZ|{l zg-oG^GKNVi*%)SQ56gq*dn0xV;Q7gvlKCnaeQ@adg@hJ*{t3vp} z@L-|{EA4I!8xjohVpx_rG8pkDk(`Wub1n<*!6Ytn24zziB$2@F^DDbBTU_9bUS%HkyHb zEL>xcq}%i>ELpkwM>?XZ>7i*E7?`+tyJO39eA`mrgi&fXU0Wk}aAh?5!Ki(<60Sx; zKg1yV+B#Z8kEobs-7N|mjK*QO-O9&G)Q2SToO8?#bdcK)Ta4$tLr<}_BjA<9x(-`*4axHa7Z2S4|w0=Y1CuW ze6sf#?H@yD^!HM<0!ju0gj4=tbPYb3BR};zumg-XMguQ6`be~2e&brIf-nhh6MJJ$e(-H| z*fQjhA!*c_*A56jB4`emsI4B~Nr)$$9{&IaE34A`0M6F?W@=c*tn97&gym~ecMueh z5^w4HgjKWxM`I`{QK;FBa9D2RKcwH&-A&Z(EsP&&{{U2N*P%Ckv2;&su|=ysNdvTH zwShD0DZ&0BFJh*e=5Pka?j&7hEa|j{4eDSK>AY+yyf*~nw5pf2wHmti;J9J%4?nC%nv|=WtYWTnlim{kiK%Nw;Hz0V z3xk+t(YBBi?c=nkLe18AFQl;KF-PTaGvB;(Ox9SqaB-Xg=2uZz^#ZTlpQI&aHh`HY z9N=^MO5K?>i}Ze;qb+G#bCLk<^TglLDp0TKAYQ~U^YJj+mn>P%8Ti1-z#b!2>s+Cx zz5*{JJ?3uOl18qlQ&m(1HWRZX{RGC*+AUVgW%5b+Bj#Omr~%*SIOp9K!_}y4%lx!k)w zrJ+*d!vHev4UCgIZ9NtGWo=ae_|7H;PL^AgU>F=OW~~-mT4vf>(4z#8cN5Q+R)hZl z^pwgyY6%B$930{&Z7Zg<2Otb%@`L<`#H)3R+RPx>G2mlAq!Zs}tN~ImE68F!vp!VD z-%sn>HBgEbP!DSg`yZ%>)%9v0Z>hS(&I9L<`Tb`nUY6v$)C!CS3167{Xie(D?XV1c zYbQAW05dx+dUe)4;Z<3(2{`aaeE$GRxo(f~H5%{LS6eU}pCqvu>}Nlvpe>A+lyxcy zDtX{{`p25bR^O)~{2qD7#AVVuul=9WY7o|{3zQjZz+czv-^Arkm2|GbD;)bX)#GCqR z>$I%sgrXcPHy@@yOvc-#>U1?0x`w6EPb6*!IRNLxzL~3~3bR*h9!}xgGp5hD<9d%s zWEJ{Wl(QtDC{=Nuc|E3h)a&W7scfOov74QPpFb15)oCi#FQ&j)nLfaI#O8Xwgi&F3 ztv?PihhASlIQaTbxaHHHcx$#AYc&9~?iO)@V8vdyo7=vKY#K0!VN##byGa0aC1jyd{% zu>SyybhT8K*?<5Z1kO`)rm?!Op>#cEmg!w?+8KERx9K#OPS@X)&U<~PC+y3MD6`aJ z%YmPy>DyY(QS_d*7~6#$%;D0{wuer$4rmoBfH}mbwxn0GhI}V|&|Nj?t+$fS-GI*- zhU*TtzSIEaa$CgX%^Tx4sMhKzt(!u{nDWZUn1y|JRZZdbMN_!RT!Hl;S&^sfhb&UU z_!&6yG=tAU2=Om3)vNT#(RBu>BDV~164ojWs(IV;X4m6=1r{+jQ~=1y+@D{Ze`uz& zq;&qFa@}5~tFcKWqVdntSN@x&5nurM;v{oO&^GB6>BZK>pl40yq0OU-Adgxo*hSGAobPN zzP)0C6#j3dz1=nH71-`Ku=gw|>H6JHnbcHDu;5JrQp(1{ z=5-6yKM$rsP~`cAQ1C5Ylnte9Q|HWortP`Nayv{f`i+*Ipz|N-*8cgG&$!N6&Bt6l zwlx|hW+gjBt*lqU8JZ1g18Vn!Z8esx@?UPzk!cOa0E=uHnm1_O2_!&zKS`1jAmaiy ziGVnjO1Tl0TnMBp;7Kyrm4QTtl1?QU%%`_vRi041scgkWyYUrvg=3f^AVQW(`Q}_# zlNd_)<}g%OUVKBXVp$ge2tWXc+M#hgc$Ov!jEYM^v++F8W=ZCA23|`JB{rFqAUTCW z1dA9&nomlxh1&&U@gT10Qe{cRuIOyM#t$+{lqZ-GQ1cG4bc`oxh2>cBEe7T*VYz{p zbIcZbf?Nq9WmLi|aS*8!-JuK|j@}K4b$rTVdlCYR2=gQ{o&+)lBa!AtHpwr#c!n#G zry1{#B_*2Yd~bGaa58%d9if=vLHQyvr>J^lxKO9uNo;ZB@8(jQ$ZRqzXN-}^5UBK9 zRz_0Z6KGR|@y=&FoiKwYkcx9v%fGUdz~{W(r%9*L(^xJ0@8Wadp7@JtRM3m+VB7&< zxa>U4-lePcHEpIj3P5f=ne+buqni4Ue+LZrOlx&D-FTv!0bbb8=K7c$D}04^C^3+u z$nquh!(!N2fpwkS4$!yIE3eeFGO!-pH$PK6@@jckrrZ9pYV_!+LRqzciJ0;~-UV$2 z%vN1g;1CzT{P~Mt)srfSH?wd~BG>f|O(zX#aC<8#kIneMjktawFexo z(g6Pea}(9oW|L4|0V)3g^ERF%PPMZOPT=RpGJp31Q`*%cHcu>Ak)McVW%RM!x~jd$ zP=(a{kGF5q2HJ`uvcDsYXEM4?MaNR0s9yn;dH(=-&FB{0p;_+6_GWMJKGV~~mgjFj zHX6Ma^xEXyi|!rOLC@Y()3Fb^3D47saZ}{l3xE&i23sKpJ!WFVa0{M3`JTCx$>J?5 zU23+JQv6{_;14W++FHx()#cdX2Vyw;_LEJh>=wmEKnY`x{C_D>U7EA+sQFDZvNpF6f8@Si2aXYG*hfF`93xyac2$0RHp7KGRan zdV=F^-3>I)Uf0?PJY<{!{Nw<%VST%?o!!Ep^Cj+U3pw4F_Kc1cK^9V@oHxN3AI@r9 zS+Yg$KoB@3ox6-;2$DU!aD2;;RG}MNJ;C^bC!XZ|=jLu#JfdW9dypr;=Xn{LVPK{4 zx4+kkR;|MkfydTdl4Vs{fn$=Q8$K!oW+MY5&LmpppYq@Vfyaqzi<=lB!6Ubm zB`LDexPqz<2{PW6l%z8fK>z_J=WKw?NgqBTUA-hFNhdx}BOgeqQpY2a^p@JQPca5m zNnx`al!AU_eo_es2a%Zsxde;=1feqdR$Ksco-rcC2%~ZLd`T)LgsB)f%#kFS;5_(} ztfQ06HEHWFw;+5;DyMl0qb6%rq;sJ@ra0zXr*!^9k$^dr>RS4#qS}(op4i7}%R2ON zV|Mw+?FO8FJ&SS&iObtsCZ&n^IGs0JxYX?Sa!ll(Qf<_G223#Co{Xq|+Ig`<9{~21 zI<$0jIRKr=3G99#^#z=h;vs@t81p?CV=V3F&plI*sBOlkPjVmP9sZNUSzu9H@h#{(0x^&zHA)pk zcZk+U-&=3tmGVN9$IdbRXSBUd#y5~vN4q>8_{>dcB@3|_9m#-oVlX=ynT|K6yt19T zLMheL?P5ssr;!b$vGn{7apyCZR}rTP;~DcOQKzoFvU?bJz~W=mds+HP)H1LZe?|*ZZh4AMuSaLCAI;UINaTaA&#?7 zr0vJIvVq+E{bqA(FhX(%`j7iS>91j^Urw?ZoR)E)kW_`eH(6fWEd|01Z5TX7t^ls> zC}6qE0W|Ko*;!^=cAn=P7@G~(4^L{A7(8Piq-7Qt_3TVDbDzpQm@Peejb5QjSFybD z#M#r)>C}~2h)6!x`1}6=Sn7IcH&l!1s45SZ@heuD8YmsHngI6WjEd zc4+l_TEM}vGl0V*`N|E>zU~U}RFjkT%;#I%q9Uhc6;=WQNg&{J<|_ABr*c-rXJM1I zw`1=STbKZ~stkC*=lR7o7TVlc09>3f;!4v4R%_L-sO_|$2h828T8!5^f4-lt_}(@+?13^Ot5uWsF|({&mRS5zFIkOx0FoVLHFCW&oIwE+mm z#l}k?zxW;Imrm5Crr%K5cPPmp5dBL}_{x1n%K$LkkU&3a(+vAwoK;m>T@7!!t~RiB z=PU7_zxbBDk?HjH3PA&$d6(4a^%&b)Y8OhUZ@#_Mwi%(p_LP~MuIfcsUs+CC=cT{+w~$+aytyHtfJM!><&e&@_xA1tAU?Z?b}S)si& zaymAi zi}VUA0axH2@!d9r(8a+N)Ob1WXE|oJ-HT{6-lqbVBRC73M|!S`sufh01decfM|yZV zT7mTez&Ivh_1#m~X?&7?2Z_TwsManr+f{cpWB@*qj-S-2VhXvg?<&WG7_*=%)NG-_ z1Gop7&h;%V`fwdraR)zHj}^DjwEE$z)0!^ESYR;tnH@t#)N~BiijCS;?&x{M!swlJ z(!nnLjmIOAJ8cfSyI0!G=tc??kVG|M-rTv=4Psa~>^SaMkJejBuGHx*Rz}?UPs)^rMnz{ zkFNT5U2dY=9cclGOasZl{U+{G1mldWqsDp+h_zS2SLehQZ$MgfhXMK0FM(V zuXOYmuB6!NjJYLwI35q*iE`C8oAFqB-3HCA$T;&JxmR7&fl9+!w;YZ?S-w2{bJL!9 z%O5rBhTS(vy0=%|W0OkmCu?^EW9ObCnl{u~qj%PUN~-QgW!lVVQ`S0m%l$?560zEq zBL#r)PwUT#&EKfg>Mskby1QE6Z|<10*MZoX_PF!heBANXWUac3Zr&Ug`ps=8S=Q6K zwkwQrGVIcoR=R_Ih&c|m_8(bY8MIyv+m#1>!xPai+-ps>OR3MQ*0n~PR!Qu8M>O`edg{h=p7DKnYqWIQgDZh33#FBlIo9L3qzr<2$js5H4h{s( z(&}m&D$5+4M*jfn3F-Sg&n8~piRn#MMY0UddUiK6L-D(~amUOBz_RWGB4m{c1i0c> zS`{G5875T7s>TK1#3hU(uFy#>BX(fPk_-y+F_R-Q!1EDSMHz=t1f~>437G}u34&&l z6`ogiV7E&H1eq9V8o|kcFe@qM5ljk*MI1y4ndxD-Z=A~o z2j(e2CBl!1aGu@6wD1y2fr&qSNyM2XJ$M_HgD=y>uH4Ix##-Lc+@w?A!u_@&09W+jpF)VsM~+B+%vT1(O-!dBo$Jwhe1pGR~)0RhfW#Q6FD0Kads$3MZSl~+|>rRrG^ z^!;X5zgjJERV#tmf$=M@)|Ao;l)ebyXRv|k=~_Cypp%^E91pzbyKcUnW-}S(P1V!M zuYS}(i{sDSz#SDR%)qw%5)M1!KkF9L?#OV5Aqel!oJu{4KnAM7&ORqxc;$1;lE!7U z1>6-khCiGxKa^R{qfPt3&eV?U-yd8}mPPeIbt*?-M-c_C!p^3)Vx0SY31yC}SY_oq zTS2Y$N$e{eh4JSGd$e9F<~t$@@dCbpk_B3-?j=`oUdV?oGQ`4oa}jKG4m**m_ks+%Un&Ir~nW zd0AXJG3hM9Qq4I)N6HQd>lVL4;=rn;h5!{_1~J+eiXOr%+Uk6{{*y|iN-G;qY(4%K zcD&v(%5Zry#+lNxJw4tq2atcsj_LYyf*RJrE1jpD$g^Bj>_df224w)`pNNa=Z6uP+ zz~^Y)-yhm?%ld8f>yPtE?AquW?3TM~j>v*HUrR+I1KuA8ujvoo7*IDm#<`oa4XT%N~5C-BwT_OUSz7V{Jbth>ZuJ?0oux?5aXU=+dU^80$Jk<~7(8vy=Fbw_8ojIlIV1w) z0sjEF#Z>AazV%Oyz5f76!f4rB_k}6|WrKiE^5$#gW8>LfDpp3~g_wOZ40};sL(*HV z_6&aP9A*yd@QqCj%n9YN1_#nBR+N`#wOkx~a(}6u{J!mV#}2h1h?+9zHWufAzA^Lj z2W6D80=gcuGD$p_&!5x$$SYNO76Q%VXy^IK6*Zpa53?8m_7eFg>G<;e0fj1som#~n z)yD(xDPje3g4M{O4vh` zqzZt~Dl)|V;$^q6T$050B5B>FTH9ez&IdR=5BGpty%pT5eMSd7c3=I?PD%8|-DwZG)^H-+y%dW9laay0}1EbQm zrsNwy8Sp07r(Se5jXf)q*y3j_wHxE(-7%@uSEbUjD`oM4iNu=^<1VuQ0LV#?U8{X3 zP&R5oIUTdwa@||2`uDO@FbVNE?{8)^#gY3*c-3kdDqQ2@WAy%9{8Q{%U+yrK3>T!b}0iJvGE~>F}sL%u%flrN{}7c4uiiqntCliq{iDxP0Cd<=RN-bq5iWO>8vhIFHW0K;dd(z`Nw#x z2cdx7y?M%>;7MBSDy4I|y`!Jy+BP~n`Xnln$MW;eerG3UVPg7~TFY%Y#ym(XW#9JB z;&KZPa3pmk{4S-DFn+@a^^f$rg*2-_C6Km$MAz2Y-+E^@g8JMehUD?*iFQl$H+n7` z40t%tGiuv5+q-BeKq?PnVzxNHSye=mLlSuULuO)JRdtl^Ye=m20Adj|!=`Fc>;b_C z^fOWHMW)Jj1tjgjJ^l<0tSq3`$ASF;7=f@B*H^hX0E3OEjE^zB6?OVvqX1ZtM?W(^ zrs-*|8t2JjpK%HSKYs|xlxVcR&#nz!qo%0Tz+VU-vMF-+}7 zg1W3enHzri0%edZs|+wn@AR5Bix!t-yCm_S6F$_#U@jXO3!H*Jk(5`o*PS)1X(t~O zLY+&nR>2^i5+_Xz$1Q+-E_uZ0I%b-n=XeC;EJul)nv~~Dc1yjZWo$A}z@1i~TU%3V z=L`P;*CD@2$bKPxHrOzo%x6!oHK{9tpz)tGIQwXAea~%|dVAGz!e=1-;N#=_PH4ZO zs%u`06|hKs!`wXM?G0nKS2dy8v&dzzVtT-AMZXq+cm~3_&yWYi^v0r$)#^@F!nhnW z6M`UaICAZsrv#W`=}~2&TD?Q5_JlcAKVP5i_mM)I3f4i(c|C}WNubkpY=@L#iI>zm z8L1ljoY~LiC&Xdg8`)@J4sulZ=4w%@TS2YGY*&Ii;#<=;Y@bc)FQTn$Enz*tUlXj* z_5DV(QB4|h&Krl^Is48NswS^Ra<~N4-T~tu6MvxVG}>BhzfV~V5M|ip^D&(3Y&*SA zP_BzBK){jT?KytA)s6a=#x{=;El$1fxgEWuwn^_ZHE(9)V*0b zSw=RuV-jsz>J-(hpqBXpFr7l?8@B?#BgA2^r%_l@fa8#H-gI+wI{x)ebo3WV!38w@YuVqctAoe}-;!o}`RZ zw(xuWiO!v0rNL$y;I4RpTa7GUrraC04UH#NoVQbQ$261XJGgqTy_(HFuEIgKx(8(JN#a8F zpHa5jL2PZV;b#guO2kJbll_0xU|`lz@awgMj=x$`g6uJrYHY2SCXuup9N0KppS{++tE&8fR+ z1P(^;krQ@m=uJzu;eoublh}W0n>#;6ziKq}QnlT6mwEwD6b@I9&Gp3K`j<_(Et-ug zi-n35IA9nr$ruD5top*8F1tyx%F&m+`R7>FUDCRnY3xlsbvENu1qakpMpXF+865o2 zqx{^K^zn1!uH3CZi_$Wy>I*0*HH&?t=wlj&sqG9)g(ptk!(4N6{AY=QUE|aVol|7kYtf71f0s2m181ERcykk%nAZbVON0> zQlcDj2CFZeNhGvd*U2!B*owPzBC8n#yrDeF1_(eiJ+@h8qN=LA!zh&lm_?9W4kaJ7 zzHtX(;wY89;@IXhZzRCum=~2{0Fa^(2`7M;0zebNhnxZOViypMfPhmx0i68DH2oh_ zW}ESPma9Xe^QhIoroX0nWA+m}Q~OiVbax#mM9^E-_5HnT)TJQ&iufdd_8>=Jqko1d z(rPt1^eKA)8DgV_`<}z~gLEkDjf|4Vr4d{euCI z+lD)-;qXGgN{CtnlGhUQ4!pM z-yOe{&Al~@>;k@m0#%0YR1@A>()4N9(@;FI<8pim!qBXDJ$HK#AdbiRLLCsU!DBO6 z0`BEV$KG+dM{IGR3ZBgcmoVq!{LIC;Y2Dvw$rvlUAO2?JqpCHG{-JE`HTJ~C)>Ovt za~D&?5zpFu?reM9eJ|(Btx{IHSix<4HzY7Q<2-^PYH}2e=ef_BoE666;D%F-{Epx4 z9noD_ASncaf=`k3_?%tT$~PkHpS5TS6t)sT+v_af6I^{{Zte60c8o zEGppRA+kT0hzs=B<9*h8pBu17f0VMD?i@1JDik(IaKw>@4WH8yojF>YV*0FdHxe)~ zJH@Nh-fk;Zdl+y{dFSsmsZUCcnt=Bf$i{Pz`SUq?b+%MnYHlvfHo4?Kk?=9vJdBlq z&BbAIPJc0(=9*deI3ZV&*q^b2)C+Fiw$p-g0bG+cE&3Xl+f_9nQggQ;WNkgBalN%w zwPdoceeLv@(}SebSlWt)<`}?k-Tweg#I)Nl)UZ2|S0Je8yw10KJxS4;)hHGL?@R% zRirGt4Vm0{$rxYfDao%xEA)Dsu8}=QO--A1|u|SD51C6 zwcK_K*h{&!vi1eo+%^wjCjMPHd~mm?O%<@O0N24_y`sxo*?t4DUz}$@tYMG2s;`wR zmTwW14YX7jDhIiVobM&Ixs1jXT`g&6@A4FWRmcR7tTPQiq$}Dzyo~XWq#;V^Pg>2M ztDkQENFQmRb;xy$Jtixh06)Au>Bk+M6;jB=XMQn&*Z}=vu9c_*c`gopq~j1*y_MP% za^#Ga&Px5_;?{dlD&P5M9LvtN;L$eMUYlrl!S`-(Fh1WiIBRUO<~ehY%m;X2s8lVN z0;}84KLQ2VZDkoCjsQH(OsKLW*4Tr#3C35sh8C9~x|6(Z8;&yo6@X?4g^B!*I{^j; zJ*rvyb%-(RzPpEUMJ?P!=5R&w12qwUzXM5i2fn zjOPPCS&F*7Mzc}sEtRb`kWGWM&CBt*j{Tc*+I<+)>Cr*PdGR@m>9Ff7vd2E+cm{NK z^$j>?*chqvj^ZQew@|gLq$*V1$Q^{SYPs5|)ao^C#Wt=;1onnwPJ)K0Ida3?N0TS2 zI?}~Z=L4LXiY&3Jw|+3Zg2U!?j8t2TPqYqumVvuVS<@r(2^{_7Nz?7Web?`X24&d{{Y-BT?vn4=(;$d6LN+*&O9t7 z$x}OSWmGG)eW5iisJvNd7_Yw8m(pofH1)lym%v~M-)6qao^%= zt-EyosAXEz6P`)>%**v*e{>ac+~kb$^C|d{X2VYd8yeBx&6H1B1W1uasZRQW%Dfj1blw!V(vQ%UJrP$L7r zFn?K}s-#n>TPd)Qc33gb5PCaKpWFF{Fxcb?ZBOs&e@q6!!By@ie$UePTtkvi8037y z)Js>iGW)qBJj~k#)>KrS?c}y{N5ITI9<5Z7a8QJK;KfQUnysu9F!V%olXAHe8I0{BAft3GxrbcTJ-dA!6A)jG0Y@nu`8V zVVbsbcY8w-wNJ!$6l*qu6aLbV^y4NaVw*yePjW~C7RtgSFgR@E8-x`~#=>`;lMVf` zkS(MyQnG?E{0F!bSEg3tvDu$E+I)|Q*Wl3UP$_N1XFc)sh*xo`*$4~B=UQv%u zg^_I26X~*T=O$xcY8xyWP^^8q@*{NXPNIQP2kL%dG@1spF2isn@=kkBQs}cOsr2l? zXLAoHw13rGs4B!E100?N_M=P~6#y1dj1e*t??)l`kMxEa_WuBxuOB70#*;#}>DwxI zDuOV6^16S-Qs%b=DEzFWf%C*aM!9I;zXWa0W}Es?SqgAK+yUc1GoPlSD_ZKLmMYdb z+6f)Ld&1Y#v|ufkFyN8_Kh(^zYgd1&=-9?91CR_j)dwJu4m=Us8#*&?mX)L0fD9JI z1$Vl+eY+J0CmqaYonD-s_Acbgj>wr5Ja>A`4xsT|Sgn zIrTUs9zgjYF>P+Xnzq!Na(m~*K>@8Pg0m*;_+(mg<=uiKMR5Gmw6; z`l`JL7pf@v81Vy8dRcH)oM6}oB>kcKp1D~A3d0U^2OrjT({W0XSZ{ka2R;6gwbZR+ zV!7Hrdzq1K!n>6MG4_lR&IBbwtNhuKz}wndJ8Js1+T4W%@tk>qK(*@`hHyQt#0t~^ zEG*<_hdAvRuTkr+{l^rX_r^>#kLm5JQKl(-9#41N@`I{Xs?9@V#9oB$bJ5xv% zn$AZdmULI#*L42?R=>=i;2-X>>OmuCh zI;&e%O`pEq?Las=9DP2&F^b6A_1Ngb;m+0Dwa1b_NRLK#8awJ!mD!QF^50*)k?7H{ ztuAr5I43YIUY4y!qRQJI;7G?~IltO&nO*vpTXiXX4J}PeEsJaB&qYZnKHc#- zy=#iCdTOw`DIhj6nYnq@rk2Okxpfw0?X8NG7{&q=?H<2SZ6EhUZ(huUGaTKBIt*H<4iGOb%% zcmtKpXIpDBfx!Dr-r76U-n+>=Fi)Akpwg9n6mS;1i(CbTKOy=Ei(_f`rpn)^C zwA#7IT!@RYnk56*n2o3bGG>=BOxcEUftbEz9x!YnWk{0Qb`qdNFqFikNkPmyh!q^e zq*laHEF=yDoZ?AoR*||%r}3`l5C z5KLr&vAe}w^ABO<#hB#6l|0KVj7bthDeVdj0>Ug+6nB>ri!LG)2WM$>1m}q&)(K2P zyrQXRR8U!%;Z;USKLATf;2{xNm?9}=V}Pm&B>Vs-#*g-D`zvep)(tHc^JYlOkNk=D zr|NU`nO#fv{{W)&Rc$Zq?wo721e7{~yVED&zvr{;yH+xOvv1>CpV{V3J(|9$s;j5z zEBxxXSlp*Pw0R6HR60Lg$q_?gVxPhS0{ z)Fz{<>g`lh`?}NxHIMv2_S8Py{bs(0rD?i-TPdv+c>s6&p6B(KmcP`@cmBle({+#9 zJxAjD?zQm8BD&n<3E@<*BT zd4Hz;K8F|bv@h0EPh=+p>T~zZTh?1titQi~&IfTZFV+6}{W^`7fJWwGKctN|px0?G z;MPcOx$I|}a%H{k>&t>`(_q(Sb`YFl2?_^^nbhg6wEIP8dHmu;`F~HDpQB4{Ur!~AA7}?4e9h&xbmV%BJwiR8e|dRdw>+_*OVOhl57}3{lg~dN z5HD_$oTXJ30CVvc)$88E61=Ma0Hk?yn{W^3s>h_;-N8E}5Tr_yg5-)!ZNT{kUcyn1IF zJDhQkpu|PoF${LF&MGv@LWt(5ev+B%a(#WvszklW0EE{oyt(rovf*W8A87`j~vQ{Bc`QR8(%r$1S^p ze$jnJ+mqg{&unp+XBE>?XUTRP0@*)#m3XrIij@R`?NR1(xmxRnFdM37>xl#^+Uhgg2ONJWxEnzrlky1fGpB8~Db(s3u(Snody(ctHTW#f zJ-=SlB;8$VF}PXuVSylDcW}x<@J42CCP>-Sx2PK1ocdtow}!`InW6ZkdL7el(Zb}; zU|Fbdb*s5YY@7iuEPGv0y=$M~2ZSGZ%c#a!%UV{OExb5vs*Gfg^5}>aGWm1M1@3;4 zdRnjs((h(JHV9Q8Kz#jT_gePa60x@@yw=5o+I{&zTRoHG=wMCSgg8Y|o<=3qEjSz; zf-s}M+{=tI5Tt~taNuJn=4xYWIb|S1=R&5;F7Clb1R^l}#WuD;4Bdu*NJgT!1T~Bf zPX&DaA%$YwDy)TkjEdLBC#McQJlW?%^sQ@f6^oP49RC0)$)dCy?^frYdz?n<2%2|% zf_t)q+GOely0$l*gQ#uSmBSih?c*qx2gNJ_MQu#(-T>$ zTJ=0!3UV+($@+Gg?XRZoPqeMp074@j%DD1IIhfV1y+*@pwHDV{sR*bN>spKvj`{jc zu9anl((74qlEnu&nEGu6KvhLmn{vDITOeo8GiUyq^fA9al_!m!Tb};_K`m%ovi(Cw zy2boRu&~J>G2u_tMf4py)1+arl*@Z>ze-`GU6nZMJGJ%OyN)JAXC@ zVf@6Jk&V_Ks+s*O0cAL6X3z4NDb_U<(#pDr!6z%XjD04DQKwedEjvI~;DWsUNr*O! z@j)qF+tiZHo&Xc3GFEb9YOl9dZpsy#Ja_J9o|j2wbq{vtJ~m_mBi%cvRJz1m?Esfy z_7d8lR~4%ez7!7s0BP5APF?IfcWLzPP5VZ9_Y!}c%WEq(8mPd|Kmg`HuUjewEU*_& zHx5BQL7CQ4Yqr3oI|kdDl;gK}!AM!5b}n)T20;0gRI;j#s63DW813B6dUQoXwU{r1 zv~YPoPy0sg($zwW*yYe?k>vB4)>kV#bK_5C@9a>yjSP@<;=;km~d$K~NL$qnwx>RT`VYML?>1kaAD^eWO*X zZx~ScNYvF*vmaWfL2&yWio;193XbX^sKATP)O{V_9@)MV8a3sKU?8Lz?eCp*;Z z{5Md*T$blJNTte~{aW#i<0rA1 zl{0Nzb~!A5W%UocN`;4G?m6rxCmU@usnhM)!9V(g%u)J|uYW8Hl~3{8iQh31~;cU9x^^h)5I;BGirY8WwbXWqkoR$^_f+E z^9-$r19seG%^f%>uKRpo4E8XyP0q>nDwQ=EyHe#F+NcHq`<#D?Z4FwIO4C&&H*yYf z`sO#OWw%O2^@l;#0_?}Oeg zX1d!o4eDm9z~^xO&{k@zM+5`u_vg>nWbEAzTH84#h&aH&67B%@tCh!cPk4&l+r6NL zx1Ko6wYpTdYl3@_%6o`uZB0cd)>`mcUywkNrexCU(b+>NJ)eMn47GZDO1dkZpn;Yg z0r3ZDRj1wVZ2NtjefgG{v9gP0w1NiWe0)MZe!X9Zx4_3D4eUB=(b0of+|3rfEI?fY zDeXBqIrUf2TB=DIJRh08qtvUQ77VHGBD$I@)YezNJAB0YZH-0VfyXC;VM|wE)9QLl z*X|wi`d6@lQONG!R1ajkMk3{ zx6^Cx6r8SkALc33Z7hKA!;bOYOW97PSmSMrfI;&zM$}bSvqzg41K@$4W^S3i9FPX& z_MCYWLc7?e>c$6^B4yudQPi@-i#z?{VU^MJt#PT>2Q@zg8MD+h^jXuJRi!NIxyp`8 zgX11&Bd1#{)NKda!*KYWH%rwQQl|>+o~3{SP=ZPS08u7dJ9J$o(@mm(P?8p&82Aw`q-|sj zq_zOvo_7K>oc{nZEcPy&OR1rVWH_sDgU|f_amvwKqWWN~X1uO@9DKs4FHOB)PhA?u zeV>&!>aQk^@UMndi5VB{YkqaMR!y5IVeJ%Ew;bRN)8Cy+c$9dWZMx%Vc(D#73{!2YwW>QrfUw%TvosI(|` zTE8#{{{RvH05LgDOKIAsqfoZdZASq?-;9y{d-wWJug8|-=kiX-ddpWi<_+w9Lk#x# zmGlv=ZKRY3f??XHTx}DpnB*;VsI2>snAV!RF115*i5gl#BY4lm*V5phO;-~+T{Vo` zN2X}2R~9}nCqY+Gr%rIZiO*G4r`ic05|&?OCuY)Tn=5|aoMe-yMbwRYtv{-r=2njF z#G`i*nk${qmm5(0`HC|l>d9Xd4K@XSqCr(-Q)V;fQjJa<8RA&7wfQqGY13$RTdTty z!dirriIt*naKW1<-`#OH7|Ic_afkxAjK4g<E0fHP)ZsZIco&W=rBq1rT+l&1NM9NbheM8 z)Oxp3Y^WM$J2UXs<8%FxD$PuO>o+F8i}r5&Rj$@q_*$#{<0Ep9{E0hL^*Q=Xex>_; z&^ik?i}rg*{{U6jkW{s7eJ}q2bN*)^LA7Tmuhf3f{?}To*7b!n;O4%aq)@1za6PN{ zJCE9G=sFb|TEZI96U}Ml{ITxethYXvuiGzFzqDVq_N#S`VQo5oO&wTDD*Xl*Kg|7R z_fP1WE}nyGE7|^{KcM=31|y&V08kVTzzH)qzioQ-u{>jpeIU%RQpW=s`kYT~QMfTA zly!XDg`IGjz^?B?z|> zyq-`n8TKfsF1P<3CAiU)#5EdkOrwM#t+(2>Y}DE>Sg-_w;1|?A6*iu^%>U*$1VuZ$dZ&L z&I`Wu@(T7by|a%@Te@{tUAm?O9!jwyKc&zIHF|`A+!iM!;%qLR8iDDm)}x#%{{VTN zs>KVIx3}dPobA)LHqkD=trZ@XLj|~5mfc5v$8Wfv6yvb@ncYLEhG0kV<|?|7KPsG+ z#t$EP&i3nuI`-1HR>AV@K8n6#mg{ztw+*gu58Sz8-&p+PQnAvUon?OVY zvT@%fqz;K|9n{-mll(iN`fjUjL-%99aAwcqYxK7>FbsQvlaDi6j9t%6YS*c0tYr4HkDjuz#p409*?mm%+yGY7et8^Xr4(Igk5o+}Ij09CV&)#Ou`ZFngt*F*- zR`gl6`6HZuvb0`KcPnkjEJic>OL_#R!oUIB?H6To@9jGmlg@r|4;vShr!?B@Hw*V4 zIlv#{EcWFYa!6CoIObCAd2+(4&IkZ)=ZGTMU~X@3eW3H19WGt$U=(fo=fB*_KAnK! z*V;LJc!W@F&!&JXjPi4d=e1Det6KTVIsDU1+FVc&?>QqpgLn|9! zb*@!0j0HFY=?Pe;0CHD3!5A5rj@OpGms{*&s#d}Q#sTglD|^>qViYdl;T^w8WlDWN zcnVvQft-G@wR%+K`XN9b)_JUW?PD%T@pdXz6#C3Nk-!oFEW>bR#s=KyKT`?n22HyF z+;}A9aXj^Azink-9C5}xb35b1KHM_q{6^a>7!8o1mLKU7`;>yC{^2=RQGiFE@he>z ztw2@;@^F7J{{VtA*`<_Rr7|}x-S+Y*3cjOm2f^nbK{RTu1r}MzY;l3W zF$&7=SFwjb!ZVOQk>yvjHqls0*F3r4{*zKOeV4zeRtVw%Hl0B?fkFNv7=en1z>muJ_T{@iaZqB~|M^(9G2U;qnT&t@C?)lmW zU@_eH=6j^{%{%L>wQGF6ZM$jb>oVx8R5dQzyHMnUcD6wivFPJ;cezM)or`v=KbLt-?4!{YXdfocHD!R}a6;5z7v=01D z-Cm}xR)Gp^sZ|I8hDOuhiOOB5MWWNDeVYp-B92=gVBa##8}?_!&LS z$JBK)TmV7_alq$~_k@hv9i1UB)Eh2xNdS6l#~)a(ruuY*)|6p8U>;BV!&%kuyHIIe zIRVZU0ObA5@zOA@?{0t456PV)7NbL?vs}nfttt))E1o~ZwfYoh^wF%b&wZ!3{{Rv% z*Zv<%RaIpOO`zkyF)KQql~?U2EPc#zoXtVAH|Pa*v)FJyg9ZadqCT~DxK*m7l203t zfHM`RsJT1YhVVV0^MU^Gho~D6mM+RU&x8K*8#%7l@XX3cQT!)^^qEIib{j0gY-f|e z`k%MNjYXDf89loY{UlmQq^MF(bD0gQYx)1*1L+aI+ zYdVtOVJ(8oO43gyn9Kg~v$yd9-UiDGqar5k_#6lzNK=d{KN;is$r`f&yNtFzxQH99 z4a<^87(OBW4c0RNcAmlcIFQ;^MHy08=q0pXyRP)vkUOjXQQr_JPpY?WScA`Fi17W{ zTE_0g2i1}z)vh-=1vHQz1J`AWM+^SpCJRFWQ@*#^GMi3G54;hwS zF=ydh_3$PX9~=@Cq2F#nuF;=kCOHL%6PViD8!Q}y$W!q< z+q8>iK5%;xIPGNzPpD-?&pU%>yz9qx(JNTV1ON;7$?f%zy-M^Az-=dK80|AR=WU9( zQ}xd<<5m$Ak%C9haXTHtx}87aZ7PlcB;*f?p{A{N){GgBJMkC2>%GBH#J2~|Cyk9& z#0AIzWpgTHQEJhFf`O0DB6^OWRhyc%*ht3{$*E&YZDEY^36o>kD_k*UX61)GOSo)= z^mQ_7KxP34<|pEAQ(`M3Iv!6XOtDEJnJXL)3_hOcPfvIR5r91A6ywqUtFiRT!Np~V z#yK74HFkwN$-&119}{a+Q)(#zIAQL=`1?dxrtj6Dz!f0c*z#o+LiHfaw$%;bmBvJ4 z@Y-8b)~)zqgOR|1w@FTyQq}HK7-Y74$~|qTO6?d6xTqiI6+>gE-|H)&HY%>%1CS=B zi>Gf^tk|N*;1bx{B04Qak5V8Epd9{_qpd>H0IjO}&f5yrJEOBvCepUR`K`k9-n)Ud#7D%j&HszJxVg#wneALSOft@>sxFx#?Aa=-eQs>e=7MPy)tKQQoRvR|v&VrlCNa+`_9Jjc3b zr52hMk1do3DB2Rsnoh1ibs!(_a1!A8dFMvtZ;06kSfM7e)HW{=d-FWqRd>J1}7ev z2lW2{n4UF?&|?fsu*mWUnf18jk>Jm|r{U?hEQ`0!+~*Si0JyD}1Z13jqLf}myLlt{ zvGEA?RjROhD9BHoaW`|3j~iNgZKduX>pMyHp z)#?!B1##vJSZUUcwpD_B;%0LGuaEJCMT9WHn|gOIUokf6AccBcCTh{5wI~??&e&%w zmUviMr8dSbpsE-31_8ksJW3kbS0sP#b(e2Ff$ift1MlVuqqZ_xQmZN67u&RUA6TKO z$l*?V^ws`;xA$16>J9NnE*>-ND?(FArwdgSdgP9x6DFu zDtO1#;%Vr5Cby_Eghs6!ki|)6A1b~_)Zl$!`mgOr?9c3LnnRwU)d2qhv8HhLXa4{z zKP_YEy!~dTKdjup3~0Y+b(*#Y&}dv|XQP`%U{{YSq;*eHMo~uczr#Zh!rs{{YjHX17cIoYVC5 z!qzGTVU-&$-~FG@*l{+g)77P~Wk?ixK1ul}xt1pow`NTQl~+;(trZk_0$ePx8H5SM zq1ZT{A{4GABo)TdkLN3ie)%Ru3IWI8G1?8Q!`k5+lY`3v*nY4z@q|Tk&%ki5Ff!AzmZ_>{T%Dd9=`=KyzNZ0gKAHi!LC84z7^RErX_wGaEDe_< zbCH3CKyi^J9T2sYX=_jhOECi@?Kyo$ zt9p_RhSB+P%7NxAy0!`e-sBoSqN*GTp zv~1P5ODhb7al0e=Ls}Ogfa*^tJP*=U9IBGa#hbGMm}R(FrCPQuquj@n`U%TT`6GSK zrC1JsgmyoqYLG+ZuKP#j2eN2~#20WkA z0dBE!6*2Gx_byKahCGsCrmZ(-pne=JYq1H&NaVo^%Wih}sO^^ENv@DcByRh7jT8%o z4jTiwJWiPL?Z+-j0vdZ0+Q*D!ZZg{ss&WGoFnGuImjr}h?AZIoH5#|9ah$gbKEE?v zVkYtx?Ml&!yCXf9%o6Kqxp9WtKZuC2OE2~Y_J0{CAJ!0b*LB()l^upXzsO2l@1tfP zW>;B5+P>lx9F_TpH|6lG7@j~L1jbdqjMTU+KZytX!zbdI3Z21m*m0k~%+~Y1*s}@S zPbP<5*K8zZi1!iy0Jv(c4z@Qu@Tg8sK1^3hK*04?nCAsl!&6(nI+p?T+$moQ#EB zf+uXTk~wkPZ(+8Ukd1Dj*yTpmMB@TrYkE=EuGOrnk0rCPeCA&LZRwTTA&RSknBn{nOb;BoOXO%F}vrYvsZx#v7~l-Bi2b()4$FJ-Uo!sLUVzzFwB zU3Q+wWYdFUk8?W#{{Z4@Nd?M_M?hM1^y)QxnTYn*SSpXMJ|Z^0lC4=fRjlLlDxWc} zdTSb^sj~DoA8)vp10CiqJtoagsY2dzNd?tFAEetBgVZ{ciyV4mVeyQDOgBu^->b5; zI62NnVa~j1meE3l0Aj|+bMO!IkoK$TYtTNWx0k-f zAhU8tWEgyB8Ihss9f?-@*h($p3h$?yV~B|=NX!jn^CCO zGi53YGBM6r{pB>b*M8FI1&Vpc$Ic^kbj21Z+8aC!cm7fnr&U$r)JCmP4(L0W;)TPQ{#`O&qWHgEcM3YuIz}+zCq zNI)J7@I(XZl74_k$Y_O1i&$G2Fb5g(A=Dc2y0yp7BPUEncv{;P1!Kksm~Ak&i)>xBI9&D|<~lUWQ!F_g zW<0~w_N7PeJiy(3XxLN4#(0cVZA!-u3#j(w9{&J;fj4_*O*<^7uU_RR86P5hsOV|E zf>uJ{3={q0`fXKulnd&k*rT1NI5S4NmiqRn#fm@S`%V^Vpmbk}sQ|S_iHsHV0jTJ! zb2FEGf(Qe-A9JDIbS15Tk`{Zg2+vG5bJdJS7wyfJ-PYg zGo04dTUTH!jC^>+&_}dXEC6ycF)6ItbmfN&jDzzTbxV%a>GFuwIof@sV``E=%oS7u zq&NoyjK(cj)7bU?WMW;3?Tlt1mxo=sBLgEoXR2=cQ%5MMA|?-hWjdbaztsG@FbN)W zFQ#C2zVB*|-zHmArnb~K4|1^N8Bc9rM3&;zuqaCXvCOqyKrY(U9!A#W_Jz?|zQV0o z8OJc%&Y4;Fl-N&|;7xV*%-Omv8nTq9wL24t4Q`z2wFnUHRXG4368d{rN(CbfSIF=n zEYw)~hjd48cMJr>ZCw_jbz9IJagp?duHd1RV|y=i^ojKPEEhOqx6eI^gw{25T2ibQ z@-yZQY%AzWEZ=c$-TsO`vIe4?cDpg!q>N**@dV@6BsP_S`Z>qyOtz0g>Aiyxt>l>~ zaV}-f#>;spl4I2CTCE_l{{T5S1IZp{Gm9C*rJv8VyXz4Urb?=DljrL*WolU2PN`dx z4pjU%69-ls@vfIxre?sksMI(3;2_*OB#@)QhRzStWq&p8{xT?Ns)Tk z4a_@Z6J~&Y2B?Jus3fmB_=syLSM;qp0Z1f|A~mF93laH%7#|{5eW=mV-|E)YA8N4t z+44WIg}&jXt^Gm0z}vKcBQtcCH9Cjdz^;EN?+?^A)q;$~u-&&A=4LdXOC+2d!NwqrHzREA5M$qie+kG`X&JWiU zw-!C^jO#b3ToTJQIv_1u2IpnNHD~HPedMON-K5g3fR--gzIo>nOZt1!cdw+?X$52g zJv~Us{{S(Du^lQ^UTRWod|)#k_!HNcC79{vnD%<0yNzFmP*3rXGjFHqG_?${3dM3W z%*WBu_>cv8MaMr`y`|8dFjuUHz{3vn&zB{2m`1jrL(@e|iV?{fIT6+Uo~Gg&g;e?} z#v)WzZKZt`0C^aMr>I+{yjNniXFMG7IbKYiW+nB%vVBYQqHp;h5q%#{YK5t&tgE2z ze9pV4_1hNlTjZ~t$NGksO|PM>uL%9{Jk5E^X5;7SuheNS^vZ6p#}jiyudeEdva&em zh%H@hrJ!%q?*p?h@XW^QO>KX;8(a=R9p-DcF}{{*sAhGrus_2HHQHi(natO9tmyYE z1#U_C1H9?_cSY1|^wj!`T^e9ZmO^)6_~*PP7c0@ZV}#y9P}Q)o))`0{PzMe4ni?G@ zvc=p{)RBT1WM?y5OVM;%ErVjU&4wy+aq*7=HK^)VXuXAMr3nD28DM_6o~$@5_VQ=_ z68a#$66*jrx2IS<;Q-Iy<~vtL>8t{a8M!AV`INg+uD{=1U{a})7;bO2I}g{HnbbPJ zps3ZQZ|aPMY-BbLJ8(h4`ibb6$n#w7tyk#UCbgGyjyV{>{XUQyj)&5CQ*+e7ch7&< z%nwX%r%z6<%WA6o$^tgNKm*1M&)0s`YHD_;QBzg7-YePpvB?CViLFL=N7R1Hs*?3; z1*qOPW0{t{A5%jUZkgSC242AZr$^H+U3Q$_spwK0J4wiFc^?8f{{T`7tOeS82nUdH zkB%mMVcUl~J1fDuw?I#Fh6tfm+(`W=&}pKreM;nPW#EN5S(|S|(HqJu+N{Tw2Zb=O z?Z=fNK_i3O4U`cy`i_Fl9TOfHvruLH&BSyVsyWQQaXJR* zHB?P)btS;uS0w)cF*5c2J@iaj-Ax`G9OvmRA`jXxXJ}VtJcGy91)x*{NR8;aZmVC; zp1<9LhPlqvQ}Lglpvn48eO(;~_Df1NeRo+}u^C0NH#7eL{yKY~yYcjllv^47Kcnk9 zfks61>SKXxf&+i`7|VWu!}VUJ`!V|vrTVU?@f!8zbjSl-PmgaaSo@OS7@T)o{{Z-x z*1FKPpV|7GrD$AjePyki>>vI6kFs|8nH@9sTFYPY7VpcPwfB#EXW)3x&vWxNxf$-7 z>tDA20JOighRrUJ9+A@A;p(A?cmDwVmVRP>oS*z>x>xL-R4Tiu1pBu-{{T>@`?h|< zbXs1YrqQbUN$vw1m<*4%f%-<iBQ$OdSg=sM;0*5lCoWmjw+-!EFFIziD?1pkA%}U*>&aV~d0ZL2)N9QvVvu)x zfgjdk+WxgbGX`Z`bHOL;J}y$@lkV|ZaV>AVN`=#LU|9bEIiA=j({4Y~W4oTU2mxXI z;2jl{^5lO|Wa+Uqx~HouP!;^76NvKaYE!&cGQ+VXfiZIH?#N=Z$zv#OnZtJnkt$sV zmef$%j4lx8YWah|SogWf2LmVlqS{?WDM9s-inm}62i{aVE}+CF(3O*C<~)pe5v@Lw zvr9VwfMc-d6Sg?p)6JH1*dL7DUK-3W;{aqr3s6^r%5%wtx`p)u<7oc?Nci!YDyXCK zWjg_k;Qak#W%XH>UqwjWj&K0L&l!dlS%S^RJCJ#jR9BYvorjRzC9=L3a?L0y^u0OZ!(Ws?{)+iD!r!JFtiL)zYo zU8;NdO4#H0v5d>5cG%cr034hMaYv-(!BdRmB%J5Dmlf|!)&UB6B;qrVV)94{rBJZ} zRflXgNc%->UsA=v?s(i|c9g20Rs8mubTyZC0q;9koP)E?stHDcs%(;}9f(Frx$FK$`J$^QW}hrJ}`mqYOF5Nd{>>>v>dr z7tkY;566jyS$3gd57_&N`1(gp%Vmrfu%*Wb8OO(IjPdE$i#=Y*^^L82r3lv6N9M@m z^z$+LuA0p*ol=&u!znF-Kr=&7x9PPOtMAxIVOZ>W=lV?R_14*`7=5*&;IaIoaeHND zB?kRC-nK4%`NriRGe)j}^|D!AgVH22?KmX;V6__aT_xJ1wX^)bapq;))UGU+Rc^&m zf_(n~DJ{CnhKokltF^a)2zMvigWwM)DOQuF-DTQ}^oWCL0D@VY>OLnqcTuraR)FpP zB73>>D`u|d1*=D_f#tFM=Az!4-iD7x71V8F4o+~x^#1^OEmv5tPCXi1(o`Fc2cN(6 zm=|i8X*XKKHv5b32RP3hd6`z~*?|QIr{#&7n6Yw=oBD~>F=xoBcvLOk8jWY+bg!|C zSmS6QbB{Qj+UnenG{XSkF>XJ!Fa1H(GLRJs8GQErBjV4TY~QR+GMlYtd$H{YfQ0=# znUQ}}3<@KKBH4t*{Pxv(9@F_u*3tCf_5wh_6XX$ z01yrWfG}ef=?6GP0FF53GYed=E|~Omm3nuzeEm#ydO%X3pxNW=G-{;QFndco6QYv)h@XYtQDCWCOIkVDqf^(bwN+C$zHBTPp&f9AkkT#wDvzTR4}DGdjq- zrUynf_xPOdt4P#4Z%pES2DJw`K`{EjO1EJVv)~>?;hq|sb;CWm4@$Sse4hq-{VTN~ ztDK+B_Kew3VY;z`JFHE^0kb48#N{h=DwQiI-dvyImM9w5F22*){1^t&65)U>eyxcu zu1YqCOb{_Ot@;aHLa52)5sXa69-gCMPIJ#QTD@IbgbRWj{{SvCGqq|Imp~Z*06DBJ z-l(T^I^V;%HF7dHo>Z7xK?NF1 zx*O5ocA+ztz+eV*@etGB{{YEq4TWq2D#Tz$^!l)Ysx9nc=fobrOG~>y10a&ZacLTg zD%5F;T&uYG5qf$E^u>k93^9i}++bC;49l^D_$*9+Ytg37+0Sjw&*&qup$3hUPKvt= zR=Ff&v{O@Vbjx0I=Q8I|6oTY%Ffd|0O4Cl7%ZO|dg#*lS-KmMKaH7!IEr6qszd8LT zL9LrCgLi4AeXMlZWl_72O^;*eKcvqWGA3OM1K^x-J-GT~Av6r(O7XWo4|%ec6Qc?U zIPdlo15Q~k*^*FXjyvXUC7zWa0uJnsC6w*c3)3uL>1r8OMn1DjolTBGbT9G=&VNWN zw2A2@z<}e7nR+#1r6hz>JTS(8S({A%0J99$`n#J+fsS0J0WhD6r4)y)g13CGB&N7N zIdD&44r1DyQpVZN7bl-59ii;)YBiO$yQ^F%>`!<#yEZwpqDJF@qA#F!uWLSbkIYE! zVK2XKrI!S%p2AvNXwh=GRSFS+z*cVX?NixX0vU0}GckV|RbYmx_KtbtY17mbQ%G%& zwfJ^9-Sm}}t;es^K9v^-aBfR4Vtxe6y_-_)wY{6Yp^svDn%3NPI%8VoxdgETjQD`j z^vhbzx`ZU;Z6cD1Uc z_Tw|0n|V>L)f+WIv%Lm$=MbeUwy;ybaAtagr>|+oN%JE|sWoa@ahlKco0#;?qfqUo zyJd}pW5M{E>ZxsDD~9@ZF$U*y_mb%P^xlg!xp1s!XHgGu^5Z7dV2CH_Sh8(sqPGL2cD7unAiBCVQ%g{Ih z5cb{2(jQde&1lW5nz4m{*T>zt)^ zrww5zZy5q}(|c&?ttgvR`v(jUW0B0hp5VKcEAxSZdmiFs*36!ol6h^!5i~TRkP^cj zpVUJ~Z4G6~_yMO&RO*>a1#ja20AnyZo3)`ekz>wHDT97~9x= z<*R3|`xv;nNFMIk82a{|@#;9`cUih#m#VtVl|&u0j&nEk-3qGf#f`_D#D5y>!Dwz_ zE4NVPh1P`%Gta~_%GEKAy6n?wy^6WVnRM#9MzAV?55&W>R)V6Y(#n5`MArq2r>Mih z#K&#TK7mHEn$y@y*nAk*;%U|>uKhZ`e2Mew?bVAEm1pP2dC}-~H>k-!m$^OXJHNK& zZ@!kQ(A`z8$i6X>Bl>+(sx7HzNgRVSsnz5tdhkf$$0O`!hy6UTR#L2cjiuD$uWyTETwO_NE-2ib_Ss!Q_Ae>;%oO{aX zT>N`xe$Vt;e*)b@s#2D-5NoTD028Lu>k{u(H5CfzK%rFt9DOk`I=4&en%ziU7N)9n z8e}Zqp+6kw7@0cF9>3L19*6to@+HFcx zcA09^x`I`fr3%1(LFYeQ#5%k6onKE!r=Yh+mHz+;W;uV;ALeGRr&H9@r0P12GN3-! zR4l;uBakuhb2v?B?H5sP70*K0?@`LSrz5!M`g1jUQ}vAxLM7gk_U7ABo1C7*Ne8)VvYcJ)GJ^G@GIPN3ChPFsnhO7T9tQ7?*By z!T9YQN;Q|#x`tJ#t8W|s05g08@i=;VF1tp$jYV4b+}`!R;ecBn;08P#@?mt{TUn#h zzNf1718INzN~kjD{DBKjou0j_*J;^E&Gip!0!C*iVia--2ki->zQIMhyPffFLK1Rt zWo<{mVlpw3M>v_KK(fJSmJ$nUVssrNrdMqXZN*(dB>+AClPgO>vei3x0NBPjiT=`c z2GMH23r?WYD?)hr{{VQzcc}i+-m7IjB9rS-&FWT(ky?)=e3)$y?PiavY0+$J=?a^i zwgAu6c$}TRb5)?xl=L%pqS0;IjDOt&>+v~WrPXZJ+j}M3i`Y)Wa1Y!5d`$;`U+pDb zJ6*kA%hvk0RBj-#1&?vt;{r8D?T&*{ci}a9K~Z%%-Zra`1QFQJob>8Cr%>00(`nw& z6e9Xe|&@Ca%@})qX#B zSmg8M{{T6|y*+>Szp0&HTGd;tupi1>*;mseg&#ru<<#~&zMYp_{{Z-@ZN;>|vN}Ia zYxaNCH7Ewb{{Vu1RQ_4|PE$+JHGN-H+WmdDYVWg_)ar>N^aqyxc=}F{r~c3BG%AM0 zTirh|7)ARm)rHEnB!o1bBTJ(p_WHMft@+2;b`jEZJkq4h7q?3YG2RtKQtBZ9kfVq| zB%V>sFoMq6?^6O>S{axD7q-KWB^u}#vocRJNG4F6$jp*YZv4*)1kVE`HG5R$q1MI7 zBD1+W5PU^b8RwY5agbCV-t7MX&d|zjtZtsQQpMUBmceWeF**H3PBj<-H`;Vji-1_RULzGso&`NF@K12(;hB#GTcp&(U>JxHs zfylrowjv*gxU(>oPPXKxTbj+q2vNrDafy1XYpe#)p;sK^6Hb$J zpH9DGcnjij%P-e_anmnt&gy+TJtgCBJi|Vmux6=F0Lk3FzLRii(xhX1iwcbLdra+5 zQl^$7ylx*)GagaBT3IVp=EVv=Fiv4=a^$6XHSW*Ge(^@?7r7)5GI6;4LRHn6k4=>} zj^`5cR|}?+)%(L)T)91m_LgE0HsTM{JP3fYw>#CCljFHER<^J(vv5(!9y2xNYoCmr zhp5f=Wih?Dz{v_BsnI5xcR2x%>SJijT}r5%6nAnv!MX)4xm!K(K0foe9vc4uCmu}q z{wjMFHwHP*I1-}3!)R_ho+aAU+7)Y6Z)jHd(zo?5?&@ z-b*kipNUWl(8%@zU$*JW;PmnGeOCdf7BebX`6S?E{KV{`*5aFk5;rbR82u+juj%ZT z7Hp^lgYB8Oty;Wx^#^nECpU5yX>DX{P_l(w2hS!YeKnxk;j+Uei)7+x7qd>Aag-+^ zU~JiKN;Wd>2e|Py1!UIjjH_C1tdD3TAfNY|I(ikW+~^2#+rbe!l_<5+%G_kG~3rNXhf?VStX}egFCq|4FD+NcNh^Drhn{`Q??Z5+)W`?TsS_;tnh~)ejt#vs* z^R6*gS$hPkEtDfYh(M*geDP^@F!dZIx^rK<&7kneB_Dq$ob(Hi70= zodv5cdhJXA#~hes(>o2El>>(V0BK4PUv?K8OJo5zYtfq>7ij%tI)rq+Ij(TKLo&H(mTv9lF-YDT!`zrC(cJ*j0jSB6uNOzw&O-Y__r z{Y-Ab%=7YN;_Zf8_T#N6TaWvb#4H`{N#)i z<}s$#+rFT@k(>rT1lCw}xTSPdEqX0>^TIH|GeNqIFJ9LPX#=xx!|Y~H<7?E`fyUM= z?K$%hI@(ilYQX0Kk7k(gtu=447Q<-31#&}p6A!3reNy$Quz`XYWDh*?4XV_;)*S-a zq#s}y@1LLZ5qoR76_xV99?g#7 z67y$mz+;awzJ@;ZKDx(E7-ivH!0FKHS8tK+ACHLH=f-o5T#6W7Utym}Vk?te3 zHmo48l6X*xGo1LAt?IVS+f%Ur09NudDDSzf3;|Za_VMSwBicei!jH7o z)Dw1^(`iC05lO5v{{X8bkB)w1TUGwj5~NPpJmMj$*4(VcY$HEeI(>ek&rfh-tZ=y) z24>4+dz;f+Pjb{18)oc#wvRs(Q%c{zQmhH%x&HGJbq=G7fX9#{`g?O_Y`~}Z`pjLr zMK#)kQL&!ZQtoq+-YVMS#MV)j1^^`I`c2#Fqo`Eq?YXnKAA>V4=!><0PR7M zELLBpZlhYxjzOnwA z<%sbYe5#fwEKV~9+M$xQdz;C`_W*m`Zu2@!Sh*R~>&gLDjDF%c>Fd>mBX@Dk=H1D1 zetxmN9<9}a6Z*?8KDtvr`)%nh!P#&lk0M>H72Nd_Z}u{I;Br$MnJ&N%vrTLBC>eLxcnKJji+s0JF{7kJ)rlR zja7P_7pr!}BlBaKk)o}ek%m#+1JClC79eWt9HH72WZ)Ax?Jc1ze!IqV^Ht4&DRP&XbXi&B6tnN$*k+&;a}@|vv3QmSaGpUM>D zk(lKg9X*H?>U;jP$)M9MzN3Ig#(Pbz9-B(t`+0J~7^UvQIoWbg}(WKMUp|qC>$jTc#iSv);G4(p0qttqZ8eKC`&wYU~xdO6J zb3KnQr(f1Jhf>Q**{HwOf6A8Ag#-=`Gc0O{SVQu`b!&{{YuL z_~vx_ai-|q7+HG_X0FxVp~8Xz)3vsJXv)QkJ+YXC(Gn zC3+9+{{U6jjUKmKXQ8IZv5Ruse0zJ0_=eW?U%>TknqVCT#W3FD%iQFAV~I|s*KX6@ zW{oUh!IrxAsXLbkkO#+^?LSRgwZmCrHO`Dl-hHZ1AfE&00tKnmdd-8SBkCT+s~I(x z9Fy_!pP7wC_iHtkU8t_Nr#VGqz#mWRGy2a`s4EKqabxW`Yz4<-{N{&4YAv$58J#zy z7FI%Vd>_03-9RG0&4+KH=ql3cdV5v4BdE%IvPaTm-BVRz$r#QuPC1Ko>(fbgm^+3j z#eGyf=Q$V{=g;$-{a2^kRif+=gl_kJW^6399AXg^vjqbr@G&F^dMUE|R=QE%!`L9_ z@AMy8X(RD=Xsz|#R0PkoFLRD3FZ*58x^9(e)oIB(t85l}kH{opJeM)URD#tN3c!LWR^1O$z*~{%_9F-bb+jE_~&?_wzIQr%~#9ZD6*Iy2=~_mK&LrXFd5KXTR2QeOsyBqoUtVvifui zwz`kvKmH7Jr*+%($@t5hrL3O7!-~`RMh6iNr`16f+{IY(a6^3J zVzmh3MrG^`TOW7Rg9J>;Niamq6Dye~frqce8m>%X6ktIJqF9+x+Etn<=4cmlJmv|E z60GJ|ah@_HR?PNgWX4doXR{+S(F4>08=Hma7$YRl1}oV25P!fvoJkBNX(7uwql8ov}%v~w8IO7@ z$8&cbgz~1tje}|U;|Jz-$A?kNE`Pvlpf!z*q5;S!^JC^5R8)GH0B0oU8J?=L`<$pC z@woYgBR(?U(saWOIc1*eh!Ku>&PX74h8RjpfrcDrQltpJr)ECc=OI*|(1_ciJEiRe z9tL?aP!$==k^$s%*hU$)jGxTI5o?rKst?TZWp`ggj!g^2r`oI&KG8JSS@>-~?#38` zH<_Bj>Bkr)V6VN3uG4}DoUd-_)@t8S-jbf*nD^!l5dx6I)OHx>#H~S3=Hor1HICC* z;kP{bJXHJ=sP`O$;}Vv133!dK{<4=!`B0StRg-ap4l%enqSuAtg|)&xE9 zVfHVjx!m9#-sb$r=prA7v3S%gA6O&NgyX2eP-t!x3(`N$I^*ODY;XQ`EL;9ciNnvtdLo`=Odo-Fr<5d zB6{%S)5ponwM#dXkUI$janCYIQ~V zSk^EI@O#bRNdS?Mf03530^wMYJm6+x;O~zujw4Y`gOga!c^?oq(Or9q$sK^i>$Mss zu?y`RyLRz2ZRxLJBDLHzkEopNvb#*>j$v!9-MvL16T;((Cqr8^+5qJ60pLyBI$?6y zT=Bv9fE_CLa8Ysxe}gabYjKC@>ODdO`$@>>v}F;&+E}?A$@!a$(HM}=jOQ54JzX}v zJ=tU#=YifG(Q}cIx_8)0qLYqP4nz%|ChJ^aF#Gd5+NmtBu>nGI3kcfj(AR47)aUOy9UhcyE)O``8yLiE)3X~vQ=gIkV-(Zrh75NxIVNsm<)0?w z)gVRM03;r8WVpfl!*+tRoKBi>*=7VDJ5M8Ek&g0SLidOhJYy0|3`*uzjFZKn%TE(1 zObX$lRieA0QK7PE86;*2Oe(}ZwpE9*zR-Y4#I@2bQN?SB;xRN3qnz?$pOVd|@{`0b zs8vW1D;xwQn2x%!W}gBdc_W!{7tgb?+ndih zhNjRK#&Mm)7?p1t_NZEwwj1ByB4<4T@XAV@cEr;FTS}u0vka~dIg1KNcT!6p;eI13 zu&ykqlW^Uhaw8kb)9nPABRs8$RZpC8DNc&_Vx6nbGoJHX0qJWmb1>r^XW~sReO6WC z#XJ%GQL8Ha6^mN{2^bz4kCm;x(s`^ENLXbhoUD=gQ5qils8nSVM zILs{_GO8T4p=^H+Te15_X`8!r4O&}T#Iyiek77A9cC8w^GipxeU(7ow5ELqUq;cu0 z`;L1E5w2?oq{@9YCmT=rnYV6|NM*EZYkFx#9Ci!;06(nH>Ka9@4?mP;aoZ+7{Yfd& zimF}N<&H79{{XOw4C}24a|H+IoS&>Q+PY_<)U|BDHV(>0}cWwy585kRi{$ljzWIL{~TH-80q;FSyT zAlpy80;&(rdx`YE7sdG)RkH3m8+hcHwU22yGorVpf`wmp1_+RyDwO^eB=RLVsPl{j z(^%jh;P>OSGW6>dscd_XV8bKtI&D6ht=mIjV2z{>-$^H=>B$1Jr01Q{_~`Rz)1bzT z@3j=za03uNV&kRXM(49&0OQ-vX4>g2-Rb)__6wYhLV6;)_PGG51xg+cJP{v;ACZyx zMs(nSN0%5f$4{&ED=rDEw#7c&O$S|4qi)zso!dy1O+Y9?LZP@A9L(MbXDc?{E~$x8 zRz`A%xgYN?y+6~m2VzH(6yqOxyITgWQq)oqV3)TE5p_;jcKzTz@NM+98ILIfDyfNx5wjw(OAZ}Fyl0Lt9`Kj$sPU>r`aQw+`S%1cSVj?k!yYg5!2bg1<&xQyYf_&u?St<$>wj6k`@#TdwGg>Ig`Eh}X>B%B*DoDU{m z^r*UA%2LJv?;HjL`c5`C?b7QyZTgbBjRk~r!w&O*+Y z)Ot3zb!1znVtq=7=2aef9})U>T8$MV)&(B6*|qrNzGmBN{{R`J^(w;_8Ff9x-Z_0% zZY1jBqY^tu|~+Q<&>{(t;VX3c4? zi$>BbbCPq6&wmg4yu6m$-m0A&>%SRQXqDAgYfvz)p6fG@;dMP-(z<;$rza>d+q04S z%;VK9(KY&2uq3XD3gu>pH51fv>mIPDiSk$Mxo9 z>AF^@U8&Rc9Ya`}3p<6zTjKzqFqL%eX17CTvi8YjWK=!W;~6=C>U~F7MwYtks^3O{ zp5RIw`HeL)zU&ZDjBsMZZGuo|o&?F<1pKL^D1eRoxL zRnU#YazQ}Bey53_tkR10z`d(u0$4iZk@g*;1l3on*4(t{g^g(EaXgdvn{@SSx<5t~ zF&~IIX54vZJQMaErmYH^FWJrPfLs%M0KBRC{fCdN#Od8XMXI&6H`Xq1Oc9g8WBvTV znCWg?*4Jun1w|zn9f4!X{$JiT)w;gzDJZ2WUMo+Q$8vndZ0Uf8y{-QM<5TY1I7K7R z_n3OUX|d}%QX5aVwm>R;AFg0d>1%B=2f2l{H+61R$I$pZ_Kex9(W57(TqX z#LTn)tE|#3i&;wR83=jF@@G4y)z-~@nw&cyb$f{egU&v((zW%C7f~0bl>o>Z8O|z=&EV8|8 zBO(GS1ODg8&w(?PGQNYYrl8BH(~xMmZ%lR>@j3lh?VTEQ>U|o!PKQ>es?$>0ColQz z{{Z%aHjf!6zA@EX=nV#zsu$B#7zjPXA36D$TF==om)CV-?OiS`MH#l2*xN_X&OXyJ zmn+nIU+lht0@2s>r%#MJXv1H1$K;X#`R;h}G9Opdw70MQH$YO?!2C|EmtsCZ0|0)L zU#;l+Uq`O(K8-9u7t?2E4%s|oAPLOqG;gcFeQwWaZ(@_n5$0?+`cA0c`joQm!5@oQ z_g{?o1jAXX*L3|g)wb%aF3-7!310<#5s!)5{?2p?^gTMK>MWHtnPl7to^VbwKUk+# z(P(2OF&FiKyvGq5yWzM{$mDeqw&2-=nsaa;nYgn+|wY zC%4zMXQ+P6*V3r5#;ocvyFjcQ?>&Y*0sUrmZGM)Nr75psjn+Fp4t~F{5~$l~G<0du zHAc(ZD^^YpetyH|YV=C)U3Q+T?q(~71UDlc&)Rb}S#z>l$JDWb+@-%D&uP{44Kp1= zYE5g~86!W+bFIkgI)nP^&$&**2OaWzk@Vtc7L4Du5UT{pJOtzQiI^1UGU_E15@FS3 z9#M#O?F1#mNiPP{@~tEpgLp$I(m;_QiDF>Mfk~1jVHPkX31G6yuz?~r5G8Uj2tC1) z89^i<`edFZ*(FGU+6gRjNhh~5>IwUKl>$tX=ZWtEO0hs#08TN;%%}v9@eLikMofZs zf=78d`5ei}#~`1il(aUjl<%m3eB-lyY+_wNUc)i~=awJuCpkT_i1_r+mudo3 z8tCEAU(m$N*AIg8dt%E2A}uzJfqN^%Z( z60};15CB2PFp;ZzYg1NKyJG{(9`NBpp;F)s=bsQEhyyFeJHQ>pZfxM3V>7(7A5Sfo z`#HX};2(~2OpV1 z$jm~U$r}jde8&{;K)Z%<{HF1{XDOxA3UUM{3aR{{d}o+El{h3G$1s>t+@3~8C!-u? zk1lywsqdaS-~la?dyWT*;0E5qC(Q8gI!t78xmuPRK--)X=1`nu4${w(6z8$+D~Yx$ zv&56%5_!fvPh;D=ILC;yw}-bR@O%kLCzFzMDqYEK?qvsY z@H=KVfs=uO;7B4`Gb&9~@T{P&4rEn8_b@;S=jbCWicWFNiJum{=r3Ozc*x{W3YC}= zxnMiO5$?>8WufI#iN`&p>>zsp!0{wWmW8z{4+Mb<3>Y%N#tO8f%gV5bEW?>m2;-SZ zq>~1Dgd;Kv$pl2B3`3RrN)?$q0uc;?ze(UAwmO`cWie&q6tNodMaNgd99AUA?kU<`2`)3E@zf#w$O9d^X)dKca5 z0l}GTwe-zXk8nIrzTLOd%IXREGoROV??HE}oEhZH9ao}V#~H2D+8B!XSi@8eD%&y$ z<0d_NErGJ1mLpyJ9jha^AQJ?PB@CPZPk|KH`^XE(K4-0SW~gmB!SG}w6fCxZv*{G2 zd&Oa#9($SXPM()d#S|Y05&Cw8;1USIA4ok;jTi>6vvZEomc?V9^w-uX4$wyg_Kmvw zbq)Y0;K+I!n?o?b`odak+6Lj1$m27%_NPx)qOg`fe0P;Mk*_tJfZTw4edW~eL63}m z;+mzk7`1Dyd>{KvH9{xx!VRXoLdaVtw|w*4wJMHICS0j(ou`9PhwSXO<)UK`py(A+Uq^ za68V6Mu?OFN^aUY+B>(w%(dEedcS$4UE~m0_>RSw>6xSI2zJ+IOmwxYQX6(RYy%wj?;YvX zz51=@vOpVHwsJlU8wwGMrk{6_?*9PP+tSo@S~IC|MtK0AnObp{du7w9w7Djr0USn= zUdC2hut#&^FqctnWqlE}daeLS+H>X>9hrWN>KxUod^eIg#&h03r0MSU zgnPkp$jIgXpIPedx|;t053UF^`9c06^9xgHls$Fc$iVIZHb!Nmu-8e{^fsSGP1SN* zwm8q&{KqS_Bh;=esZYCWpg0G|$o(O!)!AYI?u4Q4kIDTb15@f$1FEsC?JU^H{YPn& zC}~%xacY>F*$D%1CqM7^m#I?p*{RWwO#2&eb^id!e8y_&wb`ebT_aVdc3_0tX&|<8 zrF?z4>}53?-By;N>fw?7gVXnHIsVa&KU&?WVYwJD2EJP$dC!CQ#Kq!`&RRyz1@7;s z)M@Ju_z80Ij64&L$Lsa&GZM$irQA5lEB^r4c%7K8jcU6g*|~AKd*kgfdWMf&8pOYL zr%vABjCcda4{(3yGt0~Tv)p<(`Tm&xwy#g9^z`gSY;dC|Zcm@>AFEM)B-(!|J6XO@ zk&hB@(-){>>g#IjS1G(PRq%fQ08c*?F*T1xmQ{U$)Sf{nk@}7&gOiIaUaWJLI)t*G zp;4qb+ugTw&&-O5(rarY6;ykvB!UN!XFB$cIt#0~J0QrxJAm`Y{{Wc&ldaLzSJJ+) z4{%gq50VV$B{xfTr}dXoRrhIuBCt5gGnah}vQ*hV@>d*@z{&1sQPxFNX%r)|A@bY5 zhd&3*=Pv3#h5BO3EvzZ_vG$+&#QGdQwDI!$W9oWsn!Qdb+$&o)wWrV6O&vH8Hr@BF4WRn_bH9+*=V&AE;A7Bg=9R9Hg z=-QoMPtbbLLYl}RvHFwx%|4%?udC|m)#|J(q(;Ls{QLv`;$wESCYr3*P-0ji$Zsb< z1O4MV?Q3+Z6G})6Rz0dQl1VvKhMTGL)ec*om0 zKl98tP3by&sIu;(?j&)x%DtI~9>I91vZ@xjSqd(!p9Qu99 z`TKn)PKlDWuvuy{E)*7A={EF=^){0G z3BU&&XNV4^`$8;DZiiSaMyr)jz=j;)Olh~ze`)TgCWIr{OB+cfl27%7{g3^d)m_x9 z)~Qvk7b@+N9lnAu)O6KdYS}=sJ$x^wSTJVKIP;IVou}ij=-n4jrs>OIYO5`6YMX%1 zcF*#}+>EQK(sj%F(4>(?TXAKcrsK5v1Lx~A`nOQj^tOLQjaJ?^)TCu`Nysc74MAnwk_^#J>xWQu9^wOOeNWyJ-fDqQV?>RM{` zb?mOb7!l76pN>0mGxa)Gt5By)5|L;R`s$6Bx7&uAfAr zArgXj92d_A={mY8T^65e9nJ?WxD z9%Wgx^O@xDAZDS=Fc5WihaI9eqj4Y}Wr*ffNeCn!(pR*H?IasPw7VW<8b~t+(8NKJ zB&zp@K?J?wl1OC`fbj$jkkSGXDpq@ZjnPQN_9u%-%o2Zu%I14ArYT^}@^ST&4u8D1 z6eqcf_yA)ctbu4=$Uf(jGq&R3U=r9E>^ztv*(8upz|4|EF=i(wRvJ4*_i5?j!OtJ8 zeKWP!`JVzu-g_c@Bkq61j9phFB;kHwtz~-7dBMOQWReR3jy^Lg=r)s(4tVBy`6k|; zEVWp+(TiA>$pa(l6}t5!Boan3^qRWbPSW5IKfJ}gaDMaW<&TYhJTvlxSnF<>0H8P0 zEWab3aXnLHw?5JGWY#dpXe9SAY~6QU!vuh;KZ-IKs6OM!=i+0VkAhF!#;9f>_P`k= z%3XFW+ekXKK%VAK?5Nr~92kWcpHz{8F~`JZEGM6?aXRCk+_J+-Wngy^#L&{Pdh$Ty zfN>LY#d+Wu#+!k(9v8A9iP4o)DzE!(%uuv7#cr{|gD4l)i1o;(Z# z!J1~Mc$Jqt%4ZP>G6#7i&j|&TLa6NqrW76`7Xm?^6U4}b!Ucpgca$PQK(LA@fP_&Z z@`yny1h|4uWD_ScgvmT)3F+x19@3e_xr2d=gDwmRJi-GaCS>LhGugy|>nubPIE4fe z5CaZi$mSWxFl0p{dU{qt=9DwU%9&`rBNN$?kp#5xG9^nuiJ1qPP13`eBw|<&WrEL3 zNo}BlPng);C<>z^CI=H z5gEkEPUHZQ-*EFP?_*cBtM`0~tEp0hmE1g$;vr>@M#_+2X5#4P>=YBq> zH>JF;#daru7?t!dd?*KMjIrz`?4g;{+(F|YO)aNerZy;G4sn@{a#>Z5xb~B{XPi!< zmtrarfKCj<>Lb#*gAzd;=5Gu1DS8+>6$5C_LCn*uN~V&QjGTMEvmtO(au65cX2E0m zhCpi1f0@y&Z%qE3RUW;Svw(BLx87*oWmFKgZETNrN_E zT`^Nk<+h!tl3?{(+v)V$wE$y?#NX1cw@gbi2LyMAuTbo9VvPBiH1(_Q0eRd!Lw+Mu zyMN*A-S&!Z*eS4J)FADx5k=Z&EPE9OC#KbE3WRMtdyjG@t<&6ulv{E15F1?eVuNjp z`5tk|n)*FJy*DMQ=Na0^na#L1)HgdTkDkUYP4`!0HVET~5l^8eopt&nU5MEo)Q;1X zx20=zTJkDZbFQ>fKVNskO37rPBOKyopp#}7gODNUHOVg(k`FQv`_1WBv*f1_WEtf(XB zI8oTb!-#Z#)KZ$CO5Iot7iQZNE8o8pS54Hew@j^1qhJg}5c~o2gN`O2Uj2`^Ru@^* z&15p*fO5pnZC|Qaajw=iS}+)}@A^VReZHk%QLO&}{aB*FKP{Cu2>K9WTHO|tP`;*! zIMgu8Y;aje$JBpm%;@!-wEqBw*4`>n`ADaLezQkUKwnH%t4OX2S2m+nqPkqXNhaO>J+g|RQsMYGM_JpBG+T-$oK1ZJZK4xv2B{o)<)oRv( zmpcK+&m)X|%<03optjXQHD|b7lb^H(C}(?>8Czx@7lt_QGvaxP&BAY6lSgNJ=o-f z=RR}f%neSfE3~-7`@N)3pPm|e@yl_w>o&RBN_)xS0py9x^$OmrE!(w`dBbs!v~+7f z4On8q%J3L>QTB-Ht>1=H*mdVI;s$i;~;1dk` zy0$3_c^r&nLEEfW(&D|s%;$Ibp2wDonmV+#ja7-g*4(;|;bS_R^fy#$sa;EVC071zfxCqt(=x8;&Zn> zn(}`T+kHRvH#qShEl6Z~Y%gFLKLEmX{T7x>kPsI=rqw!us4HD{jg=P0?n|Gq^5S#atv}&Rn4g(exU98(gb)%n}@cfBnDjI9{=+R?+?>>K4ncC)u|+8SZn*IEw4K zg7sZ6Q1)dxM`OcqcmhBBH~TYxQEG43T%$$5^N>#OxAMTuw6@m$hUr?%SnAfA?NeW| z>=+5pGo<~dx2h(B+9;@Ow+mLr1OeEu#J^Lq`UZ(V09NfKb^EJ04~+70^qK3{->YCSbkcMzelG1!4J>glbcV8n)`xhwwwij&9tPRBt=)9Dnl<=@@gkPPQ* zaLu09@J2Yn?G|G$?X=^A_An&Om%Ers$1p>@Jz_&$M?e6)_JmO`E5Yq97nr3nJ%~-S#UO*sg3{$op;IKL z>Fv@PNR^Qg8E{k_k)MbmJ)B7hU5PnAr-|UC!R8SmrM0#ngC}8yOzj+*IEnzF9OH=$ zM}-{1xWIujNbxPZj9`87%&ugDJjkw&2XsJ?J5Dz1~~7`w&RSEf!m3T zj@!pA0c{Eh=W!kvw8YhEPpEx`fAKyiL7$EcrP^V71%0?b#CQUEd3d+m*Tu`H8F*}C zXvB=!QVqBxf=Zn6Ivcujs38vJ`pnz3EkSTAQ=Se+XO`oqw;bgr=Xv1|&`O&y011Kj zmVGlXyBLJ@n;}P@!edU8D4jF5RAT_2nED(e27D9a61q(hDY?7-W5-C#mjfdm=JCSD zGvt&y*LsCsezQiJ$Fv3jcFBn8s5{Rc&StW)QNSJWK^esKamw;?UHp#@k+3aFLQ6xSkNyM&YvN(Lnf>Q`PNFk3hnU+1E z<+398g%Ncy>SULmo}Q9;nNnoWVnrB|1dt`bkO+ttjTM4)g&=1VL&$N6AYhDdGuz;S zp3IQS{KgUS5D$oCw}}%+m?I;2$9V3W?Y|6zb(uB*HCOpABOo8yZ$r4{6&n zme#3*FL4{MQu>AW5`L2x2T75AZj04j)sPr-IeXh{J9?O5+;*MXDvFN-2ZK1?v7=Sg zGbulDJelLF?~Uo1WesIusa4&}e2BdyR)iEeEsUNpAuJYF!NCQ{7y#l6RL*H3lPpF_ zUeF;~bwP@NNe9P~EvKz)zPoX_jPQ8DnTFd?vRJAg!82c_t%~ZJh1$)Y0qqbDuoB>@ z>?S_AGwKyNJoX-D&FFWWaykB!6Ip8uzT-cv);G0%JaqM~C6HjSz%l*7P%41g85_PL zp{g|n$j6)+YFSwj5s*n<_~$tjq^lZa)Dur$7juAE@!DqT5}W@3cafABJ}`bJv42{Z zzyJg+K=Cm&p@!=Dz&Xh0GjXL)O0-seAjD^aKhMnemaj(bma7W^8%8^yGjg3suAp<0 zJK?dIEmhRqi|QdOobLyb_K%V9`(|}R?@k5)EOY$jbh?UcWfi+|&eP!iB{j6)bv7+P zwmX7E2TiMDGwg5to8##41Lma)e&>=th6y#=O95(;SB(3czuPfq zs=Ar4C3ZJE0|SqyW~SNPJi02c_pxKN{{RP{)Ihe1yB5HTp|A#J^!n!9t+93v0Pp;S zz3**~4O{lFV#hy!w2q$A=+3Wlpdo?yhtzac$0PWhfohm!lK*+{%G4(x7uVSLY z37>PP=Z?|Q8pPO-k>)nN2S-@V#2kG1nR-sHbjz_=mdBGlcU7|ln+GyYsycs8(KA}8 zq`+C$1$Sgtm>w}O$5m!R4o+f~v)yqdkiSVMSl_f&5I=B$k03_0{{XafpJP&(`%XnD zt*qtS9})WWwFDUi{;|+?Eq6=SFTR9Ae25#mA5XTw+&YCl`I(ENV_xxRKX@NcP^Uht z544che$1^rRE!jyhMv>@%IoUX3v0PZ@!}kH?M)SgAP}RumAm$kqN|HCDbQ?j`1L-g41H>MS_P)2;tPj*QYRxxI>Qy^eXQu8s#~Ff3Xu5W# z0McIA&Q(ixj8oCNf)nVKWj)tC8IC$t{Wj$^Zeia!%w>OQZqnPb*4p0($d-(!>pHHV ztFwJi>AK_(h%TX{UZ?2YQYP%@cH9*|20x?yq^`}WWqY>_5=1*G_t~c^^!H)pOtoB& zr%Cunv+)*IUB4&?ID@XU8b{P)7-T$b#i^bY0c?6_O65p_={CQ$z@`r$bGImwO1o2e<_G zoaa!|Dr$0pN)^L13}g=@oM#h%Q&y^~LsI6|UU2y5?mj=-XX|LL%9}o-SB~X#-@qR| z=MG*i&Yf`N)pD(2Q?YjXW41qtMAWf!Kn1=r9J@xeSoCi^ukhoK=6U1dDSfY|uh3Gs zC*Og|`V;UzJ7+V&WtHpR;C8K|!RZ_la8LQ3zOweRwZH+`_r$rlT|fzd+0P#_X0)}K zCk^Iz!!5@Q@zozgLtM>LFv=X4J)@e9V^>Pjr*^`4Q{2VU-@dG6^y_gL9Nj=P1H>ayzfR!ok$Hb1UaXy=an#*GzL4w$e z=+Y9wYl06acZq8CI$JfKucEsEBLGa+ybAFCD8a{`p)YSGfZLNE@cqi!^Zh*Z{@4)dzf^lN&Sj@lL7ZgmF)dr18c5_)Isww=HE9{kofHDkxxblov58g;UgsINN)!LdxN?4mIsedSPc|UxA zc)ePyIu*3gTh7%uZSXA4xxth-jY9d0`4B=0RI3m}Hn$^BrRm>NxNyp(fzN-bn|2jbwR-`9J*U#0 zE}2(?#9+2yoyf^OzK6W$w!H+$lPQ^2Y?HY&zz97WT-VOxlyhXQ*rd*lY@NOkWo zo*;piCn5<2pqR_F5t2kA_=A^Ih@nKVGJwx+kXlZ6@WtIkb z$QNXuw1AD@A8te< zfP@xd^&Rswbz8>WsOgmjLF2%SH7H3XatQ4lXDz~=tMnY1M$XJBB5}NXcbuAMogg0I zbF_9MS*oa^463B|$0jxD?)?0pnc+xLj&|pdyvM}ckw z*vvo#N^Oi3`bun)LWn>l1;o6NY#^9M7(p8Zyn`@ zpIEHAnLue=pD>YhI55GYE6gzf{o&ZaHzp}j3W8J!MI1>dw5c-22=|g$7kgfz z6%=-rX<^z`q?FDK_U> z7}L^e*|e>^WS(a))lhq?Ck#7%XH}`x`Wi(G*olL7nYxXQg#fQ76UU!*>cwkAirL>Jd>HPw^bhYz?Vh#jtqqpQOn2g}4Kj2L@?Xs8c{)#PC-ncM~sA z$4hXtoP3DEQr#h%vKG7n+<_9hoAl*U%!Nb22an}8snvr>FOmSxK+H>1V(P5)t0b{K z%;_?$4!WOF4Jm~zb7K?WOoZBPv1%1(nuw$mE|hPN3^UEn_&7JK0(T)2VB1$|G=h z1MWXa)uMufpRO4Ag49y2NUY$wJB~cS>G6AwI8ys~-Hc*7=}x+7RadP*^wg*v;Q5*g zUW5f#mL2o-g3}a1w3F&5@b=;ydz@Ow&*l0~SvunxHnsYz8nIHK!6)Qn=4a}3wpOZr zK*Z$zP-i#`A(BI*&t0fs)kAf2jWeX^$r6E-keUZ?!@FVRZdA z%g7?kXX0)x8U+Ni0~If@6I+E@_VoPvKk0SplLfcG&R|ZLb~lUC;e9LQ2sWloz_cDTF`o62B zUTb_4GcVD;#UYPqo~uCHLxbCr1g)vJrt2FQEpBCx?EU7D~b`puXe~5XUFx2WQ#@0K2^Gie4 zEl1kSMq{wiw@-GHsIKH?rG5eth)q}8e9{{RX2gc^p>R)WW4 zPlMkOF}T&O`dg!wr>e)|`!INz?Yi~9-+X3yL5NXc! z9+jhf?WV=gAg>@9lK%j++SI?+ro}{LC3D=Ie{9a~s?}~N&WMAuk~#e%`tG4ZQZZ5b zXUEz)X1IQxsnpY`TJ<)gInGrFVdt6Ebn8~@k?L#Z?$pM#Z(kmScS~?YFyX`9G}+A0!FAswn!QSUZ61-$>P+FQ-{R4MQvACDh=7^EWv^Sh#*URUsEUvd6rjWJYbL zb9c{-Ms@mHbmaPWI}Gp)%{8+Ea7HJ2+m$jkY>pU_v<_nS>n+qU!S;_5Dbl^IgOUCw z-%aVXBCFm&?}47vOo!6?Et>XIVA_qIbHD?+GpH5x?ww+3LJ|BcwK*m^`#VKtj_eAp zM_Spu@&59fy*iOjpo6&zs)BRQd%~8DdQG}#)M`RgQUmGLfwX6~eoR5SkNrbZl<7b! zpy7rHKR*%9w2NgMYki2SRe?M!A0}ebx_uUzEDLAuhq-^$-#=*?8@AV|(R8vjx7f?H zoxwm2^OKTdU$dIT)alx-LE1SWk-_FErRti_s<(=l36D*3J8_%{eK%2VkEF7Sl+do{ z35@Q~oS&RY7X6{u`h7;Rr)`yn8vC?s`u_ zT@Hu z>{P3+2iu91^zArV+5ujnHV`@U0{tIPjGRLrdG>PQzfL#Oe+p09R4E6MJ0HMll{WiK z%GGFs*jz6Yq~^x-=Xh*oqB5u?aV#25u5*tNY}`(pa`d@*<$gq5vCNR%i6g4~!j42$ z4r2;cY`KKP13uiB+E-_CVD`ZQJDm+D##0oB;ZJ{44l9hp5A_7 zgmc6RGFkH}n5BTsHE1&go#P4GL2$BXrAtXIx=^&CXh)5(A1cCa< zA&KmAev+Yz_Vea=+hOB*!I?XWAynbBpQQ0CkISFDq>v@0w49#!{REGsf!;~v=d{1>K#2(+s_rtV;c1{pbkPH`DQffOb%AgqSZ5H=Vh@7^x) zhvo!{5V6devoZn%xiBQpzI#lqNQ;v4>Iq-ln_Fpg;T@;g$=}l0 zu~y8C62)P|aAEXCN9L15S8aekNid62jY&{I0%HC;<1ewb&uY@Fpq0mYiC0mvH5--i z#Pc`rwQQ|w9ghUe9d4UVr_=!KPDdaakG)26FVwka%61OYH_(^poY4ZH)7 z=Q_@pUNjZ^M>#zAm>Se3+Zp|*`_6aUjbv*mKu;gL6DLtcmVQ!IaD4WUz56j(R@!;b z%v)CMLi4qJrhCLP3aGe-e39qiMomJ^D5!$R!TLfjm?n6Cv+Jei=)rkgbP zY+c(4NaIQI~?~AM%FeJQdqxnI!3Xl8isv8rGbERxOfvUOum)=lVy1Qsx7)ky1mtr$jHx_3pCcHNKqK~^*E2|izpik?gN!8jQyvz z7dG4!V_(OC9j!2|cX72&;yz4&rB(v^Wp>@%=k@;pc&*-@kNdH-IocVx!TF5o8Fc>u z_g^dy2?IacH_+E@TIe*RmczS{aC`wJTh-KVLUcLMh4ayP26XgTE7`{Z+59vd2bLJ*0WBlQB&<+8|0qke`tlehP0i; zs9+8cAFQT3^t|=$3DlG@-_o1kN9rfHl03}@cu<#(+pE(>m&Qb!P~Da7V*dbG-%qif zTANQe<1(^T=06Cc(yTtxmg$;N+M*dXCWx7>e-daq~IZrnXj{c^0nlaod>b zwQu(*!w5IfE~1^u+(g3Jra59`+UdEY*FWUvB5&yWh-wa>QbAMX&Nhuht$>a{S+z!_ zT(~?mU^lJMdMDP?H?!#t()x>6UD&SiAEh5)}nK)+WNG2r~j>Kb*^YLKFa56=-h zbl2&$y-!TnDJgYhhF>CL*G*ft>NU0wbFhqNMvtl;F*UM&r?`%F4J!2(r&UlTe88-& z(?*iPRflsAV8(Wzh??26U#4KniCX$ZN@)Rmm#EpCl2?=z3!%+D(kgEDMeK>^>$Ip%m)vP(56s zA~;+0Jw#KQmiX%R!`hN)9o5( zQk5KT3P>%S8P7S>c-=Ycz@R8>n+YHff#NG^U_sA$rLJGz4Yd-Xm~Z&W@iXihTu%;D zRqDp>HFPb$%F7U6x6WhTO82VQsdzZsjLve6MUi0T^0 zvnP;S07e%_W%h)@G(YpS);X*Sy9+E>TxJL$sq_fP28z09^( zfQPYcZ~I0TfHk8{sRp{2n*%o=&-9IHbbUc^)}5%`!dgG9qjbBice=Or&Q4W4f#WB< z(9@u?>(uG@Pynk=HiDzD$2geXQ&iP;S}JRCS~|I5*pHbM>aL-#t=G}G9&2HYjCPdJ z>NM8AqMNHFp93s=DDWn0QL)xE=cre6Q+ffn+}Y;<;PK?lI+e7xdJa^TPi}rb@l99b zI%ctG&idPm+?;|wB=t}piuIZPAPD+m8zX4-H)w9KetO!^Q*y8z{K{S`)iJkGsv-XX zGOc8jpB<({>#JJNJ6VtA!5AMCMRnD^NGRLuW0&2?A5WQWK=dfyuG?=S!|uuMWBQsZ zwzp=^7qKTLzPbMZ5ofp6K##RHo-jNA0KA<(qO)OCR%It~4d)&}nXN|0Md@0NZR=K* z23vxBPo-&kCYz#Z_g7Or+j+f*pE&;jo+k^U>(_Mqsq~)}yYf=LLGk|pIG;t+YIUPY zW%XDsp;sF*+v)Q$l-z1{Rnu!LYOO{9x54o#_$@W=9W%LxK_4+4W|5l9u1c9(D#1en z7wCWI5;|SB`nHP!L0LEr-#-K9c(caK+t-d$O>}#7?gLccuZe9%*4tYUxxn!v-k{dj z3uAwb`I*P4*-M6C0+ZZMIY(@6*RIAM=g1M0i|gQV2HL>&sdWVa&LL@3wwoJD*&t_Y zN^W{LSE|gvrPvs^kF?FVPi1)oPMW<+x@~13_U1vVyALWeJK@i+tA<<>_Z*X>FC<7_ z;&fV`m0gNT{{Sh2Z6=#and!$llC%*FJO2uv9 zrb$f8sK}SZfks?G75YZqU`qWZVhuzx4kcA#R7ND?LE;dOARNgpx>y*#?q!3CAxVQX z+sDkIX(hpuTZlc6Dj^7v1d!Es5vA-R^^9UL zJ4q#=nh?q~kR(bKh9X8JfeFuOMOkpaNI;xP0z?g?h|`F*f*_-g@}Xgvca_@$pb|v5 z5<*K!<>bSt;s`D!!IBnHr?+5AlbIv|%pwlbgS;Z1A{WGiJ)R{Qq*6?-WeJ`J2`vXK zWk-1!G9N|JvNC=nhM!u%>P}cd>1p2Eh#N^`H{l&?1du-S zA5)@dz0G)R38vjhJwaW2`HA1EL|M%)BHLHW}0|)$xpbn=%6HWz%m0SWF z#$xnnF;f*8axt0Fbqf(RX;}Q52OY*flLaqoDm5bxPD2sQ+^SONWohZ|{oolqc9}Q3 z*RD3!8zc?NM+RtGyH`2M@_uFX+IF{jae@#w4+eC?>b2~(dZfxBU8TlG3}H07jYUps zWo|%R9!w6qOWYvO@h;KKV?nAVcD542#N_e2iMy|+<@zq&H&T@p)K*cMvK*Y_=^R== z_{&hqZM>1@HZEOK?R8WC0Esi!ROwo>+Nc0IPbL>5qJqPvsCZyV?e6MYgWua$sQs%=*X*uA%k+G zxSIMZ1=R)K4S*bDhW>M&SGQ?3G`-EH)T2;pRb2MQb2(nG(rwc(sbB9aJe&*_KUucy z^=g)q95EcQ10elClKPU^^{N(DeXa68Nn@49>|7;cQ>v}FIad0^SEwrdG|SX!$!*BM zN-d9Hw9mJ_KRu@H)K#mmo*HG@qrXVomALUdnwLy-92$hRaMIc>UbH%eCk9he3#^Ss ziC>yY4IJtHAaTU086`U!Z$ zvNYWEI<`qs;%jO3V75WXF*};EL0zI{rnAwv9mIAWHmgpBJ!-gbGd`QZemFViGo-y= zP86TkE2^y!?_S5eLqV8oG}t_u&)$72TBjeZ(9`O6wll={Rno4ZQrgG|JpCbiAoTY1 z(UFgF!GqOQY)tLr=2)`Sfd_fxZsiJlOX$XJ#k9Ao4o7(9dek;4P*unEgVyXeDaUb% znQI!gQybCKtzA#5fP?p%DX0tr+B1&Ea=I;G$R9shsadeTGH`ygQtdSLE*mx>Qr=`MZ3H!;h z&6h2XcpMTB^@d)IE3IAX=Wqa=@Wc#{q|C6^wScF0p8br})gl@l2su;ZzD&w>MP^W^ z1Hm)J%Qaq{FG1}hvQJ~$Qk$7Sgs>hVz-B^63OK}C;%YqbXE}Rrwe;7Fk+dE?-%&Jc zyH}`PsH6Zqk-(g%;u%07;Eyo1>#hfI1MBsjW1VhUsMb?yKbQlJyRv_j)2pv)Ytwp4 z1srz>GxyBolT&-J+Sp;{dZf1Sabq5Qo+28Jso$()>Jka}I~wCnkxZS2Lu*uexFEvFsn}Do0XQ& zf-y8SkEZl1+~Ae=;w97)y+*AQPI8p;%sY3GBwpjIu_}LNcqH}{bu09`cc?;TIaf?$ z?Zg@BNV+$+mI1N2ao^+^uTNX`ZYf3vklY;q07x~q$4Rv6?OQJc(iuCJ%n_eB{RHgv zoo3#S_0;I?X6>LaBE@dlIryAa`YSXfl|n1D6T84a`;6%Nh2FI;MU7nmGBU30d}0f4 zvuL$+(M1|}(4S4z7PR0H$GmdxpVMB`Vz5i|%LCk>r1{35_JeIgzh7$v4eA^-7Vbt5 z`JG2d>fIXi__fwub?uF)P6EC@hk^d^vR`d^S?Q_&0748#qRmRK!wr`63|Pd^O8G$ztVGd{?xxw)w18G->0FsCAQX~Tj)kP z6Elw=!}4YN>zd!$`#}QA(MCZiNx}Q}mAj^+(4$6EXc)qtd-u%eHJ`Psrmyi8(3RC@ z1L?x$Tj%NeiAp-&mbOFFw=Hjr8oMpwN06ld0BB{!jp?b$l)KwkQ)S(XkQ0tXoh{pm z4Z)hWdVNEuD}c#iKzYH)`s1|a@4nq7!95m5GJww$B(XS>44{`lNs<{mLWx?_ zv@l!MGba);1gjYe2{QY030g?4T5$=C31t`pd5#1?W2bd+d8Ew#DO93RlWn@T_ zX;ENZD*$;jHJs##N|=Iv($7gNybdOPS8eVIrsRObBjO8Jp;uOF%P2nvH={+*xPgzD z&b)=I*nfs_3T)AYbqR_r?%rcy8?@!a_-&YKd06H z8&jsKFuNBdP8K?64E3c-*fJm`OvBXZUdmawg(RF7@BF7rtI{1^ps>P^cg)STmc_L( zhwh;H6E0QH>Gb}R?JNPtcr!a&U4{PucVS?D4rZmbHFa-x9^=69AvBa}>aW!#5>8y> zx6)~sH%6+~wdfTCfx$iIpGoyAz3Ox!PavMzj%!m&z*qnTb}Qay>H3E4Szk{9NFc9| zyivZUc9&2N0OLMBlb+PA)YfY8!3X`PQrcmtxe~+9PIL70GkTt#EUcpVIB8>h6^U1@ z>Z(-xjtdVHN|Z|N2RJM?{oriIlBLh)Mgty5`NXxh5`l2cJRbZ`yEv(1o$gsms-3w$ zC27)DShBFl8;M-z2 z$6$Nn39DwLdV?PA*~fUQXrJDNhdh9M$6Hr_NYra8w2IdNMe5wWh+q$c3HW6X=rw3m zoE~wHu6#;9lvi-9+CnKA&m>N=Pp2zx+I=dL+lzp3ayaoBXBF5{7YaN4XZ%6v+}%k< zKJ^?QoOp`rmB!};><8VCtml;7b(dJ}&{R_8^;m+eyEx=X)0b-vHaez1hqv;X{U+`7 z`fY-Z-*1E9;vZc_x`riK23!&Kq zi+{XAo}enTYtqWcgSh$o%}qYMu*H=tW5Or*GA?Ufqk1w|EWR z^$v3s*7Vx7DVmNm@MqoA^OCunE&+}VZ>I}g!5xfKQ=zh(w&3SK6Kh8G=_mt&*qNLA zJsqVhp?5eNz998|L5hV@h@HBC=OE?*?b>PDwdyItkPPJzJvMLn zn)+>DY{pTbF5RZL-5#${E~5euS`yH^haOB_lS#8o zxj#`Twg*rE!wH?84Q*k8C%=iCuTBG={$`y{r`p1L{U~rpaSPdEQE65j&WlB>E}L`6 zm^bMfmpJVSrM;@N8NnW9E{mz6)wOoMIr72+?wlDJ>QYGV1X8VD?&k;OM|Aqj6~ICW z@(-D<>}gFEtV=HP89CeLZ0`I9Jv)@;;J9>1TT^-C0ANV3>9c z*VeR!C-{n)8j4@i7Hz{a@7g@)%mrZA`BUAD4q;N)w$vr9I2ge_&-@b@?cdz(VV*?V za`#uTjn*J6HxC3%i|H?ofmJb*#Nk2vndRl3y%=7UR`!vc6WR||UO+waGYgv1y`t1& z*gONr^oDJwj5Y{A6VK|pdt$IB7#T9TE7-#nIUsnIwJ@O0^10YHmes%?#q%^Y)~$QC zuu}g3GZvxm;LUmp+sCv-47!D??S!9C$j5nmY&NM)Ff2$Xjz=FN5FI)IzL;DPq(=V$ z6Q|NpPMuUK?f~uc_kk^^ZBSbD>8B#r8@;^o52WeZ*6Leqdekp~ag*>qu-ZPMQ&NRk z8%Wx~_aov7dXZLo*>`~5{E~Q*wC=A@R+CV+>KKqvsZzKEeI?S}GNp}M1{ml603dt+ z0AUx_E~izbHM2NY7y{JSXTIxm~})bIDlH=|MEh zt3Vfc-GRX$NL_o;R*_P~uA^5fyKp~3Ag}7$Yt<>MA*|yAoxf~BY{sKq>d=~c!xgXs zNy$0#e(;?$Q&l>1Q#e-u?PKO=X|#=^S}wTd&OrwwXT;I!Dwb&y(5Bm3y}+pF81o{I zoV99oM$)YHSNuxA0sj90Y1rtr{W_6a(zKkRE&MDA>`y%ToHm~0q0!Qnwix3q4+G?w z-CnKwqUn1CDl@d_apxFm#YQiQc0&v&ThLmXp_e@Vn@tW74p*2VR5gjVdt zusP??>6xq6ts|%CnSd-U+Q4zfJLBWD&ev)c^lhj`1%U64cYR=LwCWuYsI&rq{#AgS zaoZv@(fW>}okfjBfo&=j`!Klp6C+K1I*lKuu2#Z5#GGK_6271kfKU%r7Y*bHr(x4S zX!UvyuT^jA6n4W54iD5$zpYWGZ4uSiSTADmFmsXQW5oH6g>2MaRrT2EYJjZ6jr`~Q z&i??U>yEQYewS2SZGEMQ$;WTfa^%KX>&KPHCqBm4E&)(#P&Na}A2ID7ukgWJ9y8uw zU7=H9vbJrkLf5$E*pB$ei3y=&U{)L+)8}Q5tLbBw<4JXktE;y$7Z_p;cTb@-jE^$b zX*nc}&qoG*YP`Joceu^Ox~Z3H_OTe2RmM1;t-On(@Hm&-E{njIE5txplbPcxV3*x2 z3>bw`#8bp(2E_9i#H%hukrr7|-Z1F`%Y!75E*c9sk{U>G(sw{+SQw#mDDpFj8O#t& zGbxC0xiZ49q>w5&mjWDB^p+KUB#RLS!aQpFK)qi`BD2O6)%2bfpNSv}6C_ekc-mE* zaSkc^Nf7itl06@!(e!?jtyj`XpXq%ky(94*^vA^arSy^|=^uzhOh$b%@gSfPNK8x0 zXA{NYQpQM?23<^$ia3%)@}4A^pcrysu@1wNDTyQjB#9zP?Ia71APD9#{h%AXLb#sW zBpGO+_WdWZ^n}Fm5FrsDfiA*O$vgzQiBM(4lKH@ccH&u}$bwULOt6qvR2)eRrWH(G zOe&txMgh!*?JBZb1yW~&VnLPjB4HaQGMvOv@*~yC4XjoU#pAR*s|<*ybFaDxq;c1t z&?PzTGyzoR948aXy!M4v;z>=ARwQgF_=9@$nIa6l&jpN{x$QmaeISd0Ji~}%SrVmW zOE5W+w8l2)#PCQUl_kYz9iak4D&xdaai?R)j0)wTV@iHx0bRy<`A`~x_>Hf}qTz|e#oowzfe>VZ*)6}Jw;KhiF= z`kH%r2jq5%bvoX?LghvQ?>XOYRx(Y%DfNg09Fy4oksI|mAR_{Yf;-}AYjmpCIb~g= zwq`b`rdmU~7Qycr#WkhV`m~F0Y$5&*+(^}WR8$hQ3yye;J8G7-qufXt3)_h%twZIk zLJUI5sA8{TI4rpPkEB_(7Jel9o%|n}Dos`ZwUDRtePc&VmZ)W72TFVvYGn`;@fEX4>D=vQp3rgw z4#sO6rgFyLM|wtut3|c!RYe4bBN>Y8oe!!i8&vYS@^jzmH7nF0p_R^kpfAv3V{6Ki z6%^&N&%4Au(|X5CbkZnx-nnlLnWy-g_JH;P;A7p!XF2Jc6WR#K$Oa9AP%ua<=X7ox zqa5_W1p^kEHug&omt>uwjy7>Z!F@?L&ywdiyFkas1YiltBk=icK)|?aVfqPGE z%8M~FPgLnr;wgTh9Y+RL+?3)5x*gkb?E-MvT`*zv?5+ZXf+E$Z9@1qjCa2t|5>48| zvTS3Ytgq!89C6^pscWr4?L7I!(iX0ESULHc1xqb10XakBO`1_*^(r{bHM(HRNQ$f3 zLv3z843ew}uaSlxA$0nc9@dn04opVv^*AH3k}<4dyAdR)yY2&l8%k`{?c4|v=cPp< zRA2}uY1+2{GnS88nnJ$}K4x>P9ZI%C4EUNhvGj0x$dWTgs9RA}orBC`iVIeEV2mJ8fNC1O z-VQj8C7zJnM{)UnVyahr6=t`dbI04?~q~Dd!l+%(>Foqa;PowSn>s=`P&$rkEC* z)Zh?)Om?je+O$=62pf`d(0B0G2vrDhk)M*wft~m_eeu8OJr(T!SeLTH`85RzC`+X!;8dlfn z-}jCwVb79J#v|HppP|tz8i-4)L5FY3=gxkxc6A@c=)-2FU@%ycq<<-$N@}%wimCNg zPX7SUo)6MWv~|5pN9cia5Urlg*#7{}5Pc_Dn%7Fn457+{{{V;_MRh$e{yEoPYkJi| zuVKI=oXcxdPo=iIX>-Oo9)9pgx`vvqbq!5y5H@f;AFuR<>6#Ov>2;QJUAe$ADX?hh z^oTZC!#N$nJ;&M~Q>gU1U5?=Hv}{m0<2|Nr(^3P}^_Ocs4LWRLNFcK?$tV8+#L8Vm z@AqL*MsUYFqS|q78j;Reu17e}C!gsrtV@C$>nCEJp?mikG9ha(8b)8kVV=hvfPvPF&e-(Z}#pg6e{-uGpf`rHmht>wg9^~5_9trTiOd1ZFHr8 zC_TZO10NvGo}2qb)%9Q$C`Q7$9PeJ;{LUXj+M?{=ky!R)gE#BCgw&AiLlz9Bd-K5% zu}b?Mm!atOD%zVh1S=iELlQ=N&mY<4rtBq~keuyi$s%(-E2>qY&{b`^s+`sjsH_=F zf4Ti8wu|8!U*?!UZn3BmsWpA((p#=1^iqMEAp6`p@oi*ON;2hklZ=k@&CAE*+=g@2*%sCKjyu>KHn92-@ zBwk}Eyh~vlWEF)AJ+~erin2hxe0Gl&5Ke+gp38{BRs?5G@;O%w5#U0LPh3%fHMS zf!K=*z95nmNo-X(6WgT;#Gz?y2+Sp5%SsoL7__N)xw2PziW!?QXPHg;mIop!7?x2a z33XWqu{=b$S!_ucW%@ytjTOwM!E*-K%T@jl_FG#P$j|-i4`hC+{V#xB@r=*VFaAYG`1xg zC^sBIViCJRn1_Jo6-N?^IfYRa_9X3=0$2!b1?hbx`hRFy=%UYG)0n#_G4>gPHRSRn ziUcyjN5o^&6UMOw1~KAD62xLelh~4~3C1UcxEPV0jH;-cpqTDCGVIw+xRu9g!z;S< z9OiYbRU9&%hv#(-mtpyJ{<_oHv;D zmfd3)DtUvd*R36J86UVmGn3P=4ni)Nsm zO95MM;Aj0YTW!@?aDf$j62@-o^dK_@UugWt%t7?E<7$n{c_5zB4As+@VV8Uk{7c=6 z6LK>H_k?D#6_&BJbL32oMf!^^7^sj>#|CP4DVNqg^peSe{;WvT({8S`wC&rD$|aXd z$yWVVRG!Eq-87L?tzZM>6XHEGLaKn)vA`;$l44y>pGEg$lh}#1uc4_^aTrGKGu~rs zS;FH37~*Ri+eN4roz__AJ9tn$LKegRQmRKIGHSGM9^9WC$g!5ZhQKYHnb&)2EVuq1 z@T7j|_8{;KZr@JeLn|@y#7@eMm7=KJQ=hnzMyM4la`K~sJczZu42bop55OD^#~2YV zsKhXC!MpuOgZ<&|qbiOT$5Y9~Quk+L$X}9UJ&e1G=v!m93C?iEp>0aupcnzT_xLbN zQ&e)^MsxECsIdT07u-9X@IGemwiI0-rr9+yK5%`rO*fsPNl<-=2al5}L2_R!khsnV z2R|lj^!gIj7P1kzVRJdwmcAyNrje*k z!MprLEK8^ajFA$rSz-`D#CU2=$<7bVcFvPlsFm-`J2Kn?k&U_Uebj?336s}15x`fY+nXNWc~(sHqD zlFHAtjj{*MsaM(r7N=ANSW(G|8#a%V;7Y09yPLdHuemKoUeipsuoLHrlI&Sbe@$np zrd9qWnb}6BuTI4)@e1v|7dhRJ61tO>Mt_`3)~9d}+rg1BqjD8@ax;S~aw2j5rp|o8 zBeixK&PLPyAhk9#U%6ZV031eDOsPb_YZV2^W6XBaR=W1FT<186Q%Jh>egIRrgD&=x z*1K8ksd&Kr#GfCjmRV{EI}1?F>G&)?yNST&W;&AKa9H=pXu+|W@6y9Fxu<=~7Mt@Vt|r)4a0BlFKV@p;h&&)OK04smGkkrm^TA zg8;0jAok49)6(yA5o;=f0U%_}I&>>-i(#>1;#%n^NeHYO~ha!8x6O?6+Ffo}emA9g&cs1-!z=E05}WJG6aH+SC;x$SQnu z+D%NlSFYBq7Z@sk%I|yTwg~U?+p>L?dq;R$X z0qzv}!G$j0pxvm~QkGU3MxBc?SLBo6PS;ld0EpA2f~yF}xROC17!&3lCe3b_YgcZ- zeWyMKCvQ!+U(=z12qokr46+I;^2KN|PhxH9SOBRdG5*@>-ox~ku5fq@sh&?72&o{T@EA64bg zhWms{K{CfZJ~NDoMnGq$9HpC&K3Li8y73N&ID;#LETe$MwYShEGrQNO-4wp$9*9u(UL5q7=}|NlO3gTGf3;q zGNxkno?~^HC%rK$)ri0n+X3z)pg`0!+DE1IkD7!VpdKZn&9=ODi?HTBd_~S6kR^eE z-HBk*BMeJmK?5zj0!ebB0@Rl_N|pr6Xq|{N09HEc34ml7l2JzxgNa;E6D6mZ!IM1g z1Y4Ocub4J6cS;!q6Dkb0+{+5gQi2f#y3~Fl1!P1j77z&`ev{Hm0WKC*Wx5ThEB>r3(Y~g!K5*vdtlde4)OG^xqGBviTyXl-wY$v11 zoZnMoxyQ`&<$HSg=S>Tqh0=^pQ(bvm2(0ln5}PfjC5+A26{Sk|nRT&|_@}DT&Pm|& zI^L1g{fnl*o-?0`$=_{hVvqqi^Wqw-O{i@~%LCjV#x^b#^K)+o}8>JcCZ@3Vn`!`W?9tKUw0Ytp9VW+DOh4Z zDDDp;WNPbHB<(mqln&Ag>viU}190EqcZyTC#ocABPIK)riPMeN2O)5L_lAC&)SRLB zd`(S#GP2o2E_VWRj!sM-n^?7V8kJACklROW`HiJ?3e{Z#wmsiSH5zIIdP+0$WCiMo zZgoNAgZGQ;saq*+%Z@vIL#wUhKMHU@vKr7SDoJiVyT7!~`!02=YEW31VmqmVFZhrg zf&n-Rz4J6~sXcYcAy0gG65@*V%*qqr<%!Jg=QOlRg|d^D;DeApAhnHAtXp0;gU%yb zgmTX< z^M&lnnCoi-w zdTWr3Ve>=r)>G82?nCYnBId25zx4e@RtMT}OunZ>yp4;8#!P-#JSY|GcVILlvK$pW z#aB*J&8#_ywoxG8)h@wR=9ZH#WR!N1zf3;9?Dicn5(Qdl|+Wv$sAV?9peM z^Y$_FS_-?_8w_}iJ%Q`v1jL@JP@U*;8>vduky$y9kkD4?Zd6w&cE#+w+psP^xtm&N zYY+t)tV`GLF&O&`Hg4-ki&ptoJbm+&Ci;l(|bo4&qw}wTB^7frI42Re2SP z+#LMO{bi+!5;=uJN+`*|?Jbk6H*RwHBzX`*#j${6!sD4$-D6<4^D@G%sIE`9<1vV1 zVxkba!=Co5FT!92^bo+3q`H zG`cPJD|d2E0U5~nfczDH+>_g!4o}iMPMvGW1nfBAaXi^%_G5<0S)#c$C5rGQE|JJ~ zurMQ&nYfI_ftoD<*bF*BB`C~}C~lWt9fW9~kawmHilO!)k7LJvx&7MFKdY%%OH^Ba9; zmcjUaK9ybAkQj}}@5}`YqSRN7tgFs&@HPG$OKR-g$WI^`?Tt3B*W07hucQ%nDN$KH zfgQ}v((5c`+gL5~+qg2e+M1`y2$e58N6+2@-Evf(ofs{g4@cb&`*K*1nKY}YBFb}$vG$!{NF{{RHX z)@p2ipXzDvuiC)A@6CbQAx@y$8C>HhBu_4RT%OK6CN2F&xv$e&X(ST3-GR!>be&`A z6*AZcWa;$*5vp^Yr-Sr?t%W+F?;vL)dF*$U+iErB)+|6tZ!mOqioXyQ+XKuMTB`5R zY*c3Ys>~n!rE@sFI^LCmxFJ#9jxuKUi>NWE zEr;x9emwHU$CHm7q31)MB>*79qlX!i%RTtr&Buc*b#+x=%OuDgNn^xWF5(Xm>@z90 zK`1*$QB0*to?|GHPb!>C?K0p2ECi5YWrD(l%pk1%Nfc^{97-BmJ?YO1rJ=G(Sq&LM zjAj$j0!A?{Ar$t7Q2?h9s?kdk7qoyOo%o4eXMC7iOI*X4jhvW!UnU0-L1US962WEF zWuvqOki!BUSLrJCoJeeoxy-0@3uR;yV78pYMSRK=Er~`EN|T5;X>ufNCQ8sxCRfuf z8m^tq+BaU9W3+=gX)=8S@imuDvfqau1XCna^p{%9(mHRXp`#um3N3$VK%97lHRHta zA4woxAc(<@mCuMKtjPhKOKy@-$Ouf5HyP~0EhP7r^s(YdE@y!-zMf~jE+mxSEE-g% z4>B<#F6kSj1t+w)t&fNWH&0~3dPm|&tLY+zCKMT6pAy)_l4KQ=oJ$U65PN0SaV@q; z5(xrH6Vf+GgzFcvF&iX6*ajqu;|38M5~On_d%&1e+7uW7WQLL+S7e1@2nfVLM!7Ojy?O5NV zduu8gPoEQ}E_r%;OQ{4R)Wp7XiPmT=I+eq&2wY}-?u|>0K`i@#=L#`8&Y!7$)IR;G zjF3)YmOpM*oz_w^SF-$k#AwmAwpgvSARgJbelgO{%Sc3C3Cy@)UA-@?zGr*|Zw>Fd4@@N*gsE zo*Jge?TBk-I)pnK-z9K9rXZS^byW?tws0}tYwQ=)Djc@n{{SW=-5{-u&WCGo2=Ohm>lFB9>PP}itwk2mrkWvV6f+v z8NnhHTfM+`*bkFD_ZsgmeOxY{l5{KdKQ+E#Y;D~H^lOV&RTN$a(R3s2y~?N zGL?u=`brRYApTQFQK+u7#9UuyrFO3}`7%BbpZzmRrzJ=7o^39h{USV?ER4Gx{ot(8 zDSymm=U0p(`YkiJzNAjA~=+rmnhiy1Pk<7g4xW_eT?B1H^+*AU7l8gH# zSZ&!ep)a(Nw96o78RSh)T&LLh@vd!uWOSjki57mz(BK2&blKWv($ILser(6Zi%vOM5r%=_a9QKkl+3hc%HSyx+TlQkJMmQkEt^Jm5xKco!i&pjam)OJu ziK+b=!FaK*65rX2O&ddgu`O3X)`~@SC$xE=Ng86{=32gwjb5sY+vVmzWoyh8<5QB zLq*|ScQ~CtP||BOnt~ZJ2b@6aGzqBFDgGncda=)ay!Q^+I@(!CTz2GQMHgdeJ-p46 z(=M}00bs>O&_L`WW2VJhDrDo$)b*K_llK33_NIRXuW0X*10EvH5E4OI_ zIqX3>2kY8-a(m`qh@)7g!4`zw&RpObu9rOsjH+^Zai8Uhk!Ma8>@|SA9zS3G%azn< zMRn>>n(@M~K734A=`#IqSFWz~!}0J>82iEM^n0vkRly^IzG9kN#hI)jn09zX*l=b!hEC#hCI2-Ww&;Pcyz_xnv$Uj)UcD)|rtbL)j zj!5|?Xw$b>P*H(lT#x-i^tLCir5M&QM+~6yKGQQ>(~Z@)qH^Tt7{NdLhSXK)*}(x^ z;2qe(`$N}e(&*K)n!wySJ^adFwU#@MX%fV8zTiBZV;^`ksIeDnC>S^-Kvx z7v3|HfbdMn)>pRHM@^ud&CJ{~;$?c)xWm2__#Pmws=|x67%k>$eiF2>Z3jKTBZC6# zlZ~XX&Ss`;-ZCv#Jm=(zWJdAG`^T1(jz9zEQq4I$r+Inpj{>Hlk5#9YHMh(54SF(j34nC>3tKY)WwAfWr_Zxaz3~-FZL^~9XqvG0Ce{I2xFPq z{xhT|{{V8csl&4ZKS7bq;hq<6*kwM$zKS&(6Grd|<9Iw~a=Psfr$mg}f`BUFRenCe zPQyji`c}VC^wCMJ`v$OJMf#qU3oz9S8yihz@QS^~FFMM}_?0a!b@ z_ygzfF&%fNM@|-9TQO2V87(%bu9W@YscpBzIrfjVs?&a^v`UexsQ&=luN)}<04M5Z zhL5IeQpLW)`!Rx~V1Fr_7n~N~#p-nGi>R=X{*$#w@{^_06I!iBp6}ykW66zctGNnS zT5Ydr&IUstyihfiCdz=O^W9q}8mDlj-9q=f(A0ZPXN++)^aC}7^-fPCF&lJ-^bdW> z>Yw5Wc@nxGS4(LZdw~3hwnS{`%Iq{3fXcgreMHh_oYzfdM-BDuI+bx?26o}gU00u< z9o?29KoQ?a5!y)8XT3QuUrq?61tY2l-?LbEQYej~z(VNekS!XJq& zxDnq>O1(o7469eXT&c{^x~}{{`eVp}E-GU^=otNIyhD0QW2DAv7#`EYq9(zmObgHf z&LE1^pm>KINGXoexyh0hOe&bTg;OLU<_)ntP9WIDA%_x$W#!cg@gNS_1p$m2jG;n1 z$pUHwO08V>j1fEpwv}3eg!G0b)JZWw8icQ=A&jY#0jkf5PeU=a88FJ35Jl5Y0*)p9 zFr&0S+z2*r5ibZ%(9G}L@Y^_CKGN13+FJU40wTPOlC&eB>;q841luYr}&b19&?FoGZBdVT0t3&Sj@;mjAzWovqo;!fc|HjlJ32nSjhuDU0LwKr=P3U zvjm(?W$4tRl2bF5>wPDOilQ`6aD91-=L{{UL9y<4A8AEaMNMZb(FOy93X zPR?H^?=&lyuS;cgkb|^A>$EE?H#waqy=$mpJI;GmPNO8?fPRy4iqEf7bdJo3q>+we zmS1L``rge#Z(s2PvI;*!J zxZ*W+VVsrt=5gI6wJl_T@yvT&TCOtSoa3?LW~J>~K3E)% zFlL!r{Y8`ydf*=>2{i!ZvHZgl9Iflv<-Cbj(+#{S9C;E;)n!DDt+a#l%y8Pr8(FsF zpLR1auG8uR16d2tiC~(Iq*(jcJQ60ndf3}l)wLDwZMG5Ie15)SqRWj|)NuQ8f^nQh z>UW~mWnwZiN75BTi@iaBI~*L#Xt~j|I)bAtc`zEFXe?2NLC!}rLG>oKU|3*|KHg?_ zsnv-pNm4oDZyVVrZ5BCJ2PEV{GPq)QHs1VrhH3?Yh|dQrj`B9Wc@|}GFjd#To^jfG zAlZ!-TgtZJH+=WU&L*4Ct+`JsIUYOA+G@35A&`PQsp4tXs#93@NnX)g^w!s2K#XbnlGq0#I4u?r1bj+lwC1e2(EF6;A4{wR;g)RK zqo&U57qg}>bOLK}Xb;9zUvFr_A1ZS)^_o_xOF?M0OpS&<*(T|nb1tlk}X9tM*FT&e4*Y$i(e*TGB?+aAt!9um&Hfop9#M7d-Hl z^6f^B%>#rQa@c~qL#NbGfB4}X_Y>(ljSUxqu7YH0e`b;FFGKV2C#Md5`*FNle1~6M zntPg(nQ%R7&U^NsQtKbG_T58=LBAXkIE3gep|PyGz3e$qM-$tJEVHMZ zE;}hsp4zv#Hp?(ifJifUsbQz5L|04 zSG?;k`mAeAlcEn|xlhfWag6@3H|Vw$!Q7#CmCs?|&05l%aMf4=9k(9wuR^nvWq=$7 z+EjD;!!9Xfc=FiGuKxhTtx8ii5(6n6hy2Gey=@k1Xs^*54=CHQ+rM+#I2F+dTI&FC z2wp_K9^E*vZbzNNAKG!e7CYnp61ihj>20lmDpZ^tclsZRxR$GF>U`sKV2t)K@1{eQ zcLSV-2ehfsyH?1!!RJ3&%PuUPapuR<)wiwtw;Y^;f3&8ugtojpG4|n=G1w0#U6t2P zpw{b-8;|6Fyf3Mf>IEjUF!=7XmaML_%H}Lk&+bP7I4C>goxWIL0zk07#Q9gIEWo77b^xL(h4{$$htk+(VV zGU=mJrmR>AP^Tmj!RNQ%{pPA_mFnB6t<@M~c*E&~892}qJiIx8VX58R> z81A0HnXyP$1D~wwi%rzKb%9DEt4(t4$%PzI*ZX$LlLuW|Yhyv7Rp7h7wm z)2OHn=aOcVUqyPev;kPbVtL8L3%jqV^!HS{h8lfBFSV5LAhNxsp(?6-=BSC z^26yqT$;jdpzZ^1K?gV>cAGjKHJv)cu1qSEk+r!|F_fBb8)bL8og%Bg>eaA#M)CB| zgA=QwjZyfyl(c)TWeb6xmxdo=^;pqISAorCp$uEyG6mLy<{4_oh)@T_l46i57?b#X&nw{b2rXPh0M%^g5`^L~>=kry(N#;6l3=mQOs>M}%j zrQ@_1(~x^ZC@LcYN(^>@u>>;`Wigp;7AZWaa~l-{;t?p%i6ZGqI1uAXaU8C&pifxJ zq9}`Lxa|aV=i)JDpbqdR;LzMU1Jl~j%h*pIWz=?-#+7c+N}HNBw<1|90OlzbRg4oZ z*d3+M0xffWVQP`)VFp?Ot>#nJIqx&Jv^!Wq;!SoE+`yu20H_*iQ?i=R%H5Q z$6+Amc%ULkLcQe00O|x@z&m0#;sB;kXd=FtJ%sn+{*kd*9w&|jw3+Qu2eiF(Zt)xl z3wDJ{z>+F7z<819fO|(91GEc%X&_BNh$f(u90+B$Ni+Wd4=QQLiKVg((~~66^!E6c z+R|qG(1D>^*y2%y*|;BCoSqd9zE%8EnCJrn3lSysq#h$ z>|@DqqWz|RyMC&|cjFw+7rON0G(z64^b(AF_JG(d(>>PrBQqzdSm#uWUsMa$(G+w zwSXY`^E7lXV-(wJC4lfj?mjU!X58g>S{+vUg0~=!{{SK#sMs37MnK?v&RavOxoT&& zm%unf&JXD~>D9H%=@3E2K{L&sI%-|Bb(@KBK;&`6$kcnr+$cEYf!=f%Q|q0i5C?OP z(=P2G77$PLn3YyFX?1sR&j;c^Ua$hH;Nv(Uj}mNTq?K;cgVT0mhj!7Kz3e@y)uu}T z0*{U&ln5!WP>q`vLmcfr<29rr*Za`Jk&H`Nqtq;;1-p-! zCAc6etZ)w*Ip#|{DF-=H2>Z;IX0$bA^MUc=TBGV$x$sZGJ>wT3y0S%N1NF?!^lVEC z$nmK5~HTD{wU9;K;X(fwjknocE*OML&q)sV=JXZ9l4n&sX87> zmR_tF;FB$#G;Pz?z=jcHqO1KP5Amb@P_^r8bAow?m0D2r`dhloCrzVp_J~t@NnJt{ zjK|821$Rr+9Qz{Q^sPUInt@ipWm`;tgv)7(Z%3@20Gax(n$#}j@j6!1+&dXRg%&0s z%z924dWE`sFmgV#Z$s8Y@X`B!XQ;LER7IQiX;qJ=aDI@~rc>+G>nq>-$0^pfm;BXon!WPo}inQ0CD0)FY0oHVSb6qM1PkhX(nf)AUi6Ah>C)c{iNz?T$F$|!bPn~N&XYJA(Dys&j zCnMs0E-q=5{{Xj(lZ#iTRr-UbwEAQU&miNpgx6f;vJglhapq`O(H-=RE-DBdn2ir! zk&2~W;lN@uJ(%)ur<&q&rg{`vslIOrxd<08C&Eam*7}WoOuOgk9wfK zxJI*SaC|GQHBXo%9w>B!Ch*AwSh>%=W^$?-YkcO+wNa^-N5=8Pr_8zGj_@mjN=n6 zmD1{ig*!k8+QT20?-}%x>VO+VcpdXIO53AhZB~O~)_qi73*h+uBN}~T-T*etnJQA=&C z?FG9`N^I$n)K{?iR{-D`y^2L8Hsus#ePdULs^zOKw1Ub{Z{_~-q5Y|D7t<%WKjL(i z!s1T5ML-KYjIY--+J##!R&vUx+qif?Slj6q+i=|BPq|J8AcmE>t(yu+BaF>T&BoSN zuCSyoILV&Uplg6^?ecz;OI4z>+KCmo$2lW0IyA<;M+6QJEx;q@cE>4W&z=@mjXEm- z02%6U+KiEx)Yi9zAPujC=Z~?OjccYo+mwu`&ob4}bt*l<2PBi?ZH+YV*Kbo;QG(bC z268dpw-Xm}r_hFvQ=}oNxvi;=ah`v#5r@=O z%{scWt~dd>Kd780llG@v_XRdH41cSzoi>ZFY4v}*vZNXAbGM(Lq$OHs>vX?@-F}IW zqFXLYXCG6ZKoRW^P*>BMY<Ax5e)VL9h#8@#h4{3VTGjk8=tk@#QnIWN`p}T=8>3txQt|XQQ zQ)WC*6(hD_l4N$3EW&$u^Bt`Ujm%{TiK}y%VCNE)Q!L;J7PBS{?rC78MLoDrD~#eV zV~|V;xPy~3Y13U8ZI)sT&aYH9!hR<`qqU)&^8(yvNO0a|m;xlA1+yuQ&=!p58Af4NphqGFo%nWXVWePliLA9K9i?0J?<6ApM3+;?ctNPh?;jzdoj?(GQd$sMx)!?k5OFQ72USGE8psi}uWXA7iEU_+ z9LWg1Bnv+J^uT*rI=qc8>wrZE@4v=vYJk!JwM+#A2MrN3sncq# z8wqnWbQdK{W3~Z3rlUG5q3P9VN!$T%CTHs{s?A~A2Hz7$U93Q@wD+9t&Wgl?pS0X^ zvPC+JR(~iN`a}Y-U!*0>805=_U4nFuSJ`zqY=90q=d^E1?2B%x(J(!u+CMjlwZvd! zw6j^dl&yWL4sZ{1^EAb2YIT=tZA3Q`E36h^c>7ImmH4J9bxgvv<7*860QsEddcR7w z42r=|pE2RqF1#%$O(?&VD{=n-xs0aNuGuwP!!!GiI9}09LfVW&0NwKyqW=I@7a1T9 z7{WC+Dtd(RINU61WePPOpapDL9+(ho86b>@D(!`O*jgbRNkaVR z#~y#2-016LXf!P8UDs_uL1i01r`By%QrdxV!=KE2z-bP4sr>mP5Sp!O(9MC!9_Btj zPcrg%#~CBWoG~f_sXs9KYE5ZwqXZ@f0}mOQ)N0o1?%S5Islg{F>nt_IqN|1|M=SCA zLv1@$)G+1{YI1PHh>aS$%E*0Pd#91e?X^{C$P+g01NwoI9d(u95YdYDb&zskiY|`g zo#%#@^f{W9-h>PZcCZnExr%50@GcHucUZ5HHIl9U)*WB;Fe52V#>a7GMM=&NcIfPYAbQ0d8fmsJ^yXtf5w2+!*_5w|ENYaF~;vRL`YR;OD= zpjYVFDf$V|>NClA-CyGLWHji^Tl6X=%KEmZJL;T*k~1=ZWpr*y>bcCkA~CdwDJTIag43 zAw6be^4v`gE~d8jyX}7j_L$n88AaZ>%KSmosyPbh+;|3VR%@oz>1ncq>ENE_gj-FW z7=Kf9@x~exMXD~FPqc{Hs=E*Zn=Au@e8fSk+pDY;d@$swz8Nu754yh&xqsO8FVhEQfjcJYq1C!IOM^m ztg7)*xGs{XU}KXkZZh6~T@gD%7g)QgS#T zaz5V^Dz!`OW2Y{zdK7CevFwC9qT zSYo}2vE5nypqJ5U`8A7HIn)+J2!#|$T&RAo~ZU&rz*$= zkBzQRIUXiIPS;(ev>WUsgMzv39qBdtrkht&n-^`UIb=QNby~4a{<>5>z!FKF4v$@> z>PYn10_T87%**PUC+XGt0~S`}*nTnp0Ombq4)kO3wc3exm4S8TmoryK)JCO3zYIJX zz%9?G)nF2*Jmz$I4SL#r+&_acH;rz|lx|+^wh;!`e8X1Wr5oG=K>l+u+HG=A%=#R8 z`A3I5bwsySwl1APOYQ#vF@0MRkW_=<%-UR=PUAaFWpcMpnyptNKSrQ2?=fgx)(k3N zv9X@fykpJO>FxlH(a%*%%ndfJc8tf9Qlxj8xY&0o;^Q(yW-1srm{d51h|q0`V%$Kx zVg)2-R8mRiLccKT{7kK&Q;!nCC5zxen+S*<6ibk%B-)UkWd^iMXGSfo9K_zEo*{n} z45M7I_LNG9Fr{z=3DhW*ZAd&uVzw`=5wRre79!-a%vHf2X66HJgw&~S^6#nvIGBlH zL*5GgcF}_$0p!rsYcO~-6Iaw*1Ot!O3hhyI#$whFQBG$}agm!#S*dGZA2U~{=*5e; z%vN2F4BF7^R)~NFyU3%hi)an~>8JacN`+qde!||z;qWo%~FsXQdGoPyF4atm+Uct=Ig+5Wg zTdy$QvVl4E>SN1Gop>YK(-vsP<7z-3`-$DH2RO-@jjSv&x_KLan0qFlQ<08hH{G5y zFLMQ6e9ZfGHLsbhc=j%GJldUy22^z|yl%`#Q>muRk~IOa6A?i@@i-yUQ1-Y_}la?6iR{*S?%b@i}V(?fXEr1OWlqD z#9ePeZW+b|KjLc~BIh$6PM;iDx4+-|NYq(PE0AKA>qWK-f{chAOM(C^oDY~>o1Ahg z-9gR1!|jgOt+CVc6NsjnGnVwmKCqT*zYGXvIX@8ctz0EK$R37w_?+#B8iLs+N0~PD zEkXU=dFQlM)G9TO{_Nz4RJ?Y)M>FcnVW63ZPW`vp!C;c1B~{J-=xU_azXC1dhH6RBo0S0 zJvy@%7$ff=9l7~A>$+ZQI{{Cr8JRZe{WmD%pPBSSr#_ZK4m|u!`?^JMrDjD<$DT~a zTynKDaOFnbrAu&+?iYtt%z2P!gcxBBhqnpm***XB#YanM(*gV|l`oev*A6 zyK+YF^qxd$*Sf*RbK~pVAI?Cedb@C0h~opXltCt)LIl_z!h2^Me)CeLO4$GztvnBy ztyEZ>@s$K%0sa2b{UsaS8I-1Y+Hi7XT`=~&V%0n81eEt2L9Ukj?)0v0xd4&}kCQa* z3q>g5HmUA2-bEMHsdB8_H;iXI!RU}JA4*D8BBOHH&UpqpKMkaBU{|#E^9)+cD?Xk! zt^puf3U=LVQd&|XbsP&d??!<)C?#^0gf7Sv1 zDF>`583tM#ZB;_40x}k{Ib+torVh@w*v8}h;iI=@SEIW-gezT2oM0c;aI4lVKg55W z^L1+h$zv@oyN|}x@g;S16NVFo{{XDFmN4{nP{bX^8e4Z8w5|cjklIRLmE%4{<*U~# z56|8|MzpR7c0h7QhJ|lnd8h(zM@+J@L^n6s7^7NRqe96ighc4gAvqp z{fE`B`b_%L%A1Bbnwou9_1fLU4wR%9tfhi;{b!}B+3KqDIWwlQkPt@`KdC8wIV0j~ zGg#UUXsm5uawkcqy)CR2Gl@X;>WFF5boK;s6w-Gj;LPr;qFsKG(2sZ-257oqXD}Hj zyw*8*-N9zx7wXz3)9Ng)Na!ceEqf6QD>&~0=e#I z*Wu;fFPq87<3p!bT^6D_`7$+Hbh9ORKOYd*Ri|qb+8bzZ1DsD-$zImn+3m#i%eN&o z8m6h(Kj&OdlS`?t#R+0=&mhiM)mkpX4{`Fs6*bq@@7!r)-6 z(bMan{@->UT1A@T+sh~)Bur(;GIqz7;}~sUrD>^7RYo^5=eK$1^qF63wm2n+AFnfD z*vnXTDTCht8G4qTVa2tEQ|FE+48Nc8-hWU1Iq(Ta+J%in+rP*?xQyxbiE=X7-r{zV z{-2H}#houqqZRRygNc;X8ns4)O6?;%)w|9a@Z4UGK53Bju}fO3>l;YHJ`Q||nRI4Ra`M_b_C)JSz2l|;8X^7?O=bT<0HN8m={ld19}XfCA|Ltc|U?( zq^)X;oD!oaGN(aqop4m4d^bFd{7cp?4f~`!dk{uf^5->hPtUZD`J^s`obivc4aJa(9^!^QOchOL2M z2Gwof&JgW8ap~H-KxJcr`o(_>T9va**DrQmn_7VKZ?fj;Ad-_#D^%U=`$XD<7{&RaltkLR4ewJvt z#&GS$gW!Y2g`19OC98O%)xg0Fg#KVb&If`be}h+cT{ew1C;(<^Z})TZ8$UC->K!ht zk(#$!>~Yv*lm7tY=45KWXDr&6X}8mTc@62D7XzKc2N>{V<#Dl)JzWW{E~4REj%vM1 zu=&Tw=^eYL*Qlke)3ZAU0>o@?U-y>K>o$7&JxztGO!{Kb9a#1p9^a%Er&Vr`SCrja z-aB{Xf93J@gBl8|FaH3kyVwau&g_Vey#nf&W2i6ZJ*Krzb=q`m3mX(<5OKI424{6@ z%YkB60-OW1=byX+^{?SE?InoGBmQ@rT`#C8zh#HoeV}K7`c6n{+O?))H8IZn1H|EAW7|NZj zdHvvAHjalFz(1t=SXYNQ=F-BH;6yIevcG&zk5kiv44{}A-4o>C zcAchjxn+%7j-h}((@RUQ+;O`TnoTw8_}hrm8)}bLnC$A-0E2fhpY;V_Gml=P>bT}f z*0KI0GjWaiS6%-AR8#y=i>Ft|lBuEiWTAjVX2IZFDAr*MCq z)YJ7Xz+}!>9Xd+aCRL9SVR>Uw%p0&*<{+Bkw;7bPQds9Rq{W+)O|>YaF$?vp z%ud};7|hLQce4P;T3c!UD=bLN^EhFg!P7%JgA4d(`Ih2~rjR}|BbDd|$?Y{x6lQ3O z*=q)7)20@yrTGG8ojtWSCrO;g}te<*Ef%OzLXaj?b*OtF_Fnboe2g*n92G7OzG1-sQRZ3kbB0oy0If4PG?x>SenmK z2?heH9l7rt>KCr~Z6RA10%@HAL2T_krkzzbGbW7zHtxrHx|%c;fsSW8lx|~LNQ?|> zirt8ejW7qaHA#m635>FuZyavLriMky5z3ct8c(db6@L^hiC&xsW5y?klRdrsTzdBR z;u}nbnQ|s4Q0v*d(_HrN4X^6Hx{4GH#$^8h(X}_~0~-d?aB(=jGf;wCsMK{(DkXz_ z&BwOwFbJ!bV}Ud@{X0q1YThda$CE0yP;V?i$Qc|d-Z5j&F&9i7wltE#AmY>;sw`q;6Tc#)~;wr||drl#^H zKD4bmp@o^R`spO;$>>?HZdv*&xkim+d_#@VsXI4R=RbSY7^9450f*d zogRbJvp`%v)*a?EFx2%0OI#JtJk2KQ_FG9cI8{C~In8HG)M+)Lg{6jh=fq@ZZEmU4 zsJ^Yba06rbe9YTLnjIk11O>_A=E7>U+HRVUkZe7VgOsM~H>VrhY@%T;%wZ*B6fN(+ER+6h^I>F z>83^FnWL*Bs?~y$JLFAB##g5JV$@{!#8Rt$D+u{9eIC79unB^4)?CK0Ngf9z%{jUX z{WlrhKUgb}SDdEgZb24?BPKotuFwDw@hIX=$Wp@{171MFxeLm9_(nlq4OuDMJrC|KO_>0k{WDa)* zd*{xmmHtcbIVcJ9Bp&yqaLU%SsZfDlahZ(_Lj07 zqOn&BN|G_p%+Rrk04F4lO!HEyU#iQu1Cie|p5BRunRZuez^bo-i?1hBykEH|0}6=Ll7Uc^bU!ks@$_UCcr^W01()fcs^L)(e1d)-0`-m$nY zFb-ntr%|T@wA#dbnWUXH-CSUC(U|FZ8f|i@h~L&g7zPvYdH(?M6}O}{ry?>7l`Usfdw5cF7WFk2 zKL%)5(A^GsKdf@C7SGIs=@GItX)J!^%UjcI!8n^Ws;6NauhU_k(%P;gQBIMtjmdus-=4K5UF zD65<509STLaASQxQ;b1%oik*S_ljsYs)A?V_vE19`NvL`_E0jJn z+GtgH)K&$UR`7mh^&KruSbc@ahMv~I1%Z5K9TAFNB%cvmu zJ>aW%MqF9#ZxDA|ZC7wfjG(|vH0`>)i2WT!(*O(~If(lER5NodEv2x|MBJ>`K~Gdi zBYJXs;6@VNHmGbz+n<@slqBY>NYB<Y@1`USPH+md0i&VIv-4^*UPh8dJ4K>iHS?mta^btweoe zYo=f~?IZ6qmOfKWaZ-l1Q?;>rbxM&f|s)MG!adrVtUO-icA8Bis{ zgI;d}u1cbV@#h{V8(pOOWu~d?eammFRxm<<7!Rob05eBR)DFc^1F0i!NdEvaHtF|S zxvXmJL+w^v1`|HLx=ll&uF8;Se&g}xT9oeM>Kmy783XJDbGTzaUeR4vy^dK|Y5|W? zMoh}nbuYvpLdx=($OLEgnp$lzxxv%f>83r($AkJAi!80orH3@VNiMBT#f{1VJaLR; zF}o|(AWdGGmgFFZTmnb=2mWSdYdUVDPFB^yR9qLq3IXjH>ALmPDh*o$O`xve-t2h9 zj@$hk_JYCb>BUQXzAGx<#qr;?+Sd&&+CchudW}o8AYw7kf76-BUr5tlbyWjqo&DQ^ z$o`#-)9IRryEU^r`?oH61bv}Z?OUy@R2IIXzKzN~x{#a^{{WB~{&7uKjkMu6y5CKC zH99L0+3$`qo;ybR%~dN6O+~D2N9I51A3-uYcCOt%v+(=PYWW)gz+vOJ_nC3ZYMIft z2K8b?1Dy8}O?$qdP}tlma0zAc^f9e%O+qb{mNk>kd;Nc`TUDqVKY4~Ya_=AAUjAsz`-+W{Y91N!K~+REs@+#eFp2m{{YH#1JZgGQ0I>L;%jXoF5E+- zWcd@N={maYuhMSwjH4^+E0PKcqtZ1dyOgmTkAP-9U7Or?hjnd%+E>(J&SizGZ6UFq zJH;)$*Wa1$;?Hj%CK?^CW5J$U5xa?%ZoPMRho@Aej_&N)Gv9g-7F%U0TOK; zjQA#Ma^pwGV=K^N_)biItp%=tm75wqoT-Ol9DAs!V~kE2{{T$J>DL||xodiDf&Tz3 zr{Ff6fG0;`OM3tyn@UQ`e@wo`;k#lh`W!Hg(Q+{akLxXVi; z^=#e(DweWflPYwqPcupGsZ#*Vo3*3VAagZ;q+07!+DSyAngG?!YNc`!C! zmgG%i{XMJ0(+g9UKWGZrFBqGTrAr6aBX3Q8+zIT*l4Ux1a^taGJtEVd;%w>buA~t? zCY9F!2+OHKlP@J8)@oRAvWux<+`*$M;s*W0@{S-)ooa!S#t7?>lkPJQP;Dm}m(ysq zHv5Kg3v{P#Ek?aUcbOLI!@dFHJ7-M2m*Qqy(y!g-IOi(3){dQPIM4QuhSQ?<#O8`> zX*+r15gO-TaX6hAZaw;Q?cxQJm;AVyV@}R+40lhYw~k@E2$bm&ZO9zQofK3?CF#-- z0qqX-uG~gZW1802S$^EWr3}gK98KGF2+UZ-?b}G&JWY8GIju)dweou5vhRn#=YNY3?*-X=43+ES7Rdv+5qRjWFX0~{dW0r3{p3TUMP zBX^US>20lVTfM>s;c@wM8`5j){9+ACFX=eywKiR;70?MEGepsKNNuo{W7x|>ZtmTB zT-#_>1Hhc`Q|f0=U<9cnz?|snI*Ofw-N(;t#O=AS%QKQk%rtiDY5k*W^z5LL$Fz#I zS8Ellb_5RjoVVg{J*wheHN*@yN643rUQyB6Nxf7C+ssp2qQS$R*z9ISol?hdX*)pT zT=h440yU&fNX^Ot)Ps7etV0ZA=4D%o$sirSCTu0tvU|BdGL|hbTLu74$151;R4@VY z1loYu#}aR>GT;u=Ik#^<>C+r#a?2|#Krn97x`k?h1ow*8%%>p6thsGXr@zc~p}vWd zOYsh%kpe~} zKw8De9R1_S%xu}SIrHC-yf%}ix~sLJ85m$5@r{e7J6FeX@j2zoHtUBC*I=gL&FVFb zgYt&tO53M5Rz*c?Zs(lw^pY8^@ONh%@DJ8fv0mfI4l{-Moc>2y!wa^VnGdW%9hSr7 z!cuCbHojT2k&ZB7tW43kh0b_JiX&rK6~+{fPIC;ix2Cb;lJ+ZV)tR;&p5VnChB;BS zyaOyhbz75Pw4Z4nU`?*uw^hF*f@U{ewGGzcJG$>9If8vtRHU~V?0dl3X|AM8Ya58n z9d4H+DO{?Ik)HBOjbEt5=W}@x%_dC?aX!*JRPAY zv>tn(5gksvdPI|gWiGeYMMZ8}d;X7Qg-hX(^PhOUj=YP z;U0GlTe2J;BOa1o zyyHJ;ZBCSJ%88zCdRr1nvqQ1BAdYFW2GjlE>(mEcWQ%}22_Y`ht)s+D^rbz3{!x*r z=4ffIRs%BH(+KG7p?hRcU^3v&nX0bQ#wK2)N)VIt@g(K-+I@#|hz&lg91}P0rEEe7 zoc^ywYqN4hR`#CNxiC*LKB1%-Er=eSaYhQ==B3!SnI;8#bYp`BZk?%p%bY_NHDAwc z%@wV0r|TjHi&De@PaGnRUd7K7MkXlb=YNakjlU2dYZ)K#2jVcB1##Xyc} zb%m21{$kpu#iYh69SBXoX{%MuSsdnPQrv?WzR%rcL?89NKxTJTqu0|mJBS@FsIks_ zaU5$E*BIc<;dOS-Q*ranZC;JLN^WmFL9Uj&TR6ZzCht?N66Ob9mB|nO;xJH3UXsrbi18};7DEhTy?Kx_x{Vw(P?Lef&QO9+O=y6T02 zYEVzfma|v|c?UW01m!8#n|moAa~yqkDcGm=k#1MjmmoTgqG-{5HUiby0W+7;x>yIe z2J$9{;k5vVM;J3LMyhHprswsSdJzGxrkzk*6HJT`aKb_h!iHgxJ|){&z-)CJ#LdQ5 zvC6_VSEZqMrkxhW;Hd8cMS9W|Q)4OMnV_PbU|@K(aTQ*lYE)59!JbU=^6}sPJ)B&U z{{ZPCx|*1Jm7%XYMQXgXExjabbmd(gLj(EOF{>?oI{yIMEQ7lNnU{Z8Yu$}bq{{Rui z)?268c5hHJ&WHyeq~~>8(@#ybzE-o0k%qzXHFS0*^wn~{-%(F<*pHv2E+$s_v+-JR zDqgaN9`+1D zaq%%=%6*A%P1T(tTXgoQ10ZeWACb)EdX-K1raqA?@DF*rr0QDAN~Mp$T%xx)9!Na@ z04RNZboCu2jY5QWt%g;uuM>8(ZCShLeydrFYUKR;i;^q19Yx|34Iq^#7LSM-Pncgv6c zPFJ^SgQQJ5e)Tq^SKL?*LH#?-ylT)8gI$sTdzzkx9H{bQ+V7>lX)s^EwIEaw=( z;%(@9-5UDMY}pC`HM3_wWAiO&+#{a4fa@4GxjhtKNn_hRhu`TYZaPk^fb5I;k8>~6LW=|L$v;Rtm01A? zF2lgdl+keP1%nE4?%?KTvhMU6g*5@=v5nuQTQb0m8Hdvusv9gE@JBOtgd+~w0vK{! zXv-bOHQF#N#zY!^={?h+yG>2sml>H_q(xwa;CxSZ9J0LGap(nbOe(uCV*$AXcK`@O zVBmN@XL)wvOnxw^!Dut6+;RysGRp2}_Ke~^r_tI|-az`z=xoAjK4E=DU^#*O3w*@2 z72iW-5=X>z&e~qUbHs{BRu|-H>RU0+QRr*vD4JlfsAsVmKC`X z<1$G<_Qdr4Hk$j99T%+? z?E)RcwXS{M(ZtBr0y{J->Pd1(If`6uZbZe$ZA`QpUiF-VE_#y>{{S#cRgB!MAkwHg zX`6#nBJZr!T22l>S$cH0TltJNwo^rX49Zp1f_5^IS|aqz3C80ty*bhGxWKT``p0X&VBrM*w$-xU%QK z97YXqVB@r6ihVxfduDP+o5t@z(V%RI(`g3WVp+9oS;uMXHA(5@HZwV2=-1P8h@pR_)crHbi)uQ)K?+YZhJ3dtTzKSY->Inv0Ezml=Qx>xtiHl?8m*N- zFwEyXy|-i$wADD}N8Sm#eQY=bA~JP!mB2Wj8j8o6*X7NN$Bkv%qiW7nLsMffaUW}I zaf8fUpe%Owo$=?Fs^ymx$6K*^6E^;)cX^Iqq-Pn-%x;+M=L59#O!0BJ*t&HBsqlm4#d%@bH0(RBu{Yu8{VkxTW%5${y8P9LC@W&gz z-7q_Fe(|jqvwAZ*BkMGE9Wxm%h>oL0>DQIud5&C|*d{j`($zTp0Z3i?80xL6L6`@Wu8vztCXX9sv=tnPg=L1y5dXFb#F77MPkfSd?@S5b3g zY1Mda!|AlPE|?$~^6X)zy!6HDx{yG^u5sMXW%V{Cxjtu0))JP=j!5oiJsJaH0da;r zVroI>Z8Ou2wvEoXP&~)9{Wi^2=IekJcm@+psYaan$IQ03tgBb)(xC=9+b31)O*V@1 z$UU)LcTyUg#bAEzekN7jLmY0;+GgFVIRQ^;Xj>z1S$!|3+qVtiiG(#}Utr;7EYuRc z^8l5z^W@D;e_6{v#@3rvmBSo}$+o6eIPoyh*+=FW2h5e$6Sxz`K4aoM{{T&TWLV6k z6o%C3_Np4mClO69qiMr-GfZkwDO_ZjX>FL7GBy}bHQwMe-d9?nF$zn=EA02fX*WKrB1< ziyKDV&20@vwpI!c(9G53XA^FM%TnxG>!^>+`8)nf!kJn+O0`;xNv@(sG8IRUpS0+- z$huuPR;gOi$A8QJ0J&`zoomt4q^j&agL92J{J@UPf5(rc+OI)fP7om2n8+q_%DZ8P z`(@}B#%8r?AwW1TKdjr+>FP8KEn-G^$?joVqQ=Hb_Z8h(zB4z#9wT^OG-MRdZ^+q^kT(w$Rj89oYq*lo2nRfzh#Kp>nbBec!xJw!6vykPkf(JOwg(S{6ugFWD@)mU;^ z9FEX>1#NS(B6|?vnYF3XrjJSj@O*t@HbUKPtx~#=YK$0elVuvx;h3l#7zMQ}sl89i z!;vqh71BkBeW$tlMJ=r9f}c?C-(c)>+)NvNYFMyR7ImWZDZH+7F@rKSNmcdO!~h3u zL2Zv!_Lui?B-v`UxL!xh(5FRPEhwiuk0v#&>2=XibzkK)C_N`jyHFj@6?rpPTdfE} zDC4li#;;aULaLme^6O!`(AFFx9GuBDlTlj?=M&6|$KrB&ZGEcPCTh{tdje+*j-GwA z%~(0_EmfNi4{1tuO5?njT%3p_M5C?@!q@53cIlzKh^DVDq{&u5v(bL=>8>MzkrBS8 z^$Gcn(WT9Kh&NWfgLj)6PNuvAIlJoghGO;W4OV&L8e4a%s9OeN`i(C9k=kKs`jGzs zRwmw`R<1cYhMH$MdRr2sS@n-=D_aziIlWfuSePe6ONSW-Fpg1%4Cgc&(~vVXTT$mE z$!*csc2H!?Si92`BT1mpENf?^Cmp_Dr_OGRBurlH@pxC8mE{{>02%CCe znYKt2G~7me$0;Q7m>p));IDW(Y$2kVopzz>oE$`Sb}fifzzzqP{{UI{+3_-}BD7+B z@jkBw-ZAs}XZZoErm&>1aR#JYPSS9GVf6r|M3KbI)4nZZiS3@AYwR?sjqIum9L|$T zqNh(-vc((zBAPK?oehKoV{$VOWr2TralkPM)>xqlMG$%|WpoC2{QeBj>RRtmZ9~V* z&s|cCFL`XYpQtszoBPbWt{ETaTZ6x`Ot(F0hd(Kyf@!E6$0Na$+Xe>IMXR2Lq6%F{5NpA&MeF0W$Q zUY){$mR3{9`_5}k)HQ=EW&Y+Jw|`NN(^pB<-%Y67se2iQF=1?D?q?g0oh95xvrlam zn{T5P=iQKSTgEf=$b(DYO-dB6kzg&Ja6Z$dx{9@xmVMr#P87cn=Z_vH9rO({F}AA+ zem0|%pV#JM#(gsmx~g>xWrYY0hh?O0IR5~iJ|M1pE3J#0du43SYjK=#-K8!~K9DwP zVQ6d$gBLvZ9sK=%BL1-yS?Ukm_j*Nv&l&Cq-)_eXMPw)}e0CqH!G~AWX;!i=8CCYK{twiQ{{Vl9 zzX+yv**8WihRw2U2(fZt$o!QuevXf-{$H?P;uPDn-z&ma!xlR1j*vc>fkNiN%t z)yk9l_A&a-q0MWh)7oy^@51o8`6qxej{HnqTbXUvXtldlDLQt-uGTiN%O4)o+B9p^ z^;!lBk@1iD&)#!dQ>{{!>IF8S$p)v%Qhuw)_?y)mY=2V6q?<4>2t1D5jNr`J*7gtK zbg0a_iIP2sBr5@roJRiud8nx?)TgIvsZiB|LHXyu z>L9fJHt<{3QRy=6SL6ad`I)YHPR7uL-?vS6LG8R_KTOTl*IaAZ+%VvS;&OFWWe;U9 zo=!t}np%BbX?O1|F2HT$wB?tz9af&I)k9g_%3q~h4t8!IF)I3o^%WIvM-7e)DP3NH zMVq&wl(s93rLWPod1F{3?EsoZEA9E_YNt?J*KWa+-crmW`*hzQw#~ioF4*4-AXVFOQv&;+)P|@y|=jx zjV`5Gm?$IYdx@Q{w^3iQQ|+E6lC7+F6OW*Px7Kf9UHS6L8z0Ue9X@f+)zlepDRPw{ z;%;efOY0;K0MB^ceJyNk+ek0reINn-L<4BGUb8CHs;D_sJ4QZ`@TO-2d%~|1BU!CJ zIGR>;-BpT3bhhF7Looejd+0R^=hdOt?!13F%i^uhNg0i0_UGbSwRAh5wA!mmDopLi z-Y(Ro?T`ujncEII9P-Z46s@Tn(xNAL?wh5f3?R#V_>KPn(JfrFSh)I5zA=lHLavtD zihu;mze*{$eSM?9gk)yg$C2hxX)VW+K4x;lQ;v318cTN83v(M9#PUC^&-^B$r`l^E z9y11XcWv6POP<1JVPjV*TIA`SJ;MaVuF|1mYkZH0nbT?)aCYPm5kJJ=Q{$dxao4rK z#*=CID?`RFDy3RoKAAUKK|kIjexpuoJ7sn1Yy|~} zFR4d^@Ufd2sSXt?}+?cA10Z}y1Itt zIhRW_*c|;Pn>L#IvN`#Lqf1II;Jk@^HXo6izM=4clp@t(A-z1!Mz(~@S&n>xDfpV+ zuocEWeq%W77dc9T#PXProkrS5AcLZ=9%pN=0g47AGdBwq$}6pJRY}i36C%G@Y5?z; z=4>Nq7zg;6;#JkuEWh}Tp$qA(v>RlUb8$S~E7ODmhy-?3~a_;j#T5+m^BAhd(|bo8 zaTN+hjb=E<1`7ME(}FC)I0O$ds~f5%`sZ|N#!gR&m|Gv)+C>_aa!PVff24p9s7Nr= ztjl6*G}YHm0W42)Vm7ZB6GE!()T8X|JGX&7cTca@L^o_KqY4nWf49umF`TlDrA(fC zMk}a3cKDkbUWc0MwLNQ1g|?Naa9nxDet)!ezh-Z&hbG6~J?!8hAKaP08poV+*Mby` z=MZ#MEnETa^EUOpI;s$RwHuiGYc(j(#!hBus)*masG$S7eCO-h7Fax`dX(GeIF~l< zZOh&_qNVjWZbd4EOBOx5c=?O&^w`%;XD9Qs4sq}YfekgV%bN~NTTp-@A~$tHqDpRc zBR#u)BD*XW;>BfbXK5VGNX}UlM(+ttC)ZJG9jE%vm!^KrzN)`bt11cOGf}^0{ivk- zSGdg6m2!0SzgW1fj^Ch$Iu)V!79@_rmxu}p^lEK=V$`^5l{Y%L$WpQZdB^}d9A zSyvl-{NUmj(Z+V@dQOV1J~b??9dh|6^Aj``eL!t4RR=7fa^GksO_~}uk9IpBA9ycW z%?(IkSblG;^5uJb@SQAfsa4qJE9W>oh>aF{*KAt=)0HPDy9kd_b-}8qB(^dmdMzl? z(x&cQkQ97Ov>*K?i=9HL6$6};6xODhn<1Dg4oEzcGw3xFCN>o~Kr87ONgJ?U>J~Z9o~^#P|jyVWy~hu5KWl`#~IH zYEyMDPRlR{(qd{f29xUUImc+K^%v8y4n2|88+Mtg(6&y}G0e`@`e8qIKUjr5R9#q5 z;18s@zuB}Lj}Xz=nglPY4s0O@gEqR3y1Mjb0Nls4=Ct=J+RX2?4#O~7&aF}thspUd z(DD-7O(9=6E!;r9vX;rk24D;>7 zNK0C#!H%Y{+ukE$@8m`IG8vlFWp507<{w~nnDtwpb2MnU;By2W+CjfF!)F{UxInx88zFTx9IklQoxR8_M2CLIha~X3Z7%5 zavX`zDK^$M(bcLB0AmNJ(yhzKnT(pUrfKQ5%EJWCZEUyTn!5>6E?tpu!?e=emI%G{ zZd`~EblT0fd`(L3^#H`??WwAKiJ_p>oPbOzZlEA?W@l4P$V_#0DnOBPi%^V7IaMS+ zCWS7*cae6JWrTqSjkTjf+P7eY=4R7*E}LPig0m}IT5J| zc@O+vXVd}%&ZM!uMW{i-=0*6ZL+}+IeO;xA@zB4B68G7-%`A=l8HH-c;b-4qG zM%1050@wJVR$k&7POF zPZB*A+qxz-A?yUrI(43+1PVCw5J~Wd1q?o(+T86tO6!TSae*^;nz_s=(`xF%0>Jxz z34y0mkOa>f$FU0J;x(roqbO@s>PHGQFIJ(ex156%sa$)3hx|fhdW(be5a<_LQj8on zV^j}SHvkv)n6;Yhr-lTGMAo%gQcgT#T03+VP~eAKFb95Sg6cxyeMQHa%Ga;BXVcE# zv?Us`>H}*K(bC(ow#yQw^xb*yBArVqDR~8cla#-#vOdtq1L9h~uFUyi4H;csx@xUO z^cXLmVxRP?fg<}6^Uo7Do~YY1x44(7n^ZOn3X?hBJY6RyMLNGtbi3CT??5{V%x2w2 zb~ZM)ap7loK(?hRbBh!EM72E)EG=amExa%jhbJ8O9r(F#s^lmdnso*2T-Thj1M;7w z)TdirwW>7?{PJ6z!<{CTZdCQ1OA@7`OA9Kpw6dd+7+?@b)_C*A?d`)ov@{)5 zBep~B2b3+_>L%8V>2<}+UIA`HH|G9xi*xut-K}i)JAgUwHR#pnc&}~;BxAJYW325x zW2KEQp3AGHnL}=7&vWyE^_co@pQu#zThwb`l918)8N z@(+)gdvxD{)oM3B`o;%rjGw6Oqp%Ev^amf(YCl5iPhM*D%?*!j=r9I3IWT|zoz`2n zt(4VM4c4s(8;8d4j(@B-D_%4=rJjO=Tc`9$!q!yON9ad?ytbQ1sHuPW&sG7;`c@Ff z>@mpuWZ?mEeXVo3U@Nc={xjfDRndB#8jA0-yz`yNz=8d7gEMs0m8YZE>S<-7 z!WOg2Iotyrp8o&_YirjTSGYOg%+9Yy$5C@w)#j1`2RSnrU8}SBo4nAkQU3tP3hDqo;(bF% z%*xfEVcQ0MALi!zuNOZZ`ecoFq(LnJ{ZW#_QMfXioeh+^^>L3OPcRMBnY62BN}S?- zDU`9|@v*0;>d5%q0VC!x>X(eD%NdoW(VDOrf)9DOr+O(W2RNCg7G8+fux+@S74*6( zMcRVLBP5=~^oO%ikP2}aT|z5kX#>eK&y}^?8H(C%ElhIN;Qs*QH8j*EZMkuXmY)9r zQ*+izf!vW9s?%bG85#48P8nRc)2=fec5Rs1g##Zksu0!Qyq|(N654GgLZx;&f3IN; zw(YsW3O=(F8}`++=rq!L)_b#)rrSrEwBbCtSZ=i~V z_huR5Ls*F0tzZr~22$RsN_(A%TwrbSFDVwSIGS33<(a_jS`D=6wBnJs&QE!lZ&+*C z06@p^lZ?YxtTqB0Bj^c-oIVulfa*d3gm!?jFR)<_6X16+e~8-TD8TW|Kdg&R`v~Xq zAn`5AwMjOb>q{IUKUK~m$EVqoifal6Nl}TYHI<)p0toDLi7Ts3xLz3d%(aT_wv|HE zN~Qq$^9IUU?9Nnp$M%ldw`nyTABc_W<@oFeIO8TDqiLTc?jU@zz?81K4?Qnpl>;2&XWOGy$Qzl5nDX5C$Ee}6oPP|1TQlWUV>#Xrh$z6;P~%YEj{T6- zC78+F0QsFIdL%AYRF=j@c;*yowg4<4jx&-wOU556%G1$V2JN{x;C#ib>MK@qgKzO* zO-r<`dKLA3H6sL~%*%A9%JvHgOyd~?Kc|_ze0?#MtnE&%^-8a%a6!-U5>_i(YA)Pf zyMP0^5xVQIr!|({yK}i4PXM17`hV_H-mcYxhRQ{!mao4b0(vpU%PUu_Nv>17XjS{h zQ(JDCb~Z~7(kgXYg{6qvm63|t&k;)%olCnb?L3t^ApG$+;VT|iG~V{NAhy$-Dv$35 z+O$Ln5v;Ht#N+KSwOv#T82WFxbB6DjG1SGtvM;4p0CJ-wvFtxtxyJ{cZdI)hUgsvX z&fF3XPty`ExeEK-fsuiXpU`=kOQ*Y5ifAUCxfuLe| z#d+4=qV#(4ZKYHLj4pq)O1(`NF3?>;G{s&Ic@h^D_%t^U?YfTWfOf>7U|0e$kq=-q5y{Tnw`}Khx%9H(Chnfp=hM{^9L0 zwf%n4sAd{_DNr-E#Uys`{{XHg=M?*5^53Uix{W0V+QQPnjjLZAcgZnrL-fTgtyN3> zVAh!N=#y8UXiRO+lfq@wLVZ;#SnTh{e9qE~ZzY{QpU#~ug3{{WoLe0Y8jJe?id zEgz=0R^=5>8(ew%cZgrnskj?XrrQT6+@LY~4F3Rmm8R8Lwr2ZR)KwS^`8gx$7;d22 zO8_moa6RXYgX9koyjYpZlP2DqsY{ljrLL~QAsAM)-oG}CJMt{sk>h@YaG_7jcw5~JX;N$yZa=cN~9!lYkoK&TT8hT8o;m#|TsY+Jxo635QB&3{_KIjn ztyF?l@<1K{nOeTNS}O{b6-mx{1P>tm#dY05^t+i-wW|!fdu2!?^*#F;mtoW1E6NX1 zfZRDOe#f|-bsTqEW2osaHL(9=)eB} zjc`+pFlOL==B9?1WcM%#L03D5GC!Zp##l<;IoRjZLfV9TF0A^9ygjz$AGdGwHFUaa z7fRZVMwKqGagEqr+!9!ixZ-R-PSQ|S?zmc`z&r?NIx7@=mu@kYJbjGk@j85te^GF= zQ}{*O=?>@+6;uBJi67tl!}O2Ya(aW^th5X_f!ut}+BEufKEYG6kIi54O$zP_P&mOn z509kfmO5`6wJuueK~+oaQH2A!neTm+S{C}wBkgX?T*`6>w?8op&Dwy(s)OW_%)>@s zSwo*l5P?Sr<|C@HYX-+|Ob*tq52S;CV9a|&?5k6Cij9o7c0GWQ+TOCgfL5i0@X9>R zZ3QTyvv9(w+`#)s)9W$@m4&3OOcNn9j`^J*Of~48sr22BHg2 zH;F)vbFO_g1Cfl*XIazjurSB8_MK~K)}*bRdx?o_?W|Z70l@J!D>gZW(%qx0b8;uE z*AjUoM|GDVT!Wc_63d;9YH^=4Qf-~-wwJC6BX)kWmeHoNqU^YAWM%_TMd|@7+Bo!C zs2etg2f31FY0}HC+MJo8_-iO*y3YKZ_?n$NK~)*CumJwB9Y~{A#~6mpja)1JVz*LM zk|uVaseo`y)TZ^lah%U4toG(z9D{=_)rT1nuTc1d2OL78UQvFnL`XktU{(q zs1qG1~3SA|YmvY0e=1G20VikUGVVoQBNblQtEupQN>l0>;%q@dJ8+ zE$uR`$vjA|_d|kAZE6hD*6Btwh@*O4X~@aQnswY9aAzsGs%%XiF0VM}nVhDY+h0*i zx@R2B!i`fBGas()!}{VDQ%`RKX5B*v-m!{_RT+VXbsr)VN2n-9 z58eQh%LbNgd`UV${;gvthz4Q~1g@F)RymcjvJPVr+FW2ob&r-ZsdmB4UsHB+({eL! zPH(P~P~hTn*H&4>3BA>IYh$tlGJPD@O3$amEXCpEyQS!`*5jGIMWmo0@j23r90D+W zO?@SXs>bs@8*&d=8MPaa<%Xz`ow6YHu3n&2d4j6T6$G|4o7qyp2N{CCu*tb$+Ow$u z5zow{)99gGlk?8paA$f4o6eo%xJ5naRv!O`x&H z^ME0|_9lU&Cm%Ps0K6IekH8 z{o?Wr^<wj$#{HzLG1;Rc7gGH3>=qWn(cDta7Dt3;3f$xl(AvG} zi_yyg+)Q5EhqXE+ZR*vj-TYXXnx2@{Qc)R|m!2l(mr`|_d}oy$LD+%QHn9O)kWBFN zO;@{zEwQWV>wc3UDY3^7*@6E6GQ*@>r%KhXN38)agyEY%6JPN4QOX6w9E_gQ29n!v zTR^VmJTN{4^W^nn&eHm9Hw830aq1TQ-*`Yi~>A=?ltK!D@F7u8TRA9 z=w*{lr?c9dy@Y^=gE{Td%wMM0*J^$lKs>sh`pEGBpRL`v=QW#Z`3E%8)dykH0 zzu?^}4m9PqjCSYknT*$4qIZA5veQ;i3`pny0K~V_nD*-J&<%3E4xXK$6S;W94#du0 z#n)GL)uq?H`v`D(#yt1pZS?-Kx}&G6l10E_!#Cz04q^K4fa(=23dNhcDZni;D|Rt- zI<~0oVxF!)Fj98^04P^&ey!{PP~kx>>;QrCOrs4&9-TpJs>S?@*TGdv;1Ks}sK9M_WKa8n$A+0ERDm#yn^983%MzJ2CDhZ4d+fQ%j1$8zntI@IW za#f$*_KGn~3n@kn$At<&Bm5ZNpLMoZxoJUA$?|{v!JM&`pQ!*Atxd04<^44Puh9I& zt*WY3omK8oc&Ug7K5&1;=&sXIaa(IHn@>3YAbar_Z9RILuB!`b0nT`Dxs#omU{zqY z^XTp$D90pyezCr$wpFJZjXhzu3^8s<`v^O9hevjD_SuHn*Z@D&eCO%<%`_tYDvRp< z;2sZo&cU{3ms&MCRb~fz@Ohdvvs+fyVsY~_zv;G9d1`C9t32&rA9<&xs`X+`Z?Fz> zJ56JCcCwqAElun-Xa4|>9fy@U{*zVurjKDuE4eoe9M-?>2WncCf{v+?k%7)9cPdORTBKDB{GS}m zqWWDOtj%i>e&>uv30*GLtdd8VajHwXQWdd*^_e>DMT1fhg(tx$j7(*gnv-H{AAh-s z9Py5FW;M@qI4O~X-!k@IS&2eO&x45zO^$aE%6tRNt(Es?z|o*q-LB=Qr5u>GMslKx6)`=btOPz4;dIfCOcZMgixyIu*P#a?yJBZ98XUl70JsywCnX+x~Wd!3}gj>{{X4fq!(STM;YKwVvTxTr%jGMO2&6F+y}_` znb-A8TG3Nfd;VoSFW>Dw9}IH&Yv@#pwG}IlTNuGSK)?7R73?HoNlJ<2kVFiz14}aEi%Z|$Fj(alCSsO_#`Nko=F)KmFccPcwOKTXyFm!TiAY{U?r&txx^b-?Z+? z9FMf`$H`-!SZ+C*wVIaI78j~jf=g#PTxY*=+Euk>EA@R=zOjrnoX z+S>?y{y^*xfsWkFD>NFqZdPP3TBi@{g;U3Td{1UM$C~4!%ha^@AtfA|j`7@q_xeQb z>h)>c)1b4Xdy20-{Z1nKifdi=Ie${V_UH3wIQIVlv;uUibyjVyl?nXWIKrRe`(v5X zXBe@$PQ67Tnc00(Rat;ien1|4O&WTzusTqu)K6i4NIc`bosHqE( zIBa~8j{auF-A3wFK`Ia&F?YdZjFLW)*ggLMS9PMpVxYkG?qRr`@;L4K&uRL;k?m1x zDz{^hTff&h{bqGqq02qmZJo*r2=SbT$0T`z{APg|wwkbAM*jd)aREsAA0Klwv!_xn zqMu@7z=I%?4%73Gq^*ml)m4h^Otu|>Q;hlgLly^2B!rS))@!54o2dBAS~(? zQW=gIg~}<&-^c{_$H_62g6(5!!ctAkTasIl?s@O@lY6G3%Ie!vC>Y$TT=(Wu{-b82 zPiUfr^zH*G#!fr-$A9q_({z1xwwmZyCek+XlB#pdO5F`?OS(-OlyOcZB#tw0N7d&NMXQ2h^q^%HuZ*=;U#_^GMV z43ds}{GNOG?=f?csf&Qp{vkseaIWX`5&4(zDb(`pp5=Y|JyBRq`G{{a60 z)4eN9rK?uCQ1Mc+!25kfR?eNK(rs$cmnV(G@@MbJ&&)nZ*T-4CJErT6C9F$RPFFjD z&T>zWryNGtNv6|V>R-CoJdvHi9}`BMmm>uNz^%0kwtt}gd(8b+8uHszS7Mgh6k`MK zBj*Q=A5E&#xhkLwSMw%vTmF1vGpAuqTY@TJ5xvevPu_F4bj@;HuU%%xCm=2rYadgd z`2O&C z_5SkIY3XY1TAf3`v@OR0LEwOYUgj-coHdNCVNlrx1AcF$kos+1mb<>Cwv6FH?Ee5v zc8yuARjmN7s>c}&IC+rz9bIlIuDzfg#1oUxl785lYtj0a7e{WMrBbDW?i^(JA|6LV z*RL4~{{U_4=Z|nB{{Y0@(`%{LQY=R(+fTC@6OPpBF3^+~hpqPEO9O`kfWt7_UZgbj z0Pe+Klk}CwQ!S3A!quFz7Vg2DApQP1j0Rij^#dpXVS(pq`^-u;HC;yGdg0CqIphJv zV^4mR0a2_gF~;I}=dfTuzuscwBU38XrD~xRFjBUXc=CS!B3g>+^%K&moRgJMbC1?I zntH3#Kdo{;dxaZ+PJ6}ew!J<*%Aj^mPDj`4Io&tk*dnqo31Yt9%27}D5xVr^tL%Ev zlz+}k;|J_d^ouQ`+bi^4s;P!ZL7%>S!}{rvH!Gs{AH+XjnT+pFm`=jeUZYiTDj#U} zf^afs7gb$d%*k~`B8=g?0r#30vL>;Y^DV;)-Hv`H2D?{XmpvCXn-JkL#(7?in9$~m zoV~k==RYudix`6Y@S~iA4yr8CJE_iB0DOth-DQH*Ur+#B7@Ct~%S{@Fs^OGhXUlmJ zr0H+HJwgB{fi)B^n?X>mK>%fmi&NBnQ)Tb@K)@3&lGy^iY0-Z0a(fKS`#OTN=Q)pc zB}*#YSPXMHr>q;WNdU=tR`c~gRe)CK7!yOKRb64Nyz}B<=<7nr2Z-hOSOyocnXTh? z(ds(UQ!dk$1BuRSHFw$>f&0b+U3+_u5Pe}R(}bak@*^p=f-gD4oaQrC$hg2Cq(@sp zs?7M7rI&68dA)Syw$(cDppBwkb+x~@w8M5e81D$WtT+WoBfK=!u9a_8uQ|`_Gxb)R af!shI1WC0|cATw5gA@&M?%9Tp-~ZW0!i`)2 diff --git a/integration-notes/make_flux3_modular_repo.py b/integration-notes/make_flux3_modular_repo.py deleted file mode 100644 index d228701b53a0..000000000000 --- a/integration-notes/make_flux3_modular_repo.py +++ /dev/null @@ -1,95 +0,0 @@ -"""Build the diffusers modular repo of FLUX.3 image-small from a BFL model package. - -Converts the transformer and the image VAE, copies the text encoder and tokenizer as they are, writes the -`modular_model_index.json`, and (optionally) pushes the result to the Hub. - - .venv/bin/python integration-notes/make_flux3_modular_repo.py \ - --package /raid/yiyi/f3-image-small --output /raid/yiyi/flux3-image-modular-release \ - [--variant sd] [--push_to_hub diffusers-internal-dev/f3-image-diffusers] - -`--variant` takes the DiT weights and config from `variants//` of the package (the shared encoders stay at the -root); a flow-map config (the step-distilled `sd` variant) gets `Flux3DistilledAutoBlocks`. - -Scheduler, guider and image processor are `from_config` components created by the blocks (shift 6.93, guidance -3.5), so they are not stored in the repo. -""" - -import argparse -import json -import os -import subprocess -import sys - - -REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) -sys.path.insert(0, os.path.join(REPO, "src")) - -import torch # noqa: E402 -from transformers import AutoProcessor, Qwen3VLForConditionalGeneration # noqa: E402 - -from diffusers import AutoencoderKLFlux2, Flux3Transformer2DModel # noqa: E402 -from diffusers.modular_pipelines import Flux3AutoBlocks, Flux3DistilledAutoBlocks # noqa: E402 - - -parser = argparse.ArgumentParser() -parser.add_argument( - "--package", - required=True, - help="the BFL package directory (config.json, model.safetensors, ae.safetensors, text_encoder/, variants/)", -) -parser.add_argument("--variant", default=None, help="package variant (variants//), e.g. sd") -parser.add_argument("--output", required=True) -parser.add_argument("--push_to_hub", default=None, help="repo id to push to (private)") -args = parser.parse_args() - -python = sys.executable -dit_dir = os.path.join(args.package, "variants", args.variant) if args.variant else args.package -with open(os.path.join(dit_dir, "config.json")) as f: - package_config = json.load(f) -blocks_class = Flux3DistilledAutoBlocks if package_config["dit"].get("flow_map") else Flux3AutoBlocks -# newer packages keep the tokenizer files in text_encoder/ -tokenizer_dir = os.path.join(args.package, package_config["text_encoder"].get("tokenizer", "tokenizer")) - -subprocess.run( - [ - python, - os.path.join(REPO, "scripts", "convert_flux3_to_diffusers.py"), - "--checkpoint_path", - os.path.join(dit_dir, "model.safetensors"), - "--config_path", - os.path.join(dit_dir, "config.json"), - "--output_path", - args.output, - ], - check=True, -) -subprocess.run( - [ - python, - os.path.join(REPO, "scripts", "convert_flux2_to_diffusers.py"), - "--vae", - "--checkpoint_path", - os.path.join(args.package, "ae.safetensors"), - "--output_path", - args.output, - ], - check=True, -) - -pipe = blocks_class().init_pipeline() -pipe.update_components( - transformer=Flux3Transformer2DModel.from_pretrained(args.output, subfolder="transformer", dtype=torch.bfloat16), - vae=AutoencoderKLFlux2.from_pretrained(args.output, subfolder="vae"), # fp32, as shipped - text_encoder=Qwen3VLForConditionalGeneration.from_pretrained( - os.path.join(args.package, "text_encoder"), dtype=torch.bfloat16 - ), - tokenizer=AutoProcessor.from_pretrained(tokenizer_dir), -) -pipe.save_pretrained( - args.output, - overwrite_modular_index=True, - push_to_hub=args.push_to_hub is not None, - repo_id=args.push_to_hub, - private=True, -) -print(f"saved {args.output}") diff --git a/integration-notes/make_tiny_flux3_repo.py b/integration-notes/make_tiny_flux3_repo.py deleted file mode 100644 index 4f149a332002..000000000000 --- a/integration-notes/make_tiny_flux3_repo.py +++ /dev/null @@ -1,125 +0,0 @@ -"""Build a tiny local FLUX.3 image modular repo for fast tests. - -Creates randomly-initialized tiny versions of the four loadable components with -self-consistent geometry (the transformer's token width derives from the tiny VAE -config the same way the real repo's does), saves them under ../tiny-flux3-image-modular -(next to this repo) with a handwritten modular_model_index.json, and smoke-tests a -2-step t2i run and a 2-reference i2i run. - -The text encoder + processor are vendored from trl-internal-testing's tiny Qwen3-VL -(hidden 16, 2 layers, real Qwen vocab/processor files); tests pass -text_encoder_out_layers=(1,), so joint_attention_dim = 16 * 1 = 16. - -With `--flow_map`, builds the step-distilled counterpart instead (a `flow_map` transformer and -`Flux3DistilledAutoBlocks`) under ../tiny-flux3-image-distilled-modular. - -Upload later as hf-internal-testing/tiny-flux3-image-modular (and -distilled-) and point the tests there. -""" - -import json -import os -import sys - -import PIL.Image -import torch -from huggingface_hub import snapshot_download - - -sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src")) - -from diffusers import AutoencoderKLFlux2, Flux3Transformer2DModel, ModularPipeline # noqa: E402 - - -FLOW_MAP = "--flow_map" in sys.argv -OUT = os.path.abspath( - os.path.join( - os.path.dirname(__file__), - "..", - "..", - "tiny-flux3-image-distilled-modular" if FLOW_MAP else "tiny-flux3-image-modular", - ) -) -TINY_QWEN3VL = "trl-internal-testing/tiny-Qwen3VLForConditionalGeneration" - -torch.manual_seed(0) - -# text encoder + processor: vendor the tiny Qwen3-VL -qwen_dir = snapshot_download(TINY_QWEN3VL) -os.makedirs(OUT, exist_ok=True) -from transformers import AutoProcessor, Qwen3VLForConditionalGeneration # noqa: E402 - - -text_encoder = Qwen3VLForConditionalGeneration.from_pretrained(qwen_dir) -text_encoder.save_pretrained(os.path.join(OUT, "text_encoder")) -tokenizer = AutoProcessor.from_pretrained(qwen_dir) -tokenizer.save_pretrained(os.path.join(OUT, "tokenizer")) -text_hidden = text_encoder.config.text_config.hidden_size - -vae = AutoencoderKLFlux2( - block_out_channels=[32, 64], - down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"], - up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], - layers_per_block=1, - latent_channels=4, - norm_num_groups=16, - patch_size=[2, 2], - sample_size=32, -) -vae.save_pretrained(os.path.join(OUT, "vae")) - -transformer = Flux3Transformer2DModel( - in_channels=vae.config.latent_channels * 4, - cond_in_channels=vae.config.latent_channels * 4, - num_layers=1, - num_single_layers=1, - attention_head_dim=8, - num_attention_heads=2, - joint_attention_dim=text_hidden, # tests pass text_encoder_out_layers=(1,) - timestep_channels=32, - axes_dims_rope=(2, 2, 2, 2), - flow_map=FLOW_MAP, -) -transformer.save_pretrained(os.path.join(OUT, "transformer")) - -index = { - "_class_name": "Flux3DistilledModularPipeline" if FLOW_MAP else "Flux3ModularPipeline", - "_blocks_class_name": "Flux3DistilledAutoBlocks" if FLOW_MAP else "Flux3AutoBlocks", - "_diffusers_version": "0.40.0.dev0", -} -for name, lib, cls in [ - ("transformer", "diffusers", "Flux3Transformer2DModel"), - ("vae", "diffusers", "AutoencoderKLFlux2"), - ("text_encoder", "transformers", "Qwen3VLForConditionalGeneration"), - ("tokenizer", "transformers", "AutoProcessor"), -]: - index[name] = [ - lib, - cls, - { - "type_hint": [lib, cls], - "pretrained_model_name_or_path": OUT, - "subfolder": name, - "variant": None, - "revision": None, - }, - ] -with open(os.path.join(OUT, "modular_model_index.json"), "w") as f: - json.dump(index, f, indent=2) - f.write("\n") - -print(f"saved tiny repo to {OUT}") - -# smoke tests: 2-step t2i, and i2i with two references of different sizes -pipe = ModularPipeline.from_pretrained(OUT) -pipe.load_components(torch_dtype=torch.float32) -common = { - "num_inference_steps": 2, - "max_sequence_length": 8, - "text_encoder_out_layers": (1,), - "generator": torch.Generator().manual_seed(0), -} -out = pipe(prompt="a tiny fox", height=64, width=64, **common, output="images") -print("t2i smoke test OK:", out[0].size) -references = [PIL.Image.new("RGB", (64, 48), "red"), PIL.Image.new("RGB", (32, 32), "blue")] -out = pipe(prompt="a tiny fox", image=references, height=64, width=64, **common, output="images") -print("i2i smoke test OK:", out[0].size) diff --git a/integration-notes/parity_bf16.py b/integration-notes/parity_bf16.py deleted file mode 100644 index ed4543023901..000000000000 --- a/integration-notes/parity_bf16.py +++ /dev/null @@ -1,125 +0,0 @@ -"""Visual parity, bf16: BFL's ImageSmallPipeline vs the diffusers modular pipeline, same seed, same prompts. - -Two runs, one per environment, handing over through PNG files: - - # 1. reference, in BFL's training environment (transformers 4.57.6) - CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 /raid/yiyi/venv-tf457/bin/python integration-notes/parity_bf16.py reference - # 2. diffusers, in this repo's venv (transformers 5.x) - CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 .venv/bin/python integration-notes/parity_bf16.py diffusers - -Cases: t2i and i2i (the fox), and BFL's two layout-control examples (bbox regions in the caption; the knight edit -with , which also exercises the i2i size derived from the reference). Writes _reference.png, -_diffusers.png and _side_by_side.png under OUT, and prints the mean pixel difference. - -`F3_VARIANT=sd` runs the step-distilled variant instead (16 steps, no CFG): BFL's flow-map branch against the converted -distilled Hub repo, under OUT-sd. -""" - -import os -import sys - -import numpy as np -import PIL.Image -import torch - - -REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) -VARIANT = os.environ.get("F3_VARIANT") # None (bf16 base) or "sd" (step-distilled) -PACKAGE = "/raid/yiyi/f3-image-small" # the BFL package -BFL_CODE = os.path.join(REPO, "..", "bfl-f3-flowmap" if VARIANT else "bfl-f3-image") # the reference implementation -MODULAR_REPO = ( - "diffusers-internal-dev/f3-small-image-distilled" if VARIANT == "sd" else "diffusers-internal-dev/f3-small-img" -) -OUT = "/raid/yiyi/diffusers-scratch/flux3-image/parity-bf16" + (f"-{VARIANT}" if VARIANT else "") - -SEED = 0 -HEIGHT = WIDTH = 1024 -PROMPT = ( - "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest clearing at golden hour. " - "The fox is centered in the frame, its thick orange winter coat contrasting with the fresh white snow, breath " - "visible in the cold air. Snow-laden pine trees fill the softly blurred background, and low warm sunlight rakes " - "across the scene from the left, casting long blue shadows." -) -EDIT_PROMPT = ( - "The same red fox in the same snow-covered forest clearing, now at night under a full moon: cool blue moonlight, " - "deep shadows, stars visible through the pine branches, the fox's eyes catching the light." -) -# BFL's layout-control examples (examples/generate.py): regions appended to the caption as JSON, references as -KNIGHT = os.path.join(REPO, "..", "bfl-f3-image", "examples", "assets", "knight.jpg") # BFL docs image, 1360 x 768 -BBOX_PROMPT = 'Minimalist graphic illustration featuring a black silhouette of a person centered against a solid, vibrant chartreuse background . The figure is captured in a dynamic, mid-stride running pose, facing toward the left side of the frame. [{"id": "background_1", "bbox": [0, 0, 1000, 1000], "desc": "A flat field of neon yellow-green with a subtle paper texture and fine, organic grain."}, {"id": "silhouette_1", "bbox": [150, 150, 850, 850], "desc": "A black silhouette of a person in motion, made of dense stippled ink. The leading edges are solid; the trailing arm and leg dissolve into coarse square pixels and scattered dots."}]' -KNIGHT_PROMPT = 'In , move the miniature grey amigurumi knight figure upwards and to the left along the yarn cliff . Keep the rest of the image completely unchanged, preserving the blurred yarn backdrop , the large knitted dragon , and the fiery orange thread . [{"id": "knight_1", "from": "ref_image_0", "src_bbox": [500, 150, 850, 350], "tgt_bbox": [194, 55, 544, 255], "desc": "A miniature amigurumi knight crocheted from thick grey woolen yarn."}, {"id": "backdrop_1", "from": "ref_image_0", "src_bbox": [0, 0, 1000, 1000], "tgt_bbox": [0, 0, 1000, 1000], "desc": "A soft, out-of-focus backdrop of cream and indigo knitted yarn."}, {"id": "cliff_1", "from": "ref_image_0", "src_bbox": [300, 0, 1000, 450], "tgt_bbox": [300, 0, 1000, 450], "desc": "A steep cliff of stacked yarn skeins in mustard, terracotta, and indigo."}, {"id": "needle_1", "from": "ref_image_0", "src_bbox": [420, 250, 550, 450], "tgt_bbox": [420, 250, 550, 450], "desc": "An oversized polished steel tapestry needle."}, {"id": "dragon_1", "from": "ref_image_0", "src_bbox": [100, 600, 950, 1000], "tgt_bbox": [100, 600, 950, 1000], "desc": "A large, fluffy knitted dragon in terracotta and mustard yarn."}, {"id": "fire_1", "from": "ref_image_0", "src_bbox": [350, 400, 650, 650], "tgt_bbox": [350, 400, 650, 650], "desc": "Curled puffs of fiery orange angora thread suspended in mid-air."}]' - -# name, prompt, reference image (None, the t2i output, or a file), size (None: the pipeline's default -- 1024 x 1024 -# for t2i, ~1 MP at the reference's aspect ratio for i2i) -CASES = [ - ("t2i", PROMPT, None, (HEIGHT, WIDTH)), - ("i2i", EDIT_PROMPT, "t2i", (HEIGHT, WIDTH)), - ("bbox_t2i", BBOX_PROMPT, None, None), - ("bbox_i2i", KNIGHT_PROMPT, KNIGHT, None), -] - - -def reference(): - sys.path.insert(0, BFL_CODE) - from f3.core.hub import resolve_package - from f3.image_small.pipeline import ImageSmallPipeline, default_size, prepare_reference, to_pil, to_tensor - - package = resolve_package(PACKAGE, variant=VARIANT, verify=False) - pipe = ImageSmallPipeline.from_package(package, device="cuda") - for name, prompt, image, size in CASES: - references = [] - if image is not None: - path = os.path.join(OUT, "t2i_reference.png") if image == "t2i" else image - references = [prepare_reference(PIL.Image.open(path))] - width, height = (size[1], size[0]) if size else default_size(references, package.config) - # defaults from the package config: 50 steps, guidance 3.5, alpha 6.93 (sd: 16 steps, no guidance) - result = to_pil(pipe.generate(prompt, [to_tensor(r) for r in references], width, height, seed=SEED)) - result.save(os.path.join(OUT, f"{name}_reference.png")) - print(f"{name}: {result.size}") - - -def diffusers(): - sys.path.insert(0, os.path.join(REPO, "src")) - from diffusers import AutoencoderKLFlux2, ModularPipeline - from diffusers.models.transformers import transformer_flux3 - - print(f"diffusers source: {transformer_flux3.__file__}") - - pipe = ModularPipeline.from_pretrained(MODULAR_REPO) - pipe.load_components(dtype=torch.bfloat16) - pipe.update_components(vae=AutoencoderKLFlux2.from_pretrained(MODULAR_REPO, subfolder="vae", dtype=torch.float32)) - pipe.to("cuda") - - for name, prompt, image, size in CASES: - inputs = {"prompt": prompt} - if image is not None: - path = os.path.join(OUT, "t2i_reference.png") if image == "t2i" else image - inputs["image"] = PIL.Image.open(path) - if size is not None: - inputs["height"], inputs["width"] = size - # the reference draws the noise token-major on the CPU from manual_seed(SEED) and casts to bf16 - # (f3.image_small.pipeline.noise_tokens); same draw here, reshaped to the (1, C, h, w) latents diffusers - # takes. The size is the one the pipeline resolves (the reference's png has it when derived). - height, width = size or PIL.Image.open(os.path.join(OUT, f"{name}_reference.png")).size[::-1] - h, w = height // 16, width // 16 - noise = torch.randn((h * w, 128), generator=torch.Generator().manual_seed(SEED)).to(torch.bfloat16) - inputs["latents"] = noise.T.reshape(1, 128, h, w) - - result = pipe(**inputs, output="images")[0] - result.save(os.path.join(OUT, f"{name}_diffusers.png")) - print(f"{name}: {result.size}") - - for name, _, _, _ in CASES: - a = PIL.Image.open(os.path.join(OUT, f"{name}_reference.png")) - b = PIL.Image.open(os.path.join(OUT, f"{name}_diffusers.png")) - diff = np.abs(np.asarray(a, dtype=np.float32) - np.asarray(b, dtype=np.float32)) - print(f"{name}: mean pixel diff {diff.mean():.2f}/255, max {diff.max():.0f}/255") - side_by_side = PIL.Image.new("RGB", (a.width * 2, a.height)) - side_by_side.paste(a, (0, 0)) - side_by_side.paste(b, (a.width, 0)) - side_by_side.save(os.path.join(OUT, f"{name}_side_by_side.png")) # reference left, diffusers right - - -if __name__ == "__main__": - os.makedirs(OUT, exist_ok=True) - {"reference": reference, "diffusers": diffusers}[sys.argv[1]]() diff --git a/src/diffusers/__init__.py b/src/diffusers/__init__.py index 35ee907960c4..287a9c8af934 100644 --- a/src/diffusers/__init__.py +++ b/src/diffusers/__init__.py @@ -134,6 +134,18 @@ else: _import_structure["quantizers.quantization_config"].append("NunchakuLiteQuantizationConfig") +try: + if not is_torch_available(): + raise OptionalDependencyNotAvailable() +except OptionalDependencyNotAvailable: + from .utils import dummy_bfl_objects + + _import_structure["utils.dummy_bfl_objects"] = [ + name for name in dir(dummy_bfl_objects) if not name.startswith("_") + ] +else: + _import_structure["quantizers.quantization_config"].append("BFLQuantizationConfig") + try: if not is_auto_round_available(): raise OptionalDependencyNotAvailable() @@ -1037,6 +1049,14 @@ else: from .quantizers.quantization_config import NunchakuLiteQuantizationConfig + try: + if not is_torch_available(): + raise OptionalDependencyNotAvailable() + except OptionalDependencyNotAvailable: + from .utils.dummy_bfl_objects import * + else: + from .quantizers.quantization_config import BFLQuantizationConfig + try: if not is_auto_round_available(): raise OptionalDependencyNotAvailable() diff --git a/src/diffusers/loaders/single_file_model.py b/src/diffusers/loaders/single_file_model.py index 44aa403b1e77..b796dc4a9c59 100644 --- a/src/diffusers/loaders/single_file_model.py +++ b/src/diffusers/loaders/single_file_model.py @@ -399,9 +399,12 @@ def from_single_file(cls, pretrained_model_link_or_path_or_dict: str | None = No user_agent=user_agent, ) if quantization_config is not None: + if not low_cpu_mem_usage: + raise ValueError("`low_cpu_mem_usage` cannot be False when using quantization.") hf_quantizer = DiffusersAutoQuantizer.from_config(quantization_config) hf_quantizer.validate_environment() torch_dtype = hf_quantizer.update_torch_dtype(torch_dtype) + checkpoint = hf_quantizer.maybe_update_state_dict(checkpoint) else: hf_quantizer = None diff --git a/src/diffusers/loaders/single_file_utils.py b/src/diffusers/loaders/single_file_utils.py index 329238f08132..e3a1eb1888ab 100644 --- a/src/diffusers/loaders/single_file_utils.py +++ b/src/diffusers/loaders/single_file_utils.py @@ -153,6 +153,7 @@ "model.diffusion_model.early_stream_modulations.image.lin.weight", "early_stream_modulations.image.lin.weight", ], + "flux3-distilled": ["model.diffusion_model.ref_time_in.in_layer.weight", "ref_time_in.in_layer.weight"], "chroma": [ "model.diffusion_model.distilled_guidance_layer.in_proj.bias", "distilled_guidance_layer.in_proj.bias", @@ -213,6 +214,7 @@ "flux-schnell": {"pretrained_model_name_or_path": "black-forest-labs/FLUX.1-schnell"}, "flux-2-dev": {"pretrained_model_name_or_path": "black-forest-labs/FLUX.2-dev"}, "flux-3-image": {"pretrained_model_name_or_path": "diffusers-internal-dev/f3-small-img"}, + "flux-3-image-distilled": {"pretrained_model_name_or_path": "diffusers-internal-dev/f3-small-image-distilled"}, "chroma": {"pretrained_model_name_or_path": "lodestones/Chroma1-HD"}, "ltx-video": {"pretrained_model_name_or_path": "diffusers/LTX-Video-0.9.0"}, "ltx-video-0.9.1": {"pretrained_model_name_or_path": "diffusers/LTX-Video-0.9.1"}, @@ -694,7 +696,11 @@ def infer_diffusers_model_type(checkpoint): model_type = "flux-2-dev" elif any(key in checkpoint for key in CHECKPOINT_KEY_NAMES["flux3"]): - model_type = "flux-3-image" + # the step-distilled (flow-map) checkpoint carries a second timestep embedder + if any(key in checkpoint for key in CHECKPOINT_KEY_NAMES["flux3-distilled"]): + model_type = "flux-3-image-distilled" + else: + model_type = "flux-3-image" elif any(key in checkpoint for key in CHECKPOINT_KEY_NAMES["chroma"]): model_type = "chroma" diff --git a/src/diffusers/quantizers/auto.py b/src/diffusers/quantizers/auto.py index ea6caf91ab80..377b46b7b361 100644 --- a/src/diffusers/quantizers/auto.py +++ b/src/diffusers/quantizers/auto.py @@ -19,12 +19,14 @@ import warnings from .autoround import AutoRoundQuantizer +from .bfl import BFLQuantizer from .bitsandbytes import BnB4BitDiffusersQuantizer, BnB8BitDiffusersQuantizer from .gguf import GGUFQuantizer from .modelopt import NVIDIAModelOptQuantizer from .nunchaku import NunchakuLiteQuantizer from .quantization_config import ( AutoRoundConfig, + BFLQuantizationConfig, BitsAndBytesConfig, GGUFQuantizationConfig, NunchakuLiteQuantizationConfig, @@ -41,6 +43,7 @@ AUTO_QUANTIZER_MAPPING = { + "bfl": BFLQuantizer, "bitsandbytes_4bit": BnB4BitDiffusersQuantizer, "bitsandbytes_8bit": BnB8BitDiffusersQuantizer, "gguf": GGUFQuantizer, @@ -53,6 +56,7 @@ } AUTO_QUANTIZATION_CONFIG_MAPPING = { + "bfl": BFLQuantizationConfig, "bitsandbytes_4bit": BitsAndBytesConfig, "bitsandbytes_8bit": BitsAndBytesConfig, "gguf": GGUFQuantizationConfig, diff --git a/src/diffusers/quantizers/bfl/__init__.py b/src/diffusers/quantizers/bfl/__init__.py new file mode 100644 index 000000000000..d5294d03a420 --- /dev/null +++ b/src/diffusers/quantizers/bfl/__init__.py @@ -0,0 +1 @@ +from .bfl_quantizer import BFLQuantizer diff --git a/src/diffusers/quantizers/bfl/bfl_quantizer.py b/src/diffusers/quantizers/bfl/bfl_quantizer.py new file mode 100644 index 000000000000..8802144fa593 --- /dev/null +++ b/src/diffusers/quantizers/bfl/bfl_quantizer.py @@ -0,0 +1,152 @@ +# Copyright 2026 The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +from ...utils import get_module_from_name, is_accelerate_available, is_accelerate_version, is_torch_available +from ..base import DiffusersQuantizer + + +if TYPE_CHECKING: + from ...models.modeling_utils import ModelMixin + + +if is_torch_available(): + import torch + import torch.nn as nn + + from .utils import ( + BFLFP8RLinear, + BFLNVFP4Linear, + BFLQuantizedParameter, + _dequantize_bfl_and_restore_linear, + _dequantize_weight, + _replace_with_bfl_linear, + ) + + +class BFLQuantizer(DiffusersQuantizer): + use_keep_in_fp32_modules = True + + def __init__(self, quantization_config, **kwargs): + super().__init__(quantization_config, **kwargs) + self.compute_dtype = quantization_config.compute_dtype + self.keep_in_fp32_modules = [] + + def validate_environment(self, *args, **kwargs): + if not is_accelerate_available() or is_accelerate_version("<", "0.26.0"): + raise ImportError( + "Loading BFL quantized checkpoints requires `accelerate` installed in your environment: `pip install 'accelerate>=0.26.0'`" + ) + + def adjust_target_dtype(self, target_dtype: "torch.dtype") -> "torch.dtype": + return torch.uint8 + + def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype": + if torch_dtype is None: + torch_dtype = self.compute_dtype + return torch_dtype + + def maybe_update_state_dict(self, state_dict: dict[str, Any]) -> dict[str, Any]: + prefixes = [key[: -len(".weight_scale")] for key in state_dict if key.endswith(".weight_scale")] + if not prefixes: + raise ValueError("The checkpoint has no `weight_scale` tensors, so it is not an fp8r or nvfp4 checkpoint.") + for prefix in prefixes: + weight_scale = state_dict.pop(f"{prefix}.weight_scale") + weight_scale_2 = state_dict.pop(f"{prefix}.weight_scale_2", None) + input_scale = state_dict.pop(f"{prefix}.input_scale", None) + quant_type = "nvfp4" if weight_scale_2 is not None else "fp8r" + weight = state_dict[f"{prefix}.weight"] + expected_dtype = torch.uint8 if quant_type == "nvfp4" else torch.float8_e4m3fn + if weight.dtype != expected_dtype: + raise ValueError( + f"{prefix}.weight is {weight.dtype}, but the {quant_type} layout expects {expected_dtype}." + ) + if quant_type == "nvfp4" and input_scale is None: + raise ValueError( + f"{prefix} has `weight_scale_2` but no `input_scale`, which the nvfp4 layout requires." + ) + state_dict[f"{prefix}.weight"] = BFLQuantizedParameter( + weight, quant_type, weight_scale, weight_scale_2, input_scale + ) + return state_dict + + def check_if_quantized_param( + self, + model: "ModelMixin", + param_value: "torch.Tensor", + param_name: str, + state_dict: dict[str, Any], + **kwargs, + ) -> bool: + return isinstance(param_value, BFLQuantizedParameter) + + def create_quantized_param( + self, + model: "ModelMixin", + param_value: "BFLQuantizedParameter", + param_name: str, + target_device: "torch.device", + state_dict: dict[str, Any] | None = None, + unexpected_keys: list[str] | None = None, + **kwargs, + ): + module, tensor_name = get_module_from_name(model, param_name) + if isinstance(module, (BFLFP8RLinear, BFLNVFP4Linear)): + module.weight = nn.Parameter(param_value.as_tensor().to(target_device), requires_grad=False) + module.weight_scale = param_value.weight_scale.to(target_device) + if isinstance(module, BFLNVFP4Linear): + module.weight_scale_2 = param_value.weight_scale_2.to(target_device) + module.input_scale = param_value.input_scale.to(target_device) + return + + # The module was excluded from quantization (for example by `_keep_in_fp32_modules`), so it stays a + # regular `nn.Linear` and receives the dequantized weight. + keep_in_fp32 = any(m in param_name.split(".") for m in self.keep_in_fp32_modules) + dtype = torch.float32 if keep_in_fp32 else self.compute_dtype + module._parameters[tensor_name] = nn.Parameter( + _dequantize_weight(param_value).to(target_device, dtype), requires_grad=False + ) + + def _process_model_before_weight_loading( + self, + model: "ModelMixin", + device_map, + keep_in_fp32_modules: list[str] = [], + **kwargs, + ): + self.keep_in_fp32_modules = [module for module in keep_in_fp32_modules if module is not None] + _replace_with_bfl_linear( + model, self.compute_dtype, kwargs.get("state_dict"), modules_to_not_convert=self.keep_in_fp32_modules + ) + + def _process_model_after_weight_loading(self, model: "ModelMixin", **kwargs): + return model + + @property + def is_serializable(self): + return False + + @property + def is_trainable(self) -> bool: + return False + + @property + def is_compileable(self) -> bool: + return True + + def _dequantize(self, model): + return _dequantize_bfl_and_restore_linear(model) diff --git a/src/diffusers/quantizers/bfl/utils.py b/src/diffusers/quantizers/bfl/utils.py new file mode 100644 index 000000000000..cb96f84852a9 --- /dev/null +++ b/src/diffusers/quantizers/bfl/utils.py @@ -0,0 +1,310 @@ +# Copyright 2026 The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import inspect + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from ...utils import is_accelerate_available, is_flashinfer_available + + +if is_accelerate_available(): + import accelerate + from accelerate import init_empty_weights + from accelerate.hooks import add_hook_to_module, remove_hook_from_module + +if is_flashinfer_available(): + import flashinfer + + +FP8_MAX = 448.0 +E2M1_MAX = 6.0 +NVFP4_BLOCK = 16 +# E2M1 values indexed by their 4-bit code: bit 3 is the sign, bits 2-0 the magnitude. +E2M1_LUT = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0]) + + +def quantize_fp8r(weight: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """`[N, K]` float weight -> E4M3 weight `[N, K]` and fp32 per-row scales `[N]`.""" + weight = weight.float() + weight_scale = (weight.abs().amax(dim=1) / FP8_MAX).clamp(min=1e-12) + weight_q = (weight / weight_scale[:, None]).clamp(-FP8_MAX, FP8_MAX).to(torch.float8_e4m3fn) + return weight_q, weight_scale + + +def dequantize_fp8r(weight: torch.Tensor, weight_scale: torch.Tensor) -> torch.Tensor: + return weight.float() * weight_scale[:, None] + + +def quantize_nvfp4(weight: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """`[N, K]` float weight -> packed E2M1 weight `[N, K/2]` (even element in the low nibble), E4M3 block scales + `[N, K/16]` and the fp32 global scale `weight_scale_2 = amax / (448 * 6)`, following the ModelOpt NVFP4 layout.""" + weight = weight.float() + out_features, in_features = weight.shape + weight_scale_2 = weight.abs().amax() / (FP8_MAX * E2M1_MAX) + blocks = weight.reshape(out_features, in_features // NVFP4_BLOCK, NVFP4_BLOCK) + weight_scale = (blocks.abs().amax(dim=-1) / E2M1_MAX / weight_scale_2).to(torch.float8_e4m3fn) + scale = (weight_scale.float() * weight_scale_2)[..., None] + scaled = torch.where(scale > 0, blocks / scale, torch.zeros_like(blocks)) + magnitude = scaled.abs().clamp(max=E2M1_MAX) + # Round to nearest even on the E2M1 grid, whose spacing is 0.5 below 2, 1 below 4 and 2 above. + magnitude = torch.where( + magnitude < 2, + torch.round(magnitude * 2) / 2, + torch.where(magnitude < 4, torch.round(magnitude), torch.round(magnitude / 2) * 2), + ) + codes = torch.where(magnitude < 2, magnitude * 2, torch.where(magnitude < 4, magnitude + 2, magnitude / 2 + 4)) + codes = codes.to(torch.uint8) | ((scaled < 0).to(torch.uint8) << 3) + codes = codes.reshape(out_features, in_features) + weight_q = codes[:, 0::2] | (codes[:, 1::2] << 4) + return weight_q, weight_scale, weight_scale_2 + + +def dequantize_nvfp4(weight: torch.Tensor, weight_scale: torch.Tensor, weight_scale_2: torch.Tensor) -> torch.Tensor: + codes = torch.stack([weight & 0x0F, weight >> 4], dim=-1).reshape(weight.shape[0], -1) + values = E2M1_LUT.to(weight.device)[codes.long()] + scale = (weight_scale.float() * weight_scale_2).repeat_interleave(NVFP4_BLOCK, dim=1) + return values * scale + + +class BFLQuantizedParameter(nn.Parameter): + """A quantized linear weight bundled with its scales. + + Single-file checkpoint converters chunk, split and concatenate weights along the output dimension (fused QKV, + swapped scale/shift). Bundling keeps the per-row scales aligned with the rows through those operations. A cast to + a floating point dtype dequantizes the weight into a plain tensor (the loader casts the parameters of modules kept + in fp32); any other shape or dtype change is refused. + """ + + def __new__(cls, data, quant_type, weight_scale, weight_scale_2=None, input_scale=None): + self = torch.Tensor._make_subclass(cls, data, False) + self.quant_type = quant_type + self.weight_scale = weight_scale + self.weight_scale_2 = weight_scale_2 + self.input_scale = input_scale + return self + + def as_tensor(self): + return torch.Tensor._make_subclass(torch.Tensor, self, False) + + def _like(self, data, weight_scale): + return BFLQuantizedParameter(data, self.quant_type, weight_scale, self.weight_scale_2, self.input_scale) + + def __repr__(self): + return f"BFLQuantizedParameter(quant_type={self.quant_type!r}, data={self.as_tensor()!r})" + + @classmethod + def __torch_function__(cls, func, types, args=(), kwargs=None): + kwargs = kwargs or {} + if func in (torch.chunk, torch.Tensor.chunk, torch.split, torch.Tensor.split): + source = args[0] + dim = kwargs.get("dim", args[2] if len(args) > 2 else 0) + if dim != 0: + raise NotImplementedError(f"{func.__name__} along dim={dim} is not supported on quantized weights.") + with torch._C.DisableTorchFunctionSubclass(): + weights = func(*args, **kwargs) + scales = func(source.weight_scale, *args[1:], **kwargs) + return tuple(source._like(w, s) for w, s in zip(weights, scales)) + + if func in (torch.cat, torch.concat, torch.concatenate): + tensors = args[0] + dim = kwargs.get("dim", args[1] if len(args) > 1 else 0) + source = tensors[0] + compatible = all( + isinstance(t, cls) + and t.quant_type == source.quant_type + and all( + (getattr(t, name) is None and getattr(source, name) is None) + or torch.equal(getattr(t, name), getattr(source, name)) + for name in ("weight_scale_2", "input_scale") + ) + for t in tensors + ) + if dim != 0 or not compatible: + raise NotImplementedError( + "Only quantized weights of the same scheme and global scales can be concatenated along dim=0." + ) + with torch._C.DisableTorchFunctionSubclass(): + weight = torch.cat(tensors, dim=0) + weight_scale = torch.cat([t.weight_scale for t in tensors], dim=0) + return source._like(weight, weight_scale) + + source = next(a for a in torch.utils._pytree.tree_leaves(args) if isinstance(a, cls)) + with torch._C.DisableTorchFunctionSubclass(): + result = func(*args, **kwargs) + if not isinstance(result, torch.Tensor) or func in torch.overrides.get_default_nowrap_functions(): + return result + if result.shape == source.shape and result.dtype != source.dtype and result.is_floating_point(): + return _dequantize_weight(source).to(result.device, result.dtype) + if result.shape != source.shape or result.dtype != source.dtype: + raise NotImplementedError( + f"{getattr(func, '__name__', func)} changes the shape or dtype of a quantized weight, which is not supported." + ) + weight_scale = source.weight_scale.to(result.device) + return source._like(result, weight_scale) + + +class BFLFP8RLinear(nn.Linear): + """Linear over an E4M3 weight `[N, K]` with fp32 per-row scales `[N]`. + + On CUDA devices with FP8 support (compute capability 8.9+) activations are quantized per row and the GEMM runs + through `torch._scaled_mm`; elsewhere the weight is dequantized to `compute_dtype`. + """ + + def __init__(self, in_features, out_features, bias=True, compute_dtype=None, device=None): + super().__init__(in_features, out_features, bias, device=device) + self.weight = nn.Parameter( + torch.empty(out_features, in_features, dtype=torch.float8_e4m3fn, device=device), requires_grad=False + ) + self.register_buffer("weight_scale", torch.empty(out_features, dtype=torch.float32, device=device)) + self.compute_dtype = compute_dtype + + def forward(self, inputs: torch.Tensor) -> torch.Tensor: + if ( + inputs.is_cuda + and self.in_features % 16 == 0 + and self.out_features % 16 == 0 + and torch.cuda.get_device_capability(inputs.device) >= (8, 9) + ): + x = inputs.reshape(-1, self.in_features).float() + x_scale = (x.abs().amax(dim=1, keepdim=True) / FP8_MAX).clamp(min=1e-12) + x_q = (x / x_scale).clamp(-FP8_MAX, FP8_MAX).to(torch.float8_e4m3fn) + output = torch._scaled_mm( + x_q, self.weight.T, x_scale, self.weight_scale[None, :], out_dtype=torch.bfloat16, use_fast_accum=True + ) + output = output.reshape(*inputs.shape[:-1], self.out_features).to(self.compute_dtype) + if self.bias is not None: + output = output + self.bias + return output + + weight = dequantize_fp8r(self.weight, self.weight_scale).to(self.compute_dtype) + return F.linear(inputs, weight, self.bias) + + +class BFLNVFP4Linear(nn.Linear): + """Linear over a packed E2M1 weight `[N, K/2]`, E4M3 block scales `[N, K/16]`, the fp32 global weight scale + `weight_scale_2` and the calibrated fp32 activation scale `input_scale`. + + On Blackwell GPUs with FlashInfer installed activations are quantized to NVFP4 with the static `input_scale` and + the GEMM runs through `flashinfer.mm_fp4`; elsewhere the weight is dequantized to `compute_dtype`. + """ + + def __init__(self, in_features, out_features, bias=True, compute_dtype=None, device=None): + super().__init__(in_features, out_features, bias, device=device) + self.weight = nn.Parameter( + torch.empty(out_features, in_features // 2, dtype=torch.uint8, device=device), requires_grad=False + ) + self.register_buffer( + "weight_scale", + torch.empty(out_features, in_features // NVFP4_BLOCK, dtype=torch.float8_e4m3fn, device=device), + ) + self.register_buffer("weight_scale_2", torch.empty((), dtype=torch.float32, device=device)) + self.register_buffer("input_scale", torch.empty((), dtype=torch.float32, device=device)) + self.compute_dtype = compute_dtype + + def forward(self, inputs: torch.Tensor) -> torch.Tensor: + if inputs.is_cuda and is_flashinfer_available() and torch.cuda.get_device_capability(inputs.device)[0] >= 10: + x = inputs.reshape(-1, self.in_features).to(torch.bfloat16).contiguous() + x_q, x_scale = flashinfer.nvfp4_quantize( + x, (1.0 / self.input_scale).reshape(1), sfLayout=flashinfer.SfLayout.layout_128x4, do_shuffle=False + ) + rows, cols = self.weight_scale.shape + weight_scale = flashinfer.block_scale_interleave(self.weight_scale.view(torch.uint8).contiguous()) + weight_scale = weight_scale.reshape(-(-rows // 128) * 128, -(-cols // 4) * 4).view(torch.float8_e4m3fn) + output = flashinfer.mm_fp4( + x_q, + self.weight.T, + x_scale, + weight_scale.T, + alpha=(self.input_scale * self.weight_scale_2).reshape(1), + out_dtype=torch.bfloat16, + backend="cutlass", + ) + output = output.reshape(*inputs.shape[:-1], self.out_features).to(self.compute_dtype) + if self.bias is not None: + output = output + self.bias + return output + + weight = dequantize_nvfp4(self.weight, self.weight_scale, self.weight_scale_2).to(self.compute_dtype) + return F.linear(inputs, weight, self.bias) + + +QUANTIZED_LINEARS = {"fp8r": BFLFP8RLinear, "nvfp4": BFLNVFP4Linear} + + +def _dequantize_weight(param: BFLQuantizedParameter) -> torch.Tensor: + if param.quant_type == "nvfp4": + return dequantize_nvfp4(param.as_tensor(), param.weight_scale, param.weight_scale_2) + return dequantize_fp8r(param.as_tensor(), param.weight_scale) + + +def _replace_with_bfl_linear(model, compute_dtype, state_dict, prefix="", modules_to_not_convert=[]): + for name, module in model.named_children(): + if name in modules_to_not_convert: + continue + module_prefix = prefix + name + "." + _replace_with_bfl_linear(module, compute_dtype, state_dict, module_prefix, modules_to_not_convert) + + weight = state_dict.get(module_prefix + "weight") + if isinstance(module, nn.Linear) and isinstance(weight, BFLQuantizedParameter): + with init_empty_weights(): + model._modules[name] = QUANTIZED_LINEARS[weight.quant_type]( + module.in_features, module.out_features, module.bias is not None, compute_dtype=compute_dtype + ) + return model + + +# Copied from diffusers.quantizers.bitsandbytes.utils._create_accelerate_new_hook +def _create_accelerate_new_hook(old_hook): + r""" + Creates a new hook based on the old hook. Use it only if you know what you are doing ! This method is a copy of: + https://github.com/huggingface/peft/blob/748f7968f3a31ec06a1c2b0328993319ad9a150a/src/peft/utils/other.py#L245 with + some changes + """ + old_hook_cls = getattr(accelerate.hooks, old_hook.__class__.__name__) + old_hook_attr = old_hook.__dict__ + filtered_old_hook_attr = {} + old_hook_init_signature = inspect.signature(old_hook_cls.__init__) + for k in old_hook_attr.keys(): + if k in old_hook_init_signature.parameters: + filtered_old_hook_attr[k] = old_hook_attr[k] + new_hook = old_hook_cls(**filtered_old_hook_attr) + return new_hook + + +def _dequantize_bfl_and_restore_linear(model): + for name, module in model.named_children(): + if isinstance(module, (BFLFP8RLinear, BFLNVFP4Linear)): + if isinstance(module, BFLNVFP4Linear): + weight = dequantize_nvfp4(module.weight, module.weight_scale, module.weight_scale_2) + else: + weight = dequantize_fp8r(module.weight, module.weight_scale) + device = module.weight.device + with init_empty_weights(): + new_module = nn.Linear(module.in_features, module.out_features, module.bias is not None) + new_module.weight = nn.Parameter(weight.to(module.compute_dtype), requires_grad=False) + if module.bias is not None: + new_module.bias = module.bias + + if hasattr(module, "_hf_hook"): + new_hook = _create_accelerate_new_hook(module._hf_hook) + remove_hook_from_module(module) + add_hook_to_module(new_module, new_hook) + + model._modules[name] = new_module.to(device) + else: + _dequantize_bfl_and_restore_linear(module) + return model diff --git a/src/diffusers/quantizers/quantization_config.py b/src/diffusers/quantizers/quantization_config.py index 30e89f53f906..5777ab45fa7a 100644 --- a/src/diffusers/quantizers/quantization_config.py +++ b/src/diffusers/quantizers/quantization_config.py @@ -44,6 +44,7 @@ class QuantizationMethod(str, Enum): BITS_AND_BYTES = "bitsandbytes" + BFL = "bfl" GGUF = "gguf" NUNCHAKU_LITE = "nunchaku_lite" TORCHAO = "torchao" @@ -431,6 +432,30 @@ def __init__(self, compute_dtype: "torch.dtype" | None = None): self.compute_dtype = torch.float32 +@dataclass +class BFLQuantizationConfig(QuantizationConfigMixin): + """Config for loading checkpoints quantized with the Black Forest Labs `fp8r` and `nvfp4` schemes. + + Both schemes are read from the checkpoint through [`~FromSingleFileMixin.from_single_file`]; quantizing a model on + the fly is not supported. A linear layer is quantized when its weight comes with scale tensors: + + - `fp8r`: `weight` is `float8_e4m3fn` with shape `[N, K]` and `weight_scale` is an fp32 per-row scale `[N]`. + - `nvfp4`: `weight` is `uint8` with shape `[N, K/2]` (two E2M1 values per byte), `weight_scale` is a + `float8_e4m3fn` block scale `[N, K/16]`, and `weight_scale_2` and `input_scale` are fp32 scalars holding the + global weight scale and the calibrated activation scale (the ModelOpt NVFP4 layout). + + Args: + compute_dtype (`torch.dtype`, defaults to `torch.bfloat16`): + The dtype of the activations and of the unquantized layers. Devices without fused FP8 or NVFP4 kernels + dequantize the weights to this dtype in every forward pass. + """ + + def __init__(self, compute_dtype: "torch.dtype" | None = None): + self.quant_method = QuantizationMethod.BFL + self.compute_dtype = compute_dtype if compute_dtype is not None else torch.bfloat16 + self.pre_quantized = True + + @dataclass class NunchakuLiteQuantizationConfig(QuantizationConfigMixin): """Configuration for loading Nunchaku Lite checkpoints. diff --git a/src/diffusers/utils/__init__.py b/src/diffusers/utils/__init__.py index 5c63a4bc7661..ed07fa585e62 100644 --- a/src/diffusers/utils/__init__.py +++ b/src/diffusers/utils/__init__.py @@ -76,6 +76,7 @@ is_flash_attn_3_available, is_flash_attn_available, is_flash_attn_version, + is_flashinfer_available, is_flashpack_available, is_ftfy_available, is_gguf_available, diff --git a/src/diffusers/utils/dummy_bfl_objects.py b/src/diffusers/utils/dummy_bfl_objects.py new file mode 100644 index 000000000000..77fd7880e901 --- /dev/null +++ b/src/diffusers/utils/dummy_bfl_objects.py @@ -0,0 +1,17 @@ +# This file is autogenerated by the command `make fix-copies`, do not edit. +from ..utils import DummyObject, requires_backends + + +class BFLQuantizationConfig(metaclass=DummyObject): + _backends = ["torch"] + + def __init__(self, *args, **kwargs): + requires_backends(self, ["torch"]) + + @classmethod + def from_config(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) + + @classmethod + def from_pretrained(cls, *args, **kwargs): + requires_backends(cls, ["torch"]) diff --git a/src/diffusers/utils/import_utils.py b/src/diffusers/utils/import_utils.py index d2cf394cd9a7..94ebb064a12d 100644 --- a/src/diffusers/utils/import_utils.py +++ b/src/diffusers/utils/import_utils.py @@ -213,6 +213,7 @@ def _is_package_available(pkg_name: str, get_dist_name: bool = False) -> tuple[b _cosmos_guardrail_available, _cosmos_guardrail_version = _is_package_available("cosmos_guardrail") _sageattention_available, _sageattention_version = _is_package_available("sageattention") _flash_attn_available, _flash_attn_version = _is_package_available("flash_attn") +_flashinfer_available, _flashinfer_version = _is_package_available("flashinfer", get_dist_name=True) _flash_attn_3_available, _flash_attn_3_version = _is_package_available("flash_attn_3") _kornia_available, _kornia_version = _is_package_available("kornia") _nvidia_modelopt_available, _nvidia_modelopt_version = _is_package_available("modelopt", get_dist_name=True) @@ -354,6 +355,10 @@ def is_gguf_available(): return _gguf_available +def is_flashinfer_available(): + return _flashinfer_available + + def is_flashpack_available(): return _flashpack_available diff --git a/tests/conftest.py b/tests/conftest.py index d0401281e37d..eaebf335c94b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -47,6 +47,7 @@ def pytest_configure(config): config.addinivalue_line("markers", "gguf: marks tests for GGUF quantization functionality") config.addinivalue_line("markers", "modelopt: marks tests for NVIDIA ModelOpt quantization functionality") config.addinivalue_line("markers", "sdnq: marks tests for SDNQ quantization functionality") + config.addinivalue_line("markers", "bfl: marks tests for BFL (fp8r / nvfp4) quantization functionality") config.addinivalue_line("markers", "context_parallel: marks tests for context parallel inference functionality") config.addinivalue_line("markers", "tensor_parallel: marks tests for tensor parallel inference functionality") config.addinivalue_line("markers", "slow: mark test as slow") diff --git a/tests/models/testing_utils/__init__.py b/tests/models/testing_utils/__init__.py index 760a3fac04e0..d1f318564b7c 100644 --- a/tests/models/testing_utils/__init__.py +++ b/tests/models/testing_utils/__init__.py @@ -26,6 +26,9 @@ AutoRoundCompileTesterMixin, AutoRoundConfigMixin, AutoRoundTesterMixin, + BFLCompileTesterMixin, + BFLConfigMixin, + BFLTesterMixin, BitsAndBytesCompileTesterMixin, BitsAndBytesConfigMixin, BitsAndBytesTesterMixin, @@ -57,6 +60,9 @@ "AttentionTesterMixin", "AutoRoundConfigMixin", "AutoRoundTesterMixin", + "BFLCompileTesterMixin", + "BFLConfigMixin", + "BFLTesterMixin", "BaseModelTesterConfig", "BitsAndBytesCompileTesterMixin", "BitsAndBytesConfigMixin", diff --git a/tests/models/testing_utils/quantization.py b/tests/models/testing_utils/quantization.py index 8285704ec2a5..2e1a846887ff 100644 --- a/tests/models/testing_utils/quantization.py +++ b/tests/models/testing_utils/quantization.py @@ -14,6 +14,7 @@ # limitations under the License. import gc +import tempfile import pytest import safetensors.torch @@ -21,6 +22,7 @@ from diffusers import ( AutoRoundConfig, + BFLQuantizationConfig, BitsAndBytesConfig, GGUFQuantizationConfig, NunchakuLiteQuantizationConfig, @@ -28,8 +30,10 @@ SDNQConfig, TorchAoConfig, ) +from diffusers.loaders.single_file_model import SINGLE_FILE_LOADABLE_CLASSES, _get_single_file_loadable_mapping_class from diffusers.utils.import_utils import ( is_bitsandbytes_available, + is_flashinfer_available, is_gguf_available, is_kernels_available, is_nvidia_modelopt_available, @@ -41,6 +45,7 @@ assert_tensors_close, backend_empty_cache, is_autoround, + is_bfl, is_bitsandbytes, is_gguf, is_modelopt, @@ -58,6 +63,7 @@ require_torchao_version_greater_or_equal, torch_device, ) +from .single_file import fetch_checkpoint_metadata if is_peft_available(): @@ -1765,3 +1771,284 @@ def test_autoround_torch_compile(self): def test_autoround_torch_compile_with_group_offload(self): self._test_torch_compile_with_group_offload(self.config_dict) + + +class BFLConfigMixin: + """ + Base mixin providing BFL (`fp8r` / `nvfp4`) quantization config and model creation. + + BFL checkpoints are prequantized single files, one per scheme, and the tests never download their weights. The + safetensors header of each checkpoint says which linears the exporter quantized; the tests quantize those same + linears of the small test model with the library's own `quantize_fp8r` / `quantize_nvfp4` and load the result + through [`~FromOriginalModelMixin.from_single_file`], exactly as the real checkpoint is loaded. + + Expected class attributes: + - model_class: The model class to test + + Expected methods to be implemented by subclasses: + - get_init_dict(): Returns the init kwargs of the small test model + - get_dummy_inputs(): Returns dict of inputs to pass to the model forward pass + + Required properties (must be implemented by subclasses): + - bfl_checkpoints: Dict mapping a scheme (`"fp8r"`, `"nvfp4"`) to the URL or path of its single-file checkpoint + """ + + BFL_CONFIGS = { + "fp8r": {"scheme": "fp8r", "compute_dtype": torch.bfloat16}, + "nvfp4": {"scheme": "nvfp4", "compute_dtype": torch.bfloat16}, + } + + @property + def bfl_checkpoints(self) -> dict[str, str]: + """Scheme -> single-file checkpoint. Must be implemented by subclasses.""" + raise NotImplementedError("Subclasses must implement the `bfl_checkpoints` property.") + + @staticmethod + def _bfl_linear_cls(scheme): + from diffusers.quantizers.bfl.utils import BFLFP8RLinear, BFLNVFP4Linear + + return {"fp8r": BFLFP8RLinear, "nvfp4": BFLNVFP4Linear}[scheme] + + def _bfl_quantized_modules(self, scheme) -> set[str]: + """The diffusers names of the linears the checkpoint quantizes, read from its safetensors header.""" + metadata = fetch_checkpoint_metadata(self.bfl_checkpoints[scheme]) + if metadata is None: + pytest.skip(f"{self.bfl_checkpoints[scheme]} is not a safetensors checkpoint") + + # one `weight_scale` per quantized linear; the model's single-file mapping renames them to diffusers keys + weights = { + f"{name[: -len('.weight_scale')]}.weight" for name in metadata.tensors if name.endswith(".weight_scale") + } + checkpoint = { + name: torch.empty(tuple(metadata.tensors[name].shape), device="meta") for name in sorted(weights) + } + mapping_fn = SINGLE_FILE_LOADABLE_CLASSES[_get_single_file_loadable_mapping_class(self.model_class)][ + "checkpoint_mapping_fn" + ] + converted = mapping_fn(checkpoint=checkpoint, config=None) + return {name[: -len(".weight")] for name in converted} + + def _bfl_quantized_linears(self, model, scheme) -> set[str]: + """The checkpoint's quantized linears that the small model has (it has fewer blocks than the checkpoint).""" + linears = {name for name, module in model.named_modules() if isinstance(module, torch.nn.Linear)} + return self._bfl_quantized_modules(scheme) & linears + + def _load_unquantized_model(self): + torch.manual_seed(0) + return self.model_class(**self.get_init_dict()).eval() + + @torch.no_grad() + def _bfl_quantized_state_dict(self, model, scheme): + from diffusers.quantizers.bfl.utils import E2M1_MAX, FP8_MAX, quantize_fp8r, quantize_nvfp4 + + quantized_modules = self._bfl_quantized_linears(model, scheme) + assert quantized_modules, "the small model has none of the linears the checkpoint quantizes" + # nvfp4 stores a calibrated activation scale per linear: record the input amax on the dummy inputs + amax = {} + if scheme == "nvfp4": + handles = [ + model.get_submodule(name).register_forward_pre_hook( + lambda module, inputs, name=name: amax.__setitem__(name, inputs[0].abs().amax().item()) + ) + for name in quantized_modules + ] + model.to(torch_device)(**self.get_dummy_inputs(), return_dict=False) + model.cpu() + for handle in handles: + handle.remove() + + state_dict = {} + for name, tensor in model.state_dict().items(): + module_name = name[: -len(".weight")] if name.endswith(".weight") else None + if module_name not in quantized_modules: + state_dict[name] = tensor + elif scheme == "fp8r": + state_dict[name], state_dict[f"{module_name}.weight_scale"] = quantize_fp8r(tensor) + else: + weight_q, weight_scale, weight_scale_2 = quantize_nvfp4(tensor) + state_dict[name] = weight_q + state_dict[f"{module_name}.weight_scale"] = weight_scale + state_dict[f"{module_name}.weight_scale_2"] = weight_scale_2 + # linears the dummy inputs do not reach (e.g. the reference-image blocks) take the largest scale seen + input_amax = amax.get(module_name, max(amax.values(), default=1.0)) + state_dict[f"{module_name}.input_scale"] = torch.tensor(input_amax / (FP8_MAX * E2M1_MAX)) + return state_dict + + def _create_quantized_model(self, config_kwargs, **extra_kwargs): + config_kwargs = dict(config_kwargs) + scheme = config_kwargs.pop("scheme") + compute_dtype = config_kwargs.get("compute_dtype", torch.bfloat16) + + model = self._load_unquantized_model().to(compute_dtype) + state_dict = self._bfl_quantized_state_dict(model, scheme) + kwargs = { + "quantization_config": BFLQuantizationConfig(**config_kwargs), + "dtype": compute_dtype, + "device": str(torch_device), + } + kwargs.update(extra_kwargs) + with tempfile.TemporaryDirectory() as config_dir: + model.save_config(config_dir) + return self.model_class.from_single_file(state_dict, config=config_dir, **kwargs) + + def _verify_if_layer_quantized(self, name, module, config_kwargs): + cls = self._bfl_linear_cls(config_kwargs["scheme"]) + assert isinstance(module, cls), f"Layer {name} is not a {cls.__name__}, got {type(module)}" + if config_kwargs["scheme"] == "fp8r": + assert module.weight.dtype == torch.float8_e4m3fn, f"{name} weight dtype should be float8_e4m3fn" + assert module.weight_scale.shape == (module.out_features,), f"{name} weight_scale is not per row" + else: + assert module.weight.dtype == torch.uint8, f"{name} weight dtype should be uint8 (packed E2M1)" + assert module.weight.shape == (module.out_features, module.in_features // 2), ( + f"{name} weight is not packed" + ) + assert module.weight_scale.dtype == torch.float8_e4m3fn, f"{name} block scales should be float8_e4m3fn" + assert module.weight_scale_2.ndim == 0 and module.input_scale.ndim == 0, f"{name} global scales" + + def _is_module_quantized(self, module, config_kwargs=None): + from diffusers.quantizers.bfl.utils import BFLFP8RLinear, BFLNVFP4Linear + + return isinstance(module, (BFLFP8RLinear, BFLNVFP4Linear)) + + +@is_quantization +@is_bfl +@require_accelerator +@require_accelerate +class BFLTesterMixin(BFLConfigMixin, QuantizationTesterMixin): + """ + Mixin class for testing BFL (`fp8r` / `nvfp4`) quantization on models. + + Expected class attributes: + - model_class: The model class to test + + Expected methods to be implemented by subclasses: + - get_init_dict(): Returns the init kwargs of the small test model + - get_dummy_inputs(): Returns dict of inputs to pass to the model forward pass + + Required properties (must be implemented by subclasses): + - bfl_checkpoints: Dict mapping a scheme (`"fp8r"`, `"nvfp4"`) to its single-file checkpoint; only the + safetensors header is fetched + + Pytest mark: bfl + Use `pytest -m "not bfl"` to skip these tests + """ + + @pytest.mark.parametrize("config_name", list(BFLConfigMixin.BFL_CONFIGS), ids=list(BFLConfigMixin.BFL_CONFIGS)) + def test_bfl_quantization_inference(self, config_name): + self._test_quantization_inference(BFLConfigMixin.BFL_CONFIGS[config_name]) + + @pytest.mark.parametrize("config_name", list(BFLConfigMixin.BFL_CONFIGS), ids=list(BFLConfigMixin.BFL_CONFIGS)) + def test_bfl_quantization_memory_footprint(self, config_name): + self._test_quantization_memory_footprint(BFLConfigMixin.BFL_CONFIGS[config_name]) + + def _test_quantized_layers(self, config_kwargs): + # The exporter quantizes a subset of the linears (embedders, output layers and some blocks stay in bf16), + # so the expected set is the one the checkpoint header describes, restricted to the small model's layers. + model = self._create_quantized_model(config_kwargs) + expected = self._bfl_quantized_linears(model, config_kwargs["scheme"]) + quantized = set() + for name, module in model.named_modules(): + if self._is_module_quantized(module): + self._verify_if_layer_quantized(name, module, config_kwargs) + quantized.add(name) + elif isinstance(module, torch.nn.Linear): + assert module.weight.dtype == config_kwargs["compute_dtype"], f"{name} is neither quantized nor bf16" + + assert quantized, "No quantized layers found in model" + assert quantized == expected, f"Quantized layers differ from the checkpoint: {quantized ^ expected}" + + @pytest.mark.parametrize("config_name", list(BFLConfigMixin.BFL_CONFIGS), ids=list(BFLConfigMixin.BFL_CONFIGS)) + def test_bfl_quantized_layers(self, config_name): + self._test_quantized_layers(BFLConfigMixin.BFL_CONFIGS[config_name]) + + @pytest.mark.parametrize("config_name", list(BFLConfigMixin.BFL_CONFIGS), ids=list(BFLConfigMixin.BFL_CONFIGS)) + @torch.no_grad() + def test_bfl_quantization_matches_unquantized(self, config_name): + """The quantized model tracks the model it was quantized from, on the fused kernel or the fallback.""" + config_kwargs = BFLConfigMixin.BFL_CONFIGS[config_name] + reference = self._load_unquantized_model().to(torch_device, config_kwargs["compute_dtype"]) + expected = reference(**self.get_dummy_inputs(), return_dict=False)[0] + + model = self._create_quantized_model(config_kwargs) + model.to(torch_device) + output = model(**self.get_dummy_inputs(), return_dict=False)[0] + + assert output.shape == expected.shape and not torch.isnan(output).any() + cosine = torch.nn.functional.cosine_similarity(output.float().flatten(), expected.float().flatten(), dim=0) + assert cosine > {"fp8r": 0.99, "nvfp4": 0.9}[config_name], f"cosine similarity {cosine.item():.4f}" + + @pytest.mark.parametrize("config_name", ["fp8r"], ids=["fp8r"]) + def test_bfl_quantization_dtype_assignment(self, config_name): + self._test_quantization_dtype_assignment(BFLConfigMixin.BFL_CONFIGS[config_name]) + + @pytest.mark.parametrize("config_name", ["fp8r"], ids=["fp8r"]) + def test_bfl_quantization_lora_inference(self, config_name): + self._test_quantization_lora_inference(BFLConfigMixin.BFL_CONFIGS[config_name]) + + @pytest.mark.parametrize("config_name", list(BFLConfigMixin.BFL_CONFIGS), ids=list(BFLConfigMixin.BFL_CONFIGS)) + def test_bfl_keep_modules_in_fp32(self, config_name): + # `to_q` is quantized in the checkpoint, so this exercises the quantizer's fp32 exclusion path rather than + # a layer the exporter already left in bf16. + _keep_in_fp32_modules = self.model_class._keep_in_fp32_modules + self.model_class._keep_in_fp32_modules = ["to_q"] + try: + self._test_keep_modules_in_fp32(BFLConfigMixin.BFL_CONFIGS[config_name]) + finally: + self.model_class._keep_in_fp32_modules = _keep_in_fp32_modules + + @pytest.mark.parametrize("config_name", list(BFLConfigMixin.BFL_CONFIGS), ids=list(BFLConfigMixin.BFL_CONFIGS)) + def test_bfl_dequantize(self, config_name): + self._test_dequantize(BFLConfigMixin.BFL_CONFIGS[config_name]) + + def test_bfl_rejects_unquantized_checkpoint(self): + model = self._load_unquantized_model().to(torch.bfloat16) + with tempfile.TemporaryDirectory() as config_dir: + model.save_config(config_dir) + with pytest.raises(ValueError, match="weight_scale"): + self.model_class.from_single_file( + model.state_dict(), config=config_dir, quantization_config=BFLQuantizationConfig() + ) + + +@is_quantization +@is_bfl +@is_torch_compile +@require_accelerator +@require_accelerate +class BFLCompileTesterMixin(BFLConfigMixin, QuantizationCompileTesterMixin): + """ + Mixin class for testing torch.compile with BFL (`fp8r` / `nvfp4`) quantized models. + + Expected class attributes: + - model_class: The model class to test + + Expected methods to be implemented by subclasses: + - get_init_dict(): Returns the init kwargs of the small test model + - get_dummy_inputs(): Returns dict of inputs to pass to the model forward pass + + Required properties (must be implemented by subclasses): + - bfl_checkpoints: Dict mapping a scheme (`"fp8r"`, `"nvfp4"`) to its single-file checkpoint; only the + safetensors header is fetched + + Pytest mark: bfl + Use `pytest -m "not bfl"` to skip these tests + """ + + @pytest.mark.parametrize("config_name", list(BFLConfigMixin.BFL_CONFIGS), ids=list(BFLConfigMixin.BFL_CONFIGS)) + def test_bfl_torch_compile(self, config_name): + if ( + config_name == "nvfp4" + and torch_device == "cuda" + and torch.cuda.get_device_capability()[0] >= 10 + and is_flashinfer_available() + ): + # Dynamo cannot trace FlashInfer's Python wrappers (they take a thread lock), and with graph breaks around + # them the remaining fragments recompile between calls. Expected to pass once the fused nvfp4 path wraps + # the FlashInfer calls as custom ops. + pytest.xfail("the fused nvfp4 path calls FlashInfer, which torch.compile cannot trace") + self._test_torch_compile(BFLConfigMixin.BFL_CONFIGS[config_name]) + + @pytest.mark.parametrize("config_name", ["fp8r"], ids=["fp8r"]) + def test_bfl_torch_compile_with_group_offload(self, config_name): + self._test_torch_compile_with_group_offload(BFLConfigMixin.BFL_CONFIGS[config_name]) diff --git a/tests/models/transformers/test_models_transformer_flux3.py b/tests/models/transformers/test_models_transformer_flux3.py index 09cfc06cabd6..cec0449d6282 100644 --- a/tests/models/transformers/test_models_transformer_flux3.py +++ b/tests/models/transformers/test_models_transformer_flux3.py @@ -23,6 +23,8 @@ from ..testing_utils import ( AttentionTesterMixin, BaseModelTesterConfig, + BFLCompileTesterMixin, + BFLTesterMixin, MemoryTesterMixin, ModelTesterMixin, SingleFileTesterMixin, @@ -167,3 +169,32 @@ def pretrained_model_kwargs(self): @property def torch_dtype(self): return torch.bfloat16 + + +class Flux3TransformerBFLTesterConfig(Flux3TransformerTesterConfig): + """Shared config for the BFL quantization tests: which linears to quantize comes from the real checkpoints' + headers; the model itself is the small test model in the compute dtype.""" + + @property + def bfl_checkpoints(self): + return { + "fp8r": "https://huggingface.co/herzogenhorn/f3-image-small/blob/main/variants/fp8r/model.safetensors", + "nvfp4": "https://huggingface.co/herzogenhorn/f3-image-small/blob/main/variants/nvfp4/model.safetensors", + } + + @property + def torch_dtype(self): + return torch.bfloat16 + + def get_dummy_inputs(self, **kwargs): + """Override to build inputs in the quantizer compute dtype (unquantized linears do strict-dtype matmuls).""" + inputs = super().get_dummy_inputs(**kwargs) + return {k: v.to(self.torch_dtype) if torch.is_floating_point(v) else v for k, v in inputs.items()} + + +class TestFlux3TransformerBFL(Flux3TransformerBFLTesterConfig, BFLTesterMixin): + """BFL fp8r / nvfp4 quantization tests for the FLUX.3 image transformer.""" + + +class TestFlux3TransformerBFLCompile(Flux3TransformerBFLTesterConfig, BFLCompileTesterMixin): + """BFL fp8r / nvfp4 + torch.compile tests for the FLUX.3 image transformer.""" diff --git a/tests/testing_utils.py b/tests/testing_utils.py index 9da89e626198..3fc14c285f80 100644 --- a/tests/testing_utils.py +++ b/tests/testing_utils.py @@ -468,6 +468,14 @@ def is_sdnq(test_case): return pytest.mark.sdnq(test_case) +def is_bfl(test_case): + """ + Decorator marking a test as a BFL (fp8r / nvfp4) quantization test. These tests can be filtered using: + pytest -m "not bfl" to skip pytest -m bfl to run only these tests + """ + return pytest.mark.bfl(test_case) + + def is_context_parallel(test_case): """ Decorator marking a test as a context parallel inference test. These tests can be filtered using: From 4430857d6d62191844be0c66b3db464c45567d9e Mon Sep 17 00:00:00 2001 From: linoytsaban Date: Mon, 12 Oct 2026 07:16:26 +0000 Subject: [PATCH 19/34] add LoRA training for Flux 3 image small --- examples/dreambooth/README_flux3_small.md | 202 ++ .../dreambooth/requirements_flux3_small.txt | 11 + .../test_dreambooth_lora_flux3_small.py | 194 ++ ...est_dreambooth_lora_flux3_small_img2img.py | 121 + .../train_dreambooth_lora_flux3_small.py | 2172 +++++++++++++++++ ...ain_dreambooth_lora_flux3_small_img2img.py | 2058 ++++++++++++++++ 6 files changed, 4758 insertions(+) create mode 100644 examples/dreambooth/README_flux3_small.md create mode 100644 examples/dreambooth/requirements_flux3_small.txt create mode 100644 examples/dreambooth/test_dreambooth_lora_flux3_small.py create mode 100644 examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py create mode 100644 examples/dreambooth/train_dreambooth_lora_flux3_small.py create mode 100644 examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py diff --git a/examples/dreambooth/README_flux3_small.md b/examples/dreambooth/README_flux3_small.md new file mode 100644 index 000000000000..4ace1414bf03 --- /dev/null +++ b/examples/dreambooth/README_flux3_small.md @@ -0,0 +1,202 @@ +# DreamBooth LoRA training for the FLUX.3 image model + +These scripts train LoRA adapters for the FLUX.3 image model (`Flux3Transformer2DModel`, served by +`Flux3ModularPipeline`) with the DreamBooth recipe: + +- `train_dreambooth_lora_flux3_small.py` — text-to-image (t2i) +- `train_dreambooth_lora_flux3_small_img2img.py` — image-to-image / instruction editing (i2i) + +> [!NOTE] +> LoRA **loading and inference** for this model live in the pipeline/loader PR (`Flux3LoraLoaderMixin`); these +> scripts only cover **training**. Adapters are saved through `Flux3LoraLoaderMixin.save_lora_weights`, so they load +> back with `Flux3ModularPipeline.load_lora_weights`. + +A few things that are specific to FLUX.3 and differ from the FLUX.1/FLUX.2 trainers: + +- **Text encoder is Qwen3-VL.** The text embedding is the stack of several Qwen3-VL hidden layers + (`--text_encoder_out_layers`, default `4 8 12 16 20 24 28 32`), concatenated along the channel dim. The prompt goes + through the Qwen chat template and the text window is right-padded to a multiple of 80 tokens (capped at + `--max_sequence_length`), matching the pipeline's `Flux3TextEncoderStep`. +- **No guidance distillation.** The model is not guidance-distilled, so there is no guidance argument in training; + classifier-free guidance happens at inference. Use `--caption_dropout` for unconditional training (t2i). +- **VAE normalization is a batch-norm.** Latents are 2×2-patchified to 128 channels and normalized with the VAE's + stored batch-norm statistics (not a scalar `scaling_factor`/`shift_factor`). +- **Flow matching with a fixed shift of 6.93** (no resolution-dependent dynamic shifting); the velocity target is + `noise - image`. +- **Validation reuses the in-memory trained transformer** via `Flux3AutoBlocks().init_pipeline()` + + `update_components(...)`, so the DiT is never loaded a second time. + +## Running locally with PyTorch + +### Installing the dependencies + +Install `diffusers` from source and keep it up to date: + +```bash +git clone https://github.com/huggingface/diffusers +cd diffusers +pip install -e . +``` + +Then `cd` into `examples/dreambooth` and install the requirements: + +```bash +pip install -r requirements_flux3_small.txt +``` + +Initialize an [🤗 Accelerate](https://github.com/huggingface/accelerate/) environment with: + +```bash +accelerate config +``` + +Or, for a default configuration: + +```bash +accelerate config default +``` + +We use PEFT as the LoRA backend, so make sure `peft>=0.11.1` is installed. + +## Text-to-image (t2i) + +### Style LoRA example + +This trains a *yarn-art* style LoRA on [Norod78/Yarn-art-style](https://huggingface.co/datasets/Norod78/Yarn-art-style), +whose captions already contain the trigger phrase `yarn art style`. + +```bash +export MODEL_NAME="" +export OUTPUT_DIR="trained-flux3-small-yarn-lora" + +accelerate launch train_dreambooth_lora_flux3_small.py \ + --pretrained_model_name_or_path=$MODEL_NAME \ + --dataset_name="Norod78/Yarn-art-style" \ + --caption_column="text" \ + --instance_prompt="yarn art style" \ + --output_dir=$OUTPUT_DIR \ + --mixed_precision="bf16" \ + --resolution=1024 \ + --train_batch_size=1 \ + --gradient_checkpointing \ + --cache_latents \ + --rank=16 \ + --lora_alpha=16 \ + --optimizer="adamW" \ + --use_8bit_adam \ + --learning_rate=1e-4 \ + --lr_scheduler="constant" \ + --lr_warmup_steps=0 \ + --max_train_steps=2000 \ + --validation_prompt="a cat, yarn art style" \ + --validation_epochs=25 \ + --seed="0" \ + --push_to_hub +``` + +For a subject/object LoRA, point `--instance_data_dir` at a local folder of images and set `--instance_prompt` to a +caption with a trigger word (e.g. `a photo of sks dog`) instead of `--dataset_name`/`--caption_column`. + +> [!NOTE] +> For long prompts, raise `--max_sequence_length` (the Qwen3-VL token cap, default 512). Longer windows use more +> memory and can slow down training. + +## Image-to-image (i2i) + +The i2i script trains an instruction-editing / relighting-style LoRA on **(condition image, target image, +instruction)** triplets. The reference image is VAE-encoded and fed to the model as a separate, clean conditioning +stream (its own position ids and timestep 0); the loss is computed on the generated image. + +This example uses [kontext-community/relighting](https://huggingface.co/datasets/kontext-community/relighting) +(`file_name` = source, `output` = relit target, `instruction` = edit prompt): + +```bash +export MODEL_NAME="" +export OUTPUT_DIR="trained-flux3-small-relight-lora" + +accelerate launch train_dreambooth_lora_flux3_small_img2img.py \ + --pretrained_model_name_or_path=$MODEL_NAME \ + --dataset_name="kontext-community/relighting" \ + --image_column="output" \ + --cond_image_column="file_name" \ + --caption_column="instruction" \ + --output_dir=$OUTPUT_DIR \ + --mixed_precision="bf16" \ + --resolution=1024 \ + --train_batch_size=1 \ + --gradient_checkpointing \ + --cache_latents \ + --rank=16 \ + --lora_alpha=16 \ + --optimizer="adamW" \ + --use_8bit_adam \ + --learning_rate=1e-4 \ + --lr_scheduler="constant" \ + --max_train_steps=1500 \ + --seed="0" \ + --validation_prompt="relight the scene with warm sunset light" \ + --validation_image="path/to/a/source/image.png" \ + --validation_epochs=25 \ + --push_to_hub +``` + +i2i reads its data through `--dataset_name` (a dataset with an image column, a conditioning-image column and a caption +column), not a plain image folder. + +## Tracking + +Pass `--report_to="wandb"` (or `tensorboard`) and `--validation_prompt` (plus `--validation_image` for i2i) to log +validation samples during training. + +## Memory optimizations + +- **CPU offloading** — `--offload` moves the VAE and text encoder to CPU except while they are used. +- **Latent caching** — `--cache_latents` pre-encodes images to latents once (the VAE is then freed), at the cost of no + random image augmentations. +- **Low-precision training** — `--do_fp8_training` (torchao FP8) or `--bnb_quantization_config_path` (bitsandbytes + QLoRA); the two are mutually exclusive. +- **Gradient checkpointing / accumulation** — `--gradient_checkpointing`, `--gradient_accumulation_steps`. +- **8-bit Adam** — `--optimizer="adamW" --use_8bit_adam` (or `--optimizer="prodigy"`). +- **Saved-LoRA precision** — `--upcast_before_saving` saves the adapter in fp32 instead of the training dtype. + +## LoRA rank, alpha and target modules + +`--rank` / `--lora_alpha` / `--lora_dropout` control the adapter. By default the scripts adapt the attention and +feed-forward projections across every block: + +``` +to_q, to_k, to_v, to_out.0, ff.linear_in, ff.linear_out +``` + +These are exactly the leaf modules the non-diffusers (ostris / ai-toolkit) FLUX.3 LoRA converter maps, so adapters +trained here stay interchangeable with that layout. These names match across the per-mode blocks (`mode_blocks.image`, +`mode_blocks.image_cond`, `txt_mode_blocks`) and the joint `single_transformer_blocks`, so a single target set reaches +every stream. Use `--lora_layers` (a comma-separated list) to target a different set, e.g. to also adapt +`context_embedder`, `x_embedders.image` or `proj_out`. + +## Aspect-ratio bucketing and caption dropout + +- `--aspect_ratio_buckets` / `--use_aspect_ratio_buckets` group images of similar aspect ratio so a batch shares a + resolution (t2i). +- `--caption_dropout` replaces a fraction of instance captions with the empty prompt during training, which improves + classifier-free guidance at inference (t2i). + +## Inference + +Load a trained adapter onto the modular pipeline and generate: + +```python +import torch +from diffusers.modular_pipelines import Flux3AutoBlocks + +pipe = Flux3AutoBlocks().init_pipeline("") +pipe.load_components(dtype=torch.bfloat16) +pipe.to("cuda") + +pipe.load_lora_weights("trained-flux3-small-yarn-lora", weight_name="pytorch_lora_weights.safetensors") + +image = pipe(prompt="a cat, yarn art style", output="images")[0] +image.save("yarn_cat.png") +``` + +For i2i, pass the reference image through the pipeline's `image=` input. diff --git a/examples/dreambooth/requirements_flux3_small.txt b/examples/dreambooth/requirements_flux3_small.txt new file mode 100644 index 000000000000..f7791342a0f4 --- /dev/null +++ b/examples/dreambooth/requirements_flux3_small.txt @@ -0,0 +1,11 @@ +accelerate>=0.31.0 +torchvision +transformers>=4.57.0 +ftfy +tensorboard +Jinja2 +peft>=0.11.1 +sentencepiece +bitsandbytes +prodigyopt +datasets>=3.1.0 diff --git a/examples/dreambooth/test_dreambooth_lora_flux3_small.py b/examples/dreambooth/test_dreambooth_lora_flux3_small.py new file mode 100644 index 000000000000..87a30ca0a492 --- /dev/null +++ b/examples/dreambooth/test_dreambooth_lora_flux3_small.py @@ -0,0 +1,194 @@ +# coding=utf-8 +# Copyright 2026 HuggingFace Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import os +import sys +import tempfile + +import safetensors + + +sys.path.append("..") +from test_examples_utils import ExamplesTestsAccelerate, run_command # noqa: E402 + + +logging.basicConfig(level=logging.DEBUG) + +logger = logging.getLogger() +stream_handler = logging.StreamHandler(sys.stdout) +logger.addHandler(stream_handler) + +# The tiny FLUX.3 image repo is built by `integration-notes/make_tiny_flux3_repo.py`. Upload it as +# hf-internal-testing/tiny-flux3-image-modular and drop the env override before opening the PR. +FLUX3_TINY_REPO = os.environ.get("FLUX3_TINY_REPO", "hf-internal-testing/tiny-flux3-image-modular") + + +class DreamBoothLoRAFlux3Small(ExamplesTestsAccelerate): + instance_data_dir = "docs/source/en/imgs" + instance_prompt = "dog" + pretrained_model_name_or_path = FLUX3_TINY_REPO + script_path = "examples/dreambooth/train_dreambooth_lora_flux3_small.py" + transformer_layer_type = "single_transformer_blocks.0.attn.to_k" + + def test_dreambooth_lora_flux3_small(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_args = f""" + {self.script_path} + --pretrained_model_name_or_path {self.pretrained_model_name_or_path} + --instance_data_dir {self.instance_data_dir} + --instance_prompt {self.instance_prompt} + --resolution 64 + --train_batch_size 1 + --gradient_accumulation_steps 1 + --max_train_steps 2 + --learning_rate 5.0e-04 + --scale_lr + --lr_scheduler constant + --lr_warmup_steps 0 + --max_sequence_length 8 + --text_encoder_out_layers 1 + --output_dir {tmpdir} + """.split() + + run_command(self._launch_args + test_args) + # save_pretrained smoke test + self.assertTrue(os.path.isfile(os.path.join(tmpdir, "pytorch_lora_weights.safetensors"))) + + # make sure the state_dict has the correct naming in the parameters. + lora_state_dict = safetensors.torch.load_file(os.path.join(tmpdir, "pytorch_lora_weights.safetensors")) + is_lora = all("lora" in k for k in lora_state_dict.keys()) + self.assertTrue(is_lora) + + # when not training the text encoder, all the parameters in the state dict should start + # with `"transformer"` in their names. + starts_with_transformer = all(key.startswith("transformer") for key in lora_state_dict.keys()) + self.assertTrue(starts_with_transformer) + + def test_dreambooth_lora_flux3_small_latent_caching(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_args = f""" + {self.script_path} + --pretrained_model_name_or_path {self.pretrained_model_name_or_path} + --instance_data_dir {self.instance_data_dir} + --instance_prompt {self.instance_prompt} + --resolution 64 + --train_batch_size 1 + --gradient_accumulation_steps 1 + --max_train_steps 2 + --cache_latents + --learning_rate 5.0e-04 + --scale_lr + --lr_scheduler constant + --lr_warmup_steps 0 + --max_sequence_length 8 + --text_encoder_out_layers 1 + --output_dir {tmpdir} + """.split() + + run_command(self._launch_args + test_args) + # save_pretrained smoke test + self.assertTrue(os.path.isfile(os.path.join(tmpdir, "pytorch_lora_weights.safetensors"))) + + # make sure the state_dict has the correct naming in the parameters. + lora_state_dict = safetensors.torch.load_file(os.path.join(tmpdir, "pytorch_lora_weights.safetensors")) + is_lora = all("lora" in k for k in lora_state_dict.keys()) + self.assertTrue(is_lora) + + starts_with_transformer = all(key.startswith("transformer") for key in lora_state_dict.keys()) + self.assertTrue(starts_with_transformer) + + def test_dreambooth_lora_flux3_small_layers(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_args = f""" + {self.script_path} + --pretrained_model_name_or_path {self.pretrained_model_name_or_path} + --instance_data_dir {self.instance_data_dir} + --instance_prompt {self.instance_prompt} + --resolution 64 + --train_batch_size 1 + --gradient_accumulation_steps 1 + --max_train_steps 2 + --cache_latents + --learning_rate 5.0e-04 + --scale_lr + --lora_layers {self.transformer_layer_type} + --lr_scheduler constant + --lr_warmup_steps 0 + --max_sequence_length 8 + --text_encoder_out_layers 1 + --output_dir {tmpdir} + """.split() + + run_command(self._launch_args + test_args) + # save_pretrained smoke test + self.assertTrue(os.path.isfile(os.path.join(tmpdir, "pytorch_lora_weights.safetensors"))) + + # make sure the state_dict has the correct naming in the parameters. + lora_state_dict = safetensors.torch.load_file(os.path.join(tmpdir, "pytorch_lora_weights.safetensors")) + is_lora = all("lora" in k for k in lora_state_dict.keys()) + self.assertTrue(is_lora) + + # only the layers named by `--lora_layers` should be trained, so every key sits under that path + starts_with_transformer = all( + key.startswith(f"transformer.{self.transformer_layer_type}") for key in lora_state_dict.keys() + ) + self.assertTrue(starts_with_transformer) + + def test_dreambooth_lora_flux3_small_checkpointing_checkpoints_total_limit(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_args = f""" + {self.script_path} + --pretrained_model_name_or_path={self.pretrained_model_name_or_path} + --instance_data_dir={self.instance_data_dir} + --output_dir={tmpdir} + --instance_prompt={self.instance_prompt} + --resolution=64 + --train_batch_size=1 + --gradient_accumulation_steps=1 + --max_train_steps=6 + --checkpoints_total_limit=2 + --max_sequence_length 8 + --checkpointing_steps=2 + --text_encoder_out_layers 1 + """.split() + + run_command(self._launch_args + test_args) + + self.assertEqual( + {x for x in os.listdir(tmpdir) if "checkpoint" in x}, + {"checkpoint-4", "checkpoint-6"}, + ) + + def test_dreambooth_lora_flux3_small_checkpointing_checkpoints_total_limit_removes_multiple_checkpoints(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_args = f""" + {self.script_path} + --pretrained_model_name_or_path={self.pretrained_model_name_or_path} + --instance_data_dir={self.instance_data_dir} + --output_dir={tmpdir} + --instance_prompt={self.instance_prompt} + --resolution=64 + --train_batch_size=1 + --gradient_accumulation_steps=1 + --max_train_steps=4 + --checkpointing_steps=2 + --max_sequence_length 8 + --text_encoder_out_layers 1 + """.split() + + run_command(self._launch_args + test_args) + + self.assertEqual({x for x in os.listdir(tmpdir) if "checkpoint" in x}, {"checkpoint-2", "checkpoint-4"}) diff --git a/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py b/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py new file mode 100644 index 000000000000..907743681dd7 --- /dev/null +++ b/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py @@ -0,0 +1,121 @@ +# coding=utf-8 +# Copyright 2026 HuggingFace Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import os +import sys +import tempfile + +import safetensors + + +sys.path.append("..") +from test_examples_utils import ExamplesTestsAccelerate, run_command # noqa: E402 + + +logging.basicConfig(level=logging.DEBUG) + +logger = logging.getLogger() +stream_handler = logging.StreamHandler(sys.stdout) +logger.addHandler(stream_handler) + +# The tiny FLUX.3 image repo is built by `integration-notes/make_tiny_flux3_repo.py`. Upload it as +# hf-internal-testing/tiny-flux3-image-modular and drop the env override before opening the PR. +FLUX3_TINY_REPO = os.environ.get("FLUX3_TINY_REPO", "hf-internal-testing/tiny-flux3-image-modular") + + +class DreamBoothLoRAFlux3SmallImg2Img(ExamplesTestsAccelerate): + # image-to-image training needs a paired conditioning image, so it reads a dataset with an image + + # conditioning_image column rather than a plain folder + dataset_name = "hf-internal-testing/fill10" + image_column = "image" + cond_image_column = "conditioning_image" + caption_column = "text" + instance_prompt = "dog" + pretrained_model_name_or_path = FLUX3_TINY_REPO + script_path = "examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py" + + def test_dreambooth_lora_flux3_small_img2img(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_args = f""" + {self.script_path} + --pretrained_model_name_or_path {self.pretrained_model_name_or_path} + --dataset_name {self.dataset_name} + --image_column {self.image_column} + --cond_image_column {self.cond_image_column} + --caption_column {self.caption_column} + --instance_prompt {self.instance_prompt} + --resolution 64 + --train_batch_size 1 + --gradient_accumulation_steps 1 + --max_train_steps 2 + --learning_rate 5.0e-04 + --lr_scheduler constant + --lr_warmup_steps 0 + --max_sequence_length 8 + --text_encoder_out_layers 1 + --output_dir {tmpdir} + """.split() + + run_command(self._launch_args + test_args) + # save_pretrained smoke test + self.assertTrue(os.path.isfile(os.path.join(tmpdir, "pytorch_lora_weights.safetensors"))) + + # make sure the state_dict has the correct naming in the parameters. + lora_state_dict = safetensors.torch.load_file(os.path.join(tmpdir, "pytorch_lora_weights.safetensors")) + is_lora = all("lora" in k for k in lora_state_dict.keys()) + self.assertTrue(is_lora) + + starts_with_transformer = all(key.startswith("transformer") for key in lora_state_dict.keys()) + self.assertTrue(starts_with_transformer) + + # the reference image is consumed by its own ("image_cond") mode blocks, so training an img2img adapter + # with the default targets reaches both the generated-image and the reference mode blocks + self.assertTrue(any("mode_blocks.image." in key for key in lora_state_dict.keys())) + self.assertTrue(any("mode_blocks.image_cond." in key for key in lora_state_dict.keys())) + + def test_dreambooth_lora_flux3_small_img2img_latent_caching(self): + with tempfile.TemporaryDirectory() as tmpdir: + test_args = f""" + {self.script_path} + --pretrained_model_name_or_path {self.pretrained_model_name_or_path} + --dataset_name {self.dataset_name} + --image_column {self.image_column} + --cond_image_column {self.cond_image_column} + --caption_column {self.caption_column} + --instance_prompt {self.instance_prompt} + --resolution 64 + --train_batch_size 1 + --gradient_accumulation_steps 1 + --max_train_steps 2 + --cache_latents + --learning_rate 5.0e-04 + --lr_scheduler constant + --lr_warmup_steps 0 + --max_sequence_length 8 + --text_encoder_out_layers 1 + --output_dir {tmpdir} + """.split() + + run_command(self._launch_args + test_args) + # save_pretrained smoke test + self.assertTrue(os.path.isfile(os.path.join(tmpdir, "pytorch_lora_weights.safetensors"))) + + lora_state_dict = safetensors.torch.load_file(os.path.join(tmpdir, "pytorch_lora_weights.safetensors")) + is_lora = all("lora" in k for k in lora_state_dict.keys()) + self.assertTrue(is_lora) + + # the cached conditioning latents must still produce an adapter on the reference mode blocks + self.assertTrue(any("image_cond" in key for key in lora_state_dict.keys())) diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_small.py b/examples/dreambooth/train_dreambooth_lora_flux3_small.py new file mode 100644 index 000000000000..3fdaafdba4e1 --- /dev/null +++ b/examples/dreambooth/train_dreambooth_lora_flux3_small.py @@ -0,0 +1,2172 @@ +#!/usr/bin/env python +# coding=utf-8 +# Copyright 2025 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# /// script +# dependencies = [ +# "diffusers @ git+https://github.com/huggingface/diffusers.git", +# "torch>=2.0.0", +# "accelerate>=0.31.0", +# "transformers>=4.41.2", +# "ftfy", +# "tensorboard", +# "Jinja2", +# "peft>=0.11.1", +# "sentencepiece", +# "torchvision", +# "datasets", +# "bitsandbytes", +# "prodigyopt", +# ] +# /// + +import argparse +import copy +import itertools +import json +import logging +import math +import os +import random +import shutil +import warnings +from contextlib import nullcontext +from pathlib import Path +from typing import Any + +import numpy as np +import torch +import transformers +from accelerate import Accelerator +from accelerate.logging import get_logger +from accelerate.utils import DistributedDataParallelKwargs, ProjectConfiguration, set_seed +from huggingface_hub import create_repo, upload_folder +from huggingface_hub.utils import insecure_hashlib +from peft import LoraConfig, prepare_model_for_kbit_training, set_peft_model_state_dict +from peft.utils import get_peft_model_state_dict +from PIL import Image +from PIL.ImageOps import exif_transpose +from torch.utils.data import Dataset +from torch.utils.data.sampler import BatchSampler +from torchvision import transforms +from torchvision.transforms import functional as TF +from tqdm.auto import tqdm +from transformers import AutoProcessor, Qwen3VLForConditionalGeneration + +import diffusers +from diffusers import ( + AutoencoderKLFlux2, + BitsAndBytesConfig, + FlowMatchEulerDiscreteScheduler, + Flux3Transformer2DModel, + ModularPipeline, +) +from diffusers.loaders import Flux3LoraLoaderMixin +from diffusers.modular_pipelines.flux3 import Flux3AutoBlocks +from diffusers.optimization import get_scheduler +from diffusers.training_utils import ( + _collate_lora_metadata, + _to_cpu_contiguous, + cast_training_params, + compute_density_for_timestep_sampling, + compute_loss_weighting_for_sd3, + find_nearest_bucket, + free_memory, + generate_aspect_ratio_buckets, + get_fsdp_kwargs_from_accelerator, + offload_models, + parse_buckets_string, + wrap_with_fsdp, +) +from diffusers.utils import ( + check_min_version, + convert_unet_state_dict_to_peft, + is_wandb_available, +) +from diffusers.utils.hub_utils import load_or_create_model_card, populate_model_card +from diffusers.utils.import_utils import is_torch_npu_available +from diffusers.utils.torch_utils import is_compiled_module + + +if getattr(torch, "distributed", None) is not None: + import torch.distributed as dist + +if is_wandb_available(): + import wandb + +# Will error if the minimal version of diffusers is not installed. Remove at your own risks. +check_min_version("0.40.0.dev0") + +logger = get_logger(__name__) + +# FLUX.3 ships no `scheduler` subfolder: the modular pipeline builds its scheduler `from_config` with a fixed +# shift of 6.93 (the `Flux3SetTimestepsStep` default, and the reference sampler's `alpha`). There is no +# resolution-dependent dynamic shifting. Constructed here for the same reason, rather than loaded. +FLOW_SHIFT = 6.93 + +# FLUX.3 image RoPE scale applied to the H/W position-id axes (see the flux3 modular pipeline). +IMAGE_ROPE_SCALE = 0.5 + +# The text window is right-padded to a multiple of this many tokens (the `Flux3TextEncoderStep` default). +TEXT_PAD_MULTIPLE = 80 + + +# --- FLUX.3 latent / text helpers (bit-identical to the modular pipeline blocks) --- +def patchify_latents(latents): + """(B, C, H, W) -> (B, C*4, H//2, W//2).""" + b, c, h, w = latents.shape + latents = latents.view(b, c, h // 2, 2, w // 2, 2).permute(0, 1, 3, 5, 2, 4) + return latents.reshape(b, c * 4, h // 2, w // 2) + + +def pack_latents(latents): + """(B, C, H, W) -> (B, H*W, C).""" + b, c, h, w = latents.shape + return latents.reshape(b, c, h * w).permute(0, 2, 1) + + +def prepare_latent_ids(height, width, device, image_rope_scale=IMAGE_ROPE_SCALE): + """(H*W, 4) position ids (t, h, w, l); H/W axes scaled by the FLUX.3 image RoPE scale.""" + ids = torch.cartesian_prod(torch.arange(1), torch.arange(height), torch.arange(width), torch.arange(1)) + ids = ids.to(dtype=torch.float32) + ids[:, 1:3] = ids[:, 1:3] * image_rope_scale + return ids.to(device) + + +def prepare_text_ids(seq_len, device): + """(seq_len, 4) position ids for text tokens; text advances along the L axis.""" + ids = torch.cartesian_prod(torch.arange(1), torch.arange(1), torch.arange(1), torch.arange(seq_len)) + return ids.to(dtype=torch.float32).to(device) + + +@torch.no_grad() +def get_qwen3vl_prompt_embeds( + text_encoder, tokenizer, prompt, device, dtype, text_encoder_out_layers, max_sequence_length +): + """Stack the FLUX.3 Qwen3-VL hidden layers -> (B, seq, len(layers) * hidden_dim). + + Mirrors `Flux3TextEncoderStep._get_qwen3_vl_prompt_embeds`: each prompt goes through the chat template and is + right-padded to a multiple of `TEXT_PAD_MULTIPLE` (capped at `max_sequence_length`). Qwen3-VL masks the padding + (so a prompt's real-token embeddings are independent of the window size), but the transformer attends to it, so + the window is part of the recipe -- for a single prompt this reproduces the pipeline window exactly. When a batch + of differing prompts is encoded together, they share the batch's largest window so the embeddings can be stacked; + with `train_batch_size=1` or a shared instance prompt every prompt keeps its own pipeline-exact window. + The position ids of the padding tokens are passed explicitly to match the Qwen3-VL 4.57 training environment. + """ + prompt = [prompt] if isinstance(prompt, str) else prompt + texts = [ + tokenizer.apply_chat_template([{"role": "user", "content": p}], tokenize=False, add_generation_prompt=True) + for p in prompt + ] + real_lengths = [ + tokenizer.tokenizer(t, return_tensors="pt", truncation=True, max_length=max_sequence_length)[ + "input_ids" + ].shape[1] + for t in texts + ] + padded_length = min(math.ceil(max(real_lengths) / TEXT_PAD_MULTIPLE) * TEXT_PAD_MULTIPLE, max_sequence_length) + enc = tokenizer.tokenizer( + texts, + return_tensors="pt", + padding="max_length", + padding_side="right", + truncation=True, + max_length=padded_length, + ) + input_ids, attn = enc["input_ids"].to(device), enc["attention_mask"].to(device) + position_ids = torch.arange(padded_length, device=device)[None].masked_fill(attn == 0, 1) + out = text_encoder( + input_ids=input_ids, + attention_mask=attn, + position_ids=position_ids, + output_hidden_states=True, + use_cache=False, + ) + stacked = torch.stack([out.hidden_states[k] for k in text_encoder_out_layers], dim=1).to( + dtype=dtype, device=device + ) + b, n, s, d = stacked.shape + return stacked.permute(0, 2, 1, 3).reshape(b, s, n * d) + + +def save_model_card( + repo_id: str, + images=None, + base_model: str = None, + instance_prompt=None, + validation_prompt=None, + repo_folder=None, + quant_training=None, +): + widget_dict = [] + if images is not None: + for i, image in enumerate(images): + image.save(os.path.join(repo_folder, f"image_{i}.png")) + widget_dict.append( + {"text": validation_prompt if validation_prompt else " ", "output": {"url": f"image_{i}.png"}} + ) + + model_description = f""" +# FLUX.3 DreamBooth LoRA - {repo_id} + + + +## Model description + +These are {repo_id} DreamBooth LoRA weights for {base_model}. + +The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3.md). + +Quant training? {quant_training} + +## Trigger words + +You should use `{instance_prompt}` to trigger the image generation. + +## Download model + +[Download the *.safetensors LoRA]({repo_id}/tree/main) in the Files & versions tab. + +## Use it with the [🧨 diffusers library](https://github.com/huggingface/diffusers) + +```py +from diffusers import ModularPipeline +import torch +pipeline = ModularPipeline.from_pretrained("{base_model}") +pipeline.load_components(torch_dtype=torch.bfloat16) +pipeline.to('cuda') +pipeline.load_lora_weights('{repo_id}', weight_name='pytorch_lora_weights.safetensors') +image = pipeline(prompt='{validation_prompt if validation_prompt else instance_prompt}', output_type="pil").values["images"][0] +``` + +For more details, including weighting, merging and fusing LoRAs, check the [documentation on loading LoRAs in diffusers](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters) + +## License + +Please adhere to the licensing terms as described in the model repository. +""" + model_card = load_or_create_model_card( + repo_id_or_path=repo_id, + from_training=True, + license="other", + base_model=base_model, + prompt=instance_prompt, + model_description=model_description, + widget=widget_dict, + ) + tags = [ + "text-to-image", + "diffusers-training", + "diffusers", + "lora", + "flux3", + "flux3-diffusers", + "template:sd-lora", + ] + + model_card = populate_model_card(model_card, tags=tags) + model_card.save(os.path.join(repo_folder, "README.md")) + + +def log_validation( + pipeline, + args, + accelerator, + pipeline_args, + epoch, + torch_dtype, + is_final_validation=False, +): + args.num_validation_images = args.num_validation_images if args.num_validation_images else 1 + logger.info( + f"Running validation... \n Generating {args.num_validation_images} images with prompt:" + f" {args.validation_prompt}." + ) + + # run inference + generator = torch.Generator(device=accelerator.device).manual_seed(args.seed) if args.seed is not None else None + autocast_ctx = torch.autocast(accelerator.device.type) if not is_final_validation else nullcontext() + + images = [] + for _ in range(args.num_validation_images): + with autocast_ctx: + output = pipeline(generator=generator, output_type="pil", **pipeline_args) + images.append(output.values["images"][0]) + + for tracker in accelerator.trackers: + phase_name = "test" if is_final_validation else "validation" + if tracker.name == "tensorboard": + np_images = np.stack([np.asarray(img) for img in images]) + tracker.writer.add_images(phase_name, np_images, epoch, dataformats="NHWC") + if tracker.name == "wandb": + tracker.log( + { + phase_name: [ + wandb.Image(image, caption=f"{i}: {args.validation_prompt}") for i, image in enumerate(images) + ] + } + ) + + del pipeline + free_memory() + + return images + + +def make_validation_pipeline(args, transformer, weight_dtype, device): + # Build a modular pipeline for validation that REUSES the in-memory trained transformer, so validation runs + # on exactly the weights being trained and the DiT is never loaded a second time. Only the small auxiliary + # components are (re)loaded (they are freed after latent caching). + vae = AutoencoderKLFlux2.from_pretrained( + args.pretrained_model_name_or_path, subfolder="vae", revision=args.revision, torch_dtype=torch.float32 + ) + text_encoder = Qwen3VLForConditionalGeneration.from_pretrained( + args.pretrained_model_name_or_path, subfolder="text_encoder", revision=args.revision, torch_dtype=weight_dtype + ) + tokenizer = AutoProcessor.from_pretrained( + args.pretrained_model_name_or_path, subfolder="tokenizer", revision=args.revision + ) + scheduler = FlowMatchEulerDiscreteScheduler(shift=FLOW_SHIFT, use_dynamic_shifting=False) + pipeline = Flux3AutoBlocks().init_pipeline() + pipeline.update_components( + transformer=transformer, vae=vae, text_encoder=text_encoder, tokenizer=tokenizer, scheduler=scheduler + ) + pipeline.to(device) + return pipeline + + +def module_filter_fn(mod: torch.nn.Module, fqn: str): + # don't convert the output module (FLUX.3's output projection) + if fqn.endswith("proj_out"): + return False + # don't convert linear modules with weight dimensions not divisible by 16 + if isinstance(mod, torch.nn.Linear): + if mod.in_features % 16 != 0 or mod.out_features % 16 != 0: + return False + return True + + +def parse_args(input_args=None): + parser = argparse.ArgumentParser(description="Simple example of a training script.") + parser.add_argument( + "--pretrained_model_name_or_path", + type=str, + default=None, + required=True, + help="Path to pretrained model or model identifier from huggingface.co/models.", + ) + parser.add_argument( + "--revision", + type=str, + default=None, + required=False, + help="Revision of pretrained model identifier from huggingface.co/models.", + ) + parser.add_argument( + "--bnb_quantization_config_path", + type=str, + default=None, + help="Quantization config in a JSON file that will be used to define the bitsandbytes quant config of the DiT.", + ) + parser.add_argument( + "--do_fp8_training", + action="store_true", + help="if we are doing FP8 training.", + ) + parser.add_argument( + "--variant", + type=str, + default=None, + help="Variant of the model files of the pretrained model identifier from huggingface.co/models, 'e.g.' fp16", + ) + parser.add_argument( + "--dataset_name", + type=str, + default=None, + help=( + "The name of the Dataset (from the HuggingFace hub) containing the training data of instance images (could be your own, possibly private," + " dataset). It can also be a path pointing to a local copy of a dataset in your filesystem," + " or to a folder containing files that 🤗 Datasets can understand." + ), + ) + parser.add_argument( + "--dataset_config_name", + type=str, + default=None, + help="The config of the Dataset, leave as None if there's only one config.", + ) + parser.add_argument( + "--instance_data_dir", + type=str, + default=None, + help=("A folder containing the training data. "), + ) + + parser.add_argument( + "--cache_dir", + type=str, + default=None, + help="The directory where the downloaded models and datasets will be stored.", + ) + + parser.add_argument( + "--image_column", + type=str, + default="image", + help="The column of the dataset containing the target image. By " + "default, the standard Image Dataset maps out 'file_name' " + "to 'image'.", + ) + parser.add_argument( + "--caption_column", + type=str, + default="text", + help="The column of the dataset containing the instance prompt for each image", + ) + + parser.add_argument("--repeats", type=int, default=1, help="How many times to repeat the training data.") + + parser.add_argument( + "--class_data_dir", + type=str, + default=None, + required=False, + help="A folder containing the training data of class images.", + ) + parser.add_argument( + "--instance_prompt", + type=str, + default=None, + required=True, + help="The prompt with identifier specifying the instance, e.g. 'photo of a TOK dog', 'in the style of TOK'", + ) + parser.add_argument( + "--class_prompt", + type=str, + default=None, + help="The prompt to specify images in the same class as provided instance images.", + ) + parser.add_argument( + "--max_sequence_length", + type=int, + default=512, + help=( + "Cap on the number of Qwen3-VL prompt tokens; longer prompts are truncated. The text window is the " + f"prompt's length rounded up to a multiple of {TEXT_PAD_MULTIPLE}, capped at this value." + ), + ) + parser.add_argument( + "--text_encoder_out_layers", + type=int, + nargs="+", + default=[4, 8, 12, 16, 20, 24, 28, 32], + help="Text encoder hidden layers to compute the final text embeddings.", + ) + parser.add_argument( + "--validation_prompt", + type=str, + default=None, + help="A prompt that is used during validation to verify that the model is learning.", + ) + parser.add_argument( + "--skip_final_inference", + default=False, + action="store_true", + help="Whether to skip the final inference step with loaded lora weights upon training completion. This will run intermediate validation inference if `validation_prompt` is provided. Specify to reduce memory.", + ) + parser.add_argument( + "--final_validation_prompt", + type=str, + default=None, + help="A prompt that is used during a final validation to verify that the model is learning. Ignored if `--validation_prompt` is provided.", + ) + parser.add_argument( + "--num_validation_images", + type=int, + default=4, + help="Number of images that should be generated during validation with `validation_prompt`.", + ) + parser.add_argument( + "--validation_epochs", + type=int, + default=50, + help=( + "Run dreambooth validation every X epochs. Dreambooth validation consists of running the prompt" + " `args.validation_prompt` multiple times: `args.num_validation_images`." + ), + ) + parser.add_argument( + "--rank", + type=int, + default=4, + help=("The dimension of the LoRA update matrices."), + ) + parser.add_argument( + "--lora_alpha", + type=int, + default=4, + help="LoRA alpha to be used for additional scaling.", + ) + parser.add_argument("--lora_dropout", type=float, default=0.0, help="Dropout probability for LoRA layers") + + parser.add_argument( + "--with_prior_preservation", + default=False, + action="store_true", + help="Flag to add prior preservation loss.", + ) + parser.add_argument("--prior_loss_weight", type=float, default=1.0, help="The weight of prior preservation loss.") + parser.add_argument( + "--num_class_images", + type=int, + default=100, + help=( + "Minimal class images for prior preservation loss. If there are not enough images already present in" + " class_data_dir, additional images will be sampled with class_prompt." + ), + ) + parser.add_argument( + "--output_dir", + type=str, + default="flux-dreambooth-lora", + help="The output directory where the model predictions and checkpoints will be written.", + ) + parser.add_argument("--seed", type=int, default=None, help="A seed for reproducible training.") + parser.add_argument( + "--resolution", + type=int, + default=512, + help=( + "The resolution for input images, all the images in the train/validation dataset will be resized to this" + " resolution" + ), + ) + parser.add_argument( + "--aspect_ratio_buckets", + type=str, + default=None, + help=( + "Aspect ratio buckets to use for training. Define as a string of 'h1,w1;h2,w2;...'. " + "e.g. '1024,1024;768,1360;1360,768;880,1168;1168,880;1248,832;832,1248'. " + "Requires --use_aspect_ratio_buckets. Images are resized to cover and cropped to the nearest " + "listed bucket (smaller images are upscaled). When set, --resolution is ignored." + ), + ) + parser.add_argument( + "--use_aspect_ratio_buckets", + action="store_true", + help=( + "Enable aspect-ratio bucketing. Without --aspect_ratio_buckets, the buckets are computed on the " + "fly from --resolution and capped to each image's own resolution, so smaller images are assigned " + "to a smaller bucket instead of being upscaled. Provide --aspect_ratio_buckets to use an explicit list." + ), + ) + parser.add_argument( + "--center_crop", + default=False, + action="store_true", + help=( + "Whether to center crop the input images to the resolution. If not set, the images will be randomly" + " cropped. The images will be resized to the resolution first before cropping." + ), + ) + parser.add_argument( + "--random_flip", + action="store_true", + help="whether to randomly flip images horizontally", + ) + parser.add_argument( + "--caption_dropout", + type=float, + default=0.0, + help=( + "Probability of replacing an instance image's caption with an empty string during training, so that" + " fraction of samples is trained unconditionally. Improves classifier-free guidance. A common value is" + " 0.1. Class/prior-preservation captions are never dropped." + ), + ) + parser.add_argument( + "--train_batch_size", type=int, default=4, help="Batch size (per device) for the training dataloader." + ) + parser.add_argument( + "--sample_batch_size", type=int, default=4, help="Batch size (per device) for sampling images." + ) + parser.add_argument("--num_train_epochs", type=int, default=1) + parser.add_argument( + "--max_train_steps", + type=int, + default=None, + help="Total number of training steps to perform. If provided, overrides num_train_epochs.", + ) + parser.add_argument( + "--checkpointing_steps", + type=int, + default=500, + help=( + "Save a checkpoint of the training state every X updates. These checkpoints can be used both as final" + " checkpoints in case they are better than the last checkpoint, and are also suitable for resuming" + " training using `--resume_from_checkpoint`." + ), + ) + parser.add_argument( + "--checkpoints_total_limit", + type=int, + default=None, + help=("Max number of checkpoints to store."), + ) + parser.add_argument( + "--resume_from_checkpoint", + type=str, + default=None, + help=( + "Whether training should be resumed from a previous checkpoint. Use a path saved by" + ' `--checkpointing_steps`, or `"latest"` to automatically select the last available checkpoint.' + ), + ) + parser.add_argument( + "--gradient_accumulation_steps", + type=int, + default=1, + help="Number of updates steps to accumulate before performing a backward/update pass.", + ) + parser.add_argument( + "--gradient_checkpointing", + action="store_true", + help="Whether or not to use gradient checkpointing to save memory at the expense of slower backward pass.", + ) + parser.add_argument( + "--learning_rate", + type=float, + default=1e-4, + help="Initial learning rate (after the potential warmup period) to use.", + ) + + parser.add_argument( + "--scale_lr", + action="store_true", + default=False, + help="Scale the learning rate by the number of GPUs, gradient accumulation steps, and batch size.", + ) + parser.add_argument( + "--lr_scheduler", + type=str, + default="constant", + help=( + 'The scheduler type to use. Choose between ["linear", "cosine", "cosine_with_restarts", "polynomial",' + ' "constant", "constant_with_warmup"]' + ), + ) + parser.add_argument( + "--lr_warmup_steps", type=int, default=500, help="Number of steps for the warmup in the lr scheduler." + ) + parser.add_argument( + "--lr_num_cycles", + type=int, + default=1, + help="Number of hard resets of the lr in cosine_with_restarts scheduler.", + ) + parser.add_argument("--lr_power", type=float, default=1.0, help="Power factor of the polynomial scheduler.") + parser.add_argument( + "--dataloader_num_workers", + type=int, + default=0, + help=( + "Number of subprocesses to use for data loading. 0 means that the data will be loaded in the main process." + ), + ) + parser.add_argument( + "--weighting_scheme", + type=str, + default="none", + choices=["sigma_sqrt", "logit_normal", "mode", "cosmap", "none"], + help=('We default to the "none" weighting scheme for uniform sampling and uniform loss'), + ) + parser.add_argument( + "--logit_mean", type=float, default=0.0, help="mean to use when using the `'logit_normal'` weighting scheme." + ) + parser.add_argument( + "--logit_std", type=float, default=1.0, help="std to use when using the `'logit_normal'` weighting scheme." + ) + parser.add_argument( + "--mode_scale", + type=float, + default=1.29, + help="Scale of mode weighting scheme. Only effective when using the `'mode'` as the `weighting_scheme`.", + ) + parser.add_argument( + "--optimizer", + type=str, + default="AdamW", + help=('The optimizer type to use. Choose between ["AdamW", "prodigy"]'), + ) + + parser.add_argument( + "--use_8bit_adam", + action="store_true", + help="Whether or not to use 8-bit Adam from bitsandbytes. Ignored if optimizer is not set to AdamW", + ) + + parser.add_argument( + "--adam_beta1", type=float, default=0.9, help="The beta1 parameter for the Adam and Prodigy optimizers." + ) + parser.add_argument( + "--adam_beta2", type=float, default=0.999, help="The beta2 parameter for the Adam and Prodigy optimizers." + ) + parser.add_argument( + "--prodigy_beta3", + type=float, + default=None, + help="coefficients for computing the Prodigy stepsize using running averages. If set to None, " + "uses the value of square root of beta2. Ignored if optimizer is adamW", + ) + parser.add_argument("--prodigy_decouple", type=bool, default=True, help="Use AdamW style decoupled weight decay") + parser.add_argument("--adam_weight_decay", type=float, default=1e-04, help="Weight decay to use for unet params") + parser.add_argument( + "--adam_weight_decay_text_encoder", type=float, default=1e-03, help="Weight decay to use for text_encoder" + ) + + parser.add_argument( + "--lora_layers", + type=str, + default=None, + help=( + 'The transformer modules to apply LoRA training on. Please specify the layers in a comma separated. E.g. - "to_k,to_q,to_v,to_out.0" will result in lora training of attention layers only' + ), + ) + + parser.add_argument( + "--adam_epsilon", + type=float, + default=1e-08, + help="Epsilon value for the Adam optimizer and Prodigy optimizers.", + ) + + parser.add_argument( + "--prodigy_use_bias_correction", + type=bool, + default=True, + help="Turn on Adam's bias correction. True by default. Ignored if optimizer is adamW", + ) + parser.add_argument( + "--prodigy_safeguard_warmup", + type=bool, + default=True, + help="Remove lr from the denominator of D estimate to avoid issues during warm-up stage. True by default. " + "Ignored if optimizer is adamW", + ) + parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.") + parser.add_argument("--push_to_hub", action="store_true", help="Whether or not to push the model to the Hub.") + parser.add_argument("--hub_token", type=str, default=None, help="The token to use to push to the Model Hub.") + parser.add_argument( + "--hub_model_id", + type=str, + default=None, + help="The name of the repository to keep in sync with the local `output_dir`.", + ) + parser.add_argument( + "--logging_dir", + type=str, + default="logs", + help=( + "[TensorBoard](https://www.tensorflow.org/tensorboard) log directory. Will default to" + " *output_dir/runs/**CURRENT_DATETIME_HOSTNAME***." + ), + ) + parser.add_argument( + "--allow_tf32", + action="store_true", + help=( + "Whether or not to allow TF32 on Ampere GPUs. Can be used to speed up training. For more information, see" + " https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices" + ), + ) + parser.add_argument( + "--cache_latents", + action="store_true", + default=False, + help="Cache the VAE latents", + ) + parser.add_argument( + "--report_to", + type=str, + default="tensorboard", + help=( + 'The integration to report the results and logs to. Supported platforms are `"tensorboard"`' + ' (default), `"wandb"` and `"comet_ml"`. Use `"all"` to report to all integrations.' + ), + ) + parser.add_argument( + "--mixed_precision", + type=str, + default=None, + choices=["no", "fp16", "bf16"], + help=( + "Whether to use mixed precision. Choose between fp16 and bf16 (bfloat16). Bf16 requires PyTorch >=" + " 1.10.and an Nvidia Ampere GPU. Default to the value of accelerate config of the current system or the" + " flag passed with the `accelerate.launch` command. Use this argument to override the accelerate config." + ), + ) + parser.add_argument( + "--upcast_before_saving", + action="store_true", + default=False, + help=( + "Whether to upcast the trained transformer layers to float32 before saving (at the end of training). " + "Defaults to precision dtype used for training to save memory" + ), + ) + parser.add_argument( + "--offload", + action="store_true", + help="Whether to offload the VAE and the text encoder to CPU when they are not used.", + ) + parser.add_argument( + "--prior_generation_precision", + type=str, + default=None, + choices=["no", "fp32", "fp16", "bf16"], + help=( + "Choose prior generation precision between fp32, fp16 and bf16 (bfloat16). Bf16 requires PyTorch >=" + " 1.10.and an Nvidia Ampere GPU. Default to fp16 if a GPU is available else fp32." + ), + ) + parser.add_argument("--local_rank", type=int, default=-1, help="For distributed training: local_rank") + parser.add_argument("--enable_npu_flash_attention", action="store_true", help="Enabla Flash Attention for NPU") + parser.add_argument("--fsdp_text_encoder", action="store_true", help="Use FSDP for text encoder") + + if input_args is not None: + args = parser.parse_args(input_args) + else: + args = parser.parse_args() + + if args.dataset_name is None and args.instance_data_dir is None: + raise ValueError("Specify either `--dataset_name` or `--instance_data_dir`") + + if args.dataset_name is not None and args.instance_data_dir is not None: + raise ValueError("Specify only one of `--dataset_name` or `--instance_data_dir`") + if args.do_fp8_training and args.bnb_quantization_config_path: + raise ValueError("Both `do_fp8_training` and `bnb_quantization_config_path` cannot be passed.") + + env_local_rank = int(os.environ.get("LOCAL_RANK", -1)) + if env_local_rank != -1 and env_local_rank != args.local_rank: + args.local_rank = env_local_rank + + if args.with_prior_preservation: + if args.class_data_dir is None: + raise ValueError("You must specify a data directory for class images.") + if args.class_prompt is None: + raise ValueError("You must specify prompt for class images.") + else: + # logger is not available yet + if args.class_data_dir is not None: + warnings.warn("You need not use --class_data_dir without --with_prior_preservation.") + if args.class_prompt is not None: + warnings.warn("You need not use --class_prompt without --with_prior_preservation.") + + return args + + +class DreamBoothDataset(Dataset): + """ + A dataset to prepare the instance and class images with the prompts for fine-tuning the model. + It pre-processes the images. + """ + + def __init__( + self, + instance_data_root, + instance_prompt, + class_prompt, + class_data_root=None, + class_num=None, + size=1024, + repeats=1, + center_crop=False, + buckets=None, + use_aspect_ratio_buckets=False, + bucket_divisibility=16, + bucket_base_resolutions=None, + ): + self.size = size + self.resolution = size + self.center_crop = center_crop + + self.instance_prompt = instance_prompt + self.custom_instance_prompts = None + self.class_prompt = class_prompt + + # Explicit user-provided bucket list (or None). The concrete list of buckets actually used is + # built from the data in `self.buckets` during preprocessing below. + self._explicit_buckets = buckets + self.use_aspect_ratio_buckets = use_aspect_ratio_buckets + self.bucket_divisibility = bucket_divisibility + self.bucket_base_resolutions = bucket_base_resolutions + + # if --dataset_name is provided or a metadata jsonl file is provided in the local --instance_data directory, + # we load the training data using load_dataset + if args.dataset_name is not None: + try: + from datasets import load_dataset + except ImportError: + raise ImportError( + "You are trying to load your data using the datasets library. If you wish to train using custom " + "captions please install the datasets library: `pip install datasets`. If you wish to load a " + "local folder containing images only, specify --instance_data_dir instead." + ) + # Downloading and loading a dataset from the hub. + # See more about loading custom images at + # https://huggingface.co/docs/datasets/v2.0.0/en/dataset_script + dataset = load_dataset( + args.dataset_name, + args.dataset_config_name, + cache_dir=args.cache_dir, + ) + # Preprocessing the datasets. + column_names = dataset["train"].column_names + + # 6. Get the column names for input/target. + if args.image_column is None: + image_column = column_names[0] + logger.info(f"image column defaulting to {image_column}") + else: + image_column = args.image_column + if image_column not in column_names: + raise ValueError( + f"`--image_column` value '{args.image_column}' not found in dataset columns. Dataset columns are: {', '.join(column_names)}" + ) + instance_images = dataset["train"][image_column] + + if args.caption_column is None: + logger.info( + "No caption column provided, defaulting to instance_prompt for all images. If your dataset " + "contains captions/prompts for the images, make sure to specify the " + "column as --caption_column" + ) + self.custom_instance_prompts = None + else: + if args.caption_column not in column_names: + raise ValueError( + f"`--caption_column` value '{args.caption_column}' not found in dataset columns. Dataset columns are: {', '.join(column_names)}" + ) + custom_instance_prompts = dataset["train"][args.caption_column] + # create final list of captions according to --repeats + self.custom_instance_prompts = [] + for caption in custom_instance_prompts: + self.custom_instance_prompts.extend(itertools.repeat(caption, repeats)) + else: + self.instance_data_root = Path(instance_data_root) + if not self.instance_data_root.exists(): + raise ValueError("Instance images root doesn't exists.") + + instance_images = [Image.open(path) for path in list(Path(instance_data_root).iterdir())] + self.custom_instance_prompts = None + + self.instance_images = [] + for img in instance_images: + self.instance_images.extend(itertools.repeat(img, repeats)) + + self.pixel_values = [] + self.buckets = [] + bucket_to_idx = {} + for i, image in enumerate(self.instance_images): + image = exif_transpose(image) + if not image.mode == "RGB": + image = image.convert("RGB") + + width, height = image.size + + # Assign the image to a bucket. + target = self._bucket_for_image(height, width) + if target not in bucket_to_idx: + bucket_to_idx[target] = len(self.buckets) + self.buckets.append(target) + bucket_idx = bucket_to_idx[target] + + # based on the bucket assignment, define the transformations + image = self.train_transform( + image, + size=target, + center_crop=args.center_crop, + random_flip=args.random_flip, + ) + self.pixel_values.append((image, bucket_idx)) + + self.num_instance_images = len(self.instance_images) + self._length = self.num_instance_images + + if class_data_root is not None: + self.class_data_root = Path(class_data_root) + self.class_data_root.mkdir(parents=True, exist_ok=True) + self.class_images_path = list(self.class_data_root.iterdir()) + if class_num is not None: + self.num_class_images = min(len(self.class_images_path), class_num) + else: + self.num_class_images = len(self.class_images_path) + self._length = max(self.num_class_images, self.num_instance_images) + else: + self.class_data_root = None + + def __len__(self): + return self._length + + def __getitem__(self, index): + example = {} + instance_image, bucket_idx = self.pixel_values[index % self.num_instance_images] + example["index"] = index + example["instance_images"] = instance_image + example["bucket_idx"] = bucket_idx + if self.custom_instance_prompts: + caption = self.custom_instance_prompts[index % self.num_instance_images] + if caption: + example["instance_prompt"] = caption + else: + example["instance_prompt"] = self.instance_prompt + + else: # custom prompts were provided, but length does not match size of image dataset + example["instance_prompt"] = self.instance_prompt + + if self.class_data_root: + class_image = Image.open(self.class_images_path[index % self.num_class_images]) + class_image = exif_transpose(class_image) + + if not class_image.mode == "RGB": + class_image = class_image.convert("RGB") + # Match the class image to the paired instance image's bucket so they can be stacked into one batch. + example["class_images"] = self.train_transform( + class_image, size=self.buckets[bucket_idx], center_crop=self.center_crop + ) + example["class_prompt"] = self.class_prompt + + return example + + def _bucket_for_image(self, height, width): + # An explicit bucket list takes priority: pick the nearest, upscaling smaller images to cover it. + if self._explicit_buckets is not None: + return self._explicit_buckets[find_nearest_bucket(height, width, self._explicit_buckets)] + # On-the-fly bucketing: cap the ladder to the image's own resolution so smaller images are + # assigned to a smaller bucket rather than being upscaled (mirrors ostris' bucketing). + if self.use_aspect_ratio_buckets: + resolution = min(self.resolution, round((height * width) ** 0.5)) + ladder = generate_aspect_ratio_buckets( + resolution, + divisibility=self.bucket_divisibility, + base_resolutions=self.bucket_base_resolutions, + ) + return ladder[find_nearest_bucket(height, width, ladder)] + # No bucketing: a single square bucket reproduces the fixed-size resize + crop. + return (self.resolution, self.resolution) + + def train_transform(self, image, size, center_crop=False, random_flip=False): + # Resize preserving aspect ratio so the image covers the bucket, then crop to the bucket size. + target_height, target_width = size + width, height = image.size + scale = max(target_height / height, target_width / width) + new_height, new_width = round(height * scale), round(width * scale) + image = TF.resize(image, [new_height, new_width], interpolation=transforms.InterpolationMode.BILINEAR) + if center_crop: + image = TF.center_crop(image, size) + else: + i, j, h, w = transforms.RandomCrop.get_params(image, output_size=size) + image = TF.crop(image, i, j, h, w) + if random_flip and random.random() < 0.5: + image = TF.hflip(image) + return TF.normalize(TF.to_tensor(image), [0.5], [0.5]) + + +def collate_fn(examples, with_prior_preservation=False): + indices = [example["index"] for example in examples] + pixel_values = [example["instance_images"] for example in examples] + # Keep instance_prompts unchanged for prompt cache precompute; prompts may be extended with class prompts below. + instance_prompts = [example["instance_prompt"] for example in examples] + prompts = [example["instance_prompt"] for example in examples] + + # Concat class and instance examples for prior preservation. + # We do this to avoid doing two forward passes. + if with_prior_preservation: + pixel_values += [example["class_images"] for example in examples] + prompts += [example["class_prompt"] for example in examples] + + pixel_values = torch.stack(pixel_values) + pixel_values = pixel_values.to(memory_format=torch.contiguous_format).float() + + batch = { + "indices": indices, + "pixel_values": pixel_values, + "instance_prompts": instance_prompts, + "prompts": prompts, + } + return batch + + +class BucketBatchSampler(BatchSampler): + def __init__(self, dataset: DreamBoothDataset, batch_size: int, drop_last: bool = False, seed: int = None): + if not isinstance(batch_size, int) or batch_size <= 0: + raise ValueError("batch_size should be a positive integer value, but got batch_size={}".format(batch_size)) + if not isinstance(drop_last, bool): + raise ValueError("drop_last should be a boolean value, but got drop_last={}".format(drop_last)) + + self.dataset = dataset + self.batch_size = batch_size + self.drop_last = drop_last + self.generator = random.Random(seed) if seed is not None else random + + # Group indices by bucket + self.bucket_indices = [[] for _ in range(len(self.dataset.buckets))] + for idx, (_, bucket_idx) in enumerate(self.dataset.pixel_values): + self.bucket_indices[bucket_idx].append(idx) + + self.sampler_len = 0 + for indices_in_bucket in self.bucket_indices: + num_batches, remainder = divmod(len(indices_in_bucket), self.batch_size) + self.sampler_len += num_batches + if remainder > 0 and not self.drop_last: + self.sampler_len += 1 + + def __iter__(self): + batches = [] + for indices_in_bucket in self.bucket_indices: + shuffled_indices = indices_in_bucket.copy() + self.generator.shuffle(shuffled_indices) + for i in range(0, len(shuffled_indices), self.batch_size): + batch = shuffled_indices[i : i + self.batch_size] + if len(batch) < self.batch_size and self.drop_last: + continue + batches.append(batch) + + self.generator.shuffle(batches) + for batch in batches: + yield batch + + def __len__(self): + return self.sampler_len + + +class PromptDataset(Dataset): + "A simple dataset to prepare the prompts to generate class images on multiple GPUs." + + def __init__(self, prompt, num_samples): + self.prompt = prompt + self.num_samples = num_samples + + def __len__(self): + return self.num_samples + + def __getitem__(self, index): + example = {} + example["prompt"] = self.prompt + example["index"] = index + return example + + +def main(args): + if args.report_to == "wandb" and args.hub_token is not None: + raise ValueError( + "You cannot use both --report_to=wandb and --hub_token due to a security risk of exposing your token." + " Please use `hf auth login` to authenticate with the Hub." + ) + + if torch.backends.mps.is_available() and args.mixed_precision == "bf16": + # due to pytorch#99272, MPS does not yet support bfloat16. + raise ValueError( + "Mixed precision training with bfloat16 is not supported on MPS. Please use fp16 (recommended) or fp32 instead." + ) + if args.do_fp8_training: + from torchao.float8 import Float8LinearConfig, convert_to_float8_training + + logging_dir = Path(args.output_dir, args.logging_dir) + + accelerator_project_config = ProjectConfiguration(project_dir=args.output_dir, logging_dir=logging_dir) + kwargs = DistributedDataParallelKwargs(find_unused_parameters=True) + accelerator = Accelerator( + gradient_accumulation_steps=args.gradient_accumulation_steps, + mixed_precision=args.mixed_precision, + log_with=args.report_to, + project_config=accelerator_project_config, + kwargs_handlers=[kwargs], + ) + + # Disable AMP for MPS. + if torch.backends.mps.is_available(): + accelerator.native_amp = False + + if args.report_to == "wandb": + if not is_wandb_available(): + raise ImportError("Make sure to install wandb if you want to use it for logging during training.") + + # Make one log on every process with the configuration for debugging. + logging.basicConfig( + format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", + datefmt="%m/%d/%Y %H:%M:%S", + level=logging.INFO, + ) + logger.info(accelerator.state, main_process_only=False) + if accelerator.is_local_main_process: + transformers.utils.logging.set_verbosity_warning() + diffusers.utils.logging.set_verbosity_info() + else: + transformers.utils.logging.set_verbosity_error() + diffusers.utils.logging.set_verbosity_error() + + # If passed along, set the training seed now. + if args.seed is not None: + set_seed(args.seed) + + # Generate class images if prior preservation is enabled. + if args.with_prior_preservation: + class_images_dir = Path(args.class_data_dir) + if not class_images_dir.exists(): + class_images_dir.mkdir(parents=True) + cur_class_images = len(list(class_images_dir.iterdir())) + + if cur_class_images < args.num_class_images: + has_supported_fp16_accelerator = torch.cuda.is_available() or torch.backends.mps.is_available() + torch_dtype = torch.float16 if has_supported_fp16_accelerator else torch.float32 + if args.prior_generation_precision == "fp32": + torch_dtype = torch.float32 + elif args.prior_generation_precision == "fp16": + torch_dtype = torch.float16 + elif args.prior_generation_precision == "bf16": + torch_dtype = torch.bfloat16 + + pipeline = ModularPipeline.from_pretrained(args.pretrained_model_name_or_path) + pipeline.load_components(torch_dtype=torch_dtype) + + num_new_images = args.num_class_images - cur_class_images + logger.info(f"Number of class images to sample: {num_new_images}.") + + sample_dataset = PromptDataset(args.class_prompt, num_new_images) + sample_dataloader = torch.utils.data.DataLoader(sample_dataset, batch_size=args.sample_batch_size) + + sample_dataloader = accelerator.prepare(sample_dataloader) + pipeline.to(accelerator.device) + + for example in tqdm( + sample_dataloader, desc="Generating class images", disable=not accelerator.is_local_main_process + ): + with torch.autocast(device_type=accelerator.device.type, dtype=torch_dtype): + images = pipeline(prompt=example["prompt"], output_type="pil").values["images"] + + for i, image in enumerate(images): + hash_image = insecure_hashlib.sha1(image.tobytes()).hexdigest() + image_filename = class_images_dir / f"{example['index'][i] + cur_class_images}-{hash_image}.jpg" + image.save(image_filename) + + del pipeline + free_memory() + + # Handle the repository creation + if accelerator.is_main_process: + if args.output_dir is not None: + os.makedirs(args.output_dir, exist_ok=True) + + if args.push_to_hub: + repo_id = create_repo( + repo_id=args.hub_model_id or Path(args.output_dir).name, + exist_ok=True, + ).repo_id + + # Load the tokenizers + tokenizer = AutoProcessor.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="tokenizer", + revision=args.revision, + ) + + # For mixed precision training we cast all non-trainable weights (vae, text_encoder and transformer) to half-precision + # as these weights are only used for inference, keeping weights in full precision is not required. + weight_dtype = torch.float32 + if accelerator.mixed_precision == "fp16": + weight_dtype = torch.float16 + elif accelerator.mixed_precision == "bf16": + weight_dtype = torch.bfloat16 + + # Load scheduler and models + noise_scheduler = FlowMatchEulerDiscreteScheduler(shift=FLOW_SHIFT, use_dynamic_shifting=False) + noise_scheduler_copy = copy.deepcopy(noise_scheduler) + vae = AutoencoderKLFlux2.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="vae", + revision=args.revision, + variant=args.variant, + ) + latents_bn_mean = vae.bn.running_mean.view(1, -1, 1, 1).to(accelerator.device) + latents_bn_std = torch.sqrt(vae.bn.running_var.view(1, -1, 1, 1) + vae.config.batch_norm_eps).to( + accelerator.device + ) + + quantization_config = None + if args.bnb_quantization_config_path is not None: + with open(args.bnb_quantization_config_path, "r") as f: + config_kwargs = json.load(f) + if "load_in_4bit" in config_kwargs and config_kwargs["load_in_4bit"]: + config_kwargs["bnb_4bit_compute_dtype"] = weight_dtype + quantization_config = BitsAndBytesConfig(**config_kwargs) + + transformer = Flux3Transformer2DModel.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="transformer", + revision=args.revision, + variant=args.variant, + quantization_config=quantization_config, + torch_dtype=weight_dtype, + ) + if args.bnb_quantization_config_path is not None: + transformer = prepare_model_for_kbit_training(transformer, use_gradient_checkpointing=False) + + text_encoder = Qwen3VLForConditionalGeneration.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="text_encoder", + revision=args.revision, + variant=args.variant, + torch_dtype=weight_dtype, + ) + text_encoder.requires_grad_(False) + + # We only train the additional adapter LoRA layers + transformer.requires_grad_(False) + vae.requires_grad_(False) + + if args.enable_npu_flash_attention: + if is_torch_npu_available(): + logger.info("npu flash attention enabled.") + transformer.set_attention_backend("_native_npu") + else: + raise ValueError("npu flash attention requires torch_npu extensions and is supported only on npu device ") + + if torch.backends.mps.is_available() and weight_dtype == torch.bfloat16: + # due to pytorch#99272, MPS does not yet support bfloat16. + raise ValueError( + "Mixed precision training with bfloat16 is not supported on MPS. Please use fp16 (recommended) or fp32 instead." + ) + + to_kwargs = {"dtype": weight_dtype, "device": accelerator.device} if not args.offload else {"dtype": weight_dtype} + # flux vae is stable in bf16 so load it in weight_dtype to reduce memory + vae.to(**to_kwargs) + # we never offload the transformer to CPU, so we can just use the accelerator device + transformer_to_kwargs = ( + {"device": accelerator.device} + if args.bnb_quantization_config_path is not None + else {"device": accelerator.device, "dtype": weight_dtype} + ) + + is_fsdp = getattr(accelerator.state, "fsdp_plugin", None) is not None + if not is_fsdp: + transformer.to(**transformer_to_kwargs) + + if args.do_fp8_training: + convert_to_float8_training( + transformer, module_filter_fn=module_filter_fn, config=Float8LinearConfig(pad_inner_dim=True) + ) + + text_encoder.to(**to_kwargs) + + if args.gradient_checkpointing: + transformer.enable_gradient_checkpointing() + + if args.lora_layers is not None: + target_modules = [layer.strip() for layer in args.lora_layers.split(",")] + else: + # Attention + feed-forward projections across every block (mode blocks and single blocks). These are exactly + # the leaf modules the non-diffusers (ostris) FLUX.3 LoRA converter maps, so adapters trained here stay + # interchangeable with that layout. Pass `--lora_layers` to target a different set. + target_modules = [ + "to_q", + "to_k", + "to_v", + "to_out.0", + "ff.linear_in", + "ff.linear_out", + ] + + # now we will add new LoRA weights the transformer layers + transformer_lora_config = LoraConfig( + r=args.rank, + lora_alpha=args.lora_alpha, + lora_dropout=args.lora_dropout, + init_lora_weights="gaussian", + target_modules=target_modules, + ) + transformer.add_adapter(transformer_lora_config) + + def unwrap_model(model): + model = accelerator.unwrap_model(model) + model = model._orig_mod if is_compiled_module(model) else model + return model + + # create custom saving & loading hooks so that `accelerator.save_state(...)` serializes in a nice format + def save_model_hook(models, weights, output_dir): + transformer_cls = type(unwrap_model(transformer)) + + # 1) Validate and pick the transformer model + modules_to_save: dict[str, Any] = {} + transformer_model = None + + for model in models: + if isinstance(unwrap_model(model), transformer_cls): + transformer_model = model + modules_to_save["transformer"] = model + else: + raise ValueError(f"unexpected save model: {model.__class__}") + + if transformer_model is None: + raise ValueError("No transformer model found in 'models'") + + # 2) Optionally gather FSDP state dict once + state_dict = accelerator.get_state_dict(model) if is_fsdp else None + + # 3) Only main process materializes the LoRA state dict + transformer_lora_layers_to_save = None + if accelerator.is_main_process: + peft_kwargs = {} + if is_fsdp: + peft_kwargs["state_dict"] = state_dict + + transformer_lora_layers_to_save = get_peft_model_state_dict( + unwrap_model(transformer_model) if is_fsdp else transformer_model, + **peft_kwargs, + ) + + if is_fsdp: + transformer_lora_layers_to_save = _to_cpu_contiguous(transformer_lora_layers_to_save) + + # make sure to pop weight so that corresponding model is not saved again + if weights: + weights.pop() + + Flux3LoraLoaderMixin.save_lora_weights( + output_dir, + transformer_lora_layers=transformer_lora_layers_to_save, + **_collate_lora_metadata(modules_to_save), + ) + + def load_model_hook(models, input_dir): + transformer_ = None + + if not is_fsdp: + while len(models) > 0: + model = models.pop() + + if isinstance(unwrap_model(model), type(unwrap_model(transformer))): + transformer_ = unwrap_model(model) + else: + raise ValueError(f"unexpected save model: {model.__class__}") + else: + transformer_ = Flux3Transformer2DModel.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="transformer", + ) + transformer_.add_adapter(transformer_lora_config) + + lora_state_dict = Flux3LoraLoaderMixin.lora_state_dict(input_dir) + + transformer_state_dict = { + f"{k.replace('transformer.', '')}": v for k, v in lora_state_dict.items() if k.startswith("transformer.") + } + transformer_state_dict = convert_unet_state_dict_to_peft(transformer_state_dict) + incompatible_keys = set_peft_model_state_dict(transformer_, transformer_state_dict, adapter_name="default") + if incompatible_keys is not None: + # check only for unexpected keys + unexpected_keys = getattr(incompatible_keys, "unexpected_keys", None) + if unexpected_keys: + logger.warning( + f"Loading adapter weights from state_dict led to unexpected keys not found in the model: " + f" {unexpected_keys}. " + ) + + # Make sure the trainable params are in float32. This is again needed since the base models + # are in `weight_dtype`. More details: + # https://github.com/huggingface/diffusers/pull/6514#discussion_r1449796804 + if args.mixed_precision == "fp16": + models = [transformer_] + # only upcast trainable parameters (LoRA) into fp32 + cast_training_params(models) + + accelerator.register_save_state_pre_hook(save_model_hook) + accelerator.register_load_state_pre_hook(load_model_hook) + + # Enable TF32 for faster training on Ampere GPUs, + # cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices + if args.allow_tf32 and torch.cuda.is_available(): + torch.backends.cuda.matmul.allow_tf32 = True + + if args.scale_lr: + args.learning_rate = ( + args.learning_rate * args.gradient_accumulation_steps * args.train_batch_size * accelerator.num_processes + ) + + # Make sure the trainable params are in float32. + if args.mixed_precision == "fp16": + models = [transformer] + # only upcast trainable parameters (LoRA) into fp32 + cast_training_params(models, dtype=torch.float32) + + transformer_lora_parameters = list(filter(lambda p: p.requires_grad, transformer.parameters())) + + # Optimization parameters + transformer_parameters_with_lr = {"params": transformer_lora_parameters, "lr": args.learning_rate} + params_to_optimize = [transformer_parameters_with_lr] + + # Optimizer creation + if not (args.optimizer.lower() == "prodigy" or args.optimizer.lower() == "adamw"): + logger.warning( + f"Unsupported choice of optimizer: {args.optimizer}.Supported optimizers include [adamW, prodigy]." + "Defaulting to adamW" + ) + args.optimizer = "adamw" + + if args.use_8bit_adam and not args.optimizer.lower() == "adamw": + logger.warning( + f"use_8bit_adam is ignored when optimizer is not set to 'AdamW'. Optimizer was " + f"set to {args.optimizer.lower()}" + ) + + if args.optimizer.lower() == "adamw": + if args.use_8bit_adam: + try: + import bitsandbytes as bnb + except ImportError: + raise ImportError( + "To use 8-bit Adam, please install the bitsandbytes library: `pip install bitsandbytes`." + ) + + optimizer_class = bnb.optim.AdamW8bit + else: + optimizer_class = torch.optim.AdamW + + optimizer = optimizer_class( + params_to_optimize, + betas=(args.adam_beta1, args.adam_beta2), + weight_decay=args.adam_weight_decay, + eps=args.adam_epsilon, + ) + + if args.optimizer.lower() == "prodigy": + try: + import prodigyopt + except ImportError: + raise ImportError("To use Prodigy, please install the prodigyopt library: `pip install prodigyopt`") + + optimizer_class = prodigyopt.Prodigy + + if args.learning_rate <= 0.1: + logger.warning( + "Learning rate is too low. When using prodigy, it's generally better to set learning rate around 1.0" + ) + + optimizer = optimizer_class( + params_to_optimize, + betas=(args.adam_beta1, args.adam_beta2), + beta3=args.prodigy_beta3, + weight_decay=args.adam_weight_decay, + eps=args.adam_epsilon, + decouple=args.prodigy_decouple, + use_bias_correction=args.prodigy_use_bias_correction, + safeguard_warmup=args.prodigy_safeguard_warmup, + ) + + # Resolve the bucketing mode. Bucketing must be enabled explicitly with --use_aspect_ratio_buckets; + # a bucket list without that flag is an error. With the flag, an explicit --aspect_ratio_buckets list + # drives assignment, otherwise buckets are computed on the fly inside the dataset. Without the flag a + # single square bucket reproduces the fixed-size resize + crop. + if args.aspect_ratio_buckets is not None and not args.use_aspect_ratio_buckets: + raise ValueError("--aspect_ratio_buckets requires --use_aspect_ratio_buckets to be set.") + if args.aspect_ratio_buckets is not None: + buckets = parse_buckets_string(args.aspect_ratio_buckets) + use_aspect_ratio_buckets = False + logger.info(f"Using explicit aspect ratio buckets: {buckets}") + elif args.use_aspect_ratio_buckets: + buckets = None + use_aspect_ratio_buckets = True + logger.info( + "No --aspect_ratio_buckets provided; auto-computing aspect ratio buckets on the fly from --resolution." + ) + else: + buckets = [(args.resolution, args.resolution)] + use_aspect_ratio_buckets = False + + # Dataset and DataLoaders creation: + train_dataset = DreamBoothDataset( + instance_data_root=args.instance_data_dir, + instance_prompt=args.instance_prompt, + class_prompt=args.class_prompt, + class_data_root=args.class_data_dir if args.with_prior_preservation else None, + class_num=args.num_class_images, + size=args.resolution, + repeats=args.repeats, + center_crop=args.center_crop, + buckets=buckets, + use_aspect_ratio_buckets=use_aspect_ratio_buckets, + ) + precompute_latents = args.cache_latents or train_dataset.custom_instance_prompts + batch_sampler = BucketBatchSampler(train_dataset, batch_size=args.train_batch_size, drop_last=True, seed=args.seed) + train_dataloader = torch.utils.data.DataLoader( + train_dataset, + batch_sampler=batch_sampler, + collate_fn=lambda examples: collate_fn(examples, args.with_prior_preservation), + num_workers=args.dataloader_num_workers, + ) + + def compute_text_embeddings(prompt, text_encoder, tokenizer): + # FLUX.3 has no classic text-encoding pipeline; encode directly with the frozen Qwen3-VL encoder. + with torch.no_grad(): + prompt_embeds = get_qwen3vl_prompt_embeds( + text_encoder, + tokenizer, + prompt, + accelerator.device, + weight_dtype, + text_encoder_out_layers=args.text_encoder_out_layers, + max_sequence_length=args.max_sequence_length, + ) + # FLUX.3 text ids are positional only (identical for every sample of a given length). We carry them + # batched here so the [instance, class] concat / repeat / caption-dropout plumbing below is unchanged; + # the transformer consumes a single (seq, 4) slice at the forward call. + text_ids = prepare_text_ids(prompt_embeds.shape[1], accelerator.device) + text_ids = text_ids.unsqueeze(0).expand(prompt_embeds.shape[0], -1, -1) + return prompt_embeds, text_ids + + # If no type of tuning is done on the text_encoder and custom instance prompts are NOT + # provided (i.e. the --instance_prompt is used for all images), we encode the instance prompt once to avoid + # the redundant encoding. + if not train_dataset.custom_instance_prompts: + with offload_models(text_encoder, device=accelerator.device, offload=args.offload): + instance_prompt_hidden_states, instance_text_ids = compute_text_embeddings( + args.instance_prompt, text_encoder, tokenizer + ) + + # Handle class prompt for prior-preservation. + if args.with_prior_preservation: + with offload_models(text_encoder, device=accelerator.device, offload=args.offload): + class_prompt_hidden_states, class_text_ids = compute_text_embeddings( + args.class_prompt, text_encoder, tokenizer + ) + + # When caption dropout is enabled, we precompute the empty ("") prompt embedding once and swap it in + # for randomly selected instance samples at training time (see the training loop below). + if args.caption_dropout > 0: + with offload_models(text_encoder, device=accelerator.device, offload=args.offload): + empty_prompt_hidden_states, empty_text_ids = compute_text_embeddings("", text_encoder, tokenizer) + + # FLUX.3 validation runs through the modular pipeline (which encodes prompts itself), so we only assemble + # the generation kwargs here rather than precomputing text embeddings. + validation_pipeline_args = {} + if args.validation_prompt is not None: + validation_pipeline_args = { + "prompt": args.validation_prompt, + "negative_prompt": "", + "height": args.resolution, + "width": args.resolution, + "num_inference_steps": 28, + # Encode the validation prompt exactly as training did (same window cap and hidden layers), so the + # modular pipeline's text embeddings match the ones the adapter was trained against. + "max_sequence_length": args.max_sequence_length, + "text_encoder_out_layers": tuple(args.text_encoder_out_layers), + } + + # Init FSDP for text encoder + if args.fsdp_text_encoder: + fsdp_kwargs = get_fsdp_kwargs_from_accelerator(accelerator) + text_encoder = wrap_with_fsdp( + model=text_encoder, + device=accelerator.device, + offload=args.offload, + limit_all_gathers=True, + use_orig_params=True, + fsdp_kwargs=fsdp_kwargs, + ) + dist.barrier() + + # If custom instance prompts are NOT provided (i.e. the instance prompt is used for all images), + # pack the statically computed variables appropriately here. This is so that we don't + # have to pass them to the dataloader. + if not train_dataset.custom_instance_prompts: + prompt_embeds = instance_prompt_hidden_states + text_ids = instance_text_ids + if args.with_prior_preservation: + prompt_embeds = torch.cat([prompt_embeds, class_prompt_hidden_states], dim=0) + text_ids = torch.cat([text_ids, class_text_ids], dim=0) + static_prompt_embeds = prompt_embeds + static_text_ids = text_ids + + # if cache_latents is set to True, we encode images to latents and store them. + # Similar to pre-encoding in the case of a single instance prompt, if custom prompts are provided + # we encode them in advance as well. + if args.cache_latents: + instance_latents_cache = [None] * train_dataset.num_instance_images + class_latents_cache = [None] * train_dataset.num_instance_images if args.with_prior_preservation else None + if train_dataset.custom_instance_prompts: + prompt_embeds_cache = [None] * train_dataset.num_instance_images + text_ids_cache = [None] * train_dataset.num_instance_images + if precompute_latents: + cache_batch_sampler = BucketBatchSampler( + train_dataset, batch_size=args.train_batch_size, drop_last=False, seed=args.seed + ) + cache_dataloader = torch.utils.data.DataLoader( + train_dataset, + batch_sampler=cache_batch_sampler, + collate_fn=lambda examples: collate_fn(examples, args.with_prior_preservation), + num_workers=args.dataloader_num_workers, + ) + for batch in tqdm(cache_dataloader, desc="Caching latents"): + with torch.no_grad(): + sample_indices = batch["indices"] + if args.cache_latents: + with offload_models(vae, device=accelerator.device, offload=args.offload): + batch["pixel_values"] = batch["pixel_values"].to( + accelerator.device, non_blocking=True, dtype=vae.dtype + ) + latents = vae.encode(batch["pixel_values"]).latent_dist.mode() + if args.with_prior_preservation: + instance_latents, class_latents = torch.chunk(latents, 2, dim=0) + else: + instance_latents = latents + for i, idx in enumerate(sample_indices): + instance_latents_cache[idx] = instance_latents[i : i + 1] + if args.with_prior_preservation: + class_latents_cache[idx] = class_latents[i : i + 1] + if train_dataset.custom_instance_prompts: + if args.fsdp_text_encoder: + prompt_embeds, text_ids = compute_text_embeddings( + batch["instance_prompts"], text_encoder, tokenizer + ) + else: + with offload_models(text_encoder, device=accelerator.device, offload=args.offload): + prompt_embeds, text_ids = compute_text_embeddings( + batch["instance_prompts"], text_encoder, tokenizer + ) + for i, idx in enumerate(sample_indices): + prompt_embeds_cache[idx] = prompt_embeds[i : i + 1] + text_ids_cache[idx] = text_ids[i : i + 1] + + if args.cache_latents: + assert all(latents is not None for latents in instance_latents_cache), "Latent cache has unfilled entries." + if args.with_prior_preservation: + assert all(latents is not None for latents in class_latents_cache), ( + "Class latent cache has unfilled entries." + ) + if train_dataset.custom_instance_prompts: + assert all(embeds is not None for embeds in prompt_embeds_cache), ( + "Prompt embedding cache has unfilled entries." + ) + assert all(ids is not None for ids in text_ids_cache), "Text ID cache has unfilled entries." + + # move back to cpu before deleting to ensure memory is freed see: https://github.com/huggingface/diffusers/issues/11376#issue-3008144624 + if args.cache_latents: + vae = vae.to("cpu") + del vae + + # move back to cpu before deleting to ensure memory is freed see: https://github.com/huggingface/diffusers/issues/11376#issue-3008144624 + text_encoder = text_encoder.to("cpu") + del text_encoder, tokenizer + free_memory() + + # Scheduler and math around the number of training steps. + # Check the PR https://github.com/huggingface/diffusers/pull/8312 for detailed explanation. + num_warmup_steps_for_scheduler = args.lr_warmup_steps * accelerator.num_processes + if args.max_train_steps is None: + len_train_dataloader_after_sharding = math.ceil(len(train_dataloader) / accelerator.num_processes) + num_update_steps_per_epoch = math.ceil(len_train_dataloader_after_sharding / args.gradient_accumulation_steps) + num_training_steps_for_scheduler = ( + args.num_train_epochs * accelerator.num_processes * num_update_steps_per_epoch + ) + else: + num_training_steps_for_scheduler = args.max_train_steps * accelerator.num_processes + + lr_scheduler = get_scheduler( + args.lr_scheduler, + optimizer=optimizer, + num_warmup_steps=num_warmup_steps_for_scheduler, + num_training_steps=num_training_steps_for_scheduler, + num_cycles=args.lr_num_cycles, + power=args.lr_power, + ) + + # Prepare everything with our `accelerator`. + transformer, optimizer, train_dataloader, lr_scheduler = accelerator.prepare( + transformer, optimizer, train_dataloader, lr_scheduler + ) + + # We need to recalculate our total training steps as the size of the training dataloader may have changed. + num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps) + if args.max_train_steps is None: + args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch + if num_training_steps_for_scheduler != args.max_train_steps: + logger.warning( + f"The length of the 'train_dataloader' after 'accelerator.prepare' ({len(train_dataloader)}) does not match " + f"the expected length ({len_train_dataloader_after_sharding}) when the learning rate scheduler was created. " + f"This inconsistency may result in the learning rate scheduler not functioning properly." + ) + # Afterwards we recalculate our number of training epochs + args.num_train_epochs = math.ceil(args.max_train_steps / num_update_steps_per_epoch) + + # We need to initialize the trackers we use, and also store our configuration. + # The trackers initializes automatically on the main process. + if accelerator.is_main_process: + tracker_name = "dreambooth-flux3-lora" + args_cp = vars(args).copy() + args_cp["text_encoder_out_layers"] = str(args_cp["text_encoder_out_layers"]) + accelerator.init_trackers(tracker_name, config=args_cp) + + # Train! + total_batch_size = args.train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps + + logger.info("***** Running training *****") + logger.info(f" Num examples = {len(train_dataset)}") + logger.info(f" Num batches each epoch = {len(train_dataloader)}") + logger.info(f" Num Epochs = {args.num_train_epochs}") + logger.info(f" Instantaneous batch size per device = {args.train_batch_size}") + logger.info(f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}") + logger.info(f" Gradient Accumulation steps = {args.gradient_accumulation_steps}") + logger.info(f" Total optimization steps = {args.max_train_steps}") + global_step = 0 + first_epoch = 0 + + # Potentially load in the weights and states from a previous save + if args.resume_from_checkpoint: + if args.resume_from_checkpoint != "latest": + path = os.path.basename(args.resume_from_checkpoint) + else: + # Get the mos recent checkpoint + dirs = os.listdir(args.output_dir) + dirs = [d for d in dirs if d.startswith("checkpoint")] + dirs = sorted(dirs, key=lambda x: int(x.split("-")[1])) + path = dirs[-1] if len(dirs) > 0 else None + + if path is None: + accelerator.print( + f"Checkpoint '{args.resume_from_checkpoint}' does not exist. Starting a new training run." + ) + args.resume_from_checkpoint = None + initial_global_step = 0 + else: + accelerator.print(f"Resuming from checkpoint {path}") + accelerator.load_state(os.path.join(args.output_dir, path)) + global_step = int(path.split("-")[1]) + + initial_global_step = global_step + first_epoch = global_step // num_update_steps_per_epoch + + else: + initial_global_step = 0 + + progress_bar = tqdm( + range(0, args.max_train_steps), + initial=initial_global_step, + desc="Steps", + # Only show the progress bar once on each machine. + disable=not accelerator.is_local_main_process, + ) + + def get_sigmas(timesteps, n_dim=4, dtype=torch.float32): + sigmas = noise_scheduler_copy.sigmas.to(device=accelerator.device, dtype=dtype) + schedule_timesteps = noise_scheduler_copy.timesteps.to(accelerator.device) + timesteps = timesteps.to(accelerator.device) + step_indices = [(schedule_timesteps == t).nonzero().item() for t in timesteps] + + sigma = sigmas[step_indices].flatten() + while len(sigma.shape) < n_dim: + sigma = sigma.unsqueeze(-1) + return sigma + + for epoch in range(first_epoch, args.num_train_epochs): + transformer.train() + + for batch in train_dataloader: + models_to_accumulate = [transformer] + sample_indices = batch["indices"] + prompts = batch["prompts"] + + with accelerator.accumulate(models_to_accumulate): + if train_dataset.custom_instance_prompts: + prompt_embeds = torch.cat([prompt_embeds_cache[idx] for idx in sample_indices], dim=0) + text_ids = torch.cat([text_ids_cache[idx] for idx in sample_indices], dim=0) + if args.with_prior_preservation: + prompt_embeds = torch.cat( + [prompt_embeds, class_prompt_hidden_states.repeat(len(sample_indices), 1, 1)], dim=0 + ) + text_ids = torch.cat([text_ids, class_text_ids.repeat(len(sample_indices), 1, 1)], dim=0) + else: + # With prior preservation, prompt_embeds/text_ids already contain [instance, class] entries, + # while collate_fn orders batches as [inst1..instB, class1..classB]. Repeat each entry along + # dim 0 to preserve that grouping instead of interleaving [inst, class, inst, class, ...]. + num_repeat_elements = len(prompts) // 2 if args.with_prior_preservation else len(prompts) + prompt_embeds = static_prompt_embeds.repeat_interleave(num_repeat_elements, dim=0) + text_ids = static_text_ids.repeat_interleave(num_repeat_elements, dim=0) + + # Caption dropout: replace a sample's caption embedding with the empty-prompt embedding so it + # trains unconditionally. Both paths order embeddings as [instance..., class...], so only the + # leading instance rows are dropped, never class/prior captions. + if args.caption_dropout > 0: + n_inst = len(sample_indices) + drop_mask = torch.rand(n_inst, device=prompt_embeds.device) < args.caption_dropout + if drop_mask.any(): + prompt_embeds[:n_inst][drop_mask] = empty_prompt_hidden_states.to( + device=prompt_embeds.device, dtype=prompt_embeds.dtype + ) + text_ids[:n_inst][drop_mask] = empty_text_ids.to(device=text_ids.device, dtype=text_ids.dtype) + + # Convert images to latent space + if args.cache_latents: + model_input = torch.cat([instance_latents_cache[idx] for idx in sample_indices], dim=0) + if args.with_prior_preservation: + model_input = torch.cat( + [model_input, torch.cat([class_latents_cache[idx] for idx in sample_indices], dim=0)], + dim=0, + ) + else: + with offload_models(vae, device=accelerator.device, offload=args.offload): + pixel_values = batch["pixel_values"].to(device=accelerator.device, dtype=vae.dtype) + model_input = vae.encode(pixel_values).latent_dist.mode() + + model_input = patchify_latents(model_input) + model_input = (model_input - latents_bn_mean) / latents_bn_std + + _, _, lh, lw = model_input.shape + # Sample noise that we'll add to the latents + noise = torch.randn_like(model_input) + bsz = model_input.shape[0] + + # Sample a random timestep for each image + # for weighting schemes where we sample timesteps non-uniformly + u = compute_density_for_timestep_sampling( + weighting_scheme=args.weighting_scheme, + batch_size=bsz, + logit_mean=args.logit_mean, + logit_std=args.logit_std, + mode_scale=args.mode_scale, + ) + indices = (u * noise_scheduler_copy.config.num_train_timesteps).long() + timesteps = noise_scheduler_copy.timesteps[indices].to(device=model_input.device) + + # Add noise according to flow matching. + # zt = (1 - texp) * x + texp * z1 + sigmas = get_sigmas(timesteps, n_dim=model_input.ndim, dtype=model_input.dtype) + noisy_model_input = (1.0 - sigmas) * model_input + sigmas * noise + + # [B, C, H, W] -> [B, H*W, C] + packed_noisy_model_input = pack_latents(noisy_model_input) + image_ids = prepare_latent_ids(lh, lw, accelerator.device) # (image_seq_len, 4) + # FLUX.3 is not guidance-distilled: no guidance argument. The flow timestep is a per-sample scalar in + # [0, 1] (the transformer scales it by 1000 and broadcasts it over the image tokens internally); the + # text context and any reference tokens sit at timestep 0. + timestep = sigmas.reshape(bsz) + + # Predict the noise residual. The transformer returns only the generated image tokens, so there is + # nothing to slice/unpack afterwards. + with accelerator.autocast(): + model_pred = transformer( + hidden_states=packed_noisy_model_input, # (B, image_seq_len, C) + img_ids=image_ids, # (image_seq_len, 4) + timestep=timestep, # (B,) + encoder_hidden_states=prompt_embeds, + txt_ids=text_ids[0], # (text_seq_len, 4), shared across the batch + return_dict=False, + )[0] + + # these weighting schemes use a uniform timestep sampling + # and instead post-weight the loss + weighting = compute_loss_weighting_for_sd3( + weighting_scheme=args.weighting_scheme, sigmas=sigmas.reshape(bsz, 1, 1) + ) + + # flow matching loss (velocity target) on the packed image tokens + target = pack_latents(noise) - pack_latents(model_input) + + if args.with_prior_preservation: + # Chunk the noise and model_pred into two parts and compute the loss on each part separately. + model_pred, model_pred_prior = torch.chunk(model_pred, 2, dim=0) + target, target_prior = torch.chunk(target, 2, dim=0) + weighting, weighting_prior = torch.chunk(weighting, 2, dim=0) + + # Compute prior loss + prior_loss = torch.mean( + (weighting_prior.float() * (model_pred_prior.float() - target_prior.float()) ** 2).reshape( + target_prior.shape[0], -1 + ), + 1, + ) + prior_loss = prior_loss.mean() + + # Compute regular loss. + loss = torch.mean( + (weighting.float() * (model_pred.float() - target.float()) ** 2).reshape(target.shape[0], -1), + 1, + ) + loss = loss.mean() + + if args.with_prior_preservation: + # Add the prior loss to the instance loss. + loss = loss + args.prior_loss_weight * prior_loss + + accelerator.backward(loss) + if accelerator.sync_gradients: + params_to_clip = transformer.parameters() + accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm) + + optimizer.step() + lr_scheduler.step() + optimizer.zero_grad() + + # Checks if the accelerator has performed an optimization step behind the scenes + if accelerator.sync_gradients: + progress_bar.update(1) + global_step += 1 + + if accelerator.is_main_process or is_fsdp: + if global_step % args.checkpointing_steps == 0: + # _before_ saving state, check if this save would set us over the `checkpoints_total_limit` + if args.checkpoints_total_limit is not None: + checkpoints = os.listdir(args.output_dir) + checkpoints = [d for d in checkpoints if d.startswith("checkpoint")] + checkpoints = sorted(checkpoints, key=lambda x: int(x.split("-")[1])) + + # before we save the new checkpoint, we need to have at _most_ `checkpoints_total_limit - 1` checkpoints + if len(checkpoints) >= args.checkpoints_total_limit: + num_to_remove = len(checkpoints) - args.checkpoints_total_limit + 1 + removing_checkpoints = checkpoints[0:num_to_remove] + + logger.info( + f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints" + ) + logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}") + + for removing_checkpoint in removing_checkpoints: + removing_checkpoint = os.path.join(args.output_dir, removing_checkpoint) + shutil.rmtree(removing_checkpoint) + + save_path = os.path.join(args.output_dir, f"checkpoint-{global_step}") + accelerator.save_state(save_path) + logger.info(f"Saved state to {save_path}") + + logs = {"loss": loss.detach().item(), "lr": lr_scheduler.get_last_lr()[0]} + progress_bar.set_postfix(**logs) + accelerator.log(logs, step=global_step) + + if global_step >= args.max_train_steps: + break + + if accelerator.is_main_process: + if args.validation_prompt is not None and epoch % args.validation_epochs == 0: + transformer_ = unwrap_model(transformer) + transformer_.eval() + pipeline = make_validation_pipeline(args, transformer_, weight_dtype, accelerator.device) + images = log_validation( + pipeline=pipeline, + args=args, + accelerator=accelerator, + pipeline_args=validation_pipeline_args, + epoch=epoch, + torch_dtype=weight_dtype, + ) + transformer_.train() + free_memory() + + # Save the lora layers + accelerator.wait_for_everyone() + + if is_fsdp: + transformer = unwrap_model(transformer) + state_dict = accelerator.get_state_dict(transformer) + if accelerator.is_main_process: + modules_to_save = {} + if is_fsdp: + if args.bnb_quantization_config_path is None: + if args.upcast_before_saving: + state_dict = { + k: v.to(torch.float32) if isinstance(v, torch.Tensor) else v for k, v in state_dict.items() + } + else: + state_dict = { + k: v.to(weight_dtype) if isinstance(v, torch.Tensor) else v for k, v in state_dict.items() + } + + transformer_lora_layers = get_peft_model_state_dict( + transformer, + state_dict=state_dict, + ) + transformer_lora_layers = { + k: v.detach().cpu().contiguous() if isinstance(v, torch.Tensor) else v + for k, v in transformer_lora_layers.items() + } + + else: + transformer = unwrap_model(transformer) + if args.bnb_quantization_config_path is None: + if args.upcast_before_saving: + transformer.to(torch.float32) + else: + transformer = transformer.to(weight_dtype) + transformer_lora_layers = get_peft_model_state_dict(transformer) + + modules_to_save["transformer"] = transformer + + Flux3LoraLoaderMixin.save_lora_weights( + save_directory=args.output_dir, + transformer_lora_layers=transformer_lora_layers, + **_collate_lora_metadata(modules_to_save), + ) + + images = [] + run_validation = (args.validation_prompt and args.num_validation_images > 0) or (args.final_validation_prompt) + should_run_final_inference = not args.skip_final_inference and run_validation + if should_run_final_inference: + pipeline = make_validation_pipeline(args, transformer, weight_dtype, accelerator.device) + + # run inference + images = [] + if args.validation_prompt and args.num_validation_images > 0: + images = log_validation( + pipeline=pipeline, + args=args, + accelerator=accelerator, + pipeline_args=validation_pipeline_args, + epoch=epoch, + is_final_validation=True, + torch_dtype=weight_dtype, + ) + images = None + del pipeline + free_memory() + + validation_prompt = args.validation_prompt if args.validation_prompt else args.final_validation_prompt + quant_training = None + if args.do_fp8_training: + quant_training = "FP8 TorchAO" + elif args.bnb_quantization_config_path: + quant_training = "BitsandBytes" + save_model_card( + (args.hub_model_id or Path(args.output_dir).name) if not args.push_to_hub else repo_id, + images=images, + base_model=args.pretrained_model_name_or_path, + instance_prompt=args.instance_prompt, + validation_prompt=validation_prompt, + repo_folder=args.output_dir, + quant_training=quant_training, + ) + + if args.push_to_hub: + upload_folder( + repo_id=repo_id, + folder_path=args.output_dir, + commit_message="End of training", + ignore_patterns=["step_*", "epoch_*"], + ) + + accelerator.end_training() + + +if __name__ == "__main__": + args = parse_args() + main(args) diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py b/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py new file mode 100644 index 000000000000..341a99ecde0d --- /dev/null +++ b/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py @@ -0,0 +1,2058 @@ +#!/usr/bin/env python +# coding=utf-8 +# Copyright 2025 The HuggingFace Inc. team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# /// script +# dependencies = [ +# "diffusers @ git+https://github.com/huggingface/diffusers.git", +# "torch>=2.0.0", +# "accelerate>=0.31.0", +# "transformers>=4.41.2", +# "ftfy", +# "tensorboard", +# "Jinja2", +# "peft>=0.11.1", +# "sentencepiece", +# "torchvision", +# "datasets", +# "bitsandbytes", +# "prodigyopt", +# ] +# /// + +import argparse +import copy +import itertools +import json +import logging +import math +import os +import random +import shutil +from contextlib import nullcontext +from pathlib import Path +from typing import Any + +import numpy as np +import torch +import transformers +from accelerate import Accelerator +from accelerate.logging import get_logger +from accelerate.utils import DistributedDataParallelKwargs, ProjectConfiguration, set_seed +from huggingface_hub import create_repo, upload_folder +from peft import LoraConfig, prepare_model_for_kbit_training, set_peft_model_state_dict +from peft.utils import get_peft_model_state_dict +from PIL import Image +from PIL.ImageOps import exif_transpose +from torch.utils.data import Dataset +from torch.utils.data.sampler import BatchSampler +from torchvision import transforms +from torchvision.transforms import functional as TF +from tqdm.auto import tqdm +from transformers import AutoProcessor, Qwen3VLForConditionalGeneration + +import diffusers +from diffusers import ( + AutoencoderKLFlux2, + BitsAndBytesConfig, + FlowMatchEulerDiscreteScheduler, + Flux3Transformer2DModel, +) +from diffusers.loaders import Flux3LoraLoaderMixin +from diffusers.modular_pipelines.flux3 import Flux3AutoBlocks +from diffusers.optimization import get_scheduler +from diffusers.pipelines.flux2.image_processor import Flux2ImageProcessor +from diffusers.training_utils import ( + _collate_lora_metadata, + _to_cpu_contiguous, + cast_training_params, + compute_density_for_timestep_sampling, + compute_loss_weighting_for_sd3, + find_nearest_bucket, + free_memory, + generate_aspect_ratio_buckets, + get_fsdp_kwargs_from_accelerator, + offload_models, + parse_buckets_string, + wrap_with_fsdp, +) +from diffusers.utils import ( + check_min_version, + convert_unet_state_dict_to_peft, + is_wandb_available, + load_image, +) +from diffusers.utils.hub_utils import load_or_create_model_card, populate_model_card +from diffusers.utils.import_utils import is_torch_npu_available +from diffusers.utils.torch_utils import is_compiled_module + + +if getattr(torch, "distributed", None) is not None: + import torch.distributed as dist + +if is_wandb_available(): + import wandb + +# Will error if the minimal version of diffusers is not installed. Remove at your own risks. +check_min_version("0.40.0.dev0") + +logger = get_logger(__name__) + +# FLUX.3 ships no `scheduler` subfolder: the modular pipeline builds its scheduler `from_config` with a fixed +# shift of 6.93 (the `Flux3SetTimestepsStep` default, and the reference sampler's `alpha`). There is no +# resolution-dependent dynamic shifting. Constructed here for the same reason, rather than loaded. +FLOW_SHIFT = 6.93 + +# FLUX.3 image RoPE scale applied to the H/W position-id axes (see the flux3 modular pipeline). +IMAGE_ROPE_SCALE = 0.5 + +# The text window is right-padded to a multiple of this many tokens (the `Flux3TextEncoderStep` default). +TEXT_PAD_MULTIPLE = 80 + + +# --- FLUX.3 latent / text helpers (bit-identical to the modular pipeline blocks) --- +def patchify_latents(latents): + """(B, C, H, W) -> (B, C*4, H//2, W//2).""" + b, c, h, w = latents.shape + latents = latents.view(b, c, h // 2, 2, w // 2, 2).permute(0, 1, 3, 5, 2, 4) + return latents.reshape(b, c * 4, h // 2, w // 2) + + +def pack_latents(latents): + """(B, C, H, W) -> (B, H*W, C).""" + b, c, h, w = latents.shape + return latents.reshape(b, c, h * w).permute(0, 2, 1) + + +def prepare_latent_ids(height, width, device, image_rope_scale=IMAGE_ROPE_SCALE): + """(H*W, 4) position ids (t, h, w, l) for the generated image; H/W axes scaled by the image RoPE scale.""" + ids = torch.cartesian_prod(torch.arange(1), torch.arange(height), torch.arange(width), torch.arange(1)) + ids = ids.to(dtype=torch.float32) + ids[:, 1:3] = ids[:, 1:3] * image_rope_scale + return ids.to(device) + + +def prepare_image_cond_ids(height, width, device, image_rope_scale=IMAGE_ROPE_SCALE): + """(H*W, 4) position ids for the conditioning image: reference sits at the L axis position -1; H/W scaled.""" + ids = torch.cartesian_prod(torch.arange(1), torch.arange(height), torch.arange(width), torch.tensor([-1])) + ids = ids.to(dtype=torch.float32) + ids[:, 1:3] = ids[:, 1:3] * image_rope_scale + return ids.to(device) + + +def prepare_text_ids(seq_len, device): + """(seq_len, 4) position ids for text tokens; text advances along the L axis.""" + ids = torch.cartesian_prod(torch.arange(1), torch.arange(1), torch.arange(1), torch.arange(seq_len)) + return ids.to(dtype=torch.float32).to(device) + + +@torch.no_grad() +def get_qwen3vl_prompt_embeds( + text_encoder, tokenizer, prompt, device, dtype, text_encoder_out_layers, max_sequence_length +): + """Stack the FLUX.3 Qwen3-VL hidden layers -> (B, seq, len(layers) * hidden_dim). + + Mirrors `Flux3TextEncoderStep._get_qwen3_vl_prompt_embeds`: each prompt goes through the chat template and is + right-padded to a multiple of `TEXT_PAD_MULTIPLE` (capped at `max_sequence_length`). Qwen3-VL masks the padding + (so a prompt's real-token embeddings are independent of the window size), but the transformer attends to it, so + the window is part of the recipe -- for a single prompt this reproduces the pipeline window exactly. When a batch + of differing prompts is encoded together, they share the batch's largest window so the embeddings can be stacked; + with `train_batch_size=1` or a shared instance prompt every prompt keeps its own pipeline-exact window. + The position ids of the padding tokens are passed explicitly to match the Qwen3-VL 4.57 training environment. + """ + prompt = [prompt] if isinstance(prompt, str) else prompt + texts = [ + tokenizer.apply_chat_template([{"role": "user", "content": p}], tokenize=False, add_generation_prompt=True) + for p in prompt + ] + real_lengths = [ + tokenizer.tokenizer(t, return_tensors="pt", truncation=True, max_length=max_sequence_length)[ + "input_ids" + ].shape[1] + for t in texts + ] + padded_length = min(math.ceil(max(real_lengths) / TEXT_PAD_MULTIPLE) * TEXT_PAD_MULTIPLE, max_sequence_length) + enc = tokenizer.tokenizer( + texts, + return_tensors="pt", + padding="max_length", + padding_side="right", + truncation=True, + max_length=padded_length, + ) + input_ids, attn = enc["input_ids"].to(device), enc["attention_mask"].to(device) + position_ids = torch.arange(padded_length, device=device)[None].masked_fill(attn == 0, 1) + out = text_encoder( + input_ids=input_ids, + attention_mask=attn, + position_ids=position_ids, + output_hidden_states=True, + use_cache=False, + ) + stacked = torch.stack([out.hidden_states[k] for k in text_encoder_out_layers], dim=1).to( + dtype=dtype, device=device + ) + b, n, s, d = stacked.shape + return stacked.permute(0, 2, 1, 3).reshape(b, s, n * d) + + +def save_model_card( + repo_id: str, + images=None, + base_model: str = None, + instance_prompt=None, + validation_prompt=None, + repo_folder=None, + fp8_training=False, +): + widget_dict = [] + if images is not None: + for i, image in enumerate(images): + image.save(os.path.join(repo_folder, f"image_{i}.png")) + widget_dict.append( + {"text": validation_prompt if validation_prompt else " ", "output": {"url": f"image_{i}.png"}} + ) + + model_description = f""" +# FLUX.3 DreamBooth LoRA - {repo_id} + + + +## Model description + +These are {repo_id} DreamBooth LoRA weights for {base_model}. + +The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3.md). + +FP8 training? {fp8_training} + +## Trigger words + +You should use `{instance_prompt}` to trigger the image generation. + +## Download model + +[Download the *.safetensors LoRA]({repo_id}/tree/main) in the Files & versions tab. + +## Use it with the [🧨 diffusers library](https://github.com/huggingface/diffusers) + +```py +from diffusers import ModularPipeline +import torch +pipeline = ModularPipeline.from_pretrained("{base_model}") +pipeline.load_components(torch_dtype=torch.bfloat16) +pipeline.to('cuda') +pipeline.load_lora_weights('{repo_id}', weight_name='pytorch_lora_weights.safetensors') +image = pipeline(prompt='{validation_prompt if validation_prompt else instance_prompt}', output_type="pil").values["images"][0] +``` + +For more details, including weighting, merging and fusing LoRAs, check the [documentation on loading LoRAs in diffusers](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters) + +## License + +Please adhere to the licensing terms as described in the model repository. +""" + model_card = load_or_create_model_card( + repo_id_or_path=repo_id, + from_training=True, + license="other", + base_model=base_model, + prompt=instance_prompt, + model_description=model_description, + widget=widget_dict, + ) + tags = [ + "text-to-image", + "diffusers-training", + "diffusers", + "lora", + "flux3", + "flux3-diffusers", + "template:sd-lora", + ] + + model_card = populate_model_card(model_card, tags=tags) + model_card.save(os.path.join(repo_folder, "README.md")) + + +def log_validation( + pipeline, + args, + accelerator, + pipeline_args, + epoch, + torch_dtype, + is_final_validation=False, +): + args.num_validation_images = args.num_validation_images if args.num_validation_images else 1 + logger.info( + f"Running validation... \n Generating {args.num_validation_images} images with prompt:" + f" {args.validation_prompt}." + ) + + # run inference + generator = torch.Generator(device=accelerator.device).manual_seed(args.seed) if args.seed is not None else None + autocast_ctx = torch.autocast(accelerator.device.type) if not is_final_validation else nullcontext() + + images = [] + for _ in range(args.num_validation_images): + with autocast_ctx: + output = pipeline(generator=generator, output_type="pil", **pipeline_args) + images.append(output.values["images"][0]) + + for tracker in accelerator.trackers: + phase_name = "test" if is_final_validation else "validation" + if tracker.name == "tensorboard": + np_images = np.stack([np.asarray(img) for img in images]) + tracker.writer.add_images(phase_name, np_images, epoch, dataformats="NHWC") + if tracker.name == "wandb": + tracker.log( + { + phase_name: [ + wandb.Image(image, caption=f"{i}: {args.validation_prompt}") for i, image in enumerate(images) + ] + } + ) + + del pipeline + free_memory() + + return images + + +def make_validation_pipeline(args, transformer, weight_dtype, device): + # Build a modular pipeline for validation that REUSES the in-memory trained transformer, so validation runs + # on exactly the weights being trained and the DiT is never loaded a second time. Only the small auxiliary + # components are (re)loaded (they are freed after latent caching). + vae = AutoencoderKLFlux2.from_pretrained( + args.pretrained_model_name_or_path, subfolder="vae", revision=args.revision, torch_dtype=torch.float32 + ) + text_encoder = Qwen3VLForConditionalGeneration.from_pretrained( + args.pretrained_model_name_or_path, subfolder="text_encoder", revision=args.revision, torch_dtype=weight_dtype + ) + tokenizer = AutoProcessor.from_pretrained( + args.pretrained_model_name_or_path, subfolder="tokenizer", revision=args.revision + ) + scheduler = FlowMatchEulerDiscreteScheduler(shift=FLOW_SHIFT, use_dynamic_shifting=False) + pipeline = Flux3AutoBlocks().init_pipeline() + pipeline.update_components( + transformer=transformer, vae=vae, text_encoder=text_encoder, tokenizer=tokenizer, scheduler=scheduler + ) + pipeline.to(device) + return pipeline + + +def module_filter_fn(mod: torch.nn.Module, fqn: str): + # don't convert the output module (FLUX.3's output projection) + if fqn.endswith("proj_out"): + return False + # don't convert linear modules with weight dimensions not divisible by 16 + if isinstance(mod, torch.nn.Linear): + if mod.in_features % 16 != 0 or mod.out_features % 16 != 0: + return False + return True + + +def parse_args(input_args=None): + parser = argparse.ArgumentParser(description="Simple example of a training script.") + parser.add_argument( + "--pretrained_model_name_or_path", + type=str, + default=None, + required=True, + help="Path to pretrained model or model identifier from huggingface.co/models.", + ) + parser.add_argument( + "--revision", + type=str, + default=None, + required=False, + help="Revision of pretrained model identifier from huggingface.co/models.", + ) + parser.add_argument( + "--bnb_quantization_config_path", + type=str, + default=None, + help="Quantization config in a JSON file that will be used to define the bitsandbytes quant config of the DiT.", + ) + parser.add_argument( + "--do_fp8_training", + action="store_true", + help="if we are doing FP8 training.", + ) + parser.add_argument( + "--variant", + type=str, + default=None, + help="Variant of the model files of the pretrained model identifier from huggingface.co/models, 'e.g.' fp16", + ) + parser.add_argument( + "--dataset_name", + type=str, + default=None, + help=( + "The name of the Dataset (from the HuggingFace hub) containing the training data of instance images (could be your own, possibly private," + " dataset). It can also be a path pointing to a local copy of a dataset in your filesystem," + " or to a folder containing files that 🤗 Datasets can understand." + ), + ) + parser.add_argument( + "--dataset_config_name", + type=str, + default=None, + help="The config of the Dataset, leave as None if there's only one config.", + ) + parser.add_argument( + "--instance_data_dir", + type=str, + default=None, + help=("A folder containing the training data. "), + ) + + parser.add_argument( + "--cache_dir", + type=str, + default=None, + help="The directory where the downloaded models and datasets will be stored.", + ) + + parser.add_argument( + "--image_column", + type=str, + default="image", + help="The column of the dataset containing the target image. By " + "default, the standard Image Dataset maps out 'file_name' " + "to 'image'.", + ) + parser.add_argument( + "--cond_image_column", + type=str, + default=None, + help="Column in the dataset containing the condition image. Must be specified when performing I2I fine-tuning", + ) + parser.add_argument( + "--caption_column", + type=str, + default=None, + help="The column of the dataset containing the instance prompt for each image", + ) + + parser.add_argument("--repeats", type=int, default=1, help="How many times to repeat the training data.") + + parser.add_argument( + "--class_data_dir", + type=str, + default=None, + required=False, + help="A folder containing the training data of class images.", + ) + parser.add_argument( + "--instance_prompt", + type=str, + default=None, + required=False, + help="The prompt with identifier specifying the instance, e.g. 'photo of a TOK dog', 'in the style of TOK'", + ) + parser.add_argument( + "--max_sequence_length", + type=int, + default=512, + help=( + "Cap on the number of Qwen3-VL prompt tokens; longer prompts are truncated. The text window is the " + f"prompt's length rounded up to a multiple of {TEXT_PAD_MULTIPLE}, capped at this value." + ), + ) + parser.add_argument( + "--text_encoder_out_layers", + type=int, + nargs="+", + default=[4, 8, 12, 16, 20, 24, 28, 32], + help="Text encoder hidden layers to compute the final text embeddings.", + ) + parser.add_argument( + "--validation_prompt", + type=str, + default=None, + help="A prompt that is used during validation to verify that the model is learning.", + ) + parser.add_argument( + "--validation_image", + type=str, + default=None, + help="path to an image that is used during validation as the condition image to verify that the model is learning.", + ) + parser.add_argument( + "--skip_final_inference", + default=False, + action="store_true", + help="Whether to skip the final inference step with loaded lora weights upon training completion. This will run intermediate validation inference if `validation_prompt` is provided. Specify to reduce memory.", + ) + parser.add_argument( + "--final_validation_prompt", + type=str, + default=None, + help="A prompt that is used during a final validation to verify that the model is learning. Ignored if `--validation_prompt` is provided.", + ) + parser.add_argument( + "--num_validation_images", + type=int, + default=4, + help="Number of images that should be generated during validation with `validation_prompt`.", + ) + parser.add_argument( + "--validation_epochs", + type=int, + default=50, + help=( + "Run dreambooth validation every X epochs. Dreambooth validation consists of running the prompt" + " `args.validation_prompt` multiple times: `args.num_validation_images`." + ), + ) + parser.add_argument( + "--rank", + type=int, + default=4, + help=("The dimension of the LoRA update matrices."), + ) + parser.add_argument( + "--lora_alpha", + type=int, + default=4, + help="LoRA alpha to be used for additional scaling.", + ) + parser.add_argument("--lora_dropout", type=float, default=0.0, help="Dropout probability for LoRA layers") + + parser.add_argument( + "--output_dir", + type=str, + default="flux-dreambooth-lora", + help="The output directory where the model predictions and checkpoints will be written.", + ) + parser.add_argument("--seed", type=int, default=None, help="A seed for reproducible training.") + parser.add_argument( + "--resolution", + type=int, + default=512, + help=( + "The resolution for input images, all the images in the train/validation dataset will be resized to this" + " resolution" + ), + ) + parser.add_argument( + "--aspect_ratio_buckets", + type=str, + default=None, + help=( + "Aspect ratio buckets to use for training. Define as a string of 'h1,w1;h2,w2;...'. " + "e.g. '1024,1024;768,1360;1360,768;880,1168;1168,880;1248,832;832,1248'. " + "Requires --use_aspect_ratio_buckets. Images are resized to cover and cropped to the nearest " + "listed bucket (smaller images are upscaled). When set, --resolution is ignored." + ), + ) + parser.add_argument( + "--use_aspect_ratio_buckets", + action="store_true", + help=( + "Enable aspect-ratio bucketing. Without --aspect_ratio_buckets, the buckets are computed on the " + "fly from --resolution and capped to each image's own resolution, so smaller images are assigned " + "to a smaller bucket instead of being upscaled. Provide --aspect_ratio_buckets to use an explicit list." + ), + ) + parser.add_argument( + "--center_crop", + default=False, + action="store_true", + help=( + "Whether to center crop the input images to the resolution. If not set, the images will be randomly" + " cropped. The images will be resized to the resolution first before cropping." + ), + ) + parser.add_argument( + "--random_flip", + action="store_true", + help="whether to randomly flip images horizontally", + ) + parser.add_argument( + "--train_batch_size", type=int, default=4, help="Batch size (per device) for the training dataloader." + ) + parser.add_argument( + "--sample_batch_size", type=int, default=4, help="Batch size (per device) for sampling images." + ) + parser.add_argument("--num_train_epochs", type=int, default=1) + parser.add_argument( + "--max_train_steps", + type=int, + default=None, + help="Total number of training steps to perform. If provided, overrides num_train_epochs.", + ) + parser.add_argument( + "--checkpointing_steps", + type=int, + default=500, + help=( + "Save a checkpoint of the training state every X updates. These checkpoints can be used both as final" + " checkpoints in case they are better than the last checkpoint, and are also suitable for resuming" + " training using `--resume_from_checkpoint`." + ), + ) + parser.add_argument( + "--checkpoints_total_limit", + type=int, + default=None, + help=("Max number of checkpoints to store."), + ) + parser.add_argument( + "--resume_from_checkpoint", + type=str, + default=None, + help=( + "Whether training should be resumed from a previous checkpoint. Use a path saved by" + ' `--checkpointing_steps`, or `"latest"` to automatically select the last available checkpoint.' + ), + ) + parser.add_argument( + "--gradient_accumulation_steps", + type=int, + default=1, + help="Number of updates steps to accumulate before performing a backward/update pass.", + ) + parser.add_argument( + "--gradient_checkpointing", + action="store_true", + help="Whether or not to use gradient checkpointing to save memory at the expense of slower backward pass.", + ) + parser.add_argument( + "--learning_rate", + type=float, + default=1e-4, + help="Initial learning rate (after the potential warmup period) to use.", + ) + + parser.add_argument( + "--scale_lr", + action="store_true", + default=False, + help="Scale the learning rate by the number of GPUs, gradient accumulation steps, and batch size.", + ) + parser.add_argument( + "--lr_scheduler", + type=str, + default="constant", + help=( + 'The scheduler type to use. Choose between ["linear", "cosine", "cosine_with_restarts", "polynomial",' + ' "constant", "constant_with_warmup"]' + ), + ) + parser.add_argument( + "--lr_warmup_steps", type=int, default=500, help="Number of steps for the warmup in the lr scheduler." + ) + parser.add_argument( + "--lr_num_cycles", + type=int, + default=1, + help="Number of hard resets of the lr in cosine_with_restarts scheduler.", + ) + parser.add_argument("--lr_power", type=float, default=1.0, help="Power factor of the polynomial scheduler.") + parser.add_argument( + "--dataloader_num_workers", + type=int, + default=0, + help=( + "Number of subprocesses to use for data loading. 0 means that the data will be loaded in the main process." + ), + ) + parser.add_argument( + "--weighting_scheme", + type=str, + default="none", + choices=["sigma_sqrt", "logit_normal", "mode", "cosmap", "none"], + help=('We default to the "none" weighting scheme for uniform sampling and uniform loss'), + ) + parser.add_argument( + "--logit_mean", type=float, default=0.0, help="mean to use when using the `'logit_normal'` weighting scheme." + ) + parser.add_argument( + "--logit_std", type=float, default=1.0, help="std to use when using the `'logit_normal'` weighting scheme." + ) + parser.add_argument( + "--mode_scale", + type=float, + default=1.29, + help="Scale of mode weighting scheme. Only effective when using the `'mode'` as the `weighting_scheme`.", + ) + parser.add_argument( + "--optimizer", + type=str, + default="AdamW", + help=('The optimizer type to use. Choose between ["AdamW", "prodigy"]'), + ) + + parser.add_argument( + "--use_8bit_adam", + action="store_true", + help="Whether or not to use 8-bit Adam from bitsandbytes. Ignored if optimizer is not set to AdamW", + ) + + parser.add_argument( + "--adam_beta1", type=float, default=0.9, help="The beta1 parameter for the Adam and Prodigy optimizers." + ) + parser.add_argument( + "--adam_beta2", type=float, default=0.999, help="The beta2 parameter for the Adam and Prodigy optimizers." + ) + parser.add_argument( + "--prodigy_beta3", + type=float, + default=None, + help="coefficients for computing the Prodigy stepsize using running averages. If set to None, " + "uses the value of square root of beta2. Ignored if optimizer is adamW", + ) + parser.add_argument("--prodigy_decouple", type=bool, default=True, help="Use AdamW style decoupled weight decay") + parser.add_argument("--adam_weight_decay", type=float, default=1e-04, help="Weight decay to use for unet params") + parser.add_argument( + "--adam_weight_decay_text_encoder", type=float, default=1e-03, help="Weight decay to use for text_encoder" + ) + + parser.add_argument( + "--lora_layers", + type=str, + default=None, + help=( + 'The transformer modules to apply LoRA training on. Please specify the layers in a comma separated. E.g. - "to_k,to_q,to_v,to_out.0" will result in lora training of attention layers only' + ), + ) + + parser.add_argument( + "--adam_epsilon", + type=float, + default=1e-08, + help="Epsilon value for the Adam optimizer and Prodigy optimizers.", + ) + + parser.add_argument( + "--prodigy_use_bias_correction", + type=bool, + default=True, + help="Turn on Adam's bias correction. True by default. Ignored if optimizer is adamW", + ) + parser.add_argument( + "--prodigy_safeguard_warmup", + type=bool, + default=True, + help="Remove lr from the denominator of D estimate to avoid issues during warm-up stage. True by default. " + "Ignored if optimizer is adamW", + ) + parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.") + parser.add_argument("--push_to_hub", action="store_true", help="Whether or not to push the model to the Hub.") + parser.add_argument("--hub_token", type=str, default=None, help="The token to use to push to the Model Hub.") + parser.add_argument( + "--hub_model_id", + type=str, + default=None, + help="The name of the repository to keep in sync with the local `output_dir`.", + ) + parser.add_argument( + "--logging_dir", + type=str, + default="logs", + help=( + "[TensorBoard](https://www.tensorflow.org/tensorboard) log directory. Will default to" + " *output_dir/runs/**CURRENT_DATETIME_HOSTNAME***." + ), + ) + parser.add_argument( + "--allow_tf32", + action="store_true", + help=( + "Whether or not to allow TF32 on Ampere GPUs. Can be used to speed up training. For more information, see" + " https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices" + ), + ) + parser.add_argument( + "--cache_latents", + action="store_true", + default=False, + help="Cache the VAE latents", + ) + parser.add_argument( + "--report_to", + type=str, + default="tensorboard", + help=( + 'The integration to report the results and logs to. Supported platforms are `"tensorboard"`' + ' (default), `"wandb"` and `"comet_ml"`. Use `"all"` to report to all integrations.' + ), + ) + parser.add_argument( + "--mixed_precision", + type=str, + default=None, + choices=["no", "fp16", "bf16"], + help=( + "Whether to use mixed precision. Choose between fp16 and bf16 (bfloat16). Bf16 requires PyTorch >=" + " 1.10.and an Nvidia Ampere GPU. Default to the value of accelerate config of the current system or the" + " flag passed with the `accelerate.launch` command. Use this argument to override the accelerate config." + ), + ) + parser.add_argument( + "--upcast_before_saving", + action="store_true", + default=False, + help=( + "Whether to upcast the trained transformer layers to float32 before saving (at the end of training). " + "Defaults to precision dtype used for training to save memory" + ), + ) + parser.add_argument( + "--offload", + action="store_true", + help="Whether to offload the VAE and the text encoder to CPU when they are not used.", + ) + + parser.add_argument("--local_rank", type=int, default=-1, help="For distributed training: local_rank") + parser.add_argument("--enable_npu_flash_attention", action="store_true", help="Enabla Flash Attention for NPU") + parser.add_argument("--fsdp_text_encoder", action="store_true", help="Use FSDP for text encoder") + + if input_args is not None: + args = parser.parse_args(input_args) + else: + args = parser.parse_args() + + if args.cond_image_column is None: + raise ValueError( + "you must provide --cond_image_column for image-to-image training. Otherwise please see Flux2 text-to-image training example." + ) + else: + assert args.image_column is not None + assert args.caption_column is not None + + if args.dataset_name is None and args.instance_data_dir is None: + raise ValueError("Specify either `--dataset_name` or `--instance_data_dir`") + + if args.dataset_name is not None and args.instance_data_dir is not None: + raise ValueError("Specify only one of `--dataset_name` or `--instance_data_dir`") + + env_local_rank = int(os.environ.get("LOCAL_RANK", -1)) + if env_local_rank != -1 and env_local_rank != args.local_rank: + args.local_rank = env_local_rank + + return args + + +class DreamBoothDataset(Dataset): + """ + A dataset to prepare the instance and class images with the prompts for fine-tuning the model. + It pre-processes the images. + """ + + def __init__( + self, + instance_data_root, + instance_prompt, + size=1024, + repeats=1, + center_crop=False, + buckets=None, + use_aspect_ratio_buckets=False, + bucket_divisibility=16, + bucket_base_resolutions=None, + ): + self.size = size + self.resolution = size + self.center_crop = center_crop + + self.instance_prompt = instance_prompt + self.custom_instance_prompts = None + + # Explicit user-provided bucket list (or None). The concrete list of buckets actually used is + # built from the data in `self.buckets` during preprocessing below. + self._explicit_buckets = buckets + self.use_aspect_ratio_buckets = use_aspect_ratio_buckets + self.bucket_divisibility = bucket_divisibility + self.bucket_base_resolutions = bucket_base_resolutions + + # if --dataset_name is provided or a metadata jsonl file is provided in the local --instance_data directory, + # we load the training data using load_dataset + if args.dataset_name is not None: + try: + from datasets import load_dataset + except ImportError: + raise ImportError( + "You are trying to load your data using the datasets library. If you wish to train using custom " + "captions please install the datasets library: `pip install datasets`. If you wish to load a " + "local folder containing images only, specify --instance_data_dir instead." + ) + # Downloading and loading a dataset from the hub. + # See more about loading custom images at + # https://huggingface.co/docs/datasets/v2.0.0/en/dataset_script + dataset = load_dataset( + args.dataset_name, + args.dataset_config_name, + cache_dir=args.cache_dir, + ) + # Preprocessing the datasets. + column_names = dataset["train"].column_names + + # 6. Get the column names for input/target. + if args.cond_image_column is not None and args.cond_image_column not in column_names: + raise ValueError( + f"`--cond_image_column` value '{args.cond_image_column}' not found in dataset columns. Dataset columns are: {', '.join(column_names)}" + ) + if args.image_column is None: + image_column = column_names[0] + logger.info(f"image column defaulting to {image_column}") + else: + image_column = args.image_column + if image_column not in column_names: + raise ValueError( + f"`--image_column` value '{args.image_column}' not found in dataset columns. Dataset columns are: {', '.join(column_names)}" + ) + instance_images = dataset["train"][image_column] + cond_images = None + cond_image_column = args.cond_image_column + if cond_image_column is not None: + cond_images = [dataset["train"][i][cond_image_column] for i in range(len(dataset["train"]))] + assert len(instance_images) == len(cond_images) + + if args.caption_column is None: + logger.info( + "No caption column provided, defaulting to instance_prompt for all images. If your dataset " + "contains captions/prompts for the images, make sure to specify the " + "column as --caption_column" + ) + self.custom_instance_prompts = None + else: + if args.caption_column not in column_names: + raise ValueError( + f"`--caption_column` value '{args.caption_column}' not found in dataset columns. Dataset columns are: {', '.join(column_names)}" + ) + custom_instance_prompts = dataset["train"][args.caption_column] + # create final list of captions according to --repeats + self.custom_instance_prompts = [] + for caption in custom_instance_prompts: + self.custom_instance_prompts.extend(itertools.repeat(caption, repeats)) + else: + self.instance_data_root = Path(instance_data_root) + if not self.instance_data_root.exists(): + raise ValueError("Instance images root doesn't exists.") + + instance_images = [Image.open(path) for path in list(Path(instance_data_root).iterdir())] + self.custom_instance_prompts = None + + self.instance_images = [] + self.cond_images = [] + for i, img in enumerate(instance_images): + self.instance_images.extend(itertools.repeat(img, repeats)) + if args.dataset_name is not None and cond_images is not None: + self.cond_images.extend(itertools.repeat(cond_images[i], repeats)) + + self.pixel_values = [] + self.cond_pixel_values = [] + self.buckets = [] + bucket_to_idx = {} + for i, image in enumerate(self.instance_images): + image = exif_transpose(image) + if not image.mode == "RGB": + image = image.convert("RGB") + dest_image = None + if self.cond_images: # todo: take care of max area for buckets + dest_image = self.cond_images[i] + image_width, image_height = dest_image.size + if image_width * image_height > 1024 * 1024: + dest_image = Flux2ImageProcessor._resize_to_target_area(dest_image, 1024 * 1024) + image_width, image_height = dest_image.size + + multiple_of = 2 ** (4 - 1) # 2 ** (len(vae.config.block_out_channels) - 1), temp! + image_width = (image_width // multiple_of) * multiple_of + image_height = (image_height // multiple_of) * multiple_of + image_processor = Flux2ImageProcessor() + dest_image = image_processor.preprocess( + dest_image, height=image_height, width=image_width, resize_mode="crop" + ) + # Convert back to PIL + dest_image = dest_image.squeeze(0) + if dest_image.min() < 0: + dest_image = (dest_image + 1) / 2 + dest_image = (torch.clamp(dest_image, 0, 1) * 255).byte().cpu() + + if dest_image.shape[0] == 1: + # Gray scale image + dest_image = Image.fromarray(dest_image.squeeze().numpy(), mode="L") + else: + # RGB scale image: (C, H, W) -> (H, W, C) + dest_image = TF.to_pil_image(dest_image) + + dest_image = exif_transpose(dest_image) + if not dest_image.mode == "RGB": + dest_image = dest_image.convert("RGB") + + width, height = image.size + + # Assign the image to a bucket. + target = self._bucket_for_image(height, width) + if target not in bucket_to_idx: + bucket_to_idx[target] = len(self.buckets) + self.buckets.append(target) + bucket_idx = bucket_to_idx[target] + + # based on the bucket assignment, define the transformations + image, dest_image = self.paired_transform( + image, + dest_image=dest_image, + size=target, + center_crop=args.center_crop, + random_flip=args.random_flip, + ) + self.pixel_values.append((image, bucket_idx)) + if dest_image is not None: + self.cond_pixel_values.append((dest_image, bucket_idx)) + + self.num_instance_images = len(self.instance_images) + self._length = self.num_instance_images + + def __len__(self): + return self._length + + def __getitem__(self, index): + example = {} + instance_image, bucket_idx = self.pixel_values[index % self.num_instance_images] + example["instance_images"] = instance_image + example["bucket_idx"] = bucket_idx + if self.cond_pixel_values: + dest_image, _ = self.cond_pixel_values[index % self.num_instance_images] + example["cond_images"] = dest_image + + if self.custom_instance_prompts: + caption = self.custom_instance_prompts[index % self.num_instance_images] + if caption: + example["instance_prompt"] = caption + else: + example["instance_prompt"] = self.instance_prompt + + else: # custom prompts were provided, but length does not match size of image dataset + example["instance_prompt"] = self.instance_prompt + + return example + + def _bucket_for_image(self, height, width): + # An explicit bucket list takes priority: pick the nearest, upscaling smaller images to cover it. + if self._explicit_buckets is not None: + return self._explicit_buckets[find_nearest_bucket(height, width, self._explicit_buckets)] + # On-the-fly bucketing: cap the ladder to the image's own resolution so smaller images are + # assigned to a smaller bucket rather than being upscaled (mirrors ostris' bucketing). + if self.use_aspect_ratio_buckets: + resolution = min(self.resolution, round((height * width) ** 0.5)) + ladder = generate_aspect_ratio_buckets( + resolution, + divisibility=self.bucket_divisibility, + base_resolutions=self.bucket_base_resolutions, + ) + return ladder[find_nearest_bucket(height, width, ladder)] + # No bucketing: a single square bucket reproduces the fixed-size resize + crop. + return (self.resolution, self.resolution) + + def paired_transform(self, image, dest_image=None, size=(224, 224), center_crop=False, random_flip=False): + # Resize preserving aspect ratio so the image covers the bucket, then crop to the bucket size. + # The same geometry is applied to the conditioning image so the pair stays aligned. + target_height, target_width = size + width, height = image.size + scale = max(target_height / height, target_width / width) + new_size = [round(height * scale), round(width * scale)] + image = TF.resize(image, new_size, interpolation=transforms.InterpolationMode.BILINEAR) + if dest_image is not None: + dest_image = TF.resize(dest_image, new_size, interpolation=transforms.InterpolationMode.BILINEAR) + if center_crop: + image = TF.center_crop(image, size) + if dest_image is not None: + dest_image = TF.center_crop(dest_image, size) + else: + i, j, h, w = transforms.RandomCrop.get_params(image, output_size=size) + image = TF.crop(image, i, j, h, w) + if dest_image is not None: + dest_image = TF.crop(dest_image, i, j, h, w) + if random_flip and random.random() < 0.5: + image = TF.hflip(image) + if dest_image is not None: + dest_image = TF.hflip(dest_image) + image = TF.normalize(TF.to_tensor(image), [0.5], [0.5]) + if dest_image is not None: + dest_image = TF.normalize(TF.to_tensor(dest_image), [0.5], [0.5]) + return (image, dest_image) if dest_image is not None else (image, None) + + +def collate_fn(examples): + pixel_values = [example["instance_images"] for example in examples] + prompts = [example["instance_prompt"] for example in examples] + + pixel_values = torch.stack(pixel_values) + pixel_values = pixel_values.to(memory_format=torch.contiguous_format).float() + + batch = {"pixel_values": pixel_values, "prompts": prompts} + if any("cond_images" in example for example in examples): + cond_pixel_values = [example["cond_images"] for example in examples] + cond_pixel_values = torch.stack(cond_pixel_values) + cond_pixel_values = cond_pixel_values.to(memory_format=torch.contiguous_format).float() + batch.update({"cond_pixel_values": cond_pixel_values}) + return batch + + +class BucketBatchSampler(BatchSampler): + def __init__( + self, + dataset: DreamBoothDataset, + batch_size: int, + drop_last: bool = False, + shuffle_batches_each_epoch: bool = True, + ): + if not isinstance(batch_size, int) or batch_size <= 0: + raise ValueError("batch_size should be a positive integer value, but got batch_size={}".format(batch_size)) + if not isinstance(drop_last, bool): + raise ValueError("drop_last should be a boolean value, but got drop_last={}".format(drop_last)) + + self.dataset = dataset + self.batch_size = batch_size + self.drop_last = drop_last + self.shuffle_batches_each_epoch = shuffle_batches_each_epoch + + # Group indices by bucket + self.bucket_indices = [[] for _ in range(len(self.dataset.buckets))] + for idx, (_, bucket_idx) in enumerate(self.dataset.pixel_values): + self.bucket_indices[bucket_idx].append(idx) + + self.sampler_len = 0 + self.batches = [] + + # Pre-generate batches for each bucket + for indices_in_bucket in self.bucket_indices: + # Shuffle indices within the bucket + random.shuffle(indices_in_bucket) + # Create batches + for i in range(0, len(indices_in_bucket), self.batch_size): + batch = indices_in_bucket[i : i + self.batch_size] + if len(batch) < self.batch_size and self.drop_last: + continue # Skip partial batch if drop_last is True + self.batches.append(batch) + self.sampler_len += 1 # Count the number of batches + + if not self.shuffle_batches_each_epoch: + # Shuffle the precomputed batches once to mix buckets while keeping + # the order stable across epochs for step-indexed caches. + random.shuffle(self.batches) + + def __iter__(self): + if self.shuffle_batches_each_epoch: + random.shuffle(self.batches) + for batch in self.batches: + yield batch + + def __len__(self): + return self.sampler_len + + +class PromptDataset(Dataset): + "A simple dataset to prepare the prompts to generate class images on multiple GPUs." + + def __init__(self, prompt, num_samples): + self.prompt = prompt + self.num_samples = num_samples + + def __len__(self): + return self.num_samples + + def __getitem__(self, index): + example = {} + example["prompt"] = self.prompt + example["index"] = index + return example + + +def main(args): + if args.report_to == "wandb" and args.hub_token is not None: + raise ValueError( + "You cannot use both --report_to=wandb and --hub_token due to a security risk of exposing your token." + " Please use `hf auth login` to authenticate with the Hub." + ) + + if torch.backends.mps.is_available() and args.mixed_precision == "bf16": + # due to pytorch#99272, MPS does not yet support bfloat16. + raise ValueError( + "Mixed precision training with bfloat16 is not supported on MPS. Please use fp16 (recommended) or fp32 instead." + ) + if args.do_fp8_training: + from torchao.float8 import Float8LinearConfig, convert_to_float8_training + + logging_dir = Path(args.output_dir, args.logging_dir) + + accelerator_project_config = ProjectConfiguration(project_dir=args.output_dir, logging_dir=logging_dir) + kwargs = DistributedDataParallelKwargs(find_unused_parameters=True) + accelerator = Accelerator( + gradient_accumulation_steps=args.gradient_accumulation_steps, + mixed_precision=args.mixed_precision, + log_with=args.report_to, + project_config=accelerator_project_config, + kwargs_handlers=[kwargs], + ) + + # Disable AMP for MPS. + if torch.backends.mps.is_available(): + accelerator.native_amp = False + + if args.report_to == "wandb": + if not is_wandb_available(): + raise ImportError("Make sure to install wandb if you want to use it for logging during training.") + + # Make one log on every process with the configuration for debugging. + logging.basicConfig( + format="%(asctime)s - %(levelname)s - %(name)s - %(message)s", + datefmt="%m/%d/%Y %H:%M:%S", + level=logging.INFO, + ) + logger.info(accelerator.state, main_process_only=False) + if accelerator.is_local_main_process: + transformers.utils.logging.set_verbosity_warning() + diffusers.utils.logging.set_verbosity_info() + else: + transformers.utils.logging.set_verbosity_error() + diffusers.utils.logging.set_verbosity_error() + + # If passed along, set the training seed now. + if args.seed is not None: + set_seed(args.seed) + + # Handle the repository creation + if accelerator.is_main_process: + if args.output_dir is not None: + os.makedirs(args.output_dir, exist_ok=True) + + if args.push_to_hub: + repo_id = create_repo( + repo_id=args.hub_model_id or Path(args.output_dir).name, + exist_ok=True, + ).repo_id + + # Load the tokenizers + tokenizer = AutoProcessor.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="tokenizer", + revision=args.revision, + ) + + # For mixed precision training we cast all non-trainable weights (vae, text_encoder and transformer) to half-precision + # as these weights are only used for inference, keeping weights in full precision is not required. + weight_dtype = torch.float32 + if accelerator.mixed_precision == "fp16": + weight_dtype = torch.float16 + elif accelerator.mixed_precision == "bf16": + weight_dtype = torch.bfloat16 + + # Load scheduler and models + noise_scheduler = FlowMatchEulerDiscreteScheduler(shift=FLOW_SHIFT, use_dynamic_shifting=False) + noise_scheduler_copy = copy.deepcopy(noise_scheduler) + vae = AutoencoderKLFlux2.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="vae", + revision=args.revision, + variant=args.variant, + ) + latents_bn_mean = vae.bn.running_mean.view(1, -1, 1, 1).to(accelerator.device) + latents_bn_std = torch.sqrt(vae.bn.running_var.view(1, -1, 1, 1) + vae.config.batch_norm_eps).to( + accelerator.device + ) + + quantization_config = None + if args.bnb_quantization_config_path is not None: + with open(args.bnb_quantization_config_path, "r") as f: + config_kwargs = json.load(f) + if "load_in_4bit" in config_kwargs and config_kwargs["load_in_4bit"]: + config_kwargs["bnb_4bit_compute_dtype"] = weight_dtype + quantization_config = BitsAndBytesConfig(**config_kwargs) + + transformer = Flux3Transformer2DModel.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="transformer", + revision=args.revision, + variant=args.variant, + quantization_config=quantization_config, + torch_dtype=weight_dtype, + ) + if args.bnb_quantization_config_path is not None: + transformer = prepare_model_for_kbit_training(transformer, use_gradient_checkpointing=False) + + text_encoder = Qwen3VLForConditionalGeneration.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="text_encoder", + revision=args.revision, + variant=args.variant, + torch_dtype=weight_dtype, + ) + text_encoder.requires_grad_(False) + + # We only train the additional adapter LoRA layers + transformer.requires_grad_(False) + vae.requires_grad_(False) + + if args.enable_npu_flash_attention: + if is_torch_npu_available(): + logger.info("npu flash attention enabled.") + transformer.set_attention_backend("_native_npu") + else: + raise ValueError("npu flash attention requires torch_npu extensions and is supported only on npu device ") + + if torch.backends.mps.is_available() and weight_dtype == torch.bfloat16: + # due to pytorch#99272, MPS does not yet support bfloat16. + raise ValueError( + "Mixed precision training with bfloat16 is not supported on MPS. Please use fp16 (recommended) or fp32 instead." + ) + + to_kwargs = {"dtype": weight_dtype, "device": accelerator.device} if not args.offload else {"dtype": weight_dtype} + # flux vae is stable in bf16 so load it in weight_dtype to reduce memory + vae.to(**to_kwargs) + # we never offload the transformer to CPU, so we can just use the accelerator device + transformer_to_kwargs = ( + {"device": accelerator.device} + if args.bnb_quantization_config_path is not None + else {"device": accelerator.device, "dtype": weight_dtype} + ) + + is_fsdp = getattr(accelerator.state, "fsdp_plugin", None) is not None + if not is_fsdp: + transformer.to(**transformer_to_kwargs) + + if args.do_fp8_training: + convert_to_float8_training( + transformer, module_filter_fn=module_filter_fn, config=Float8LinearConfig(pad_inner_dim=True) + ) + + text_encoder.to(**to_kwargs) + + if args.gradient_checkpointing: + transformer.enable_gradient_checkpointing() + + if args.lora_layers is not None: + target_modules = [layer.strip() for layer in args.lora_layers.split(",")] + else: + # Attention + feed-forward projections across every block (the generated-image, reference and text mode + # blocks, and the joint single blocks). These are exactly the leaf modules the non-diffusers (ostris) + # FLUX.3 LoRA converter maps, so adapters trained here stay interchangeable with that layout. The reference + # ("image_cond") mode blocks are adapted too, so the LoRA can adjust how the condition image is consumed. + # Pass `--lora_layers` to target a different set. + target_modules = [ + "to_q", + "to_k", + "to_v", + "to_out.0", + "ff.linear_in", + "ff.linear_out", + ] + + # now we will add new LoRA weights the transformer layers + transformer_lora_config = LoraConfig( + r=args.rank, + lora_alpha=args.lora_alpha, + lora_dropout=args.lora_dropout, + init_lora_weights="gaussian", + target_modules=target_modules, + ) + transformer.add_adapter(transformer_lora_config) + + def unwrap_model(model): + model = accelerator.unwrap_model(model) + model = model._orig_mod if is_compiled_module(model) else model + return model + + # create custom saving & loading hooks so that `accelerator.save_state(...)` serializes in a nice format + def save_model_hook(models, weights, output_dir): + transformer_cls = type(unwrap_model(transformer)) + + # 1) Validate and pick the transformer model + modules_to_save: dict[str, Any] = {} + transformer_model = None + + for model in models: + if isinstance(unwrap_model(model), transformer_cls): + transformer_model = model + modules_to_save["transformer"] = model + else: + raise ValueError(f"unexpected save model: {model.__class__}") + + if transformer_model is None: + raise ValueError("No transformer model found in 'models'") + + # 2) Optionally gather FSDP state dict once + state_dict = accelerator.get_state_dict(model) if is_fsdp else None + + # 3) Only main process materializes the LoRA state dict + transformer_lora_layers_to_save = None + if accelerator.is_main_process: + peft_kwargs = {} + if is_fsdp: + peft_kwargs["state_dict"] = state_dict + + transformer_lora_layers_to_save = get_peft_model_state_dict( + unwrap_model(transformer_model) if is_fsdp else transformer_model, + **peft_kwargs, + ) + + if is_fsdp: + transformer_lora_layers_to_save = _to_cpu_contiguous(transformer_lora_layers_to_save) + + # make sure to pop weight so that corresponding model is not saved again + if weights: + weights.pop() + + Flux3LoraLoaderMixin.save_lora_weights( + output_dir, + transformer_lora_layers=transformer_lora_layers_to_save, + **_collate_lora_metadata(modules_to_save), + ) + + def load_model_hook(models, input_dir): + transformer_ = None + + if not is_fsdp: + while len(models) > 0: + model = models.pop() + + if isinstance(unwrap_model(model), type(unwrap_model(transformer))): + transformer_ = unwrap_model(model) + else: + raise ValueError(f"unexpected save model: {model.__class__}") + else: + transformer_ = Flux3Transformer2DModel.from_pretrained( + args.pretrained_model_name_or_path, + subfolder="transformer", + ) + transformer_.add_adapter(transformer_lora_config) + + lora_state_dict = Flux3LoraLoaderMixin.lora_state_dict(input_dir) + + transformer_state_dict = { + f"{k.replace('transformer.', '')}": v for k, v in lora_state_dict.items() if k.startswith("transformer.") + } + transformer_state_dict = convert_unet_state_dict_to_peft(transformer_state_dict) + incompatible_keys = set_peft_model_state_dict(transformer_, transformer_state_dict, adapter_name="default") + if incompatible_keys is not None: + # check only for unexpected keys + unexpected_keys = getattr(incompatible_keys, "unexpected_keys", None) + if unexpected_keys: + logger.warning( + f"Loading adapter weights from state_dict led to unexpected keys not found in the model: " + f" {unexpected_keys}. " + ) + + # Make sure the trainable params are in float32. This is again needed since the base models + # are in `weight_dtype`. More details: + # https://github.com/huggingface/diffusers/pull/6514#discussion_r1449796804 + if args.mixed_precision == "fp16": + models = [transformer_] + # only upcast trainable parameters (LoRA) into fp32 + cast_training_params(models) + + accelerator.register_save_state_pre_hook(save_model_hook) + accelerator.register_load_state_pre_hook(load_model_hook) + + # Enable TF32 for faster training on Ampere GPUs, + # cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices + if args.allow_tf32 and torch.cuda.is_available(): + torch.backends.cuda.matmul.allow_tf32 = True + + if args.scale_lr: + args.learning_rate = ( + args.learning_rate * args.gradient_accumulation_steps * args.train_batch_size * accelerator.num_processes + ) + + # Make sure the trainable params are in float32. + if args.mixed_precision == "fp16": + models = [transformer] + # only upcast trainable parameters (LoRA) into fp32 + cast_training_params(models, dtype=torch.float32) + + transformer_lora_parameters = list(filter(lambda p: p.requires_grad, transformer.parameters())) + + # Optimization parameters + transformer_parameters_with_lr = {"params": transformer_lora_parameters, "lr": args.learning_rate} + params_to_optimize = [transformer_parameters_with_lr] + + # Optimizer creation + if not (args.optimizer.lower() == "prodigy" or args.optimizer.lower() == "adamw"): + logger.warning( + f"Unsupported choice of optimizer: {args.optimizer}.Supported optimizers include [adamW, prodigy]." + "Defaulting to adamW" + ) + args.optimizer = "adamw" + + if args.use_8bit_adam and not args.optimizer.lower() == "adamw": + logger.warning( + f"use_8bit_adam is ignored when optimizer is not set to 'AdamW'. Optimizer was " + f"set to {args.optimizer.lower()}" + ) + + if args.optimizer.lower() == "adamw": + if args.use_8bit_adam: + try: + import bitsandbytes as bnb + except ImportError: + raise ImportError( + "To use 8-bit Adam, please install the bitsandbytes library: `pip install bitsandbytes`." + ) + + optimizer_class = bnb.optim.AdamW8bit + else: + optimizer_class = torch.optim.AdamW + + optimizer = optimizer_class( + params_to_optimize, + betas=(args.adam_beta1, args.adam_beta2), + weight_decay=args.adam_weight_decay, + eps=args.adam_epsilon, + ) + + if args.optimizer.lower() == "prodigy": + try: + import prodigyopt + except ImportError: + raise ImportError("To use Prodigy, please install the prodigyopt library: `pip install prodigyopt`") + + optimizer_class = prodigyopt.Prodigy + + if args.learning_rate <= 0.1: + logger.warning( + "Learning rate is too low. When using prodigy, it's generally better to set learning rate around 1.0" + ) + + optimizer = optimizer_class( + params_to_optimize, + betas=(args.adam_beta1, args.adam_beta2), + beta3=args.prodigy_beta3, + weight_decay=args.adam_weight_decay, + eps=args.adam_epsilon, + decouple=args.prodigy_decouple, + use_bias_correction=args.prodigy_use_bias_correction, + safeguard_warmup=args.prodigy_safeguard_warmup, + ) + + # Resolve the bucketing mode. Bucketing must be enabled explicitly with --use_aspect_ratio_buckets; + # a bucket list without that flag is an error. With the flag, an explicit --aspect_ratio_buckets list + # drives assignment, otherwise buckets are computed on the fly inside the dataset. Without the flag a + # single square bucket reproduces the fixed-size resize + crop. + if args.aspect_ratio_buckets is not None and not args.use_aspect_ratio_buckets: + raise ValueError("--aspect_ratio_buckets requires --use_aspect_ratio_buckets to be set.") + if args.aspect_ratio_buckets is not None: + buckets = parse_buckets_string(args.aspect_ratio_buckets) + use_aspect_ratio_buckets = False + logger.info(f"Using explicit aspect ratio buckets: {buckets}") + elif args.use_aspect_ratio_buckets: + buckets = None + use_aspect_ratio_buckets = True + logger.info( + "No --aspect_ratio_buckets provided; auto-computing aspect ratio buckets on the fly from --resolution." + ) + else: + buckets = [(args.resolution, args.resolution)] + use_aspect_ratio_buckets = False + + # Dataset and DataLoaders creation: + train_dataset = DreamBoothDataset( + instance_data_root=args.instance_data_dir, + instance_prompt=args.instance_prompt, + size=args.resolution, + repeats=args.repeats, + center_crop=args.center_crop, + buckets=buckets, + use_aspect_ratio_buckets=use_aspect_ratio_buckets, + ) + has_step_indexed_caches = precompute_latents = args.cache_latents or train_dataset.custom_instance_prompts + batch_sampler = BucketBatchSampler( + train_dataset, + batch_size=args.train_batch_size, + drop_last=True, + shuffle_batches_each_epoch=not has_step_indexed_caches, + ) + train_dataloader = torch.utils.data.DataLoader( + train_dataset, + batch_sampler=batch_sampler, + collate_fn=lambda examples: collate_fn(examples), + num_workers=args.dataloader_num_workers, + ) + + def compute_text_embeddings(prompt, text_encoder, tokenizer): + # FLUX.3 has no classic text-encoding pipeline; encode directly with the frozen Qwen3-VL encoder. + with torch.no_grad(): + prompt_embeds = get_qwen3vl_prompt_embeds( + text_encoder, + tokenizer, + prompt, + accelerator.device, + weight_dtype, + text_encoder_out_layers=args.text_encoder_out_layers, + max_sequence_length=args.max_sequence_length, + ) + # FLUX.3 text ids are positional only (identical for every sample of a given length). We carry them + # batched here so the repeat plumbing below is unchanged; the transformer consumes a single (seq, 4) + # slice at the forward call. + text_ids = prepare_text_ids(prompt_embeds.shape[1], accelerator.device) + text_ids = text_ids.unsqueeze(0).expand(prompt_embeds.shape[0], -1, -1) + return prompt_embeds, text_ids + + # If no type of tuning is done on the text_encoder and custom instance prompts are NOT + # provided (i.e. the --instance_prompt is used for all images), we encode the instance prompt once to avoid + # the redundant encoding. + if not train_dataset.custom_instance_prompts: + with offload_models(text_encoder, device=accelerator.device, offload=args.offload): + instance_prompt_hidden_states, instance_text_ids = compute_text_embeddings( + args.instance_prompt, text_encoder, tokenizer + ) + + # FLUX.3 validation runs through the modular pipeline (which encodes prompts itself), so we only assemble + # the generation kwargs here. The reference/condition image is routed to the pipeline's `image` input. + validation_pipeline_args = {} + if args.validation_prompt is not None: + validation_image = load_image(args.validation_image).convert("RGB") + validation_pipeline_args = { + "prompt": args.validation_prompt, + "image": validation_image, + "negative_prompt": "", + "height": args.resolution, + "width": args.resolution, + "num_inference_steps": 28, + # Encode the validation prompt exactly as training did (same window cap and hidden layers), so the + # modular pipeline's text embeddings match the ones the adapter was trained against. + "max_sequence_length": args.max_sequence_length, + "text_encoder_out_layers": tuple(args.text_encoder_out_layers), + } + + # Init FSDP for text encoder + if args.fsdp_text_encoder: + fsdp_kwargs = get_fsdp_kwargs_from_accelerator(accelerator) + text_encoder = wrap_with_fsdp( + model=text_encoder, + device=accelerator.device, + offload=args.offload, + limit_all_gathers=True, + use_orig_params=True, + fsdp_kwargs=fsdp_kwargs, + ) + dist.barrier() + + # If custom instance prompts are NOT provided (i.e. the instance prompt is used for all images), + # pack the statically computed variables appropriately here. This is so that we don't + # have to pass them to the dataloader. + if not train_dataset.custom_instance_prompts: + prompt_embeds = instance_prompt_hidden_states + text_ids = instance_text_ids + + # if cache_latents is set to True, we encode images to latents and store them. + # Similar to pre-encoding in the case of a single instance prompt, if custom prompts are provided + # we encode them in advance as well. + if precompute_latents: + prompt_embeds_cache = [] + text_ids_cache = [] + latents_cache = [] + cond_latents_cache = [] + for batch in tqdm(train_dataloader, desc="Caching latents"): + with torch.no_grad(): + if args.cache_latents: + with offload_models(vae, device=accelerator.device, offload=args.offload): + batch["pixel_values"] = batch["pixel_values"].to( + accelerator.device, non_blocking=True, dtype=vae.dtype + ) + latents_cache.append(vae.encode(batch["pixel_values"]).latent_dist) + batch["cond_pixel_values"] = batch["cond_pixel_values"].to( + accelerator.device, non_blocking=True, dtype=vae.dtype + ) + cond_latents_cache.append(vae.encode(batch["cond_pixel_values"]).latent_dist) + if train_dataset.custom_instance_prompts: + if args.fsdp_text_encoder: + prompt_embeds, text_ids = compute_text_embeddings(batch["prompts"], text_encoder, tokenizer) + else: + with offload_models(text_encoder, device=accelerator.device, offload=args.offload): + prompt_embeds, text_ids = compute_text_embeddings( + batch["prompts"], text_encoder, tokenizer + ) + prompt_embeds_cache.append(prompt_embeds) + text_ids_cache.append(text_ids) + + # move back to cpu before deleting to ensure memory is freed see: https://github.com/huggingface/diffusers/issues/11376#issue-3008144624 + if args.cache_latents: + vae = vae.to("cpu") + del vae + + # move back to cpu before deleting to ensure memory is freed see: https://github.com/huggingface/diffusers/issues/11376#issue-3008144624 + text_encoder = text_encoder.to("cpu") + del text_encoder, tokenizer + free_memory() + + # Scheduler and math around the number of training steps. + # Check the PR https://github.com/huggingface/diffusers/pull/8312 for detailed explanation. + num_warmup_steps_for_scheduler = args.lr_warmup_steps * accelerator.num_processes + if args.max_train_steps is None: + len_train_dataloader_after_sharding = math.ceil(len(train_dataloader) / accelerator.num_processes) + num_update_steps_per_epoch = math.ceil(len_train_dataloader_after_sharding / args.gradient_accumulation_steps) + num_training_steps_for_scheduler = ( + args.num_train_epochs * accelerator.num_processes * num_update_steps_per_epoch + ) + else: + num_training_steps_for_scheduler = args.max_train_steps * accelerator.num_processes + + lr_scheduler = get_scheduler( + args.lr_scheduler, + optimizer=optimizer, + num_warmup_steps=num_warmup_steps_for_scheduler, + num_training_steps=num_training_steps_for_scheduler, + num_cycles=args.lr_num_cycles, + power=args.lr_power, + ) + + # Prepare everything with our `accelerator`. + transformer, optimizer, train_dataloader, lr_scheduler = accelerator.prepare( + transformer, optimizer, train_dataloader, lr_scheduler + ) + + # We need to recalculate our total training steps as the size of the training dataloader may have changed. + num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps) + if args.max_train_steps is None: + args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch + if num_training_steps_for_scheduler != args.max_train_steps: + logger.warning( + f"The length of the 'train_dataloader' after 'accelerator.prepare' ({len(train_dataloader)}) does not match " + f"the expected length ({len_train_dataloader_after_sharding}) when the learning rate scheduler was created. " + f"This inconsistency may result in the learning rate scheduler not functioning properly." + ) + # Afterwards we recalculate our number of training epochs + args.num_train_epochs = math.ceil(args.max_train_steps / num_update_steps_per_epoch) + + # We need to initialize the trackers we use, and also store our configuration. + # The trackers initializes automatically on the main process. + if accelerator.is_main_process: + tracker_name = "dreambooth-flux3-image2img-lora" + args_cp = vars(args).copy() + args_cp["text_encoder_out_layers"] = str(args_cp["text_encoder_out_layers"]) + accelerator.init_trackers(tracker_name, config=args_cp) + + # Train! + total_batch_size = args.train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps + + logger.info("***** Running training *****") + logger.info(f" Num examples = {len(train_dataset)}") + logger.info(f" Num batches each epoch = {len(train_dataloader)}") + logger.info(f" Num Epochs = {args.num_train_epochs}") + logger.info(f" Instantaneous batch size per device = {args.train_batch_size}") + logger.info(f" Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}") + logger.info(f" Gradient Accumulation steps = {args.gradient_accumulation_steps}") + logger.info(f" Total optimization steps = {args.max_train_steps}") + global_step = 0 + first_epoch = 0 + + # Potentially load in the weights and states from a previous save + if args.resume_from_checkpoint: + if args.resume_from_checkpoint != "latest": + path = os.path.basename(args.resume_from_checkpoint) + else: + # Get the mos recent checkpoint + dirs = os.listdir(args.output_dir) + dirs = [d for d in dirs if d.startswith("checkpoint")] + dirs = sorted(dirs, key=lambda x: int(x.split("-")[1])) + path = dirs[-1] if len(dirs) > 0 else None + + if path is None: + accelerator.print( + f"Checkpoint '{args.resume_from_checkpoint}' does not exist. Starting a new training run." + ) + args.resume_from_checkpoint = None + initial_global_step = 0 + else: + accelerator.print(f"Resuming from checkpoint {path}") + accelerator.load_state(os.path.join(args.output_dir, path)) + global_step = int(path.split("-")[1]) + + initial_global_step = global_step + first_epoch = global_step // num_update_steps_per_epoch + + else: + initial_global_step = 0 + + progress_bar = tqdm( + range(0, args.max_train_steps), + initial=initial_global_step, + desc="Steps", + # Only show the progress bar once on each machine. + disable=not accelerator.is_local_main_process, + ) + + def get_sigmas(timesteps, n_dim=4, dtype=torch.float32): + sigmas = noise_scheduler_copy.sigmas.to(device=accelerator.device, dtype=dtype) + schedule_timesteps = noise_scheduler_copy.timesteps.to(accelerator.device) + timesteps = timesteps.to(accelerator.device) + step_indices = [(schedule_timesteps == t).nonzero().item() for t in timesteps] + + sigma = sigmas[step_indices].flatten() + while len(sigma.shape) < n_dim: + sigma = sigma.unsqueeze(-1) + return sigma + + for epoch in range(first_epoch, args.num_train_epochs): + transformer.train() + + for step, batch in enumerate(train_dataloader): + models_to_accumulate = [transformer] + prompts = batch["prompts"] + + with accelerator.accumulate(models_to_accumulate): + if train_dataset.custom_instance_prompts: + prompt_embeds = prompt_embeds_cache[step] + text_ids = text_ids_cache[step] + else: + num_repeat_elements = len(prompts) + prompt_embeds = prompt_embeds.repeat(num_repeat_elements, 1, 1) + text_ids = text_ids.repeat(num_repeat_elements, 1, 1) + + # Convert images to latent space + if args.cache_latents: + model_input = latents_cache[step].mode() + cond_model_input = cond_latents_cache[step].mode() + else: + with offload_models(vae, device=accelerator.device, offload=args.offload): + pixel_values = batch["pixel_values"].to(device=accelerator.device, dtype=vae.dtype) + cond_pixel_values = batch["cond_pixel_values"].to(device=accelerator.device, dtype=vae.dtype) + model_input = vae.encode(pixel_values).latent_dist.mode() + cond_model_input = vae.encode(cond_pixel_values).latent_dist.mode() + + model_input = patchify_latents(model_input) + model_input = (model_input - latents_bn_mean) / latents_bn_std + + cond_model_input = patchify_latents(cond_model_input) + cond_model_input = (cond_model_input - latents_bn_mean) / latents_bn_std + + _, _, lh, lw = model_input.shape + _, _, ch, cw = cond_model_input.shape + + # Sample noise that we'll add to the latents + noise = torch.randn_like(model_input) + bsz = model_input.shape[0] + + # Sample a random timestep for each image + # for weighting schemes where we sample timesteps non-uniformly + u = compute_density_for_timestep_sampling( + weighting_scheme=args.weighting_scheme, + batch_size=bsz, + logit_mean=args.logit_mean, + logit_std=args.logit_std, + mode_scale=args.mode_scale, + ) + indices = (u * noise_scheduler_copy.config.num_train_timesteps).long() + timesteps = noise_scheduler_copy.timesteps[indices].to(device=model_input.device) + + # Add noise according to flow matching. + # zt = (1 - texp) * x + texp * z1 + sigmas = get_sigmas(timesteps, n_dim=model_input.ndim, dtype=model_input.dtype) + noisy_model_input = (1.0 - sigmas) * model_input + sigmas * noise + + # [B, C, H, W] -> [B, H*W, C]: the noisy generated-image tokens. + packed_noisy_model_input = pack_latents(noisy_model_input) + image_ids = prepare_latent_ids(lh, lw, accelerator.device) # (image_seq_len, 4) + # FLUX.3 is not guidance-distilled: no guidance argument. The flow timestep is a per-sample scalar in + # [0, 1] (the transformer scales it by 1000 and broadcasts it over the image tokens internally). + timestep = sigmas.reshape(bsz) + + # The clean reference image is fed as a SEPARATE conditioning stream: its own packed tokens and + # position ids at the l=-1 axis. The transformer holds the reference (and the text) at timestep 0 + # internally, so no cond timestep is passed. It does not concatenate the reference onto the image + # stream and returns only the generated image tokens, so there is nothing to slice off afterwards. + image_cond_hidden_states = pack_latents(cond_model_input) + image_cond_ids = prepare_image_cond_ids(ch, cw, accelerator.device) # (cond_seq_len, 4) + + # Predict the noise residual. The transformer returns only the generated image tokens, so there is + # nothing to slice/unpack afterwards. + with accelerator.autocast(): + model_pred = transformer( + hidden_states=packed_noisy_model_input, # (B, image_seq_len, C) + img_ids=image_ids, # (image_seq_len, 4) + timestep=timestep, # (B,) + image_cond_hidden_states=image_cond_hidden_states, # (B, cond_seq_len, C) + image_cond_ids=image_cond_ids, # (cond_seq_len, 4) + encoder_hidden_states=prompt_embeds, + txt_ids=text_ids[0], # (text_seq_len, 4), shared across the batch + return_dict=False, + )[0] + + # these weighting schemes use a uniform timestep sampling + # and instead post-weight the loss + weighting = compute_loss_weighting_for_sd3( + weighting_scheme=args.weighting_scheme, sigmas=sigmas.reshape(bsz, 1, 1) + ) + + # flow matching loss (velocity target) on the packed image tokens + target = pack_latents(noise) - pack_latents(model_input) + + # Compute regular loss. + loss = torch.mean( + (weighting.float() * (model_pred.float() - target.float()) ** 2).reshape(target.shape[0], -1), + 1, + ) + loss = loss.mean() + + accelerator.backward(loss) + if accelerator.sync_gradients: + params_to_clip = transformer.parameters() + accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm) + + optimizer.step() + lr_scheduler.step() + optimizer.zero_grad() + + # Checks if the accelerator has performed an optimization step behind the scenes + if accelerator.sync_gradients: + progress_bar.update(1) + global_step += 1 + + if accelerator.is_main_process or is_fsdp: + if global_step % args.checkpointing_steps == 0: + # _before_ saving state, check if this save would set us over the `checkpoints_total_limit` + if args.checkpoints_total_limit is not None: + checkpoints = os.listdir(args.output_dir) + checkpoints = [d for d in checkpoints if d.startswith("checkpoint")] + checkpoints = sorted(checkpoints, key=lambda x: int(x.split("-")[1])) + + # before we save the new checkpoint, we need to have at _most_ `checkpoints_total_limit - 1` checkpoints + if len(checkpoints) >= args.checkpoints_total_limit: + num_to_remove = len(checkpoints) - args.checkpoints_total_limit + 1 + removing_checkpoints = checkpoints[0:num_to_remove] + + logger.info( + f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints" + ) + logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}") + + for removing_checkpoint in removing_checkpoints: + removing_checkpoint = os.path.join(args.output_dir, removing_checkpoint) + shutil.rmtree(removing_checkpoint) + + save_path = os.path.join(args.output_dir, f"checkpoint-{global_step}") + accelerator.save_state(save_path) + logger.info(f"Saved state to {save_path}") + + logs = {"loss": loss.detach().item(), "lr": lr_scheduler.get_last_lr()[0]} + progress_bar.set_postfix(**logs) + accelerator.log(logs, step=global_step) + + if global_step >= args.max_train_steps: + break + + if accelerator.is_main_process: + if args.validation_prompt is not None and epoch % args.validation_epochs == 0: + transformer_ = unwrap_model(transformer) + transformer_.eval() + pipeline = make_validation_pipeline(args, transformer_, weight_dtype, accelerator.device) + images = log_validation( + pipeline=pipeline, + args=args, + accelerator=accelerator, + pipeline_args=validation_pipeline_args, + epoch=epoch, + torch_dtype=weight_dtype, + ) + transformer_.train() + free_memory() + + # Save the lora layers + accelerator.wait_for_everyone() + + if is_fsdp: + transformer = unwrap_model(transformer) + state_dict = accelerator.get_state_dict(transformer) + if accelerator.is_main_process: + modules_to_save = {} + if is_fsdp: + if args.bnb_quantization_config_path is None: + if args.upcast_before_saving: + state_dict = { + k: v.to(torch.float32) if isinstance(v, torch.Tensor) else v for k, v in state_dict.items() + } + else: + state_dict = { + k: v.to(weight_dtype) if isinstance(v, torch.Tensor) else v for k, v in state_dict.items() + } + + transformer_lora_layers = get_peft_model_state_dict( + transformer, + state_dict=state_dict, + ) + transformer_lora_layers = { + k: v.detach().cpu().contiguous() if isinstance(v, torch.Tensor) else v + for k, v in transformer_lora_layers.items() + } + + else: + transformer = unwrap_model(transformer) + if args.bnb_quantization_config_path is None: + if args.upcast_before_saving: + transformer.to(torch.float32) + else: + transformer = transformer.to(weight_dtype) + transformer_lora_layers = get_peft_model_state_dict(transformer) + + modules_to_save["transformer"] = transformer + + Flux3LoraLoaderMixin.save_lora_weights( + save_directory=args.output_dir, + transformer_lora_layers=transformer_lora_layers, + **_collate_lora_metadata(modules_to_save), + ) + + images = [] + run_validation = (args.validation_prompt and args.num_validation_images > 0) or (args.final_validation_prompt) + should_run_final_inference = not args.skip_final_inference and run_validation + if should_run_final_inference: + pipeline = make_validation_pipeline(args, transformer, weight_dtype, accelerator.device) + + # run inference + images = [] + if args.validation_prompt and args.num_validation_images > 0: + images = log_validation( + pipeline=pipeline, + args=args, + accelerator=accelerator, + pipeline_args=validation_pipeline_args, + epoch=epoch, + is_final_validation=True, + torch_dtype=weight_dtype, + ) + del pipeline + free_memory() + + validation_prompt = args.validation_prompt if args.validation_prompt else args.final_validation_prompt + save_model_card( + (args.hub_model_id or Path(args.output_dir).name) if not args.push_to_hub else repo_id, + images=images, + base_model=args.pretrained_model_name_or_path, + instance_prompt=args.instance_prompt, + validation_prompt=validation_prompt, + repo_folder=args.output_dir, + fp8_training=args.do_fp8_training, + ) + + if args.push_to_hub: + upload_folder( + repo_id=repo_id, + folder_path=args.output_dir, + commit_message="End of training", + ignore_patterns=["step_*", "epoch_*"], + ) + + accelerator.end_training() + + +if __name__ == "__main__": + args = parse_args() + main(args) From 64777c39d0fece3fa988a8ca4870cd9810cfd1c3 Mon Sep 17 00:00:00 2001 From: linoytsaban Date: Mon, 12 Oct 2026 17:06:32 +0000 Subject: [PATCH 20/34] tests: clarify FLUX3_TINY_REPO comment The tiny repo is uploaded to hf-internal-testing and the env var is a local-dev escape hatch, not a pre-PR TODO. --- examples/dreambooth/test_dreambooth_lora_flux3_small.py | 4 ++-- .../dreambooth/test_dreambooth_lora_flux3_small_img2img.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/examples/dreambooth/test_dreambooth_lora_flux3_small.py b/examples/dreambooth/test_dreambooth_lora_flux3_small.py index 87a30ca0a492..3c3fcaf2003a 100644 --- a/examples/dreambooth/test_dreambooth_lora_flux3_small.py +++ b/examples/dreambooth/test_dreambooth_lora_flux3_small.py @@ -31,8 +31,8 @@ stream_handler = logging.StreamHandler(sys.stdout) logger.addHandler(stream_handler) -# The tiny FLUX.3 image repo is built by `integration-notes/make_tiny_flux3_repo.py`. Upload it as -# hf-internal-testing/tiny-flux3-image-modular and drop the env override before opening the PR. +# Tiny randomly-initialized FLUX.3 image modular repo used for fast CI. The env var is an escape hatch for +# running against a local build (see `integration-notes/make_tiny_flux3_repo.py`). FLUX3_TINY_REPO = os.environ.get("FLUX3_TINY_REPO", "hf-internal-testing/tiny-flux3-image-modular") diff --git a/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py b/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py index 907743681dd7..03fa913d6bee 100644 --- a/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py +++ b/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py @@ -31,8 +31,8 @@ stream_handler = logging.StreamHandler(sys.stdout) logger.addHandler(stream_handler) -# The tiny FLUX.3 image repo is built by `integration-notes/make_tiny_flux3_repo.py`. Upload it as -# hf-internal-testing/tiny-flux3-image-modular and drop the env override before opening the PR. +# Tiny randomly-initialized FLUX.3 image modular repo used for fast CI. The env var is an escape hatch for +# running against a local build (see `integration-notes/make_tiny_flux3_repo.py`). FLUX3_TINY_REPO = os.environ.get("FLUX3_TINY_REPO", "hf-internal-testing/tiny-flux3-image-modular") From a4e826173b87deffacba022debb75e83700ced6d Mon Sep 17 00:00:00 2001 From: linoytsaban Date: Mon, 12 Oct 2026 17:35:55 +0000 Subject: [PATCH 21/34] examples/flux3: drop unused args and dead class Remove the text-encoder weight-decay arg from both trainers (neither trains the text encoder) and, from the img2img trainer, the prior-preservation leftovers (--class_data_dir, --sample_batch_size, PromptDataset) that it never uses. --- .../train_dreambooth_lora_flux3_small.py | 3 -- ...ain_dreambooth_lora_flux3_small_img2img.py | 30 ------------------- 2 files changed, 33 deletions(-) diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_small.py b/examples/dreambooth/train_dreambooth_lora_flux3_small.py index 3fdaafdba4e1..c538fa88c4be 100644 --- a/examples/dreambooth/train_dreambooth_lora_flux3_small.py +++ b/examples/dreambooth/train_dreambooth_lora_flux3_small.py @@ -733,9 +733,6 @@ def parse_args(input_args=None): ) parser.add_argument("--prodigy_decouple", type=bool, default=True, help="Use AdamW style decoupled weight decay") parser.add_argument("--adam_weight_decay", type=float, default=1e-04, help="Weight decay to use for unet params") - parser.add_argument( - "--adam_weight_decay_text_encoder", type=float, default=1e-03, help="Weight decay to use for text_encoder" - ) parser.add_argument( "--lora_layers", diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py b/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py index 341a99ecde0d..35a63b11666d 100644 --- a/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py +++ b/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py @@ -451,13 +451,6 @@ def parse_args(input_args=None): parser.add_argument("--repeats", type=int, default=1, help="How many times to repeat the training data.") - parser.add_argument( - "--class_data_dir", - type=str, - default=None, - required=False, - help="A folder containing the training data of class images.", - ) parser.add_argument( "--instance_prompt", type=str, @@ -587,9 +580,6 @@ def parse_args(input_args=None): parser.add_argument( "--train_batch_size", type=int, default=4, help="Batch size (per device) for the training dataloader." ) - parser.add_argument( - "--sample_batch_size", type=int, default=4, help="Batch size (per device) for sampling images." - ) parser.add_argument("--num_train_epochs", type=int, default=1) parser.add_argument( "--max_train_steps", @@ -720,9 +710,6 @@ def parse_args(input_args=None): ) parser.add_argument("--prodigy_decouple", type=bool, default=True, help="Use AdamW style decoupled weight decay") parser.add_argument("--adam_weight_decay", type=float, default=1e-04, help="Weight decay to use for unet params") - parser.add_argument( - "--adam_weight_decay_text_encoder", type=float, default=1e-03, help="Weight decay to use for text_encoder" - ) parser.add_argument( "--lora_layers", @@ -1160,23 +1147,6 @@ def __len__(self): return self.sampler_len -class PromptDataset(Dataset): - "A simple dataset to prepare the prompts to generate class images on multiple GPUs." - - def __init__(self, prompt, num_samples): - self.prompt = prompt - self.num_samples = num_samples - - def __len__(self): - return self.num_samples - - def __getitem__(self, index): - example = {} - example["prompt"] = self.prompt - example["index"] = index - return example - - def main(args): if args.report_to == "wandb" and args.hub_token is not None: raise ValueError( From 8af9ae4441edc701f330181e8a87cd011260facf Mon Sep 17 00:00:00 2001 From: linoytsaban Date: Wed, 14 Oct 2026 12:01:53 +0000 Subject: [PATCH 22/34] bump yarn example rank to 32 in flux3 small readme --- examples/dreambooth/README_flux3_small.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/dreambooth/README_flux3_small.md b/examples/dreambooth/README_flux3_small.md index 4ace1414bf03..5d4efc48b5ac 100644 --- a/examples/dreambooth/README_flux3_small.md +++ b/examples/dreambooth/README_flux3_small.md @@ -80,8 +80,8 @@ accelerate launch train_dreambooth_lora_flux3_small.py \ --train_batch_size=1 \ --gradient_checkpointing \ --cache_latents \ - --rank=16 \ - --lora_alpha=16 \ + --rank=32 \ + --lora_alpha=32 \ --optimizer="adamW" \ --use_8bit_adam \ --learning_rate=1e-4 \ From c870eb871277c66952dc2fea0927543799039312 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?apolin=C3=A1rio?= Date: Wed, 14 Oct 2026 14:34:42 +0200 Subject: [PATCH 23/34] Document references and layout prompts for FLUX.3 Matches the prompt grammar in BFL's reference inference code (bfl-hf-flux3-confidential#3): references are addressed as in the order passed to image, and regions can be placed with a trailing JSON list of id / bbox / desc (from / src_bbox / tgt_bbox when editing). --- docs/source/en/api/pipelines/flux3.md | 31 +++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index 6bd1e3295a40..83d4159bae7a 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -60,11 +60,11 @@ night = pipe( )[0] ``` -With several references, the prompt can refer to them in order: +With several references, refer to them in the prompt as ``, ``, ... in the order they are passed to `image`: ```python combined = pipe( - prompt="The fox from the first image sitting on the wooden bench from the second image, in the same snowy clearing.", + prompt="The fox from sitting on the wooden bench from , in the same snowy clearing.", image=[image, bench], height=768, width=1360, @@ -74,6 +74,33 @@ combined = pipe( Pass `height` and/or `width` to override the derived size. +## Layout control + +To place content explicitly, tag regions in the caption with `` and append a JSON list of regions to the prompt. Each region has an `id`, a `bbox` of `[y0, x0, y1, x1]` (top-left and bottom-right corners, origin top-left) normalized to 0-1000 on both axes so it is independent of the output size, and a `desc`: + +```python +prompt = ( + "Minimalist graphic illustration featuring a black silhouette of a person centered against a " + "solid, vibrant chartreuse background . " + '[{"id": "background_1", "bbox": [0, 0, 1000, 1000], "desc": "A flat field of neon yellow-green."}, ' + '{"id": "silhouette_1", "bbox": [150, 150, 850, 850], "desc": "A black silhouette of a person in motion."}]' +) +``` + +When editing, each region names its source reference with `from` (for example `"ref_image_0"`), where it is in that image with `src_bbox`, and where it should end up with `tgt_bbox`. Equal boxes keep a region in place: + +```python +prompt = ( + "In , move the knight figure upwards and to the left. Keep the rest of the image " + "unchanged, preserving the backdrop . " + '[{"id": "knight_1", "from": "ref_image_0", "src_bbox": [500, 150, 850, 350], "tgt_bbox": [194, 55, 544, 255], ' + '"desc": "A miniature amigurumi knight crocheted from grey yarn."}, ' + '{"id": "backdrop_1", "from": "ref_image_0", "src_bbox": [0, 0, 1000, 1000], "tgt_bbox": [0, 0, 1000, 1000], ' + '"desc": "A soft, out-of-focus backdrop of knitted yarn."}]' +) +image = pipe(prompt=prompt, image=knight, output="images")[0] +``` + ## Guidance scale The pipeline applies classifier-free guidance through its `guider` component and does not take a `guidance_scale` argument at call time. The default guider is [`ClassifierFreeGuidance`] with `guidance_scale=3.5`, the recommended setting for both workflows. To change it, create an updated copy of the guider: From 6d57817b887219e77ae90b7635417949101f2e1d Mon Sep 17 00:00:00 2001 From: linoytsaban Date: Wed, 14 Oct 2026 13:05:38 +0000 Subject: [PATCH 24/34] examples/flux3: set klein model id, document ref image prompts, clean leftover refs --- examples/dreambooth/README_flux3_small.md | 28 +++++++++++++++++-- .../train_dreambooth_lora_flux3_small.py | 5 ++-- ...ain_dreambooth_lora_flux3_small_img2img.py | 20 ++++++++----- 3 files changed, 40 insertions(+), 13 deletions(-) diff --git a/examples/dreambooth/README_flux3_small.md b/examples/dreambooth/README_flux3_small.md index 5d4efc48b5ac..8dcf7f5a7132 100644 --- a/examples/dreambooth/README_flux3_small.md +++ b/examples/dreambooth/README_flux3_small.md @@ -66,7 +66,7 @@ This trains a *yarn-art* style LoRA on [Norod78/Yarn-art-style](https://huggingf whose captions already contain the trigger phrase `yarn art style`. ```bash -export MODEL_NAME="" +export MODEL_NAME="black-forest-labs/flux-3-image-klein-base" export OUTPUT_DIR="trained-flux3-small-yarn-lora" accelerate launch train_dreambooth_lora_flux3_small.py \ @@ -111,7 +111,7 @@ This example uses [kontext-community/relighting](https://huggingface.co/datasets (`file_name` = source, `output` = relit target, `instruction` = edit prompt): ```bash -export MODEL_NAME="" +export MODEL_NAME="black-forest-labs/flux-3-image-klein-base" export OUTPUT_DIR="trained-flux3-small-relight-lora" accelerate launch train_dreambooth_lora_flux3_small_img2img.py \ @@ -143,6 +143,28 @@ accelerate launch train_dreambooth_lora_flux3_small_img2img.py \ i2i reads its data through `--dataset_name` (a dataset with an image column, a conditioning-image column and a caption column), not a plain image folder. +### Addressing the reference image in the prompt + +FLUX.3 refers to each conditioning image passed to `image=` as ``, ``, ... in the order +they are passed. i2i training uses a single conditioning image, so captions address it as ``, e.g. +`"Relight with warm sunset light."` Matching this grammar in your captions keeps the adapter aligned +with how the base model reads references. + +For explicit placement you can tag regions with an `` and append a JSON list of regions to the caption. Each +region has an `id`, a `bbox` of `[y0, x0, y1, x1]` (top-left and bottom-right corners, origin top-left) normalized to +0-1000 on both axes, and a `desc`. When editing, a region also names its source reference with `from` (e.g. +`"ref_image_0"`), where it starts with `src_bbox`, and where it should end up with `tgt_bbox` (equal boxes keep it in +place): + +```text +In , move the knight up and to the left, keeping everything else. +[{"id": "knight_1", "from": "ref_image_0", "src_bbox": [500, 150, 850, 350], "tgt_bbox": [194, 55, 544, 255], + "desc": "A miniature knight crocheted from grey yarn."}] +``` + +See the [FLUX.3 pipeline docs](https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/flux3.md) +for the full prompt grammar. + ## Tracking Pass `--report_to="wandb"` (or `tensorboard`) and `--validation_prompt` (plus `--validation_image` for i2i) to log @@ -189,7 +211,7 @@ Load a trained adapter onto the modular pipeline and generate: import torch from diffusers.modular_pipelines import Flux3AutoBlocks -pipe = Flux3AutoBlocks().init_pipeline("") +pipe = Flux3AutoBlocks().init_pipeline("black-forest-labs/flux-3-image-klein-base") pipe.load_components(dtype=torch.bfloat16) pipe.to("cuda") diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_small.py b/examples/dreambooth/train_dreambooth_lora_flux3_small.py index c538fa88c4be..ce8d2ff274ea 100644 --- a/examples/dreambooth/train_dreambooth_lora_flux3_small.py +++ b/examples/dreambooth/train_dreambooth_lora_flux3_small.py @@ -227,7 +227,7 @@ def save_model_card( These are {repo_id} DreamBooth LoRA weights for {base_model}. -The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3.md). +The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3_small.md). Quant training? {quant_training} @@ -1957,7 +1957,7 @@ def get_sigmas(timesteps, n_dim=4, dtype=torch.float32): image_ids = prepare_latent_ids(lh, lw, accelerator.device) # (image_seq_len, 4) # FLUX.3 is not guidance-distilled: no guidance argument. The flow timestep is a per-sample scalar in # [0, 1] (the transformer scales it by 1000 and broadcasts it over the image tokens internally); the - # text context and any reference tokens sit at timestep 0. + # text context sits at timestep 0. timestep = sigmas.reshape(bsz) # Predict the noise residual. The transformer returns only the generated image tokens, so there is @@ -2133,7 +2133,6 @@ def get_sigmas(timesteps, n_dim=4, dtype=torch.float32): is_final_validation=True, torch_dtype=weight_dtype, ) - images = None del pipeline free_memory() diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py b/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py index 35a63b11666d..5230d4b48f69 100644 --- a/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py +++ b/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py @@ -215,7 +215,7 @@ def save_model_card( instance_prompt=None, validation_prompt=None, repo_folder=None, - fp8_training=False, + quant_training=None, ): widget_dict = [] if images is not None: @@ -234,9 +234,9 @@ def save_model_card( These are {repo_id} DreamBooth LoRA weights for {base_model}. -The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3.md). +The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3_small.md). -FP8 training? {fp8_training} +Quant training? {quant_training} ## Trigger words @@ -818,7 +818,7 @@ def parse_args(input_args=None): if args.cond_image_column is None: raise ValueError( - "you must provide --cond_image_column for image-to-image training. Otherwise please see Flux2 text-to-image training example." + "you must provide --cond_image_column for image-to-image training. Otherwise use the FLUX.3 text-to-image training script (train_dreambooth_lora_flux3_small.py)." ) else: assert args.image_column is not None @@ -953,14 +953,15 @@ def __init__( if not image.mode == "RGB": image = image.convert("RGB") dest_image = None - if self.cond_images: # todo: take care of max area for buckets + if self.cond_images: dest_image = self.cond_images[i] image_width, image_height = dest_image.size if image_width * image_height > 1024 * 1024: dest_image = Flux2ImageProcessor._resize_to_target_area(dest_image, 1024 * 1024) image_width, image_height = dest_image.size - multiple_of = 2 ** (4 - 1) # 2 ** (len(vae.config.block_out_channels) - 1), temp! + # VAE spatial compression: 2 ** (len(vae.config.block_out_channels) - 1) = 8 for the Flux2 VAE + multiple_of = 2 ** (4 - 1) image_width = (image_width // multiple_of) * multiple_of image_height = (image_height // multiple_of) * multiple_of image_processor = Flux2ImageProcessor() @@ -2002,6 +2003,11 @@ def get_sigmas(timesteps, n_dim=4, dtype=torch.float32): free_memory() validation_prompt = args.validation_prompt if args.validation_prompt else args.final_validation_prompt + quant_training = None + if args.do_fp8_training: + quant_training = "FP8 TorchAO" + elif args.bnb_quantization_config_path: + quant_training = "BitsandBytes" save_model_card( (args.hub_model_id or Path(args.output_dir).name) if not args.push_to_hub else repo_id, images=images, @@ -2009,7 +2015,7 @@ def get_sigmas(timesteps, n_dim=4, dtype=torch.float32): instance_prompt=args.instance_prompt, validation_prompt=validation_prompt, repo_folder=args.output_dir, - fp8_training=args.do_fp8_training, + quant_training=quant_training, ) if args.push_to_hub: From 062061e3b6e561db05787074246c4f6cd0350bbe Mon Sep 17 00:00:00 2001 From: linoytsaban Date: Wed, 14 Oct 2026 13:05:45 +0000 Subject: [PATCH 25/34] point flux3 image docs at the klein release model ids --- .../source/en/api/models/flux3_transformer.md | 2 +- docs/source/en/api/pipelines/flux3.md | 22 +++++++++---------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/source/en/api/models/flux3_transformer.md b/docs/source/en/api/models/flux3_transformer.md index b923f28268a8..e87b20ef4860 100644 --- a/docs/source/en/api/models/flux3_transformer.md +++ b/docs/source/en/api/models/flux3_transformer.md @@ -21,7 +21,7 @@ import torch from diffusers import Flux3Transformer2DModel transformer = Flux3Transformer2DModel.from_pretrained( - "black-forest-labs/FLUX.3-image-small", subfolder="transformer", dtype=torch.bfloat16 + "black-forest-labs/flux-3-image-klein-base", subfolder="transformer", dtype=torch.bfloat16 ) ``` diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index 6bd1e3295a40..f001745fb9e2 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -23,7 +23,7 @@ FLUX.3 ships as a modular pipeline. Prompting matters: the model responds best t import torch from diffusers import ModularPipeline -pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small") +pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") pipe.load_components(dtype=torch.bfloat16) pipe.to("cuda") @@ -90,7 +90,7 @@ The step-distilled checkpoint generates in 16 steps without classifier-free guid ```python -pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small-distilled") +pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein") pipe.load_components(dtype=torch.bfloat16) pipe.to("cuda") @@ -124,7 +124,7 @@ from diffusers.modular_pipelines import Flux3AutoBlocks print(Flux3AutoBlocks().available_workflows) # ["text2image", "image_conditioned"] -t2i = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small", workflow="text2image") +t2i = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base", workflow="text2image") print(t2i.blocks.doc) ``` @@ -133,7 +133,7 @@ print(t2i.blocks.doc) Loading everything in bf16 takes about 20 GB of weights (the transformer is 5.5B parameters, the Qwen3-VL-4B text encoder 4B), with the image VAE in fp32. For GPUs with less memory, enable component offloading: ```python -pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small") +pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") pipe.load_components(dtype=torch.bfloat16) pipe.enable_auto_cpu_offload(device="cuda") ``` @@ -147,13 +147,13 @@ import torch from diffusers import Flux3Transformer2DModel, ModularPipeline transformer = Flux3Transformer2DModel.from_single_file( - "https://huggingface.co/black-forest-labs/FLUX.3-image-small/blob/main/model.safetensors", - config="black-forest-labs/FLUX.3-image-small", + "https://huggingface.co/black-forest-labs/flux-3-image-klein-base/blob/main/model.safetensors", + config="black-forest-labs/flux-3-image-klein-base", subfolder="transformer", dtype=torch.bfloat16, ) -pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.3-image-small") +pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") pipe.load_components(names=["text_encoder", "tokenizer", "vae"], dtype=torch.bfloat16) pipe.update_components(transformer=transformer) pipe.to("cuda") @@ -167,8 +167,8 @@ Loading the FP8r and NVFP4 single-file checkpoints are also supported. Both load from diffusers import BFLQuantizationConfig transformer = Flux3Transformer2DModel.from_single_file( - "https://huggingface.co/black-forest-labs/FLUX.3-image-small/blob/main/variants/nvfp4/model.safetensors", - config="black-forest-labs/FLUX.3-image-small", + "https://huggingface.co/black-forest-labs/flux-3-image-klein-base/blob/main/variants/nvfp4/model.safetensors", + config="black-forest-labs/flux-3-image-klein-base", subfolder="transformer", quantization_config=BFLQuantizationConfig(compute_dtype=torch.bfloat16), dtype=torch.bfloat16, @@ -181,8 +181,8 @@ configs (`flow_map=True`): ```python transformer = Flux3Transformer2DModel.from_single_file( - "https://huggingface.co/black-forest-labs/FLUX.3-image-small/blob/main/variants/sd-nvfp4/model.safetensors", - config="black-forest-labs/FLUX.3-image-small-distilled", + "https://huggingface.co/black-forest-labs/flux-3-image-klein-base/blob/main/variants/sd-nvfp4/model.safetensors", + config="black-forest-labs/flux-3-image-klein", subfolder="transformer", quantization_config=BFLQuantizationConfig(compute_dtype=torch.bfloat16), dtype=torch.bfloat16, From 1f913d4390a1c6247028b66f620d24f535adfb68 Mon Sep 17 00:00:00 2001 From: Sayak Paul Date: Thu, 15 Oct 2026 08:54:45 +0530 Subject: [PATCH 26/34] fix: TestFlux3DistilledModularPipelineWorkflow::test_workflow_defaults test (#35) --- .../modular_pipelines/flux3/test_modular_pipeline_flux3.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py index 22febdd6872a..138d58564d44 100644 --- a/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py +++ b/tests/modular_pipelines/flux3/test_modular_pipeline_flux3.py @@ -146,8 +146,10 @@ } FLUX3_DISTILLED_COMPONENTS = {name: cls for name, cls in FLUX3_COMPONENTS.items() if name != "guider"} FLUX3_DISTILLED_SAMPLING = {"scheduler": {"shift": 6.93}} -# no negative prompt, 16 steps by default -FLUX3_DISTILLED_INPUTS = {name: value for name, value in FLUX3_INPUTS.items() if name != "negative_prompt"} +# no negative prompt and no attention kwargs, 16 steps by default +FLUX3_DISTILLED_INPUTS = { + name: value for name, value in FLUX3_INPUTS.items() if name not in ("negative_prompt", "attention_kwargs") +} FLUX3_DISTILLED_INPUTS["num_inference_steps"] = 16 FLUX3_DISTILLED_WORKFLOW_DEFAULTS = { "text2image": { From fb51c15f4538e6252ce5ced1fd1f58fe04f3b58f Mon Sep 17 00:00:00 2001 From: Sayak Paul Date: Thu, 15 Oct 2026 10:26:10 +0530 Subject: [PATCH 27/34] fix checkpoint ids in the flux3 docs. (#38) --- docs/source/en/api/pipelines/flux3.md | 10 +++------- docs/source/en/quantization/bfl.md | 28 +++++++++++++-------------- 2 files changed, 17 insertions(+), 21 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index bda58b0c3a6b..918ebec88542 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -18,7 +18,6 @@ FLUX.3 is an image generation model from Black Forest Labs: a single transformer FLUX.3 ships as a modular pipeline. Prompting matters: the model responds best to full scene captions that describe the subject, composition, lighting, and atmosphere — see the [FLUX.3 prompting guide](https://docs.bfl.ai) for details. Get started with a text-to-image generation: - ```python import torch from diffusers import ModularPipeline @@ -115,7 +114,6 @@ See the [guiders guide](../../using-diffusers/guiders) to switch to a different The step-distilled checkpoint generates in 16 steps without classifier-free guidance. Its transformer is a flow map: at every step it is told the current timestep and the one the step jumps to. It loads like the base model (the repo's `modular_model_index.json` resolves to [`Flux3DistilledModularPipeline`] and [`Flux3DistilledAutoBlocks`]); the pipeline has no `guider` and does not take `negative_prompt`: - ```python pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein") pipe.load_components(dtype=torch.bfloat16) @@ -128,9 +126,8 @@ It serves the same two workflows as the base model. `num_inference_steps` defaul ## Reproducible generation -A `generator` fixes the noise the pipeline draws (see [Reproducibility](../../using-diffusers/reusing_seeds)). To reuse noise drawn elsewhere, pass it as `latents` with shape `(batch_size, 128, height // 16, width // 16)`. For example, the [BFL implementation](https://github.com/black-forest-labs/TODO) draws its noise as image tokens on the CPU and casts it to bf16; this reproduces its sample for a given seed: +A `generator` fixes the noise the pipeline draws (see [Reproducibility](../../using-diffusers/reusing_seeds)). To reuse noise drawn elsewhere, pass it as `latents` with shape `(batch_size, 128, height // 16, width // 16)`. For example, the [BFL implementation](https://github.com/black-forest-labs/flux-3) draws its noise as image tokens on the CPU and casts it to bf16; this reproduces its sample for a given seed: - ```python height, width = 1024, 1024 h, w = height // 16, width // 16 @@ -181,7 +178,7 @@ transformer = Flux3Transformer2DModel.from_single_file( ) pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") -pipe.load_components(names=["text_encoder", "tokenizer", "vae"], dtype=torch.bfloat16) +pipe.load_components(dtype=torch.bfloat16) pipe.update_components(transformer=transformer) pipe.to("cuda") ``` @@ -205,10 +202,9 @@ transformer = Flux3Transformer2DModel.from_single_file( When loading the step-distilled checkpoints, we need their configs (`flow_map=True`): - ```python transformer = Flux3Transformer2DModel.from_single_file( - "https://huggingface.co/black-forest-labs/flux-3-image-klein-base/blob/main/variants/sd-nvfp4/model.safetensors", + "https://huggingface.co/black-forest-labs/flux-3-image-klein/blob/main/variants/nvfp4/model.safetensors", config="black-forest-labs/flux-3-image-klein", subfolder="transformer", quantization_config=BFLQuantizationConfig(compute_dtype=torch.bfloat16), diff --git a/docs/source/en/quantization/bfl.md b/docs/source/en/quantization/bfl.md index 6bc90ab0506b..fae0ccaf1ed9 100644 --- a/docs/source/en/quantization/bfl.md +++ b/docs/source/en/quantization/bfl.md @@ -20,32 +20,32 @@ specific language governing permissions and limitations under the License. Diffusers loads both through [`~FromSingleFileMixin.from_single_file`] with a [`BFLQuantizationConfig`]. The scheme is read from the checkpoint, so the same config works for either file. Only loading prequantized checkpoints is supported; pipeline-level loading is not. -The example below loads a `fp8r` checkpoint of [FLUX.2 klein 4B](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B). Pass `config` so that the model config is read from the Diffusers repository. +The example below loads the `fp8r` checkpoint of [FLUX 3 Image Klein Base](https://huggingface.co/black-forest-labs/flux-3-image-klein-base), which lives at `variants/fp8r/model.safetensors` in the model repository. Pass `config` so that the model config is read from the Diffusers folder of that repository, then load the other components by name (see [Single-file loading](../api/pipelines/flux3#single-file-loading)). ```python import torch -from diffusers import BFLQuantizationConfig, Flux2KleinPipeline, Flux2Transformer2DModel +from diffusers import BFLQuantizationConfig, Flux3Transformer2DModel, ModularPipeline -ckpt_path = "https://huggingface.co/sayakpaul/FLUX.2-klein-4B-fp8r-nvfp4/blob/main/flux-2-klein-4b-fp8r.safetensors" -transformer = Flux2Transformer2DModel.from_single_file( +ckpt_path = "https://huggingface.co/black-forest-labs/flux-3-image-klein-base/blob/main/variants/fp8r/model.safetensors" +transformer = Flux3Transformer2DModel.from_single_file( ckpt_path, - config="black-forest-labs/FLUX.2-klein-4B", + config="black-forest-labs/flux-3-image-klein-base", subfolder="transformer", quantization_config=BFLQuantizationConfig(compute_dtype=torch.bfloat16), dtype=torch.bfloat16, ) -pipe = Flux2KleinPipeline.from_pretrained( - "black-forest-labs/FLUX.2-klein-4B", - transformer=transformer, - dtype=torch.bfloat16, -).to("cuda") -prompt = "A cat holding a sign that says hello world" -image = pipe(prompt=prompt, num_inference_steps=4, guidance_scale=1.0, generator=torch.manual_seed(0)).images[0] -image.save("flux2-klein-fp8r.png") +pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") +pipe.load_components(dtype=torch.bfloat16) +pipe.update_components(transformer=transformer) +pipe.to("cuda") + +prompt = "A wide-angle wildlife photograph of a red fox standing alert in a snow-covered forest clearing at golden hour." +image = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="images")[0] +image.save("flux3-klein-base-fp8r.png") ``` -Replace the filename with `flux-2-klein-4b-nvfp4.safetensors` to load the `nvfp4` variant. +Replace `variants/fp8r` with `variants/nvfp4` to load the `nvfp4` variant. The step-distilled [FLUX 3 Image Klein](https://huggingface.co/black-forest-labs/flux-3-image-klein) ships the same two variants; load them with its repository id as `config` and pipeline. ## Kernels From c0ddc02d1823abc3696d75393fd731810a3567dd Mon Sep 17 00:00:00 2001 From: Aritra Roy Gosthipaty Date: Thu, 15 Oct 2026 15:21:16 +0530 Subject: [PATCH 28/34] [docs] update pipe loading with vae fp32 dtype (#39) * update pipe loading with vae fp32 dtype * updating `load_componenets` --- docs/source/en/api/pipelines/flux3.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index 918ebec88542..0ecf52987f12 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -23,7 +23,7 @@ import torch from diffusers import ModularPipeline pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") -pipe.load_components(dtype=torch.bfloat16) +pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) pipe.to("cuda") image = pipe( @@ -158,7 +158,7 @@ Loading everything in bf16 takes about 20 GB of weights (the transformer is 5.5B ```python pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") -pipe.load_components(dtype=torch.bfloat16) +pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) pipe.enable_auto_cpu_offload(device="cuda") ``` @@ -178,7 +178,7 @@ transformer = Flux3Transformer2DModel.from_single_file( ) pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein-base") -pipe.load_components(dtype=torch.bfloat16) +pipe.load_components(dtype={"vae": torch.float32, "default": torch.bfloat16}) pipe.update_components(transformer=transformer) pipe.to("cuda") ``` From ac36f1f4e7110c9f817121e7dadca99f2c014dd6 Mon Sep 17 00:00:00 2001 From: Sayak Paul Date: Thu, 15 Oct 2026 15:26:01 +0530 Subject: [PATCH 29/34] use actual ckpts. (#42) --- src/diffusers/loaders/single_file_utils.py | 4 ++-- .../models/transformers/test_models_transformer_flux3.py | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/diffusers/loaders/single_file_utils.py b/src/diffusers/loaders/single_file_utils.py index e3a1eb1888ab..5182d15f4388 100644 --- a/src/diffusers/loaders/single_file_utils.py +++ b/src/diffusers/loaders/single_file_utils.py @@ -213,8 +213,8 @@ "flux-depth": {"pretrained_model_name_or_path": "black-forest-labs/FLUX.1-Depth-dev"}, "flux-schnell": {"pretrained_model_name_or_path": "black-forest-labs/FLUX.1-schnell"}, "flux-2-dev": {"pretrained_model_name_or_path": "black-forest-labs/FLUX.2-dev"}, - "flux-3-image": {"pretrained_model_name_or_path": "diffusers-internal-dev/f3-small-img"}, - "flux-3-image-distilled": {"pretrained_model_name_or_path": "diffusers-internal-dev/f3-small-image-distilled"}, + "flux-3-image": {"pretrained_model_name_or_path": "black-forest-labs/flux-3-image-klein-base"}, + "flux-3-image-distilled": {"pretrained_model_name_or_path": "black-forest-labs/flux-3-image-klein"}, "chroma": {"pretrained_model_name_or_path": "lodestones/Chroma1-HD"}, "ltx-video": {"pretrained_model_name_or_path": "diffusers/LTX-Video-0.9.0"}, "ltx-video-0.9.1": {"pretrained_model_name_or_path": "diffusers/LTX-Video-0.9.1"}, diff --git a/tests/models/transformers/test_models_transformer_flux3.py b/tests/models/transformers/test_models_transformer_flux3.py index cec0449d6282..66f9f8149685 100644 --- a/tests/models/transformers/test_models_transformer_flux3.py +++ b/tests/models/transformers/test_models_transformer_flux3.py @@ -156,11 +156,11 @@ class TestFlux3TransformerAttention(Flux3TransformerTesterConfig, AttentionTeste class TestFlux3SingleFile(Flux3TransformerTesterConfig, SingleFileTesterMixin): @property def ckpt_path(self): - return "https://huggingface.co/herzogenhorn/f3-image-small/blob/main/model.safetensors" + return "https://huggingface.co/black-forest-labs/flux-3-image-klein-base/blob/main/model.safetensors" @property def pretrained_model_name_or_path(self): - return "diffusers-internal-dev/f3-small-img" + return "black-forest-labs/flux-3-image-klein-base" @property def pretrained_model_kwargs(self): @@ -178,8 +178,8 @@ class Flux3TransformerBFLTesterConfig(Flux3TransformerTesterConfig): @property def bfl_checkpoints(self): return { - "fp8r": "https://huggingface.co/herzogenhorn/f3-image-small/blob/main/variants/fp8r/model.safetensors", - "nvfp4": "https://huggingface.co/herzogenhorn/f3-image-small/blob/main/variants/nvfp4/model.safetensors", + "fp8r": "https://huggingface.co/black-forest-labs/flux-3-image-klein-base/blob/main/variants/fp8r/model.safetensors", + "nvfp4": "https://huggingface.co/black-forest-labs/flux-3-image-klein-base/blob/main/variants/nvfp4/model.safetensors", } @property From dc356dc7662b79295a30e06f8024ba0154f9fea3 Mon Sep 17 00:00:00 2001 From: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com> Date: Thu, 15 Oct 2026 14:15:37 +0300 Subject: [PATCH 30/34] flux3 klein example: rename from "small" and fix caption dropout for long prompts (#41) * examples/flux3: rename small trainer, tests and readme files to klein * examples/flux3: fix caption dropout shape mismatch for prompts longer than the pad window * examples/flux3: add caption dropout to the i2i trainer * examples/flux3: complete the klein rename (test class names, example output dirs) * examples/flux3: guard against train_batch_size>1 with variable-length captions --- ...E_flux3_small.md => README_flux3_klein.md} | 22 ++++---- ...small.txt => requirements_flux3_klein.txt} | 0 ...py => test_dreambooth_lora_flux3_klein.py} | 14 ++--- ...st_dreambooth_lora_flux3_klein_img2img.py} | 8 +-- ...y => train_dreambooth_lora_flux3_klein.py} | 48 +++++++++++++---- ...in_dreambooth_lora_flux3_klein_img2img.py} | 54 +++++++++++++++++-- 6 files changed, 110 insertions(+), 36 deletions(-) rename examples/dreambooth/{README_flux3_small.md => README_flux3_klein.md} (92%) rename examples/dreambooth/{requirements_flux3_small.txt => requirements_flux3_klein.txt} (100%) rename examples/dreambooth/{test_dreambooth_lora_flux3_small.py => test_dreambooth_lora_flux3_klein.py} (95%) rename examples/dreambooth/{test_dreambooth_lora_flux3_small_img2img.py => test_dreambooth_lora_flux3_klein_img2img.py} (96%) rename examples/dreambooth/{train_dreambooth_lora_flux3_small.py => train_dreambooth_lora_flux3_klein.py} (97%) rename examples/dreambooth/{train_dreambooth_lora_flux3_small_img2img.py => train_dreambooth_lora_flux3_klein_img2img.py} (96%) diff --git a/examples/dreambooth/README_flux3_small.md b/examples/dreambooth/README_flux3_klein.md similarity index 92% rename from examples/dreambooth/README_flux3_small.md rename to examples/dreambooth/README_flux3_klein.md index 8dcf7f5a7132..2649356e2988 100644 --- a/examples/dreambooth/README_flux3_small.md +++ b/examples/dreambooth/README_flux3_klein.md @@ -3,8 +3,8 @@ These scripts train LoRA adapters for the FLUX.3 image model (`Flux3Transformer2DModel`, served by `Flux3ModularPipeline`) with the DreamBooth recipe: -- `train_dreambooth_lora_flux3_small.py` — text-to-image (t2i) -- `train_dreambooth_lora_flux3_small_img2img.py` — image-to-image / instruction editing (i2i) +- `train_dreambooth_lora_flux3_klein.py` — text-to-image (t2i) +- `train_dreambooth_lora_flux3_klein_img2img.py` — image-to-image / instruction editing (i2i) > [!NOTE] > LoRA **loading and inference** for this model live in the pipeline/loader PR (`Flux3LoraLoaderMixin`); these @@ -18,7 +18,7 @@ A few things that are specific to FLUX.3 and differ from the FLUX.1/FLUX.2 train through the Qwen chat template and the text window is right-padded to a multiple of 80 tokens (capped at `--max_sequence_length`), matching the pipeline's `Flux3TextEncoderStep`. - **No guidance distillation.** The model is not guidance-distilled, so there is no guidance argument in training; - classifier-free guidance happens at inference. Use `--caption_dropout` for unconditional training (t2i). + classifier-free guidance happens at inference. Use `--caption_dropout` for unconditional training (t2i and i2i). - **VAE normalization is a batch-norm.** Latents are 2×2-patchified to 128 channels and normalized with the VAE's stored batch-norm statistics (not a scalar `scaling_factor`/`shift_factor`). - **Flow matching with a fixed shift of 6.93** (no resolution-dependent dynamic shifting); the velocity target is @@ -41,7 +41,7 @@ pip install -e . Then `cd` into `examples/dreambooth` and install the requirements: ```bash -pip install -r requirements_flux3_small.txt +pip install -r requirements_flux3_klein.txt ``` Initialize an [🤗 Accelerate](https://github.com/huggingface/accelerate/) environment with: @@ -67,9 +67,9 @@ whose captions already contain the trigger phrase `yarn art style`. ```bash export MODEL_NAME="black-forest-labs/flux-3-image-klein-base" -export OUTPUT_DIR="trained-flux3-small-yarn-lora" +export OUTPUT_DIR="trained-flux3-klein-yarn-lora" -accelerate launch train_dreambooth_lora_flux3_small.py \ +accelerate launch train_dreambooth_lora_flux3_klein.py \ --pretrained_model_name_or_path=$MODEL_NAME \ --dataset_name="Norod78/Yarn-art-style" \ --caption_column="text" \ @@ -112,9 +112,9 @@ This example uses [kontext-community/relighting](https://huggingface.co/datasets ```bash export MODEL_NAME="black-forest-labs/flux-3-image-klein-base" -export OUTPUT_DIR="trained-flux3-small-relight-lora" +export OUTPUT_DIR="trained-flux3-klein-relight-lora" -accelerate launch train_dreambooth_lora_flux3_small_img2img.py \ +accelerate launch train_dreambooth_lora_flux3_klein_img2img.py \ --pretrained_model_name_or_path=$MODEL_NAME \ --dataset_name="kontext-community/relighting" \ --image_column="output" \ @@ -200,8 +200,8 @@ every stream. Use `--lora_layers` (a comma-separated list) to target a different - `--aspect_ratio_buckets` / `--use_aspect_ratio_buckets` group images of similar aspect ratio so a batch shares a resolution (t2i). -- `--caption_dropout` replaces a fraction of instance captions with the empty prompt during training, which improves - classifier-free guidance at inference (t2i). +- `--caption_dropout` replaces a fraction of captions (t2i) or instructions (i2i) with the empty prompt during + training, which improves classifier-free guidance at inference. For i2i the conditioning image is kept. ## Inference @@ -215,7 +215,7 @@ pipe = Flux3AutoBlocks().init_pipeline("black-forest-labs/flux-3-image-klein-bas pipe.load_components(dtype=torch.bfloat16) pipe.to("cuda") -pipe.load_lora_weights("trained-flux3-small-yarn-lora", weight_name="pytorch_lora_weights.safetensors") +pipe.load_lora_weights("trained-flux3-klein-yarn-lora", weight_name="pytorch_lora_weights.safetensors") image = pipe(prompt="a cat, yarn art style", output="images")[0] image.save("yarn_cat.png") diff --git a/examples/dreambooth/requirements_flux3_small.txt b/examples/dreambooth/requirements_flux3_klein.txt similarity index 100% rename from examples/dreambooth/requirements_flux3_small.txt rename to examples/dreambooth/requirements_flux3_klein.txt diff --git a/examples/dreambooth/test_dreambooth_lora_flux3_small.py b/examples/dreambooth/test_dreambooth_lora_flux3_klein.py similarity index 95% rename from examples/dreambooth/test_dreambooth_lora_flux3_small.py rename to examples/dreambooth/test_dreambooth_lora_flux3_klein.py index 3c3fcaf2003a..31260859abc6 100644 --- a/examples/dreambooth/test_dreambooth_lora_flux3_small.py +++ b/examples/dreambooth/test_dreambooth_lora_flux3_klein.py @@ -36,14 +36,14 @@ FLUX3_TINY_REPO = os.environ.get("FLUX3_TINY_REPO", "hf-internal-testing/tiny-flux3-image-modular") -class DreamBoothLoRAFlux3Small(ExamplesTestsAccelerate): +class DreamBoothLoRAFlux3Klein(ExamplesTestsAccelerate): instance_data_dir = "docs/source/en/imgs" instance_prompt = "dog" pretrained_model_name_or_path = FLUX3_TINY_REPO - script_path = "examples/dreambooth/train_dreambooth_lora_flux3_small.py" + script_path = "examples/dreambooth/train_dreambooth_lora_flux3_klein.py" transformer_layer_type = "single_transformer_blocks.0.attn.to_k" - def test_dreambooth_lora_flux3_small(self): + def test_dreambooth_lora_flux3_klein(self): with tempfile.TemporaryDirectory() as tmpdir: test_args = f""" {self.script_path} @@ -77,7 +77,7 @@ def test_dreambooth_lora_flux3_small(self): starts_with_transformer = all(key.startswith("transformer") for key in lora_state_dict.keys()) self.assertTrue(starts_with_transformer) - def test_dreambooth_lora_flux3_small_latent_caching(self): + def test_dreambooth_lora_flux3_klein_latent_caching(self): with tempfile.TemporaryDirectory() as tmpdir: test_args = f""" {self.script_path} @@ -110,7 +110,7 @@ def test_dreambooth_lora_flux3_small_latent_caching(self): starts_with_transformer = all(key.startswith("transformer") for key in lora_state_dict.keys()) self.assertTrue(starts_with_transformer) - def test_dreambooth_lora_flux3_small_layers(self): + def test_dreambooth_lora_flux3_klein_layers(self): with tempfile.TemporaryDirectory() as tmpdir: test_args = f""" {self.script_path} @@ -147,7 +147,7 @@ def test_dreambooth_lora_flux3_small_layers(self): ) self.assertTrue(starts_with_transformer) - def test_dreambooth_lora_flux3_small_checkpointing_checkpoints_total_limit(self): + def test_dreambooth_lora_flux3_klein_checkpointing_checkpoints_total_limit(self): with tempfile.TemporaryDirectory() as tmpdir: test_args = f""" {self.script_path} @@ -172,7 +172,7 @@ def test_dreambooth_lora_flux3_small_checkpointing_checkpoints_total_limit(self) {"checkpoint-4", "checkpoint-6"}, ) - def test_dreambooth_lora_flux3_small_checkpointing_checkpoints_total_limit_removes_multiple_checkpoints(self): + def test_dreambooth_lora_flux3_klein_checkpointing_checkpoints_total_limit_removes_multiple_checkpoints(self): with tempfile.TemporaryDirectory() as tmpdir: test_args = f""" {self.script_path} diff --git a/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py b/examples/dreambooth/test_dreambooth_lora_flux3_klein_img2img.py similarity index 96% rename from examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py rename to examples/dreambooth/test_dreambooth_lora_flux3_klein_img2img.py index 03fa913d6bee..845d956ea3c4 100644 --- a/examples/dreambooth/test_dreambooth_lora_flux3_small_img2img.py +++ b/examples/dreambooth/test_dreambooth_lora_flux3_klein_img2img.py @@ -36,7 +36,7 @@ FLUX3_TINY_REPO = os.environ.get("FLUX3_TINY_REPO", "hf-internal-testing/tiny-flux3-image-modular") -class DreamBoothLoRAFlux3SmallImg2Img(ExamplesTestsAccelerate): +class DreamBoothLoRAFlux3KleinImg2Img(ExamplesTestsAccelerate): # image-to-image training needs a paired conditioning image, so it reads a dataset with an image + # conditioning_image column rather than a plain folder dataset_name = "hf-internal-testing/fill10" @@ -45,9 +45,9 @@ class DreamBoothLoRAFlux3SmallImg2Img(ExamplesTestsAccelerate): caption_column = "text" instance_prompt = "dog" pretrained_model_name_or_path = FLUX3_TINY_REPO - script_path = "examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py" + script_path = "examples/dreambooth/train_dreambooth_lora_flux3_klein_img2img.py" - def test_dreambooth_lora_flux3_small_img2img(self): + def test_dreambooth_lora_flux3_klein_img2img(self): with tempfile.TemporaryDirectory() as tmpdir: test_args = f""" {self.script_path} @@ -86,7 +86,7 @@ def test_dreambooth_lora_flux3_small_img2img(self): self.assertTrue(any("mode_blocks.image." in key for key in lora_state_dict.keys())) self.assertTrue(any("mode_blocks.image_cond." in key for key in lora_state_dict.keys())) - def test_dreambooth_lora_flux3_small_img2img_latent_caching(self): + def test_dreambooth_lora_flux3_klein_img2img_latent_caching(self): with tempfile.TemporaryDirectory() as tmpdir: test_args = f""" {self.script_path} diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_small.py b/examples/dreambooth/train_dreambooth_lora_flux3_klein.py similarity index 97% rename from examples/dreambooth/train_dreambooth_lora_flux3_small.py rename to examples/dreambooth/train_dreambooth_lora_flux3_klein.py index ce8d2ff274ea..9cc4b6a47d5d 100644 --- a/examples/dreambooth/train_dreambooth_lora_flux3_small.py +++ b/examples/dreambooth/train_dreambooth_lora_flux3_klein.py @@ -153,7 +153,7 @@ def prepare_text_ids(seq_len, device): @torch.no_grad() def get_qwen3vl_prompt_embeds( - text_encoder, tokenizer, prompt, device, dtype, text_encoder_out_layers, max_sequence_length + text_encoder, tokenizer, prompt, device, dtype, text_encoder_out_layers, max_sequence_length, pad_to_length=None ): """Stack the FLUX.3 Qwen3-VL hidden layers -> (B, seq, len(layers) * hidden_dim). @@ -177,6 +177,10 @@ def get_qwen3vl_prompt_embeds( for t in texts ] padded_length = min(math.ceil(max(real_lengths) / TEXT_PAD_MULTIPLE) * TEXT_PAD_MULTIPLE, max_sequence_length) + if pad_to_length is not None: + # Force a specific window (caption dropout encodes the empty prompt at a dropped caption's window). Never + # shrink below the prompt's own real length, never exceed max_sequence_length. + padded_length = min(max(padded_length, pad_to_length), max_sequence_length) enc = tokenizer.tokenizer( texts, return_tensors="pt", @@ -227,7 +231,7 @@ def save_model_card( These are {repo_id} DreamBooth LoRA weights for {base_model}. -The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3_small.md). +The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3_klein.md). Quant training? {quant_training} @@ -1615,7 +1619,7 @@ def load_model_hook(models, input_dir): num_workers=args.dataloader_num_workers, ) - def compute_text_embeddings(prompt, text_encoder, tokenizer): + def compute_text_embeddings(prompt, text_encoder, tokenizer, pad_to_length=None): # FLUX.3 has no classic text-encoding pipeline; encode directly with the frozen Qwen3-VL encoder. with torch.no_grad(): prompt_embeds = get_qwen3vl_prompt_embeds( @@ -1626,6 +1630,7 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer): weight_dtype, text_encoder_out_layers=args.text_encoder_out_layers, max_sequence_length=args.max_sequence_length, + pad_to_length=pad_to_length, ) # FLUX.3 text ids are positional only (identical for every sample of a given length). We carry them # batched here so the [instance, class] concat / repeat / caption-dropout plumbing below is unchanged; @@ -1650,12 +1655,6 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer): args.class_prompt, text_encoder, tokenizer ) - # When caption dropout is enabled, we precompute the empty ("") prompt embedding once and swap it in - # for randomly selected instance samples at training time (see the training loop below). - if args.caption_dropout > 0: - with offload_models(text_encoder, device=accelerator.device, offload=args.offload): - empty_prompt_hidden_states, empty_text_ids = compute_text_embeddings("", text_encoder, tokenizer) - # FLUX.3 validation runs through the modular pipeline (which encodes prompts itself), so we only assemble # the generation kwargs here rather than precomputing text embeddings. validation_pipeline_args = {} @@ -1759,6 +1758,34 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer): ) assert all(ids is not None for ids in text_ids_cache), "Text ID cache has unfilled entries." + # The training loop re-batches the cached per-caption embeddings with torch.cat, which needs a uniform + # sequence window across the batch. FLUX.3 pads each caption to its own window, and the bucket sampler + # reshuffles every epoch, so train_batch_size > 1 with variable-length captions would group mismatched + # windows and crash mid-training. Fail fast with a clear message instead. + if args.train_batch_size > 1 and len({embeds.shape[1] for embeds in prompt_embeds_cache}) > 1: + raise ValueError( + "train_batch_size > 1 with variable-length captions is not supported: FLUX.3 pads each caption to" + " its own window and the cached embeddings cannot be batched together. Use --train_batch_size=1," + " or a dataset whose captions all share a length." + ) + + # Caption dropout swaps the empty-prompt embedding in for a dropped caption. FLUX.3 pads each prompt to its own + # window (a multiple of TEXT_PAD_MULTIPLE) and the transformer attends to it, so the empty prompt must match the + # dropped sample's window. Precompute it for every window in use here, while the text encoder is still loaded + # (it is deleted just below, so this cannot be done in the training loop). + empty_prompt_by_window = {} + if args.caption_dropout > 0: + dropout_windows = ( + {embeds.shape[1] for embeds in prompt_embeds_cache} + if train_dataset.custom_instance_prompts + else {static_prompt_embeds.shape[1]} + ) + for window in sorted(dropout_windows): + with offload_models(text_encoder, device=accelerator.device, offload=args.offload): + empty_prompt_by_window[window] = compute_text_embeddings( + "", text_encoder, tokenizer, pad_to_length=window + ) + # move back to cpu before deleting to ensure memory is freed see: https://github.com/huggingface/diffusers/issues/11376#issue-3008144624 if args.cache_latents: vae = vae.to("cpu") @@ -1909,7 +1936,8 @@ def get_sigmas(timesteps, n_dim=4, dtype=torch.float32): n_inst = len(sample_indices) drop_mask = torch.rand(n_inst, device=prompt_embeds.device) < args.caption_dropout if drop_mask.any(): - prompt_embeds[:n_inst][drop_mask] = empty_prompt_hidden_states.to( + empty_hidden_states, empty_text_ids = empty_prompt_by_window[prompt_embeds.shape[1]] + prompt_embeds[:n_inst][drop_mask] = empty_hidden_states.to( device=prompt_embeds.device, dtype=prompt_embeds.dtype ) text_ids[:n_inst][drop_mask] = empty_text_ids.to(device=text_ids.device, dtype=text_ids.dtype) diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py b/examples/dreambooth/train_dreambooth_lora_flux3_klein_img2img.py similarity index 96% rename from examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py rename to examples/dreambooth/train_dreambooth_lora_flux3_klein_img2img.py index 5230d4b48f69..0fec7b2ffcda 100644 --- a/examples/dreambooth/train_dreambooth_lora_flux3_small_img2img.py +++ b/examples/dreambooth/train_dreambooth_lora_flux3_klein_img2img.py @@ -160,7 +160,7 @@ def prepare_text_ids(seq_len, device): @torch.no_grad() def get_qwen3vl_prompt_embeds( - text_encoder, tokenizer, prompt, device, dtype, text_encoder_out_layers, max_sequence_length + text_encoder, tokenizer, prompt, device, dtype, text_encoder_out_layers, max_sequence_length, pad_to_length=None ): """Stack the FLUX.3 Qwen3-VL hidden layers -> (B, seq, len(layers) * hidden_dim). @@ -184,6 +184,10 @@ def get_qwen3vl_prompt_embeds( for t in texts ] padded_length = min(math.ceil(max(real_lengths) / TEXT_PAD_MULTIPLE) * TEXT_PAD_MULTIPLE, max_sequence_length) + if pad_to_length is not None: + # Force a specific window (caption dropout encodes the empty prompt at a dropped instruction's window). Never + # shrink below the prompt's own real length, never exceed max_sequence_length. + padded_length = min(max(padded_length, pad_to_length), max_sequence_length) enc = tokenizer.tokenizer( texts, return_tensors="pt", @@ -234,7 +238,7 @@ def save_model_card( These are {repo_id} DreamBooth LoRA weights for {base_model}. -The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3_small.md). +The weights were trained using [DreamBooth](https://dreambooth.github.io/) with the [FLUX.3 diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_flux3_klein.md). Quant training? {quant_training} @@ -450,6 +454,16 @@ def parse_args(input_args=None): ) parser.add_argument("--repeats", type=int, default=1, help="How many times to repeat the training data.") + parser.add_argument( + "--caption_dropout", + type=float, + default=0.0, + help=( + "Probability of replacing a sample's instruction with an empty string during training, so that fraction" + " of samples is trained unconditionally on the text (the conditioning image is kept). Improves" + " classifier-free guidance. A common value is 0.1." + ), + ) parser.add_argument( "--instance_prompt", @@ -818,7 +832,7 @@ def parse_args(input_args=None): if args.cond_image_column is None: raise ValueError( - "you must provide --cond_image_column for image-to-image training. Otherwise use the FLUX.3 text-to-image training script (train_dreambooth_lora_flux3_small.py)." + "you must provide --cond_image_column for image-to-image training. Otherwise use the FLUX.3 text-to-image training script (train_dreambooth_lora_flux3_klein.py)." ) else: assert args.image_column is not None @@ -1557,7 +1571,7 @@ def load_model_hook(models, input_dir): num_workers=args.dataloader_num_workers, ) - def compute_text_embeddings(prompt, text_encoder, tokenizer): + def compute_text_embeddings(prompt, text_encoder, tokenizer, pad_to_length=None): # FLUX.3 has no classic text-encoding pipeline; encode directly with the frozen Qwen3-VL encoder. with torch.no_grad(): prompt_embeds = get_qwen3vl_prompt_embeds( @@ -1568,6 +1582,7 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer): weight_dtype, text_encoder_out_layers=args.text_encoder_out_layers, max_sequence_length=args.max_sequence_length, + pad_to_length=pad_to_length, ) # FLUX.3 text ids are positional only (identical for every sample of a given length). We carry them # batched here so the repeat plumbing below is unchanged; the transformer consumes a single (seq, 4) @@ -1654,6 +1669,23 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer): prompt_embeds_cache.append(prompt_embeds) text_ids_cache.append(text_ids) + # Caption dropout replaces the instruction with the empty prompt for a fraction of samples (the conditioning + # image is kept). FLUX.3 pads each prompt to its own window (a multiple of TEXT_PAD_MULTIPLE) and the transformer + # attends to it, so the empty prompt must match the dropped sample's window. Precompute it for every window in + # use here, while the text encoder is still loaded (it is deleted below, so this cannot be done in the loop). + empty_prompt_by_window = {} + if args.caption_dropout > 0: + dropout_windows = ( + {embeds.shape[1] for embeds in prompt_embeds_cache} + if train_dataset.custom_instance_prompts + else {instance_prompt_hidden_states.shape[1]} + ) + for window in sorted(dropout_windows): + with offload_models(text_encoder, device=accelerator.device, offload=args.offload): + empty_prompt_by_window[window] = compute_text_embeddings( + "", text_encoder, tokenizer, pad_to_length=window + ) + # move back to cpu before deleting to ensure memory is freed see: https://github.com/huggingface/diffusers/issues/11376#issue-3008144624 if args.cache_latents: vae = vae.to("cpu") @@ -1788,6 +1820,20 @@ def get_sigmas(timesteps, n_dim=4, dtype=torch.float32): prompt_embeds = prompt_embeds.repeat(num_repeat_elements, 1, 1) text_ids = text_ids.repeat(num_repeat_elements, 1, 1) + # Caption dropout: replace the instruction with the empty prompt for a fraction of samples so they + # train unconditionally on the text (the conditioning image is kept). Clone first so the in-place + # write never mutates the cached embeddings. + if args.caption_dropout > 0: + drop_mask = torch.rand(prompt_embeds.shape[0], device=prompt_embeds.device) < args.caption_dropout + if drop_mask.any(): + empty_hidden_states, empty_text_ids = empty_prompt_by_window[prompt_embeds.shape[1]] + prompt_embeds = prompt_embeds.clone() + text_ids = text_ids.clone() + prompt_embeds[drop_mask] = empty_hidden_states.to( + device=prompt_embeds.device, dtype=prompt_embeds.dtype + ) + text_ids[drop_mask] = empty_text_ids.to(device=text_ids.device, dtype=text_ids.dtype) + # Convert images to latent space if args.cache_latents: model_input = latents_cache[step].mode() From 4d3b9f6ff094aeb588e83e42c399c3d85aa04577 Mon Sep 17 00:00:00 2001 From: Yumeng Li <109459218+YumengLi007@users.noreply.github.com> Date: Thu, 15 Oct 2026 14:11:16 +0200 Subject: [PATCH 31/34] update flux3.md (#43) --- docs/source/en/api/pipelines/flux3.md | 50 +++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/docs/source/en/api/pipelines/flux3.md b/docs/source/en/api/pipelines/flux3.md index 0ecf52987f12..b33b86c479ed 100644 --- a/docs/source/en/api/pipelines/flux3.md +++ b/docs/source/en/api/pipelines/flux3.md @@ -16,7 +16,7 @@ FLUX.3 is an image generation model from Black Forest Labs: a single transformer ([`Flux3Transformer2DModel`]) generates images from a text prompt, alone or together with one or more reference images. Text conditioning comes from Qwen3-VL, and images use the FLUX.2 packed autoencoder ([`AutoencoderKLFlux2`]). -FLUX.3 ships as a modular pipeline. Prompting matters: the model responds best to full scene captions that describe the subject, composition, lighting, and atmosphere — see the [FLUX.3 prompting guide](https://docs.bfl.ai) for details. Get started with a text-to-image generation: +FLUX.3 ships as a modular pipeline. Prompting matters: the model responds best to full scene captions that describe the subject, composition, lighting, and atmosphere. Black Forest Labs provides a [prompt upsampler](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup) that expands short prompts automatically; see [Prompt upsampling](#prompt-upsampling). Get started with a text-to-image generation: ```python import torch @@ -42,9 +42,55 @@ image.save("fox.png") `prompt` is the only required input: the pipeline carries the recommended sampling settings (50 steps, guidance scale 3.5, scheduler shift 6.93) and generates 1024 x 1024 images by default. It takes one prompt per call; use `num_images_per_prompt` for several images of the same prompt. +## Prompt upsampling + +The prompt upsampler, [black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup), turns the user prompt, together with the reference images when editing, into a full scene caption. It is a separate model; run it with Transformers and pass its output as `prompt`: + +```python +import json + +import torch +from huggingface_hub import hf_hub_download +from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration + + +model_id = "black-forest-labs/flux-3-image-klein-pup" +processor = AutoProcessor.from_pretrained(model_id) +upsampler = Qwen3_5ForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto").eval() +system_prompts = json.load(open(hf_hub_download(model_id, "system_prompts.json"))) + + +def upsample(prompt, images=()): + if images: + text = ( + "Reference images follow in order (the first image, the second image, ...).\n" + f"User request: {prompt}\nOutput the caption." + ) + content = [{"type": "image", "image": image} for image in images] + [{"type": "text", "text": text}] + messages = [{"role": "system", "content": system_prompts["i2i"]}, {"role": "user", "content": content}] + else: + messages = [{"role": "system", "content": system_prompts["t2i"]}, {"role": "user", "content": prompt}] + inputs = processor.apply_chat_template( + messages, + add_generation_prompt=True, + tokenize=True, + return_dict=True, + return_tensors="pt", + enable_thinking=False, + ).to(upsampler.device) + with torch.no_grad(): + output = upsampler.generate(**inputs, max_new_tokens=3584, do_sample=False) + return processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True).strip() + + +image = pipe(prompt=upsample("An astronaut riding a horse"), output="images")[0] +``` + +For an edit, pass the same reference images to both: `pipe(prompt=upsample(instruction, [image]), image=image, ...)`. The upsampler also writes the [layout](#layout-control) captions from a list of boxes; see its model card. + ## Image-to-image -Pass one or more reference images with `image`. Every reference keeps its own resolution (center-cropped to the 16-pixel grid; references above 16 megapixels are downscaled first), and the generation size defaults to one megapixel at the first reference's aspect ratio. Describe the *target* image as a complete caption of the reference with your edit folded in, rather than a one-line instruction: +Pass one or more reference images with `image`. Every reference keeps its own resolution (center-cropped to the 16-pixel grid; references above 16 megapixels are downscaled first), and the generation size defaults to one megapixel at the first reference's aspect ratio. Describe the *target* image as a complete caption of the reference with your edit folded in, rather than a one-line instruction, or let the [prompt upsampler](#prompt-upsampling) write it from the instruction and the references: ```python night = pipe( From df5e606c26e80675e721dcef8b8a19774f89475d Mon Sep 17 00:00:00 2001 From: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com> Date: Thu, 15 Oct 2026 17:08:08 +0300 Subject: [PATCH 32/34] examples/flux3: support train_batch_size>1 and prior preservation with variable-length captions (#44) --- .../train_dreambooth_lora_flux3_klein.py | 42 ++++++++++++------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/examples/dreambooth/train_dreambooth_lora_flux3_klein.py b/examples/dreambooth/train_dreambooth_lora_flux3_klein.py index 9cc4b6a47d5d..cd1f4ba7798d 100644 --- a/examples/dreambooth/train_dreambooth_lora_flux3_klein.py +++ b/examples/dreambooth/train_dreambooth_lora_flux3_klein.py @@ -1639,6 +1639,31 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer, pad_to_length=None) text_ids = text_ids.unsqueeze(0).expand(prompt_embeds.shape[0], -1, -1) return prompt_embeds, text_ids + # Prompt embeddings get stacked across a batch (train_batch_size > 1) and concatenated with the class prompt + # (prior preservation). FLUX.3 pads each prompt to its own window, so stacking requires a shared window. Compute + # one dataset-wide window here and pad all custom captions (and the class prompt) to it. bs=1 without prior + # preservation never stacks, so it keeps each prompt's exact per-prompt window. + uniform_window = None + if train_dataset.custom_instance_prompts and (args.train_batch_size > 1 or args.with_prior_preservation): + windowed_prompts = list(train_dataset.custom_instance_prompts) + if args.with_prior_preservation: + windowed_prompts.append(args.class_prompt) + real_lengths = [ + len( + tokenizer.tokenizer( + tokenizer.apply_chat_template( + [{"role": "user", "content": p}], tokenize=False, add_generation_prompt=True + ), + truncation=True, + max_length=args.max_sequence_length, + )["input_ids"] + ) + for p in windowed_prompts + ] + uniform_window = min( + math.ceil(max(real_lengths) / TEXT_PAD_MULTIPLE) * TEXT_PAD_MULTIPLE, args.max_sequence_length + ) + # If no type of tuning is done on the text_encoder and custom instance prompts are NOT # provided (i.e. the --instance_prompt is used for all images), we encode the instance prompt once to avoid # the redundant encoding. @@ -1652,7 +1677,7 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer, pad_to_length=None) if args.with_prior_preservation: with offload_models(text_encoder, device=accelerator.device, offload=args.offload): class_prompt_hidden_states, class_text_ids = compute_text_embeddings( - args.class_prompt, text_encoder, tokenizer + args.class_prompt, text_encoder, tokenizer, pad_to_length=uniform_window ) # FLUX.3 validation runs through the modular pipeline (which encodes prompts itself), so we only assemble @@ -1735,12 +1760,12 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer, pad_to_length=None) if train_dataset.custom_instance_prompts: if args.fsdp_text_encoder: prompt_embeds, text_ids = compute_text_embeddings( - batch["instance_prompts"], text_encoder, tokenizer + batch["instance_prompts"], text_encoder, tokenizer, pad_to_length=uniform_window ) else: with offload_models(text_encoder, device=accelerator.device, offload=args.offload): prompt_embeds, text_ids = compute_text_embeddings( - batch["instance_prompts"], text_encoder, tokenizer + batch["instance_prompts"], text_encoder, tokenizer, pad_to_length=uniform_window ) for i, idx in enumerate(sample_indices): prompt_embeds_cache[idx] = prompt_embeds[i : i + 1] @@ -1758,17 +1783,6 @@ def compute_text_embeddings(prompt, text_encoder, tokenizer, pad_to_length=None) ) assert all(ids is not None for ids in text_ids_cache), "Text ID cache has unfilled entries." - # The training loop re-batches the cached per-caption embeddings with torch.cat, which needs a uniform - # sequence window across the batch. FLUX.3 pads each caption to its own window, and the bucket sampler - # reshuffles every epoch, so train_batch_size > 1 with variable-length captions would group mismatched - # windows and crash mid-training. Fail fast with a clear message instead. - if args.train_batch_size > 1 and len({embeds.shape[1] for embeds in prompt_embeds_cache}) > 1: - raise ValueError( - "train_batch_size > 1 with variable-length captions is not supported: FLUX.3 pads each caption to" - " its own window and the cached embeddings cannot be batched together. Use --train_batch_size=1," - " or a dataset whose captions all share a length." - ) - # Caption dropout swaps the empty-prompt embedding in for a dropped caption. FLUX.3 pads each prompt to its own # window (a multiple of TEXT_PAD_MULTIPLE) and the transformer attends to it, so the empty prompt must match the # dropped sample's window. Precompute it for every window in use here, while the text encoder is still loaded From e09ac3662ab9d26707beafd8d82c766597f47bc6 Mon Sep 17 00:00:00 2001 From: sayakpaul Date: Thu, 15 Oct 2026 21:03:21 +0530 Subject: [PATCH 33/34] doc --- docs/source/en/api/loaders/lora.md | 4 ++++ src/diffusers/modular_pipelines/flux3/denoise.py | 4 +++- src/diffusers/modular_pipelines/flux3/encoders.py | 4 +++- 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/source/en/api/loaders/lora.md b/docs/source/en/api/loaders/lora.md index 4722d082822c..4c8eb20b299a 100644 --- a/docs/source/en/api/loaders/lora.md +++ b/docs/source/en/api/loaders/lora.md @@ -68,6 +68,10 @@ LoRA is a fast and lightweight training method that inserts and trains a signifi [[autodoc]] loaders.lora_pipeline.Flux2LoraLoaderMixin +## Flux3LoraLoaderMixin + +[[autodoc]] loaders.lora_pipeline.Flux3LoraLoaderMixin + ## ErnieImageLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.ErnieImageLoraLoaderMixin diff --git a/src/diffusers/modular_pipelines/flux3/denoise.py b/src/diffusers/modular_pipelines/flux3/denoise.py index d31310d664b5..17aea9c0a03d 100644 --- a/src/diffusers/modular_pipelines/flux3/denoise.py +++ b/src/diffusers/modular_pipelines/flux3/denoise.py @@ -233,7 +233,9 @@ def intermediate_outputs(self) -> list[OutputParam]: return [OutputParam("latents", type_hint=torch.Tensor, description="The denoised latents")] @torch.no_grad() - def __call__(self, components: Flux3ModularPipeline, block_state: BlockState, i: int, t: torch.Tensor): + def __call__( + self, components: Flux3ModularPipeline, block_state: BlockState, i: int, t: torch.Tensor + ) -> tuple[Flux3ModularPipeline, BlockState]: latents_dtype = block_state.latents.dtype block_state.latents = components.scheduler.step( block_state.noise_pred, diff --git a/src/diffusers/modular_pipelines/flux3/encoders.py b/src/diffusers/modular_pipelines/flux3/encoders.py index b336233254ca..18c6a454a63f 100644 --- a/src/diffusers/modular_pipelines/flux3/encoders.py +++ b/src/diffusers/modular_pipelines/flux3/encoders.py @@ -73,7 +73,9 @@ def intermediate_outputs(self) -> list[OutputParam]: ] @torch.no_grad() - def __call__(self, components: Flux3ModularPipeline, state: PipelineState): + def __call__( + self, components: Flux3ModularPipeline, state: PipelineState + ) -> tuple[Flux3ModularPipeline, PipelineState]: block_state = self.get_block_state(state) images = block_state.image if isinstance(block_state.image, list) else [block_state.image] From ec288418a5efa32e9fd49eb7b193babce920a106 Mon Sep 17 00:00:00 2001 From: sayakpaul Date: Thu, 15 Oct 2026 21:13:36 +0530 Subject: [PATCH 34/34] autodocstrings. --- src/diffusers/modular_pipelines/flux3/denoise.py | 2 ++ .../modular_pipelines/flux3/modular_blocks_flux3.py | 8 ++++++++ 2 files changed, 10 insertions(+) diff --git a/src/diffusers/modular_pipelines/flux3/denoise.py b/src/diffusers/modular_pipelines/flux3/denoise.py index 17aea9c0a03d..ed91f8569cb5 100644 --- a/src/diffusers/modular_pipelines/flux3/denoise.py +++ b/src/diffusers/modular_pipelines/flux3/denoise.py @@ -333,6 +333,8 @@ class Flux3DenoiseStep(Flux3DenoiseLoopWrapper): The image latent tokens to denoise. Shape: (B, seq_len, C) **denoiser_input_fields (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + attention_kwargs (`dict`, *optional*): + Additional kwargs for attention processors. Outputs: latents (`Tensor`): diff --git a/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py b/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py index 2ef1b48c3596..2c7889a5cda3 100644 --- a/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py +++ b/src/diffusers/modular_pipelines/flux3/modular_blocks_flux3.py @@ -151,6 +151,8 @@ class Flux3CoreDenoiseStep(SequentialPipelineBlocks): Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) **denoiser_input_fields (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + attention_kwargs (`dict`, *optional*): + Additional kwargs for attention processors. Outputs: latents (`Tensor`): @@ -223,6 +225,8 @@ class Flux3ImageConditionedCoreDenoiseStep(SequentialPipelineBlocks): Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) **denoiser_input_fields (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + attention_kwargs (`dict`, *optional*): + Additional kwargs for attention processors. Outputs: latents (`Tensor`): @@ -285,6 +289,8 @@ class Flux3AutoCoreDenoiseStep(AutoPipelineBlocks): Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) **denoiser_input_fields (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + attention_kwargs (`dict`, *optional*): + Additional kwargs for attention processors. Outputs: latents (`Tensor`): @@ -368,6 +374,8 @@ class Flux3AutoBlocks(SequentialPipelineBlocks): Custom sigmas for the denoising process. (overrides `num_inference_steps` when provided) **denoiser_input_fields (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc. + attention_kwargs (`dict`, *optional*): + Additional kwargs for attention processors. output_type (`str`, *optional*, defaults to pil): Output format: 'pil', 'np', 'pt'.