diff --git a/docs/serviceTemplates/README.md b/docs/serviceTemplates/README.md index da37c18b8..7921c147f 100644 --- a/docs/serviceTemplates/README.md +++ b/docs/serviceTemplates/README.md @@ -28,9 +28,32 @@ Templates are validated against `ServiceTemplateSchema` | `image` + exactly one of `tag` / `checksum` / `dockerfile` | Image spec (`dockerfile` triggers a build and is gated per daemon by `allowImageBuild`) | | `exposedPorts` | Container ports forwarded to host ports and returned as service endpoints | | `command` / `entrypoint` | Docker CMD / ENTRYPOINT overrides | +| `commandFile` | Path to a script, resolved relative to this directory and inlined as `command[0]` at load time (mutually exclusive with `command`) | | `envVars` | Fixed operator-set env vars (values never returned to callers) | -| `userConfigurableEnvVars` | Env vars the consumer supplies via ECIES-encrypted `userData` (optional regex `validation`, `sensitive` UI hint) | +| `userConfigurableEnvVars` | Env vars the consumer supplies via ECIES-encrypted `userData` (optional regex `validation`, `sensitive` UI hint, `required` hint — advisory, never enforced node-side) | | `requiredResources` / `recommendedResources` | Gate/score environment selection (`min` is enforced at `SERVICE_START`) | +| `workflows` | Selectable graphs for UI-driven templates; each entry is `id` / `name` / `file` (a path to the graph JSON, resolved relative to this directory and inlined into `graph` at load time) | + +### Catalogue metadata (services vs bundles) + +Optional, purely descriptive fields. They change nothing about how the container runs — +they travel to clients through the `getServiceTemplates` sanitizer and only affect how the +entry is presented in a catalogue. A template that omits all of them is a plain **service**. + +| Field | Meaning | +| ---------- | ---------------------------------------------------------------------------------------------------------------- | +| `kind` | `service` (bare app, the default when absent) or `bundle` (the same app whose command pre-downloads a curated model set) | +| `service` | Bundles only, **required on them**: `id` of the service template this is a variant of (may not exist on this node) | +| `outcome` | Bundles only: the one concrete thing this gets done, one sentence | +| `category` | One of `image` / `video` / `llm` / `serving` / `notebook` / `embeddings` / `app` | +| `includes` | Bundles only: manifest of what the command downloads — `name`, `kind` (`model` / `workflow` / `customnode` / `other`), optional `sizeGb`, `repoId`, `url`. Display metadata; nothing here is fetched or verified by the node | + +Keep `includes` in step with what the script actually downloads: clients use its length as +the denominator for a "preparing models — N of M" progress line, driven by `[models]` +markers on the container's stdout (`downloading ` / `ready: ` / +`already present: ` / `WARNING: could not download ` / `bundle complete`). The +markers are a convention, not a protocol — a script that prints none simply shows no +progress. ## Templates in this folder @@ -101,6 +124,331 @@ so no `command` override is needed. Bundles ComfyUI-Manager for installing check custom nodes from the UI; `HF_TOKEN` / `CIVITAI_TOKEN` are optional user env vars for gated downloads. ~10 GB VRAM for SDXL. +### `ltx-video-ugc-product.json` — ComfyUI, LTX-2.3 product video (GPU) + +A product photo becomes a 9:16 vertical clip (720×1280, 5 s) with camera motion and ambient +audio, using the fixed workflow `workflows/ocean_ugc_product.json`. Needs a CUDA GPU with +48 GB+ VRAM. + +Pick a persistent-storage bucket: it holds ComfyUI's whole base directory, so the ~38 GiB +weight set downloads only on the first launch, and clips are written to the bucket root where +the storage API's `listFiles` (top-level files only) can see them. Without a bucket everything +goes to `/tmp` and is lost on stop. + +### `ltx-video-ugc-multishot.json` — ComfyUI, LTX-2.3 multishot UGC reel (GPU) + +Same image, bucket behavior and **the same generator** as the product template — both +workflows embed the identical `Image to Video (LTX-2.3)` subgraph (22B dev fp8 + distilled +LoRA, base pass at 360×640, `LTXVLatentUpsampler`, then a refine pass at 720×1280), so shot +quality matches. It builds a vertical reel one 720×1280 / 5 s shot at a time. Ships two +workflows: + +- `workflows/ocean_ugc_multishot.json` — renders one shot per Run. +- `workflows/ocean_ugc_assemble.json` — concatenates rendered shots into one reel. + +Three inputs, following what the reference-image tools converged on: a **first frame** +(optional — a character photo; with none, flip *No character image* and the style prompt +builds the character instead), a **style box** holding everything that must not change +between shots (who the character is, wardrobe, room, lighting, palette, camera look), and a +**scene box** holding only what does (action, camera move, dialogue, ambient sound). Keeping +the style box byte-identical across runs is what holds the reel together — so it is a +separate box from the scene text rather than something you retype and drift. + +Click **Run** for a clip, edit the scene box, Run again. Clips save to the bucket root with +ComfyUI's own save counter — `shot_00001…`, `shot_00002…` — so re-takes never overwrite and +you pick the good ones at assembly time. + +**Frame chaining.** Every run also drops that clip's last frame beside it as +`lastframe_…png`. Refresh, pick it in the image slot, and the next shot starts exactly where +the last one ended — identity carried in pixels rather than in prose, which is what the +reference-image tools rely on and what text-only character descriptions cannot do. + +**Hard cuts.** The *Shot start* dial controls how literally that image is taken. It drives +the `strength` of both `LTXVImgToVideoInplace` passes, promoted out of the subgraph: the +refine pass directly, the base pass through `a * 0.7` so the dial at **1.0** reproduces the +strengths the graph shipped with (1.0 refine / 0.7 base) exactly. At **1.0** frame 0 *is* the +image and the take continues seamlessly. Around **0.35** the image only guides look and +character while the model composes a new frame — a hard cut to a new scene with the same +face and wardrobe. The separate *No character image* toggle still bypasses the image +entirely (`Switch to Text to Video?` inside the subgraph) for a shot built from the prompt +alone. + +Three ways to start a shot, then: continue it, cut to it, or generate it fresh. + +Mechanically: the generator subgraph exposes its decoded frames as a second `IMAGE` output +alongside `VIDEO`; `GetImageSize` → `a - 1` → `ImageFromBatch` takes the final frame and +`SaveImage` writes it. It costs one PNG per run and needs no interaction — and because the +bootstrap points ComfyUI's `--input-directory` and `--output-directory` at the same bucket +root, the PNG shows up in `LoadImage`'s dropdown with no copying. + +Once every shot looks good, switch to the **Assemble reel** workflow, pick the clips in its +`LoadVideo` slots, bypass the slots you don't need, and Run — it concatenates them into one +clip inside ComfyUI, no external editor needed. + +**Voice.** LTX builds each clip's audio from a fresh `LTXVEmptyLatentAudio` conditioned only +on the text — there is no speaker embedding, so a spoken line is a different person on every +cut and no amount of prompting locks it. Frame chaining carries the face; nothing carries the +voice. + +Assemble concatenates the clips' own audio, which keeps ambience and picture in sync, then +runs the stitched track through **voice conversion** — the same thing dubbing pipelines do. +`UnifiedVoiceChangerNode` remaps every line to one reference speaker while leaving the words, +their timing and the room tone alone. That is why conversion beats dubbing a voiceover over +the footage: the phonemes never move, so ambience survives and the lips still match. Upload a +few seconds of clean speech as the reference voice; `refinement_passes` 1 is usually right +(above ~5 it distorts) and `max_chunk_duration` bounds VRAM, rejoining chunks with +timing-preserving concatenation. + +Scripting still helps and costs nothing: dialogue in one or two shots, action and ambience in +the rest, means fewer conversions to get right. + +If the voice nodes load red, the pack failed to install — delete both and wire the last +`AudioConcat` straight into `CreateVideo`, which is the plain stitched audio the workflow +used before. The node's own how-to says so too. + +### `minimax-h3-video-ugc-multishot.json` — ComfyUI, MiniMax H3 UGC reel (GPU) + +Same image and bucket behavior as the LTX templates, but a different generator — MiniMax H3 +(open-weights, int8) — and a different unit of work: **one Run produces ~30 s with six shots and +one consistent character**, versus LTX's one continuous take per Run. Needs a CUDA GPU; see +**Hardware** below — it is built for one 80 GB+ card (H200/H100 class), which holds the whole +chain resident, and 24 GB still runs it at a lower resolution. Ships two workflows: + +- `workflows/ocean_h3_ugc_multishot.json` — one Run, two chained beats, ~30 s out. +- `workflows/ocean_h3_ugc_assemble.json` — concatenates up to 8 rendered clips into one reel. + +**Native audio, no voice-conversion pass.** H3 generates voice, room tone and music in the +same forward pass as the video, so there is no separate TTS step and nothing analogous to the +LTX templates' `UnifiedVoiceChangerNode` pass. The bootstrap's TTS-Audio-Suite install is gated +on the installed graph containing `UnifiedVoiceChangerNode`; this workflow never does, so the +install never runs for this template — no custom node pack rides along. + +**Two chained stages, not two Runs.** H3 caps at 15 s per generation, so 30 s needs two passes. +The graph chains them. Beat 1 is **FL2VA**, the only mode whose `first_frame` literally becomes +frame 0, so the user's photo opens the video. Beat 2 is **Ref2VA**, the only mode that takes a +reference *video* — which is what carries face, wardrobe, room, camera style and voice into the +second half. Chaining beats two manual Runs because `ref_video_0` is typed `IMAGE` and +`ref_video_audio_0` is `AUDIO`: beat 1's `VAEDecode` / `VAEDecodeAudio` feed beat 2 **directly**, +with no file, no re-encode and strictly higher fidelity than referencing a saved MP4. ComfyUI +caches by input hash, so once beat 1 is good, editing only beat 2's text leaves beat 1's subtree +untouched and it is not recomputed. + +H3 truncates an over-long reference batch **from the start** (`frames[:frame_count]`), so a raw +pass-through would reference beat 1's opening rather than its end. Beat 2 slices the tail +explicitly: `GetImageSize` → `ComfyMathExpression("a - 72")` → `ImageFromBatch(length 72)` for +the frames, and `TrimAudioDuration(start_index -3, duration 3)` for the audio. 72 frames is 3 s +at 24 fps, inside H3's documented 2–15 s reference window and far cheaper than referencing all +362 — reference tokens ride every sampling step. Both values are derived rather than pinned +(the frame index from the actual batch size, the audio offset from the end of the clip), so +changing the duration widget cannot break the slice. + +**The seam is a hard cut.** 15 s is the model's ceiling per pass and no wiring changes that. +Beat 2 is a new generation that *references* beat 1, not a continuation of the same take, so the +join at 00:15 is a hard cut — same person, same room, same voice, new camera setup. The how-to +note says to write beat 2 as a deliberate cut, which reads as editing rather than as a glitch. + +**One boolean, and it really does skip beat 2.** `Second beat — 30 s total` drives two +`ComfySwitchNode`s (video and audio) plus the last-frame picker. `ComfySwitchNode` is **lazy**, +so `false` means the `ImageBatch` / `AudioConcat` branch never evaluates and **beat 2 never runs +at all** — no ref2va checkpoint loaded, no second sampling pass. That is the fast-iteration +path: a 15 s beat in about half the time. The `SaveImage` last-frame output hangs off the +*switched* stream for the same reason; wiring it straight to beat 2 would make it an output +dependency and force the second pass even with the boolean off. + +**Four prompt boxes, one per contract.** H3 mandates this section order: + + subject_definitions → summary → retention_analysis → + detailed_description → overall_soundscape → non_diegetic_music + +but the two modes do not take the same sections, so the boxes split by *stage* as well as by +set-once/per-beat: + +| Box | When | Sections | Used by | +|---|---|---|---| +| **A · Cast & references** | set once | `subject_definitions` | beat 2 | +| **B · Beat 1** | per beat | `integrated_multimodal_description` | beat 1 | +| **C · Beat 2** | per beat | `summary`, `retention_analysis`, `detailed_description` | beat 2 | +| **D · Audio & music** | set once | `overall_soundscape`, `non_diegetic_music` | both | + +Two `StringFormat` nodes join them, one per stage, each a pure join because every section label +is already written literally in its box. Beat 1's `f_string` bakes in the base guide's fixed +I2VA instruction line (`For the target video, at 0.00 seconds …`), which is constant and so gets +no box, then `{a}` = B and `{b}` = D. Beat 2's is `{a}\n\n{b}\n\n{c}` = A, C, D, which lands all +six sections in the mandated order. `retention_analysis` is the section actually holding the +character's face steady; it names the chained tail as `