AV latent support for LTXVLoopingSampler and LTXVExtendSampler

[Jean J. de Jong (jjdejong)]

Summary

• Add audio-visual latent support to LTXVLoopingSampler and LTXVExtendSampler (and the LTXVBaseSampler / LTXVInContextSampler building blocks), removing the ValueError guard that previously rejected AV latents.
• Audio is carried jointly with video through the temporal tiling loop — separated on input, sliced/extended per tile alongside the video overlap geometry, accumulated across tiles, and reassembled into an AV NestedTensor on output.
• For stage-2 refinement (low-sigma upscale), the input audio data initializes each tile's audio frames instead of zeros, so lipsync and AV coherence are refined jointly with video — matching the behavior of the single-pass SamplerCustomAdvanced workflow.
• Adds example workflows + docs for two-pass I2V looping (single-tile and 30s 3-tile variants with MultiPromptProvider) and a V2V detailer doc.

Implementation notes

• New helpers _make_av_latent_dict() / _split_av_latent_dict() in easy_samplers.py handle NestedTensor packing/unpacking with proper noise-mask propagation for both modalities.
• Audio temporal compression differs from video; frame_overlap (expressed in video latent frames) is converted via the audio VAE stride before audio slicing.
• Backward compatible: with no audio component in the input latent, both samplers behave exactly as before.

Motivation

Previously, the looping and extend samplers raised ValueError: LoopingSampler currently does not support Audio Visual latents., forcing users to generate audio in a separate low-sigma pass on top of a video-only result. That workaround produces inferior lipsync because audio is never refined jointly with video across temporal tiles. This change makes joint AV generation possible in long-form clips.

Testing status

• ✅ LTXVLoopingSampler AV path — confirmed working end-to-end. Audio is generated jointly with video and stays synchronised across tile boundaries; verified with the included two-pass workflow.
• ⚠️ LTXVExtendSampler AV path — implemented but not fully validated. The extend pass runs without errors when handed a source AV latent, but joint AV continuity across the extend boundary has not been rigorously compared against the workaround pipeline.
• ⚠️ optional_negative_index_strength — wired through all samplers but not extensively tested. The default (1.0) preserves prior behavior; intermediate values to soften reference-image influence have not been validated on a reference set.

Test plan

• Run LTX-2.3_Two_Pass_I2V_Looping.json with an AV latent and confirm decoded audio is in sync with video.
• Run LTX-2.3_Two_Pass_I2V_Looping_30s.json (3 tiles) and confirm audio continuity across both tile boundaries.
• Run a video-extension workflow with a source AV latent through LTXVExtendSampler and confirm audio continuation is coherent across the overlap region.
• Sweep optional_negative_index_strength across {0.0, 0.5, 1.0} on a reference-conditioned generation to confirm monotonic influence reduction.
• Confirm video-only inputs still produce identical output (no regression for the non-AV path).

🤖 Generated with Claude Code

[Jean J. de Jong (jjdejong)]

Friendly ping on this PR 🙂 — it's been open since late April and I'd love to get it reviewed whenever someone has bandwidth. Happy to address any changes, and if there's a preferred contribution/review process I should follow, just let me know.

One scope heads-up: this branch also includes a small, general STG bug fix — a crash and an attention-index miscount that occur when cond-image guides with strength != 1.0 are combined with STG perturbation (triggered by the guide-mask self-attention split introduced in CORE-166). Since that fix isn't AV-specific and helps any STG + cond-image workflow, I've extracted it into a standalone PR for easier, independent review: #503. Happy to drop it from this PR if you'd prefer to keep the two from overlapping.

Thanks!