Skip to content

refactor(audio): split Sortformer and Silero VAD into stage adapters - #2455

Draft
mohammadaaftabv wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
mohammadaaftabv:agent/sortformer-silero-adapters
Draft

mohammadaaftabv wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
mohammadaaftabv:agent/sortformer-silero-adapters

Conversation

@mohammadaaftabv

@mohammadaaftabv mohammadaaftabv commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR moves Sortformer diarization and Silero VAD to the shared stage/adapter architecture already used by ASR and SED.

  • Adds canonical DiarizationAdapter and VADAdapter contracts.
  • Keeps task I/O, normalization, fan-out/result assembly, errors, sidecars, and resume behavior in the stages.
  • Moves model acquisition, worker-local lifecycle, preprocessing, and inference into reusable model adapters.
  • Adds the NeMo Sortformer adapter and the Silero TorchScript/ONNX adapter.
  • Reuses AdapterInferenceStage for node prefetch, worker setup, GPU allocation, and teardown.
  • Adds bounded long-audio Sortformer preprocessing and safe RTTM output handling.
  • Makes VAD adapter and worker settings YAML-selectable in AudioDataFilterStage.

TensorRT Sortformer and Silero support is intentionally split into follow-up draft #2457.

Compatibility

  • Preserves the existing Sortformer import path through a compatibility re-export.
  • Preserves current-main constructor ordering and legacy Sortformer fields.
  • Retains Silero fan-out and nested-output behavior, with optional audit placeholders for empty/error rows.
  • Uses lazy public imports so Silero and diarization do not eagerly load each other's optional dependencies.

Documentation scope

The in-repository audio developer READMEs and tutorials are updated for the adapter boundary and output contract. This PR contains no Fern changes.

Validation

  • Affected audio suite: 190 passed, 1 warning.
  • Focused post-split regression suite: 80 passed.
  • Ruff 0.14.10 check and format validation passed for all changed Python files.
  • git diff --check passed.
  • GitHub pre-commit, Ruff, secrets-detector, and DCO checks pass.
  • Fern diff against main: zero files.

Local parity

A frozen local target/reference experiment ran the PR head (3aa1b4a1cab24d83710f70e4a8fc0779339d4650) against nithinraok/Curator:nkoluguri/integration-test (1e2d639946f28dfd289a0a9456480385d4480c28) on one RTX 3080 Ti. Both arms used the same two original whole-recording AMI WAVs, model, dependency environment, resource contract, and comparator.

  • Target execution: 2a773e80-00f1-4bab-b5c6-edb0e7a186f1 (SUCCEEDED).
  • Reference execution: fda2fef9-6d9a-4292-bfd0-68c7ff4fd9d3 (SUCCEEDED).
  • Comparator: accepted; exact sample identities; no duplicates, errors, empty rows, or forbidden serialized state.
  • Output: 252 JSONL rows per arm, with the same shard-content hash multiset.
  • VAD segments: 92 and 160; maximum boundary delta 0.0 ms; total-speech delta 0.0 ms.
  • Diarization: maximum boundary delta 0.0 ms; pseudo-DER 0.0; speaker cardinality 4/4 for both recordings.

Earlier environment-only failures are preserved in the local evidence. The successful replacement pair corrected the Ray/NVML harness without changing either Curator source tree, feature configuration, model, inputs, backend, hardware, comparator, or acceptance thresholds.

Signed-off-by: V Mohammad Aaftab <aaftaabv@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: V Mohammad Aaftab <aaftaabv@gmail.com>
Signed-off-by: V Mohammad Aaftab <aaftaabv@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant