Skip to content

feat: add source-aligned text normalization APIs and CLI - #88

Open
HungerDeng wants to merge 3 commits into
FluidInference:mainfrom
HungerDeng:feat/source-aligned-tn
Open

HungerDeng wants to merge 3 commits into
FluidInference:mainfrom
HungerDeng:feat/source-aligned-tn

Conversation

@HungerDeng

Copy link
Copy Markdown

Summary

Add source-aligned text normalization for compiled NeMo FST grammars and expose it across the native and Swift interfaces. Callers now receive the complete normalized sentence together with source ranges, original text, normalized span text, and semantic classes, enabling downstream processing without string-diff heuristics.

Changes

  • Preserve input progress through the selected classifier FST path and return aligned spans with stable UTF-8 byte offsets.
  • Add Rust APIs for English, French, Spanish, German, Chinese, Hindi, and Japanese.
  • Expose the alignment through the C ABI, Swift wrapper, and WASM binding.
  • Add a SwiftPM nemo-tn-aligned JSON/JSONL CLI.
  • Select the host Rust archive in SwiftPM so the existing and aligned CLIs link statically on Apple Silicon and Intel Macs.
  • Document the API, build flow, and CLI output.

Testing

  • cargo test --features fst-engine
  • cargo check --locked --features ffi,fst-engine
  • cargo fmt --all --check
  • cargo clippy --all-targets --features ffi,fst-engine
  • swift build and swift build --product nemo-tn-aligned
  • CLI smoke tests for sentence arguments and JSONL stdin
  • git diff --check

Notes for Reviewers

The per-span normalized value is direct verbalizer output; sentence-level spacing and punctuation cleanup are represented by the top-level normalized field. Offsets are half-open UTF-8 byte ranges into the original input.

Expose source spans and semantic classes alongside compiled NeMo TN output across Rust, C, Swift, and WASM. Preserve selected FST input progress so callers can map normalized tokens without string diffing.
Provide a macOS SwiftPM executable that emits aligned TN results as JSON or JSONL. Select the host Rust archive and link it statically so the existing and aligned CLIs build on both Apple Silicon and Intel Macs.
Verify full-sentence mappings preserve semantic token kinds and UTF-8 byte ranges across English and Spanish inputs. Exercise currency, email, date, phone, unsupported-language behavior, and C FFI cleanup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant