feat: add source-aligned text normalization APIs and CLI - #88
Open
HungerDeng wants to merge 3 commits into
Open
HungerDeng wants to merge 3 commits into
HungerDeng wants to merge 3 commits into
Conversation
Expose source spans and semantic classes alongside compiled NeMo TN output across Rust, C, Swift, and WASM. Preserve selected FST input progress so callers can map normalized tokens without string diffing.
Provide a macOS SwiftPM executable that emits aligned TN results as JSON or JSONL. Select the host Rust archive and link it statically so the existing and aligned CLIs build on both Apple Silicon and Intel Macs.
Verify full-sentence mappings preserve semantic token kinds and UTF-8 byte ranges across English and Spanish inputs. Exercise currency, email, date, phone, unsupported-language behavior, and C FFI cleanup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add source-aligned text normalization for compiled NeMo FST grammars and expose it across the native and Swift interfaces. Callers now receive the complete normalized sentence together with source ranges, original text, normalized span text, and semantic classes, enabling downstream processing without string-diff heuristics.
Changes
nemo-tn-alignedJSON/JSONL CLI.Testing
cargo test --features fst-enginecargo check --locked --features ffi,fst-enginecargo fmt --all --checkcargo clippy --all-targets --features ffi,fst-engineswift buildandswift build --product nemo-tn-alignedgit diff --checkNotes for Reviewers
The per-span
normalizedvalue is direct verbalizer output; sentence-level spacing and punctuation cleanup are represented by the top-levelnormalizedfield. Offsets are half-open UTF-8 byte ranges into the original input.