Skip to content

feat(en): expand Unicode symbol normalization - #490

Open
HungerDeng wants to merge 2 commits into
NVIDIA:mainfrom
HungerDeng:feat/en-unicode-symbol-normalization
Open

HungerDeng wants to merge 2 commits into
NVIDIA:mainfrom
HungerDeng:feat/en-unicode-symbol-normalization

Conversation

@HungerDeng

@HungerDeng HungerDeng commented Sep 15, 2026

Copy link
Copy Markdown

Summary

Expand English text normalization for common Unicode math operators, currency symbols, micro-sign units, and general symbols, referencing the text replacement of the https://github.com/espeak-ng/espeak-ng/blob/master/dictsource/en_emoji

Changes

  • Add mathematical operators and comparisons, including , , , and .
  • Add currency symbols and micro-sign unit aliases under their semantic grammars.
  • Add general arrows, typographic symbols, and music notation to the whitelist.
  • Add shared English fixtures used by both pytest and Sparrowhawk normalization tests.

Testing

  • PASS: python -m pytest -q tests/nemo_text_processing/en/test_math.py tests/nemo_text_processing/en/test_measure.py tests/nemo_text_processing/en/test_money.py tests/nemo_text_processing/en/test_whitelist.py --cpu --use_local_test_data --tn_cache_dir=/private/tmp/nemo_tn_cache (492 passed).
  • Full explicit pytest collection found 1,874 tests but was interrupted after 10:34 during global grammar construction; no failures were reported before interruption.
  • Sparrowhawk grammar export completed and produced the English FAR files. The Docker-backed Sparrowhawk test could not run because this host has no Docker daemon; the macOS Bash 3.2 script also reports unsupported ${VAR,,} syntax.

Notes for Reviewers

The distinct Unicode multiplication aliases (×, *, ·, , , and ) intentionally remain mapped to times in math_operation.tsv; they are different input code points.

Expand English normalization coverage for mathematical operators, currency signs, micro units, and common standalone symbols. Categorize each symbol under the grammar that matches its meaning.

Signed-off-by: hungerdeng <hungerdeng@gmail.com>
Exercise the added math operators, micro-sign units, currency symbols, and whitelist symbols in the shared English fixtures used by pytest and Sparrowhawk.

Signed-off-by: hungerdeng <hungerdeng@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant