Skip to content

Add Tamil (ta) text normalization and inverse text normalization support - #495

Open
veera-puthiran-14082 wants to merge 1 commit into
NVIDIA:mainfrom
veera-puthiran-14082:ta-support
Open

veera-puthiran-14082 wants to merge 1 commit into
NVIDIA:mainfrom
veera-puthiran-14082:ta-support

Conversation

@veera-puthiran-14082

Copy link
Copy Markdown

What does this PR do ?

Add Tamil (ta) text normalization and inverse text normalization support.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

Summary

This PR adds --lang=ta support to both normalize.py and inverse_normalize.py, covering cardinal,
ordinal, decimal, fraction, date, time, money, measure, telephone, whitelist, punctuation and word for
both directions, plus electronic, roman, serial and range for TN only (Tamil has no ITN reading for those
four). ITN inverts the TN cardinal grammar directly rather than duplicating the number morphology, so the
two directions cannot drift apart; this mirrors the existing precedent of hi ITN importing TN utilities,
extended here to a full tagger.

15 pytest modules were added under tests/nemo_text_processing/ta/ (523 TN + 289 ITN golden cases, all
passing), along with both Sparrowhawk shell scripts. The existing hi suite (1111 cases) still passes
unchanged.

Notes for reviewers

  • Sparrowhawk tests are written but not executedbash -n confirms both scripts parse, but running
    them needs NVIDIA's Sparrowhawk container, which isn't available in the environment this was built in.
  • Commits are not yet signed off (git commit -s) — will add before merge if required, flagging here
    since it changes the commit hash and needs a force-push.
  • No change to the NeMo documentation repo (NVIDIA/NeMo) is included; that's a separate repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant