Extend Swedish TN by jimregan · Pull Request #468 · NVIDIA/NeMo-text-processing

jimregan · 2026-08-07T16:12:40Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

This incorporates additional items imported from Sardin, the open source version of the internal phonetisation system from MTM (Myndigheten för tillgängliga medier/Swedish Agency for Accessible Media)

Before your PR is "Ready for review"

Pre checks:

PR Type:

New Feature
Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

Signed-off-by: Jim O'Regan <joregan@kth.se>

Signed-off-by: Jim O'Regan <joregan@kth.se> Signed-off-by: "Christina Tånnander" <christina.tannander@mtm.se>

Signed-off-by: Jim O'Regan <joregan@kth.se>

for more information, see https://pre-commit.ci

Signed-off-by: Jim O'Regan <joregan@kth.se>

for more information, see https://pre-commit.ci

Signed-off-by: Jim O'Regan <joregan@kth.se>

…ssing into swedish-updates Signed-off-by: Jim O'Regan <joregan@kth.se>

github-actions · 2026-08-22T01:59:54Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

Signed-off-by: Jim O'Regan <joregan@kth.se>

tbartley94 · 2026-09-11T20:07:16Z

@jimregan Thanks for the contributions. Will be reviewing over next week, just sourcing language speakers to evaluate the test cases (you'll see additional reviewer beyond myself, they'll be going over the test cases).

j-l-thorsson · 2026-09-18T20:55:47Z

The spoken expansion looks correct. Can we also cover the standard spelling e.Kr. (capital K and final period) if it isn't tested already?

j-l-thorsson · 2026-09-18T21:17:33Z

+~kl. 3
+klockan tre
+klass tre
+~3 kl.


It may be worth considering an extra test case here. "3:e kl." is commonly used similarly to 3 kl.

j-l-thorsson · 2026-09-18T21:25:37Z

 cwt	centner
 da	atommassenhet
 db	decibel
+dba	decibel


Is dropping "A-vägd" for the trailing "a" intentional?

It was inherited from Sardin (the open source version of MTM's normalisation system), fixed now

j-l-thorsson

Reviewed the Swedish test cases for language correctness, with TTS in mind. Left a few comments on written conventions and additional examples.

Signed-off-by: Jim O'Regan <joregan@kth.se>

tbartley94 · 2026-09-25T19:22:25Z

@jimregan doing technical review aspect, please let me know if any conflict with ongoing language changes.

tbartley94

Stylistic changes and implementation discussion. Need use of aliases instead of relying on f-string formatting in order to aid maintenance.

tbartley94 · 2026-09-25T19:24:08Z

@@ -11,8 +11,8 @@ mån	måndag
 mån	månad
 ons	ons
 ons	onsdag
-AB	AB


why wouldn't this be in abbreviations alternatives?

also can include the upper case variations.

tbartley94 · 2026-09-25T20:41:58Z

@@ -22,4 +22,3 @@
 Χ	chi


Honestly would just generate the unicode codepoints instead of adding file churn. Ditto for lower.

It's mapping the characters to the Swedish version of their names. I don't see how generating unicode codepoints can achieve that.

tbartley94 · 2026-09-25T20:44:41Z

@@ -9,6 +9,9 @@ celsiusgrad	celsiusgrader
 centimeter


sanity confirm: aware this transforms everything in the file to an fst mapping?

tbartley94 · 2026-09-25T20:45:16Z

 ₧	peseta
 ptas	peseta
 usd	amerikansk dollar
+us$	amerikansk dollar


hmm, swedes add spacing at all after the currency or firmly non-delineated?

tbartley94 · 2026-09-25T21:17:52Z

        optional_minus_graph = pynini.closure(pynutil.insert("negative: ") + pynini.cross("-", "\"true\" "), 0, 1)

        final_graph = optional_minus_graph + pynutil.insert("integer: \"") + self.graph + pynutil.insert("\"")
+        reference_graph = pynini.Fst()


rationale for building an empty fst? backend autocasts

tbartley94 · 2026-09-25T21:20:41Z


        graph_unit = pynini.string_file(get_abs_path("data/measure/unit.tsv"))
+        graph_unit_optional_dot = pynini.string_file(get_abs_path("data/measure/unit_optional_dot.tsv"))
+        graph_unit |= graph_unit_optional_dot + pynini.closure(pynutil.delete("."), 0, 1)


wouldn't this just be two instances of the optional dot?

I've renamed that to 'graph_optionally_dotted_units'

tbartley94 · 2026-09-25T21:21:32Z

        final_graph = (graph_integer_only + optional_delete_fractional_zeros) | graph_decimal

+        currency_quantity = pynini.string_file(get_abs_path("data/money/currency_quantity.tsv"))
+        amount = pynini.closure(NEMO_DIGIT | pynini.union(",", ".", "-", " "), 1)


I'd just alias the strings as "PUNCTUATION" and NEMO_SPACE in the graph_utils. Avoids editing errors.

tbartley94 · 2026-09-25T21:22:09Z

        self.bare_ordinals = cleaned_graph
-        kapitlet_word = pynini.union("kapitlet", pynini.cross("kap", "kapitlet"))
-        kapitlet = cleaned_graph + NEMO_SPACE + kapitlet_word
+        reference_graph = pynini.Fst()


curious about the reference_graph decision

tbartley94 · 2026-09-25T21:22:38Z

+                    optional_dot = pynini.closure(pynutil.delete("."), 0, 1) if written.isalpha() else pynini.accep("")
+                    unit = pynutil.delete(written) + optional_dot
+                    reference = reference_number + delete_space + unit
+                    reference_graph |= reference + pynutil.insert(f" {definite}")


NEMO_SPACE with definite would be cleaner

Signed-off-by: Jim O'Regan <joregan@kth.se>

…ssing into swedish-updates Signed-off-by: Jim O'Regan <joregan@kth.se>

Signed-off-by: Jim O'Regan <joregan@kth.se>

jimregan added 4 commits August 7, 2026 14:39

Plus / minus

2eaf203

Signed-off-by: Jim O'Regan <joregan@kth.se>

Add Swedish legal reference readings

dce05a1

Signed-off-by: Jim O'Regan <joregan@kth.se>

Expand abbreviations from Sardin

66ccc05

Signed-off-by: Jim O'Regan <joregan@kth.se> Signed-off-by: "Christina Tånnander" <christina.tannander@mtm.se>

Add header to __init__.py

98869d5

Signed-off-by: Jim O'Regan <joregan@kth.se>

jimregan changed the title ~~Plus / minus~~ Extend Swedish TN Aug 7, 2026

pre-commit-ci Bot and others added 6 commits August 7, 2026 16:15

[pre-commit.ci] auto fixes from pre-commit.com hooks

5b50a18

for more information, see https://pre-commit.ci

Move Swedish units into measure grammar

cdb85eb

Signed-off-by: Jim O'Regan <joregan@kth.se>

extend non-det ordinal readings to also accept an article

70e2767

Signed-off-by: Jim O'Regan <joregan@kth.se>

[pre-commit.ci] auto fixes from pre-commit.com hooks

b28f04e

for more information, see https://pre-commit.ci

fixes for sparrowhawk

3f3ab4e

Signed-off-by: Jim O'Regan <joregan@kth.se>

Merge branch 'swedish-updates' of github.com:jimregan/NeMo-text-proce…

bc1c02c

…ssing into swedish-updates Signed-off-by: Jim O'Regan <joregan@kth.se>

jimregan force-pushed the swedish-updates branch from e4f7d5a to bc1c02c Compare August 7, 2026 21:57

github-actions Bot added the Stale label Aug 22, 2026

jimregan added 2 commits August 29, 2026 08:10

lowercase abbreviations

29575b1

Signed-off-by: Jim O'Regan <joregan@kth.se>

sort abbreviations into existing categories

6d0432b

Signed-off-by: Jim O'Regan <joregan@kth.se>

github-actions Bot removed the Stale label Aug 30, 2026

tbartley94 assigned tbartley94 and svenchilton and unassigned tbartley94 Sep 11, 2026

tbartley94 assigned j-l-thorsson and loreskar Sep 14, 2026

j-l-thorsson reviewed Sep 18, 2026

View reviewed changes

Comment thread tests/nemo_text_processing/sv/data_text_normalization/test_cases_measure.txt

j-l-thorsson reviewed Sep 18, 2026

View reviewed changes

address review comments

559b374

Signed-off-by: Jim O'Regan <joregan@kth.se>

jimregan added 3 commits September 21, 2026 21:40

mm3

fe2e51f

Signed-off-by: Jim O'Regan <joregan@kth.se>

version without article

0dce2a5

Signed-off-by: Jim O'Regan <joregan@kth.se>

test fix

263846b

Signed-off-by: Jim O'Regan <joregan@kth.se>

tbartley94 changed the base branch from main to staging/swe_tn September 25, 2026 19:21

tbartley94 requested changes Sep 25, 2026

View reviewed changes

jimregan added 10 commits September 29, 2026 16:07

year

016e412

Signed-off-by: Jim O'Regan <joregan@kth.se>

insert_space/remove fstrings

99476cc

Signed-off-by: Jim O'Regan <joregan@kth.se>

simplify optional space

3de5c1d

Signed-off-by: Jim O'Regan <joregan@kth.se>

make better use of existing items

3d0ab54

Signed-off-by: Jim O'Regan <joregan@kth.se>

this also needs to work when deterministic

eb42c03

Signed-off-by: Jim O'Regan <joregan@kth.se>

Merge branch 'swedish-updates' of github.com:jimregan/NeMo-text-proce…

0548c24

…ssing into swedish-updates Signed-off-by: Jim O'Regan <joregan@kth.se>

make clear that this is a non-breaking space

4f2fa69

Signed-off-by: Jim O'Regan <joregan@kth.se>

change comment, hopefully this is clearer

195a83b

Signed-off-by: Jim O'Regan <joregan@kth.se>

rename variable, hopefully clearer

cee0c9f

Signed-off-by: Jim O'Regan <joregan@kth.se>

maybe overkill, but name the optional dot

7e5eb93

Signed-off-by: Jim O'Regan <joregan@kth.se>

Conversation

jimregan commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Before your PR is "Ready for review"

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

tbartley94 commented Sep 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

j-l-thorsson left a comment

Choose a reason for hiding this comment

Uh oh!

tbartley94 commented Sep 25, 2026

Uh oh!

tbartley94 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jimregan commented Aug 7, 2026 •

edited

Loading