Skip to content

pl: mark the unicode translations as verified (t -> T) - #741

Open
michaldziwisz wants to merge 3 commits into
daisy:mainfrom
michaldziwisz:pl-verified-keys
Open

pl: mark the unicode translations as verified (t -> T)#741
michaldziwisz wants to merge 3 commits into
daisy:mainfrom
michaldziwisz:pl-verified-keys

Conversation

@michaldziwisz

@michaldziwisz michaldziwisz commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

What this does

Raises the "needs review" key (t/ot/ctT/OT/CT) on Polish entries that have in fact been translated all along, and adds tests for one rule that had none. No wording is changed anywhere in this PR.

Three commits, reviewable separately:

commit what lines
5f1a8dda single-line entries in unicode.yaml / unicode-full.yaml 2435
845be3df multi-line entries in the same two files 312
b4e8ca92 rule files (ClearSpeak_Rules, navigate, SharedRules/*, overview) + new tests 248

Why

The Polish rules were written before the lower/upper-case convention was in use, so translated entries still carried the "needs review" key. The audit tool counted every one of them as untranslated:

audit-translations pl, untranslated text:  3162 -> 2

The 3162 made the real gaps invisible. Working out that only ~80 unicode entries were genuinely untranslated (handled in #738) required a separate tool comparing the text against the English source, because key case alone says nothing about whether the work was done.

Why the second commit exists

The tool I used for the first pass only recognised entries written on one line (- "±": [t: "..."]). Entries whose body is a block were invisible to it:

 - "°":
    - test:
        if: "parent::*[name(.)='msup' ...]"
        then: [t: "stopni"]
        else: [t: "stopień"]

Those are exactly the entries carrying inflection or a TTS condition — 312 of the most deliberate translations in the file were left looking unreviewed. Fixing the tool and re-running it over my own earlier work is what surfaced them.

What is not raised

Only entries whose text differs from the English source are touched — that is the evidence a translation exists. 18 entries deliberately keep a lower-case key:

  • t: "" for U+00A0 (the @data-empty-in-2D else-branch). That is silence, not text; an upper-case key asserts "a translator checked this wording" and there is no wording to check. English keeps a lower-case key there too.
  • the letters C, N, Q, R, Z in dimension-number-sets — the letter is the wording, as in English.
  • ct: "s" and ct: "-" in the unit rule: the plural suffix and the prefix separator, not words. (The "s" is an English plural ending on Polish unit names; pl: inflect unit names according to the numeral, not with an English "s" #744 removes it.)
  • navigate.yaml:176, a Polish euphony variant ("ze" before "st-"), which happens to equal a Russian letter name in the English corpus.
  • seven XPath expressions rather than speech.

Entries whose text legitimately equals English (Roman numerals, ligatures, proper names, Greek and Cyrillic letter names) were reviewed individually in #738.

Tests for the number-set rules

ℂ ℕ ℚ ℝ ℤ had no Polish test at all, in a file where two silent defects have already been found. New tests/Languages/pl/number_sets.rs covers the bare symbols, the dimension form, both sign variants, and ClearSpeak agreeing with SimpleSpeak.

number_set_with_variable_exponent is the negative control: dimension-number-sets fires on count(*)=2, so a non-numeric exponent must fall through to the full phrase ("liczby całkowite do potęgi n") instead of being read as a dimension. Without it, a rule that swallowed every superscript would still pass.

Polish keeps the letter with a dimension ("R 3"), matching en, de, fr, nb, sv, fi and es; Hungarian is the only language that translates in that position too. Checked against the SAPI voice NVDA actually uses: "R 3" renders byte-identical to a phonetic "er 3", and "N 4" to "en 4", so the letters really are spoken as their Polish names.

Verification

cargo test --test languages Languages::pl    614 passed, 0 failed

validity control - breaking the rule must fail the new tests:
  dimension_number_sets              FAILED
  number_sets_clearspeak_matches     FAILED
  simple_number_sets                     ok   <- different rule
  pos_neg_number_sets                    ok   <- different rule
  number_set_with_variable_exponent      ok   <- negative control

Zero changes outside key case. Each removed line was paired with its added line and compared after normalising the key name; all 248 rule-file differences are t -> T with content, comments and indentation identical character for character. For the unicode pass the same check was done with:

git diff -U0 | grep -E '^[-+]' | grep -v '^[-+][-+]' \
  | sed -E 's/^.//; s/\b(t|ot|ct|T|OT|CT):/KEY:/' | sort | uniq -c | awk '$1%2!=0'

Without the normalisation 4710 unpaired lines remain, so the check discriminates rather than trivially passing.

Kept separate from #738 on purpose: 2995 lines of key case should not bury five reviewable fixes.

@NSoiffer

NSoiffer commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

There are conflicts in unicode-full.yaml that appear to be becasue of the t->T change. Can you resolve them. I don't accidentally want to break something

Mechanical change: 2435 text keys raised from t/ot/ct to T/OT/CT in the
two unicode files. No text is touched.

The Polish rules were written before the lowercase/uppercase convention
was in use, so entries that have been translated all along still carried
the "needs review" key. The audit tool therefore reported them as
untranslated, which made the real gaps impossible to see:

  audit-translations pl, untranslated text:  3162 -> 727

The remaining 727 are in the rule files, where the entries need reading
rather than a mechanical pass, so they are deliberately left alone.

Every raised entry was checked to differ from the English source, i.e.
it really is translated. Entries whose text legitimately equals English
(Roman numerals, ligatures, proper names, whitespace) were handled
separately in the previous PR; entries still holding English text are
NOT raised, since that would assert a translation that does not exist.

Verification:

  cargo test --test languages Languages::pl   609 passed, 0 failed
  diff: 2436 insertions, 2436 deletions, zero changes outside key case

The last point is checked by normalising the key case in the diff and
confirming no unpaired lines remain (without normalising, 4710 remain,
so the check does discriminate).
@michaldziwisz

Copy link
Copy Markdown
Contributor Author

Done - rebased onto current main (which now includes #738), force-pushed, and the PR is mergeable again as a single commit.

One clarification on the cause, because it matters for "I don't accidentally want to break something": the conflicts were not the t->T change disagreeing with anything in main. They came from how the branch was stacked. #741 sat on top of #738, and #738 was merged as a squash (9ea98c23), so its individual commits no longer exist in main. Git therefore had to merge the old pl content (via #738's commits) against the new squashed pl content (in main) - byte-identical text, different history - and that surfaced as 12 conflict blocks in unicode-full.yaml. I reproduced it in a throwaway worktree: every block was the same lines on both sides, no real content disagreement. Rebasing --onto main drops the duplicated history and the conflicts disappear.

Measurements, so the safety claim is checkable rather than asserted:

  • git diff of Rules/Languages/pl/ between the pre-rebase and post-rebase branch: 0 lines. The rebase changed the base, not one character of the translation. Validity control on the same measurement: the same diff against main is 5482 lines, so it does discriminate.
  • Diff against main outside Rules/Languages/pl/: 0 lines. Only unicode.yaml and unicode-full.yaml are touched: 2436 insertions, 2436 deletions.
  • Key-case-only check: normalising t|ot|ct|T|OT|CT to one token in the diff leaves 0 unpaired lines; without normalising, 4710 remain - so the change really is nothing but the case of the key.
  • cargo test --test languages Languages::pl: 609 passed, 0 failed.
  • Your audit tool for pl, per file (main -> this PR): unicode-full.yaml 2742 -> 374, unicode.yaml 155 -> 88, total 3162 -> 727. The rule files are untouched.

To be precise about what is not fixed here, since the numbers do not go to zero: the remainder in those two files is small keys inside the rule blocks at the top of each file (letter/greek/capital handling, e.g. t: "wielka", t: "gwiazda"), plus one structure difference in the * rule that predates this branch. Those need reading entry by entry, not a mechanical pass, so I left them out - together with the 727 in SharedRules, definitions.yaml and navigate.yaml they are a separate PR.

One note for the release process: release-candidate has no commits touching Rules/Languages/pl/, so this should not conflict there either.

The previous commit raised the key on entries written on a single line
(`- "±": [t: "..."]`), but not on entries whose body is a block:

    - "°":
       - test:
           if: "parent::*[name(.)='msup' ...]"
           then: [t: "stopni"]
           else: [t: "stopień"]

Those are exactly the entries carrying inflection or a TTS condition, so
312 of the most deliberate translations in the file were left looking
unreviewed. This raises them: 64 in unicode.yaml, 247 in unicode-full.yaml
and one single-line entry (U+23E6) missed earlier.

One entry is deliberately left with a lower-case key: `t: ""` for U+00A0,
the else-branch of the `@data-empty-in-2D` test. That is silence rather
than text, and an upper-case key means "a translator checked this
wording" - there is no wording to check. English keeps a lower-case key
there too.

No content changes; only key case. `cargo test --test languages
Languages::pl` gives 609 passed, 0 failed, which is the same result as
before this commit, as expected for a change the engine treats as
equivalent (src/speech.rs matches "t"|"T" in one match arm).
Two loose ends in the Polish localization.

**Rule-file keys.** 248 entries in the rule files still used a lower-case
key while carrying translated Polish text. Content is untouched - every
one of the 248 changed lines differs from its predecessor only in the
case of the key name, verified line by line after normalizing the key.

17 entries are deliberately left lower-case: five are the letters C, N,
Q, R, Z in `dimension-number-sets` (the letter itself is the wording, as
in English); `ct: "s"`/`ct: "-"` in the unit rule are the plural suffix
and prefix separator; `navigate.yaml:176` is a Polish euphony variant
("ze" before "st-"); the remaining seven are XPath expressions rather
than speech.

**Tests for the number-set rules.** These rules had no Polish test at
all, in a file where two silent defects were already found. New
`tests/Languages/pl/number_sets.rs` covers the bare symbols, the
dimension form, both sign variants, and ClearSpeak agreeing with
SimpleSpeak.

The `number_set_with_variable_exponent` case is the negative control:
`dimension-number-sets` fires on `count(*)=2`, so a non-numeric exponent
must fall through to the full phrase ("liczby całkowite do potęgi n")
rather than being read as a dimension. Without it, a rule that swallowed
every superscript would still pass.

Polish keeps the letter with a dimension ("R 3"), matching en, de, fr,
nb, sv, fi and es; Hungarian is the only language that translates in that
position too. Measured with the SAPI voice NVDA actually uses: "R 3" is
byte-identical to a phonetic "er 3" and "N 4" to "en 4", so the letters
are pronounced as their Polish names.

    cargo test --test languages Languages::pl   614 passed, 0 failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

2 participants