Skip to content

Add nine languages: Polish, Czech, Greek, Hebrew, Turkish, Bulgarian, Italian, Portuguese (PT + BR) - #18

Merged
vstrelnikof merged 3 commits into
mainfrom
lang/nine-more-languages
Aug 1, 2026
Merged

Add nine languages: Polish, Czech, Greek, Hebrew, Turkish, Bulgarian, Italian, Portuguese (PT + BR)#18
vstrelnikof merged 3 commits into
mainfrom
lang/nine-more-languages

Conversation

@vstrelnikof

Copy link
Copy Markdown
Member

Closes #2.

Fifteen bundled layouts instead of six. Each language is a layout TOML
plus a full dictionary, so the detection that makes uk-UA ↔ en-US
reliable now applies to cs-CZ, tr-TR, it-IT and the rest. The
active-layout filter is untouched — a two-keyboard user still loads two.

Layout mappings were generated from xkeyboard-config rather than
transcribed from keyboard diagrams. The generator reproduced the
existing de_de.toml byte-for-byte as a control before any new file
was kept; the technique is written up in docs/ADDING_A_LANGUAGE.md
for the next person.

Three bugs found on the way

All three produced a green run, which is why they had survived.

Polish and Greek would have shipped as mojibake. Hunspell declares
a dictionary pair's encoding once, in the .aff — the .dic has no
SET line of its own. We scanned each file separately and fell back to
Latin-1 when there wasn't one, so every .dic hit that fallback. That
was invisible for the six languages bundled so far (four are UTF-8, and
German genuinely is Latin-1), and fatal for anything else: słowo
landed in the FST as s³owo, which matches no lookup and looks enough
like a word to survive a skim. Fixed by reading the .aff once and
decoding both halves with it, with real 96-entry tables for ISO-8859-2
and ISO-8859-7 generated from Python's codecs rather than typed by hand.
An unknown or absent SET is now an error — guessing is the failure
mode here, not a convenience. Polish went 2.28 M → 3.96 M forms once
the affix conditions actually matched.

The French source had been 404ing. Upstream moved it from fr_FR/
to fr_FR/dictionaries/; the fetch printed one line on stderr and
exited 0, leaving the stale wordlist in place. Fixed, and the command
now tallies failures and exits non-zero. The refresh gains 11 701 forms
and loses 26, all elision artifacts like d'pick-up.

FLAG num was rejected outright on the grounds that no bundled
dictionary used it. Turkish does.

Regenerating also turned up a UTF-8 BOM on pt_BR.aff, previously
invisible because valid-UTF-8 files short-circuited before the SET
scan.

Two caveats worth reading before merge

Polish maps to the same characters as US English. The layout
essentially every Polish user has enabled is the "programmer's" one —
QWERTY with every diacritic on AltGr, which PolterType doesn't track.
So there is no pl-PL ↔ en-US mistake to make and none to correct. The
file still earns its place: it gives Polish an id the OS switcher can
target, and it hangs the Polish wordlist off a layout, which is what
stops Polish prose being dragged toward whichever other layout is
active. Stated in the TOML header, CREDITS.md and the changelog
rather than left to be discovered.

Hebrew's Hspell is AGPL-3.0-or-later — the strictest licence in the
tree and the only one with a network clause. It attaches to the data,
not the engine, and we build no network service, so a locally-installed
desktop app is within terms; but a downstream packager needs to know.
Checking it prompted an audit of all thirteen against their upstream
licence files, which found the old blanket hedge ("most often
GPL-2-or-later or LGPL/MPL") was also wrong about Russian — it's BSD.
CREDITS.md now carries a per-language table.

Cost

+73 MB in the repo, +57 MB in every installer — 30 MB of that is
Turkish alone, whose agglutination expands to 5.8 M forms and two 15 MB
FSTs. That is the price of the detection quality, and nothing is read at
runtime without the matching keyboard enabled.

Hebrew would have been 141 MB expanded faithfully — its affix table is
3335 prefix rules and no suffix rules, encoding the clitic particles as
affixes — so it ships stems only via a new ExpandMode. Its script
already separates it from everything else bundled, so the dictionary is
a refinement there rather than the load-bearing signal.

Verification

  • 390 tests pass; clippy --all-features -D warnings, fmt --check
    and cargo deny check all clean.
  • New tests cover the codepage decoders, the BOM, FLAG num, the
    distinctive keys of every new layout, and — the one that would have
    caught the mojibake — that each bundled dictionary contains ordinary
    words of its own including diacritics, since the ASCII-only words
    were correct the whole time.
  • A test now pins build.rs::LAYOUTS against BUNDLED_LAYOUT_STEMS;
    that drift previously only produced a startup warning.
  • Ran the app on this machine: all fifteen are registered, the two
    active in the OS load, the other thirteen are skipped cleanly with no
    missing-TOML warnings.

docs/ADDING_A_LANGUAGE.md gains the two steps it never had — the WiX
entries (skip them and the language works everywhere except Windows,
silently) and registering the id with the OS switchers.

All three produced a green run and wrong (or stale) data, which is
why they survived: the command's only failure signal was a line on
stderr that nothing read.

Encoding. Hunspell declares a dictionary pair's encoding once, in the
.aff — the .dic carries no SET line of its own. We scanned each file
separately and fell back to Latin-1 when a file had no SET, so every
.dic hit that fallback. Invisible for the languages bundled so far
(four are UTF-8, and German genuinely is Latin-1), fatal for anything
else: `ISO8859-*` was also matched as a prefix and decoded as Latin-1,
so Polish (ISO-8859-2) and Greek (ISO-8859-7) came out as plausible
mojibake — `słowo` as `s³owo`, which matches no lookup and trips no
check. Now the .aff is read once and its encoding decodes both halves,
Latin-2 and Greek get real 96-entry tables generated from Python's
codecs rather than typed by hand, and an unknown or missing SET is an
error. Guessing is the failure mode here, not a convenience. A UTF-8
BOM (pt_BR ships one) no longer hides the SET line either.

Dead source. The French dictionary moved upstream from fr_FR/ to
fr_FR/dictionaries/ and the old URL had been 404ing, leaving the
committed wordlist to quietly go stale. Fixed, and the command now
tallies per-language failures and exits non-zero, so the next dead
URL can't pass for a successful run.

FLAG num. Comma-separated numeric affix flags were rejected outright
on the grounds that no bundled dictionary used them. Turkish does —
its affix table is effectively one flag per surface form and runs to
six figures.

Also splits the source list into a HUNSPELL_SOURCES table so adding a
language is one entry rather than another copy-pasted call, and gives
it an ExpandMode so a dictionary whose affix table is combinatorial
rather than inflectional can ship stems.
…n and Portuguese

Fifteen layouts instead of six. Each language is a layout TOML plus a
full dictionary, so the detection that makes uk-UA ↔ en-US reliable
now applies to cs-CZ, tr-TR, it-IT and the rest. The active-layout
filter is unchanged, so a two-keyboard user still pays for two.

Mappings were generated from xkeyboard-config rather than transcribed
from keyboard diagrams — the generator reproduced the existing
de_de.toml byte-for-byte as a control before any new file was kept.
Dead keys are surfaced as their spacing equivalents, as the other
bundled layouts already do.

Two carry caveats, documented in their TOML headers rather than left
for a user to discover:

Polish maps to exactly the same characters as US English. The layout
essentially every Polish user has enabled is the "programmer's" one —
QWERTY with every diacritic on AltGr, which we don't track — so there
is no pl-PL ↔ en-US mistake to make and none to correct. The file
still earns its place: it gives Polish an id the OS switcher can
target and hangs the Polish wordlist off a layout, which is what stops
Polish prose being dragged toward whichever other layout is active.

Hebrew ships dictionary stems without affix expansion. Its Hunspell
table is 3335 prefix rules and no suffix rules — it encodes the clitic
particles as affixes — and expanding it faithfully gives 60.6 M forms:
a 141 MB wordlist and a far larger FST in every installer. Hebrew
shares its script with nothing else bundled, so plausibility already
separates it and the dictionary is a refinement rather than the
load-bearing signal.

derive_vowels learns the new vowel sets. Two aren't what a script
default gives: ъ is a full vowel in Bulgarian (unlike Russian, where
the same glyph is a silent sign), and Turkish's dotless ı is a vowel
the bare Latin set would score as a consonant.

The MSI enumerates every data file by hand, so it gets four entries
per language; the AppImage and macOS bundle copy the tree and need
nothing. Installers grow ~57 MB, 30 MB of it Turkish — agglutination
expands to 5.8 M forms and two 15 MB FSTs. That is the cost of the
detection quality, and none of it is read at runtime without the
matching keyboard enabled.

CREDITS.md now states each dictionary's licence per language instead
of hedging. The old blanket "most often GPL-2-or-later or LGPL/MPL"
was wrong about Russian, which is BSD — and Hebrew's Hspell is
AGPL-3.0-or-later, the strictest thing in the tree and worth knowing
before redistributing a build.

fr_fr.txt.gz is refreshed as a side effect of the URL fix in the
previous commit: +11701 forms, -26, the losses all elision artifacts
like `d'pick-up`.
ADDING_A_LANGUAGE.md gains the two steps it never had. The MSI
enumerates every data file by hand, so a bundled language needs four
WiX entries or it works everywhere except Windows — silently. And a
new layout id has to be registered with the OS switchers in
poltertype-layout, or the app can name the language but never switch
to it. Both were discovered by doing this rather than by reading, and
the count of steps a contributor has to get right is exactly the thing
that doc exists to state.

It also now says how to get a mapping right without guessing: on Linux
xkbcommon already knows what every layout produces per key, and the
XKB key names map onto Set-1 scancodes in four straight runs.

DECISIONS.md marks the encoding rule superseded rather than editing it
away — the old text describes real reasoning that turned out to be
wrong in a specific, instructive way, and the note explains why
guessing an encoding is the failure mode rather than a convenience.
@vstrelnikof
vstrelnikof merged commit 88c90a2 into main Aug 1, 2026
4 checks passed
@vstrelnikof
vstrelnikof deleted the lang/nine-more-languages branch August 1, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add your language: layout TOMLs and wordlists wanted

1 participant