Add nine languages: Polish, Czech, Greek, Hebrew, Turkish, Bulgarian, Italian, Portuguese (PT + BR) - #18
Merged
Merged
Conversation
All three produced a green run and wrong (or stale) data, which is why they survived: the command's only failure signal was a line on stderr that nothing read. Encoding. Hunspell declares a dictionary pair's encoding once, in the .aff — the .dic carries no SET line of its own. We scanned each file separately and fell back to Latin-1 when a file had no SET, so every .dic hit that fallback. Invisible for the languages bundled so far (four are UTF-8, and German genuinely is Latin-1), fatal for anything else: `ISO8859-*` was also matched as a prefix and decoded as Latin-1, so Polish (ISO-8859-2) and Greek (ISO-8859-7) came out as plausible mojibake — `słowo` as `s³owo`, which matches no lookup and trips no check. Now the .aff is read once and its encoding decodes both halves, Latin-2 and Greek get real 96-entry tables generated from Python's codecs rather than typed by hand, and an unknown or missing SET is an error. Guessing is the failure mode here, not a convenience. A UTF-8 BOM (pt_BR ships one) no longer hides the SET line either. Dead source. The French dictionary moved upstream from fr_FR/ to fr_FR/dictionaries/ and the old URL had been 404ing, leaving the committed wordlist to quietly go stale. Fixed, and the command now tallies per-language failures and exits non-zero, so the next dead URL can't pass for a successful run. FLAG num. Comma-separated numeric affix flags were rejected outright on the grounds that no bundled dictionary used them. Turkish does — its affix table is effectively one flag per surface form and runs to six figures. Also splits the source list into a HUNSPELL_SOURCES table so adding a language is one entry rather than another copy-pasted call, and gives it an ExpandMode so a dictionary whose affix table is combinatorial rather than inflectional can ship stems.
…n and Portuguese Fifteen layouts instead of six. Each language is a layout TOML plus a full dictionary, so the detection that makes uk-UA ↔ en-US reliable now applies to cs-CZ, tr-TR, it-IT and the rest. The active-layout filter is unchanged, so a two-keyboard user still pays for two. Mappings were generated from xkeyboard-config rather than transcribed from keyboard diagrams — the generator reproduced the existing de_de.toml byte-for-byte as a control before any new file was kept. Dead keys are surfaced as their spacing equivalents, as the other bundled layouts already do. Two carry caveats, documented in their TOML headers rather than left for a user to discover: Polish maps to exactly the same characters as US English. The layout essentially every Polish user has enabled is the "programmer's" one — QWERTY with every diacritic on AltGr, which we don't track — so there is no pl-PL ↔ en-US mistake to make and none to correct. The file still earns its place: it gives Polish an id the OS switcher can target and hangs the Polish wordlist off a layout, which is what stops Polish prose being dragged toward whichever other layout is active. Hebrew ships dictionary stems without affix expansion. Its Hunspell table is 3335 prefix rules and no suffix rules — it encodes the clitic particles as affixes — and expanding it faithfully gives 60.6 M forms: a 141 MB wordlist and a far larger FST in every installer. Hebrew shares its script with nothing else bundled, so plausibility already separates it and the dictionary is a refinement rather than the load-bearing signal. derive_vowels learns the new vowel sets. Two aren't what a script default gives: ъ is a full vowel in Bulgarian (unlike Russian, where the same glyph is a silent sign), and Turkish's dotless ı is a vowel the bare Latin set would score as a consonant. The MSI enumerates every data file by hand, so it gets four entries per language; the AppImage and macOS bundle copy the tree and need nothing. Installers grow ~57 MB, 30 MB of it Turkish — agglutination expands to 5.8 M forms and two 15 MB FSTs. That is the cost of the detection quality, and none of it is read at runtime without the matching keyboard enabled. CREDITS.md now states each dictionary's licence per language instead of hedging. The old blanket "most often GPL-2-or-later or LGPL/MPL" was wrong about Russian, which is BSD — and Hebrew's Hspell is AGPL-3.0-or-later, the strictest thing in the tree and worth knowing before redistributing a build. fr_fr.txt.gz is refreshed as a side effect of the URL fix in the previous commit: +11701 forms, -26, the losses all elision artifacts like `d'pick-up`.
ADDING_A_LANGUAGE.md gains the two steps it never had. The MSI enumerates every data file by hand, so a bundled language needs four WiX entries or it works everywhere except Windows — silently. And a new layout id has to be registered with the OS switchers in poltertype-layout, or the app can name the language but never switch to it. Both were discovered by doing this rather than by reading, and the count of steps a contributor has to get right is exactly the thing that doc exists to state. It also now says how to get a mapping right without guessing: on Linux xkbcommon already knows what every layout produces per key, and the XKB key names map onto Set-1 scancodes in four straight runs. DECISIONS.md marks the encoding rule superseded rather than editing it away — the old text describes real reasoning that turned out to be wrong in a specific, instructive way, and the note explains why guessing an encoding is the failure mode rather than a convenience.
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2.
Fifteen bundled layouts instead of six. Each language is a layout TOML
plus a full dictionary, so the detection that makes uk-UA ↔ en-US
reliable now applies to cs-CZ, tr-TR, it-IT and the rest. The
active-layout filter is untouched — a two-keyboard user still loads two.
Layout mappings were generated from
xkeyboard-configrather thantranscribed from keyboard diagrams. The generator reproduced the
existing
de_de.tomlbyte-for-byte as a control before any new filewas kept; the technique is written up in
docs/ADDING_A_LANGUAGE.mdfor the next person.
Three bugs found on the way
All three produced a green run, which is why they had survived.
Polish and Greek would have shipped as mojibake. Hunspell declares
a dictionary pair's encoding once, in the
.aff— the.dichas noSETline of its own. We scanned each file separately and fell back toLatin-1 when there wasn't one, so every
.dichit that fallback. Thatwas invisible for the six languages bundled so far (four are UTF-8, and
German genuinely is Latin-1), and fatal for anything else:
słowolanded in the FST as
s³owo, which matches no lookup and looks enoughlike a word to survive a skim. Fixed by reading the
.affonce anddecoding both halves with it, with real 96-entry tables for ISO-8859-2
and ISO-8859-7 generated from Python's codecs rather than typed by hand.
An unknown or absent
SETis now an error — guessing is the failuremode here, not a convenience. Polish went 2.28 M → 3.96 M forms once
the affix conditions actually matched.
The French source had been 404ing. Upstream moved it from
fr_FR/to
fr_FR/dictionaries/; the fetch printed one line on stderr andexited 0, leaving the stale wordlist in place. Fixed, and the command
now tallies failures and exits non-zero. The refresh gains 11 701 forms
and loses 26, all elision artifacts like
d'pick-up.FLAG numwas rejected outright on the grounds that no bundleddictionary used it. Turkish does.
Regenerating also turned up a UTF-8 BOM on
pt_BR.aff, previouslyinvisible because valid-UTF-8 files short-circuited before the
SETscan.
Two caveats worth reading before merge
Polish maps to the same characters as US English. The layout
essentially every Polish user has enabled is the "programmer's" one —
QWERTY with every diacritic on AltGr, which PolterType doesn't track.
So there is no pl-PL ↔ en-US mistake to make and none to correct. The
file still earns its place: it gives Polish an id the OS switcher can
target, and it hangs the Polish wordlist off a layout, which is what
stops Polish prose being dragged toward whichever other layout is
active. Stated in the TOML header,
CREDITS.mdand the changelograther than left to be discovered.
Hebrew's Hspell is AGPL-3.0-or-later — the strictest licence in the
tree and the only one with a network clause. It attaches to the data,
not the engine, and we build no network service, so a locally-installed
desktop app is within terms; but a downstream packager needs to know.
Checking it prompted an audit of all thirteen against their upstream
licence files, which found the old blanket hedge ("most often
GPL-2-or-later or LGPL/MPL") was also wrong about Russian — it's BSD.
CREDITS.mdnow carries a per-language table.Cost
+73 MB in the repo, +57 MB in every installer — 30 MB of that is
Turkish alone, whose agglutination expands to 5.8 M forms and two 15 MB
FSTs. That is the price of the detection quality, and nothing is read at
runtime without the matching keyboard enabled.
Hebrew would have been 141 MB expanded faithfully — its affix table is
3335 prefix rules and no suffix rules, encoding the clitic particles as
affixes — so it ships stems only via a new
ExpandMode. Its scriptalready separates it from everything else bundled, so the dictionary is
a refinement there rather than the load-bearing signal.
Verification
clippy --all-features -D warnings,fmt --checkand
cargo deny checkall clean.FLAG num, thedistinctive keys of every new layout, and — the one that would have
caught the mojibake — that each bundled dictionary contains ordinary
words of its own including diacritics, since the ASCII-only words
were correct the whole time.
build.rs::LAYOUTSagainstBUNDLED_LAYOUT_STEMS;that drift previously only produced a startup warning.
active in the OS load, the other thirteen are skipped cleanly with no
missing-TOML warnings.
docs/ADDING_A_LANGUAGE.mdgains the two steps it never had — the WiXentries (skip them and the language works everywhere except Windows,
silently) and registering the id with the OS switchers.