Add nine languages: Polish, Czech, Greek, Hebrew, Turkish, Bulgarian, Italian, Portuguese (PT + BR)

CHANGELOG.md

@@ -4,6 +4,86 @@ All notable changes to PolterType are recorded here. The format is
 loosely based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and the project follows [Semantic Versioning](https://semver.org/).
 
+## [Unreleased] — 0.8.0
+
+### Added
+
+- **Nine more languages: Polish, Czech, Greek, Hebrew, Turkish,
+  Bulgarian, Italian, and Portuguese in both its orthographies.**
+  PolterType now bundles fifteen layouts instead of six. Each one is a
+  layout TOML plus a full dictionary, so the same detection that makes
+  uk-UA ↔ en-US reliable applies to cs-CZ, tr-TR, it-IT and the rest.
+  Nothing loads that your OS doesn't have enabled — the active-layout
+  filter still means a two-keyboard user pays for two.
+
+  Layout mappings were generated from `xkeyboard-config` rather than
+  transcribed from keyboard pictures, and then reviewed; the trick is
+  written up in [docs/ADDING_A_LANGUAGE.md](docs/ADDING_A_LANGUAGE.md)
+  for the next person. Closes [#2].
+
+  Two of them carry a caveat worth reading before you expect magic:
+
+  * **Polish** maps to exactly the same characters as US English. The
+    standard Polish layout is the "programmer's" one — QWERTY with
+    every diacritic on AltGr, which PolterType doesn't track — so
+    there is no pl-PL ↔ en-US mistake to correct, and none is
+    possible. The Polish *wordlist* still does real work: it stops
+    Polish prose being dragged toward whichever other layout you have
+    active.
+  * **Hebrew** ships dictionary stems without affix expansion. Its
+    Hunspell table encodes the clitic prefixes as 3335 prefix rules,
+    which expand to 60.6 M forms — a 141 MB wordlist and a far bigger
+    FST in every installer. Hebrew shares its script with nothing else
+    bundled, so plausibility already separates it and the dictionary
+    is a refinement. See
+    [data/wordlists/CREDITS.md](data/wordlists/CREDITS.md).
+
+  **Installers grow by about 57 MB**, most of it Turkish — an
+  agglutinative language expands to 5.8 M forms and two 15 MB FSTs.
+  That is the price of the detection quality; nothing about it is
+  loaded at runtime unless you have a Turkish keyboard enabled.
+
+### Fixed
+
+- **Polish and Greek dictionaries would have shipped as mojibake, and
+  the French one hadn't been refreshed in a year.** Three separate
+  faults in `cargo xtask wordlists fetch`, all of which failed
+  quietly:
+
+  * Hunspell declares a dictionary's encoding once, in the `.aff` —
+    the `.dic` has no `SET` line of its own. We looked for one in each
+    file separately and fell back to Latin-1 when there wasn't any, so
+    Polish (ISO-8859-2) and Greek (ISO-8859-7) decoded into plausible
+    nonsense: `słowo` became `s³owo`, which neither matches a lookup
+    nor trips a check. German survived only because German really is
+    Latin-1. The `.aff`'s declared encoding is now what decodes both
+    halves, ISO-8859-2 and ISO-8859-7 have real tables, and an
+    unrecognised or absent `SET` is an error instead of a guess.
+  * The French source moved upstream (`fr_FR/` → `fr_FR/dictionaries/`)
+    and the old URL had been 404ing. The fetch printed one stderr line
+    and exited 0, so the stale wordlist just stayed. Fixed, and the
+    command now exits non-zero when any source fails. The refreshed
+    French list gains 11,701 forms and loses 26, all of them elision
+    artifacts like `d'pick-up`.
+  * `FLAG num` dictionaries (comma-separated numeric affix flags) were
+    rejected outright. Turkish needs them — its affix table runs to
+    six figures of distinct flags. Now supported.
+
+### Changed
+
+- `derive_vowels` learned the vowel sets of the new languages, which
+  the plausibility detector counts. Two are not what a script default
+  would give you: `ъ` is a full vowel in Bulgarian, and Turkish's
+  dotless `ı` is a vowel that the bare Latin set scores as a
+  consonant.
+- `data/wordlists/CREDITS.md` now states each dictionary's licence
+  per-language instead of hedging. The old blanket "most often
+  GPL-2-or-later or LGPL/MPL" was wrong about at least Russian, which
+  is BSD — and **Hebrew's Hspell is AGPL-3.0-or-later**, the strictest
+  thing in the tree and worth knowing before redistributing a build.
+
+[#2]: https://github.com/Just-Code-NET/PolterType/issues/2
+
 ## [0.8.0] — Windows learns to hold your keys, the AI seam comes alive, and the app gets its own face
 
 ### Added

crates/poltertype-core/build.rs

@@ -71,6 +71,15 @@ const LAYOUTS: &[(&str, &str)] = &[
     ("de_de", "de-DE"),
     ("es_es", "es-ES"),
     ("fr_fr", "fr-FR"),
+    ("pl_pl", "pl-PL"),
+    ("cs_cz", "cs-CZ"),
+    ("el_gr", "el-GR"),
+    ("he_il", "he-IL"),
+    ("tr_tr", "tr-TR"),
+    ("bg_bg", "bg-BG"),
+    ("it_it", "it-IT"),
+    ("pt_pt", "pt-PT"),
+    ("pt_br", "pt-BR"),
 ];
 
 fn main() {

crates/poltertype-core/src/layouts/consts.rs

@@ -11,4 +11,7 @@
 ///
 /// Kept in lock-step with `build.rs::LAYOUTS`. A mismatch shows up
 /// as a "missing TOML" warning at startup, never silently.
-pub const BUNDLED_LAYOUT_STEMS: &[&str] = &["en_us", "uk_ua", "ru_ru", "de_de", "es_es", "fr_fr"];
+pub const BUNDLED_LAYOUT_STEMS: &[&str] = &[
+    "en_us", "uk_ua", "ru_ru", "de_de", "es_es", "fr_fr", "pl_pl", "cs_cz", "el_gr", "he_il",
+    "tr_tr", "bg_bg", "it_it", "pt_pt", "pt_br",
+];

crates/poltertype-core/src/layouts/helpers.rs

@@ -30,6 +30,22 @@ pub fn derive_vowels(id: &LayoutId, script: Script) -> Vec<char> {
         "de-DE" => "aeiouäöü".chars().collect(),
         "es-ES" => "aeiouáéíóúü".chars().collect(),
         "fr-FR" => "aeiouyàâéèêëîïôûùüÿ".chars().collect(),
+        "pl-PL" => "aeiouyąęó".chars().collect(),
+        "cs-CZ" => "aeiouyáéíóúůýě".chars().collect(),
+        // Greek needs the accented forms spelled out: the script
+        // default covers the bare vowels only, and Greek marks the
+        // stressed vowel of almost every polysyllabic word.
+        "el-GR" => "αεηιουωάέήίόύώϊϋΐΰ".chars().collect(),
+        // `ı` (dotless i) and `ı`'s dotted twin are both vowels, as
+        // are ö and ü — the bare Latin default would score four of
+        // Turkish's eight vowels as consonants.
+        "tr-TR" => "aeıioöuü".chars().collect(),
+        // `ъ` is a full vowel in Bulgarian (unlike Russian, where the
+        // same glyph is a silent sign), so the generic Cyrillic set
+        // is wrong here in both directions.
+        "bg-BG" => "аеиоуъюя".chars().collect(),
+        "it-IT" => "aeiouàèéìíîòóùú".chars().collect(),
+        "pt-PT" | "pt-BR" => "aeiouáàâãéêíóôõúü".chars().collect(),
         _ => match script {
             Script::Latin => "aeiouy".chars().collect(),
             Script::Cyrillic => "аеиіоуюяєїыэё".chars().collect(),

crates/poltertype-core/src/layouts/tests.rs

@@ -16,17 +16,37 @@ use super::*;
 #[test]
 fn embedded_layouts_load() {
     let db = LayoutDb::load_embedded();
-    for id in ["en-US", "uk-UA", "ru-RU", "de-DE", "es-ES", "fr-FR"] {
+    for id in [
+        "en-US", "uk-UA", "ru-RU", "de-DE", "es-ES", "fr-FR", "pl-PL", "cs-CZ", "el-GR", "he-IL",
+        "tr-TR", "bg-BG", "it-IT", "pt-PT", "pt-BR",
+    ] {
         assert!(
             db.get(&LayoutId::from(id)).is_some(),
             "embedded layout `{id}` did not load"
         );
     }
 }
 
+/// `build.rs::LAYOUTS` and [`BUNDLED_LAYOUT_STEMS`] are two lists that
+/// have to agree; when they drift the runtime only *warns*, so nothing
+/// fails until a user notices their language is missing. Pin the count
+/// against what actually loads.
+#[test]
+fn every_bundled_stem_resolves_to_a_layout() {
+    let db = LayoutDb::load_embedded();
+    assert_eq!(
+        db.len(),
+        BUNDLED_LAYOUT_STEMS.len(),
+        "{} bundled stems but {} layouts loaded — build.rs::LAYOUTS and \
+         BUNDLED_LAYOUT_STEMS have drifted, or a TOML failed to parse",
+        BUNDLED_LAYOUT_STEMS.len(),
+        db.len()
+    );
+}
+
 /// The active-filter feature: only requested layouts enter memory.
-/// This is what saves RAM for users who don't have all six bundled
-/// languages installed in the OS.
+/// This is what saves RAM for users who don't have all fifteen
+/// bundled languages installed in the OS — which is everyone.
 #[test]
 fn active_filter_drops_unrequested_layouts() {
     let want = [LayoutId::from("en-US"), LayoutId::from("uk-UA")];
@@ -86,6 +106,36 @@ fn new_languages_translate_distinctive_keys() {
         ("es-ES", 0x27, false, 'ñ'),
         ("fr-FR", 0x10, false, 'a'),
         ("fr-FR", 0x03, false, 'é'),
+        // Czech puts letters on the unshifted number row and swaps
+        // Y/Z like German.
+        ("cs-CZ", 0x03, false, 'ě'),
+        ("cs-CZ", 0x15, false, 'z'),
+        ("cs-CZ", 0x2C, false, 'y'),
+        // Greek: Q and W positions carry `;` and final sigma.
+        ("el-GR", 0x10, false, ';'),
+        ("el-GR", 0x11, false, 'ς'),
+        ("el-GR", 0x1E, false, 'α'),
+        // Hebrew is unicase — the letter positions are RTL glyphs.
+        ("he-IL", 0x1E, false, 'ש'),
+        ("he-IL", 0x14, false, 'א'),
+        // Turkish's dotless/dotted i pair, the one that bites.
+        ("tr-TR", 0x17, false, 'ı'),
+        ("tr-TR", 0x17, true, 'I'),
+        ("tr-TR", 0x28, false, 'i'),
+        ("tr-TR", 0x28, true, 'İ'),
+        // Bulgarian BDS is frequency-ordered, not phonetic: the `Q`
+        // position is a comma and `ъ` sits on `C`.
+        ("bg-BG", 0x10, false, ','),
+        ("bg-BG", 0x2E, false, 'ъ'),
+        ("bg-BG", 0x56, false, 'ѝ'),
+        ("it-IT", 0x27, false, 'ò'),
+        ("it-IT", 0x28, false, 'à'),
+        // Portuguese: ç on the `;` position in both orthographies,
+        // and the dead keys surfaced as spacing forms.
+        ("pt-PT", 0x27, false, 'ç'),
+        ("pt-PT", 0x1B, false, '´'),
+        ("pt-BR", 0x27, false, 'ç'),
+        ("pt-BR", 0x28, false, '~'),
     ];
     for (id, sc, shift, expected) in cases {
         let mapping = db.get(&LayoutId::from(id)).unwrap_or_else(|| {
@@ -124,6 +174,50 @@ fn wordlists_loaded_with_layouts() {
     }
 }
 
+/// Every bundled language must recognise ordinary words of its own,
+/// **including ones that carry the script's diacritics**.
+///
+/// That second half is the point. Polish and Greek shipped through a
+/// `.dic` decoded with the wrong codepage: the ASCII-only words were
+/// all present and correct, so any spot-check that stuck to `jest` or
+/// `kai` passed, while `słowo` sat in the FST as `s³owo`. Each row
+/// below therefore mixes plain words with accented / non-Latin ones.
+#[test]
+fn every_bundled_dictionary_knows_its_own_words() {
+    let db = LayoutDb::load_embedded();
+    let cases: &[(&str, &[&str])] = &[
+        ("pl-PL", &["jest", "bardzo", "dzień", "słowo", "książka"]),
+        ("cs-CZ", &["ahoj", "slovo", "dobrý", "protože", "děkuji"]),
+        ("el-GR", &["λέξη", "είναι", "καλημέρα", "ευχαριστώ"]),
+        ("he-IL", &["שלום", "מילה", "ספר", "תודה"]),
+        (
+            "tr-TR",
+            &["kitap", "merhaba", "kelime", "güzel", "teşekkür"],
+        ),
+        ("bg-BG", &["дума", "книга", "здравей", "благодаря"]),
+        ("it-IT", &["parola", "grazie", "perché", "buongiorno"]),
+        ("pt-PT", &["palavra", "obrigado", "coração", "português"]),
+        ("pt-BR", &["palavra", "obrigado", "você", "português"]),
+    ];
+    for (id, words) in cases {
+        let layout = db
+            .get(&LayoutId::from(*id))
+            .unwrap_or_else(|| panic!("layout `{id}` not loaded"));
+        let dict = layout
+            .dictionary
+            .as_ref()
+            .unwrap_or_else(|| panic!("`{id}` has no dictionary — did the fetch fail for it?"));
+        for w in *words {
+            assert!(
+                dict.contains(w),
+                "{id} dictionary missing `{w}` — if the plain-ASCII words in \
+                 this row pass and only the accented ones fail, suspect the \
+                 codepage the .dic was decoded with"
+            );
+        }
+    }
+}
+
 #[test]
 fn round_trip_hello_through_uk() {
     let db = LayoutDb::load_embedded();

crates/poltertype-layout/src/linux/shared.rs

@@ -46,6 +46,7 @@ pub fn xkb_to_bcp47(code: &str) -> Option<&'static str> {
         "br" => "pt-BR",
         "tr" => "tr-TR",
         "cz" => "cs-CZ",
+        "bg" => "bg-BG",
         "sk" => "sk-SK",
         "ro" => "ro-RO",
         "hu" => "hu-HU",
@@ -83,6 +84,7 @@ pub fn bcp47_to_xkb(bcp: &str) -> Option<&'static str> {
         "pt-BR" => "br",
         "tr-TR" => "tr",
         "cs-CZ" => "cz",
+        "bg-BG" => "bg",
         "sk-SK" => "sk",
         "ro-RO" => "ro",
         "hu-HU" => "hu",

crates/poltertype-layout/src/macos.rs

@@ -272,7 +272,23 @@ fn tis_id_to_bcp47(id: &str) -> Option<String> {
             "com.apple.keylayout.French" => "fr-FR",
             "com.apple.keylayout.Spanish" => "es-ES",
             "com.apple.keylayout.Polish" => "pl-PL",
+            "com.apple.keylayout.PolishPro" => "pl-PL",
             "com.apple.keylayout.Greek" => "el-GR",
+            "com.apple.keylayout.GreekPolytonic" => "el-GR",
+            "com.apple.keylayout.Czech" => "cs-CZ",
+            "com.apple.keylayout.Czech-QWERTY" => "cs-CZ",
+            "com.apple.keylayout.Hebrew" => "he-IL",
+            "com.apple.keylayout.Hebrew-PC" => "he-IL",
+            "com.apple.keylayout.Hebrew-QWERTY" => "he-IL",
+            "com.apple.keylayout.Turkish" => "tr-TR",
+            "com.apple.keylayout.Turkish-Standard" => "tr-TR",
+            "com.apple.keylayout.Turkish-QWERTY-PC" => "tr-TR",
+            "com.apple.keylayout.Bulgarian" => "bg-BG",
+            "com.apple.keylayout.Bulgarian-Phonetic" => "bg-BG",
+            "com.apple.keylayout.Italian" => "it-IT",
+            "com.apple.keylayout.Italian-Pro" => "it-IT",
+            "com.apple.keylayout.Portuguese" => "pt-PT",
+            "com.apple.keylayout.Brazilian" => "pt-BR",
             _ => return None,
         }
         .to_owned(),
@@ -282,6 +298,15 @@ fn tis_id_to_bcp47(id: &str) -> Option<String> {
 /// Inverse of [`tis_id_to_bcp47`] — best-effort reverse lookup so
 /// `switch_to(LayoutId("uk-UA"))` finds the right TIS source. Falls
 /// back to the input string if no entry exists.
+///
+/// Deliberately narrower than the forward table: this one names the
+/// source we *ask macOS to select*, so only the base id each language
+/// is certain to have gets an entry. A wrong guess in the forward
+/// direction costs nothing (the id falls through unmapped and stays a
+/// stable opaque `LayoutId`), whereas a wrong guess here is a switch
+/// that silently targets a source the user doesn't have. Like the
+/// rest of the macOS backend, none of this has been exercised on real
+/// hardware — see the known-gaps list in CLAUDE.md.
 fn bcp47_to_tis_id(id: &str) -> Option<String> {
     Some(
         match id {
@@ -294,6 +319,13 @@ fn bcp47_to_tis_id(id: &str) -> Option<String> {
             "es-ES" => "com.apple.keylayout.Spanish",
             "pl-PL" => "com.apple.keylayout.Polish",
             "el-GR" => "com.apple.keylayout.Greek",
+            "cs-CZ" => "com.apple.keylayout.Czech",
+            "he-IL" => "com.apple.keylayout.Hebrew",
+            "tr-TR" => "com.apple.keylayout.Turkish",
+            "bg-BG" => "com.apple.keylayout.Bulgarian",
+            "it-IT" => "com.apple.keylayout.Italian",
+            "pt-PT" => "com.apple.keylayout.Portuguese",
+            "pt-BR" => "com.apple.keylayout.Brazilian",
             _ => return None,
         }
         .to_owned(),