diff --git a/CHANGELOG.md b/CHANGELOG.md index 15faa4b..d9381ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,86 @@ All notable changes to PolterType are recorded here. The format is loosely based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project follows [Semantic Versioning](https://semver.org/). +## [Unreleased] — 0.8.0 + +### Added + +- **Nine more languages: Polish, Czech, Greek, Hebrew, Turkish, + Bulgarian, Italian, and Portuguese in both its orthographies.** + PolterType now bundles fifteen layouts instead of six. Each one is a + layout TOML plus a full dictionary, so the same detection that makes + uk-UA ↔ en-US reliable applies to cs-CZ, tr-TR, it-IT and the rest. + Nothing loads that your OS doesn't have enabled — the active-layout + filter still means a two-keyboard user pays for two. + + Layout mappings were generated from `xkeyboard-config` rather than + transcribed from keyboard pictures, and then reviewed; the trick is + written up in [docs/ADDING_A_LANGUAGE.md](docs/ADDING_A_LANGUAGE.md) + for the next person. Closes [#2]. + + Two of them carry a caveat worth reading before you expect magic: + + * **Polish** maps to exactly the same characters as US English. The + standard Polish layout is the "programmer's" one — QWERTY with + every diacritic on AltGr, which PolterType doesn't track — so + there is no pl-PL ↔ en-US mistake to correct, and none is + possible. The Polish *wordlist* still does real work: it stops + Polish prose being dragged toward whichever other layout you have + active. + * **Hebrew** ships dictionary stems without affix expansion. Its + Hunspell table encodes the clitic prefixes as 3335 prefix rules, + which expand to 60.6 M forms — a 141 MB wordlist and a far bigger + FST in every installer. Hebrew shares its script with nothing else + bundled, so plausibility already separates it and the dictionary + is a refinement. See + [data/wordlists/CREDITS.md](data/wordlists/CREDITS.md). + + **Installers grow by about 57 MB**, most of it Turkish — an + agglutinative language expands to 5.8 M forms and two 15 MB FSTs. + That is the price of the detection quality; nothing about it is + loaded at runtime unless you have a Turkish keyboard enabled. + +### Fixed + +- **Polish and Greek dictionaries would have shipped as mojibake, and + the French one hadn't been refreshed in a year.** Three separate + faults in `cargo xtask wordlists fetch`, all of which failed + quietly: + + * Hunspell declares a dictionary's encoding once, in the `.aff` — + the `.dic` has no `SET` line of its own. We looked for one in each + file separately and fell back to Latin-1 when there wasn't any, so + Polish (ISO-8859-2) and Greek (ISO-8859-7) decoded into plausible + nonsense: `słowo` became `s³owo`, which neither matches a lookup + nor trips a check. German survived only because German really is + Latin-1. The `.aff`'s declared encoding is now what decodes both + halves, ISO-8859-2 and ISO-8859-7 have real tables, and an + unrecognised or absent `SET` is an error instead of a guess. + * The French source moved upstream (`fr_FR/` → `fr_FR/dictionaries/`) + and the old URL had been 404ing. The fetch printed one stderr line + and exited 0, so the stale wordlist just stayed. Fixed, and the + command now exits non-zero when any source fails. The refreshed + French list gains 11,701 forms and loses 26, all of them elision + artifacts like `d'pick-up`. + * `FLAG num` dictionaries (comma-separated numeric affix flags) were + rejected outright. Turkish needs them — its affix table runs to + six figures of distinct flags. Now supported. + +### Changed + +- `derive_vowels` learned the vowel sets of the new languages, which + the plausibility detector counts. Two are not what a script default + would give you: `ъ` is a full vowel in Bulgarian, and Turkish's + dotless `ı` is a vowel that the bare Latin set scores as a + consonant. +- `data/wordlists/CREDITS.md` now states each dictionary's licence + per-language instead of hedging. The old blanket "most often + GPL-2-or-later or LGPL/MPL" was wrong about at least Russian, which + is BSD — and **Hebrew's Hspell is AGPL-3.0-or-later**, the strictest + thing in the tree and worth knowing before redistributing a build. + +[#2]: https://github.com/Just-Code-NET/PolterType/issues/2 + ## [0.8.0] — Windows learns to hold your keys, the AI seam comes alive, and the app gets its own face ### Added diff --git a/crates/poltertype-core/build.rs b/crates/poltertype-core/build.rs index 37ed996..d8be42d 100644 --- a/crates/poltertype-core/build.rs +++ b/crates/poltertype-core/build.rs @@ -71,6 +71,15 @@ const LAYOUTS: &[(&str, &str)] = &[ ("de_de", "de-DE"), ("es_es", "es-ES"), ("fr_fr", "fr-FR"), + ("pl_pl", "pl-PL"), + ("cs_cz", "cs-CZ"), + ("el_gr", "el-GR"), + ("he_il", "he-IL"), + ("tr_tr", "tr-TR"), + ("bg_bg", "bg-BG"), + ("it_it", "it-IT"), + ("pt_pt", "pt-PT"), + ("pt_br", "pt-BR"), ]; fn main() { diff --git a/crates/poltertype-core/src/layouts/consts.rs b/crates/poltertype-core/src/layouts/consts.rs index 9443281..38f3ae3 100644 --- a/crates/poltertype-core/src/layouts/consts.rs +++ b/crates/poltertype-core/src/layouts/consts.rs @@ -11,4 +11,7 @@ /// /// Kept in lock-step with `build.rs::LAYOUTS`. A mismatch shows up /// as a "missing TOML" warning at startup, never silently. -pub const BUNDLED_LAYOUT_STEMS: &[&str] = &["en_us", "uk_ua", "ru_ru", "de_de", "es_es", "fr_fr"]; +pub const BUNDLED_LAYOUT_STEMS: &[&str] = &[ + "en_us", "uk_ua", "ru_ru", "de_de", "es_es", "fr_fr", "pl_pl", "cs_cz", "el_gr", "he_il", + "tr_tr", "bg_bg", "it_it", "pt_pt", "pt_br", +]; diff --git a/crates/poltertype-core/src/layouts/helpers.rs b/crates/poltertype-core/src/layouts/helpers.rs index 2d24a9b..1c4ea9c 100644 --- a/crates/poltertype-core/src/layouts/helpers.rs +++ b/crates/poltertype-core/src/layouts/helpers.rs @@ -30,6 +30,22 @@ pub fn derive_vowels(id: &LayoutId, script: Script) -> Vec { "de-DE" => "aeiouäöü".chars().collect(), "es-ES" => "aeiouáéíóúü".chars().collect(), "fr-FR" => "aeiouyàâéèêëîïôûùüÿ".chars().collect(), + "pl-PL" => "aeiouyąęó".chars().collect(), + "cs-CZ" => "aeiouyáéíóúůýě".chars().collect(), + // Greek needs the accented forms spelled out: the script + // default covers the bare vowels only, and Greek marks the + // stressed vowel of almost every polysyllabic word. + "el-GR" => "αεηιουωάέήίόύώϊϋΐΰ".chars().collect(), + // `ı` (dotless i) and `ı`'s dotted twin are both vowels, as + // are ö and ü — the bare Latin default would score four of + // Turkish's eight vowels as consonants. + "tr-TR" => "aeıioöuü".chars().collect(), + // `ъ` is a full vowel in Bulgarian (unlike Russian, where the + // same glyph is a silent sign), so the generic Cyrillic set + // is wrong here in both directions. + "bg-BG" => "аеиоуъюя".chars().collect(), + "it-IT" => "aeiouàèéìíîòóùú".chars().collect(), + "pt-PT" | "pt-BR" => "aeiouáàâãéêíóôõúü".chars().collect(), _ => match script { Script::Latin => "aeiouy".chars().collect(), Script::Cyrillic => "аеиіоуюяєїыэё".chars().collect(), diff --git a/crates/poltertype-core/src/layouts/tests.rs b/crates/poltertype-core/src/layouts/tests.rs index 04bc0e2..7788f2c 100644 --- a/crates/poltertype-core/src/layouts/tests.rs +++ b/crates/poltertype-core/src/layouts/tests.rs @@ -16,7 +16,10 @@ use super::*; #[test] fn embedded_layouts_load() { let db = LayoutDb::load_embedded(); - for id in ["en-US", "uk-UA", "ru-RU", "de-DE", "es-ES", "fr-FR"] { + for id in [ + "en-US", "uk-UA", "ru-RU", "de-DE", "es-ES", "fr-FR", "pl-PL", "cs-CZ", "el-GR", "he-IL", + "tr-TR", "bg-BG", "it-IT", "pt-PT", "pt-BR", + ] { assert!( db.get(&LayoutId::from(id)).is_some(), "embedded layout `{id}` did not load" @@ -24,9 +27,26 @@ fn embedded_layouts_load() { } } +/// `build.rs::LAYOUTS` and [`BUNDLED_LAYOUT_STEMS`] are two lists that +/// have to agree; when they drift the runtime only *warns*, so nothing +/// fails until a user notices their language is missing. Pin the count +/// against what actually loads. +#[test] +fn every_bundled_stem_resolves_to_a_layout() { + let db = LayoutDb::load_embedded(); + assert_eq!( + db.len(), + BUNDLED_LAYOUT_STEMS.len(), + "{} bundled stems but {} layouts loaded — build.rs::LAYOUTS and \ + BUNDLED_LAYOUT_STEMS have drifted, or a TOML failed to parse", + BUNDLED_LAYOUT_STEMS.len(), + db.len() + ); +} + /// The active-filter feature: only requested layouts enter memory. -/// This is what saves RAM for users who don't have all six bundled -/// languages installed in the OS. +/// This is what saves RAM for users who don't have all fifteen +/// bundled languages installed in the OS — which is everyone. #[test] fn active_filter_drops_unrequested_layouts() { let want = [LayoutId::from("en-US"), LayoutId::from("uk-UA")]; @@ -86,6 +106,36 @@ fn new_languages_translate_distinctive_keys() { ("es-ES", 0x27, false, 'ñ'), ("fr-FR", 0x10, false, 'a'), ("fr-FR", 0x03, false, 'é'), + // Czech puts letters on the unshifted number row and swaps + // Y/Z like German. + ("cs-CZ", 0x03, false, 'ě'), + ("cs-CZ", 0x15, false, 'z'), + ("cs-CZ", 0x2C, false, 'y'), + // Greek: Q and W positions carry `;` and final sigma. + ("el-GR", 0x10, false, ';'), + ("el-GR", 0x11, false, 'ς'), + ("el-GR", 0x1E, false, 'α'), + // Hebrew is unicase — the letter positions are RTL glyphs. + ("he-IL", 0x1E, false, 'ש'), + ("he-IL", 0x14, false, 'א'), + // Turkish's dotless/dotted i pair, the one that bites. + ("tr-TR", 0x17, false, 'ı'), + ("tr-TR", 0x17, true, 'I'), + ("tr-TR", 0x28, false, 'i'), + ("tr-TR", 0x28, true, 'İ'), + // Bulgarian BDS is frequency-ordered, not phonetic: the `Q` + // position is a comma and `ъ` sits on `C`. + ("bg-BG", 0x10, false, ','), + ("bg-BG", 0x2E, false, 'ъ'), + ("bg-BG", 0x56, false, 'ѝ'), + ("it-IT", 0x27, false, 'ò'), + ("it-IT", 0x28, false, 'à'), + // Portuguese: ç on the `;` position in both orthographies, + // and the dead keys surfaced as spacing forms. + ("pt-PT", 0x27, false, 'ç'), + ("pt-PT", 0x1B, false, '´'), + ("pt-BR", 0x27, false, 'ç'), + ("pt-BR", 0x28, false, '~'), ]; for (id, sc, shift, expected) in cases { let mapping = db.get(&LayoutId::from(id)).unwrap_or_else(|| { @@ -124,6 +174,50 @@ fn wordlists_loaded_with_layouts() { } } +/// Every bundled language must recognise ordinary words of its own, +/// **including ones that carry the script's diacritics**. +/// +/// That second half is the point. Polish and Greek shipped through a +/// `.dic` decoded with the wrong codepage: the ASCII-only words were +/// all present and correct, so any spot-check that stuck to `jest` or +/// `kai` passed, while `słowo` sat in the FST as `s³owo`. Each row +/// below therefore mixes plain words with accented / non-Latin ones. +#[test] +fn every_bundled_dictionary_knows_its_own_words() { + let db = LayoutDb::load_embedded(); + let cases: &[(&str, &[&str])] = &[ + ("pl-PL", &["jest", "bardzo", "dzień", "słowo", "książka"]), + ("cs-CZ", &["ahoj", "slovo", "dobrý", "protože", "děkuji"]), + ("el-GR", &["λέξη", "είναι", "καλημέρα", "ευχαριστώ"]), + ("he-IL", &["שלום", "מילה", "ספר", "תודה"]), + ( + "tr-TR", + &["kitap", "merhaba", "kelime", "güzel", "teşekkür"], + ), + ("bg-BG", &["дума", "книга", "здравей", "благодаря"]), + ("it-IT", &["parola", "grazie", "perché", "buongiorno"]), + ("pt-PT", &["palavra", "obrigado", "coração", "português"]), + ("pt-BR", &["palavra", "obrigado", "você", "português"]), + ]; + for (id, words) in cases { + let layout = db + .get(&LayoutId::from(*id)) + .unwrap_or_else(|| panic!("layout `{id}` not loaded")); + let dict = layout + .dictionary + .as_ref() + .unwrap_or_else(|| panic!("`{id}` has no dictionary — did the fetch fail for it?")); + for w in *words { + assert!( + dict.contains(w), + "{id} dictionary missing `{w}` — if the plain-ASCII words in \ + this row pass and only the accented ones fail, suspect the \ + codepage the .dic was decoded with" + ); + } + } +} + #[test] fn round_trip_hello_through_uk() { let db = LayoutDb::load_embedded(); diff --git a/crates/poltertype-layout/src/linux/shared.rs b/crates/poltertype-layout/src/linux/shared.rs index da32fcb..c9dea18 100644 --- a/crates/poltertype-layout/src/linux/shared.rs +++ b/crates/poltertype-layout/src/linux/shared.rs @@ -46,6 +46,7 @@ pub fn xkb_to_bcp47(code: &str) -> Option<&'static str> { "br" => "pt-BR", "tr" => "tr-TR", "cz" => "cs-CZ", + "bg" => "bg-BG", "sk" => "sk-SK", "ro" => "ro-RO", "hu" => "hu-HU", @@ -83,6 +84,7 @@ pub fn bcp47_to_xkb(bcp: &str) -> Option<&'static str> { "pt-BR" => "br", "tr-TR" => "tr", "cs-CZ" => "cz", + "bg-BG" => "bg", "sk-SK" => "sk", "ro-RO" => "ro", "hu-HU" => "hu", diff --git a/crates/poltertype-layout/src/macos.rs b/crates/poltertype-layout/src/macos.rs index 028ebba..20ba74b 100644 --- a/crates/poltertype-layout/src/macos.rs +++ b/crates/poltertype-layout/src/macos.rs @@ -272,7 +272,23 @@ fn tis_id_to_bcp47(id: &str) -> Option { "com.apple.keylayout.French" => "fr-FR", "com.apple.keylayout.Spanish" => "es-ES", "com.apple.keylayout.Polish" => "pl-PL", + "com.apple.keylayout.PolishPro" => "pl-PL", "com.apple.keylayout.Greek" => "el-GR", + "com.apple.keylayout.GreekPolytonic" => "el-GR", + "com.apple.keylayout.Czech" => "cs-CZ", + "com.apple.keylayout.Czech-QWERTY" => "cs-CZ", + "com.apple.keylayout.Hebrew" => "he-IL", + "com.apple.keylayout.Hebrew-PC" => "he-IL", + "com.apple.keylayout.Hebrew-QWERTY" => "he-IL", + "com.apple.keylayout.Turkish" => "tr-TR", + "com.apple.keylayout.Turkish-Standard" => "tr-TR", + "com.apple.keylayout.Turkish-QWERTY-PC" => "tr-TR", + "com.apple.keylayout.Bulgarian" => "bg-BG", + "com.apple.keylayout.Bulgarian-Phonetic" => "bg-BG", + "com.apple.keylayout.Italian" => "it-IT", + "com.apple.keylayout.Italian-Pro" => "it-IT", + "com.apple.keylayout.Portuguese" => "pt-PT", + "com.apple.keylayout.Brazilian" => "pt-BR", _ => return None, } .to_owned(), @@ -282,6 +298,15 @@ fn tis_id_to_bcp47(id: &str) -> Option { /// Inverse of [`tis_id_to_bcp47`] — best-effort reverse lookup so /// `switch_to(LayoutId("uk-UA"))` finds the right TIS source. Falls /// back to the input string if no entry exists. +/// +/// Deliberately narrower than the forward table: this one names the +/// source we *ask macOS to select*, so only the base id each language +/// is certain to have gets an entry. A wrong guess in the forward +/// direction costs nothing (the id falls through unmapped and stays a +/// stable opaque `LayoutId`), whereas a wrong guess here is a switch +/// that silently targets a source the user doesn't have. Like the +/// rest of the macOS backend, none of this has been exercised on real +/// hardware — see the known-gaps list in CLAUDE.md. fn bcp47_to_tis_id(id: &str) -> Option { Some( match id { @@ -294,6 +319,13 @@ fn bcp47_to_tis_id(id: &str) -> Option { "es-ES" => "com.apple.keylayout.Spanish", "pl-PL" => "com.apple.keylayout.Polish", "el-GR" => "com.apple.keylayout.Greek", + "cs-CZ" => "com.apple.keylayout.Czech", + "he-IL" => "com.apple.keylayout.Hebrew", + "tr-TR" => "com.apple.keylayout.Turkish", + "bg-BG" => "com.apple.keylayout.Bulgarian", + "it-IT" => "com.apple.keylayout.Italian", + "pt-PT" => "com.apple.keylayout.Portuguese", + "pt-BR" => "com.apple.keylayout.Brazilian", _ => return None, } .to_owned(), diff --git a/data/layout-mappings/bg_bg.toml b/data/layout-mappings/bg_bg.toml new file mode 100644 index 0000000..edb26db --- /dev/null +++ b/data/layout-mappings/bg_bg.toml @@ -0,0 +1,78 @@ +# Layout mapping: Български (Bulgarian, traditional BDS). +# +# Keyed by Win SC Set-1 scancode (hex). BDS is *not* a phonetic +# transliteration of QWERTY the way the Russian and Ukrainian layouts +# are — letters are placed by frequency, so 0x10 (US `Q`) is a comma +# and the vowels cluster on the home row. Bulgarian typed with en-US +# active therefore comes out as Latin noise with no relation to the +# intended word, which the dictionary detector picks off easily. +# +# 0x56 — the extra key next to the left Shift on 105-key boards — +# carries `ѝ` (grave-accented и, a real Bulgarian word: "her"), so it +# is listed even though every other bundled layout skips that +# scancode. +# +# `ъ` is a vowel in Bulgarian, and the generic Cyrillic vowel set +# doesn't include it — hence the bg-BG entry in `derive_vowels`. + +id = "bg-BG" +name = "Български" +script = "Cyrillic" + +[keys] +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "?" } +0x04 = { plain = "3", shift = "+" } +0x05 = { plain = "4", shift = "\"" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "=" } +0x08 = { plain = "7", shift = ":" } +0x09 = { plain = "8", shift = "/" } +0x0A = { plain = "9", shift = "–" } +0x0B = { plain = "0", shift = "№" } +0x0C = { plain = "-", shift = "$" } +0x0D = { plain = ".", shift = "€" } + +# Top row +0x10 = { plain = ",", shift = "ы" } +0x11 = { plain = "у", shift = "У" } +0x12 = { plain = "е", shift = "Е" } +0x13 = { plain = "и", shift = "И" } +0x14 = { plain = "ш", shift = "Ш" } +0x15 = { plain = "щ", shift = "Щ" } +0x16 = { plain = "к", shift = "К" } +0x17 = { plain = "с", shift = "С" } +0x18 = { plain = "д", shift = "Д" } +0x19 = { plain = "з", shift = "З" } +0x1A = { plain = "ц", shift = "Ц" } +0x1B = { plain = ";", shift = "§" } + +# Home row +0x1E = { plain = "ь", shift = "ѝ" } +0x1F = { plain = "я", shift = "Я" } +0x20 = { plain = "а", shift = "А" } +0x21 = { plain = "о", shift = "О" } +0x22 = { plain = "ж", shift = "Ж" } +0x23 = { plain = "г", shift = "Г" } +0x24 = { plain = "т", shift = "Т" } +0x25 = { plain = "н", shift = "Н" } +0x26 = { plain = "в", shift = "В" } +0x27 = { plain = "м", shift = "М" } +0x28 = { plain = "ч", shift = "Ч" } + +# Bottom row +0x2B = { plain = "„", shift = "“" } +0x2C = { plain = "ю", shift = "Ю" } +0x2D = { plain = "й", shift = "Й" } +0x2E = { plain = "ъ", shift = "Ъ" } +0x2F = { plain = "э", shift = "Э" } +0x30 = { plain = "ф", shift = "Ф" } +0x31 = { plain = "х", shift = "Х" } +0x32 = { plain = "п", shift = "П" } +0x33 = { plain = "р", shift = "Р" } +0x34 = { plain = "л", shift = "Л" } +0x35 = { plain = "б", shift = "Б" } + +# ISO extra key (105-key boards) +0x56 = { plain = "ѝ", shift = "Ѝ" } diff --git a/data/layout-mappings/cs_cz.toml b/data/layout-mappings/cs_cz.toml new file mode 100644 index 0000000..abe6238 --- /dev/null +++ b/data/layout-mappings/cs_cz.toml @@ -0,0 +1,70 @@ +# Layout mapping: Čeština (Czech, QWERTZ). +# +# Keyed by Win SC Set-1 scancode (hex). Two things make this layout +# unusually productive for poltertype: the number row carries +# *letters* unshifted (ě š č ř ž ý á í é — the digits need Shift), +# and Y/Z are swapped as in German. A Czech word typed with en-US +# active comes out as a run of digits, which is about as loud a +# signal as the detector ever gets. +# +# Dead keys (´ acute, ˇ caron, ¨ diaeresis) are surfaced as their +# spacing equivalents. + +id = "cs-CZ" +name = "Čeština" +script = "Latin" + +[keys] +# Number row +0x02 = { plain = "+", shift = "1" } +0x03 = { plain = "ě", shift = "2" } +0x04 = { plain = "š", shift = "3" } +0x05 = { plain = "č", shift = "4" } +0x06 = { plain = "ř", shift = "5" } +0x07 = { plain = "ž", shift = "6" } +0x08 = { plain = "ý", shift = "7" } +0x09 = { plain = "á", shift = "8" } +0x0A = { plain = "í", shift = "9" } +0x0B = { plain = "é", shift = "0" } +0x0C = { plain = "=", shift = "%" } +0x0D = { plain = "´", shift = "ˇ" } + +# Top row +0x10 = { plain = "q", shift = "Q" } +0x11 = { plain = "w", shift = "W" } +0x12 = { plain = "e", shift = "E" } +0x13 = { plain = "r", shift = "R" } +0x14 = { plain = "t", shift = "T" } +0x15 = { plain = "z", shift = "Z" } +0x16 = { plain = "u", shift = "U" } +0x17 = { plain = "i", shift = "I" } +0x18 = { plain = "o", shift = "O" } +0x19 = { plain = "p", shift = "P" } +0x1A = { plain = "ú", shift = "/" } +0x1B = { plain = ")", shift = "(" } + +# Home row +0x1E = { plain = "a", shift = "A" } +0x1F = { plain = "s", shift = "S" } +0x20 = { plain = "d", shift = "D" } +0x21 = { plain = "f", shift = "F" } +0x22 = { plain = "g", shift = "G" } +0x23 = { plain = "h", shift = "H" } +0x24 = { plain = "j", shift = "J" } +0x25 = { plain = "k", shift = "K" } +0x26 = { plain = "l", shift = "L" } +0x27 = { plain = "ů", shift = "\"" } +0x28 = { plain = "§", shift = "!" } + +# Bottom row +0x2B = { plain = "¨", shift = "'" } +0x2C = { plain = "y", shift = "Y" } +0x2D = { plain = "x", shift = "X" } +0x2E = { plain = "c", shift = "C" } +0x2F = { plain = "v", shift = "V" } +0x30 = { plain = "b", shift = "B" } +0x31 = { plain = "n", shift = "N" } +0x32 = { plain = "m", shift = "M" } +0x33 = { plain = ",", shift = "?" } +0x34 = { plain = ".", shift = ":" } +0x35 = { plain = "-", shift = "_" } diff --git a/data/layout-mappings/el_gr.toml b/data/layout-mappings/el_gr.toml new file mode 100644 index 0000000..72c7382 --- /dev/null +++ b/data/layout-mappings/el_gr.toml @@ -0,0 +1,69 @@ +# Layout mapping: Ελληνικά (Greek). +# +# Keyed by Win SC Set-1 scancode (hex). Greek has no q/w/j, so those +# positions carry something else: 0x10 is `;` (which doubles as the +# Greek question mark), 0x11 is final sigma ς, and 0x24 is ξ. Accented +# vowels are composed with the dead acute on 0x27, surfaced here as +# its spacing form ´. +# +# Script alone separates Greek from every Latin layout, so detection +# is solid here even before the wordlist gets a look in. + +id = "el-GR" +name = "Ελληνικά" +script = "Greek" + +[keys] +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "@" } +0x04 = { plain = "3", shift = "#" } +0x05 = { plain = "4", shift = "$" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "^" } +0x08 = { plain = "7", shift = "&" } +0x09 = { plain = "8", shift = "*" } +0x0A = { plain = "9", shift = "(" } +0x0B = { plain = "0", shift = ")" } +0x0C = { plain = "-", shift = "_" } +0x0D = { plain = "=", shift = "+" } + +# Top row +0x10 = { plain = ";", shift = ":" } +0x11 = { plain = "ς", shift = "Σ" } +0x12 = { plain = "ε", shift = "Ε" } +0x13 = { plain = "ρ", shift = "Ρ" } +0x14 = { plain = "τ", shift = "Τ" } +0x15 = { plain = "υ", shift = "Υ" } +0x16 = { plain = "θ", shift = "Θ" } +0x17 = { plain = "ι", shift = "Ι" } +0x18 = { plain = "ο", shift = "Ο" } +0x19 = { plain = "π", shift = "Π" } +0x1A = { plain = "[", shift = "{" } +0x1B = { plain = "]", shift = "}" } + +# Home row +0x1E = { plain = "α", shift = "Α" } +0x1F = { plain = "σ", shift = "Σ" } +0x20 = { plain = "δ", shift = "Δ" } +0x21 = { plain = "φ", shift = "Φ" } +0x22 = { plain = "γ", shift = "Γ" } +0x23 = { plain = "η", shift = "Η" } +0x24 = { plain = "ξ", shift = "Ξ" } +0x25 = { plain = "κ", shift = "Κ" } +0x26 = { plain = "λ", shift = "Λ" } +0x27 = { plain = "´", shift = "¨" } +0x28 = { plain = "'", shift = "\"" } + +# Bottom row +0x2B = { plain = "\\", shift = "|" } +0x2C = { plain = "ζ", shift = "Ζ" } +0x2D = { plain = "χ", shift = "Χ" } +0x2E = { plain = "ψ", shift = "Ψ" } +0x2F = { plain = "ω", shift = "Ω" } +0x30 = { plain = "β", shift = "Β" } +0x31 = { plain = "ν", shift = "Ν" } +0x32 = { plain = "μ", shift = "Μ" } +0x33 = { plain = ",", shift = "<" } +0x34 = { plain = ".", shift = ">" } +0x35 = { plain = "/", shift = "?" } diff --git a/data/layout-mappings/he_il.toml b/data/layout-mappings/he_il.toml new file mode 100644 index 0000000..765dc3f --- /dev/null +++ b/data/layout-mappings/he_il.toml @@ -0,0 +1,76 @@ +# Layout mapping: עברית (Hebrew, standard Israeli). +# +# Keyed by Win SC Set-1 scancode (hex). Hebrew is unicase, so the +# letter keys have no upper/lower pair. What sits at the shifted +# level is what xkeyboard-config's `il` puts there — the Latin +# capital on letter positions, punctuation elsewhere. We transcribe +# that rather than invent a mapping: the field answers "which +# character does the OS produce", and on Linux this is what it +# produces. +# +# Watch two positions that are not where a Latin typist expects +# them: `,` and `.` move to 0x28 and 0x35, while the US comma/period +# keys (0x33 / 0x34) carry ת and ץ. +# +# Hebrew has no vowel letters, so the plausibility detector scores it +# on script fit alone — `derive_vowels` returns an empty set for the +# Hebrew script. The wordlist is what makes he-IL ↔ en-US sharp. + +id = "he-IL" +name = "עברית" +script = "Hebrew" + +[keys] +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "@" } +0x04 = { plain = "3", shift = "#" } +0x05 = { plain = "4", shift = "$" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "^" } +0x08 = { plain = "7", shift = "&" } +0x09 = { plain = "8", shift = "*" } +0x0A = { plain = "9", shift = ")" } +0x0B = { plain = "0", shift = "(" } +0x0C = { plain = "-", shift = "_" } +0x0D = { plain = "=", shift = "+" } + +# Top row +0x10 = { plain = "/", shift = "Q" } +0x11 = { plain = "'", shift = "W" } +0x12 = { plain = "ק", shift = "E" } +0x13 = { plain = "ר", shift = "R" } +0x14 = { plain = "א", shift = "T" } +0x15 = { plain = "ט", shift = "Y" } +0x16 = { plain = "ו", shift = "U" } +0x17 = { plain = "ן", shift = "I" } +0x18 = { plain = "ם", shift = "O" } +0x19 = { plain = "פ", shift = "P" } +0x1A = { plain = "]", shift = "}" } +0x1B = { plain = "[", shift = "{" } + +# Home row +0x1E = { plain = "ש", shift = "A" } +0x1F = { plain = "ד", shift = "S" } +0x20 = { plain = "ג", shift = "D" } +0x21 = { plain = "כ", shift = "F" } +0x22 = { plain = "ע", shift = "G" } +0x23 = { plain = "י", shift = "H" } +0x24 = { plain = "ח", shift = "J" } +0x25 = { plain = "ל", shift = "K" } +0x26 = { plain = "ך", shift = "L" } +0x27 = { plain = "ף", shift = ":" } +0x28 = { plain = ",", shift = "\"" } + +# Bottom row +0x2B = { plain = "\\", shift = "|" } +0x2C = { plain = "ז", shift = "Z" } +0x2D = { plain = "ס", shift = "X" } +0x2E = { plain = "ב", shift = "C" } +0x2F = { plain = "ה", shift = "V" } +0x30 = { plain = "נ", shift = "B" } +0x31 = { plain = "מ", shift = "N" } +0x32 = { plain = "צ", shift = "M" } +0x33 = { plain = "ת", shift = ">" } +0x34 = { plain = "ץ", shift = "<" } +0x35 = { plain = ".", shift = "?" } diff --git a/data/layout-mappings/it_it.toml b/data/layout-mappings/it_it.toml new file mode 100644 index 0000000..a2a86ff --- /dev/null +++ b/data/layout-mappings/it_it.toml @@ -0,0 +1,66 @@ +# Layout mapping: Italiano (Italian). +# +# Keyed by Win SC Set-1 scancode (hex). The accented vowels sit on the +# punctuation positions right of the home row — è à ò ù, plus ì on the +# number row — and, unlike French or German, the base levels carry no +# dead keys at all: every accent is its own key. That makes it-IT ↔ +# en-US a clean swap on exactly those positions. + +id = "it-IT" +name = "Italiano" +script = "Latin" + +[keys] +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "\"" } +0x04 = { plain = "3", shift = "£" } +0x05 = { plain = "4", shift = "$" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "&" } +0x08 = { plain = "7", shift = "/" } +0x09 = { plain = "8", shift = "(" } +0x0A = { plain = "9", shift = ")" } +0x0B = { plain = "0", shift = "=" } +0x0C = { plain = "'", shift = "?" } +0x0D = { plain = "ì", shift = "^" } + +# Top row +0x10 = { plain = "q", shift = "Q" } +0x11 = { plain = "w", shift = "W" } +0x12 = { plain = "e", shift = "E" } +0x13 = { plain = "r", shift = "R" } +0x14 = { plain = "t", shift = "T" } +0x15 = { plain = "y", shift = "Y" } +0x16 = { plain = "u", shift = "U" } +0x17 = { plain = "i", shift = "I" } +0x18 = { plain = "o", shift = "O" } +0x19 = { plain = "p", shift = "P" } +0x1A = { plain = "è", shift = "é" } +0x1B = { plain = "+", shift = "*" } + +# Home row +0x1E = { plain = "a", shift = "A" } +0x1F = { plain = "s", shift = "S" } +0x20 = { plain = "d", shift = "D" } +0x21 = { plain = "f", shift = "F" } +0x22 = { plain = "g", shift = "G" } +0x23 = { plain = "h", shift = "H" } +0x24 = { plain = "j", shift = "J" } +0x25 = { plain = "k", shift = "K" } +0x26 = { plain = "l", shift = "L" } +0x27 = { plain = "ò", shift = "ç" } +0x28 = { plain = "à", shift = "°" } + +# Bottom row +0x2B = { plain = "ù", shift = "§" } +0x2C = { plain = "z", shift = "Z" } +0x2D = { plain = "x", shift = "X" } +0x2E = { plain = "c", shift = "C" } +0x2F = { plain = "v", shift = "V" } +0x30 = { plain = "b", shift = "B" } +0x31 = { plain = "n", shift = "N" } +0x32 = { plain = "m", shift = "M" } +0x33 = { plain = ",", shift = ";" } +0x34 = { plain = ".", shift = ":" } +0x35 = { plain = "-", shift = "_" } diff --git a/data/layout-mappings/pl_pl.toml b/data/layout-mappings/pl_pl.toml new file mode 100644 index 0000000..9431a1e --- /dev/null +++ b/data/layout-mappings/pl_pl.toml @@ -0,0 +1,78 @@ +# Layout mapping: Polski (Polish — the "programmer's" layout, which +# is the one essentially every Polish user actually has enabled). +# +# Keyed by Win SC Set-1 scancode (hex). The notable thing about this +# file: at the unshifted and shifted levels it is *identical to +# en-US*. Every Polish diacritic (ą ć ę ł ń ó ś ź ż) lives on AltGr, +# and poltertype tracks only plain / shift — so there is deliberately +# no pl-PL ↔ en-US correction signal here, and none is possible. +# +# It still earns its place. The file gives Polish an id the OS +# switcher can target, and it is what a Polish wordlist hangs off: +# without it, Polish prose reads as gibberish to the detector and +# gets pulled toward whatever other layout is active. Against a +# different script (uk-UA, ru-RU, el-GR …) detection works normally. +# +# The "Polish (QWERTZ)" typewriter variant does put the diacritics on +# the base level, but almost nobody types on it. If you do, copy this +# file to /poltertype/layouts/ and edit — a user TOML +# with the same id wins over the bundled one. + +id = "pl-PL" +name = "Polski" +script = "Latin" + +[keys] +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "@" } +0x04 = { plain = "3", shift = "#" } +0x05 = { plain = "4", shift = "$" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "^" } +0x08 = { plain = "7", shift = "&" } +0x09 = { plain = "8", shift = "*" } +0x0A = { plain = "9", shift = "(" } +0x0B = { plain = "0", shift = ")" } +0x0C = { plain = "-", shift = "_" } +0x0D = { plain = "=", shift = "+" } + +# Top row +0x10 = { plain = "q", shift = "Q" } +0x11 = { plain = "w", shift = "W" } +0x12 = { plain = "e", shift = "E" } +0x13 = { plain = "r", shift = "R" } +0x14 = { plain = "t", shift = "T" } +0x15 = { plain = "y", shift = "Y" } +0x16 = { plain = "u", shift = "U" } +0x17 = { plain = "i", shift = "I" } +0x18 = { plain = "o", shift = "O" } +0x19 = { plain = "p", shift = "P" } +0x1A = { plain = "[", shift = "{" } +0x1B = { plain = "]", shift = "}" } + +# Home row +0x1E = { plain = "a", shift = "A" } +0x1F = { plain = "s", shift = "S" } +0x20 = { plain = "d", shift = "D" } +0x21 = { plain = "f", shift = "F" } +0x22 = { plain = "g", shift = "G" } +0x23 = { plain = "h", shift = "H" } +0x24 = { plain = "j", shift = "J" } +0x25 = { plain = "k", shift = "K" } +0x26 = { plain = "l", shift = "L" } +0x27 = { plain = ";", shift = ":" } +0x28 = { plain = "'", shift = "\"" } + +# Bottom row +0x2B = { plain = "\\", shift = "|" } +0x2C = { plain = "z", shift = "Z" } +0x2D = { plain = "x", shift = "X" } +0x2E = { plain = "c", shift = "C" } +0x2F = { plain = "v", shift = "V" } +0x30 = { plain = "b", shift = "B" } +0x31 = { plain = "n", shift = "N" } +0x32 = { plain = "m", shift = "M" } +0x33 = { plain = ",", shift = "<" } +0x34 = { plain = ".", shift = ">" } +0x35 = { plain = "/", shift = "?" } diff --git a/data/layout-mappings/pt_br.toml b/data/layout-mappings/pt_br.toml new file mode 100644 index 0000000..38b1c9c --- /dev/null +++ b/data/layout-mappings/pt_br.toml @@ -0,0 +1,69 @@ +# Layout mapping: Português (Brasil, ABNT2). +# +# Keyed by Win SC Set-1 scancode (hex). ABNT2 keeps the US number row +# and the QWERTY block, then differs where it counts: ç on 0x27, the +# acute/grave dead key on 0x1A, tilde/circumflex on 0x28 (both +# surfaced as spacing forms). +# +# ABNT2 also has a 107th key — the `/` next to the right Shift — that +# no other layout here has. It falls outside the scancode block +# poltertype tracks and is deliberately left out. + +id = "pt-BR" +name = "Português (Brasil)" +script = "Latin" + +[keys] +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "@" } +0x04 = { plain = "3", shift = "#" } +0x05 = { plain = "4", shift = "$" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "¨" } +0x08 = { plain = "7", shift = "&" } +0x09 = { plain = "8", shift = "*" } +0x0A = { plain = "9", shift = "(" } +0x0B = { plain = "0", shift = ")" } +0x0C = { plain = "-", shift = "_" } +0x0D = { plain = "=", shift = "+" } + +# Top row +0x10 = { plain = "q", shift = "Q" } +0x11 = { plain = "w", shift = "W" } +0x12 = { plain = "e", shift = "E" } +0x13 = { plain = "r", shift = "R" } +0x14 = { plain = "t", shift = "T" } +0x15 = { plain = "y", shift = "Y" } +0x16 = { plain = "u", shift = "U" } +0x17 = { plain = "i", shift = "I" } +0x18 = { plain = "o", shift = "O" } +0x19 = { plain = "p", shift = "P" } +0x1A = { plain = "´", shift = "`" } +0x1B = { plain = "[", shift = "{" } + +# Home row +0x1E = { plain = "a", shift = "A" } +0x1F = { plain = "s", shift = "S" } +0x20 = { plain = "d", shift = "D" } +0x21 = { plain = "f", shift = "F" } +0x22 = { plain = "g", shift = "G" } +0x23 = { plain = "h", shift = "H" } +0x24 = { plain = "j", shift = "J" } +0x25 = { plain = "k", shift = "K" } +0x26 = { plain = "l", shift = "L" } +0x27 = { plain = "ç", shift = "Ç" } +0x28 = { plain = "~", shift = "^" } + +# Bottom row +0x2B = { plain = "]", shift = "}" } +0x2C = { plain = "z", shift = "Z" } +0x2D = { plain = "x", shift = "X" } +0x2E = { plain = "c", shift = "C" } +0x2F = { plain = "v", shift = "V" } +0x30 = { plain = "b", shift = "B" } +0x31 = { plain = "n", shift = "N" } +0x32 = { plain = "m", shift = "M" } +0x33 = { plain = ",", shift = "<" } +0x34 = { plain = ".", shift = ">" } +0x35 = { plain = ";", shift = ":" } diff --git a/data/layout-mappings/pt_pt.toml b/data/layout-mappings/pt_pt.toml new file mode 100644 index 0000000..4bf9ce8 --- /dev/null +++ b/data/layout-mappings/pt_pt.toml @@ -0,0 +1,69 @@ +# Layout mapping: Português (Portugal). +# +# Keyed by Win SC Set-1 scancode (hex). ç has a dedicated key at 0x27 +# (where en-US has `;`), and the accents are dead keys: acute/grave on +# 0x1B, tilde/circumflex on 0x2B — surfaced here as their spacing +# equivalents ´ ` ~ ^. +# +# See pt_br.toml for the Brazilian ABNT2 keyboard. The two are +# different arrangements of the same alphabet, not variants of one +# layout, so both ship. + +id = "pt-PT" +name = "Português" +script = "Latin" + +[keys] +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "\"" } +0x04 = { plain = "3", shift = "#" } +0x05 = { plain = "4", shift = "$" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "&" } +0x08 = { plain = "7", shift = "/" } +0x09 = { plain = "8", shift = "(" } +0x0A = { plain = "9", shift = ")" } +0x0B = { plain = "0", shift = "=" } +0x0C = { plain = "'", shift = "?" } +0x0D = { plain = "«", shift = "»" } + +# Top row +0x10 = { plain = "q", shift = "Q" } +0x11 = { plain = "w", shift = "W" } +0x12 = { plain = "e", shift = "E" } +0x13 = { plain = "r", shift = "R" } +0x14 = { plain = "t", shift = "T" } +0x15 = { plain = "y", shift = "Y" } +0x16 = { plain = "u", shift = "U" } +0x17 = { plain = "i", shift = "I" } +0x18 = { plain = "o", shift = "O" } +0x19 = { plain = "p", shift = "P" } +0x1A = { plain = "+", shift = "*" } +0x1B = { plain = "´", shift = "`" } + +# Home row +0x1E = { plain = "a", shift = "A" } +0x1F = { plain = "s", shift = "S" } +0x20 = { plain = "d", shift = "D" } +0x21 = { plain = "f", shift = "F" } +0x22 = { plain = "g", shift = "G" } +0x23 = { plain = "h", shift = "H" } +0x24 = { plain = "j", shift = "J" } +0x25 = { plain = "k", shift = "K" } +0x26 = { plain = "l", shift = "L" } +0x27 = { plain = "ç", shift = "Ç" } +0x28 = { plain = "º", shift = "ª" } + +# Bottom row +0x2B = { plain = "~", shift = "^" } +0x2C = { plain = "z", shift = "Z" } +0x2D = { plain = "x", shift = "X" } +0x2E = { plain = "c", shift = "C" } +0x2F = { plain = "v", shift = "V" } +0x30 = { plain = "b", shift = "B" } +0x31 = { plain = "n", shift = "N" } +0x32 = { plain = "m", shift = "M" } +0x33 = { plain = ",", shift = ";" } +0x34 = { plain = ".", shift = ":" } +0x35 = { plain = "-", shift = "_" } diff --git a/data/layout-mappings/tr_tr.toml b/data/layout-mappings/tr_tr.toml new file mode 100644 index 0000000..e0079ae --- /dev/null +++ b/data/layout-mappings/tr_tr.toml @@ -0,0 +1,72 @@ +# Layout mapping: Türkçe (Turkish-Q — the common one; Turkish-F is a +# different typewriter layout and is not covered here). +# +# Keyed by Win SC Set-1 scancode (hex). The alphabet's extra letters +# take punctuation positions: ı ğ ü ş i ö ç. +# +# Note the dotted/dotless pair — 0x17 is ı/I (dotless) and 0x28 is +# i/İ (dotted). That pair is also the one place Turkish text bites +# downstream: Unicode's *default* case folding maps I→i and İ→"i" plus +# a combining dot, not the Turkish way round. Both the build-time FST +# and the runtime lookup use that same default folding, so the two +# agree and lookups match — which is the property that matters. Don't +# "fix" one side on its own. + +id = "tr-TR" +name = "Türkçe" +script = "Latin" + +[keys] +# Number row +0x02 = { plain = "1", shift = "!" } +0x03 = { plain = "2", shift = "'" } +0x04 = { plain = "3", shift = "^" } +0x05 = { plain = "4", shift = "+" } +0x06 = { plain = "5", shift = "%" } +0x07 = { plain = "6", shift = "&" } +0x08 = { plain = "7", shift = "/" } +0x09 = { plain = "8", shift = "(" } +0x0A = { plain = "9", shift = ")" } +0x0B = { plain = "0", shift = "=" } +0x0C = { plain = "*", shift = "?" } +0x0D = { plain = "-", shift = "_" } + +# Top row +0x10 = { plain = "q", shift = "Q" } +0x11 = { plain = "w", shift = "W" } +0x12 = { plain = "e", shift = "E" } +0x13 = { plain = "r", shift = "R" } +0x14 = { plain = "t", shift = "T" } +0x15 = { plain = "y", shift = "Y" } +0x16 = { plain = "u", shift = "U" } +0x17 = { plain = "ı", shift = "I" } +0x18 = { plain = "o", shift = "O" } +0x19 = { plain = "p", shift = "P" } +0x1A = { plain = "ğ", shift = "Ğ" } +0x1B = { plain = "ü", shift = "Ü" } + +# Home row +0x1E = { plain = "a", shift = "A" } +0x1F = { plain = "s", shift = "S" } +0x20 = { plain = "d", shift = "D" } +0x21 = { plain = "f", shift = "F" } +0x22 = { plain = "g", shift = "G" } +0x23 = { plain = "h", shift = "H" } +0x24 = { plain = "j", shift = "J" } +0x25 = { plain = "k", shift = "K" } +0x26 = { plain = "l", shift = "L" } +0x27 = { plain = "ş", shift = "Ş" } +0x28 = { plain = "i", shift = "İ" } + +# Bottom row +0x2B = { plain = ",", shift = ";" } +0x2C = { plain = "z", shift = "Z" } +0x2D = { plain = "x", shift = "X" } +0x2E = { plain = "c", shift = "C" } +0x2F = { plain = "v", shift = "V" } +0x30 = { plain = "b", shift = "B" } +0x31 = { plain = "n", shift = "N" } +0x32 = { plain = "m", shift = "M" } +0x33 = { plain = "ö", shift = "Ö" } +0x34 = { plain = "ç", shift = "Ç" } +0x35 = { plain = ".", shift = ":" } diff --git a/data/wordlists/CREDITS.md b/data/wordlists/CREDITS.md index 62f5ace..33d81f4 100644 --- a/data/wordlists/CREDITS.md +++ b/data/wordlists/CREDITS.md @@ -5,10 +5,13 @@ The dictionaries shipped in this directory power the license; we list them here so contributors and downstream packagers can audit the project's overall license posture. -The PolterType binary is MIT-licensed — embedding these wordlists -is permissible because every source allows verbatim redistribution -under non-restrictive terms (MIT-compatible) or under terms that -attach only to the data, not to consuming code (MPL). +The PolterType binary is MIT-licensed, and stays MIT: every source +here permits verbatim redistribution, and each licence attaches to +**the wordlist data**, not to the code that reads it. What differs +between sources is what redistributing *the bundle* obliges you to do +— which ranges from "nothing" (Public Domain, BSD, MPL) to +AGPL-3.0-or-later for Hebrew. The table below is the per-language +detail; read it before shipping a build to anyone else. ## en_us.txt @@ -35,30 +38,77 @@ attach only to the data, not to consuming code (MPL). de-duplicated. * **Size:** ~333k entries. -## ru_ru.txt, de_de.txt, es_es.txt, fr_fr.txt - -* **Source:** [LibreOffice/dictionaries](https://github.com/LibreOffice/dictionaries): - * `ru_RU/ru_RU.dic` - * `de/de_DE_frami.dic` - * `es/es_ES.dic` - * `fr_FR/fr.dic` -* **License:** Each upstream dictionary ships its own licence. As of - 2026 LibreOffice's bundled Hunspell dictionaries are most often - GPL-2-or-later or LGPL/MPL — review the per-language `README*.txt` - next to the `.dic` for the exact terms before redistributing a - built PolterType binary outside personal use. The licence applies - to **the wordlist data** only; the PolterType engine remains MIT. -* **Processing:** the same generic Hunspell pipeline as `uk_ua.txt` - (strip affix flags, lowercase, keep letter-only stems). -* **Size:** varies — typically 100–300k stems each, expanded to - millions of inflected forms by Hunspell at runtime, but we keep - only the stems for FST size. -* **Status (2026-05-06):** layout TOML and short-stop word list - shipped; the bulk dictionaries are populated by `cargo xtask - wordlists fetch`. Until run, the FST is empty and detection falls - back to plausibility scoring (vowel ratio + script fit) — which - works well for cross-script pairs (RU vs EN) and adequately for - same-script pairs (DE vs EN, …) on prose, less so on short tokens. +## The other LibreOffice-derived lists + +Everything below comes from +[LibreOffice/dictionaries](https://github.com/LibreOffice/dictionaries) +through one pipeline: download `.dic` + `.aff`, expand the affix rules +into surface forms (`xtask/src/hunspell/`), lowercase, keep entries +made of letters plus apostrophe and hyphen, sort, de-duplicate, gzip. +The exact URLs are in `xtask/src/consts.rs::HUNSPELL_SOURCES`; re-run +them with `cargo xtask wordlists fetch`. + +**Every licence in this table attaches to the wordlist data only.** +The PolterType engine stays MIT. What the licence constrains is +redistribution of a *built binary that embeds these lists* — read the +row before shipping PolterType anywhere but your own machine. Each +entry was checked against the upstream licence file on 2026-08-01; +the generic "probably GPL, go and look" hedge that stood here before +was wrong about at least Russian, which is BSD. + +| Stem | Upstream `.dic` | Licence | Forms | +|---|---|---|---| +| `ru_ru` | `ru_RU/ru_RU.dic` | BSD-style, 4-clause (A. I. Lebedev) | 1.4 M | +| `de_de` | `de/de_DE_frami.dic` | GPL-2 or GPL-3 | 789 k | +| `es_es` | `es/es_ES.dic` | GPL-3+ / LGPL-3+ / MPL-1.1+ (disjoint; pick one) | 652 k | +| `fr_fr` | `fr_FR/dictionaries/fr.dic` | MPL-2.0 (Dicollecte / Grammalecte) | 2.2 M | +| `pl_pl` | `pl_PL/pl_PL.dic` | GPL / LGPL / MPL / Apache-2.0 / CC-SA (pick one) | 4.0 M | +| `cs_cz` | `cs_CZ/cs_CZ.dic` | GPL, with GNU FDL 1.1 portions | 3.1 M | +| `el_gr` | `el_GR/el_GR.dic` | MPL-1.1 / GPL-2.0 / LGPL-2.1 (pick one) | 827 k | +| `he_il` | `he_IL/he_IL.dic` | **AGPL-3.0-or-later** (Hspell) — see below | 469 k | +| `tr_tr` | `tr_TR/tr_TR.dic` | MPL-2.0 | 5.8 M | +| `bg_bg` | `bg_BG/bg_BG.dic` | GPL-2 | 2.1 M | +| `it_it` | `it_IT/it_IT.dic` | GPL-3 | 3.3 M | +| `pt_pt` | `pt_PT/pt_PT.dic` | GPL-2-or-later | 1.3 M | +| `pt_br` | `pt_BR/pt_BR.dic` | LGPL-3 / MPL | 8.1 M | + +### Hebrew is the row to look at twice + +Hspell is **AGPL-3.0-or-later** — the strictest licence in this tree +and the only one with a network clause. It still attaches to the data +rather than to the engine, and PolterType builds no network service +out of it, so bundling it in a locally-installed desktop app is +within the terms. But it is a materially heavier obligation than +anything else here, and a downstream packager or a company shipping a +modified build needs to know it is present. + +If it ever becomes awkward, `he_il` is the cheapest language in the +set to drop: nothing else bundled uses the Hebrew script, so the +plausibility detector separates he-IL from every other layout on +script alone and the dictionary is a refinement, not the load-bearing +signal. + +### Hebrew also ships stems, not surface forms + +`he_IL.aff` is not an inflection table — it is 3335 **prefix** rules +and zero suffix rules, encoding the clitic particles (ב ל כ מ ש ו ה, +and the legal pairs of them) as affixes. Expanding it faithfully +produces 60.6 M forms: a 141 MB `.txt.gz` here and a far larger FST +inside every installer. So `he_il` alone is processed with +`ExpandMode::StemsOnly` (`xtask/src/enums.rs`) and ships its 469 k +stems. A Hebrew word typed with a clitic prefix therefore misses the +dictionary and falls through to plausibility — which, for a script +nothing else here shares, is enough. + +### Polish carries the wordlist on its own + +`pl_pl.toml` maps to exactly the same characters as `en_us.toml`: the +standard Polish "programmer's" layout is US QWERTY with the diacritics +on AltGr, which PolterType does not track. So there is no pl-PL ↔ +en-US correction to make, and the Polish wordlist exists for the other +direction — stopping Polish prose from being dragged toward whichever +other layout the user has active. See the header of +`data/layout-mappings/pl_pl.toml`. ## Refreshing diff --git a/data/wordlists/bg_bg-stop.txt b/data/wordlists/bg_bg-stop.txt new file mode 100644 index 0000000..d943882 --- /dev/null +++ b/data/wordlists/bg_bg-stop.txt @@ -0,0 +1,58 @@ +# Hand-curated 1- and 2-letter Bulgarian stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `bg_bg-stop.txt` → merged into THIS list at startup / reload +# * `bg_bg-extras.txt` → merged into the user-overlay HashSet +# * `bg_bg.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — prepositions, conjunctions and the copula. +а +в +е +и +с +у +я + +# 2-letter — clitic pronouns, prepositions, particles. +аз +бе +би +ви +ги +го +да +до +за +из +им +ли +ме +ми +му +на +не +ни +но +от +по +са +се +си +та +те +ти +то +тя +че +ще diff --git a/data/wordlists/bg_bg.txt.gz b/data/wordlists/bg_bg.txt.gz new file mode 100644 index 0000000..8611639 Binary files /dev/null and b/data/wordlists/bg_bg.txt.gz differ diff --git a/data/wordlists/cs_cz-stop.txt b/data/wordlists/cs_cz-stop.txt new file mode 100644 index 0000000..f335f6e --- /dev/null +++ b/data/wordlists/cs_cz-stop.txt @@ -0,0 +1,62 @@ +# Hand-curated 1- and 2-letter Czech stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `cs_cz-stop.txt` → merged into THIS list at startup / reload +# * `cs_cz-extras.txt` → merged into the user-overlay HashSet +# * `cs_cz.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — Czech prepositions are single consonants, so +# this list is unusually long for a Latin-script language. +a +i +k +o +s +u +v +z + +# 2-letter — pronouns, prepositions, particles. +až +by +co +či +do +ho +je +jí +jo +ke +ku +mi +mu +my +na +ne +ní +no +od +on +po +se +si +ta +tě +ti +to +ty +už +ve +za +ze +že diff --git a/data/wordlists/cs_cz.txt.gz b/data/wordlists/cs_cz.txt.gz new file mode 100644 index 0000000..b2930e5 Binary files /dev/null and b/data/wordlists/cs_cz.txt.gz differ diff --git a/data/wordlists/el_gr-stop.txt b/data/wordlists/el_gr-stop.txt new file mode 100644 index 0000000..c0be639 --- /dev/null +++ b/data/wordlists/el_gr-stop.txt @@ -0,0 +1,42 @@ +# Hand-curated 1- and 2-letter Greek stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `el_gr-stop.txt` → merged into THIS list at startup / reload +# * `el_gr-extras.txt` → merged into the user-overlay HashSet +# * `el_gr.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — the two singular articles and `or`. +η +ο +ή + +# 2-letter — articles, pronouns, prepositions, the future +# and negation particles. +αν +ας +δε +εν +θα +κι +μα +με +μη +να +οι +σα +σε +τα +τη +τι +το +ως diff --git a/data/wordlists/el_gr.txt.gz b/data/wordlists/el_gr.txt.gz new file mode 100644 index 0000000..173c0fe Binary files /dev/null and b/data/wordlists/el_gr.txt.gz differ diff --git a/data/wordlists/fr_fr.txt.gz b/data/wordlists/fr_fr.txt.gz index 87d0245..e125283 100644 Binary files a/data/wordlists/fr_fr.txt.gz and b/data/wordlists/fr_fr.txt.gz differ diff --git a/data/wordlists/he_il-stop.txt b/data/wordlists/he_il-stop.txt new file mode 100644 index 0000000..abe6352 --- /dev/null +++ b/data/wordlists/he_il-stop.txt @@ -0,0 +1,65 @@ +# Hand-curated 1- and 2-letter Hebrew stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `he_il-stop.txt` → merged into THIS list at startup / reload +# * `he_il-extras.txt` → merged into the user-overlay HashSet +# * `he_il.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — deliberately empty. Hebrew's one-letter forms +# (ב ל כ מ ש ו ה) are bound prefixes that attach to the +# following word; they are not typed as standalone tokens, +# and listing them would veto legitimate corrections. + +# 2-letter — a dense band in Hebrew: particles, pronouns, +# prepositions and a good number of ordinary nouns. +אב +אז +אח +אי +אך +אל +אם +אף +את +או +בה +בו +בן +בת +גם +דם +הם +הן +זה +זו +יד +יש +כי +כל +כן +לא +לב +לו +לי +לך +מה +מי +מן +עד +על +עם +עץ +פה +רק +שם +של diff --git a/data/wordlists/he_il.txt.gz b/data/wordlists/he_il.txt.gz new file mode 100644 index 0000000..f0afc2f Binary files /dev/null and b/data/wordlists/he_il.txt.gz differ diff --git a/data/wordlists/it_it-stop.txt b/data/wordlists/it_it-stop.txt new file mode 100644 index 0000000..3813bd9 --- /dev/null +++ b/data/wordlists/it_it-stop.txt @@ -0,0 +1,65 @@ +# Hand-curated 1- and 2-letter Italian stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `it_it-stop.txt` → merged into THIS list at startup / reload +# * `it_it-extras.txt` → merged into the user-overlay HashSet +# * `it_it.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — prepositions, conjunctions, the plural +# article, and `è` (is), which carries its accent. +a +e +è +i +o + +# 2-letter — articles, clitics, prepositions and the short +# irregular verb forms (ha, ho, fa, fu, sa, so, va). +ai +al +ce +ci +da +dà +di +do +fa +fu +ha +ho +il +in +là +le +li +lo +ma +me +mi +ne +né +no +re +sa +se +si +sì +so +su +te +tè +ti +tu +un +va +vi diff --git a/data/wordlists/it_it.txt.gz b/data/wordlists/it_it.txt.gz new file mode 100644 index 0000000..ff538f3 Binary files /dev/null and b/data/wordlists/it_it.txt.gz differ diff --git a/data/wordlists/pl_pl-stop.txt b/data/wordlists/pl_pl-stop.txt new file mode 100644 index 0000000..428d95b --- /dev/null +++ b/data/wordlists/pl_pl-stop.txt @@ -0,0 +1,60 @@ +# Hand-curated 1- and 2-letter Polish stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `pl_pl-stop.txt` → merged into THIS list at startup / reload +# * `pl_pl-extras.txt` → merged into the user-overlay HashSet +# * `pl_pl.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — conjunctions and prepositions, all real prose. +a +i +o +u +w +z + +# 2-letter — pronouns, prepositions, particles, conjunctions. +aż +bo +by +ci +co +do +go +im +iż +je +ku +ma +mi +mu +my +na +ni +no +od +on +po +są +ta +te +to +tu +ty +we +wy +za +ze +że +ów +oś diff --git a/data/wordlists/pl_pl.txt.gz b/data/wordlists/pl_pl.txt.gz new file mode 100644 index 0000000..e8ac74f Binary files /dev/null and b/data/wordlists/pl_pl.txt.gz differ diff --git a/data/wordlists/pt_br-stop.txt b/data/wordlists/pt_br-stop.txt new file mode 100644 index 0000000..524b1a3 --- /dev/null +++ b/data/wordlists/pt_br-stop.txt @@ -0,0 +1,61 @@ +# Hand-curated 1- and 2-letter Portuguese (Brazilian) stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `pt_br-stop.txt` → merged into THIS list at startup / reload +# * `pt_br-extras.txt` → merged into the user-overlay HashSet +# * `pt_br.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — articles, prepositions, conjunctions. +a +à +e +é +o + +# 2-letter — contractions, clitics, and the short verb +# forms. Identical to pt_pt-stop.txt by design: the two +# orthographies differ on longer words, not on these. +ai +aí +ao +as +às +cá +da +dá +de +dê +do +dó +em +eu +fé +há +ir +já +lá +me +na +no +nó +os +ou +pé +se +só +te +tu +um +vá +vê +vi diff --git a/data/wordlists/pt_br.txt.gz b/data/wordlists/pt_br.txt.gz new file mode 100644 index 0000000..356e685 Binary files /dev/null and b/data/wordlists/pt_br.txt.gz differ diff --git a/data/wordlists/pt_pt-stop.txt b/data/wordlists/pt_pt-stop.txt new file mode 100644 index 0000000..6e798d2 --- /dev/null +++ b/data/wordlists/pt_pt-stop.txt @@ -0,0 +1,62 @@ +# Hand-curated 1- and 2-letter Portuguese stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `pt_pt-stop.txt` → merged into THIS list at startup / reload +# * `pt_pt-extras.txt` → merged into the user-overlay HashSet +# * `pt_pt.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — articles, prepositions, conjunctions. +a +à +e +é +o + +# 2-letter — contractions, clitics, and the short verb +# forms. Shared with pt_br-stop.txt: European and +# Brazilian Portuguese have the same short-word lexicon, +# and the orthographic agreement left it untouched. +ai +aí +ao +as +às +cá +da +dá +de +dê +do +dó +em +eu +fé +há +ir +já +lá +me +na +no +nó +os +ou +pé +se +só +te +tu +um +vá +vê +vi diff --git a/data/wordlists/pt_pt.txt.gz b/data/wordlists/pt_pt.txt.gz new file mode 100644 index 0000000..b29dff3 Binary files /dev/null and b/data/wordlists/pt_pt.txt.gz differ diff --git a/data/wordlists/tr_tr-stop.txt b/data/wordlists/tr_tr-stop.txt new file mode 100644 index 0000000..b912cdb --- /dev/null +++ b/data/wordlists/tr_tr-stop.txt @@ -0,0 +1,56 @@ +# Hand-curated 1- and 2-letter Turkish stop words (compile-time +# baseline). +# +# `DictionaryDetector` consults this list **instead of** the embedded +# FST for buffers ≤2 letters long, because the FST ships too many +# spurious short "words" that would block real corrections. +# +# Runtime extension (no rebuild needed): drop matching files into +# `/poltertype/wordlists/`: +# * `tr_tr-stop.txt` → merged into THIS list at startup / reload +# * `tr_tr-extras.txt` → merged into the user-overlay HashSet +# * `tr_tr.txt` → same as -extras, just a different filename +# All three accept the same format and reload via "Reload Settings". +# +# Format: one lowercase word per line; blank lines and `#` comments +# ignored. Edit freely — entries are pure data. + +# 1-letter — only the third-person pronoun / distal +# demonstrative. +o + +# 2-letter — particles (including all four vowel-harmony +# variants of the question clitic), plus the short nouns +# and verb stems common enough to matter. +ad +ak +al +an +at +ay +az +bu +da +de +el +en +ev +iç +il +ip +iş +ki +mi +mı +mu +mü +ne +ok +ol +on +öz +su +şu +ün +ve +ya diff --git a/data/wordlists/tr_tr.txt.gz b/data/wordlists/tr_tr.txt.gz new file mode 100644 index 0000000..3f6eb99 Binary files /dev/null and b/data/wordlists/tr_tr.txt.gz differ diff --git a/docs/ADDING_A_LANGUAGE.md b/docs/ADDING_A_LANGUAGE.md index 64ca9e4..e507d0e 100644 --- a/docs/ADDING_A_LANGUAGE.md +++ b/docs/ADDING_A_LANGUAGE.md @@ -42,12 +42,31 @@ script = "Latin" # Latin / Cyrillic / Greek / Armenian / Hebrew / Arabi ### Which scancodes matter Cover at least the alphanumeric block plus the punctuation that -serves as a word boundary in your language. The bundled files -(`en_us.toml`, `uk_ua.toml`, `ru_ru.toml`, `de_de.toml`, `es_es.toml`, -`fr_fr.toml`) are good copy-paste templates. The detector ignores +serves as a word boundary in your language. The fifteen bundled files +in `data/layout-mappings/` are good copy-paste templates — they all +list the same 46 scancodes (number row, three letter rows, and the +key left of `Z`), plus `0x56` where a layout puts a letter on the +extra 105-key board key, as Bulgarian does. The detector ignores scancodes you don't list, so omitting `0x29` (backtick) for a layout that doesn't use it is fine. +### Getting the mapping right without guessing + +On Linux the authoritative answer is already installed. `xkbcommon` +knows what every layout produces per key, so rather than transcribing +a keyboard picture, compile the keymap and read the levels off it — +`xkb_keymap_key_by_name` for `AD01`…`AB10`, then +`xkb_keymap_key_get_syms_by_level` for group 1 levels 1 and 2, then +`xkb_keysym_to_utf8`. XKB key names map onto Set-1 scancodes in three +straight runs: `AE01`–`AE12` → `0x02`–`0x0D`, `AD01`–`AD12` → +`0x10`–`0x1B`, `AC01`–`AC11` → `0x1E`–`0x28`, `AB01`–`AB10` → +`0x2C`–`0x35`, plus `BKSL` → `0x2B` and `LSGT` → `0x56`. + +The nine layouts added in 0.9.0 were generated this way and then +hand-reviewed; the generator is small enough to rewrite in an +afternoon and is not worth vendoring, but knowing the trick is worth +a paragraph. + ### Dead keys PolterType tracks *the character produced per keystroke*, not the @@ -111,13 +130,21 @@ To bundle a new language so users get it without touching their config directory: 1. Drop the layout TOML into `data/layout-mappings/.toml`. -2. Drop the wordlists into `data/wordlists/`: - * `.txt.gz` — the large dictionary, gzipped (a plain - `.txt` is also accepted, but every bundled language ships - `.gz` — the uncompressed files are big) - * `-extras.txt` — your tech-vocab extras (optional) - * `-stop.txt` — short stop words (optional) - * `-weak.txt` — valid-but-rare entries (optional) +2. Add the dictionary source to + `xtask/src/consts.rs::HUNSPELL_SOURCES` and run `cargo xtask + wordlists fetch`, which downloads the `.dic` + `.aff`, expands the + affix rules and writes `data/wordlists/.txt.gz` for you. + (Hand-made lists work too — drop `.txt.gz` in yourself and + skip the entry.) The optional companions all live in the same + directory: + * `-extras.txt` — your tech-vocab extras + * `-stop.txt` — short stop words + * `-weak.txt` — valid-but-rare entries + + Watch the reported form count. Anything in the tens of millions + means the `.aff` is combinatorial rather than inflectional and + wants `ExpandMode::StemsOnly` — Hebrew is the worked example, see + `data/wordlists/CREDITS.md`. 3. Add the stem to `crates/poltertype-core/build.rs::LAYOUTS`, which is what copies the data into the dist tree: @@ -135,23 +162,46 @@ config directory: ```rust pub const BUNDLED_LAYOUT_STEMS: &[&str] = - &["en_us", "uk_ua", "ru_ru", "de_de", "es_es", "fr_fr", "pl_pl"]; + &["en_us", "uk_ua", "ru_ru", /* … */ "pt_br"]; ``` Nothing is baked into the binary — the TOML is read from `/layout-mappings/` at run time. The stem is all the runtime needs. -5. Optional but recommended: extend `derive_vowels` with the +5. **Add the four files to `installers/wix/main.wxs`.** The MSI is + the one installer that enumerates every data file by hand + (`` and friends); the AppImage and the macOS + bundle `cp -R` the whole tree and need nothing. Four entries per + language — the layout TOML, `.fst`, `-stop.txt`, + `-surface.fst` — and WiX **errors on a `Source=` that + doesn't exist**, so only list files the build actually produces + (that is why only `uk_ua` has a `-weak.txt` entry). Skip this step + and the language works everywhere except Windows, where it + silently isn't installed. + +6. Register the layout id with the OS switchers in + `poltertype-layout`, or the app can name your language but never + switch to it: + * Linux — `linux/shared.rs`, both directions of the XKB table. + * macOS — `macos.rs`, `tis_id_to_bcp47` (list every variant you + know) and `bcp47_to_tis_id` (base id only; a wrong guess here + targets a keyboard the user doesn't have). + * Windows needs nothing — it resolves BCP-47 from the HKL. + +7. Optional but recommended: extend `derive_vowels` with the language's vowel set (especially if it uses accented vowels — the plausibility detector counts vowel ratio, and unrecognised vowels score as consonants). -6. `cargo build` and `cargo test --workspace`. +8. `cargo build` and `cargo test --workspace`. Keep (3) and (4) in lock-step. If they drift, the runtime logs a "missing TOML" warning at startup for the stem it can't find — noisy rather than silent, but it is not a build failure, so read the first few log lines after wiring a new language. +`every_bundled_stem_resolves_to_a_layout` in +`crates/poltertype-core/src/layouts/tests.rs` fails on that drift, so +in practice `cargo test` catches it first. --- diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index d930ab7..5c16a67 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -917,9 +917,26 @@ first 2 KB, and decodes byte-for-byte as Latin-1 if the source says `ISO8859-*` / `LATIN1` / `WINDOWS-1252`. Adding a new dictionary in another encoding is a single match arm. +> **Superseded 2026-08-01 — that rule was wrong, and it shipped +> mojibake.** Two faults. `ISO8859-*` was matched as a prefix and +> then decoded as Latin-1, so ISO-8859-2 and ISO-8859-7 came out +> scrambled; and the scan ran per-file, but a Hunspell `.dic` +> carries no `SET` of its own — the `.aff` declares the encoding +> for the pair. The `.dic` therefore hit the "no SET, assume +> Latin-1" default every time, which was invisible for the five +> languages bundled then because four are UTF-8 and German is +> genuinely Latin-1. Adding Polish and Greek exposed it: `słowo` +> became `s³owo`, which matches nothing and looks like a word. +> Now `encoding_of_aff` reads the `.aff` once and that encoding +> decodes both halves, Latin-2 and Greek have real 96-entry tables +> (generated from Python's codecs, not typed by hand), and an +> unknown or missing `SET` is a hard error. Guessing an encoding is +> the failure mode here, not a convenience. + Storage on disk: bulk wordlists ship as `data/wordlists/.txt.gz` -rather than raw `.txt`. Raw, the six languages total ~165 MB -(uk_ua alone is 84 MB after expansion); gzipped they're ~24 MB. +rather than raw `.txt`. Raw, the fifteen bundled languages total +583 MB (uk_ua alone is 84 MB after expansion); gzipped they're +97 MB. Both `poltertype-core/build.rs` (`flate2::read::GzDecoder`) and the xtask generator (`flate2::write::GzEncoder`) handle the format transparently, and the build script falls back to a plain `.txt` diff --git a/docs/PLAN.md b/docs/PLAN.md index 1169533..1ce3732 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -842,8 +842,8 @@ Levels: Hunspell-expanded lists (not `lingua`, which was dropped). - [x] `poltertype-core`: WordBuffer, DecisionPolicy, Corrector, AudioPlayer. -- [x] The EN↔UK map in `data/layout-mappings/` (six are bundled - today: EN·UK·RU·DE·ES·FR). +- [x] The EN↔UK map in `data/layout-mappings/` (fifteen are bundled + today: EN·UK·RU·DE·ES·FR·PL·CS·EL·HE·TR·BG·IT·PT-PT·PT-BR). - [x] Pause / switch-last hotkeys. - [x] Settings: saving/loading `config.toml`. diff --git a/installers/wix/main.wxs b/installers/wix/main.wxs index 68034ae..0318b57 100644 --- a/installers/wix/main.wxs +++ b/installers/wix/main.wxs @@ -166,6 +166,33 @@ + + + + + + + + + @@ -199,6 +226,33 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + +