whatlanguage

Pure Ruby natural language detection for 160+ languages.

require 'whatlanguage'

WhatLanguage.language("Que linguagem é essa? É uma pergunta sobre a língua portuguesa.")
# => :portuguese

• gem install whatlanguage
• No runtime dependencies.
• Supports 20+ writing systems.
• Ships a compact ~220 KB trigram model.
• Requires Ruby 3.0+ (JRuby and TruffleRuby also good)
• Best on sentence-length text or longer. Short texts can have ambiguous results.

Important

Version 3.0 changes nine ISO output codes, speeds up detection, and reduces allocations. See Upgrading to 3.0 for the code changes.

How it works

Detection is in two stages. First, the dominant Unicode script is detected from letters; scripts used by a single supported language (Greek, Korean, Thai, Japanese using Hiragana/Katakana) resolve immediately. When kana is present, Han, Hiragana, and Katakana count together as Japanese, while a dominant unrelated script can still win. For scripts shared by several languages (e.g. Latin, Cyrillic, Arabic, Hebrew) trigrams are ranked by frequency and compared against candidate language profiles.

Digits, punctuation, and emoji do not count toward the minimum text length and are treated as separators when extracting trigrams. Combining marks are preserved in trigrams but do not count as additional letters. Unicode script coverage follows the Unicode version supported by your Ruby runtime.

The trigram profiles are vendored from whatlang, a port of Franc, whose models are built from the public-domain UDHR corpus (see Credits). The model is a ~220 KB JSON file.

Repeated calls over the full language set use a shared trigram index. A rolling window counts input trigrams without building a character array. Scores and rankings are unchanged. See the indexing benchmarks and long-input benchmarks for timings, allocations, and memory costs.

Usage

Return a full detection result:

wl = WhatLanguage.new
text = "Die Stadt plant neue Investitionen in den öffentlichen Verkehr"
result = wl.detect(text)
result.language   # => :german
result.iso        # => :de
result.score      # => 79018
result.ranked     # => [[:german, 79018], [:dutch, 77631], ... ]

result.iso and language_iso return ISO 639-1 symbols with an ISO 639-3 fallback. Selected individual languages use CLDR's macrolanguage aliases, such as Swahili swh → sw, Mandarin cmn → zh, and Western Farsi pes → fa; historical code mappings are also preserved. These aliases are a normalization policy, not a BCP 47 requirement: swh remains a valid individual-language subtag. Languages without a selected alias retain their individual codes. The CLI uses the same mapping. Codes are not necessarily unique: both :northern_uzbek and :uzbek return :uz. Use result.language to distinguish the language names.

Return ranked scores, or the score hash:

wl.ranked(text)       # => [[:german, 79018], [:dutch, 77631], ... ]
wl.score_hash(text)   # => { german: 79018, dutch: 77631, ... }

Restrict candidate languages:

wl = WhatLanguage.new(only: [:english, :german, :french])

For short texts, English, Chinese (Mandarin), Hindi, Spanish, and French receive a built-in preference. These five were selected using the 2025 total-speaker ranking (including second-language speakers). Eligible trigram candidates receive the same bonus: 600 ranking points through 10 letters, decreasing linearly to zero at 50 letters. Longer texts keep their original scores. The preference is not configurable; it respects only: and script detection. Chinese already resolves from its script, so it does not need a statistical bonus. Scores exposed by the API include the bonus and remain ranking values, not probabilities.

Short shared-script fragments are ignored by default because there is not enough signal to rank their languages reliably. The threshold applies to the statistical trigram stage; scripts that identify a single supported language, such as Greek, Korean, or Thai, can still resolve from shorter text. The threshold can be adjusted:

wl = WhatLanguage.new(min_chars: 0)

Upgrading to 3.0

Update comparisons against result.iso, language_iso, and CLI output:

Language	Previous code	3.0 code
Swahili	swh	sw
Central Aymara	ayr	ay
Central Kanuri	knc	kr
Cusco Quechua	quz	qu
Halh Mongolian	khk	mn
Koongo	kng	kg
Logudorese Sardinian	src	sc
Northern Uzbek	uzn	uz
Tosk Albanian	als	sq

Both Northern Uzbek and Uzbek now return uz; use result.language to distinguish them. Language-name symbols, detection scores, rankings, options, and return types are unchanged from 2.1.

If upgrading from 1.x, see the 2.0 changelog too. Version 2.0 replaced the Bloom-filter engine with trigram profiles and removed the String#language monkeypatch.

Command line

Installing the gem also installs a whatlanguage executable that reads from files given as arguments (or stdin) and prints the ISO 639 code of the detected language, or und if undetermined:

$ whatlanguage README.md
en
$ echo "Wie geht es dir heute?" | whatlanguage
de

Known limitations

• Short texts can be ambiguous. Built-in preferences help some phrases but can bias others toward preferred languages. Statistical detection returns nil below 10 letters by default.
• Scores indicate relative rank, not probability or confidence. Use #ranked to inspect alternatives.
• Similar written languages, such as Norwegian Bokmål/Danish and Hebrew/Yiddish, can be confused.
• Japanese without kana is classified as Chinese when Han is the dominant script.
• Romanized text is compared with Latin-script profiles, so it may not resolve to its original language.

Credits

Contributions from Konrad Reiche, Salimane Adjao Moustapha, Andrew Cone, Lasse Skindstad Ebert, Henrik Nyh, Daniel Sandbecker, Michael Hartl, Pedro Lambert, Tobias Preuss, Pepijn Looije, Keith Bennett, and others appreciated.

The trigram language profiles in lib/whatlanguage/trigrams.json are taken from whatlang (MIT, © Sergey Potapov), itself a derivative of Franc (MIT, © Titus Wormer). Those profiles are derived from the public-domain Universal Declaration of Human Rights translations.