Skip to content
BranchesTags

Repository files navigation

wikipedia-data

npm

Wikipedia language and disambiguation data for every open Wikipedia edition — the localized template name, the localized category, and the localized parenthetical title suffix (Paris (dezambiguizare)).

Everything is generated from structured Wikimedia JSON APIs (Wikidata sitelinks + the MediaWiki sitematrix and querypage APIs). No HTML scraping.

  • Zero runtime dependencies
  • Dual ESM + CommonJS build, TypeScript types included
  • Data also shipped as plain JSON in data/ for non-JS consumers

Install

npm install wikipedia-data

Requires Node.js >=20.19.0.

Usage

ESM

import {
  getLanguage,
  getDisambiguationTemplate,
  getDisambiguationCategory,
  getDisambiguationTitleSuffix,
  isDisambiguationTitle,
} from "wikipedia-data";

getLanguage("ro");
// { code: "ro", name: "Romanian", localName: "română",
//   url: "https://ro.wikipedia.org", dbName: "rowiki" }

getDisambiguationTemplate("de"); // "Begriffsklärung"
getDisambiguationCategory("ru"); // "Категория:Страницы значений"
getDisambiguationTitleSuffix("zh"); // "消歧义"

isDisambiguationTitle("Paris (dezambiguizare)", "ro"); // true
isDisambiguationTitle("Paris", "ro"); // false

CommonJS

const { getDisambiguationTitleSuffix } = require("wikipedia-data");

getDisambiguationTitleSuffix("sv"); // "olika betydelser"

Raw JSON

The data/*.json files are shipped in the tarball for non-JavaScript consumers. They are not exposed as package subpaths — read them from disk:

import { readFileSync } from "node:fs";
import { createRequire } from "node:module";
import { pathToFileURL } from "node:url";

const require = createRequire(import.meta.url);
const pkg = pathToFileURL(require.resolve("wikipedia-data/package.json"));
const languages = JSON.parse(readFileSync(new URL("data/languages.json", pkg), "utf8"));

From JavaScript you normally want the typed accessors above instead.

API

Languages

interface WikipediaLanguage {
  /** language code, e.g. "ro" */
  code: string;
  /** English name, e.g. "Romanian" */
  name: string;
  /** autonym, e.g. "română" */
  localName: string;
  /** e.g. "https://ro.wikipedia.org" */
  url: string;
  /** e.g. "rowiki" */
  dbName: string;
}
Function Returns
getLanguages() Readonly<Record<string, WikipediaLanguage>>
getLanguage(lang) WikipediaLanguage | undefined
isValidLanguage(lang) boolean

Disambiguation templates

Function Returns
getDisambiguationTemplates() Readonly<Record<string, string>>
getDisambiguationTemplate(lang) string | undefined

Namespace prefix stripped: "Format:Dezambiguizare" is shipped as "Dezambiguizare".

Disambiguation categories

Function Returns
getDisambiguationCategories() Readonly<Record<string, string>>
getDisambiguationCategory(lang) string | undefined

Namespace prefix kept: "Categorie:Dezambiguizare".

Disambiguation title suffixes

Function Returns
getDisambiguationTitleSuffixes() Readonly<Record<string, readonly string[]>>
getDisambiguationTitleSuffixList(lang) readonly string[] (most common first, [] if unknown)
getDisambiguationTitleSuffix(lang) string | undefined (the most common one)

Predicates

isDisambiguationTitle(title: string, lang: string): boolean;
isDisambiguationCategory(category: string, lang: string): boolean;

isDisambiguationTitle matches the trailing parenthetical of a title against the localized suffix list. isDisambiguationCategory matches the exact localized category and also any category that contains the localized disambiguation word (or the localized template name) as a whole word, so "Category:All article disambiguation pages" matches for en.

Both normalize _ to a space, compare case-insensitively (but diacritics-sensitively) and return false — never throw — for unknown languages and empty input.

isDisambiguationTitle("Paris_(Dezambiguizare)", "ro"); // true
isDisambiguationTitle("Madrid (desambiguacion)", "es"); // false — missing the accent
isDisambiguationCategory("Category:Disambiguation_pages", "en"); // true
isDisambiguationCategory("Category:Cities in France", "en"); // false

Metadata

interface DataMeta {
  generatedAt: string; // ISO 8601
  sources: Record<string, string>;
  counts: Record<string, number>;
  /** { lang: { secondarySuffix: measuredPrecision } } */
  suffixPrecision: Record<string, Record<string, number>>;
}

getDataMeta();

Language code normalization

Every getter normalizes its lang argument: it is trimmed, lowercased and _ is mapped to -. " RO ", "RO" and "ro" are the same language; "be_tarask" resolves to "be-tarask".

Deprecated (v0.x compatibility)

Function Replacement
getDisambiguationNames() getDisambiguationTemplates()
getDisambiguationNames2() getDisambiguationTemplates()
getDisambiguationCategories2() getDisambiguationCategories()

The *2 variants return only entries whose language code is exactly two characters, matching the old behaviour.

Datasets

All files live in data/, are pretty-printed with 2-space indentation and are sorted by language code.

data/languages.json — 348 entries

Every open Wikipedia language edition.

{
  "ro": {
    "code": "ro",
    "name": "Romanian",
    "localName": "română",
    "url": "https://ro.wikipedia.org",
    "dbName": "rowiki"
  }
}

Source: meta.wikimedia.org action=sitematrix. Only groups with a site entry whose code is "wiki" and which is not closed are kept.

Note: the sitematrix API returns the autonym in its name field and the English name in localname. This package normalizes that: name is always the English name, localName is always the autonym.

data/disambiguation-templates.json — 234 entries

The localized Template:Disambiguation name, without namespace prefix.

{
  "de": "Begriffsklärung",
  "fr": "Homonymie",
  "ro": "Dezambiguizare",
  "ru": "Неоднозначность"
}

Source: Wikidata item Q6148868 (Template:Disambiguation) sitelinks, via action=wbgetentities&props=sitelinks. Sitelinks are keyed by database name (rowiki), mapped back to language codes through languages.json; non-Wikipedia sites are ignored.

data/disambiguation-categories.json — 215 entries

The localized Category:Disambiguation pages title, namespace prefix included.

{
  "en": "Category:Disambiguation pages",
  "ro": "Categorie:Dezambiguizare",
  "ru": "Категория:Страницы значений",
  "zh": "Category:消歧义"
}

Source: Wikidata item Q1982926 (Category:Disambiguation pages) sitelinks.

data/disambiguation-title-suffixes.json — 172 entries

The localized parenthetical used in disambiguation page titles, most common first. This dataset is new in v1.0.0 and has no v0.x equivalent.

{
  "en": ["disambiguation"],
  "ro": ["dezambiguizare"],
  "sv": ["olika betydelser"],
  "zh": ["消歧义", "消歧義"]
}

Built in two passes.

1. Frequency. Each wiki's own action=query&list=querypage&qppage=DisambiguationPages (the Disambiguator extension's list of disambiguation pages) supplies up to 500 titles. The trailing (...) of every title is extracted with /\(([^()]+)\)\s*$/, counted case-insensitively, and a candidate is kept when it occurs at least 3 times and in at least 5% of the parenthetical-bearing titles. The most frequent original casing is preserved (hence de ships "Begriffsklärung", not "begriffsklärung") — comparisons in this package are case-insensitive anyway.

2. Precision. Frequency alone cannot tell a disambiguation marker from a merely popular parenthetical: the Tamil word for "film" is common enough to pass pass 1, but it is not a disambiguation marker. So every candidate beyond the top-ranked one is scored against action=query&generator=search&gsrsearch=intitle:"(suffix)"&prop=pageprops&ppprop=disambiguation — of the namespace-0 titles that really end with (suffix), what share does MediaWiki flag as disambiguation pages? A secondary suffix is kept only with at least 5 sampled titles and precision ≥ 0.8; otherwise it is dropped (too little evidence counts as a drop). The top-ranked suffix of a language is always kept, so no language is ever left with an empty list.

The measured precision of every kept secondary suffix is recorded in data/meta.json under suffixPrecision, so runs can be compared over time:

{
  "suffixPrecision": {
    "zh": { "消歧義": 1 },
    "tg": { "Ҳамнасаб": 0.97, "Ҳамном": 0.89 }
  }
}

Of 42 secondary candidates found by pass 1, 19 survived pass 2. Dropped examples: ta "film" (0/9 sampled were disambiguation pages), ky "metro station" (0.72), mn and os place-name parentheticals (0.00).

data/meta.json

Provenance: generation timestamp, the source of every dataset, entry counts, and the measured precision of every kept secondary title suffix.

Regenerating the data

npm run generate                                  # everything
npm run generate -- --only=languages,templates    # a subset
npm run generate -- --force                       # skip the shrink guard

--only accepts any of languages, templates, categories, suffixes.

The generator sends a descriptive User-Agent, fetches at most 8 wikis in parallel, honours Retry-After on HTTP 429 with a shared throttle, retries transient failures with exponential backoff, and skips wikis that keep failing instead of aborting the run. It refuses to overwrite a dataset whose new entry count is below 80% of the previous one unless --force is passed. After writing data/*.json it regenerates the typed src/data/*.ts modules and prints a per-dataset summary of counts and missing languages, including every secondary title suffix dropped by the precision filter with its measured precision and sample size.

Development

npm install
npm run build         # dual tsc build -> dist/esm + dist/cjs
npm test              # vitest, offline
npm run test:live     # vitest, spot-checks against the live APIs
npm run lint          # oxlint
npm run format        # prettier --write
npm run format:check  # prettier --check

Changelog

v1.0.0

Complete rewrite. Breaking.

  • Data is now generated from structured JSON APIs (Wikidata sitelinks, MediaWiki sitematrix and querypage) instead of regex-scraping the HTML of en.wikipedia.org pages. The v0.x data had been stale since 2015 and was partly wrong: it stored template redirects as if they were the canonical localized name (zh was "Disambiguation", nl "Dp", uk "Disambig").
  • New dataset: disambiguation-title-suffixes.json — the localized parenthetical used in article titles (Paris (dezambiguizare)), which is what you actually need to detect a disambiguation page from its title. 172 languages, each candidate beyond the primary validated against MediaWiki's own disambiguation page property (precision ≥ 0.8 over ≥ 5 sampled titles).
  • New dataset: languages.json — all 348 open Wikipedia editions with English name, autonym, url and database name.
  • Renamed disambiguation-names.json to disambiguation-templates.json; dropped the *2.json files (the *2() accessors remain, deprecated).
  • Rewritten in TypeScript, published as a dual ESM + CommonJS package with type declarations and zero runtime dependencies.
  • New helpers: getLanguage, isValidLanguage, getDisambiguationTemplate, getDisambiguationCategory, getDisambiguationTitleSuffix(es|List), isDisambiguationTitle, isDisambiguationCategory, getDataMeta.
  • All getters normalize the language code (trim, lowercase, _ → -).
  • getDisambiguationNames, getDisambiguationNames2 and getDisambiguationCategories2 are kept as deprecated shims so v0.1.1 consumers keep working.
  • Requires Node.js >=20.19.0.

v0.1.1

Last of the 2015 CommonJS releases: four JSON files scraped from Wikipedia HTML.

License

ISC © Dumitru Cantea

About

Wikipedia useful data

Resources

Readme
ISC license
Activity
Custom properties

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages