Skip to content

About

Data pipeline to harvest, transform, reconcile, enrich and export Linked Art data for LUX (or other system)

Resources

Stars

31 stars

Watchers

2 watching

Forks

Latest commit

 

History

2,002 Commits

Folders and files

Repository files navigation

Data Transformation Pipeline Code

Installation

See docs/install.md: system packages, pip install -e ., lux initialize, lux testinstall, downloading and loading data. docs/rebuild.md describes a rebuild step by step and docs/codebase.md is a tour of the code. docs/loader-dsl.md is the reference for a source's input_files block (lux guess-dump-type, lux load).

Running

Everything runs through the lux command (lux help lists them all):

lux download --source all --type records --max-workers 10   # fetch the sources' dump files
lux load --source wikidata --max-workers 24                 # load a dump into its datacache
lux reconcile --source all --max-workers 24                 # reconcile, then identify, metatypes, write-refs
lux merge --source all --max-workers 24                     # one merged record per entity
lux export --max-workers 24                                 # JSONL for MarkLogic

lux build runs a whole build, the phases of run-all.sh in its order, each as its own lux process, stopping at the first failure:

lux build --clear --max-workers 24     # clear caches and refs, new token, then build
lux build --from merge                 # pick up again at a phase
lux build --skip sheets,portal         # leave phases out
lux build --dry-run                    # print the steps, run nothing
lux build --clear --background         # start it, return, and leave it running

--background (-b) works for any command that is a run: instead of nohup lux ... &, lux starts the command as a process of its own (closing the terminal does not stop it) and prints where its output goes (<prefix>/stdout.log), the lux watch <prefix> that follows it, and the kill -- -<pid> that stops it with its workers. A mistake on the command line is still reported at once, with its exit status.

The phases are clear (only with --clear), sheets, reconcile (with identify, metatypes, write-refs), merge, portal and export, and each is a command of its own: lux sheets reads the curators' spreadsheet and loads its same-as / different-from pairs and fixes (--fetch-only, --load-only). By default that is the Google Sheet (pip install -e '.[google]'); --file reads the same tabs from an .xlsx workbook (.[xlsx]) or a directory of <tab name>.csv files instead. And lux portal tags the merged records the portals need: a portal is an internal source with "portal": true in its configuration (tagged with its name) or "portal": "<tag>" (a renamed source keeps the tag the exports know, e.g. "ypm"). A full lux reconcile, merge or export first removes what an earlier run left (metatypes slice files, done flags, old export slices); one --my-worker slice never does. The files the phases hand to each other (metatypes-*.json, reference_uris.txt) live under the configuration's temp_dir, so the phases may be run from any directory. ./run-all2.sh --all is the same build as shell (ENGINE=mp for one lux per phase instead of one per slice); the legacy run-all.sh beside it is kept until the two have been compared on the build machine.

Every run is a directory, <log_dir>/<command>-<timestamp>/ (its "prefix"; --prefix names another). Each worker writes w<n>.status and w<n>.log in it; lux watch --latest reads them to attach to a running (or finished) build from another terminal, and lux watch <build prefix> follows a lux build step by step (its steps are runs in its directory: <build prefix>/reconcile, ...). The command itself leaves driver.log (its own log: what it does before, between and after its workers) and run.json (its command line and how it ended) beside them. --log DEBUG (or --debug) turns up the logging, and --no-ui gives plain output instead of the console view. Other commands worth knowing:

lux list                                     # the configured sources, maps and indexes
lux counts --source all                      # what is in the caches, maps and indexes
lux harvest --source aat                     # pull a source's change feed into its caches
lux load-index --source wikidata             # build a source's reconciliation indexes
lux backup --map idmap                       # back up a store, whatever its backend
lux backup --all                             # every cache, map and index (not MarkLogic)
lux restore <backup-dir> --map idmap --force # put it back
lux clear --map idmap --force                # empty a cache, map or index
lux viz --recid <from>,<to>                  # draw how reconcile joined two records
lux merge --source all --profile             # cProfile each slice; `python timing-report.py --profile`

lux serve does all of this in a browser (see The web UI).

lux <command> --help lists a command's options. Backups go to backup_dir (default <data_dir>/backups), each with a manifest.json written last; lux restore refuses one without it. lux viz needs the optional extras (pip install -e '.[ui-extras]'). The steps between reconcile and merge (identify, metatypes, write-refs, new-token) are commands of their own too, and refuse to run while a build is running unless given --force.

The legacy run-*.py, manage-data.py, run-all.sh and *_parallel.sh scripts still work; retiring them is the last item (33) of the port of the rob_refactor branch, which is planned and tracked in refactor_docs/ (start with its README.md and 00-INDEX.md). CLAUDE.md describes the working conventions.

The web UI

lux serve (pip install -e .[web]) serves a browser view of the installation at http://127.0.0.1:8321/: the configuration (read and edited in its own files), any record through every cache and its identity, every run with its workers' progress and logs live, the lux commands as forms (the whole build too), timing, counts, Postgres maintenance and the installation checks. See docs/web.md.

Architecture

architecture diagram

Pipeline Components

Future ITS Owned Components:

  • Harvester: ActivityStreams based record harvester that stores to a Cache.
  • Cache: Record cache for storing local copies of JSON data. Currently postgres or filesystem.
  • IdMap: Identifier Map that mints, manages and retrieves external/internal identifier sets. Currently Postgres, Redis or in-memory.

Pipeline Components:

  • config: Configuration of the pipeline, as JSON records in a cache.
  • cli/: the lux command: one module per command (lux help lists them), the console view and lux watch.
  • progress, runtime, log: the worker status protocol (.status files, null_progress), configuration loading and logging setup. Library code logs to the lux_pipeline logger and never prints.
  • process/tasks/: the managers that run a phase's workers on an engine (mp, null, ray), one per parallel command.
  • process/phases/: the body of each build phase (reconcile, identify, merge, export), shared by lux and the legacy scripts.
  • process/utils/timing: PhaseTimer, the per-stage timings (and, with --profile, cProfile output) that timing-report.py reads.
  • process/Collector: Recursively collects identifiers for a given record.
  • process/Reconciler: Reconcile records
  • process/Merger: Merges two Linked Art records representing the same entity together.
  • process/Reidentifier: Recursively external rewrite URIs in a record to internal identifiers, given an IdMap.
  • process/ReferenceManager: Manages inter-record connections
  • process/UpdateManager: Manages harvesting and updates to caches
  • sources/*/: one directory per source (plus yale/ for the Yale units and lux/ for LUX's own outputs), each holding the classes below.
  • sources/*/Acquirer: Wraps Fetcher and Mapper to acquire a record either from the network or a cache
  • sources/*/Fetcher: Fetches identified record from external source to a Cache.
  • sources/*/Harvester: Retrieve multiple records via IIIF Change Discovery or OAI-PMH
  • sources/*/Mapper: Maps from source format into Linked Art, or from Linked Art to discovery layer format (Marklogic, QLever)
  • sources/*/Reconciler: Determine if the entity in the given record is described in the external source.
  • sources/*/Loader: Load a dump of the data into the data cache.
  • sources/*/IndexLoader: Create an inverted index to reconcile records against this dataset.
  • storage/MarkLogic: Marklogic storage of data
  • storage/cache: Data caches (Postgresql, file system, Redis); abstract.AbstractCache is the API they share.
  • storage/idmap: Identifier maps and key/value stores (Postgresql, Redis, LMDB, file system, in-memory, and the subsidiary idmap).
  • storage/backup: lux backup / lux restore: native copies (Postgres COPY, LMDB hot copy) or portable gzipped JSON Lines.

External Sources: Implementation Status

Source Fetch Map Reconcile Load Index
AAT ✅ ✅ ✅ N/A ✅
TGN ✅ ✅ - N/A -
ULAN ✅ ✅ ✅ N/A ✅
LCNAF ✅ ✅ ✅ ✅ ✅
LCSH ✅ ✅ ✅ ✅ ✅
LCDGT ✅ ✅ - - -
VIAF ✅ ✅ ✅ ✅ ✅
FAST - ✅ - ✅ -
Wikidata ✅ ✅ ✅ ✅ ✅
Wikimedia API ✅ ✅ - N/A -
Wikipedia - N/A - ✅ -
Geonames ✅ ✅ - ✅ -
Who's on First ✅ ✅ - N/A -
Pleiades - ✅ - ✅ -
DNB ✅ ✅ - ✅ -
BNF ✅ ✅ - ✅ -
BNE ✅ ✅ - ✅ -
Japan NL ✅ ✅ - ✅ -
GBIF ✅ ✅ - N/A -
ORCID ✅ ✅ - ✅ -
ROR ✅ ✅ - ✅ -
Homosaurus ✅ ✅ - - -
Nomisma ✅ ✅ - - -
SNAC ✅ ✅ - - -
Getty Museum ✅ - - N/A -

✅ = Done ; - = Not started ; N/A = Can't/Won't be done

  • AAT, TGN, ULAN: Dump files are NTriples based. More effort to reconstruct than it would be worth. Instead we use IIIF Change Discovery (lux harvest) to synchronize. Getty Museum is harvested the same way (ActivityStreams).
  • WOF: Dump file is a 33Gb sqlite db... we just use it as the cache directly
  • Wikipedia: article texts are loaded for lookup only; the mapper produces no records.
  • Index: the reconciliation indexes built by lux load-index (LMDB).
  • Besides the Yale units (sources/yale/), sources/ also holds loaders and mappers for other record sources: Ashmolean (ashmol), Europeana / MDS (mds), ethan, and an Art Institute of Chicago experiment (artic).

Fetching external source dump files

lux download --source <name> fetches the files a source's configuration names (remoteDumpFile and dumpFilePath, or its input_files block) into place; --source all --max-workers 10 fetches every source's, and --no-overwrite keeps what is already there. Wikidata's is huge: start it with --background and get on with life in the mean time.

About

Data pipeline to harvest, transform, reconcile, enrich and export Linked Art data for LUX (or other system)

Resources

Stars

31 stars

Watchers

2 watching

Forks

Releases

Packages

Used by

Contributors

Languages