LDE – Linked Data Elements

Shared building blocks for the full Linked Data lifecycle.

Every organization working with Linked Data ends up building the same infrastructure from scratch: endpoint management, data import, transformation pipelines, dataset discovery.

LDE covers the full Linked Data lifecycle – from discovery and ingestion through transformation to publication – as an open-source toolkit of composable building blocks for Node.js.

Data transformations are expressed as plain SPARQL queries: portable, transparent and free of vendor lock-in.

Key capabilities

• Discover datasets from DCAT-AP 3.0 registries.
• Download and import data dumps to a local SPARQL endpoint for querying.
• Transform datasets with pure SPARQL CONSTRUCT queries: composable stages with fan-out item selection.
• Analyze datasets with VoID statistics and SPARQL monitoring.
• Publish results to SPARQL endpoints or local files.
• Serve RDF data over HTTP with content negotiation (Fastify plugin).

Standards

Standard	Usage
DCAT-AP 3.0 (EU)	Dataset discovery and registry queries
SPARQL 1.1	Data transformations, dataset queries and endpoint management
SHACL	Per-class sampling (@lde/pipeline-shacl-sampler), validation (@lde/pipeline-shacl-validator) and documentation generation (@lde/docgen)
VoID	Statistical analysis of RDF datasets (@lde/pipeline-void)
RDF/JS	Internal data model (N3)
LDES (EU)	Event stream consumption and publication (planned)

Quick example

import {
  Pipeline,
  Stage,
  SparqlConstructReader,
  SparqlItemSelector,
  SparqlUpdateWriter,
  ManualDatasetSelection,
} from '@lde/pipeline';

const pipeline = new Pipeline({
  datasetSelector: new ManualDatasetSelection([dataset]),
  stages: [
    new Stage({
      name: 'per-class',
      itemSelector: new SparqlItemSelector({
        query: 'SELECT DISTINCT ?class WHERE { ?s a ?class }',
      }),
      readers: new SparqlConstructReader({
        query:
          'CONSTRUCT { ?class a <http://example.org/Class> } WHERE { ?s a ?class }',
      }),
    }),
  ],
  writers: new SparqlUpdateWriter({
    endpoint: new URL('http://localhost:7200/repositories/lde/statements'),
  }),
});

await pipeline.run();

Packages

Discovery – Find and retrieve dataset descriptions from registries
@lde/dataset	Core dataset and distribution objects
@lde/dataset-registry-client	Retrieve dataset descriptions from DCAT-AP 3.0 registries
Processing – Transform, enrich and analyse datasets with SPARQL pipelines
@lde/pipeline	Build pipelines that query, transform and enrich Linked Data
@lde/pipeline-shacl-sampler	Per-class sampling stages derived from SHACL shapes
@lde/pipeline-shacl-validator	SHACL validation for pipeline stages
@lde/pipeline-void	VoID statistical analysis for RDF datasets
@lde/distribution-downloader	Download distributions for local processing
@lde/distribution-health	Derive distribution usability from reachability and RDF validity
@lde/distribution-probe	Probe distributions for availability and metadata
@lde/iiif-validator	Validate that a URL resolves to a valid IIIF Presentation Manifest
@lde/sparql-importer	Import data dumps to a local SPARQL endpoint for querying
Publication – Serve and document your data
@lde/fastify-rdf	Fastify plugin for RDF content negotiation and request body parsing
@lde/docgen	Generate documentation from RDF such as SHACL shapes
@lde/search	The search core: projects RDF into engine-agnostic search documents (framing + a declarative field spec)
@lde/search-api-graphql	Engine- and domain-agnostic GraphQL surface for search: builds an executable GraphQL schema from a SearchSchema at runtime and serves it as a framework-agnostic fetch handler with a self-contained playground
@lde/search-api-server	The served search API as a bootable process and prebuilt Docker image: mounts a schema-declaration module, binds the GraphQL handler to a Typesense engine, and serves /graphql plus /health from environment config
@lde/search-indexer	The search indexer as a bootable process and prebuilt Docker image: mounts the same schema-declaration module as the API server, selects datasets from a registry, and rebuilds the Typesense collections from environment config
@lde/search-pipeline	Applies the @lde/search projection inside an @lde/pipeline run, fanning each document out to the transactional engine writer for its type’s collection (owns no projection itself)
@lde/search-typesense	Typesense engine adapter: transactional Blue/green (build fresh, swap atomically) and In-place (update the live index) rebuild writers
@lde/text-normalization	Text folding (diacritic stripping and transliteration) for search index and query normalization
Monitoring – Observe pipeline runs and endpoint health
@lde/distribution-monitor	Monitor DCAT distributions (SPARQL endpoints and data dumps) with periodic probes
@lde/pipeline-console-reporter	Console progress reporter for pipelines
Infrastructure – Manage SPARQL servers and run tasks
@lde/local-sparql-endpoint	Quickly start a local SPARQL endpoint for testing and development
@lde/sparql-server	Start, stop and control SPARQL servers
@lde/sparql-qlever	QLever SPARQL adapter for importing and serving data
@lde/wait-for-sparql	Wait for a SPARQL endpoint to become available
@lde/task-runner	Task runner core classes and interfaces
@lde/task-runner-docker	Run tasks in Docker containers
@lde/task-runner-native	Run tasks natively on the host system

Architecture

graph TD
  subgraph Discovery
    dataset
    dataset-registry-client --> dataset
  end

  subgraph Processing
    pipeline --> dataset-registry-client
    pipeline --> sparql-server
    pipeline --> sparql-importer
    pipeline-shacl-sampler --> pipeline
    pipeline-shacl-validator --> pipeline
    pipeline-void --> pipeline
    distribution-downloader --> dataset
    distribution-probe --> dataset
    pipeline --> distribution-probe
    sparql-importer --> dataset
    distribution-health --> distribution-probe
    distribution-health --> sparql-importer
  end

  subgraph Publication
    fastify-rdf
    docgen
    search --> text-normalization
    search-api-graphql --> search
    search-api-server --> search-api-graphql
    search-api-server --> search-typesense
    search-typesense --> search
    search-typesense --> text-normalization
    search-typesense --> pipeline
    search-pipeline --> search
    search-pipeline --> pipeline
    search-indexer --> search-pipeline
    search-indexer --> search-typesense
    search-indexer --> sparql-qlever
    search-indexer --> pipeline-console-reporter
  end

  subgraph Monitoring
    pipeline-console-reporter --> pipeline
    distribution-monitor --> distribution-probe
  end

  subgraph Infrastructure
    sparql-qlever --> sparql-importer
    sparql-qlever --> sparql-server
    sparql-qlever --> task-runner-docker
    task-runner-docker --> task-runner
    task-runner-native --> task-runner
    sparql-server
    local-sparql-endpoint
    wait-for-sparql
  end

Loading

Who uses LD Elements

 Netwerk Digitaal Erfgoed – Dutch national digital heritage infrastructure, commissioned by the Ministry of Education, Culture and Science

 DC4EU – Dutch Collections for Europe, the accredited national aggregator connecting Dutch heritage institutions with Europeana

Linked Open Limburg – Coöperatie Erfgoed Limburg’s regional heritage programme, built on LDE’s pipeline components

Comparison

	LD Elements	TriplyETL	rdf-connect	OpenLDES / LDI
Focus	SPARQL-native pipelines	RDF ETL platform	RDF stream processing	LDES ingestion & publication
Pipeline language	SPARQL + TypeScript	TypeScript DSL	Declarative (RML)	YAML (Spring Boot)
Lock-in	None – plain SPARQL files	Proprietary platform	Framework-specific	Framework-specific
License	MIT	Proprietary	MIT	EUPL-1.2

Development

Prerequisites: Node.js (LTS) and npm.

npm install
npx nx run-many -t build
npx nx run-many -t test
npx nx affected -t lint test typecheck build  # only changed packages

See CONTRIBUTING.md for the full development workflow.

License

MIT – see LICENSE.

Acknowledgements

LD Elements originated at the Dutch national infrastructure for digital heritage (NDE).