feat(search-api-server): serve the search API as a bootable process and Docker image#646
Merged
Conversation
ddeboer
force-pushed
the
feat/search-graphql-docker
branch
2 times, most recently
from
July 23, 2026 14:06
1a3a7af to
41e2650
Compare
ddeboer
force-pushed
the
feat/search-graphql-docker
branch
from
July 24, 2026 11:05
41e2650 to
fcb4c9a
Compare
- validateSearchType silently accepted a kind outside the FieldKind union: every kind-dependent rule passed vacuously, deferring the failure to whatever consumer read the declaration first - guards the plain-JS declaration path (a mounted schema module, a SHACL generator) that the runtime validation exists for - adds the 'unknown-kind' issue reason
ddeboer
force-pushed
the
feat/search-graphql-docker
branch
3 times, most recently
from
July 24, 2026 12:19
dbf5cd6 to
fa245f9
Compare
…nd Docker image - new composition package binding the engine-agnostic search-api-graphql handler to a Typesense engine: environment config, a mounted plain-data schema-declaration module (validated at boot), and a node:http server answering /graphql, /health and a root redirect - the search-api-server bin boots from environment variables alone and reports every configuration problem in one error - the image is built from the workspace's own outputs, never from npm: docker:build (inferred by @nx/docker) stages the compiled package, the same-commit builds of its @lde/* dependencies (copy-workspace-modules) and a pruned lockfile (prune-lockfile plus an npm --package-lock-only repair step for the Nx pruning fallback); the Dockerfile is npm ci + COPY - docker:smoke boots each affected image in CI with a fixture schema and probes /health and the SDL over HTTP, so runtime-only failures (the ERR_MODULE_NOT_FOUND class) cannot pass a green build - each release tag rebuilds, smoke-tests and pushes ghcr.io/ldelements/search-api-server:<version>, independent of the npm publish - the read-only image omits the write-side peers (@lde/pipeline, @lde/dataset): every runtime import is 'import type' - recorded as ADR 15
ddeboer
force-pushed
the
feat/search-graphql-docker
branch
from
July 24, 2026 12:27
fa245f9 to
70b8185
Compare
…ge (#655) * feat(search): add the schema-module loader as @lde/search/module - loadSchemaModule() loads and validates a mounted schema-declaration module (an .mjs default-exporting SearchType declarations as plain data) and returns the validated schema plus the raw exports, so each consumer validates its own optional exports - Node-only, so it lives in a separate entry point like ./adapter and ./testing, keeping the main entry point runtime-agnostic - The one loader both the indexer image and the served-API image boot from, so the write and the read side cannot disagree about the schema * refactor(search-api-server): load the schema module via @lde/search/module - The generic load-and-validate logic moves to the shared loader; what stays here is validating the read side's optional exports (schemaOptions, engineOptions) - Add a test for the non-object optional export failure mode * feat(search-indexer): ship the search indexer as a CLI and Docker image - New @lde/search-indexer package: a Commander CLI wrapping searchIndexerPipeline, configured entirely from environment variables (boot-or-fail, all problems reported in one error) - Mounts the same schema-declaration module as the search-api-server image, loaded with the shared @lde/search/module loader - Dataset selection against a DCAT registry: explicit IRIs (DATASETS), search criteria (DATASET_CRITERIA), or the whole registry - Typesense rebuild writers per root type: in-place (default) or blue-green, with an optional derived-collection-name prefix - Optional file provenance (PROVENANCE_FILE + PIPELINE_VERSION); rejected with blue-green rebuilds, which would drop skipped datasets - Optional QLever import path behind QLEVER_IMAGE: data dumps are imported into pipeline-controlled sibling containers over the Docker socket (see ADR 16 for why a static QLever service cannot work) - --check validates configuration and schema module, then exits; the Docker smoke test boots the image with it to catch pruned-image module-resolution failures in CI - The docker.yml release workflow now derives the package from the tag, serving both search-api-server and search-indexer images
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #600 (layer 3 of 3: the prebuilt images; the handler, layer 2, shipped in #637).
Ships the search stack as two prebuilt, workspace-built Docker images that boot from the same mounted schema-declaration module, loaded by the shared
@lde/search/moduleloader so the write side and the read side cannot disagree about the schema:ghcr.io/ldelements/search-api-server(@lde/search-api-server, new): the served search API – mount a schema module, setTYPESENSE_*env vars, get/graphql(POST execution, self-contained playground,?sdl) plus/health, with CORS and depth/cost limits on by default. The composition layer binding the deliberately engine-agnostic@lde/search-api-graphqlhandler to@lde/search-typesense.ghcr.io/ldelements/search-indexer(@lde/search-indexer, new): the write side – a Commander CLI wrappingsearchIndexerPipeline, configured entirely from environment variables. Dataset selection against a DCAT registry (explicit IRIs, search criteria, or the whole registry), in-place (default) or blue-green Typesense rebuild writers, optional file provenance, and an optional QLever import path behindQLEVER_IMAGE(ADR 16).--checkvalidates config + schema module and exits; the Docker smoke boots the image with it.Design (ADR 15, ADR 16)
search:definitions, but that generator is GraphQL search API #495’s still-open scope. The mount is an.mjsdefault-exportingSearchTypedeclarations, loaded and validated at boot by the new@lde/search/moduleentry point (Node-only, like./adapterand./testing) – the “declarations built outside TypeScript” path the runtime validation exists for. Functions (derive,transform) remain expressible: the indexer projects with them, the server ignores them – one module drives both images. When GraphQL search API #495 lands, mounted SHACL becomes a second source for the same schema.@lde/search-api-graphqlwithout breaking its engine-agnosticism.docker:build(inferred by the new@nx/dockerplugin) stages the compiled package, the same-commit builds of its@lde/*dependencies (@nx/js:copy-workspace-modules) and a pruned lockfile (@nx/js:prune-lockfile+ annpm install --package-lock-onlyrepair step, because Nx’s pruning cannot map this workspace’s graph and falls back to the root lockfile); the Dockerfiles are a plainnpm ci --omit=dev+COPY. No bundling, deliberately: bundling hid missing transitives until boot in the Dataset Register (#2128).docker:smokegates CI for both images: every affected PR builds the image and boots it (the server probed over real HTTP on/health+/graphql?sdl; the indexer via--check), so runtime-only failures – theERR_MODULE_NOT_FOUNDclass – cannot pass a green build. The smoke step runs after the test suites so image builds cannot starve timing-sensitive tests on the shared runner.@lde/<package>@<version>tag triggers a workflow that derives the package from the tag, checks out that commit, rebuilds, smoke-tests and pushes:<version>+:latest. No waiting on the npm publish; image tags stay aligned with the package semver.nx release’s own Docker support (experimental, calendar-versioned, cannot npm- and Docker-publish one project) is deliberately not used.@lde/search-typesense’s write-side peers (@lde/pipeline,@lde/dataset): every runtime import isimport type, and the smoke test guards that assumption.Also
feat(search):validateSearchTypenow rejects an unknown fieldkind(newunknown-kindissue reason). Previously a typo’d kind in a plain-JS declaration passed every kind-dependent rule vacuously – exactly the path the mounted schema module relies on.Acceptance criteria covered (of #600)
/graphql+ playground from mounted definitions and engine config+server.ts– downstream (Replace the hand-rolled /graphql endpoint with the LDE search GraphQL handler netwerk-digitaal-erfgoed/dataset-register#2282)After merge
@lde/search-api-serverand@lde/search-indexerare new on npm: each first version needs the manual Trusted-Publishing bootstrap for npm consumers. The Docker images are unaffected – they build from the workspace.