Skip to content

About

eXistdb application for dracor.org

Resources

Contributing

Stars

13 stars

Watchers

4 watching

Forks

Latest commit

 

History

1,138 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DraCor API

This is the eXistdb application providing the API for https://dracor.org.

The API Documentation is available at https://dracor.org/doc/api/.

Getting Started

git clone https://github.com/dracor-org/dracor-api.git
cd dracor-api
docker compose up
# load data, see below

We provide a compose.yml that allows to run an eXist database with dracor-api locally, together with the supporting dracor-metrics service and a triple store. With Docker installed simply run:

docker compose up

This pulls the necessary images from Docker Hub and starts the respective containers. The eXist database will become available under http://localhost:8080/. To check that the DraCor API is up run

curl http://localhost:8088/api/v1/info

By default, when you run docker compose up for the first time, a password for the admin user of the eXist database is generated and printed to the console. If you instead want to use a specific password use the EXIST_PASSWORD environment variable like this:

EXIST_PASSWORD=mysecret docker compose up

To use the database with an empty password, e.g. on a local machine, run:

EXIST_PASSWORD= docker compose up

The docker-compose setup also includes a DraCor frontend connected to the local eXist instance. It can be accessed by opening http://localhost:8088/ in a browser.

Configuration of the Docker image

When running the dracor/api image without the compose files of this repository, set the following environment variables for the container. The image itself doesn't set any of them.

Variable Default if never set
DRACOR_API_BASE https://dracor.org/api/v1
DRACOR_ID_BASE https://dracor.org/id
FUSEKI_SERVER http://localhost:3030/dracor/
METRICS_SERVER http://localhost:8030/metrics/
FUSEKI_SECRET empty
GITHUB_WEBHOOK_SECRET empty (webhook disabled)

The values are stored in /db/dracor/config-v1.xml and /db/dracor/secrets.xml whenever the package is installed. Variables that are set replace the stored values; unset (or empty) variables keep them.

In the Docker image the package is only installed on the first start, i.e. with an empty data volume. With an existing data volume eXist DB keeps the installed version: neither a restart nor a newer image installs the package again, so changed variables are not applied either. To apply them, or to upgrade the API in an existing database, install the package with deploy.sh (see Deployment).

Load Data

To load corpus data into the database use the DraCor API calls. First add a corpus:

curl https://raw.githubusercontent.com/dracor-org/testdracor/main/corpus.xml | \
curl -X POST \
  -u admin: \
  -d@- \
  -H 'Content-type: text/xml' \
  http://localhost:8088/api/v1/corpora

Then load the TEI files for the newly added corpus (in this case test):

curl -X POST -u admin: http://localhost:8088/api/v1/corpora/test

This may take a while. Eventually the added plays can be listed with

curl http://localhost:8088/api/v1/corpora/test

With jq installed you can pretty print the JSON output like this:

curl http://localhost:8088/api/v1/corpora/test | jq

Test Setup

For running the schemathesis test suite (see test/schemathesis.sh) or for iterating on the code with a small, reproducible dataset, use the test compose overlay compose.t.yml. Unlike the default setup, this one builds the API image locally from the current working tree and automatically loads the test corpus.

COMPOSE_FILE=compose.yml:compose.t.yml docker compose up --build --wait
  • --build forces a rebuild of the API image from the current sources.
  • --wait blocks until the loadtest service has finished registering and loading the testdracor corpus, so the stack is only reported ready once test data is available.

In this setup the API is served on port 8081 (not 8088):

curl http://localhost:8081/exist/restxq/v1/info
curl http://localhost:8081/exist/restxq/v1/corpora/test

Once the stack is up you can run the schemathesis suite locally:

./test/schemathesis.sh

The DTS endpoints are additionally tested with the DTS validator and with DraCor's own DTS contract tests in test/dts, which check rules of the DTS 1.0 specification the validator does not cover. Both need Python 3.10 or newer; the script installs the validator into test/.dts-validator and writes HTML and JUnit reports to dts-reports/:

./test/dts-validator.sh
# or against another DTS API; the contract tests check the `test` corpus
# unless other corpora are given with --corpus (or --all-corpora)
./test/dts-validator.sh http://localhost:8088/api/v1/dts --corpus bash

A deployed API (staging by default) can be validated with the manually triggered DTS Validator workflow.

Reset from scratch

The API image built by the test overlay is tagged dracor/api-test and may persist across runs, along with the eXist-db data volume. To start over with a freshly compiled image and an empty database:

COMPOSE_FILE=compose.yml:compose.t.yml docker compose down --volumes --rmi local
COMPOSE_FILE=compose.yml:compose.t.yml docker compose up --build --wait
  • --volumes deletes the named volumes (including the eXist-db data volume).
  • --rmi local removes the locally built dracor/api-test image so the next up recompiles it from scratch.

Linting the OpenAPI Document

Changes to api.yaml are checked with Redocly CLI in CI (see lint-openapi.yml). The rules are configured in redocly.yaml. To run the linter locally:

npx @redocly/cli lint api.yaml

VS Code Integration

For the Visual Studio Code editor an eXist-db extension is available that allows syncing a local working directory with an eXist database thus enabling comfortable development of XQuery code.

We provide a configuration template to connect your dracor-api working copy to the dracor-v1 workspace in a local eXist database (e.g. the one started with docker compose up).

After installing the VS Code extension copy the template to create an .existdb.json configuration file:

cp .existdb.json.tmpl .existdb.json

Adjust the settings if necessary and restart VS Code. You should now be able to start the synchronization from a button in the status bar at the bottom of the editor window.

XAR Package

To build a dracor-api XAR EXPath package that can be installed via the dashboard of any eXist DB instance you can just run:

./build.sh

The package is written to the build directory. It only requires git, sed and zip.

Deployment

The deploy.sh script installs a XAR package into a running eXist DB using its REST interface, replacing any installed version of dracor-api. The corpus data remains untouched. To deploy to a local instance:

EXIST_PASSWORD= ./deploy.sh dracor-v1-latest.xar

For a remote server open an SSH tunnel to its eXist port first and pass the tunnel URL:

ssh -N -L 8888:localhost:8080 user@dev.dracor.org &
EXIST_PASSWORD=secret ./deploy.sh dracor-v1-latest.xar http://localhost:8888/exist

The Deploy GitHub workflow does the same for the DraCor servers:

Environment Deployed when
dev run manually from the Actions tab
staging pushed to main
production a release is published (not a pre-release)

The deployment branch rules of the environments restrict staging to main and production to main and tags (which only organization admins can create), so a manual run can deploy other branches only to dev.

Everything specific to a server is configured in a GitHub environment of the same name:

Name Type Description
DEPLOY_KEY secret SSH key of the user github on the server
EXIST_PASSWORD secret eXist DB password
EXIST_USER secret eXist DB user (optional, default admin)
DEPLOY_HOST variable server host name, e.g. dev.dracor.org
DEPLOY_KNOWN_HOSTS variable the server's SSH host key(s) in known_hosts format
EXIST_PORT variable eXist port on the server (default 8080)

Each environment has its own DEPLOY_KEY, which the server only accepts for a tunnel to localhost:8080. It is generated with deploy-key.sh of dracor-org-deployment, which also sets DEPLOY_KNOWN_HOSTS. The other values can be set in the repository settings under Environments or with the GitHub CLI, e.g.:

gh secret set EXIST_PASSWORD --env staging
gh variable set DEPLOY_HOST --env staging --body staging.dracor.org

staging and production only deploy main (and, for production, release tags), which only org owners can push or tag without a review.

Webhook

The DraCor API provides a webhook (/webhook/github) that can trigger an update of the corpus data when the configured GitHub repository for the corpus changes.

Note: For the webhook to work, the shared secret between DraCor and GitHub needs to be stored as gh-webhook in /db/dracor/secrets.xml in the database. When the GITHUB_WEBHOOK_SECRET environment variable of eXist DB is set, the package installation writes it there.

How to Contribute

See our CONTRIBUTING document. For ways to get in touch with the team see https://dracor.org/doc/get-in-touch.

License

dracor-api is MIT licensed.

About

eXistdb application for dracor.org

Resources

Contributing

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Used by

Contributors

Languages