From e6bf95cf6562eef29661b431eb3484f1346f7ab9 Mon Sep 17 00:00:00 2001 From: Jason Grey Date: Fri, 28 Aug 2026 01:14:17 -0500 Subject: [PATCH] docs: document reproducible E2E setup --- README.md | 166 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 115 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index c18e4c7..729e519 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,23 @@ # HTMLTrust end-to-end harness -This repository runs a Docker-backed HTMLTrust simulation. It creates trust-directory records, publishes signed WordPress and Hugo pages, verifies them with Playwright, records votes and reports, and writes a report under `results/`. +Run the HTMLTrust simulation from a fresh checkout, or run its unit tests and +TypeScript build. This repository uses local packages from sibling checkouts. -The harness is public and expects sibling repositories. Install and run it from the layout below. +Status: active integration harness +Primary readers: contributors and CI maintainers +Start here: use the pinned v0.2.2-compatible layout below -## Required checkout layout +## Choose a path -Clone these repositories into one parent directory: +- For a quick local check, prepare the sibling packages, then run `npm test` + and `npm run build`. +- For the integration smoke test, install Docker, Hugo, and Ollama, then run + `npm start -- scenario-small.yaml`. +- For the browser phase in a Playwright container, use the split flow below. + +## Checkout layout and compatible revisions + +The package manifest uses these local paths: ``` htmltrust/ @@ -18,11 +29,10 @@ htmltrust/ └── htmltrust-server-reference/ ``` -The Docker build reads the canonicalization and server directories from the parent. Compose mounts the WordPress plugin from `htmltrust-cms-reference`. The browser phase mounts `htmltrust-browser-reference/build/chromium`. - -From an empty parent directory, clone the required repositories with: +Create that layout from an empty parent directory: ```bash +mkdir htmltrust && cd htmltrust git clone https://github.com/HTMLTrust/htmltrust-canonicalization.git git clone https://github.com/HTMLTrust/htmltrust-browser-client.git git clone https://github.com/HTMLTrust/htmltrust-browser-reference.git @@ -31,58 +41,92 @@ git clone https://github.com/HTMLTrust/htmltrust-e2e.git git clone https://github.com/HTMLTrust/htmltrust-server-reference.git ``` +This harness and its CI currently use the v0.2.2-compatible stack. Pin the two +JavaScript dependencies to the revisions used by CI before installing: + +```bash +git -C htmltrust-canonicalization checkout 79b0d52fecd958f8fc7ade713fe0799ca1e79626 +git -C htmltrust-browser-client checkout 09e8c7552c8111a2cedd83fa45f4ffe3811bf5ca +``` + +The current canonicalization `main` contains the newer 0.3.x package, while +the browser client still declares a 0.2.2 peer dependency. Do not combine +those mains with this harness unless you have verified and updated the full +stack together. + ## Prerequisites -- Docker Engine with Compose v2 +For tests and the TypeScript build: + - Node.js 22 and npm -- Hugo on the host, because the publish phase builds generated Hugo sites -- Chromium installed for host-side Playwright runs, or the Playwright image for the split flow below -- Ollama with the configured model for a full simulation +- the sibling canonicalization and browser-client checkouts above -The unit tests, TypeScript build, and browser-extension build do not need Docker, Hugo, or Ollama. +For the full simulation, also install: -## Setup +- Docker Engine with Compose v2 +- Hugo on the host +- Ollama with the model named by the scenario + +The extension-aware browser phase also needs the browser-reference checkout +and its Chromium build. + +## Install and check the harness -Build the local browser-client dependency before installing this repository. Its `dist/` directory is ignored by Git and is needed by the file dependency: +Build both local package dependencies before installing this repository. The +browser client's `dist/` directory is ignored by Git, and npm needs it when it +installs the local `file:` dependency. ```bash cd ../htmltrust-browser-client npm ci npm run build -cd ../htmltrust-browser-reference -npm ci -npm run build:chromium - cd ../htmltrust-e2e npm ci npm test npm run build ``` -The browser-reference build is needed only when running the extension-aware Playwright phase. The E2E package installs canonicalization from `../htmltrust-canonicalization/javascript` and browser-client from `../htmltrust-browser-client`, so keep those paths unchanged. +These checks still need the two sibling directories. They do not start Docker, +Hugo, or Ollama. Install the browser-reference extension only for the browser +flow: -## Small simulation +```bash +cd ../htmltrust-browser-reference +npm ci --ignore-scripts=false +npm run build:chromium +cd ../htmltrust-e2e +``` + +The explicit flag allows the pinned Git dependency to build its `dist/` +directory when npm is configured globally to skip lifecycle scripts. + +## Run the small simulation -The small scenario is the integration smoke test. It uses three authors, five consumers, and a short article set. Start Ollama in another terminal and load the model named by the scenario: +Start Ollama in another terminal and load the model configured in +`scenario-small.yaml`: ```bash ollama serve ollama pull llama3.2:3b ``` -Run the complete host-side simulation: +From `htmltrust-e2e`, run the complete host-side simulation: ```bash -cd ../htmltrust-e2e npm start -- scenario-small.yaml ``` -The orchestrator builds and starts the Compose stack, runs all five phases, and invokes the optional Python analysis when `uv` is installed. A missing `uv` analysis does not fail the simulation; phase failures do. +The orchestrator builds the Compose stack, publishes the test sites, runs the +consumer and researcher phases, validates the results, and writes ignored +output under `results/`, `hugo-sources/`, and `hugo-sites/`. It runs the Python +analysis when `uv` is installed; that optional analysis does not decide the +simulation exit status. -## Full simulation +## Run the full simulation -The full scenario uses ten authors and 1,000 consumers. Copy it if you need to change the Ollama endpoint or model: +The checked-in full scenario uses ten authors and 1,000 consumers. Copy it +before changing the Ollama endpoint or model: ```bash cp scenario.yaml scenario-local.yaml @@ -90,11 +134,13 @@ cp scenario.yaml scenario-local.yaml npm start -- scenario-local.yaml ``` -When the orchestrator runs on the host, use `http://localhost:11434` for a host Ollama service. The checked-in full scenario uses `host.docker.internal` for Docker-oriented runs. +For a host-side run, use `http://localhost:11434` as the Ollama host. The +checked-in full scenario uses `host.docker.internal` for Docker-oriented runs. -## Split Docker and browser run +## Run browser phases in Docker -Use this flow when the host does not have a Playwright browser. It runs setup and publishing from the host, then runs browser phases in the pinned Playwright image. +Use this flow when the host lacks a Playwright browser. Run it from this +repository after `npm ci` and the browser-reference Chromium build: ```bash docker compose up -d --build --wait @@ -102,31 +148,32 @@ npx tsx src/smoke-test.ts scenario-small.yaml docker compose run --rm playwright npx tsx src/run-phases-3-5.ts scenario-small.yaml ``` -The smoke test creates `results/ground-truth.json`. The second command reads it, runs consumer browsing, researcher reports, post-report visits, and validation. Keep `npm ci` completed first so the mounted `/workspace/node_modules` contains `tsx`, `yaml`, and the E2E dependencies. +The smoke test creates `results/ground-truth.json`. The second command runs +consumer browsing, researcher reports, post-report visits, and validation. -## Individual services and checks +## Run individual checks and services -Start only the trust directory and MongoDB when working on the API image: +Run the local checks again at any time: ```bash -docker compose up -d --build mongodb trust-server -docker compose logs -f trust-server +npm test +npm run build ``` -Start the WordPress database and one site when working on the plugin mount. The harness still needs to run its infrastructure phase to create the author and install the site: +Start only the trust directory and MongoDB while working on the server image: ```bash -docker compose up -d --build wp-db wp-1 +docker compose up -d --build mongodb trust-server +docker compose logs -f trust-server ``` -Run the TypeScript checks without any services: +Start the WordPress database and one site while working on the CMS mount: ```bash -npm test -npm run build +docker compose up -d --build wp-db wp-1 ``` -Run the Python result analyzer after a simulation has produced `results/data.json`, `results/session-logs.json`, and `results/ground-truth.json`: +Analyze results after a simulation has produced the three input files: ```bash uv run python analysis/analyze.py results @@ -134,7 +181,7 @@ uv run python analysis/analyze.py results ## Configuration -Both scenarios are YAML files. The harness accepts these environment overrides: +Both scenarios are YAML files. Override local service settings for one run: ```bash HTMLTRUST_TRUST_SERVER_URL=http://localhost:3000 \ @@ -143,18 +190,25 @@ HTMLTRUST_ADMIN_API_KEY=my-admin-key \ npm start -- scenario-small.yaml ``` -Compose also accepts `HTMLTRUST_TRUST_PORT`, `HTMLTRUST_PROXY_PORT`, `HTMLTRUST_DIRECTORY_BASE_URL`, `WP_DB_ROOT_PASSWORD`, `WP_DB_PASSWORD`, and the two API-key variables. The checked-in simulation keys are for local testing only. +Compose also accepts `HTMLTRUST_TRUST_PORT`, `HTMLTRUST_PROXY_PORT`, +`HTMLTRUST_DIRECTORY_BASE_URL`, `WP_DB_ROOT_PASSWORD`, `WP_DB_PASSWORD`, +`HTMLTRUST_GENERAL_API_KEY`, and `HTMLTRUST_ADMIN_API_KEY`. The checked-in +credentials are for local testing only. ## Troubleshooting -- `npm ci` reports a missing browser-client `dist` file: build `../htmltrust-browser-client` first, then rerun `npm ci` here. -- Docker cannot copy a sibling directory: check the layout above and run Compose from this directory. -- Hugo publication fails: install Hugo on the host and confirm `hugo version` works from this shell. -- Article generation fails: check `curl http://localhost:11434/api/tags`, pull the model, and use a scenario whose `ollama.host` is reachable from the process running `npm start`. -- Browser phases fail to resolve author hosts: run them through `docker compose run --rm playwright ...`; Docker DNS provides the `*.htmltrust.test` aliases. -- A previous run left stale databases: remove the simulation volumes with the cleanup command below, then rerun from a clean stack. +- `npm ci` cannot find a browser-client file: build + `../htmltrust-browser-client` first, then rerun `npm ci` here. +- Docker cannot copy a sibling directory: check the layout and run Compose + from `htmltrust-e2e`. +- Hugo publication fails: confirm that `hugo version` works on the host. +- Article generation fails: run `curl http://localhost:11434/api/tags`, pull + `llama3.2:3b`, and check that the scenario's Ollama host is reachable. +- Browser phases cannot resolve author hosts: run them in the Playwright + service with `docker compose run --rm playwright ...`. +- A previous run left stale databases: use the cleanup command below. -Inspect service state and logs with: +Inspect service state and logs: ```bash docker compose ps @@ -163,10 +217,20 @@ docker compose logs trust-server nginx wp-1 wp-2 wp-3 ## Cleanup -The simulation writes ignored output to `results/`, `hugo-sources/`, and `hugo-sites/`. Remove services and their databases after a run: +Remove this Compose project's containers, network, and named volumes: ```bash docker compose down -v ``` -The command removes only this Compose project's containers, network, and named volumes. Rebuild the stack with `docker compose up -d --build --wait` for the next run. +The command does not remove source checkouts. The next run rebuilds the stack +with `docker compose up -d --build --wait`. + +## Related files + +- [`scenario-small.yaml`](scenario-small.yaml) defines the integration smoke + test. +- [`scenario.yaml`](scenario.yaml) defines the larger simulation. +- [`analysis/analyze.py`](analysis/analyze.py) analyzes simulation output. +- [CI workflow](.github/workflows/ci.yml) records the currently tested package + revisions.