Skip to content

docs: document reproducible E2E setup #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jt55401 merged 1 commit into from
Aug 28, 2026
Merged

docs: document reproducible E2E setup #4

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 115 additions & 51 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,23 @@
# HTMLTrust end-to-end harness

This repository runs a Docker-backed HTMLTrust simulation. It creates trust-directory records, publishes signed WordPress and Hugo pages, verifies them with Playwright, records votes and reports, and writes a report under `results/`.
Run the HTMLTrust simulation from a fresh checkout, or run its unit tests and
TypeScript build. This repository uses local packages from sibling checkouts.

The harness is public and expects sibling repositories. Install and run it from the layout below.
Status: active integration harness
Primary readers: contributors and CI maintainers
Start here: use the pinned v0.2.2-compatible layout below

## Required checkout layout
## Choose a path

Clone these repositories into one parent directory:
- For a quick local check, prepare the sibling packages, then run `npm test`
and `npm run build`.
- For the integration smoke test, install Docker, Hugo, and Ollama, then run
`npm start -- scenario-small.yaml`.
- For the browser phase in a Playwright container, use the split flow below.

## Checkout layout and compatible revisions

The package manifest uses these local paths:

```
htmltrust/
Expand All @@ -18,11 +29,10 @@ htmltrust/
└── htmltrust-server-reference/
```

The Docker build reads the canonicalization and server directories from the parent. Compose mounts the WordPress plugin from `htmltrust-cms-reference`. The browser phase mounts `htmltrust-browser-reference/build/chromium`.

From an empty parent directory, clone the required repositories with:
Create that layout from an empty parent directory:

```bash
mkdir htmltrust && cd htmltrust
git clone https://github.com/HTMLTrust/htmltrust-canonicalization.git
git clone https://github.com/HTMLTrust/htmltrust-browser-client.git
git clone https://github.com/HTMLTrust/htmltrust-browser-reference.git
Expand All @@ -31,110 +41,147 @@ git clone https://github.com/HTMLTrust/htmltrust-e2e.git
git clone https://github.com/HTMLTrust/htmltrust-server-reference.git
```

This harness and its CI currently use the v0.2.2-compatible stack. Pin the two
JavaScript dependencies to the revisions used by CI before installing:

```bash
git -C htmltrust-canonicalization checkout 79b0d52fecd958f8fc7ade713fe0799ca1e79626
git -C htmltrust-browser-client checkout 09e8c7552c8111a2cedd83fa45f4ffe3811bf5ca
```

The current canonicalization `main` contains the newer 0.3.x package, while
the browser client still declares a 0.2.2 peer dependency. Do not combine
those mains with this harness unless you have verified and updated the full
stack together.

## Prerequisites

- Docker Engine with Compose v2
For tests and the TypeScript build:

- Node.js 22 and npm
- Hugo on the host, because the publish phase builds generated Hugo sites
- Chromium installed for host-side Playwright runs, or the Playwright image for the split flow below
- Ollama with the configured model for a full simulation
- the sibling canonicalization and browser-client checkouts above

The unit tests, TypeScript build, and browser-extension build do not need Docker, Hugo, or Ollama.
For the full simulation, also install:

## Setup
- Docker Engine with Compose v2
- Hugo on the host
- Ollama with the model named by the scenario

The extension-aware browser phase also needs the browser-reference checkout
and its Chromium build.

## Install and check the harness

Build the local browser-client dependency before installing this repository. Its `dist/` directory is ignored by Git and is needed by the file dependency:
Build both local package dependencies before installing this repository. The
browser client's `dist/` directory is ignored by Git, and npm needs it when it
installs the local `file:` dependency.

```bash
cd ../htmltrust-browser-client
npm ci
npm run build

cd ../htmltrust-browser-reference
npm ci
npm run build:chromium

cd ../htmltrust-e2e
npm ci
npm test
npm run build
```

The browser-reference build is needed only when running the extension-aware Playwright phase. The E2E package installs canonicalization from `../htmltrust-canonicalization/javascript` and browser-client from `../htmltrust-browser-client`, so keep those paths unchanged.
These checks still need the two sibling directories. They do not start Docker,
Hugo, or Ollama. Install the browser-reference extension only for the browser
flow:

## Small simulation
```bash
cd ../htmltrust-browser-reference
npm ci --ignore-scripts=false
npm run build:chromium
cd ../htmltrust-e2e
```

The explicit flag allows the pinned Git dependency to build its `dist/`
directory when npm is configured globally to skip lifecycle scripts.

## Run the small simulation

The small scenario is the integration smoke test. It uses three authors, five consumers, and a short article set. Start Ollama in another terminal and load the model named by the scenario:
Start Ollama in another terminal and load the model configured in
`scenario-small.yaml`:

```bash
ollama serve
ollama pull llama3.2:3b
```

Run the complete host-side simulation:
From `htmltrust-e2e`, run the complete host-side simulation:

```bash
cd ../htmltrust-e2e
npm start -- scenario-small.yaml
```

The orchestrator builds and starts the Compose stack, runs all five phases, and invokes the optional Python analysis when `uv` is installed. A missing `uv` analysis does not fail the simulation; phase failures do.
The orchestrator builds the Compose stack, publishes the test sites, runs the
consumer and researcher phases, validates the results, and writes ignored
output under `results/`, `hugo-sources/`, and `hugo-sites/`. It runs the Python
analysis when `uv` is installed; that optional analysis does not decide the
simulation exit status.

## Full simulation
## Run the full simulation

The full scenario uses ten authors and 1,000 consumers. Copy it if you need to change the Ollama endpoint or model:
The checked-in full scenario uses ten authors and 1,000 consumers. Copy it
before changing the Ollama endpoint or model:

```bash
cp scenario.yaml scenario-local.yaml
# Edit scenario-local.yaml, including ollama.host or ollama.model.
npm start -- scenario-local.yaml
```

When the orchestrator runs on the host, use `http://localhost:11434` for a host Ollama service. The checked-in full scenario uses `host.docker.internal` for Docker-oriented runs.
For a host-side run, use `http://localhost:11434` as the Ollama host. The
checked-in full scenario uses `host.docker.internal` for Docker-oriented runs.

## Split Docker and browser run
## Run browser phases in Docker

Use this flow when the host does not have a Playwright browser. It runs setup and publishing from the host, then runs browser phases in the pinned Playwright image.
Use this flow when the host lacks a Playwright browser. Run it from this
repository after `npm ci` and the browser-reference Chromium build:

```bash
docker compose up -d --build --wait
npx tsx src/smoke-test.ts scenario-small.yaml
docker compose run --rm playwright npx tsx src/run-phases-3-5.ts scenario-small.yaml
```

The smoke test creates `results/ground-truth.json`. The second command reads it, runs consumer browsing, researcher reports, post-report visits, and validation. Keep `npm ci` completed first so the mounted `/workspace/node_modules` contains `tsx`, `yaml`, and the E2E dependencies.
The smoke test creates `results/ground-truth.json`. The second command runs
consumer browsing, researcher reports, post-report visits, and validation.

## Individual services and checks
## Run individual checks and services

Start only the trust directory and MongoDB when working on the API image:
Run the local checks again at any time:

```bash
docker compose up -d --build mongodb trust-server
docker compose logs -f trust-server
npm test
npm run build
```

Start the WordPress database and one site when working on the plugin mount. The harness still needs to run its infrastructure phase to create the author and install the site:
Start only the trust directory and MongoDB while working on the server image:

```bash
docker compose up -d --build wp-db wp-1
docker compose up -d --build mongodb trust-server
docker compose logs -f trust-server
```

Run the TypeScript checks without any services:
Start the WordPress database and one site while working on the CMS mount:

```bash
npm test
npm run build
docker compose up -d --build wp-db wp-1
```

Run the Python result analyzer after a simulation has produced `results/data.json`, `results/session-logs.json`, and `results/ground-truth.json`:
Analyze results after a simulation has produced the three input files:

```bash
uv run python analysis/analyze.py results
```

## Configuration

Both scenarios are YAML files. The harness accepts these environment overrides:
Both scenarios are YAML files. Override local service settings for one run:

```bash
HTMLTRUST_TRUST_SERVER_URL=http://localhost:3000 \
Expand All @@ -143,18 +190,25 @@ HTMLTRUST_ADMIN_API_KEY=my-admin-key \
npm start -- scenario-small.yaml
```

Compose also accepts `HTMLTRUST_TRUST_PORT`, `HTMLTRUST_PROXY_PORT`, `HTMLTRUST_DIRECTORY_BASE_URL`, `WP_DB_ROOT_PASSWORD`, `WP_DB_PASSWORD`, and the two API-key variables. The checked-in simulation keys are for local testing only.
Compose also accepts `HTMLTRUST_TRUST_PORT`, `HTMLTRUST_PROXY_PORT`,
`HTMLTRUST_DIRECTORY_BASE_URL`, `WP_DB_ROOT_PASSWORD`, `WP_DB_PASSWORD`,
`HTMLTRUST_GENERAL_API_KEY`, and `HTMLTRUST_ADMIN_API_KEY`. The checked-in
credentials are for local testing only.

## Troubleshooting

- `npm ci` reports a missing browser-client `dist` file: build `../htmltrust-browser-client` first, then rerun `npm ci` here.
- Docker cannot copy a sibling directory: check the layout above and run Compose from this directory.
- Hugo publication fails: install Hugo on the host and confirm `hugo version` works from this shell.
- Article generation fails: check `curl http://localhost:11434/api/tags`, pull the model, and use a scenario whose `ollama.host` is reachable from the process running `npm start`.
- Browser phases fail to resolve author hosts: run them through `docker compose run --rm playwright ...`; Docker DNS provides the `*.htmltrust.test` aliases.
- A previous run left stale databases: remove the simulation volumes with the cleanup command below, then rerun from a clean stack.
- `npm ci` cannot find a browser-client file: build
`../htmltrust-browser-client` first, then rerun `npm ci` here.
- Docker cannot copy a sibling directory: check the layout and run Compose
from `htmltrust-e2e`.
- Hugo publication fails: confirm that `hugo version` works on the host.
- Article generation fails: run `curl http://localhost:11434/api/tags`, pull
`llama3.2:3b`, and check that the scenario's Ollama host is reachable.
- Browser phases cannot resolve author hosts: run them in the Playwright
service with `docker compose run --rm playwright ...`.
- A previous run left stale databases: use the cleanup command below.

Inspect service state and logs with:
Inspect service state and logs:

```bash
docker compose ps
Expand All @@ -163,10 +217,20 @@ docker compose logs trust-server nginx wp-1 wp-2 wp-3

## Cleanup

The simulation writes ignored output to `results/`, `hugo-sources/`, and `hugo-sites/`. Remove services and their databases after a run:
Remove this Compose project's containers, network, and named volumes:

```bash
docker compose down -v
```

The command removes only this Compose project's containers, network, and named volumes. Rebuild the stack with `docker compose up -d --build --wait` for the next run.
The command does not remove source checkouts. The next run rebuilds the stack
with `docker compose up -d --build --wait`.

## Related files

- [`scenario-small.yaml`](scenario-small.yaml) defines the integration smoke
test.
- [`scenario.yaml`](scenario.yaml) defines the larger simulation.
- [`analysis/analyze.py`](analysis/analyze.py) analyzes simulation output.
- [CI workflow](.github/workflows/ci.yml) records the currently tested package
revisions.
Loading