diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..a328b2e --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +source/ +node_modules/ +dist/ +.astro/ +.wrangler/ +screenshots/ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..89f7071 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,66 @@ +# CLAUDE.md — docs.opendialog.ai migration + +Read `MIGRATION-BRIEF.md` before doing anything. This file is the short list of rules that hold on every run. + +## What this repo is + +Migrating OpenDialog's product docs off GitBook onto Astro Starlight, deployed to Cloudflare Workers static assets. 208 markdown pages, 1,589 image assets, must keep every existing URL. + +## Hard rules + +1. **`documentation` branch is read-only.** GitBook syncs to it bidirectionally. Writing to it corrupts the live site. Work on a feature branch. +2. **Never hand-edit files in `src/content/docs/`.** They are generated. Found a bug? Fix `scripts/convert.mjs` and re-run. A manual edit is silently destroyed on the next run and creates a bug that reappears at cutover. +3. **Scripts must be idempotent.** `convert.mjs` and `assets.mjs` run repeatedly, right up to cutover day, against fresh GitBook syncs. Same input must give byte-identical output. +4. **`source/` is pristine and git-ignored.** All scripts read from `source/`, write to `src/`. Never mutate `source/` in place. +5. **URLs do not change.** Not the ugly ones either. Every path in the live `sitemap.xml` must resolve. This is the acceptance test. +6. **Do not edit documentation prose.** Not to fix typos, not to improve clarity. Log it in `MIGRATION-NOTES.md`. +7. **Stop at phase gates.** Report and wait for a human. Do not chain phases. + +## Gotchas that will bite + +- **`{ attr | filter }` in prose breaks MDX builds.** OpenDialog's docs are full of this template syntax. In `.mdx` the braces parse as JSX expressions. Safe inside code fences, fatal outside them. Prefer `.md` — Starlight asides (`:::note`) work there. Only promote to `.mdx` when a Starlight component is genuinely required. +- **Images must live in `src/assets/`, never `public/`.** Files in `public/` bypass `astro:assets` entirely and ship unoptimised. With 541 MB of source images this is the whole performance story. +- **Raw `` in markdown is not optimised.** The source has 430 `
` blocks. They must become markdown `![]()` syntax or the optimisation never happens. Highest-value transformation in the project. +- **1,317 asset filenames contain spaces or parentheses**, and GitBook wraps those paths in angle brackets: `![](<../.gitbook/assets/image (149).png>)`. Handle that form or you will silently drop images. +- **50 assets have no file extension.** Sniff magic bytes and rename, or they get served with the wrong MIME type. +- **One 28 MB GIF exceeds Cloudflare's 25 MiB per-file limit** and will fail deployment. Re-encode to MP4. +- **Do not set `run_worker_first`** in `wrangler.jsonc`. Static asset requests are free and unlimited; Worker invocations are metered at 100k/day on the free plan. +- **Never make DNS changes.** Pat repoints `docs.opendialog.ai` manually at cutover. Cloudflare API access is available for Workers and deployments only. +- **Pat owns the GitBook analytics export.** Do not attempt it. +- **Routes derive from nav position in `SUMMARY.md`, not from file paths.** See `MIGRATION-NOTES.md`. Four pages sit on disk somewhere other than their nav position; path-based derivation puts them at URLs that 404 today. +- **`reference/` holds committed snapshots of the live GitBook site** — the acceptance oracle. Never fetch these live in a verification script; they must outlive GitBook. Note `sitemap.xml` is an *index* pointing at `sitemap-pages.xml`, which holds the 204 real URLs. + +## Expected counts + +If your script's numbers diverge sharply from these, the script is wrong — do not adjust the expectation to match the output. + +| | | +|---|---| +| Markdown files | 208 | +| `{% hint %}` blocks | 258 | +| `{% content-ref %}` | 51 | +| `{% embed %}` | 38 | +| Relative `.md` links | 499 | +| ` ` artefacts | 1,203 | +| Asset files | 1,589 | +| Content pages (excl. `SUMMARY.md` and 3 files under `.gitbook/`) | 204 | +| `SUMMARY.md` nav entries | 204 | +| Live URLs in `sitemap-pages.xml` | 204 | + +The brief's "~131 live URLs" was wrong — 131 is the `llms.txt` nav count, not the published +URL set. Measured against the live sitemap: 204. Route parity is judged against 204. + +## Commands + +```bash +npm run dev # local dev server +npm run build # astro build -> dist/ +node scripts/convert.mjs # source/ -> src/content/docs/ +node scripts/assets.mjs # rename, re-encode, rewrite refs +node scripts/verify-routes.mjs # built routes vs live sitemap.xml +npx wrangler deploy # deploy to Cloudflare +``` + +## Definition of done for any change + +`astro build` succeeds, `verify-routes.mjs` passes, no new broken internal links, and anything ambiguous is written up in `MIGRATION-NOTES.md`. diff --git a/MIGRATION-BRIEF.md b/MIGRATION-BRIEF.md new file mode 100644 index 0000000..d101cad --- /dev/null +++ b/MIGRATION-BRIEF.md @@ -0,0 +1,324 @@ +# Brief: migrate docs.opendialog.ai from GitBook to Astro Starlight + +**Audience:** Claude Code (or an engineer working alongside it) +**Repo:** `opendialogai/docs`, branch `documentation` (live content, last synced 29 Apr 2026) +**Target:** Astro Starlight, deployed to Cloudflare Workers static assets +**Estimated effort:** 9–13 engineering days across 6 phases + +--- + +## Mission + +Replace GitBook with a self-hosted Astro Starlight site serving the same content at the same URLs, at zero hosting cost. Success means: every URL that resolves today resolves after cutover, the site looks like OpenDialog, search works, and the page weight drops by an order of magnitude. + +This is a **content transformation** job, not a writing job. Do not rewrite, reword, restructure or "improve" documentation prose. If content looks wrong, log it in `MIGRATION-NOTES.md` and move on. + +--- + +## Non-negotiables + +1. **URLs must not change.** Every path in the live `sitemap.xml` must resolve on the new site, either directly or via a 301. This is the acceptance test that matters most. +2. **The conversion must be a script, not manual edits.** `scripts/convert.mjs` must be idempotent and re-runnable from a fresh GitBook sync at any point up to cutover day. Never hand-edit a converted file — fix the script and re-run. +3. **Never write to the `documentation` branch.** GitBook syncs to it bidirectionally. Work on a new branch. Treat `documentation` as read-only source. +4. **Never touch the live GitBook site or its settings.** It stays running until cutover. +5. **Stop at each phase gate** for human review. Do not run phases end to end unattended. + +--- + +## Ground truth: what is in the source repo + +Audited 29 Jul 2026. Use these numbers as expected counts — if your script produces wildly different figures, the script is wrong. + +### Content +| Item | Count | +|---|---| +| Markdown files | 208 | +| Pages in live nav (`llms.txt`) | 131 | +| Pages not referenced in `SUMMARY.md` | 3 | +| Relative `.md` links to rewrite | 499 | +| ` ` entity artefacts to strip | 1,203 | +| Files with existing frontmatter (`description`) | 74 | + +### GitBook block syntax +| Block | Count | +|---|---| +| `{% hint %}` — info 106, success 113, warning 35, danger 4 | 258 | +| `{% content-ref %}` | 51 | +| `{% embed %}` — Loom 19, YouTube 17, other 3 | 38 | +| `{% stepper %}` / `{% step %}` | 3 / 15 | +| `{% code %}` | 7 | +| `{% columns %}` / `{% column %}` | 1 / 2 | +| `{% file %}` | 1 | + +### Raw HTML already embedded in the markdown +`
` × 430, `` × 436, `` × 131, `
` × 52, `
` × 22, `` × 18, `
`/`` × 14. + +### Assets — `.gitbook/assets/` +| Item | Value | +|---|---| +| Total files | 1,589 | +| Total weight | **541 MB** (repo is 543 MB) | +| PNGs | 1,506 (20 over 1 MB, largest 14 MB) | +| Filenames with spaces or parentheses | **1,317** | +| Files with **no extension** | **50** | +| Files over Cloudflare's 25 MiB cap | 1 — a 28 MB GIF | +| Markdown image refs `![](...)` | 113 | + +### Other +- `.gitbook/includes/navigate-to-the-next-level-....md` — one reusable snippet, transcluded somewhere. Find its usages and inline it. +- Branches `3.9` and `release/3.10` are version snapshots. **Out of scope — single-version site only.** Do not build versioning. + +--- + +## Target stack + +| Concern | Choice | +|---|---| +| Framework | Astro + Starlight (latest) | +| Content | `.md` by default, `.mdx` only where a Starlight component is required | +| Search | Pagefind (Starlight built-in, zero config, free) | +| Hosting | Cloudflare Workers static assets via `wrangler` | +| CI | Cloudflare Workers Builds (Git-connected) | +| Images | `astro:assets` build-time optimisation | +| `llms.txt` | `starlight-llms-txt` plugin | + +### Why Workers and not Pages +Cloudflare now recommends Workers for new projects; Pages is in maintenance for new work. Relevant limits on the free plan: + +- 20,000 files per version (we need ~1,800) ✅ +- **25 MiB max per file** — the 28 MB GIF violates this and must be re-encoded ⚠️ +- Requests to static assets are free and unlimited +- `_redirects`: 2,000 static + 100 dynamic rules +- The 100,000 requests/day free-plan limit applies to *Worker script invocations*, not static asset requests. Do not set `run_worker_first`; this site needs no Worker logic. + +`wrangler.jsonc`: +```jsonc +{ + "name": "opendialog-docs", + "compatibility_date": "2026-07-29", + "assets": { "directory": "./dist" } +} +``` +Build command `npx astro build`, output `./dist`. + +--- + +## Repo layout to produce + +``` +/ +├── CLAUDE.md # standing rules — read this first +├── MIGRATION-BRIEF.md # this file +├── MIGRATION-NOTES.md # append-only log of decisions + anomalies +├── astro.config.mjs # Starlight config, sidebar generated from SUMMARY.md +├── wrangler.jsonc +├── package.json +├── scripts/ +│ ├── convert.mjs # GitBook markdown -> Starlight (idempotent) +│ ├── assets.mjs # rename, sniff, re-encode, rewrite refs +│ ├── sidebar.mjs # SUMMARY.md -> sidebar config +│ └── verify-routes.mjs # generated routes vs live sitemap.xml +├── src/ +│ ├── assets/ # ALL images live here (NOT public/) so Astro optimises them +│ ├── components/ # Embed.astro, and any Starlight overrides +│ ├── content/docs/ # converted pages — path == URL +│ ├── content.config.ts +│ └── styles/custom.css # OpenDialog brand tokens +├── public/ +│ └── _redirects # only for URL mismatches found in Phase 5 +└── source/ # pristine copy of the GitBook export (git-ignored) +``` + +**Critical:** images must live under `src/`, not `public/`. Assets in `public/` are copied verbatim and are **not** optimised. Given 541 MB of source images, this single decision is most of the performance win. + +--- + +## Phases + +Each phase ends at a gate. Stop, report, wait for a human. + +--- + +### Phase 1 — Scaffold and theme +**Goal:** an ugly but deployed site, so every later phase has a real target. + +1. Scaffold Starlight: `npm create astro@latest -- --template starlight` +2. Copy the GitBook export into `source/` (git-ignored) as the pristine input. All scripts read from `source/` and write to `src/`. This is what makes re-running safe. +3. Configure `astro.config.mjs`: site URL `https://docs.opendialog.ai`, title, logo, social links. +4. Brand theming in `src/styles/custom.css` using Starlight's CSS custom properties (`--sl-color-accent`, `--sl-color-gray-*`, font stack). Pull actual colours and fonts from opendialog.ai — do not invent a palette. +5. Add `wrangler.jsonc`, connect Cloudflare Workers Builds, get a preview URL live with a handful of manually copied pages. + +**Gate:** a Cloudflare preview URL renders 3–5 real pages with OpenDialog branding. + +--- + +### Phase 2 — Conversion script +**Goal:** `node scripts/convert.mjs` turns all 208 source files into valid Starlight content. + +Build it as a unified/remark pipeline where practical, regex only for the `{% %}` block syntax. It must be **idempotent**: running twice produces identical output. + +#### Format decision: `.md` vs `.mdx` +Default to `.md`. Starlight's aside syntax (`:::note`) works in plain `.md`, which covers 258 of the 341 block conversions with no MDX needed. + +Promote a file to `.mdx` **only** if it needs a Starlight component (``, ``, ``). When you do promote a file, its raw HTML becomes JSX and must be normalised: +- self-close void elements: `
` → `
`, `` → `` +- `class=` → `className=` +- `style="a: b"` → `style={{ a: 'b' }}` +- unescaped `{` and `<` in prose must be escaped + +The OpenDialog content is full of `{ attribute | filter }` template syntax in prose and code. **In `.mdx` those braces will be parsed as expressions and break the build.** Inside fenced code blocks they are safe. Outside them, they are not. Audit for this specifically — it is the single most likely cause of build failures. + +#### Conversion mapping + +| GitBook | Starlight | Notes | +|---|---|---| +| `{% hint style="info" %}` | `:::note` | | +| `{% hint style="success" %}` | `:::tip` | | +| `{% hint style="warning" %}` | `:::caution` | | +| `{% hint style="danger" %}` | `:::danger` | | +| `{% content-ref url="x" %}` | `` | file becomes `.mdx`; title from target page's H1 | +| `{% embed url="loom/youtube" %}` | `` | custom component, see below | +| `{% stepper %}` / `{% step %}` | `` + ordered list | file becomes `.mdx` | +| `{% code title="x" %}` | ` ```lang title="x" ` | stays `.md` | +| `{% columns %}` / `{% column %}` | `` | only 3 instances — hand-check output | +| `{% file src="x" %}` | plain markdown link | 1 instance | +| `{% include "..." %}` | inline the snippet content | 1 include file | +| `
` | `![alt](src)` + `
` or caption text | **see below — important** | +| ` ` | a single space | 1,203 occurrences | +| `` | keep as-is | add a CSS rule in `custom.css` | +| `[x](./page.md)` | `[x](/route/path)` | 499 links; resolve against the route map | + +#### `
`/`` — do not skip this +430 `
` blocks wrap raw `` tags. **Raw HTML `` in markdown is not processed by `astro:assets`** — those images ship at full original size and the whole performance benefit evaporates. Convert them to markdown image syntax (`![alt](relative/path.png)`) so Astro's pipeline picks them up. Preserve `
` text as a caption below the image. This is the highest-value single transformation in the whole migration. + +#### `` component +Write `src/components/Embed.astro`. Takes a `url`, detects Loom vs YouTube, renders a lazy `