Feat/ai page access - #392
Open
TomShawn wants to merge 2 commits into
Open
Conversation
Appending `.md` to any page URL now returns clean Markdown instead of an 82 KB HTML document. Measured over the 503 released doc pages, that is 38.4 MB of HTML against 3.5 MB of Markdown -- 9%, and closer to 2% on the short reference pages, where a 79 KB page carries 1.6 KB of content. The rest is navigation, scripts and styling that an LLM pays for and cannot use. The output comes from the Markdown source rather than from the rendered HTML, so tables, admonitions and code samples survive verbatim. That matters here: the docs contain shell samples with `export VAR=...`, Java samples with `import java.sql.*;`, and pg_filedump output with literal `<Header>` markers, all of which a line-oriented stripper corrupts. The sanitiser tracks code fences and passes them through untouched, treating only Docusaurus' `mdx-code-block` fences as transparent, since their contents are evaluated rather than displayed. A build-time self-check flags components that survive sanitising. It derives the component list from each file's own imports, so a component introduced later is audited without touching this plugin. Scanning for bare capitalised tags instead is unusable -- the docs are full of `<SEGID>`, `<PID>`, `<YYYYMMDD>` placeholders and Rust generics like `<T>` that are prose, not JSX. Two exclusion lists, both keyed by docs plugin id because version names are only unique within an instance -- the unreleased version of every instance is named `current`, so a flat list would take PXF down with `docs/next`. `excludeVersions` skips a version outright (1.x, legacy); `excludeFromSitemap` still exports and links the twin but keeps it out of sitemap.xml, wired up in the next commit. Nothing about the rendered site changes; this only adds files to the build output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The twins added in the previous commit had no entry point: nothing on the site linked to a `.md` URL, and sitemap.xml listed only HTML. This adds both, for the two audiences separately. For readers, a "Copy page" menu on every doc, PXF and blog page: copy the Markdown to the clipboard, open the plain-text source, or hand the page to Claude or ChatGPT as context. The deep links use the canonical origin, since a dev-server URL would be unreachable to a third party. For crawlers, `<link rel="alternate" type="text/markdown">` in the head of every page that has a twin, plus the `.md` URLs in sitemap.xml. The head link is what makes discovery possible at all -- the menu is behind an `open &&` guard, so its links never reach the server-rendered HTML. `docs/next` gets the menu but stays out of sitemap.xml. A contributor reading the dev docs should get the dev docs; a crawler should not be answering user questions out of an unreleased version, and 491 of its 516 pages are byte-identical to 2.x anyway. Cost of listing the twins at all: sitemap.xml grows from 1456 entries to 2086, and ASF's static hosting gives us no way to send `X-Robots-Tag: noindex` on the Markdown half. Deleting the `sitemap` block returns to HTML-only. The sitemap reads the exported permalinks through a module-level set rather than a file, because `postBuild` hooks run concurrently and would race; `allContentLoaded` strictly precedes all of them. Two layout notes: on narrow viewports the actions wrap onto their own line instead of being pushed out of the viewport, and they stay flush right once wrapped -- the dropdown is anchored to the trigger's right edge, so a left-aligned trigger would send the panel off-screen. Blog pages align the panel from the left instead, where the trigger sits. Verified at 375/768/1280/1440/1728/1920 px: the panel stays inside the viewport and the actions never overflow it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TomShawn
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #ISSUE_Number
Change logs
Appending
.mdto any page URL now returns clean Markdown, and both readers and crawlers have a way to find it.src/plugins/markdown-export/) — every doc, blog post and Markdown page gets a plain-Markdown twin:/docs/introduction/cbdb-overview→/docs/introduction/cbdb-overview.md. 1146 files, 8.6 MB. Nothing about the rendered site changes.<link rel="alternate" type="text/markdown">in each page's head, plus the.mdURLs insitemap.xml. Without this nothing can find the files: the menu is behind anopen &&guard, so its links never reach the server-rendered HTML.Why
Readers increasingly paste a docs URL into an AI assistant, which today spends most of the context window on markup. Across the 503 released doc pages:
sql-stmts/create-tablesql-stmts/closeThe 50× gap on short reference pages is typical — a 79 KB page carries 1.6 KB of content. The whole
sql-stmtsset is now 1.73 MB, small enough to hand over at once instead of being truncated.The twins come from the Markdown source, not from scraping the rendered HTML, so tables, admonitions and code samples survive verbatim — including cases an HTML-to-Markdown converter mangles, like shell samples with
export VAR=...or pg_filedump output containing literal<Header>markers.Preview
Cost
docstree is already 141 MB).asf-siteis content-addressed, so an unchanged.mdproduces no new object: roughly 1.2 MB in git once, then ~0.44 MB/year of churn.sitemap.xmlgrows 1456 → 2086 entries. Every page appears twice, and ASF's static hosting gives us no way to sendX-Robots-Tag: noindexon the Markdown half. Duplicate-content risk looks low (non-HTMLtext/*is generally not indexed) but is real — deleting thesitemapblock indocusaurus.config.tsreturns to HTML-only.STRUCTURAL_TAGS/OPAQUE_TAGS. The plugin warns at build time when a component leaks into the output, deriving the list from each file's own imports so new components are covered automatically.Deliberate omissions
docs/1.xskipped entirely — legacy, not worth the weight on everyasf-sitecommit. Its HTML is untouched.docs/nextgets the menu but stays out ofsitemap.xml— a contributor reading the dev docs should get the dev docs; a crawler should not answer user questions out of an unreleased version. 491 of its 516 pages are byte-identical to 2.x anyway.llms.txt— Ahrefs' server-log study of 137,000 domains found 97% ofllms.txtfiles get zero requests, AI bots never probe for the path, and no major vendor has committed to reading it.sitemap.xmlis the one discovery file crawlers demonstrably fetch.Verification
1146 twins served as
text/markdown; charset=utf-8, 0 failures, 0 residual component markup.sitemap.xmlhas 2086 entries, none pointing at a missing file and none fromdocs/nextordocs/1.x. Menu and head link present on docs 2.x, docs/next, PXF, blog and Markdown pages; absent on docs/1.x, blog list pages and.tsxpages. Layout checked at 375–1920 px: the dropdown stays inside the viewport, and article and TOC widths are unchanged frommain.Contributor's checklist
Here are some reminders before you submit your pull request: