SOF-7992: docs agent plans and the Ask AI widget - #391
Merged
Conversation
Adds two planning documents and consolidates the vocabulary across the set: - docs-agent-implementation.md: the execution plan tying the strategy and architecture plans together. Defines the shared phase numbering (0-5), milestones M1-M8 with acceptance criteria, and a decision register. - docs-agent-platform-actions.md: Phase 5, letting the agent execute actions in the user's platform session. The action vocabulary is the Cypress/Gherkin step catalog, so the agent can only do what the test suite proves works. Naming is unified: "the platform" is platform.mat3ra.com (repository `web-app`), and docs-agent-web-app.md is renamed to docs-agent-web-delivery.md, which describes browser delivery of the documentation agent and was easily confused with the platform repository. Records two decisions taken since the plans were written: the agent core now lives in the documentation-agent repository rather than scripts/rag (D6), and the model backend is a provider abstraction defaulting to Gemini with Claude selectable (D5). scripts/rag is marked superseded accordingly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 1 is complete. The evaluation harness and golden set are built in the documentation-agent repository, so the plan now carries the measured BM25 baseline rather than an intention to measure one. Sets D8 from that measurement instead of aspiration: recall@5 >= 0.65 and MRR >= 0.45, just below the recorded 0.688 and 0.494, plus zero hallucinated URLs and correct refusal on the unanswerable questions. Records two findings the numbers make concrete. The baseline is a lower bound on the agent rather than a verdict on it, because the agent reformulates and re-searches - a page at rank 8 for the user's original phrasing is often still cited correctly. And the paraphrase weakness predicted in the RAG plan now has a specific failing case to beat: a question about REST API authentication retrieves the JupyterLite authentication page first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Corrects the retrieval numbers after the golden set's expected pages were broadened to list every page that genuinely answers a question: recall@1 0.406, recall@5 0.688, MRR 0.543, so the D8 gate on MRR rises to 0.50. The set holds 37 questions, 32 of them answerable. Records what the answer harness found on its first run. Refusal scored 4 of 5: asked whether the platform is faster than VASP on a 64 GB laptop, the agent said it is significantly faster and justified it with real hardware specifications - documented specifications, undocumented comparison. The dangerous shape is a question adjacent to documented material, where retrieval returns something plausible and the model completes the argument itself, which no retrieval improvement fixes. Faithfulness scored 0.892 on four answers that invented interface details. No URL was ever invented, so half the grounding rule holds while the half covering UI element names does not. Both are prompt problems rather than retrieval problems, so tightening the prompt and re-measuring becomes the first next action, ahead of new features. Section 8 listed hallucinated UI paths as a risk to be verified by the judge rather than assumed; it now has been. Also refreshes two statements overtaken by the work: the status line, and the site count in the definition of done, which is eight rather than four. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Milestone M4. A launcher on every page opens a chat that streams grounded answers from the agent service, with the pages it used as links. Wired once in mkdocs-base.yml. Every site config inherits from it and none override the asset lists, so this reaches all eight builds without touching them individually. Written as a framework-free module - DocsAgent.mount(element, options) - so the platform application can host the same code in its own shell rather than growing a second implementation. It assumes nothing about MkDocs; the documentation site is just its first caller. Model output is rendered by building DOM nodes and setting textContent, and innerHTML is never used for anything the model or the corpus produced. That makes injection structurally impossible instead of filtered, which is why this carries a small Markdown renderer for the subset the agent emits rather than a Markdown parser plus a sanitiser. Links are only made clickable when they are https, so a poisoned URL cannot become a trap. The launcher only appears once the service answers its health check, and a failed request degrades to a sentence pointing at the documentation search box. A documentation page must never break, or show a control that leads nowhere, because the assistant is down. The widget shows the pages retrieval touched only when the answer has not already cited its own. Those two lists differ - the answer cites what the model used, the event carries everything retrieval returned, including near-misses it correctly ignored - so printing both duplicated the useful list and dressed the near-misses up as sources. Verified in a browser against the running service: a streamed multi-search answer with headings, a code block and working citation links; the degraded path with the service unreachable; and the panel at phone width. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
M3 and M4 are built and verified end to end locally, so Phase 2 is under way rather than planned. What remains before a public beta is deployment and the launch gate, not features, and the next actions are reordered to say so. Adds the point that the faithfulness number cannot gate a launch until it is measured across repeated runs or a larger set, since one run cannot separate a prompt change from sampling noise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
timurbazhirov
changed the title
…eshes The agent now runs on Cloud Run, so the widget has a real endpoint. It is the default Cloud Run hostname for the beta; a mat3ra.com subdomain replaces it before general availability. Adds the workflow that tells the agent repository to rebuild when this repository's documentation changes, since the agent answers from a snapshot baked into its image and that snapshot goes stale on every merge. The workflow does no ingestion of its own - all agent logic lives in the agent repository, which checks this one out at the commit it is given - so the two pipelines stay decoupled. It needs a cross-repository dispatch token and skips with a message when that secret is absent, rather than failing the documentation build. Note that merging this to main does not launch anything by itself: the widget only appears once the service answers its health check, so the launch decision remains the hardening milestone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sion M5 is done and the agent is deployed and answering, so only the launch gate remains. Records what was verified against the deployed service rather than locally, including the rollback drill, and three findings worth not rediscovering: a commit SHA cannot be passed to git clone --branch, so pinning the index silently needed a different checkout; builds submit --tag cannot pass the build argument that pinning requires; and a brand-new project fails its first deploy on permission propagation rather than on any policy. The consequence for sequencing is now stated plainly. The service is live and the widget appears only once it answers a health check, so merging the widget to main is the launch, and it waits on the hardening gate rather than on anything technical. Marks D3 done - Vertex confirmed end to end from Cloud Run - and D8's refusal bar as reached rather than pending. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things the first real use made obvious. The Mat3ra mark now sits beside the "Ask AI" label in the launcher and the panel header. It is inlined and drawn in currentColor, so it works on the purple launcher and the light header without shipping two assets or depending on an image path that exists only on the documentation site. Source URLs are now links. The model lists them as bare URLs and the renderer only understood the bracketed Markdown form, so every citation arrived as dead text - a source you cannot click is not much of a citation. Emphasised product terms - Materials Bank, Materials Designer - now link to the page that defines them, which is what a reader wants to open. The mapping comes from the service, which derives it from its index, so a link can only ever name a page that exists; the model is never asked to produce one. Terms stay styled as emphasis with a dotted underline rather than turning the answer into a field of links. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sources and glossary terms opened in a new tab. A documentation link is the continuation of the answer rather than a detour, and spawning tabs behind the reader is a habit the documentation itself does not have. The consequence is worth stating: the conversation is held in memory only, so following a link ends it, and the back button returns to the documentation page rather than to the exchange. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sources and glossary terms rendered in the surrounding text colour, so nothing signalled they could be clicked. They now use the documentation's own link colour where the theme defines one, and a conventional blue wherever else the widget is embedded, since the service's try-it page and the platform shell have no such variable. This drops the earlier treatment of glossary terms as emphasis with a dotted underline. Reading as prose is worth less than being visibly clickable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Answers cite documentation pages and those links now open in place, so every citation was ending the exchange that produced it: the reader followed a source, the page reloaded, and the panel came back empty. The conversation now outlives the page. It is stored in the browser, restored on load, and the panel reopens if it was open, so following a link reads as continuing rather than starting over. Storage is bounded the same way the service bounds a request - the last twenty messages, capped by total size - and expires after a week, which is what makes "relive through multiple visits" tolerable rather than indefinite. A conversation that survives navigation also has to be endable, so the header gains "New chat" and the footer says plainly that the exchange is kept in the browser until then. Nothing new is sent anywhere: this is text the page already displayed, on the reader's own machine. Restoration waits for the glossary, so a restored answer gets the same links a fresh one would. Storage being full or disabled is not an error - the widget simply behaves as it did before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Following a source on a deploy preview jumped to production docs.mat3ra.com, because the corpus stores canonical URLs and the widget rendered them verbatim. That left the preview entirely, and took the conversation with it - stored sessions are per-origin, so the panel came back empty on the other side. A documentation URL is now rewritten to the origin currently serving the documentation, which is a no-op in production and makes previews and local servers behave like it. Rewriting only happens where the widget mounts itself onto a documentation build; the platform shell serves no documentation, so its citations keep pointing at the real site. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The widget ships to every documentation page, and until now nothing guarded it. Each fix in it so far was verified by hand once and then left unprotected, which is how the same bug comes back. Seventeen Playwright tests run in under three seconds and need neither cloud access nor a model call. The widget under test is the file the site ships; only the agent service is faked, at the network boundary, because a test that called a language model would be neither deterministic nor free and the service has its own suite. The fixture server answers every path with a page that mounts the widget, which is what makes the navigation tests possible: answers cite canonical production URLs that the widget rewrites onto the origin being read, so following a citation has to land somewhere the widget mounts again. Both mounting styles are covered, since they differ - the documentation site lets the script mount itself only after a health check, while the platform will mount it explicitly. Coverage is deliberately weighted towards the properties that are easy to break silently: markup in an answer is displayed rather than executed, a non-https link never becomes clickable, unknown emphasis stays plain rather than linking somewhere plausible, and the conversation survives a followed citation without resurrecting a week-old one. Both regressions these tests were written for were reintroduced on purpose and confirmed to fail the run, so the suite is known to have teeth rather than assumed to. The README says to do the same when adding to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
M4 is finished and now has browser tests, so the plan records what the first real use taught rather than only that it shipped: citations had to become clickable and same-tab, emphasised terms link through a glossary the service derives from its index, citations must stay on the build being read, and the conversation has to outlive the page once links open in place. Adds the local Google Cloud SDK to the outstanding items. It is still the June 2023 release, and Google rejects that client's token refresh, so a fresh login lasts about an hour before every call fails. Re-authenticating treats it hourly; upgrading fixes it. Nothing deployed depends on it, since Cloud Run authenticates as its own service account, but no deploy can be driven from a laptop in this state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three stacked causes, not the one the plan guessed: the machine driving builds is a different computer from the laptop whose logins kept not helping; its 2023-era gcloud minted tokens Google began rejecting outright on 2026-08-01; and the account session now requires interactive reauthentication, which no non-interactive shell can satisfy. Deploys run again - a current SDK from a local directory, fed tokens minted from application-default credentials, which still refresh. CI with federated identity bypasses the whole class of problem, which is one more reason to merge the pipeline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of SOF-7992. Plans + the Ask AI widget (M4).
Stacked on
feat/agent-rag(#389), which merges first.The widget (M4)
A launcher on every page opens a chat that streams grounded answers from the agent service, with the pages it used as links.
Wired once in
mkdocs-base.yml. Every site config inherits from it and none override the asset lists, so this reaches all eight builds without touching them individually. (Note the site count has grown past the four inAGENTS.md.)Framework-free module —
DocsAgent.mount(element, options)— so the platform application can host the same code in its own shell rather than growing a second implementation. It assumes nothing about MkDocs; the docs site is just its first caller.On rendering model output safely
The plan called for vendoring
marked+ DOMPurify. I did neither: this builds DOM nodes and setstextContent, neverinnerHTML, for anything the model or corpus produced. Injection becomes structurally impossible rather than filtered, and the widget ships with zero dependencies. Links are only made clickable when they'rehttps, so a poisoned URL can't become a trap.The cost is a small Markdown renderer covering the subset the agent emits (headings, lists, fenced code, inline code, bold, links) — worth it against a parser-plus-sanitiser pair to keep patched.
It must never break a documentation page
The launcher appears only once the service answers its health check, and a failed request degrades to one sentence pointing at the docs search box. Verified: with the endpoint pointed at a dead port, no launcher, no host element, page renders normally.
One thing the browser caught
The widget originally appended a "Pages used" list under every answer — which duplicated the answer's own Sources: section and listed pages that were retrieved but never cited (
index-dev/, an unrelated auth page). Those two lists aren't the same thing: the answer cites what the model used; the event carries everything retrieval returned. The widget now shows them only when the answer didn't cite its own, labelled "Pages searched".Verified in a browser against the running service: a streamed multi-search answer with headings, a code block and working citation links; the degraded path; and the panel at phone width.
Plans
docs-agent-implementation.mddocs-agent-platform-actions.mdNaming is unified — "the platform" is platform.mat3ra.com (repository
web-app) — anddocs-agent-web-app.mdis renamed todocs-agent-web-delivery.md, which described browser delivery of the documentation agent and read as though it were about the platform repository.The plans now carry measured results rather than intentions: the BM25 baseline, the D8 launch thresholds set from it, and what the evaluation harness found.
Notes
Internal planning documents only — the MkDocs builds read
lang/en/docs/, so nothing inplans/reaches docs.mat3ra.com..gitignoreaddsreference/, a local clone of the service repository.Link check: the 7 reported breaks are pre-existing cross-site links (
/guide/,/reference/, …) absent because only the main site was built locally, not caused by this change.🤖 Generated with Claude Code