URLSource caches raw page HTML: scripts, comments and attributes reach a committed cache

[cmungall]

URLSource caches the raw HTML it fetched, including <script> blocks, inline styles, comments, and every tag attribute. For a project that commits its reference cache to a public git repository, that stores a lot of material that is not the reference text — and page scripts and data attributes routinely carry things that should not be copied into somebody's repo, including signed asset URLs and API-key parameters.

In the dismech cache, 150 of 199 content_type: url entries contain a <script> block.

The exposure there turns out to be benign on inspection — the signed URLs are public CDN figure links with Expires=2147483647, and the one api_key is an empty template parameter — so this is a hygiene issue rather than an incident. But the general shape is not benign: whatever a publisher puts in a page script ends up verbatim in a committed cache file, and nobody reviews 199 HTML dumps.

Unlike the HTML full text path, which goes through HTMLExtractor, URLSource does no extraction at all — it stores the response body. I checked 0.3.0rc1: url.py has no script removal, no comment stripping, and no attribute filtering.

Suggested fix

Sanitize before caching, keeping what a quoted excerpt can legitimately need:

• drop script, style, noscript, template elements and HTML comments;
• drop all tag attributes except table-structural ones (rowspan, colspan, scope), which carry meaning for a quoted table row;
• leave plain text, XML, and extracted PDF content alone.

This is source extraction rather than a rendering decision, so it does not need to model what a browser would show. Keeping body markup and table structure is enough for excerpt validation.

Context

dismech currently does this as a runtime patch over URLSource, which is how I found it. That patch is the last one it still carries — the other ten were retired once their fixes landed here (#66-74, #85, #87, #88) — and it will be deleted when this lands. Filing it rather than keeping the workaround quiet, per the rule that a patch needs an upstream issue and an exit.

[github-actions]

Summary

Verified against main (src/linkml_reference_validator/etl/sources/url.py): the report is accurate. URLSource.fetch for non-PDF content goes ContentAcquirer().fetch_bytes → self._decode → straight into ReferenceContent.content (content_type="url"), with only a regex <title> pull for the title. No tag stripping, no attribute filtering, no comment removal anywhere in that path — the raw response body is what gets written to the cache file.

That's a genuinely different code path from the one that might look like it already handles this: HTMLExtractor (etl/extract/html.py), used for content_type: full_text via extract_full_text/extract, does call tag.decompose() on script/style (line 214) — but it then collapses everything to plain text via get_text(). It was never built to preserve markup, so it's not reusable as-is for this fix; URLSource needs its own sanitize-in-place step that keeps table structure (rowspan/colspan/scope) rather than flattening to text. Worth keeping the two paths conceptually distinct in the PR: content_type: full_text = extracted plain text, content_type: url = sanitized-but-structural HTML.

This is #12 of the dismech-patch series

Same shape and origin as the recent run of upstream issues from the dismech runtime patches: #66, #67–70, #71, #72, #73, #74, #85, #87, #88 — all filed the same way (patch found a bug → issue filed → fix landed here → patch retired), all closed within hours to a day of filing, all labeled bug. Per the issue body, this is the last patch dismech still carries, so closing it out removes the last of that scaffolding. I've applied the bug label to match the pattern.

Note for whoever picks this up

The suggested fix is well-scoped and self-contained (drop script/style/noscript/template + HTML comments, strip all attributes except rowspan/colspan/scope, leave plain text/XML/PDF paths untouched). Given how consistently issues in this series have gone from filed → fixed same-day:

@dragon-ai-agent please implement the sanitization described in this issue in URLSource (src/linkml_reference_validator/etl/sources/url.py). Only the non-PDF HTML/text branch of fetch is affected. Use BeautifulSoup (already a project dependency, see etl/extract/html.py for precedent) to: (1) decompose script, style, noscript, and template elements; (2) remove HTML comments (bs4.Comment instances); (3) strip every tag attribute except rowspan, colspan, and scope. Apply this only to HTML content (skip plain text) before it is written into ReferenceContent.content. Follow this repo's TDD convention from CLAUDE.md — add tests first (e.g. a fixture with a <script> block, an inline onclick attribute, an HTML comment, and a table with colspan/rowspan) — and add a docstring example per the project's doctest convention. Open a PR against main.