SEO cleanup: sitemap, 404 metadata, coming-soon heading

[GuanzhouSong]

Fixes #129. Three independent fixes, each small.

1. Sitemap advertised /_not-found/

collectPages() emits a URL for any directory containing an index.html and filters only top-level names in an explicit deny-list. With trailingSlash: true the export writes out/_not-found/index.html, and _not-found was not on the list.

#123 (9505e9c) fixed exactly this class for 404, but only added that one name. Note the two are not symmetric: 404 exists in both forms (/404.html and /404/ both 200) while _not-found only exists as a directory.

This matters more than a stray URL: /_not-found/ serves <meta name="robots" content="noindex">, so the sitemap was actively submitting a URL that tells crawlers not to index it — the Search Console "submitted URL marked noindex" warning class.

Verified by running the real script against a synthetic export containing _not-found/, 404/, 404.html, docs/architecture/, and packages/:

Wrote out/sitemap.xml with 3 URLs
  https://documentdb.io/
  https://documentdb.io/docs/architecture/
  https://documentdb.io/packages/

Both not-found forms are excluded and /packages/ still survives, which is the regression #123 was guarding.

Update: filtering on noindex instead of on the name

Revised after review. The name list was growing one framework route at a time — #123 added 404, this branch added _not-found, and the versioned-docs work in progress adds a third hand-written skip for the archived version directories. Each is the same rule rediscovered: do not advertise a page that tells crawlers not to index it.

collectPages() now reads each index.html and skips the ones carrying a noindex robots meta. The name list keeps only the directories holding no pages at all — _next, deb, rpm, images — where it is a traversal concern rather than an indexing decision. The tag is matched in either attribute order and tolerates noindex,nofollow, since Next.js and hand-written metadata do not agree on either form. The final log line now reports how many pages were skipped, so a filter that starts matching too much shows up in the build output instead of silently shrinking the sitemap.

Re-verified against a synthetic export — indexable root, /docs, /docs/versions, /packages, /samples, three noindex pages written three different ways, and an excluded _next:

Wrote out/sitemap.xml with 5 URLs (skipped 3 noindex pages) and advertised it in out/robots.txt.
  https://documentdb.io/            https://documentdb.io/docs/
  https://documentdb.io/docs/versions/   https://documentdb.io/packages/
  https://documentdb.io/samples/

out/404.html is left alone, being a file rather than a directory with an index.html. Note this also covers the archived-version case, so the docs/versions skip on the versioned-docs branch can go when that lands.

2. The 404 page emitted two conflicting robots tags

$ curl -s https://documentdb.io/this-page-does-not-exist/ | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="noindex"/>
<meta name="robots" content="index, follow"/>

app/not-found.tsx exported no metadata, so it inherited the root layout's — and metadataService.ts:58-61 hardcodes index: true, follow: true. Next.js separately injects noindex for the not-found route, hence both tags. The page also carried the homepage title byte for byte (DocumentDB - Open Source Document Database).

Crawlers generally honour the most restrictive directive, so the practical impact was low; this is mostly about not shipping a self-contradicting page with a misleading title.

3. /docs/architecture shipped with no heading element

$ curl -s https://documentdb.io/docs/architecture/ | grep -c '<h1'
0

Zero h1-h6, not just no h1. Not a hydration artifact — output: "export", and the placeholder prose is present in the served HTML.

The cause is a contract that the coming-soon layout does not meet. The sidebar section label is deliberately a <p>, with the comment "Not a heading: the article's h1 comes from the markdown content" — but architecture/index.md is frontmatter plus one placeholder paragraph, no markdown heading. The h1 now comes from the frontmatter title (Architecture under the hood), guarded with a check so a future coming-soon page that does start with # will not end up with two.

The card on /docs was also unmarked — class-for-class identical to the six finished sections. Marked it via a title suffix, the same mechanism the Kubernetes Operator card already uses for (Preview).

Left alone deliberately: the issue also floats excluding coming-soon pages from the sitemap. With a real h1 and a useful description the page is reasonable to index, so I would rather not hide it — happy to add it if maintainers disagree.

Not addressed here

The issue mentions in passing that the navbar logo renders loading="lazy" above the fold. Since the image is unoptimized, adding priority only buys a preload hint — different concern, so I left it for a separate change.

Validation

npm is blocked on this machine, so lint and the Next.js build are left to CI. The sitemap script is dependency-free and was executed directly, as shown above.

Review notes

Two things a reviewer should weigh, neither of which changed the diff:

The (Coming soon) marker and the page it describes live in different repos. The card title is in this repo's articles/content.yml, but articles/architecture/index.md and its layout: coming-soon frontmatter come from documentdb/docs. If the architecture content is filled in upstream, nothing here notices and the suffix goes stale. Deriving the marker from the article's frontmatter would be self-maintaining, but getArticleContent() only reads content.yml, so that is a larger change than this fix warrants. Flagging it as a known trade-off.

The 404 page may still emit two robots tags. Next.js injects its own noindex for the not-found route, and whether an explicit metadata export replaces that tag or sits alongside it was not verified locally (npm is blocked on this machine, so no local build). Either way the substantive defect is fixed: the page previously emitted noindex and index, follow — a contradiction — and now emits at worst noindex plus noindex, follow, which agree. Worth a quick check of the deployed HTML after merge if a single tag is wanted.