[IMPROVEMENT] SEO cleanup: sitemap includes /_not-found/, 404 page emits conflicting robots tags, architecture placeholder unlabelled

[GuanzhouSong]

Three small SEO / content-presentation problems found while validating the live site.

Revised. Corrected the line range in §1 (21-27, not 20-27) and the card count in §3.

1. Sitemap advertises the Next.js not-found page

Root cause

scripts/generate-sitemap.mjs emits a URL for any directory containing an index.html (collectPages(), :43-67), filtering only top-level names in an explicit deny-list. With trailingSlash: true the export emits out/_not-found/index.html, and _not-found is not in the list — scripts/generate-sitemap.mjs:21-27:

const excludedTopLevelDirectories = new Set([
  '_next',
  'deb',
  'rpm',
  'images',
  '404',
]);

#123 (9505e9c) fixed exactly this class for 404 — its commit message reads "/404/ was listed. With trailingSlash the export emits out/404/index.html alongside out/404.html; the directory form needs an explicit exclusion." — but only added '404'. _not-found was missed.

Note the two are not symmetric: 404 exists in both forms (/404.html and /404/ both 200), whereas _not-found exists only as the directory form (/_not-found.html → 404).

Reproduce

curl -s https://documentdb.io/sitemap.xml | grep '_not-found'
# <loc>https://documentdb.io/_not-found/</loc>
curl -s -o /dev/null -w "%{http_code}\n" https://documentdb.io/_not-found/
# 200

Fix: add '_not-found' to the exclusion set.

Why it matters more than it looks

/_not-found/ is listed in the sitemap and serves <meta name="robots" content="noindex">. The sitemap is actively submitting a URL that tells crawlers not to index it — the Search Console "submitted URL marked noindex" warning class. Fixing this section resolves that regardless of §2.

2. The 404 page emits two conflicting robots meta tags

Root cause

app/not-found.tsx exports no metadata, so it inherits the root layout's (app/layout.tsx:9), which calls getMetadata(...) — and that hardcodes indexability at app/services/metadataService.ts:58-61:

  robots: {
    index: true,
    follow: true
  }

Next.js separately injects noindex for the not-found route, so both tags are emitted.

Reproduce

curl -s https://documentdb.io/this-page-does-not-exist/ | grep -o '<meta name="robots"[^>]*>'

<meta name="robots" content="noindex"/>
<meta name="robots" content="index, follow"/>

Crawlers generally honour the most restrictive directive, so practical impact is low. But the page also inherits the generic homepage title, byte-identical to /:

<title>DocumentDB - Open Source Document Database</title>

Fix: give not-found.tsx its own metadata export with a distinct title and robots: { index: false }, so only one directive is emitted.

3. /docs/architecture is an unlabelled placeholder

articles/architecture/index.md:4 sets layout: coming-soon; line 7 begins "We're building something amazing! The technical architecture documentation…".

Two problems with how it surfaces:

No "coming soon" marker on the docs index. /docs shows 7 cards, one of which is Architecture. Its markup is class-for-class identical to the others — no badge. The Kubernetes Operator card carries a (Preview) suffix (source: articles/content.yml:7), so the precedent for marking non-final sections exists; architecture just doesn't use it. Worth noting the 7 cards aren't all docs sections — /kubernetes-operator/ and /samples/ leave /docs entirely.

The page has no heading element at all. Not just no h1 — zero h1-h6:

curl -s https://documentdb.io/docs/architecture/ | grep -c '<h1'
# 0

This is not a client-rendering artifact: next.config.ts sets output: "export", and the coming-soon paragraph is present in the served HTML, so the content column is fully rendered. No h1 can appear on hydration either — app/components/Markdown.tsx only emits one from a leading # in the markdown, and articles/architecture/index.md has none.

The visible "Architecture" text in the sidebar is deliberately not a heading — app/docs/[section]/[[...slug]]/page.tsx:151-155 renders it as a <p> with an explicit comment: "Not a heading: the article's h1 comes from the markdown content". That contract is simply unmet here. (The sidebar also shows Architecture, derived from the slug, while the page title is Architecture under the hood.)

The page is also in sitemap.xml, so search engines are pointed at a headingless placeholder.

Fix: mark the card as coming soon on the docs index, add an h1 to the coming-soon layout, and optionally exclude coming-soon pages from the sitemap until they have content.

Validation

Spot-checked 6 other pages (/, /docs/, /docs/getting-started/, /docs/reference/, /docs/documentdb-local/, /kubernetes-operator/) — all have an h1. /docs/architecture/ is the outlier.

---

Also noticed, not worth its own issue: the navbar logo (app/components/Navbar.tsx:105-112) renders loading="lazy" on every page despite being above the fold. The source sets no loading prop — it's next/image's default because priority is absent. Since the image is unoptimized, adding priority only buys a preload hint and drops the lazy attribute.