Add a daily IndexNow submission workflow

[David Pine (IEvangelist)]

Summary

Adds a scheduled workflow that tells IndexNow search engines (Bing, Yandex, Naver, Seznam, Yep, Amazon) about aspire.dev URLs, so new docs are discovered quickly, especially right after a release ships.

How it works (.github/workflows/indexnow.yml + src/frontend/scripts/indexnow.ts, also runnable locally with pnpm indexnow)

• Every run reads the live sitemap (about 32k URLs) and compares it with the URL list saved by the previous run. It sends one request (IndexNow takes at most 10,000 URLs in one) with the removed and new URLs first, then the next window of a rotation through the whole sitemap: English pages, then the API reference, then localized pages.
• Daily (06:43 UTC): the window is 1,000 URLs. Its position is a cursor over a stable order, so it survives pages being added or removed.
• Release: merging a release/X.Y pull request into main waits up to 45 minutes for that release's "What's new" page to be live (the docs slot swap), then starts a burst of up to 10,000 URLs per run that the daily runs finish (about 4 runs today). After that the slow rotation resumes. The sitemap has no lastmod, so the script can't tell an updated page from an unchanged one; the burst sends all English pages first, which is where the new and updated docs are.
• Manual: workflow_dispatch with mode (daily or flush), limit, release, dry_run and allow_shrink.
• The state is a single state.json in the indexnow-state Actions artifact (90 days). Losing it only causes a cold start that re-seeds the known list without submitting everything.
• A failed request (a 429 included) fails the job and leaves the progress alone, so the next run tries again. A failed flush still starts its release pass, and a failed cold start still seeds the known list.

What the script assumes about the site (listed at the top of indexnow.ts; everything else is in src/frontend/config/indexnow.json)

• /reference/api/ is the generated API reference, and the locale folders in config/locales.ts hold the translations. Both are sent after the English pages.
• Release X.Y is merged from a release/X.Y branch and publishes /whats-new/aspire-X-Y/ (aspire-X for X.0).

No repository secret is needed. An IndexNow key is public by design: the key file has to be downloadable from the site root so engines can verify ownership. The key is committed (src/frontend/config/indexnow.json and src/frontend/public/<key>.txt). The workflow only uses the automatic token with contents and actions read access.

Safety rails: a key-file preflight (a cold start skips softly until the key file has deployed), refusal of a sitemap that shrank by more than 20% (allow_shrink overrides), state read from an artifact is checked (and artifacts from fork runs are ignored), --limit is a hard cap, and 5xx and network errors retry with back-off.

Rollout

1. Merge, and let the deploy publish https://aspire.dev/c5f794ab28dc0cb92f48bd970500daa3.txt (until then the first run skips).
2. Run the workflow manually with dry_run to see the plan.
3. The first real run seeds the state and submits the first 1,000-URL window.
4. The next release merge triggers the burst, or dispatch mode: flush with release: X.Y. Verify in Bing Webmaster Tools under IndexNow.

Notes

• IndexNow doesn't publish its rate limits, so the budgets (release 10,000 and rotation 1,000) live in config/indexnow.json. A test keeps them within one request.
• A page that is new while a pass is running can be sent twice: when it appears and when the rotation reaches it.
• The release-merge path (the closed pull request event) first runs on the next release and can't be exercised before merging. The trigger matches a merged pull request from release/X.Y in this repository, which is what the 13.4, 13.5 and 13.6 content pull requests are. The pull request that merges the release branch back is not a trigger; a manual flush covers repairs.
• The job-level concurrency uses queue: max, which GitHub documents. actionlint 1.7.12 doesn't know it yet, and the repository doesn't run actionlint in CI.
• Follow-up: accurate lastmod values in the sitemap would let search engines (Bing and Google use them to schedule crawls) and this script tell updated pages apart.

Third-party links and affiliations

• https://www.indexnow.org/documentation (referenced in a code comment) and the https://api.indexnow.org/indexnow endpoint (in config/indexnow.json) are the IndexNow protocol's documentation and submission endpoint. IndexNow is an open protocol launched by Microsoft Bing and Yandex, and aspire.dev is also a Microsoft site. This change adds no sponsorship or other affiliation.

Validation

From src/frontend:

• pnpm exec vitest run --config vitest.config.ts tests/unit/indexnow.vitest.test.ts: passed, 83 tests.
• pnpm test:unit: passed, 81 files and 1,121 tests (1 skipped).
• pnpm lint: passed.
• actionlint 1.7.12 on the workflow, with the queue: max line removed because actionlint doesn't know it: passed.
• Mutation checks: nine deliberate breakages of the behavior that matters (a flush sending its new URLs ahead of its window, a failed flush forgetting them, no shrink guard, a failed flush or cold start not saving state, retrying client errors, no one-request clamp, state entries not checked against the host, a fork's release pull request accepted, a cold start not skipping while the key is missing) each fail at least one test.
• Real data: the planner against the live sitemap (32,336 URLs) runs a release burst in 4 requests (10,000, 10,000, 10,000 and 2,336 URLs), covers every URL once, and sends all 611 English docs first (the 13.6 What's new page is at position 602 of day 1).
• The workflow's submit step run under bash with a stubbed pnpm for the schedule, pull_request and workflow_dispatch triggers builds the expected arguments.
• The real CLI against the live site as dry runs: a daily run plans 1,000 URLs, --mode flush --release 13.6 plans 10,000, neither writes state, and an invalid --limit exits 1. The 404 warning for the key file is expected until it deploys.

Not run: pnpm build (left to CI), a real IndexNow submission (the key file isn't deployed yet) and the pull-request-triggered path (first exercised by the next release merge).

[Copilot]

🟡 Changes recommended

Priority submissions can exceed the configured per-run rotation budget.

1 open finding

• Priority cap ignores configured per-run budget · New

What changed in this PR

Adds automated IndexNow submissions to improve discovery of new, updated, and removed aspire.dev pages.

Changes:

• Adds scheduled, release-triggered, and manual submission modes.
• Implements sitemap planning, persisted state, retries, release prioritization, and safety checks.
• Adds configuration, public verification key, package command, and extensive unit coverage.

File	Description
.github/​workflows/​indexnow.yml	Runs submissions and persists state.
src/​frontend/​scripts/​indexnow.ts	Implements IndexNow orchestration.
src/​frontend/​tests/​unit/​indexnow.vitest.test.ts	Tests planning, state, triggers, and submissions.
src/​frontend/​config/​indexnow.json	Configures endpoint, key, and budgets.
src/​frontend/​public/​c5f794ab28dc0cb92f48bd970500daa3.txt	Publishes the verification key.
src/​frontend/​package.json	Adds the indexnow command.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[aspire-repo-bot]

Frontend HTML artifact ready

The latest frontend build uploaded the frontend-dist artifact for PR #1882. Use the VS Code button below to open this PR with GitHub Artifacts Explorer and browse the built HTML locally.

This comment updates automatically when a new frontend build artifact is uploaded.