llms.txt overhaul: full-content index, markdown mirrors, crawler policy

[rivassec]

What

Ten improvements to the AI-crawler index surface, plus one independent SEO fix.

llms.txt / llms-full.txt

• llms-full.txt now carries full post content (was expanded metadata only; the 400-char summary truncate was inert since every summary is <=160 chars). 159KB, under the new 384KB soft-warn and existing 512KB hard cap. Header notes formatting is flattened and points at the Markdown mirrors.
• Per-post Markdown mirrors: new plugins/md_mirror.py copies each published article's source to output/{slug}.md (reads generator.articles, so drafts structurally cannot leak; runs after the sitemap generator so mirrors stay out of the sitemap). Both llms files link them.
• Topical grouping by category, per-entry (YYYY-MM) dates, provenance line (Generated: <date> | N posts).
• Expanded Key pages (IAM Blast Radius tool, Categories, Accessibility) and new Tools and code section (secure-iam-lint, iam-safe-defaults, elasticsearch-tools, pwnagotchi plugins - all URLs verified live).
• Meta section: author, security.txt pointer, Atom/RSS feeds, citation preference.
• Header description single-sourced (LLMS_DESCRIPTION in pelicanconf) and refreshed to cover the AI-security, threat-intel/OSINT, and hiring-security threads.
• .well-known/llms.txt alias (byte-identical include shim) for crawlers that probe there first.

robots.txt

Explicit allow groups for GPTBot, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended - each with the same /drafts/ exclusion - and comments naming both llms URLs. Publishing an llms.txt means AI readership is wanted; this makes the policy deliberate instead of implicit.

CI guard hardening (check_wellknown_llms.py)

Now also fails on: any /drafts/ URL in either llms file, a post missing from llms-full (parity with the short index), a missing markdown mirror, or a missing .well-known alias. Soft size warning at 384KB.

Independent fix

• testing-an-iam-analyzer-against-its-own-claims.md: Summary trimmed 212 -> 157 chars (meta descriptions truncate at ~160; check_descriptions flagged it). This changes the live meta description - review the wording.

Verification

• Publish build clean (20 articles + 3 drafts, drafts in no index file)
• check_wellknown_llms.py: 0 issues (including the new guards)
• canonical/noindex: 0 bad; link graph: 0 issues; em-dash: clean
• .well-known/llms.txt byte-identical to /llms.txt; 20 markdown mirrors written