Publish docs crawler entry points and social previews

[GeorgianDusk]

Summary

Addresses audit finding AR-09 and the documentation social-preview gap.

• publishes /robots.txt with an allow policy and the real sitemap-index URL
• publishes the conventional /sitemap.xml compatibility URL as an exact build-time copy of Astro's generated sitemap index
• adds default Open Graph and Twitter preview images with alt text
• adds repeatable machine-file and metadata tests
• makes CI run the complete verification gate, including the new tests

Root cause and impact

The documentation sitemap existed at /sitemap-index.xml, but the standard /robots.txt and /sitemap.xml discovery paths returned 404. Agents and crawlers that probe only conventional entry points could therefore miss an otherwise complete sitemap.

The compatibility URL is generated from Astro's actual sitemap index after every build. It cannot silently drift if Astro later splits the sitemap into additional child files or changes their numbering.

Validation

• clean locked install with lifecycle scripts disabled
• all 362 installed registry packages have verified signatures; 85 have verified attestations
• the approved esbuild dependency was explicitly rebuilt
• npm run verify: 3 tests passed; 64 pages built
• Starlight internal-link validation and Pagefind search indexing passed
• local rendered QA passed on desktop, mobile, and JavaScript-disabled documentation pages
• local /robots.txt, /sitemap.xml, and /sitemap-index.xml returned the expected outputs
• the existing Astro /404 route-conflict warning is unchanged

Security and remaining risk

This introduces no secrets, write permissions, deployment actions, or publication behavior. The workflow retains read-only repository permissions and does not persist checkout credentials.

The repository's existing dependency audit findings are not introduced by this PR and should be handled separately.

Deployment and rollback

No content migration is required. Deploy through the normal docs pipeline. Roll back by reverting this PR.

Post-deployment checks

• verify all three URLs return 200: /robots.txt, /sitemap.xml, and /sitemap-index.xml
• confirm /sitemap.xml exactly matches the generated sitemap index
• repeat verified-provider crawler checks against the deployed sitemap index
• inspect a shared docs URL in social-preview debuggers

Deliberately not included

This PR does not choose product statuses, replace legacy client-side redirect shells, invent page review dates, or synthesize sitemap lastmod values. Those require content ownership and/or routing changes.

[Copilot]

Pull request overview

Publishes standard crawler discovery entry points and adds default social-preview metadata so documentation pages are reliably discoverable by crawlers and render consistent link previews when shared.

Changes:

• Add public/robots.txt advertising the sitemap index and permitting crawling.
• Add public/sitemap.xml as a conventional sitemap entry point.
• Add default Open Graph / Twitter preview images via Starlight head config and introduce Node test coverage plus a verify script.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/machine-files.test.mjs	Adds Node tests to validate robots.txt, sitemap.xml, and presence of social preview meta tags.
public/sitemap.xml	Introduces a conventional /sitemap.xml sitemap index entry point.
public/robots.txt	Introduces a permissive robots.txt that advertises the sitemap index URL.
package.json	Adds test and verify scripts to run Node tests and build.
astro.config.js	Adds global Open Graph and Twitter image meta tags for social previews.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[HDauven]

LGTM