feat: add agent-friendly 404 page and llms.txt

[RobbieTheWagner]

Replaces #3496, which was stuck in a broken stack state after #3494 was declined. This branch is exactly landing-trust-pages + one commit; merge #3499 first, then this.

• Custom 404.astro in the site's design with recovery links (docs, pricing, blog, llms.txt, sitemap, contact). The Vercel build output already routes unmatched paths to it with a real HTTP 404 status.
• llms.txt with an explicit 'When to use Shepherd' section naming best-fit jobs (onboarding walkthroughs, feature announcements, complex-form guidance, in-app training), install/API guidance, what Shepherd is not for, and links to docs and trust pages.

Test plan: e2e tests assert nonexistent paths return HTTP 404 with recovery links and that /llms.txt serves the when-to-use guidance; build-output tests assert 404.html and llms.txt are emitted — 31 tests passing on this branch.

After everything lands, live verification:

curl -s -o /dev/null -w "%{http_code}" https://www.shepherdjs.dev/some-path-that-does-not-exist
curl -sI -H "Accept: text/markdown" https://www.shepherdjs.dev/ | grep -iE "content-type|vary"

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
shepherd-docs	Ready	Preview	Aug 25, 2026 3:14am
shepherd-landing	Ready	Preview	Aug 25, 2026 3:14am

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3123871e-f5ba-462c-8496-544af1e790dc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[qltysh]

Coverage Impact

This PR will not change total coverage.

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.