feat(web): add the web stacklet with SearXNG search

docs/design/web/plan.md

@@ -222,7 +222,7 @@ confirm neither produces a fabricated entry.
 loads a fixture and asserts a gate verdict, so adding a site profile later
 costs one fixture plus one line.
 
-### Phase 2 — The `web` stacklet, search half (about half a day)
+### Phase 2 — The `web` stacklet, search half — SHIPPED
 
 - `stacklets/web/` with `stacklet.toml`, compose, `config/settings.yml`
   (`use_default_settings`, generated `secret_key`, `formats: [html, json]`),
@@ -266,14 +266,72 @@ Opt-in, off by default.
 - Pin `browserforge`. The spike hit an import-time failure without a pin.
 
 **Verification gate, and this gates the phase.** Must be proven inside a
-`linux/arm64` container, not on macOS. The spike ran on macOS arm64. If Chromium
-or Camoufox does not run natively in the container, this phase does not ship as
-designed, because x86 emulation is what made the SeleniumBase path cost 40s.
-Then: decathlon and geizhals both return real content through the service.
+`linux/arm64` container, **headless**, not on macOS. The spike ran on macOS
+arm64 and headed; every public stealth benchmark is also headed, so headless is
+the untested axis and the one we ship. Then: decathlon and geizhals both return
+real content through the service.
+
+Two ceilings to write into the gate rather than discover in it:
+
+- **`real_chrome` can never be true on arm64.** Patchright's own guidance is to
+  run real Google Chrome via `channel="chrome"`, and Chrome for Testing
+  publishes no `linux-arm64` build. On arm64 Playwright falls back to a
+  Chromium `headless_shell`. The weaker configuration is permanent, not a
+  setup mistake.
+- **Tier 3 is a treadmill, not a milestone.** Cloudflare turned on default
+  AI-crawler blocking for free plans on 2026-09-15, with Web Bot Auth
+  (Ed25519-signed requests, a published JWKS, an application process) as the
+  sanctioned alternative. A self-hosted family stack cannot join that
+  programme, so the web is splitting into "identify yourself cryptographically"
+  and "be indistinguishable from a browser", and famstack is structurally on
+  the second path. Keep tier 3 opt-in, behind the `Transport` seam, degrading
+  to an honest link card. Budget for it breaking.
 
 **Harness improvement.** A `stacktests` case that asserts the gate escalates
 exactly once and never loops between tier 2 and tier 3.
 
+## Landscape check, 2026-09-15
+
+A survey of the agentic-browser and agent-web-access space, assessed against
+this stack's constraints (arm64 only, nothing hosted, AGPLv3-compatible,
+container weight, a local ~30B model). Three things changed a decision; the
+rest confirmed one.
+
+**Structured data is the right long bet, and the competing standard is not.**
+JSON-LD now appears on about 41% of mobile pages and is still growing, which
+is why tier 1 reads a recipe deterministically and never asks a model. By
+contrast **`llms.txt` is a dud**: across 137,000 domains surveyed, 97% of
+`llms.txt` files received zero requests in a month, and most of the fetches
+that did happen were not AI tools. Not worth implementing. `NLWeb` (sites
+answering natural-language queries over their own schema.org data) is the one
+to watch, because it makes the JSON-LD reader more valuable rather than
+obsolete. `WebMCP` is a browser-side JavaScript API in a Chrome origin trial,
+so it does nothing for a server-side fetcher.
+
+**Nothing beats trafilatura inside these constraints.** Everything that
+measurably wins on extraction quality is a 0.6B transformer needing 1.5 GB of
+weights, or x86-only, or non-commercially licensed. Worth knowing that every
+benchmark in this space is published by someone shipping a competitor, and the
+same library scores 0.924 and 0.6402 depending on who counts. One adjacent
+finding: `html2text` (used by `stack.email_message`, not by this module) is the
+weakest dependency we have, and `html-to-markdown` replaces it at MIT, zero
+Python dependencies, native arm64 wheels, ~7 MB.
+
+**The capability bar for browser agents is far lower than the marketing.**
+ClawBench, 153 everyday tasks across 144 live sites: the best score ever
+recorded is 33.3%, and the same models score 65 to 75% on traditional web
+benchmarks. Princeton's cost-instrumented leaderboard puts real-web multi-step
+success at 40 to 42%, at hundreds of dollars per benchmark run. This is the
+evidence behind phase 3 being search snippets plus one model call rather than
+multi-step tool use: nobody has a reliable web agent, least of all on a local
+model.
+
+**Licence traps, in the PriceBuddy category.** `Notte` is SSPL. `DrissionPage`
+permits non-commercial use only, in Chinese, while GitHub reports it as
+`NOASSERTION` so an automated check will not catch it. `rebrowser-patches` has
+no licence file at all. `SurfSense` is Apache-2.0 except the directory
+containing its SearXNG connector, which is BSL 1.1.
+
 ## What we are explicitly NOT building
 
 - **lightpanda.** Solved the challenges on every blocked page and still returned
@@ -298,9 +356,20 @@ exactly once and never loops between tier 2 and tier 3.
 
 ## Open decisions
 
-1. **Image size budget for `stack-web-fetch`.** Roughly 250 MB estimated, not
-   measured. If it lands materially higher, consider Camoufox directly instead
-   of the full Scrapling browser set.
+1. ~~**Image size budget for `stack-web-fetch`.**~~ **Resolved, and the
+   fallback was backwards.** Measured: the official `pyd4vinci/scrapling`
+   `linux/arm64` image is **644 MB compressed**, of which 441 MB is
+   `playwright install chromium` and 138 MB is `uv sync --all-extras`.
+   Installing only `[fetchers]` and `playwright install --only-shell chromium`
+   puts the floor around **400 MB**. Chromium's own apt dependencies rule out
+   250 MB, so the budget moves rather than the design.
+
+   Camoufox is no longer the escape hatch: **Scrapling dropped it entirely at
+   v0.3.13**, and `StealthyFetcher` is now patchright over Playwright Chromium
+   with a built-in Turnstile solver. Camoufox's `lin.arm64` asset is **623 MB
+   zipped on its own**, so "use Camoufox directly" is now a step backwards.
+   The arm64 story is patchright and Playwright shipping native aarch64
+   wheels, not Camoufox's builds.
 2. **Does tier 3 stay synchronous?** At 3.4s to 19.8s it fits in a chat round
    trip behind the existing 👀 ack. If real-world pages cluster at the slow end,
    it becomes a background job and the reply becomes "fetching, will file it".

lib/stack/web/__init__.py

@@ -14,7 +14,7 @@
 """
 
 from stack.web.content import SourceContent
-from stack.web.fetch import FetchOutcome, fetch_url
+from stack.web.fetch import FetchOutcome, fetch_url, urllib_transport
 from stack.web.profiles import canonicalize, profile_for
 from stack.web.quality import Page, Verdict, assess
 
@@ -27,4 +27,5 @@
     "canonicalize",
     "fetch_url",
     "profile_for",
+    "urllib_transport",
 ]

lib/stack/web/fetch.py

@@ -277,6 +277,55 @@ def _from_url_only(url: str, title: str, profile: Profile) -> FetchOutcome:
     )
 
 
+# ── Transports ────────────────────────────────────────────────────────
+#
+# Two, because the ladder has two callers with incompatible
+# environments. The bot runs inside a container that already has
+# aiohttp and an open session. The host CLI runs on a Mac with no
+# virtualenv and possibly nothing up at all, so it gets the standard
+# library. Neither is imported at module level.
+
+def urllib_transport(*, timeout: int = 30) -> Transport:
+    """A transport over the standard library, for the host CLI.
+
+    `stack web fetch` has to work with no containers running and no
+    third-party packages installed, which rules out aiohttp. urllib is
+    synchronous, so the request goes to a worker thread and the ladder
+    stays async for both callers.
+
+    An HTTP error is returned rather than raised. A 403 carrying a
+    Cloudflare challenge is not a transport failure — it is a page, and
+    it is exactly the page the gate needs to look at.
+    """
+    def _blocking(url: str, headers: dict) -> Response | None:
+        import urllib.error
+        import urllib.request
+
+        request = urllib.request.Request(url, headers=headers)
+        try:
+            with urllib.request.urlopen(request, timeout=timeout) as resp:
+                return Response(
+                    url=resp.url,
+                    status=resp.status,
+                    html=resp.read().decode("utf-8", errors="replace"),
+                    content_type=resp.headers.get_content_type(),
+                )
+        except urllib.error.HTTPError as err:
+            return Response(
+                url=err.url,
+                status=err.code,
+                html=err.read().decode("utf-8", errors="replace"),
+                content_type=err.headers.get_content_type() if err.headers else "text/html",
+            )
+
+    async def _fetch(url: str, headers: dict) -> Response | None:
+        import asyncio
+
+        return await asyncio.to_thread(_blocking, url, headers)
+
+    return _fetch
+
+
 # ── aiohttp transport ─────────────────────────────────────────────────
 
 def aiohttp_transport(session, *, timeout: int = 30) -> Transport:

stacklets/web/caddy.snippet

@@ -0,0 +1,12 @@
+# stacklets/web/caddy.snippet
+#
+# The family-facing search page. Caddy runs on the stack network, so
+# the backend is the container name and the in-container port (8080),
+# not the published host port.
+#
+# No auth, matching the other family-facing stacklets. The LAN is the
+# perimeter.
+
+search.{$FAMSTACK_DOMAIN} {
+    reverse_proxy stack-web-search:8080
+}

stacklets/web/cli/fetch.py

@@ -0,0 +1,88 @@
+"""stack web fetch <url> — read a web page the way the archivist does.
+
+Host-native. Same code path the bot takes for a pasted link, with
+nothing running: canonicalize the URL, try the page's own structured
+data, fall back to HTTP plus extraction, and put the result through the
+quality gate. No container, no model, no network hop to a service.
+
+That is the point of the command. When a link files badly, the question
+is always "what did we actually get back", and answering it should not
+require the stack to be up or a bot to be restarted.
+
+    stack web fetch https://www.essen-und-trinken.de/rezepte/48816-...
+      Griechischer Salat Rezept
+      tier 1 (default) — the page published its own structured data
+
+      **Servings:** 4 · **Total time:** 35 min
+      ## Ingredients
+      - 500 g rote und gelbe Paprika
+      ...
+
+A page we cannot read says so, and says which obstacle it was:
+
+    stack web fetch https://www.decathlon.de/
+      https://www.decathlon.de/
+      challenge — bot protection served 'just a moment...' instead of the page
+
+A refusal is a successful read of a blocked page, not a command
+failure, so the exit code stays 0. The branch point for a script is
+`--json`, whose `verdict` field carries which of the five refusals it
+was — more than an exit code could say — alongside the tier, the
+profile, the title and the body.
+"""
+
+HELP = "Read a web page and print it as Markdown"
+
+import argparse
+import asyncio
+import json
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "lib"))
+
+from stack.web import fetch_url  # noqa: E402
+from stack.web.fetch import urllib_transport  # noqa: E402
+
+
+def run(args, stacklet, config):
+    parser = argparse.ArgumentParser(prog="stack web fetch", add_help=False)
+    parser.add_argument("url", nargs="?")
+    parser.add_argument("--json", action="store_true")
+    parser.add_argument("--timeout", type=int, default=30)
+    parser.add_argument("-h", "--help", action="store_true")
+    opts = parser.parse_args(args)
+
+    if opts.help or not opts.url:
+        print(__doc__.strip())
+        return {"ok": True}
+
+    outcome = asyncio.run(fetch_url(
+        opts.url, transport=urllib_transport(timeout=opts.timeout),
+    ))
+
+    if opts.json:
+        print(json.dumps({
+            "url": outcome.url,
+            "verdict": outcome.verdict.name,
+            "detail": outcome.verdict.detail,
+            "tier": outcome.tier,
+            "profile": outcome.profile,
+            "title": outcome.content.title_hint if outcome.content else None,
+            "text": outcome.content.text if outcome.content else None,
+        }, indent=2, ensure_ascii=False))
+        return {"ok": outcome.ok}
+
+    # The header is the same two lines either way -- what we ended up
+    # reading, and how we got there. A refusal is not an error report,
+    # it is the same shape with no body under it.
+    print()
+    print(f"  {outcome.content.title_hint if outcome.content else outcome.url}")
+    if outcome.ok:
+        print(f"  tier {outcome.tier} ({outcome.profile}) — {outcome.verdict.detail}")
+        print()
+        print(outcome.content.text)
+    else:
+        print(f"  {outcome.verdict.name} — {outcome.verdict.detail}")
+    print()
+    return {"ok": outcome.ok}

stacklets/web/cli/search.py

@@ -0,0 +1,114 @@
+"""stack web search <query> — search the internet from the terminal.
+
+Queries the instance's own SearXNG, which forwards to upstream engines
+and merges the results. Nothing about the family reaches Google as a
+logged-in profile; the query itself still reaches an upstream engine,
+and that is the honest description rather than a claim of anonymity.
+
+    stack web search "immich vs photoprism"
+      1. Immich vs PhotoPrism: which self-hosted photo manager
+         https://example.com/immich-vs-photoprism
+         Both index a library and serve it over the LAN, but they ...
+
+    stack web search "immich vs photoprism" --json    # for an agent
+    stack web search "mac mini idle watts" --count 5
+
+Unlike `stack web fetch`, this needs the stacklet up: search is a
+service, not a library. When it is down the command says so and points
+at `stack up web` rather than printing an empty result list, because
+"no results" and "nothing is running" are different problems and a
+family that cannot tell them apart will retype the query.
+
+The JSON API this reads is off in SearXNG's shipped defaults
+(`search.formats` is `[html]`). The stacklet's settings overlay turns
+it on; if that ever regresses the web UI keeps working perfectly and
+only this command breaks, which is why there is a test for it.
+"""
+
+HELP = "Search the internet through your own SearXNG"
+
+import argparse
+import json
+import urllib.error
+import urllib.parse
+import urllib.request
+
+SEARCH_URL = "http://localhost:42080/search"
+DEFAULT_COUNT = 8
+
+
+def search(query: str, *, count: int = DEFAULT_COUNT, timeout: int = 20) -> list[dict]:
+    """Results as a list of `{title, url, content, engine}`. Raises OSError
+    when the service cannot be reached."""
+    params = urllib.parse.urlencode({"q": query, "format": "json"})
+    request = urllib.request.Request(
+        f"{SEARCH_URL}?{params}",
+        headers={"Accept": "application/json"},
+    )
+    with urllib.request.urlopen(request, timeout=timeout) as response:
+        payload = json.loads(response.read().decode("utf-8", errors="replace"))
+
+    results = []
+    for item in payload.get("results", [])[:count]:
+        results.append({
+            "title": (item.get("title") or "").strip(),
+            "url": (item.get("url") or "").strip(),
+            "content": (item.get("content") or "").strip(),
+            "engine": item.get("engine") or "",
+        })
+    return results
+
+
+def run(args, stacklet, config):
+    parser = argparse.ArgumentParser(prog="stack web search", add_help=False)
+    parser.add_argument("query", nargs="*")
+    parser.add_argument("--json", action="store_true")
+    parser.add_argument("--count", type=int, default=DEFAULT_COUNT)
+    parser.add_argument("-h", "--help", action="store_true")
+    opts = parser.parse_args(args)
+
+    query = " ".join(opts.query).strip()
+    if opts.help or not query:
+        print(__doc__.strip())
+        return {"ok": True}
+
+    try:
+        results = search(query, count=opts.count)
+    except urllib.error.HTTPError as err:
+        # 403 here is the shipped-defaults failure, and it is worth
+        # naming: the web UI works, so nothing looks broken until a
+        # programmatic query is tried.
+        if err.code == 403:
+            return {"error": (
+                "SearXNG refused the JSON API (403). Its `search.formats` "
+                "setting is back to html-only — check "
+                "stacklets/web/config/settings.yml is mounted."
+            )}
+        return {"error": f"search failed: HTTP {err.code}"}
+    except OSError:
+        return {"error": "Search needs the web stacklet running. Run `stack up web`."}
+
+    if opts.json:
+        print(json.dumps(results, indent=2, ensure_ascii=False))
+        return {"ok": True}
+
+    if not results:
+        print(f"\n  No results for \"{query}\".\n")
+        return {"ok": True}
+
+    print()
+    for n, item in enumerate(results, 1):
+        print(f"  {n}. {item['title']}")
+        print(f"     {item['url']}")
+        if item["content"]:
+            print(f"     {_wrap(item['content'])}")
+        print()
+    return {"ok": True}
+
+
+def _wrap(text: str, width: int = 88) -> str:
+    """One-line snippet, truncated rather than reflowed — the list reads
+    as a scannable column, and a three-line snippet per hit buries the
+    next result."""
+    flat = " ".join(text.split())
+    return flat if len(flat) <= width else flat[: width - 1] + "…"