Skip to content

feat(web): add the web stacklet with SearXNG search - #97

Merged
arthware-dev merged 2 commits into
feat/web-capture-gatefrom
feat/web-stacklet-search
Sep 16, 2026
Merged

arthware-dev merged 2 commits into
feat/web-capture-gatefrom
feat/web-stacklet-search

Conversation

@arthware-dev

Copy link
Copy Markdown
Contributor

Stacked on #96 - review that one first; this PR's diff is against it.

Adds the optional web stacklet. SearXNG runs on the family's own hardware, so a search reaches upstream engines as a query from the house rather than from a logged-in browser. Not anonymity, and the docs say so.

stack web search needs the stacklet up. stack web fetch does not, and that split is the point: link capture never needed a container, so a family that never searches downloads none of it.

Phase 2 of docs/design/web/plan.md.

@arthware-dev

Copy link
Copy Markdown
Contributor Author

Verified on the live rig.

stack web fetch with the stacklet down - the Phase 2 gate, ingredients from a recipe with no container running:

  Griechischer Salat Rezept
  tier 1 (default) — the page published its own structured data

  **Servings:** 4 · **Total time:** 35 min
  ## Ingredients
  - 500 g rote und gelbe Paprikaschote
  - 300 g Tomate
  ...

Refusals name the obstacle, including the reddit drift found in #96:

$ stack web fetch https://www.reddit.com/r/selfhosted/
  https://old.reddit.com/login/?reason=lor2&dest=...
  login — the site redirected to a sign-in page (old.reddit.com/login/)
        — reddit no longer serves posts to readers who are not signed in

stack web search - 1.4s end to end, titles and URLs non-empty:

$ stack web search "immich vs photoprism"
  1. Immich vs photoprism : r/selfhosted - Reddit
     https://www.reddit.com/r/selfhosted/comments/1doatsa/immich_vs_photoprism/
  2. Jellyfin vs Immich vs Photoprism vs Nextcloud Photos 2026 ...

Lifecycle - stack up web pulls, starts and passes its health check on linux/arm64; stack down web stops it; no cron entry is added at any point (crontab -l | grep -c famstack is 0 before and after).

Config verified against the image, not from memory. searxng/searxng:latest ships search.formats: [html], so the JSON API is off and every programmatic query 403s. server.secret_key is overridden by ${SEARXNG_SECRET}, which is why no key is templated into the repo. tests/integration/test_web_search_e2e.py guards both, because the web UI keeps working when the overlay stops applying - nothing looks broken until an agent asks a question.

Tests: 81 framework web tests, 4 e2e, all green.

@arthware-dev

Copy link
Copy Markdown
Contributor Author

Follow-up commit after a landscape survey of agent web access. Three things it turned up that bear on this PR:

1. The image needed pinning, and the audit that should have caught it had a hole. SearXNG carried three builds on 2026-09-15 alone, and each one changes which upstream engines work - chasing other people's bot detection is the project's whole job. Now pinned to 2026.9.15-ca4965040, which is the exact digest the verification above ran against.

The repo already has test_compose_pins.py for precisely this, and it passed my :${WEB_SEARCH_VERSION:-latest}, because the literal tag is not the string latest. Fixed, with three regression tests covering the escape, the correct overridable-pin form, and the whole-ref ${SOME_IMAGE} case that must keep deferring.

2. Only two engines actually answer. Measured on the running instance:

results: 29   engines answering: {'google cse': 20, 'brave': 19}

Google's web engine and Bing ship disabled: true upstream; Startpage is inactive behind a proof-of-work captcha. Two engines is one bad week from one, and it degrades as fewer results, not as an error - the family would just find the answers getting worse. The e2e lane now asserts 2+ engines answer.

3. number_of_results is a trap. It reports null while results holds a full page. Nothing here branches on it, and there is now a test to keep it that way.

Also recorded in the plan, affecting Phase 4 rather than this PR: Scrapling dropped Camoufox entirely at v0.3.13, which inverts the plan's open decision - Camoufox's lin.arm64 asset is 623 MB on its own, so "fall back to Camoufox" is now a step backwards. The real measured number is 644 MB compressed for the Scrapling arm64 image, ~400 MB floor with [fetchers] + --only-shell. And real_chrome can never be true on arm64 (Chrome for Testing publishes no linux-arm64), so patchright's weaker config is a permanent ceiling.

Lane: 6 e2e green. Pin tests: 6 green.

@arthware-dev
arthware-dev marked this pull request as draft September 15, 2026 21:09
An optional stacklet runs SearXNG on the family's own hardware, so a
search reaches upstream engines from the instance rather than from a
signed-in browser. This is not anonymity and the documentation says so.

stack web search queries it from the terminal, and a search page is
served on the LAN. stack web fetch reads a single page through the same
ladder the archivist uses.

The stacklet is opt-in. Link capture does not require it.

Refs docs/design/web/plan.md (phase 2).
SearXNG publishes several builds a day and each changes which upstream
engines respond, so an unpinned image made search behaviour vary between
restarts.

Few engines answer on a default install: the Google web engine and Bing
ship disabled upstream, and Startpage is behind a captcha. Losing one
more leaves search returning fewer results rather than an error, which
is invisible to the family. The search lane now fails when fewer than
two engines respond, and nothing reads number_of_results, which the API
does not populate.

The compose audit also treated image: repo/name:${VAR:-latest} as
pinned, because the literal tag is not the word "latest".
@arthware-dev
arthware-dev force-pushed the feat/web-stacklet-search branch from 89d3ea5 to a721c7e Compare September 16, 2026 08:51
@arthware-dev arthware-dev changed the title feat(web): search the internet from your own stack feat(web): add the web stacklet with SearXNG search Sep 16, 2026
@arthware-dev
arthware-dev marked this pull request as ready for review September 16, 2026 09:28
@arthware-dev
arthware-dev merged commit 3b17066 into feat/web-capture-gate Sep 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant