Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion docs/design/web/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,37 @@ Other measurements:
Camoufox publishes `lin.arm64`; Playwright's Chromium builds arm64. This is the
single biggest reason Scrapling wins.

## Re-validation, 2026-09-15

Re-probed before building Phase 1, same two UAs, anonymous, German IP.
Three findings held, one moved.

| Claim | Still true? |
|---|---|
| decathlon.de blocks a plain fetch | yes — 403, Cloudflare `Just a moment...` |
| Recipe JSON-LD on essen-und-trinken / einfachkochen | yes — full `Recipe`, yield, totalTime, 11 and 14 ingredients, steps, nutrition |
| geizhals.de returns 403 | **no** — the *homepage* serves 200 and its real listing. The measured 403 was a product/listing URL; the row overstated it as the whole domain |
| `old.reddit.com` + Chrome UA returns the post | **no — reversed** |

**Reddit now walls anonymous readers.** `old.reddit.com` answers a 302 to
`/login/?reason=lor2`, and `www.reddit.com` serves an 8 KB JavaScript
shell. The `.json` endpoint is 403 with a 190 KB HTML block page.

This is worth more than a corrected row, because of the *shape* of the
failure. The login redirect ends on HTTP 200 with a 320 KB body and a
friendly `<title>Welcome to Reddit</title>` — which is exactly what a
"did the extractor return a string?" success check reads as an article.
The drift did not break the gate's design, it validated it: `login` was
already in the verdict set, and the landing URL is the only honest
signal on that page. The gate reads the URL a fetch *ended* on for
precisely this reason.

The reddit profile keeps its `old.reddit` rewrite anyway. It no longer
recovers the post, but it moves the reported reason from `empty` ("the
page was blank") to `login` ("reddit wants you signed in"), which is the
one a person can act on — and it starts working again unchanged if
reddit relaxes.

## Decision: one stacklet, but not for everything

The tempting version is a `web` stacklet that owns all web operations including
Expand Down Expand Up @@ -158,7 +189,7 @@ construction: a family that never pastes a shop link never downloads Chromium.

## Phases

### Phase 1 — Framework module and the gate (about 1 day)
### Phase 1 — Framework module and the gate — SHIPPED

The whole fix for both reported bugs, with no new container.

Expand Down
30 changes: 30 additions & 0 deletions lib/stack/web/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
"""How famstack reads the web.

A pasted link becomes a good vault entry or an honest link card, and
never a cookie policy. The ladder that decides which is in `fetch`; the
rule that nothing unjudged reaches the vault is in `quality`.

content SourceContent, the shape every capture source produces
quality the gate: ok / challenge / login / consent / paywall / empty
profiles per-domain rules — canonical URL, extraction tuning
structured JSON-LD Recipe and Article, read without an LLM
fetch the ladder that runs them in order

See `docs/design/web/plan.md` for the measurements behind each tier.
"""

from stack.web.content import SourceContent
from stack.web.fetch import FetchOutcome, fetch_url
from stack.web.profiles import canonicalize, profile_for
from stack.web.quality import Page, Verdict, assess

__all__ = [
"FetchOutcome",
"Page",
"SourceContent",
"Verdict",
"assess",
"canonicalize",
"fetch_url",
"profile_for",
]
37 changes: 37 additions & 0 deletions lib/stack/web/content.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
"""`SourceContent` — the classifier's input, normalized across sources.

Promoted here from `stacklets/docs/bot/extractors.py` because it stopped
being one stacklet's type the moment a second consumer needed it: the
host CLI (`stack web fetch`) builds one without the archivist running at
all. Same move as `stack.email_message`, same reason -- the framework
owns the shared shape, each stacklet owns its own mapping into it.

The archivist's classifier does not care whether text arrived from a
photographed receipt that Paperless OCR'd, a pasted URL that trafilatura
rendered, or a wall of text somebody typed. This is what all of those
agree to produce.
"""

from __future__ import annotations

from dataclasses import dataclass


@dataclass
class SourceContent:
"""The classifier's input, normalized across source types.

`text` is the body the classifier reads — Markdown when the
extractor can produce it, plain text otherwise. `title_hint` is
whatever the source advertised as a title (HTML `<title>`, first
body line, filename); the classifier may overwrite it with
something more useful. `source_uri` is the canonical pointer
back to the origin (`https://...`, `paperless://42`,
`matrix:<event-id>` — caller decides the scheme), captured into
the mirror's frontmatter for round-tripping. None means the
capture has no upstream pointer (a pure pasted note).
"""
text: str
mime: str = "text/plain"
title_hint: str | None = None
source_uri: str | None = None
Loading
Loading