docs(factories): editorial pass on scorer documentation

.agents/references/terminology.md

@@ -340,8 +340,8 @@ Docs match the screen; the fix belongs in the app.
 - **runner** — A factory resource, defined by a `runners/<name>.yaml` file, that defines the compute a run executes on: operating system, architecture, sandbox image, and instance shape. Agents and automations select a runner by name, or inherit the factory's default.
   *Usage note:* Lowercase common noun. Scoped to a factory's definition; distinct from the general [cloud agent runner](/platform/runners/) reference, which covers the same concept for standalone cloud agents outside a factory.
 
-- **Scorer** — An LLM judge, configured per factory, that classifies completed agent conversations against criteria a team defines, scoped to chosen agents and sampled at a set rate. Feeds the **Dashboard** page's Scorer cards, benchmarks, and Self-improvement.
-  *Usage note:* Capitalize, since it names a specific configured entity in the factory dashboard's UI (parallel to **Dashboard**), not a generic industry term.
+- **Scorer** — A configured LLM judge, scoped per factory to chosen agents and sampled at a set rate, that classifies completed runs against criteria you write, such as "did the agent run the tests before opening a PR?" A Scorer assigns a label (a classification with a score), not a freeform numeric grade. Feeds the **Dashboard** page's Scorer cards, benchmarks, and Self-improvement.
+  *Usage note:* Capitalize "Scorer"/"Scorers" when referring to the feature or a configured instance ("create a Scorer," "Scorer cards"); lowercase only for a generic instance count or file listing ("two scorers," alongside "skills" in an example tree). Say "classify," never "grade" — the docs draw this distinction deliberately. The unit a Scorer evaluates is a **run** (a single agent execution), not a "conversation" or "completed work." `measure-and-improve.mdx` is the canonical page for what a Scorer is and how to configure one; other pages link there rather than repeating the definition.
 
 - **AI sovereignty** — Warp Factories' positioning around customer ownership and control of inference, hosting, and data exhaust (agent conversations, evals, memories) for their factory.
 

.agents/skills/check_for_broken_links/test_check_links.py

@@ -37,6 +37,29 @@
 ]
 
 
+# (description, heading text as it appears after '## ', expected slug)
+# A code span's content renders as literal text -- markdown never parses
+# `<name>` inside backticks as an HTML/JSX tag, so these placeholders must
+# stay in the anchor id instead of being stripped like real markup.
+SLUG_CASES = [
+    (
+        "backtick heading with an angle-bracket placeholder keeps the placeholder text",
+        "`scorers/<name>/scorer.md`",
+        "scorersnamescorermd",
+    ),
+    (
+        "a different backtick+placeholder heading keeps its own placeholder text too",
+        "`agents/<name>/agent.md`",
+        "agentsnameagentmd",
+    ),
+    (
+        "plain-text heading is unaffected by the code-span handling",
+        "Configure Scorers",
+        "configure-scorers",
+    ),
+]
+
+
 def main() -> int:
     failures = 0
 
@@ -59,7 +82,13 @@ def main() -> int:
             failures += 0 if ok else 1
             print(f"  [{'PASS' if ok else 'FAIL'}] {description:<58} broken={broken_urls}")
 
-    total = len(CASES) + 1
+        for description, heading, expected_slug in SLUG_CASES:
+            slug = check_links.slugify_heading(heading)
+            ok = slug == expected_slug
+            failures += 0 if ok else 1
+            print(f"  [{'PASS' if ok else 'FAIL'}] {description:<58} slug={slug!r}")
+
+    total = len(CASES) + len(SLUG_CASES) + 1
     print()
     if failures:
         print(f"{failures} of {total} cases regressed.")

src/content/docs/factories/factory-as-code.mdx

@@ -2,13 +2,13 @@
 title: Factory definition syntax
 description: >-
   Look up every file and key in a factory definition: factory.yaml, agents,
-  automations, runners, and skills.
+  automations, runners, scorers, and skills.
 sidebar:
   label: "Definitions as code"
 ---
 import { VARS } from '@data/vars';
 
-Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, and runners, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition.
+Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, and skills, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition.
 
 Definition files are YAML and Markdown. Keys are case-sensitive.
 
@@ -46,6 +46,9 @@ automations/
     automation.md
 runners/
   linux-build.yaml
+scorers/
+  tests-run/
+    scorer.md
 skills/
   repository-conventions/
     SKILL.md
@@ -319,6 +322,66 @@ Optional. The compute size, as `vcpus` and `memoryGb`. Omit it to take the works
 
 The operating system and architecture. `os` is `linux` (the default) or `macos`, and `arch` is `x86_64` (the default on Linux) or `aarch64` (the only option on macOS). Linux runners require `linux.dockerImage`, the container image the sandbox boots, so every Linux runner declares a `platform` section. macOS runners accept an optional `mac.version` (`"14"`, `"15"`, `"26"`, or `"27"`; quote it, and it defaults to `"26"`).
 
+## `scorers/<name>/scorer.md`
+
+Optional. Each file defines a scorer: an LLM judge that classifies a sample of an agent's finished runs against a rubric. The directory segment is only a stable filesystem slug — the required `name` field is the scorer's identity. The YAML frontmatter declares the classification contract, and the Markdown body after the closing `---` fence is the rubric. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers) for how scores are used.
+
+```markdown title="scorers/tests-run/scorer.md"
+---
+name: tests-run
+description: Checks whether implementation runs include test evidence.
+agents:
+  - reviewer
+labels:
+  - value: tests_run
+    description: The transcript contains a test command and its result.
+    score: 1
+  - value: tests_skipped
+    score: 0
+passingScore: 1
+samplingRate: 25
+model: claude-4-5-haiku
+---
+Evaluate whether the agent ran the relevant tests before finishing. Return
+exactly one declared label.
+```
+
+### `name`
+
+Required. The scorer's identity. Renaming it is a content edit, not a directory move.
+
+### `description`
+
+Optional. A short summary of what the scorer checks.
+
+### `agents`
+
+Required. The agents whose runs this scorer evaluates, as a list of one or more agent names. Each name matches an agent defined under [`agents/`](#agentsnameagentmd).
+
+### `output`
+
+Optional. The scorer output form. `classification` is the current supported value.
+
+### `labels`
+
+Required. The classifications the judge may return, each with a `value`, a numeric `score` from 0 through 1, and an optional `description`. At least one label must score at or above `passingScore` and at least one below it.
+
+### `passingScore`
+
+Required. The threshold, from 0 through 1, at or above which a run counts as passing.
+
+### `samplingRate`
+
+Optional. The percentage of eligible runs to score. Defaults to 25.
+
+### `model`
+
+Required. The model that judges the runs.
+
+### `selfImprovement`
+
+Optional. When `true`, failing scores can feed the factory's self-improvement flow, which proposes definition changes as pull requests. Defaults to `false`.
+
 ## Skills
 
 A skill is a directory containing a `SKILL.md`, not a YAML key. Skills under `skills/` are available to every agent in the factory; skills under `agents/<name>/skills/` are available only to that agent. See [factory skills](/factories/factory-skills/) for when to add one, and [Skills](/agents/capabilities/skills/) for the file format.

src/content/docs/factories/factory-dashboard.mdx

@@ -88,9 +88,7 @@ When an agent proposes a change to a Warp-managed definition, its work item on *
 
 ## Score and benchmark
 
-A Scorer is an LLM judge that classifies completed runs against a rubric you define, scoped to the agents you choose and sampled at a rate you set. Each Scorer has a **Self-improvement** toggle: when it's on, the factory periodically triages the runs that Scorer marks as failing and files fixes for recurring issues. The **Self-improvement** page tracks the pull requests those fixes open, linking each one to the run that produced it.
-
-**Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks with success criteria. Benchmark results are only as reliable as the Scorers behind them, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/).
+**Scorers** is where you create Scorers and read their results. **Self-improvement** lists the pull requests the self-improvement flow opens after analyzing runs your Scorers mark as failing, and **Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers), [Configure and review Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement), and [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for what each one is and how to set it up.
 
 ## Change factory settings
 

src/content/docs/factories/how-factories-work.mdx

@@ -65,7 +65,7 @@ The first two are workflow policy, written into the foreman's instructions; edit
 
 ## How the factory improves itself
 
-Your factory is self-improving, and you define what "better" means. [Scorers](/factories/measure-and-improve/) grade completed runs against criteria you write, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) groups the failures they flag into follow-up runs that propose fixes — to the application code or to the factory's own definition. Every proposal arrives as a change for your review; nothing is adopted on its own.
+Your factory is self-improving, and you define what "better" means. [Scorers](/factories/measure-and-improve/) classify completed runs against criteria you write, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) groups the failures they flag into follow-up runs that propose fixes — to the application code or to the factory's own definition. Every proposal arrives as a change for your review; nothing is adopted on its own.
 
 The factory's definition is open to the same loop. Anyone on the team, or an agent, can propose changes to its instructions, skills, models, or other [definition files](/factories/factory-as-code/), and definitions stored in GitHub go through pull request review and [configuration checks](/factories/factory-as-code/#pull-request-checks-for-github-backed-factories) before a change reaches the production branch.
 

src/content/docs/factories/index.mdx

@@ -49,7 +49,7 @@ Warp Factories is designed for engineering teams with repeatable work that exten
 * **Definitions as code** - [Version-controlled definition files](/factories/factory-as-code/) describe your repositories, agents, automations, runners, [skills](/factories/factory-skills/), and MCP servers, so factory changes get the same review, history, and rollback as code changes.
 * **Integrations and the Factory MCP** - Work flows in from [Slack](/factories/integrations/slack/), [GitHub](/factories/integrations/github/), [GitLab](/factories/integrations/gitlab/), [Linear](/factories/integrations/linear/), and [Jira](/factories/integrations/jira/), plus direct runs and schedules. The [Factory MCP](/factories/factory-mcp/) connects coding agents and other MCP clients.
 * **Model and harness choice** - Each agent can use a different model and [supported harness](/platform/harnesses/), including the Warp Agent, Claude Code, and Codex.
-* **Measurement and self-improvement** - The [factory dashboard](/factories/factory-dashboard/) shows work-item status, runs, automations, costs, and benchmarks. [Scorers](/factories/measure-and-improve/) grade completed work, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) turns repeated failures into follow-up work the factory proposes for review.
+* **Measurement and self-improvement** - The [factory dashboard](/factories/factory-dashboard/) shows work-item status, runs, automations, costs, and benchmarks. [Scorers](/factories/measure-and-improve/) classify completed runs, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) turns repeated failures into follow-up work the factory proposes for review.
 * **Infrastructure control** - Run on Warp-hosted infrastructure, or self-host execution on an eligible Enterprise plan. Teams can also connect supported inference providers, scope secrets, and (if eligible) store transcripts, artifacts, and run attachments in their own S3 or GCS buckets. See [infrastructure and security](/factories/infrastructure-and-security/) for the available controls.
 
 ## How Warp Factories relates to other Warp products

src/content/docs/factories/measure-and-improve.mdx

@@ -1,7 +1,7 @@
 ---
 title: Measure and improve a factory
 description: >-
-  Measure factory activity and costs, evaluate completed conversations, compare
+  Measure factory activity and costs, evaluate completed runs, compare
   agent configurations, and turn failures into follow-up work.
 sidebar:
   label: "Measure and improve"
@@ -12,7 +12,7 @@ Warp Factories tracks what your factory produces and how well it performs, so yo
 | Feature | What it tells you |
 | --- | --- |
 | Dashboard metrics | How much work the factory produced, and what it cost. |
-| Scorers | Whether completed conversations meet criteria you define. |
+| Scorers | Whether completed runs meet criteria you define. |
 | Benchmarks | How different configurations perform on the same tasks. |
 | Self-improvement | Which repeated failures get investigated and turned into follow-up work. |
 
@@ -42,22 +42,22 @@ Use the **Dashboard** page to pick which runs to investigate, not to conclude wh
 
 ## Configure Scorers
 
-A **Scorer** uses an LLM judge to classify completed conversations against criteria you write, such as "did the agent run the tests before opening a PR?" Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix. For Scorers defined as files in a factory definition, see the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository.
+A **Scorer** uses an LLM judge to classify completed runs against criteria you write — for example, "did the agent run the tests before opening a PR?" It assigns a label, not a numeric grade, so keep each Scorer focused on one question its failures can point back to. Create Scorers on the factory dashboard's **Scorers** page, which also holds each Scorer's results. For Scorers defined as files in a factory definition, see the [`scorers/<name>/scorer.md` syntax](/factories/factory-as-code/#scorersnamescorermd) and the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository.
 
 Configure these fields:
 
 * **Agent(s) to evaluate** - The agents this Scorer applies to. Select at least one.
 * **Judge instructions** - The criteria the judge checks for.
 * **Judge model** - The model that acts as the judge.
 * **Classifications** - The labels the judge can assign, each with a score.
-* **Pass threshold** - The score a conversation needs to pass.
-* **Sample rate** - The share of the selected agents' completed conversations to evaluate.
+* **Pass threshold** - The score a run needs to pass.
+* **Sample rate** - The share of the selected agents' completed runs to evaluate.
 
 {/* VISUAL: The Scorer create/edit form (judge instructions, judge model, classifications, pass threshold, sample rate). */}
 
-While the sample rate is above 0, scoring runs automatically: shortly after a sampled conversation completes, the judge evaluates it and records a classification, a score, and its reasoning. To stop automatic scoring, set the sample rate to 0.
+While the sample rate is above 0, scoring happens automatically: shortly after a sampled run completes, the judge evaluates it and records a classification, a score, and its reasoning. To stop automatic scoring, set the sample rate to 0.
 
-You can also score any single conversation on demand, which is useful for testing new judge instructions before raising the sample rate. Scoring a conversation again replaces its previous result from that Scorer.
+You can also score any single run on demand, which is useful for testing new judge instructions before raising the sample rate. Scoring a run again replaces its previous result from that Scorer.
 
 Changing **Pass threshold** updates how past scores display as pass or fail; the recorded results don't change.
 
@@ -81,7 +81,7 @@ Every benchmark also runs **Correctness**, a built-in Scorer that marks each tri
 
 Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code. It can also improve the factory itself: when you manage your factory as [definitions as code](/factories/factory-as-code/), its prompts, skills, and configuration are version-controlled files, so a follow-up run can open a pull request against the factory definition the same way it would against application code. Nothing is adopted without your review.
 
-Each Self-improvement pull request includes a **Regressions addressed** section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence.
+The factory dashboard's **Self-improvement** page lists the pull requests these follow-up runs open. Each pull request includes a **Regressions addressed** section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence.
 
 ## Run a practical improvement loop
 
@@ -99,9 +99,9 @@ flowchart LR
   Improve -.-> Adopt
 ```
 
-1. **Define a Scorer.** Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then score a few conversations manually and compare the judge's results against your own review.
+1. **Define a Scorer.** Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then score a few runs manually and compare the judge's results against your own review.
 2. **Collect a baseline.** Let automatic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs.
-3. **Inspect failures.** Read the judge's reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating.
+3. **Inspect failures.** Read the judge's reasoning and the underlying runs. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating.
 4. **Benchmark a candidate.** Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference.
 5. **Review and adopt.** If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones.
 6. **Keep monitoring.** Leave the Scorer active and compare new results against your baseline. Revise the Scorer, or set its sample rate to 0, when its criteria no longer match what your team needs.