feat(record_cli): serve a different canned response per invocation

.claude/harness-candidates.md

@@ -455,3 +455,28 @@ with the two `action.yml` items above — one considered change to the action's
   `final_status`, which does not exist in `run.json` and would have made a new
   assertion dead on arrival. Guard: assert the key set that non-Python consumers
   depend on, mirroring how CE030 pins doc/schema parity.
+- [ ] **A probe task's detector must not be satisfiable from the sandbox-readable task
+  YAML.** `docker_runner._stage_inputs` serialises the post-override `TaskDefinition` to
+  `/work/input/task.yaml` and mounts `tasks/` again at `/work/task_dir`, both agent-readable.
+  Two probes now depend on NOT being satisfiable from that text — `anti_cheat_reference` via a
+  regex that cannot match its own source, `record_cli_responses` via a log-derived detector —
+  and nothing enforces it. `record_cli_responses` originally shipped (in review) with
+  `file_contains` needles that were verbatim in its own YAML, which would have let a
+  transcribing agent pass while dispatch was dead. Guard: for every `smoke-pass` task, assert no
+  `file_contains` needle / `file_matches_regex` pattern on a must-match criterion appears in the
+  serialised task YAML. Deferred: needs per-criterion-type handling and a real false-positive
+  pass (paths and generic words will collide), so well over 30 min.
+- [ ] **A new shim failure mode must still RECORD the invocation.** A generated shim that dies
+  before `record()` leaves a log byte-identical to "the agent never ran it", which passes a
+  `max_count: 0` guard. The sidecar import was exactly that, caught only in review. Guard:
+  render each shim shape, break each external dependency in turn, assert the log is non-empty.
+  Deferred: "each external dependency" has no enumeration today, so the rule needs a seam
+  (a declared list of what a shim depends on) before it can be mechanical rather than a
+  hand-maintained list that decays.
+- [ ] **A generated-artifact invariant must be asserted against a real run of that artifact,
+  not against the config that produced it.** `TestRecordCliProbeIntegrity` first shipped
+  comparing the task YAML with itself and hardcoding `"rule": 0` — a spelling `json.dumps`'s
+  default separators own — so a separator change would have left it green while the blocking CI
+  probe failed. Now fixed for this case by running a real shim. Deferred as a general guard:
+  "derives its expectation from the thing it checks" is not mechanically detectable; it belongs
+  in the review rubric rather than a lint rule.

.github/workflows/pr-checks.yml

@@ -469,16 +469,16 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
-      # tasks_run for --tags smoke-pass. 7 task files (hello_date, dataset_example,
+      # tasks_run for --tags smoke-pass. 8 task files (hello_date, dataset_example,
       # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test,
-      # anti_cheat_reference); dataset_example fans out to 2 inline rows, so 8
-      # sub-tasks. If you add/remove a smoke-pass task or change the dataset row
-      # count, bump these.
+      # anti_cheat_reference, record_cli_responses); dataset_example fans out to 2
+      # inline rows, so 9 sub-tasks. If you add/remove a smoke-pass task or change
+      # the dataset row count, bump these.
       #
       # anti_cheat_reference lives in a SUBDIRECTORY, which `tasks/*.yaml` does not
       # match — the smoke-pass step names its path explicitly. Keep that in sync.
-      EXPECTED_SMOKE_PASS_RUN: "8"
-      EXPECTED_SMOKE_PASS_SUCCEEDED: "8"
+      EXPECTED_SMOKE_PASS_RUN: "9"
+      EXPECTED_SMOKE_PASS_SUCCEEDED: "9"
       # smoke-fail bucket: three tasks expected to fail.
       #   1. smoke_negative_path: file_contains criterion is unsatisfiable
       #      (sentinel-string regression detection for success-checker).
@@ -549,6 +549,9 @@ jobs:
       # explicitly. anti_cheat_reference is the adversarial probe that the agent
       # cannot read the reference solution during its turn; it needs the
       # coder-eval-agent image built above (it is a driver: docker task).
+      # record_cli_responses is the record_cli per-invocation-response probe and
+      # is also driver: docker, so it needs that same image; it is flat in
+      # tasks/, so the glob already matches it.
       - name: Run smoke-pass bucket (expect all to succeed)
         run: |
           .venv/bin/coder-eval run tasks/*.yaml tasks/anti_cheat_reference/*.yaml \

CLAUDE.md

@@ -27,6 +27,8 @@ coder_eval/
 ├── fs_permissions.py              # set_permissions: stacked chmod window (via Sandbox.set_permissions)
 ├── pricing.py                     # Model pricing / cost calculation (ModelPricing, calculate_cost, register_pricing)
 ├── litellm_cost.py                # Join proxy-captured ACTUAL per-call cost/cache onto turns (LiteLLM backend; apply_actual_cost)
+├── invocation_log.py              # record_cli recording shim: renders it (emitting a sibling import of the argv_match sidecar), and parse_log reads its JSON Lines back
+├── argv_match.py                  # Structured argv matcher. STDLIB-ONLY (CE048): this file is copied into the recorder dir as a SIDECAR beside every response-serving shim, which imports it as a sibling, so `cli_called` and a `record_cli` response rule dispatch on ONE semantic
 ├── utils.py                       # Version info helpers
 │
 ├── agents/
@@ -38,10 +40,11 @@ coder_eval/
 │   ├── criteria.py                # 15 success criterion types + base + union
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
+│   ├── cli_match.py               # FlagMatch + CliMatch (a `when:` pattern) + the shared verb/flag validators (cycle-free leaf: criteria.py and sandbox.py both import it)
 │   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template)
 │   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
-│   ├── sandbox.py                 # SandboxConfig, ResourceLimits
+│   ├── sandbox.py                 # SandboxConfig, ResourceLimits, RecordedCli + CliResponse (per-invocation stub responses)
 │   ├── tasks.py                   # TaskDefinition, AgentConfig, Dataset (dataset fan-out + sample)
 │   ├── telemetry.py               # CommandTelemetry, CommandStatistics, TokenUsage, ProviderCallCost, ReconciliationMessage, TranscriptMessage
 │   └── templates.py               # RepoSource, TemplateDirSource, StarterFilesSource
@@ -51,6 +54,7 @@ coder_eval/
 │   ├── base.py                    # BaseCriterion (async _check_impl_async is primary; sync _check_impl derives from it, or vice versa) + @handle_criterion_errors(_async)
 │   ├── _classification_aggregate.py  # Shared overlay: accuracy / P/R/F1 / confusion matrix
 │   ├── classification_match.py    # File-based label matcher
+│   ├── cli_called.py              # Structured match over the record_cli invocation log (matching engine: argv_match.py)
 │   ├── command_executed.py
 │   ├── commands_efficiency.py
 │   ├── file_check.py
@@ -216,11 +220,11 @@ make plugin-reference  # the plugin's bundled criteria reference from the models
 
 Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction.
 
-Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed).
+Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed), **CE048** (a module copied into the recorder directory beside a generated sandbox shim — `models.sandbox.SIDECAR_MODULES`, currently `argv_match.py` — may import stdlib only. The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not installed, so one package import makes every shadowed CLI die with an ImportError the agent reads as "the tool is broken", costing a whole run to diagnose. The rule derives its target set from that exported tuple and a test asserts it matches a file that exists — a lint rule guarding zero files must fail, not pass).
 
 When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033–CE036 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE036 enforces the `live_verdict` determinism + monotonicity contract (`criteria/base.py`) that `EarlyStopWatcher`'s latching, deferred fail-stop, and flip-attribution silently depend on: monotonicity over arbitrary Python is undecidable, so instead of a static check it REPLAYS each live criterion against every prefix of recorded trajectories (`tests/lint/live_verdict_contract.py::CASES`) — on the authored ordering AND under seeded shuffles (`permuted_violations`, which catch order-sensitive bugs the authored walk misses) — and asserts the property directly, plus registry-derived coverage — every `LiveSuccessCriterion` in the union must have cases, and every polarity its instances claim via `live_decidable_polarities()` must actually be reached by one (otherwise a single always-`undecided` fixture would "cover" a type while proving nothing). Adding a live criterion therefore means adding `ContractCase`s in the same change. CE035 resolves every `steps.<id>.outputs.<key>` / `needs.<job>.outputs.<key>` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.)
 
-Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
+Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig`, `RecordedCli`, `CliResponse` — see `tests/lint/doc_schema_parity.py`, which is the SSOT; this list is a convenience copy) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
 
 **Docs index SSOT.** `nav:` plus `extra.docs_index` (blurbs) in `mkdocs.yml` are the single source of truth for the flat index surfaces — `README.md`'s Documentation table, `docs/index.md`'s "Where to go next" table, and the `## Docs` / `## Tutorials` sections of `docs/llms.txt`. Regenerate all three with `make docs-indexes`; **CE028** fails the build if any drifts, if a nav page lacks a blurb (or vice-versa), or if a `docs/*.md` page is missing from the nav. The website sidebar derives from the same `nav:`. When adding or renaming a docs page, edit `nav:` + `extra.docs_index` and run `make docs-indexes` — never hand-edit the generated tables (they sit between `<!-- docs-index:start -->` / `<!-- docs-index:end -->` markers).