Add static prompt evaluation and red-team framework

.gitignore

@@ -14,9 +14,12 @@ yarn.lock
 .vscode/settings.json
 .azure/
 .venv/
+__pycache__/
+*.pyc
 downloads/
 ctrf/
 coverage/
+evaluations/static-prompts/results/
 .auth/
 test-videos/
 test-snapshots/

.gitleaks.toml

@@ -63,3 +63,13 @@ paths = [
   '''.*\.test\.ts$''',
   '''.*\.integration\.test\.ts$''',
 ]
+
+# The harvested OpenAPI fingerprint is a content hash, not an API credential.
+# Match only this verified digest in the dataset manifest, including history.
+[[allowlists]]
+description = "Static prompt dataset OpenAPI SHA-256 fingerprint"
+targetRules = ["generic-api-key"]
+condition = "AND"
+regexTarget = "secret"
+paths = ['''^evaluations/static-prompts/datasets/manifest\.json$''']
+regexes = ['''^4f6776ac4602e279c8723acd9a6cb22fa504c95538ab9491e0b5acab267c6a6b$''']

AGENTS.md

@@ -179,6 +179,26 @@ pnpm test:integration             # Integration tests (requires .env + Docker)
 
 > **Portal Storybook stories run under Vitest**: `apps/portal/src/components/ui/stories.play.test.tsx` composes the `ui/*` stories and executes their `play` (interaction) functions inside the regular Vitest suite (no `@storybook/addon-vitest` required). It binds a Testing Library `canvas` to the rendered container, so story `play` functions must keep depending only on `canvas` plus values imported directly from `storybook/test` (`userEvent`, `screen`, `expect`). When you add a new `ui/*` story with a `play` function, register its module in that harness so it's covered.
 
+### Static prompt evaluations
+
+Read [docs/architecture/prompt-evaluations.md](docs/architecture/prompt-evaluations.md)
+before changing any AI-facing instruction surface.
+
+- Whenever hardcoded system/user prompt text, prompt-building logic, output
+  instructions, or AI-facing tool descriptions are added or modified, update
+  the corresponding production target adapter, curated cases, deterministic
+  checks, composition contract, and/or rubric.
+- Register every new runtime static prompt family in the documented inventory,
+  committed evaluation manifest, and JSONL adapter registry.
+- Run the smallest relevant explicit prompt-evaluation command before treating
+  a prompt change as complete. These suites are developer-invoked and are not
+  part of normal `pnpm test` or CI.
+- A change to user-authored/configurable prompt content does not by itself
+  create a static prompt family.
+- Whenever a user-controlled text field that reaches an AI is added or its
+  insertion point, trusted wrapper, role, tools, or security boundary changes,
+  add or update its red-team surface profile and benign composition contract.
+
 ## Contributing (Pull Requests)
 
 This repository is commonly worked on from a **fork**. When opening a pull
@@ -216,6 +236,7 @@ not open the PR against the fork unless the user explicitly asks you to.
 |----------|-------------|
 | [docs/architecture/overview.md](docs/architecture/overview.md) | System architecture, component interactions, data flow |
 | [docs/architecture/app-design.md](docs/architecture/app-design.md) | Data models, API design, package dependency graph |
+| [docs/architecture/prompt-evaluations.md](docs/architecture/prompt-evaluations.md) | Static prompt quality, user-controlled AI red teaming, datasets, commands, and maintenance rules |
 | [docs/architecture/data-organization-projects.md](docs/architecture/data-organization-projects.md) | Projects (a single container) to isolate/group data within a cluster; composes with data-tags and auth-rbac |
 | [docs/architecture/vscode-web-worker.md](docs/architecture/vscode-web-worker.md) | XState chat machine, GitHub auth flow, ARIA snapshots |
 | [docs/architecture/token-manager.md](docs/architecture/token-manager.md) | Token storage, validation, round-robin distribution |

ENV_VARIABLES.md

@@ -167,6 +167,144 @@ inference errors at runtime. Learned compatibility is cached in each API
 process by endpoint and deployment name. It is relearned after a process
 restart or when Azure rejects a previously accepted request shape.
 
+## Prompt Evaluation Configuration
+
+These variables are consumed only by the developer-run tooling in
+`evaluations/static-prompts`. They do not enable the suite in normal tests or
+CI. See [Prompt Evaluations](docs/architecture/prompt-evaluations.md) for the
+quality/red-team split, commands, artifact policy, and cloud limitations.
+
+Run `az login` before cloud evaluation. The Python tooling uses Azure Identity;
+the quality graders also accept an explicit Azure OpenAI API key when required.
+Do not commit credentials, endpoints, tenant/subscription IDs, or a populated
+environment file.
+
+### PROMPT_EVAL_MODEL
+**Default:** resolved inference credential model, then `LLM_MODEL`, then `gpt-4.1`
+**Type:** string
+**Used by:** Static prompt TypeScript generation adapters
+
+Model/deployment used to generate production-path responses for the quality
+track. This setting is independent from the Azure-assisted evaluator deployment
+so generator and grader identities are explicit in result metadata.
+
+Generation uses the existing inference credential chain described in
+[LLM Configuration](#llm-configuration-portal-ai-features), including
+`AZURE_AI_INFERENCE_ENDPOINT` and `AZURE_AI_INFERENCE_API_KEY` when configured.
+
+### SCOPE_EVAL_AZURE_OPENAI_ENDPOINT
+**Required:** AI-assisted quality evaluation
+**Fallback:** `AZURE_OPENAI_ENDPOINT`
+**Type:** URL string
+**Used by:** Azure AI Evaluation SDK quality graders
+
+Azure OpenAI resource endpoint used by built-in and configurable quality
+graders. This is the model resource endpoint, not the Foundry project endpoint.
+
+### SCOPE_EVAL_AZURE_OPENAI_DEPLOYMENT
+**Required:** AI-assisted quality evaluation
+**Fallback:** `AZURE_OPENAI_DEPLOYMENT`
+**Type:** string
+**Used by:** Azure AI Evaluation SDK quality graders
+
+Deployment used to grade generated quality rows. Keep it distinct from
+`PROMPT_EVAL_MODEL` when evaluating one model's output with another.
+
+### SCOPE_EVAL_AZURE_OPENAI_API_KEY
+**Default:** unset
+**Fallback:** `AZURE_OPENAI_API_KEY`
+**Type:** string
+**Used by:** Azure AI Evaluation SDK quality graders
+
+Optional API key for the quality grader endpoint. When neither key variable is
+set, the runner uses `DefaultAzureCredential`; `az login` is the recommended
+local authentication path.
+
+### SCOPE_EVAL_AZURE_OPENAI_API_VERSION
+**Default:** SDK default
+**Fallback:** `AZURE_OPENAI_API_VERSION`
+**Type:** string
+**Used by:** Azure AI Evaluation SDK quality graders
+
+Optional Azure OpenAI API version override.
+
+### SCOPE_EVAL_AZURE_AI_PROJECT_ENDPOINT
+**Default:** unset
+**Fallback:** `AZURE_AI_PROJECT_ENDPOINT`
+**Type:** URL string
+**Used by:** Azure AI Evaluation SDK quality runs
+
+Optional Foundry project endpoint for publishing a quality evaluation portal
+view. Local JSON/JSONL artifacts remain the source of truth.
+
+### AZURE_AI_PROJECT_ENDPOINT
+**Required:** Cloud red-team runs
+**Type:** URL string
+**Used by:** Azure AI Projects red-team runner
+
+Foundry project endpoint used to create, poll, download, and clean up cloud
+red-team evaluations. It is also the fallback for
+`SCOPE_EVAL_AZURE_AI_PROJECT_ENDPOINT` in quality runs.
+
+Use the project endpoint for the intended Foundry project, not an inference
+`/models` endpoint. Red teaming authenticates only through
+`DefaultAzureCredential`; it does not read an API-key environment variable.
+
+### AZURE_AI_MODEL_DEPLOYMENT_NAME
+**Required:** Cloud red teaming
+**Type:** string
+**Used by:** Azure AI Projects red-team runner
+
+Foundry/Azure OpenAI deployment used by `builtin.task_adherence` and as the
+target model behind cloud red-team model or temporary prompt-agent targets. The
+default reviewed configuration expects the verified `gpt-5.4-mini` deployment,
+but the deployment name is always recorded in local run metadata.
+
+### SCOPE_RED_TEAM_KEEP_REMOTE
+**Default:** unset / false
+**Type:** boolean (`1`, `true`, `yes`, or `on`)
+**Used by:** Cloud red-team runner
+
+Preserves temporary Foundry targets created for a red-team run. By default the
+runner deletes only targets it created, after native results are downloaded.
+The shared Foundry project and model deployment are never deleted.
+
+Use this only for debugging. Preserved targets consume cloud resources and must
+be removed manually.
+
+### SCOPE_RED_TEAM_SURFACES
+**Default:** unset (all reviewed profiles)
+**Type:** comma-separated surface IDs
+**Used by:** Cloud red-team runner
+
+Restricts a red-team invocation to selected profiles. Supported IDs are
+`task-scenario-prompt`, `gate-prompt`, `agents-md`, `criterion-prompt`,
+`prompt-feature-definition`, `persona-instructions`, `report-user-prompt`, and
+`report-system-prompt`.
+
+Example:
+
+```bash
+SCOPE_RED_TEAM_SURFACES=criterion-prompt,report-user-prompt pnpm eval:red-team
+```
+
+### Quality runner diagnostic overrides
+
+`SCOPE_EVAL_GENERATOR_COMMAND` overrides the production JSONL generator command
+using `{dataset}`, `{output}`, `{samples}`, and `{smoke}` placeholders.
+`SCOPE_EVAL_GENERATOR_TIMEOUT_SECONDS` changes its 1,800-second timeout.
+`SCOPE_EVAL_GENERATED_ROWS` copies an existing JSONL file instead of invoking
+generation. These are test/debugging controls; normal baseline runs should use
+the declared package generator.
+
+### Harvester bearer token
+
+The dataset harvester accepts the **name** of an environment variable through
+its token option rather than reading a fixed secret name. This supports
+protected integration environments without establishing a repository-wide
+credential variable. Never pass the token value on the command line and never
+write it to curated rows, provenance, logs, or results.
+
 ## Prompt Storage Configuration
 
 ### PROMPT_INLINE_MAX_BYTES

README.md

@@ -143,6 +143,7 @@ The default Portal address is `http://localhost:5100`. In a Git worktree,
 | Discover available run commands | `pnpm cli run --help` |
 | Start all available workers locally | `pnpm docker:dev:all` |
 | Run unit tests | `pnpm test` |
+| Evaluate Scope prompts | `pnpm eval:prompts -- --mode quality\|red-team\|both` |
 | Configure Portal AI assistance | [Environment variables](./ENV_VARIABLES.md#llm-configuration-portal-ai-features) |
 
 ## Shape the benchmark
@@ -170,6 +171,7 @@ model together.
 apps/       API, Portal, CLI, scheduler, Judge, gateway, token manager, workers
 packages/   Shared types, storage clients, migrations, auth, model scanning, evaluation
 config/     Portable examples for scenarios, personas, criteria, and prompt features
+evaluations/ Developer-run static prompt quality and cloud red-team tooling
 docs/       Architecture, operations, research, and design documentation
 ```
 
@@ -181,6 +183,7 @@ docs/       Architecture, operations, research, and design documentation
 | API, domain models, and project boundaries | [Application design](./docs/architecture/app-design.md) |
 | Queue priority, recovery, and scaling | [Queue scheduler](./docs/architecture/queue-scheduler.md) |
 | Criteria DAGs and evaluation providers | [Criteria provider](./docs/architecture/criteria-provider.md) |
+| Static prompt quality and red teaming | [Prompt evaluations](./docs/architecture/prompt-evaluations.md) |
 | Writing and delivering agent skills | [Skills architecture](./docs/architecture/skills.md) |
 | CLI installation and automation | [CLI distribution](./docs/architecture/cli-distribution.md) |
 | Environment settings | [Environment variable reference](./ENV_VARIABLES.md) |