Add static prompt evaluation and red-team framework - #1398

Open

Cedric Vidal (cedricvidal) wants to merge 23 commits into

microsoft:mainfrom

cedricvidal:cedricvidal-plan-static-prompt-evals

Commits on Sep 9, 2026

refactor: expose prompt composition seams

Cedric Vidal (cedricvidal)
and
Copilot
committed
9f5511a
feat: add static prompt evaluation suites

Cedric Vidal (cedricvidal)
and
Copilot
committed
2bceee9
docs: document prompt evaluation maintenance

Cedric Vidal (cedricvidal)
and
Copilot
committed
5123706
test: use synthetic redaction fixture

Cedric Vidal (cedricvidal)
and
Copilot
committed
a3b86de
fix: isolate concurrent judge evaluation workspaces

Cedric Vidal (cedricvidal)
and
Copilot
committed
768b627
fix: delete temporary red-team agents

Cedric Vidal (cedricvidal)
and
Copilot
committed
da10457
fix: report actual red-team target mode

Cedric Vidal (cedricvidal)
and
Copilot
committed
7d9b913
feat: guide developers when uv is missing

Cedric Vidal (cedricvidal)
and
Copilot
committed
bdf91dc
feat: generate markdown evaluation reports

Cedric Vidal (cedricvidal)
and
Copilot
committed
0751e51
docs: explain evaluation policy gating in reports

Cedric Vidal (cedricvidal)
and
Copilot
committed
65b06e7
style: remove trailing blank lines

Cedric Vidal (cedricvidal)
and
Copilot
committed
20ad442
fix(evals): cap aggregate pass-rate policy at eighty percent

Cedric Vidal (cedricvidal)
and
Copilot
committed
b7d5b6a
fix(evals): parse native grader outputs and preserve task context

Cedric Vidal (cedricvidal)
and
Copilot
committed
a182160
feat(evals): add auditable gate decisions and immutable selective replay

Cedric Vidal (cedricvidal)
and
Copilot
committed
4ace354
fix(evals): standardize decision summary and read-only legacy exports

Cedric Vidal (cedricvidal)
and
Copilot
committed
170260d
fix(evals): keep eighty-percent policy solely in rubric configuration

Cedric Vidal (cedricvidal)
and
Copilot
committed
e43a420

Commits on Sep 10, 2026

fix(evals): handle pnpm separator in report CLI

Cedric Vidal (cedricvidal)
and
Copilot
committed
4a1061a
fix(evals): resolve incomplete sample votes and reject source overlap early

Cedric Vidal (cedricvidal)
and
Copilot
committed
c0fbd7a
fix(evals): normalize SDK tool-call fields and retain evaluator diagnostics

Cedric Vidal (cedricvidal)
and
Copilot
committed
1ad33a6
fix(evals): explain unresolved report gates and format scores

Cedric Vidal (cedricvidal)
and
Copilot
committed
8e2813b
docs(evals): explain case context in review clients

Cedric Vidal (cedricvidal)
and
Copilot
committed
e4b0195
Merge upstream main and preserve prompt evaluation builders

Cedric Vidal (cedricvidal)
and
Copilot
committed
7485982
fix: restore clean evaluation builds and scope hash scan exemption

Cedric Vidal (cedricvidal)
and
Copilot
committed
9aba7f0

Navigation Menu

Add static prompt evaluation and red-team framework - #1398

Add static prompt evaluation and red-team framework#1398
Cedric Vidal (cedricvidal) wants to merge 23 commits into
microsoft:mainfrom
cedricvidal:cedricvidal-plan-static-prompt-evals

Commits on Sep 9, 2026

refactor: expose prompt composition seams

feat: add static prompt evaluation suites

docs: document prompt evaluation maintenance

test: use synthetic redaction fixture

fix: isolate concurrent judge evaluation workspaces

fix: delete temporary red-team agents

fix: report actual red-team target mode

feat: guide developers when uv is missing

feat: generate markdown evaluation reports

docs: explain evaluation policy gating in reports

style: remove trailing blank lines

fix(evals): cap aggregate pass-rate policy at eighty percent

fix(evals): parse native grader outputs and preserve task context

feat(evals): add auditable gate decisions and immutable selective replay

fix(evals): standardize decision summary and read-only legacy exports

fix(evals): keep eighty-percent policy solely in rubric configuration

Commits on Sep 10, 2026

fix(evals): handle pnpm separator in report CLI

fix(evals): resolve incomplete sample votes and reject source overlap early

fix(evals): normalize SDK tool-call fields and retain evaluator diagnostics

fix(evals): explain unresolved report gates and format scores

docs(evals): explain case context in review clients

Merge upstream main and preserve prompt evaluation builders

fix: restore clean evaluation builds and scope hash scan exemption