fix: Settle the severity threshold contract and retry analyzer crashes

[tablackburn]

Summary

Settles the FailBuildOnSeverityLevel / SeverityThreshold contract before the public API
freezes at 1.0.0, and removes the analyzer flake folded in from #147.

• ParseError records now fail the gate. They were counted by no threshold at all, so a
  file that does not parse escaped even the strictest validated value. Behavioral change —
  has a migration-guide entry.
• Any is now an accepted SeverityThreshold. It was documented but missing from the
  ValidateSet, so setting it failed parameter binding. Additive.
• The documentation comment no longer contradicts itself, and now describes every value
  including Information, which it never explained.
• A crashed analyzer rule is retried rather than producing a random red build (Test-PSBuildScriptAnalysis does not fail on the PSScriptAnalyzer rule crash from #136 #147).

Closes #144. Closes #147.

The four problems

1. ParseError escaped every threshold

PSScriptAnalyzer's severity enum has four members; Test-PSBuildScriptAnalysis counted three.
A ParseError — a file that does not parse at all — satisfied no threshold, including
Information, the strictest validated value. It was printed in the results table and the build
passed. The most severe possible finding was the only one that could never fail a build.

ParseError is now counted with Error.

2. 'Any' was documented but rejected

build.properties.ps1 promised "Any" will fail the build on any diagnostic record, but the
ValidateSet was 'None', 'Error', 'Warning', 'Information'. Setting it failed parameter
binding. The catch-all branch that implements the behavior existed but was reachable only when
no threshold was supplied — and the consumer-facing Analyze task always supplies one, so no
consumer could ever reach it. 'Any' is now accepted and wired to that branch.

3. The comment contradicted itself

It opened with "Valid values are Error, Warning, Information and None", then described Any,
and never said what Information does. Rewritten to describe all five values plus ParseError.

4. Analyzer rule crashes (#147)

PSScriptAnalyzer runs script rules in parallel against a process-wide unsynchronised singleton,
so a rule can crash on a race unrelated to the code under analysis
(PSScriptAnalyzer#1538, deferred
to 2.0). A re-run succeeds. The analysis is now retried up to three times on RULE_ERROR.

Errors are captured during the retries rather than allowed to surface, because a consumer with
$ErrorActionPreference = 'Stop' — what this repository itself does — would otherwise
terminate on the first crash and never reach the second attempt. Whatever the final attempt
still reports is re-emitted, so a persistent failure behaves exactly as it did before.

I took #147's preferred approach, not its alternative. The issue measured that a crashed rule
does not cost findings (100 cold runs, never fewer than the known-good count), so a crash is
not by itself a reason to fail — making a hard failure out of it would have cost some consumers
something to help others. This removes the flake without changing what a real failure does.

Test Plan

• Full suite — 440 passed / 0 failed (up from 428; 12 new tests)
• ParseError fails at the Error, Warning, Information, and Any thresholds, and
  still only reports at None
• Any fails on Error, Warning, and Information findings, and passes with no findings
• ValidateSet assertion updated to Any,Error,Information,None,Warning
• Retry: succeeds after one crash (2 invocations), gives up after 3 and still surfaces the
  error, and does not retry a non-RULE_ERROR failure (1 invocation)
• Existing severity-gating, end-to-end, and settings-file coverage unchanged and passing

Run locally under psake 4.9.1, matching this branch's pin.

One note for reviewers on the retry tests: their mocks set $ErrorActionPreference = 'SilentlyContinue' inside the mock body. That is deliberate, not cargo cult — the real cmdlet
honours the -ErrorAction SilentlyContinue the function passes, but a mock body inherits the
ambient preference, which build.ps1 sets to Stop process-wide. Without it the mock throws on
the first attempt and the retry can never be exercised.

Breaking Changes

Yes — one. ParseError records now fail every threshold except None. A consumer whose
build has an unparsable file was passing on 0.8.x and will now correctly fail. Documented in
docs/migration-v0.8-to-v1.0.md with a detection command:

Invoke-ScriptAnalyzer -Path ./Output/MyModule/1.0.0 -Recurse |
    Where-Object Severity -eq 'ParseError'

Any is additive and the retry only removes a flake, so neither requires consumer action.

Reviewer notes

• Why this belongs in 1.0.0: SeverityThreshold is part of the public API being frozen.
  Adding an accepted value is additive and safe now; removing or redefining one after 1.0.0
  would not be.
• The default switch branch (unsupplied threshold) is deliberately left alone — changing it is
  outside this issue.

Part of #120.

[Copilot]

Pull request overview

This PR finalizes the FailBuildOnSeverityLevel / SeverityThreshold contract ahead of the 1.0.0 API freeze by (1) making ParseError findings participate in severity gating, (2) accepting the documented-but-rejected Any threshold value, and (3) reducing flakiness by retrying PSScriptAnalyzer rule-crash (RULE_ERROR) failures.

Changes:

• Count ParseError alongside Error so unparsable files fail all thresholds except None.
• Add Any to the SeverityThreshold ValidateSet and implement the “fail on any diagnostic record” behavior.
• Retry script analysis up to 3 times on RULE_ERROR analyzer crashes, re-surfacing persistent errors after the final attempt.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
PowerShellBuild/Public/Test-PSBuildScriptAnalysis.ps1	Implements Any threshold, counts ParseError with Error, and adds bounded retry for RULE_ERROR analyzer crashes.
tests/Test-PSBuildScriptAnalysis.tests.ps1	Adds/updates Pester coverage for Any, ParseError gating, and retry behavior.
PowerShellBuild/en-US/Messages.psd1	Adds localized warning text for analyzer crash retry.
PowerShellBuild/build.properties.ps1	Updates consumer-facing documentation for severity threshold values and ParseError behavior.
docs/migration-v0.8-to-v1.0.md	Documents the behavioral change: unparsable files now fail the script-analysis gate.
CHANGELOG.md	Records the breaking/added/fixed user-facing changes for #144 and #147.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results

    4 files  ± 0    716 suites  +20   3m 34s ⏱️ -7s
  448 tests +12    446 ✅ +12   2 💤 ±0  0 ❌ ±0 
1 796 runs  +48  1 749 ✅ +48  47 💤 ±0  0 ❌ ±0 

Results for commit b6d3146. ± Comparison against base commit f726f48.

[tablackburn]

CI is green on all legs, including Windows PowerShell 5.1.

Two notes for whoever reviews:

Squash-merge, please. The branch commit subject is 76 characters, over the repo's 72-character limit (git-workflow.instructions.md). Rather than force-push a rewrite, I shortened the PR title to 70 characters — squashing uses that as the commit subject, so the merged history is compliant. If you merge with a merge-commit or rebase instead, the long subject lands as-is.

On #147, I took the issue's preferred approach rather than its alternative. #147 offered a hard failure (-ErrorAction Stop + retry) as the more conventional option, but its own measurements showed a crashed rule does not cost findings — 100 cold runs, never fewer than the known-good count. So a crash is not by itself a reason to fail, and making it one would have cost some consumers a red build to help others. The retry captures errors only for the duration of the attempts and re-emits anything the final attempt still reports, so a persistent failure behaves exactly as it did before. Happy to switch to the hard-fail variant if you would rather have the symmetry with the Analyze task's retry.