Skip to content

Show every region that differs when Should-BeString fails on a big string - #3010

Merged
nohwnd merged 3 commits into
pester:mainfrom
nohwnd:nohwnd-string-diff-engine
Aug 25, 2026
Merged

Show every region that differs when Should-BeString fails on a big string#3010
nohwnd merged 3 commits into
pester:mainfrom
nohwnd:nohwnd-string-diff-engine

Conversation

@nohwnd

@nohwnd nohwnd commented Aug 24, 2026

Copy link
Copy Markdown
Member

Supersedes #3003, which scanned for the first differing line. That answers "one thing changed". Comparing a snapshot is "many things changed", and a scan stops at the first one.

Fix #2951
Fix #3006

What it prints

Four lines changed in a package.json:

Expected strings to be the same, but they were different.
Expected length: 236
Actual length:   243
Expected 16 line(s), actual 16 line(s).
2 regions differ.

   1  1 |   {
   2  2 |     "name": "widget",
   3    | -   "version": "1.2.0",
      3 | +   "version": "1.3.0",
   4    | -   "license": "MIT",
      4 | +   "license": "Apache-2.0",
   5  5 |     "main": "index.js",
   6  6 |     "scripts": {
  ...
   9  9 |     },
  10 10 |     "dependencies": {
  11    | -     "left-pad": "^1.3.0"
     11 | +     "left-pad": "^1.3.1"
  12 12 |     },
  13 13 |     "engines": {
  14    | -     "node": ">=18"
     14 | +     "node": ">=20"
  15 15 |     }
  16 16 |   }

Expected line numbers on the left, actual on the right, so it stays readable when lines are added and the two sides stop lining up:

   1  1 |   Describe 'Api' {
      2 | +     BeforeAll {
      3 | +         Start-TestServer
      4 | +     }
      5 | + 
   2  6 |       It 'returns 200' {
  ...
   9    | -         $r.ContentType | Should-BeString 'application/json'
  10 13 |       }
  11 14 |   }

Only the lines that differ are expanded, so a tab or a trailing space is visible without turning the context into escape codes:

  1 1 |   first
  2   | -   indented with spaces
    2 | + ␉indented with a tab
  3   | - no trailing space
    3 | + no trailing space···
  4 4 |   last

The one difference you cannot work out by reading the output gets named instead:

Every line is the same, only the line endings differ.
Expected: 4 CRLF
But was:  4 LF
Use -NormalizeLineEnding to ignore this.

At most five regions are printed, then it says how many are left:

12 regions differ, showing the first 5.
...
  7 more region(s) differ, not shown.

Short strings are untouched. The full text with a caret under the difference is still the most precise thing we can show, and a long single line still gets the excerpt with ellipses.

Two DiffPlex defaults that fail quietly

Both are set in StringDiff.cs and both have tests, because getting them wrong produces no output rather than wrong output.

Setting Default Why the default is wrong here
ignoreWhitespace true A difference that is only a trailing space is reported as no difference at all.
line chunker LineChunker It throws the line endings away, so CRLF against LF comes back as two identical strings. LineEndingsPreservingChunker keeps them.

SideBySideDiffBuilder pairs changed lines and gives positions on both sides, so Pester does not have to do that bookkeeping.

Public surface: one method

Pester.StringDiff.Format(expected, actual, caseSensitive, context, maxRegions) returns the part of the failure message that describes how the two strings differ. That is the whole public API this PR adds.

Every vendored type is internal, so DiffPlex is not part of Pester's compatibility surface and can be updated, trimmed further or swapped out without that being a breaking change. StringDiffResult and Compare are internal too; the unit tests reach them through InternalsVisibleTo.

Vendored, not referenced

Shipping DiffPlex.dll next to Pester.dll puts a second assembly identity in the user's session. Windows PowerShell 5.1 has no load context isolation, so a module that already loaded a different DiffPlex version wins and Pester cannot do anything about it.

  • 14 files out of the 29 in the library. ThreeWayDiffer, UnidiffRenderer and InlineDiffBuilder are left behind, and so are five more files that only existed to serve members Pester never calls.
  • Apache-2.0, the same licence Pester uses.
  • Three mechanical rewrites (namespace, internal, licence header) plus pester.patch, which removes the members Pester does not call. The patch touches two files, removes 99 lines and adds 7.
  • src/csharp/Pester/DiffPlex/VENDORING.md records the version and the commit (3cb6415, what the 1.9.0 nuspec says the package was built from), and says what is left out and why.
  • Update-VendoredDiffPlex.ps1 rebuilds the whole copy from upstream and -Verify compares the result against what is on disk. Deleting code is what usually turns a vendored copy into a fork nobody can update, so the removals are kept as a patch rather than applied by hand: run it against a newer DiffPlex and either the patch applies or git says which hunk failed.

The licence obligations

Apache-2.0 asks for three things and the second commit adds two of them.

Where
A copy of the licence with the source src/csharp/Pester/DiffPlex/LICENSE.txt, unmodified
A copy of the licence with the binary, section 4(a) ThirdPartyNotices.txt, which build.ps1 now copies into the built module. Pester.dll has DiffPlex compiled into it, and the module previously shipped no licence file at all
Modified files saying they are modified, section 4(b) a four line header on each of the 19 files, carrying source, author, licence and the change

There is no NOTICE file upstream, so section 4(d) does not apply. The upstream source files carry no copyright header of their own, so there was nothing to retain under 4(c); the header does the attribution instead, naming Matthew Manela as the nuspec does.

Update-VendoredDiffPlex.ps1 -Verify strips the header before comparing, so it still proves every file is upstream apart from the namespace.

Upstream releases about once a year (1.7.2 in 2023-12, 1.8.0 in 2025-05, 1.9.0 in 2025-09), so this is not something we need to track closely.

Cost with embedded symbols, which is how Pester.csproj builds:

Before After Added
Pester.dll net8.0 123 904 B 145 920 B 21.5 KB
Pester.dll net462 117 760 B 139 776 B 21.5 KB

Embedded symbols carry the source text, so the licence headers cost about 5 KB and dropping the unused members gave 3.5 KB back.

The full DiffPlex.dll is 33.3 KB per target framework, so vendoring the part we use is also the smaller option.

Verification

  • Full suite on macOS: 2922 passed, 0 failed, 3 skipped.
  • src/csharp/PesterTests: 78 passed, 0 failed. 20 of those are new, in StringDiffTests.cs.
  • Update-VendoredDiffPlex.ps1 -Verify passes against DiffPlex 3cb6415.
  • One differing line in a 10 000 line file takes 2 ms and prints 6 lines.

Note that CI does not currently run src/csharp/PesterTests at all, so the C# tests here only run locally. That is worth fixing separately.

🤖

…ring

Should-BeString printed both strings in full when they were different. For the
10 000 line file in pester#2951 that is a hundred thousand lines of output and a
character offset nobody can act on.

Scanning for the first differing line fixes that when one thing changed. It does
not fix comparing a snapshot, where several things change at once and the useful
output is every changed region with its context. A scan stops at the first one.

So the diffing is done by DiffPlex now, and Pester renders it:

- Strings small enough to read are left exactly as they were, the full text with
  a caret under the difference is the most precise thing we can show.
- Bigger ones get the changed regions with two lines of context each, expected
  line numbers on the left and actual on the right, at most five regions and a
  count of how many were left out.
- Only the lines that differ are expanded, so a trailing space or a stray CR is
  visible without turning every other line into escape codes. Trailing spaces
  show up as the middle dot, control characters keep the Control Pictures that
  Formatter.EscapeControlChars already uses.
- When every line has the same text and only the endings differ it says so and
  names the endings on each side, because printing the lines would show two
  blocks that look identical.

Two DiffPlex defaults are wrong for this and both of them fail quietly.
ignoreWhitespace defaults to true, and with it a difference that is only a
trailing space is reported as no difference at all. The default line chunker
throws the endings away, so CRLF against LF comes back as two identical strings.
StringDiff.cs sets both, and there are tests for both, because a regression there
is silent.

DiffPlex is vendored, not referenced. Shipping DiffPlex.dll next to Pester.dll
puts a second assembly identity in the session, and Windows PowerShell 5.1 has no
load context isolation, so another module that already loaded a different DiffPlex
version wins. 19 files, 1080 lines, Apache-2.0 which is the same licence Pester
uses, and the only change to them is the namespace. VENDORING.md records the
version and Update-VendoredDiffPlex.ps1 re-copies it, or checks it with -Verify.
It costs 19 KB of Pester.dll on net8.0 and 18.5 KB on net462, against 33 KB per
target framework for the full library.

Fix pester#2951
Fix pester#3006

🤖

@github-advanced-security github-advanced-security AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PSScriptAnalyzer found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

The first commit put DiffPlex's LICENSE.txt next to the vendored source and stated the
one modification in VENDORING.md. That covers the repository. It does not cover two
things Apache-2.0 asks for.

Section 4(b) asks that modified files say they are modified. All 19 files are modified,
the namespace is Pester.DiffPlex, and none of them said so. They now carry a four line
header with the source, the author, the licence and the change. The upstream files have
no copyright header of their own, so the same header does the attribution, which the
nuspec gives as Matthew Manela.

Section 4(a) asks that whoever gets the code gets the licence, in source or in binary
form. Pester.dll has DiffPlex compiled into it and the built module shipped no licence
file at all, so anyone installing from the gallery was getting DiffPlex without it. The
module now ships ThirdPartyNotices.txt, and build.ps1 copies it in.

Update-VendoredDiffPlex.ps1 takes the header off before comparing, so -Verify still
proves every file is upstream apart from the namespace.

The recorded commit is now 3cb6415, which is what the 1.9.0 nuspec says the package was
built from. The DiffPlex folder there is identical to 8821ff9, which is what I recorded
before, so the vendored code did not change.

The headers cost bytes because the assemblies are built with embedded symbols. Pester.dll
grows by 24 KB per target framework now, not the 19 KB the first commit reported.

🤖
@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

Vendoring put 20 DiffPlex types into Pester's public API. Nothing asked for that, and
it would make the vendored library part of the compatibility surface, so replacing it
or updating it across a breaking change becomes our problem.

Every vendored type is internal now. Pester exposes one method,
Pester.StringDiff.Format, which takes the two strings and returns the part of the
failure message that describes how they differ. StringDiffResult and Compare are
internal, and PesterTests sees them through InternalsVisibleTo.

The files were already modified, so there was no reason to keep code we never call.
pester.patch removes Differ's convenience methods, IDiffer, ISideBySideDiffBuilder,
and the SideBySideDiffBuilder constructors and static helpers that default to
LineChunker. Five files then have nothing referring to them and are not vendored at
all. 19 files became 14, and Pester.dll grows by 22 KB per target framework instead
of 24 KB.

WordChunker and DelimiterChunker stay. SideBySideDiffBuilder uses them for the word
level sub pieces on a changed line, which we do not render yet, and highlighting the
changed part inside a line is the obvious next thing to do to this message. Cutting
them means editing the core of SideBySideDiffBuilder rather than deleting whole
members.

Deleting code is what breaks a vendored copy, because a hand-trimmed one diverges
quietly and nobody can tell what was changed on purpose. So the removals are kept as
a patch, not applied by hand. Update-VendoredDiffPlex.ps1 rebuilds the whole thing
from upstream, applies the namespace rewrite, the internal rewrite and the header,
then applies pester.patch, and -Verify compares that against what is on disk. Running
it against a newer DiffPlex either works or git says which hunk failed. -Regenerate
rewrites the patch after a deliberate hand edit.

🤖
@nohwnd
nohwnd merged commit bfcbd4d into pester:main Aug 25, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Diff engine for string comparisons with many differences (snapshot testing) Should-BeString does not truncate identical prefix in diff message

2 participants