verify-results-pgtap.sh correctly scans for real not ok lines and plan mismatches first, but
then unconditionally fails make results if $TESTOUT/regression.diffs exists at all, regardless
of content:
# Also check regression.diffs (output mismatch even if pgtap all passed)
if [ -r "$TESTOUT/regression.diffs" ]; then
echo "ERROR: Tests are failing. Cannot run 'make results'."
...
exit 1
fi
That makes it impossible to ever use make results to seed the first expected-output file for a
brand-new test: going from no baseline to any content is always a diff, even when every actual
pgtap assertion inside it says ok. The check can never pass for that case no matter how many
times it's rerun -- it's not a transient failure, it's structural.
Repro: add a new test/sql/foo.sql whose assertions all pass, run make test once so
test/results/foo.out exists, then make results. It refuses with "Tests are failing," even
though foo.out contains no not ok lines and no plan mismatch -- the only diff is
--- test/expected/foo.out not existing yet.
Worked around this time by hand-copying test/results/security.out to
test/expected/security.out directly, which the project's own docs otherwise say never to do.
Possible fix: when a given results/*.out file's expected/*.out counterpart doesn't exist yet,
skip the regression.diffs fallback for that file (the earlier not ok/plan-mismatch scan already
covers real regressions in its actual content).
verify-results-pgtap.shcorrectly scans for realnot oklines and plan mismatches first, butthen unconditionally fails
make resultsif$TESTOUT/regression.diffsexists at all, regardlessof content:
That makes it impossible to ever use
make resultsto seed the first expected-output file for abrand-new test: going from no baseline to any content is always a diff, even when every actual
pgtap assertion inside it says
ok. The check can never pass for that case no matter how manytimes it's rerun -- it's not a transient failure, it's structural.
Repro: add a new
test/sql/foo.sqlwhose assertions all pass, runmake testonce sotest/results/foo.outexists, thenmake results. It refuses with "Tests are failing," eventhough
foo.outcontains nonot oklines and no plan mismatch -- the only diff is--- test/expected/foo.outnot existing yet.Worked around this time by hand-copying
test/results/security.outtotest/expected/security.outdirectly, which the project's own docs otherwise say never to do.Possible fix: when a given
results/*.outfile'sexpected/*.outcounterpart doesn't exist yet,skip the regression.diffs fallback for that file (the earlier
not ok/plan-mismatch scan alreadycovers real regressions in its actual content).