Skip to content

verify-results-pgtap.sh can never accept a brand-new test file's first expected output #119

Description

@jnasbyupgrade

verify-results-pgtap.sh correctly scans for real not ok lines and plan mismatches first, but
then unconditionally fails make results if $TESTOUT/regression.diffs exists at all, regardless
of content:

# Also check regression.diffs (output mismatch even if pgtap all passed)
if [ -r "$TESTOUT/regression.diffs" ]; then
	echo "ERROR: Tests are failing. Cannot run 'make results'."
	...
	exit 1
fi

That makes it impossible to ever use make results to seed the first expected-output file for a
brand-new test: going from no baseline to any content is always a diff, even when every actual
pgtap assertion inside it says ok. The check can never pass for that case no matter how many
times it's rerun -- it's not a transient failure, it's structural.

Repro: add a new test/sql/foo.sql whose assertions all pass, run make test once so
test/results/foo.out exists, then make results. It refuses with "Tests are failing," even
though foo.out contains no not ok lines and no plan mismatch -- the only diff is
--- test/expected/foo.out not existing yet.

Worked around this time by hand-copying test/results/security.out to
test/expected/security.out directly, which the project's own docs otherwise say never to do.

Possible fix: when a given results/*.out file's expected/*.out counterpart doesn't exist yet,
skip the regression.diffs fallback for that file (the earlier not ok/plan-mismatch scan already
covers real regressions in its actual content).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions