Skip to content

Assert the debug output's value, not its colour - #21

Merged
cjbarth merged 1 commit into
masterfrom
fix-debug-test-asserts-ansi
Sep 17, 2026
Merged

cjbarth merged 1 commit into
masterfrom
fix-debug-test-asserts-ansi

Conversation

@cjbarth

@cjbarth cjbarth commented Sep 17, 2026

Copy link
Copy Markdown
Owner

What broke

npm run release fails at npm test, on a test I added in #20:

1) Gren release membership
     Options
       Should not print the token in the debug output:
   AssertionError: The option is still reported: expected '...Options: ...' to include 'Token:'

The redaction itself works — the assert.notInclude(output, "<the token>") beside it
passes. What fails is the next assertion. _outputOptions writes
chalk.yellow("Token") + ": <hidden>", so with colour on there is an ANSI reset between
the name and the colon and the literal Token: never appears.

chalk goes quiet when stdout is not a TTY. That covers CI and every scripted run, so the
test passed on the PR, passed on the merge to master, and passed every time I ran it. It
failed the first time it ran in a terminal, which was your release.

This is exactly what AGENTS.md says not to do:

Don't assert another tool's formatting. Normalise at the boundary and assert the
value, not the spelling it arrived in.

My mistake, and the reason the release stopped.

The fix

Normalise at the boundary with stripVTControlCharacters from node:util — a built-in,
available well under the >= 24 engines floor, so no new dependency and no
no-control-regex suppression. The assertion then reads Token: <hidden>, the value it
was always about.

No change to lib/. The redaction behaviour merged in #20 is unaffected.

Verification

Reproduced first, on master, before the fix:

npm test                -> 138 passing
FORCE_COLOR=1 npm test  -> 137 passing, 1 failing   <- your failure

After:

npm test                                       -> 138 passing
FORCE_COLOR=1 npm test                         -> 138 passing
env -u GREN_GITHUB_TOKEN npx mocha             ->  99 passing, 39 pending
env -u GREN_GITHUB_TOKEN FORCE_COLOR=1 mocha   ->  99 passing, 39 pending

The whole suite is clean under forced colour, not just this test, so nothing else in it
asserts on chalk's rendering. npm run lint is clean apart from the pre-existing
mocha/no-pending-tests warning in test/Gren.spec.js.

Follow-up worth doing separately

CI only ever sees a pipe, so it cannot catch this class of bug by construction: a
coloured-string assertion is always green there and always red for you. Running the suite
once more with FORCE_COLOR set closes that gap. I could not include it here — pushing a
.github/workflows/ change needs a token with workflow scope, which this one does not
have — so the diff is below if you want it:

       - run: npm ci
       - run: npm test
+      - name: Test with colour forced on
+        run: npm test
+        env:
+          # chalk goes quiet when stdout is a pipe, which is all CI ever sees. A test that
+          # asserts on a coloured string passes here and fails for whoever runs it by hand.
+          FORCE_COLOR: 1
       - run: npm run lint

It is additive: the existing uncoloured runs are untouched, and I verified the full suite
passes under forced colour on both the with-token and without-token paths.

🤖 Generated with Claude Code

The token test looked for "Token:" in what _outputOptions wrote. chalk
colours the option name, so an ANSI reset sits between the name and the
colon, and the assertion only held where chalk had gone quiet -- any run
whose stdout is a pipe, meaning CI and every scripted run. It failed the
first time the suite ran in a terminal, during a release.

The captured output now goes through stripVTControlCharacters, so the
test asserts the value it was always about rather than the spelling it
arrived in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cjbarth
cjbarth merged commit e37d667 into master Sep 17, 2026
2 checks passed
@cjbarth
cjbarth deleted the fix-debug-test-asserts-ansi branch September 17, 2026 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant