Skip to content

Stop the cutover and rollback scripts reporting success while shadow mode is unresolved #261

Description

@NathanTarbert

scripts/cutover/execute-cutover.ts and scripts/cutover/rollback.ts are the two scripts that flip shadow mode around a cutover. Both report the step succeeded without confirming the flag actually moved, and in opposite directions from each other.

The cutover path passes while shadow mode is still on

disableShadowMode has two exits and both are PASS. The second one is reached precisely when shadow mode is still engaged:

console.log('  ACTION REQUIRED: Set SHADOW_MODE=false in Railway environment variables');
console.log('  Then redeploy the discord-bot service.');

return { step, status: 'PASS', message: 'Shadow mode disable instruction issued', timestamp: new Date() };

Steps 1-3 in executeCutover each gate on status === 'FAIL' and return early. Step 4 pushes its result and falls straight through to the announcement step and log.outcome = 'SUCCESS'.

So with the flag still on, --confirm prints:

  [OK] Disable shadow mode: Shadow mode disable instruction issued
  [OK] Post announcement: Announcement: "..."

  Cutover COMPLETED

and exits 0. The announcement goes to the community, the incumbent gets turned off, and Outpost answers nobody — because shadow mode never came off.

Also, currentValue is interpolated into the already-disabled message but dropped from the still-on one. The branch where the raw value is the whole diagnosis is the branch that omits it.

The rollback path never reads the flag at all

enableShadowMode prints three ACTION REQUIRED lines and returns status: 'DONE'. It does not call isShadowMode(), does not read process.env.SHADOW_MODE, and verifies nothing. executeRollback gates only step 1, so the outcome is SUCCESS regardless.

This is the worse half, because it is the incident path. Production is double-posting, the operator runs rollback --confirm, reads Rollback COMPLETED, and moves on to the 30-minute monitoring step while Outpost keeps posting at real reporters the whole time.

Suggested shape

Make the unresolved branch not a pass, carry the raw value, and gate the step the way the three above it are gated:

return {
    step,
    status: 'FAIL',
    message:
        `Shadow mode is still ON (SHADOW_MODE=${currentValue ?? 'unset'}). ` +
        'Set it to false in Railway, redeploy, and re-run — cutover is not complete while the flag is engaged.',
    timestamp: new Date(),
};

If a manual step reading as a hard failure is wrong, a third status (ACTION_REQUIRED) treated as non-SUCCESS in the aggregation works too. "PASS / COMPLETED" for a step that just verified the flag is still on is the part that needs to go.

The rollback fix mirrors it, and isShadowMode() helps there: an unrecognized value reads as ON, which is the right answer for a rollback.

Test gap that goes with it

scripts/__tests__/cutover.test.ts covers runHealthChecks, verifyTicketData, executeCutover and executeRollback, but there is no test for disableShadowMode and nothing in the file sets SHADOW_MODE. Worth adding the same fence the queue suites got in #233it.each(['TRUE', '1', 'yes', 'on', ' true ', '', 'flase']) asserting the step does not report the cutover done.

Use the delete-when-absent restore from packages/outpost/queue/src/__tests__/ai-response.test.ts: assigning undefined stores the string "undefined", which is an unrecognized value under the current rules and would leak shadow-mode-ON into every later test in the file.

Both behaviours predate #233 — that PR routed the comparison through isShadowMode(), which fixed which direction the message is wrong in and is what made this visible. Filed separately so that diff stays about the predicate.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: infrastructureWorker, queue, CI, deploy, containers, observabilityroadmap: nowRoadmap horizon: V1 launch critical path / safety

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions