Skip to content

fix(api): confine the instrument token to upload, and open a new connection per gateway request - #1561

Merged
joshunrau merged 3 commits into
DouglasNeuroInformatics:mainfrom
joshunrau:fixes
Sep 23, 2026
Merged

joshunrau merged 3 commits into
DouglasNeuroInformatics:mainfrom
joshunrau:fixes

Conversation

@joshunrau

Copy link
Copy Markdown
Collaborator

What changed

1. fix(api): refuse the instrument token everywhere but instrument upload (security audit, high: "GET /v1/auth/create-instrument-token mints a principal-less token that is sufficient authorization to mint its own successor")

The token the playground uploads with carried manage Instrument, the same permission the mint route requires. So a minted token could mint its successor every hour and never expire. It carried no user id, so disabling the admin who minted it did nothing, and only rotating SECRET_KEY would retire it. It also reached every other route that permission satisfies: deleting unused series instruments in any group, the gateway healthcheck, and the instrument read routes, which answered 500.

Every token now carries a kind, login or instrument. JwtAuthGuard refuses an instrument token on any route not marked @AcceptsInstrumentToken(), and only POST /v1/instruments is marked. A token signed before kind existed passes as a login token, so sessions open at deploy keep working. A leaked pre-fix token can renew only once, into a token that cannot renew again, so no SECRET_KEY rotation is needed.

2. fix(api): open a new connection for every gateway request

The API's gateway client reused idle keep-alive connections (Node's default), and the gateway's Node server closes one after 5 idle seconds. If that close arrives while the API's event loop is busy, the API sends its next gateway request on the dead connection and gets ECONNRESET. In production that is a 500 when a clinician creates a remote assignment, or a failed synchronizer pass. The client now turns keep-alive off for HTTP and HTTPS. Its options moved to apps/api/src/gateway/gateway.http.ts so this can be tested.

3. test(testing): stop running the default duration test in firefox. Test-only change, explained below.

How the gateway bug was found

After commit 1, a full pnpm test:e2e failed where main passed, and the failing test changed from run to run. I first called it a pre-existing flake without testing main, which was wrong. So I compared the two branches directly on one machine:

  • Found the error. A trace of a failing gateway test showed POST /v1/assignments returning 500 after about 4.5 s. The API log had read ECONNRESET from GatewayService.createRemoteAssignment. It showed up in 2 of 7 runs on this branch and 0 of 6 on main. No code path from commit 1 reaches it, so the way to settle it was to find the mechanism.
  • Reproduced it outside the suite. I used axios against a default Node HTTP server in a separate process. When the client's event loop was blocked across the server's 5 s idle close, the next request failed with ECONNRESET every time. With a free event loop it never did. The e2e API is regularly this busy, for example evaluating instrument bundles server-side.
  • Separated out a second failure. The duration test also failed by reading back another run's value. Repeated 6× per browser in parallel, it fails 6 of 18 on main and on this branch alike, so that race is not from this branch.

Why the test changed

The default assignment duration is one value shared by the whole instance. The test was tagged @smoke, so it also ran in Firefox. Outside CI both copies ran at once, and each overwrote the other's value between its save and its reload. CI hides this with workers: 1. The three other tests in admin-settings.spec.ts also write instance-wide settings and already run in Chromium only, so this now matches them. testing/AGENTS.md records the rule.

Verification

  • Unit:

    • apps/api/src/auth/guards/__tests__/jwt-auth.guard.spec.ts: an instrument token is refused on unmarked routes and admitted on marked ones; a token with no kind passes as a login token.
    • apps/api/src/auth/__tests__/auth.service.spec.ts: the minted token is signed with kind: 'instrument'.
    • apps/api/src/auth/__tests__/auth.controller.spec.ts: the mint route is unmarked.
    • apps/api/src/instruments/__tests__/instruments.controller.spec.ts: create is the only marked handler.
    • apps/api/src/gateway/__tests__/gateway.http.spec.ts: two requests open two connections. Without the fix they share one.

    Each new test was watched failing against deliberately broken code.

  • E2E: testing/src/specs/authorization.spec.ts checks that a minted token gets 403 when it tries to mint a successor or to delete an instrument.

  • Suites: pnpm lint passes, and pnpm test passes (1379 tests). After both fixes, 4 of 5 full pnpm test:e2e runs passed, with no ECONNRESET.

Not covered

  • No e2e test for the connection reset. It depends on timing no test can force. The existing testing/src/specs/gateway-assignment.spec.ts exercises the path.
  • One unrelated failure. subject-detail.spec.ts "should plot a selected measure…" failed once in the five runs, and its trace was overwritten. It passed 8 of 8 on its own, and graph.page.ts already documents flakiness in this flow.
  • Language cross-talk, not addressed. admin-settings.spec.ts's language test turns French off for the whole instance. In a run of just those two specs on main, that broke the gateway French test once. It was never seen in a full run.
  • Minted tokens still carry no user id. Attribution is left to a change that audits instrument creation.

Co-Authored-By: Claude Opus 5.5 (1M context) noreply@anthropic.com

joshunrau and others added 3 commits September 23, 2026 13:23
The token `GET /v1/auth/create-instrument-token` mints carried `manage Instrument`, the permission
that route itself requires, so a minted token could mint its successor every hour and never
expire. It carried no user id, so disabling the admin who minted it did nothing, and rotating
`SECRET_KEY` was the only way to retire it. It also reached every route its permission satisfied:
deleting unused series instruments in any group, the gateway healthcheck, and the instrument read
routes, which answered 500 on its missing groups.

Every token now carries a `kind`, `login` or `instrument`, and the guard refuses an instrument
token on any route not marked `@AcceptsInstrumentToken()`. Only `POST /v1/instruments`, the
playground's upload, is marked. A token signed before `kind` existed passes as a login token, so
sessions open at deploy survive it, and a minted one of those can renew only into a token that
cannot renew again. No `SECRET_KEY` rotation is needed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Node reuses idle HTTP connections by default, and the gateway's server closes one after five idle
seconds. When that close arrived while the API's event loop was busy, the API sent its next gateway
request on the dead connection and it failed with ECONNRESET: a 500 for a clinician creating a
remote assignment, or a failed synchronizer pass. The gateway client now turns keep-alive off for
both HTTP and HTTPS, so no request goes out on a connection the gateway has already closed. Its
options move to `gateway.http.ts` so that property can be tested.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The default assignment duration is one instance-wide value. Tagged `@smoke`, its test also ran in
firefox, and outside CI, where `workers: 1` keeps them apart, both copies ran at once and each
overwrote the other's value between its save and its reload. Repeated in parallel it fails 6 of 18
on main as well. The three other tests in the file, which also write instance-wide settings,
already run in chromium only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@joshunrau
joshunrau merged commit f1decef into DouglasNeuroInformatics:main Sep 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant