fix(api): confine the instrument token to upload, and open a new connection per gateway request - #1561
Merged
Conversation
The token `GET /v1/auth/create-instrument-token` mints carried `manage Instrument`, the permission that route itself requires, so a minted token could mint its successor every hour and never expire. It carried no user id, so disabling the admin who minted it did nothing, and rotating `SECRET_KEY` was the only way to retire it. It also reached every route its permission satisfied: deleting unused series instruments in any group, the gateway healthcheck, and the instrument read routes, which answered 500 on its missing groups. Every token now carries a `kind`, `login` or `instrument`, and the guard refuses an instrument token on any route not marked `@AcceptsInstrumentToken()`. Only `POST /v1/instruments`, the playground's upload, is marked. A token signed before `kind` existed passes as a login token, so sessions open at deploy survive it, and a minted one of those can renew only into a token that cannot renew again. No `SECRET_KEY` rotation is needed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Node reuses idle HTTP connections by default, and the gateway's server closes one after five idle seconds. When that close arrived while the API's event loop was busy, the API sent its next gateway request on the dead connection and it failed with ECONNRESET: a 500 for a clinician creating a remote assignment, or a failed synchronizer pass. The gateway client now turns keep-alive off for both HTTP and HTTPS, so no request goes out on a connection the gateway has already closed. Its options move to `gateway.http.ts` so that property can be tested. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The default assignment duration is one instance-wide value. Tagged `@smoke`, its test also ran in firefox, and outside CI, where `workers: 1` keeps them apart, both copies ran at once and each overwrote the other's value between its save and its reload. Repeated in parallel it fails 6 of 18 on main as well. The three other tests in the file, which also write instance-wide settings, already run in chromium only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
1.
fix(api): refuse the instrument token everywhere but instrument upload (security audit, high: "GET /v1/auth/create-instrument-tokenmints a principal-less token that is sufficient authorization to mint its own successor")The token the playground uploads with carried
manage Instrument, the same permission the mint route requires. So a minted token could mint its successor every hour and never expire. It carried no user id, so disabling the admin who minted it did nothing, and only rotatingSECRET_KEYwould retire it. It also reached every other route that permission satisfies: deleting unused series instruments in any group, the gateway healthcheck, and the instrument read routes, which answered 500.Every token now carries a
kind,loginorinstrument.JwtAuthGuardrefuses aninstrumenttoken on any route not marked@AcceptsInstrumentToken(), and onlyPOST /v1/instrumentsis marked. A token signed beforekindexisted passes as a login token, so sessions open at deploy keep working. A leaked pre-fix token can renew only once, into a token that cannot renew again, so noSECRET_KEYrotation is needed.2.
fix(api): open a new connection for every gateway requestThe API's gateway client reused idle keep-alive connections (Node's default), and the gateway's Node server closes one after 5 idle seconds. If that close arrives while the API's event loop is busy, the API sends its next gateway request on the dead connection and gets
ECONNRESET. In production that is a 500 when a clinician creates a remote assignment, or a failed synchronizer pass. The client now turns keep-alive off for HTTP and HTTPS. Its options moved toapps/api/src/gateway/gateway.http.tsso this can be tested.3.
test(testing): stop running the default duration test in firefox. Test-only change, explained below.How the gateway bug was found
After commit 1, a full
pnpm test:e2efailed wheremainpassed, and the failing test changed from run to run. I first called it a pre-existing flake without testingmain, which was wrong. So I compared the two branches directly on one machine:POST /v1/assignmentsreturning 500 after about 4.5 s. The API log hadread ECONNRESETfromGatewayService.createRemoteAssignment. It showed up in 2 of 7 runs on this branch and 0 of 6 onmain. No code path from commit 1 reaches it, so the way to settle it was to find the mechanism.ECONNRESETevery time. With a free event loop it never did. The e2e API is regularly this busy, for example evaluating instrument bundles server-side.mainand on this branch alike, so that race is not from this branch.Why the test changed
The default assignment duration is one value shared by the whole instance. The test was tagged
@smoke, so it also ran in Firefox. Outside CI both copies ran at once, and each overwrote the other's value between its save and its reload. CI hides this withworkers: 1. The three other tests inadmin-settings.spec.tsalso write instance-wide settings and already run in Chromium only, so this now matches them.testing/AGENTS.mdrecords the rule.Verification
Unit:
apps/api/src/auth/guards/__tests__/jwt-auth.guard.spec.ts: an instrument token is refused on unmarked routes and admitted on marked ones; a token with nokindpasses as a login token.apps/api/src/auth/__tests__/auth.service.spec.ts: the minted token is signed withkind: 'instrument'.apps/api/src/auth/__tests__/auth.controller.spec.ts: the mint route is unmarked.apps/api/src/instruments/__tests__/instruments.controller.spec.ts:createis the only marked handler.apps/api/src/gateway/__tests__/gateway.http.spec.ts: two requests open two connections. Without the fix they share one.Each new test was watched failing against deliberately broken code.
E2E:
testing/src/specs/authorization.spec.tschecks that a minted token gets 403 when it tries to mint a successor or to delete an instrument.Suites:
pnpm lintpasses, andpnpm testpasses (1379 tests). After both fixes, 4 of 5 fullpnpm test:e2eruns passed, with noECONNRESET.Not covered
testing/src/specs/gateway-assignment.spec.tsexercises the path.subject-detail.spec.ts"should plot a selected measure…" failed once in the five runs, and its trace was overwritten. It passed 8 of 8 on its own, andgraph.page.tsalready documents flakiness in this flow.admin-settings.spec.ts's language test turns French off for the whole instance. In a run of just those two specs onmain, that broke the gateway French test once. It was never seen in a full run.Co-Authored-By: Claude Opus 5.5 (1M context) noreply@anthropic.com