Run the skill-harness suite in CI; fix pnpm 11 installs - #36
Open
oschaaf wants to merge 3 commits into
Open
Conversation
pnpm 11 no longer reads the `pnpm` key in package.json, so the esbuild override that lived there was silently ignored and `pnpm install --frozen-lockfile` refused the committed lockfile. Move the override to skill-harness/pnpm-workspace.yaml, which is where pnpm 11 looks for it. While there, declare `allowBuilds: esbuild: false`. esbuild's postinstall is only a fallback for platforms without a prebuilt optional dependency; on every platform we run on the binary comes from the optional package, so the build script is not needed and the install stays free of arbitrary postinstall execution. Pin the toolchain in package.json so CI and local runs agree: `packageManager: pnpm@11.18.0` (read by pnpm/action-setup) and `engines.node: 24` (read by actions/setup-node). Add a `typecheck` script (`tsc --noEmit`) so CI can run it by name. The lockfile is unchanged: a frozen install accepts it byte-for-byte once the override is read from the right place. Closes #30 Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
Add .github/workflows/skill-harness.yml. On every pull request and every push to main it runs the offline checks from skill-harness/: the schema digest `--check` (first, on the runner's default Node, since the script needs no dependencies), a frozen-lockfile install, a lockfile-stability guard (`git diff --exit-code` on the lockfile and workspace file), the test suite, `biome check`, and `tsc --noEmit`. The checks after install are gated on the install step rather than on the previous step, so one run reports every failure. The live bench (`pnpm bench`) is deliberately not run: it calls a model API and needs credentials. No paths filter, so the job can later be made a required check. Token is read-only and checkout does not persist credentials; nothing in the job needs more. Actions are pinned to commit SHAs with the tag in a comment. The dependabot config keeps those pins current (weekly, grouped) and bumps the harness npm dependencies in one weekly PR. zod majors are ignored: the harness asserts against a vendored validator bundle built for the current zod major, so that bump is a deliberate re-vendor. Closes #33 Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
The schema-digest `--check` and the harness suite are now run by .github/workflows/skill-harness.yml on every PR and push to main, so the docs that described `pnpm test` as the only enforcement point now name the workflow. Also list `pnpm typecheck` with the other offline commands and note that the live bench is not run in CI. Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
Member
Author
|
Drift the workflow now enforces on every PR/push (these guards existed but only ran if someone ran pnpm test locally):
Drift the PR adds new guards for:
What it does not guard:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #33. Closes #30.
The
skill-harnessguards (digest drift, tests, biome) currently only run locally. This adds a workflow that runs them on every PR and every push tomain, and first fixes the pnpm 11 install failure that would have made the workflow red on day one.fix(skill-harness): pnpm 11 frozen installs (#30)pnpmfield ofpackage.json, so theesbuild >=0.28.1override silently vanished andpnpm install --frozen-lockfilefailed withERR_PNPM_LOCKFILE_CONFIG_MISMATCH. The override now lives inskill-harness/pnpm-workspace.yaml, the place pnpm 11 reads it from.allowBuilds: esbuild: false— pnpm 11 hard-fails underCI=trueon an un-reviewed build script. esbuild's postinstall is only a fallback for when the platform binary isn't present; the binary ships as the optional dep that pnpm installs anyway, so nothing needs to run.packageManager: pnpm@11.18.0+engines.node: "24"pin the toolchain in one place; the workflow reads both frompackage.json, so local == CI.pnpm-lock.yamlis unchanged: the frozen install accepts it byte-for-byte once the override is visible again.ci:.github/workflows/skill-harness.yml(#33)Single
checkjob onubuntu-24.04, read-only token, no secrets, actions pinned by commit SHA (adependabot.ymlkeeps the pins and the harness deps current — one grouped PR a week, zod majors excluded because the validator bundle is byte-pinned to this major). Steps:node scripts/generate-schema-digest.mjs --check— runs first from the repo root, before any toolchain setup (onlynode:builtins), so a stale digest is reported even when install failspnpm install --frozen-lockfilegit diff --exit-codeon the lockfile + workspace file, so the class of bug behind pnpm install --frozen-lockfile and pnpm test are broken on pnpm 11 (overrides moved) #30 (pnpm wanting to rewrite committed files) fails loudly instead of passing unfrozenpnpm test,pnpm biome:check,pnpm typecheck(new script,tsc --noEmit; green today) — each gated on install only, so one run reports every failureNot run in CI:
pnpm bench(calls a model API; needs a key and a cost decision — separate discussion if wanted). No paths filter, so the job can later be made a required check without docs-only PRs getting stuck on "expected".docsCLAUDE.md / README wording that said
pnpm testis the enforcing gate now points at the workflow. Noplugin.jsonbump — CI config is non-distributed.Verification
From a fresh clone on pnpm 11.18.0 / Node 24: frozen install clean with an empty
git statusafterwards, 177/177 tests, biome clean,tsc --noEmitclean, digest in sync,actionlintclean. All three action SHAs re-resolved against their tags. Mutation-probed: removingallowBuilds→ERR_PNPM_IGNORED_BUILDSunderCI=true; removing the workspace file → the original #30 error; a one-character digest edit →--checkexits 1.Follow-up worth filing after this lands:
scripts/generate-schema-digest.mjssits outside biome's scope and has a whole-file format diff pending; kept out of this PR to keep the review focused.Suggest making
checka required status check onmainonce this first run is green.