Skip to content

Run the skill-harness suite in CI; fix pnpm 11 installs - #36

Open
oschaaf wants to merge 3 commits into
mainfrom
ci/skill-harness-workflow
Open

Run the skill-harness suite in CI; fix pnpm 11 installs#36
oschaaf wants to merge 3 commits into
mainfrom
ci/skill-harness-workflow

Conversation

@oschaaf

@oschaaf oschaaf commented Aug 21, 2026

Copy link
Copy Markdown
Member

Closes #33. Closes #30.

The skill-harness guards (digest drift, tests, biome) currently only run locally. This adds a workflow that runs them on every PR and every push to main, and first fixes the pnpm 11 install failure that would have made the workflow red on day one.

fix(skill-harness): pnpm 11 frozen installs (#30)

  • pnpm 11 no longer reads the pnpm field of package.json, so the esbuild >=0.28.1 override silently vanished and pnpm install --frozen-lockfile failed with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH. The override now lives in skill-harness/pnpm-workspace.yaml, the place pnpm 11 reads it from.
  • allowBuilds: esbuild: false — pnpm 11 hard-fails under CI=true on an un-reviewed build script. esbuild's postinstall is only a fallback for when the platform binary isn't present; the binary ships as the optional dep that pnpm installs anyway, so nothing needs to run.
  • packageManager: pnpm@11.18.0 + engines.node: "24" pin the toolchain in one place; the workflow reads both from package.json, so local == CI.
  • pnpm-lock.yaml is unchanged: the frozen install accepts it byte-for-byte once the override is visible again.

ci: .github/workflows/skill-harness.yml (#33)

Single check job on ubuntu-24.04, read-only token, no secrets, actions pinned by commit SHA (a dependabot.yml keeps the pins and the harness deps current — one grouped PR a week, zod majors excluded because the validator bundle is byte-pinned to this major). Steps:

  1. node scripts/generate-schema-digest.mjs --check — runs first from the repo root, before any toolchain setup (only node: builtins), so a stale digest is reported even when install fails
  2. pnpm install --frozen-lockfile
  3. Lockfile is stablegit diff --exit-code on the lockfile + workspace file, so the class of bug behind pnpm install --frozen-lockfile and pnpm test are broken on pnpm 11 (overrides moved) #30 (pnpm wanting to rewrite committed files) fails loudly instead of passing unfrozen
  4. pnpm test, pnpm biome:check, pnpm typecheck (new script, tsc --noEmit; green today) — each gated on install only, so one run reports every failure

Not run in CI: pnpm bench (calls a model API; needs a key and a cost decision — separate discussion if wanted). No paths filter, so the job can later be made a required check without docs-only PRs getting stuck on "expected".

docs

CLAUDE.md / README wording that said pnpm test is the enforcing gate now points at the workflow. No plugin.json bump — CI config is non-distributed.

Verification

From a fresh clone on pnpm 11.18.0 / Node 24: frozen install clean with an empty git status afterwards, 177/177 tests, biome clean, tsc --noEmit clean, digest in sync, actionlint clean. All three action SHAs re-resolved against their tags. Mutation-probed: removing allowBuildsERR_PNPM_IGNORED_BUILDS under CI=true; removing the workspace file → the original #30 error; a one-character digest edit → --check exits 1.

Follow-up worth filing after this lands: scripts/generate-schema-digest.mjs sits outside biome's scope and has a whole-file format diff pending; kept out of this PR to keep the review focused.

Suggest making check a required status check on main once this first run is green.

pnpm 11 no longer reads the `pnpm` key in package.json, so the esbuild
override that lived there was silently ignored and `pnpm install
--frozen-lockfile` refused the committed lockfile. Move the override to
skill-harness/pnpm-workspace.yaml, which is where pnpm 11 looks for it.

While there, declare `allowBuilds: esbuild: false`. esbuild's postinstall
is only a fallback for platforms without a prebuilt optional dependency;
on every platform we run on the binary comes from the optional package,
so the build script is not needed and the install stays free of
arbitrary postinstall execution.

Pin the toolchain in package.json so CI and local runs agree:
`packageManager: pnpm@11.18.0` (read by pnpm/action-setup) and
`engines.node: 24` (read by actions/setup-node). Add a `typecheck`
script (`tsc --noEmit`) so CI can run it by name.

The lockfile is unchanged: a frozen install accepts it byte-for-byte
once the override is read from the right place.

Closes #30

Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
Add .github/workflows/skill-harness.yml. On every pull request and every
push to main it runs the offline checks from skill-harness/: the schema
digest `--check` (first, on the runner's default Node, since the script
needs no dependencies), a frozen-lockfile install, a lockfile-stability
guard (`git diff --exit-code` on the lockfile and workspace file), the
test suite, `biome check`, and `tsc --noEmit`. The checks after install
are gated on the install step rather than on the previous step, so one
run reports every failure.

The live bench (`pnpm bench`) is deliberately not run: it calls a model
API and needs credentials. No paths filter, so the job can later be made
a required check. Token is read-only and checkout does not persist
credentials; nothing in the job needs more.

Actions are pinned to commit SHAs with the tag in a comment. The
dependabot config keeps those pins current (weekly, grouped) and bumps
the harness npm dependencies in one weekly PR. zod majors are ignored:
the harness asserts against a vendored validator bundle built for the
current zod major, so that bump is a deliberate re-vendor.

Closes #33

Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
The schema-digest `--check` and the harness suite are now run by
.github/workflows/skill-harness.yml on every PR and push to main, so
the docs that described `pnpm test` as the only enforcement point now
name the workflow. Also list `pnpm typecheck` with the other offline
commands and note that the live bench is not run in CI.

Signed-off-by: Otto van der Schaaf <otto@dtwo.ai>
@oschaaf
oschaaf requested a review from pareilly August 21, 2026 15:54
@oschaaf

oschaaf commented Aug 21, 2026

Copy link
Copy Markdown
Member Author

Drift the workflow now enforces on every PR/push (these guards existed but only ran if someone ran pnpm test locally):

  • SKILL.md ↔ schema-reference.json — generate-schema-digest.mjs --check fails if the generated digest section is stale (runs first, even if install breaks).
  • Vendored validator bundle — suite asserts the byte sha256 of config-validator.bundle.mjs and its 2.0.0 version pin; schema artifact likewise pinned by sha256 + 1.1.0.
  • Hand-written safe defaults ↔ artifact's gatewayDefault — the mismatched/shadowed/unresolved buckets throw at build.
  • Digest coverage gates — every target/constraint/reserved key/default flavor must render; a dangling schemaPath is fatal.
  • Type drift — tsc --noEmit now runs for the first time in any gate.

Drift the PR adds new guards for:

  • Lockfile/config drift — the Lockfile is stable step fails if a frozen install wants to rewrite pnpm-lock.yaml or pnpm-workspace.yaml (the pnpm install --frozen-lockfile and pnpm test are broken on pnpm 11 (overrides moved) #30 bug class, which previously silently
    re-resolved esbuild back to a vulnerable version).
  • Toolchain drift — pnpm and Node are pinned in package.json; CI and local read the same values.
  • Action drift — SHA-pinned actions plus dependabot so the pins are bumped deliberately rather than floating.

What it does not guard:

  • Upstream drift — whether the vendored bundle/schema still match the product repo's current artifact. The harness pins what's committed; detecting that the source moved needs a
    producer-side check (tracked as a follow-up on the product side).
  • Behavioral drift of the skill itself — that's the live bench, which stays out of CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant