feat(self-hosting): sign and attest published images, and correct the self-hosting docs - #7437
feat(self-hosting): sign and attest published images, and correct the self-hosting docs#7437waleedlatif1 wants to merge 15 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds signing, provenance, and architecture-specific SBOM publication for GHCR images, generates a complete Helm image inventory, and substantially corrects the self-hosting and enterprise documentation.
Confidence Score: 5/5The PR appears safe to merge because no blocking failure remains. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| .github/workflows/ci.yml | Adds digest resolution, keyless image signing, provenance, per-architecture SBOM publication, and release gating. |
| scripts/generate-image-manifest.ts | Generates the chart’s complete source-to-mirror image inventory from default and registry-rewritten Helm renders. |
| helm/sim/images.yaml | Documents every rendered image and now clearly distinguishes the NVIDIA device-plugin inventory path from its bare-repository fallback destination. |
| helm/sim/templates/gpu-device-plugin.yaml | Routes the device-plugin image through the common registry-aware image helper. |
| apps/docs/content/docs/platform/self-hosting/verify.mdx | Expands installation verification guidance and documents the capabilities and limitations of the setup doctor command. |
Sequence Diagram
sequenceDiagram
participant Build as Image build jobs
participant Publish as Manifest publisher
participant Resolve as Subject resolver
participant Attest as Attestation matrix
participant GHCR as GHCR
Build->>GHCR: Push immutable per-architecture images
Publish->>GHCR: Publish SHA, version, and eligible latest indexes
Resolve->>GHCR: Resolve published tags to distinct digests
Resolve->>Attest: Emit subject matrix
Attest->>GHCR: Push signatures and provenance
Attest->>GHCR: Push SBOMs for architecture subjects
Attest-->>Publish: Gate GitHub release on success
Reviews (11): Last reviewed commit: "fix: correct the audit pass's own errors..." | Re-trigger Greptile
…ory the chart's images Enterprise security reviews ask for artifacts Sim did not publish: a signature proving who built an image, SLSA provenance describing how, and an SBOM listing what is inside. A new attest-subjects job resolves the tags this run actually published to their digests, and attest-images signs each with cosign keyless signing and attaches provenance and, for single-architecture subjects, a Syft SBOM. All of it is pushed to the registry as referrers, so a mirrored registry carries the attestations with the image. Resolving tags rather than assuming them is what makes the coverage complete. `imagetools create` writes an index even from one manifest, so `:version-amd64` has a different digest from the `:sha-amd64` manifest it wraps, and which tags exist depends on whether the run is a release and whether the latest guard held. A moving alias is taken only when it resolves to this run's index, and that read is retried so registry lag is not mistaken for a deliberate skip. The SBOM is generated from the plain `<sha>-<arch>` manifest rather than the subject digest. Syft resolves an index against the runner's platform, so an arm64-only index fails outright on an amd64 runner — and the plain manifest is identical content, shared by both subjects for that architecture. helm/sim/images.yaml is the set of images a complete install pulls, generated from the rendered chart because an image can be referenced from a template that no values key names. Each entry carries the reference to pull from and the path the chart resolves to under a mirror; those differ, and the second is derived from a second render rather than described as a rule that could drift. The NVIDIA device plugin was hardcoded, so an air-gapped cluster could not redirect it. It now takes its image from `ollama.gpu.devicePlugin.image` through the shared helper, and the chart minor is bumped for the new key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
…rrect the existing pages An audit of 27 pages checked every concrete claim against the code it describes. It found claims that made a documented path fail outright, and claims that understated what ships. Errors that broke a documented path: - `openssl rand -hex 32` prints 64 hex characters, not the "32 hex chars" five entries claimed, and only two of those secrets are length-validated at all. - `API_ENCRYPTION_KEY` was missing from the whole Kubernetes path. It is optional, and unset means API keys are stored in plain text behind one warning. - Bucket fallback behaviour was wrong for all three object-storage providers, in both directions. - The remote-sandbox table omitted the immutable template and snapshot refs, so the documented configuration could not work. - The GKE snippet set a values key no template renders. - Retention deletion is scheduled by neither shipped deployment, so a deployment that followed the docs deleted nothing. - Every "Settings → Enterprise/Security → X" path named nav groups that do not exist, across eleven pages. Claims that understated what ships: - The outbound-proxy section said no platform traffic can be proxied. The server runs on Bun, whose fetch honors the proxy variables, so provider and Resend traffic already proxies; the guarded egress, SMTP, S3 and OTLP do not. Replaced with a per-path table and the NO_PROXY guidance internal services need. - Telemetry is off by default on Helm, not on. - The CronJob count was 18 in two places; there are 22. - The permission-group table was stale throughout, named a deployment key that does not exist, and inverted the checkbox semantic. Adds two pages. Sandboxes carries the base-image runbook that was buried in an enterprise enablement page. Reference architectures answers what an operator provisions versus what the chart provisions, per cloud — the question behind "do you have Terraform or CloudFormation templates?". Also documents around 25 environment variables nothing described, the migration advisory lock and its operational knobs, and `sim-setup doctor`. Removes redundancy where a page restated itself: FAQ blocks repeating their own body, and four drifting copies of the same tables. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
The mirror override in the inventory header double-prefixed. It told operators to override the device plugin with `<your-registry>/nvidia/k8s-device-plugin` while global rewriting stayed on, so `sim.image` prepended the registry a second time and the DaemonSet landed in ImagePullBackOff. Verified by rendering both forms; the override has to be the bare repository. The plugin's DaemonSet also now carries `global.imagePullSecrets`, which it was the only workload to omit — an authenticated mirror could not pull it. The resolver skipped `latest*` whenever the alias did not resolve to this run's index, which conflated "the guard withheld it" with "the read was stale". create-ghcr-manifests now publishes its guard decision and the resolver fails instead of skipping when the guard says latest was published this run. Docs corrections, each verified against source: - The loopback carve-out does lift plain HTTP and the private-address block; it leaves only the denied-port list in force. `egress.ts` skips the port check for allowlist vouching alone, which is what makes `http://localhost:5432` stay refused. - Not every guarded response is capped — the pinned fetch used by provider SDKs sets no limit. - A partially configured storage backend is skipped when a later one is complete; it fails only when none is. - GCE Ingress can serve a provisioned TLS Secret, so cert-manager is an alternative to a ManagedCertificate rather than requiring a different controller. - Workload Identity needs roles/iam.serviceAccountTokenCreator to sign GCS URLs. - NEXT_PUBLIC_BRAND_BACKGROUND_COLOR is a contrast flag, not a background. - NEXT_PUBLIC_SANDBOXES_ENABLED is a browser gate, not provider readiness. - An incomplete certificate chain is not fixed by NODE_EXTRA_CA_CERTS; the endpoint has to serve the intermediate. - Script migrations run once — the eight retries are for SQL lock timeouts. - NO_PROXY belongs in the application environment, not a shell assignment, and the GKE annotate command no longer hard-codes a release name and namespace. - Pi's own image, the Microsoft login callback, the custom-block identity split, the whitelabeling surfaces that ignore configured URLs, and sim-setup's Compose-only scope are all now stated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
546aac6 to
4f9c34f
Compare
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
Two chart defects, both the same shape as the uploads-path bug this branch already documents: a path that looks right but sits outside where the app actually serves from. - `branding.mountPath` defaulted to `/app/public/branding`, while the standalone build places the public directory at `/app/apps/sim/public`. Every branded asset the docs tell operators to reference would have 404'd. Corrected in the chart and the example. - The device plugin was the only `sim.image` call site not passing `chartAppVersion`, so `tag: ""` — a natural way to ask for the chart default — errored instead of falling back. Docs, each verified against source: - Local disk is selected unless a backend's *activating* variables are set; credentials alone activate nothing. - Six backoff retries follow the initial failure, so up to seven attempts. - The migrations init container reads only the database Secret, and the chart's Secret templates carry just the password — so `MIGRATION_DATABASE_URL` needs an operator-supplied Secret, not the generated one. - The Ollama stack's missing Redis also breaks CLI authentication outright, while the idempotency store and progress markers fall back to Postgres. - `sim-setup reset` leaves the PostgreSQL PVC on Kubernetes. - The copilot ingress route renders only when `copilot.enabled` is set. - The Postgres exposure applies to whichever Compose file started the install. - The GKE annotate command resolves the Service name from the release rather than assuming it. - `API_ENCRYPTION_KEY` warns on every use, not once, and the whitelabeling coverage claim now describes the two surfaces that only partially honour it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
Two of these were fixes from earlier rounds that never landed — the edits were in batches that aborted partway, and I reported them applied without re-verifying. The LM Studio recreate command still hard-coded the production Compose file, and the proxy table still claimed "four exceptions" above a row listing fifteen providers. The rest: - The NO_PROXY example omitted `ollama`, so on the Ollama stack local model requests would go to the external proxy. - The `lock_timeout` sentence had its trade-off backwards: it trades a table-wide stall for a failed migration, not the reverse. - `NEXT_PUBLIC_CUSTOM_CSS_URL` has to be same-origin or CSP-allowlisted; `style-src` blocks a stylesheet from another origin. - Any of a storage backend's keys activates it — a dedicated bucket, `S3_ENDPOINT`, an Azure credential — so a half-configured backend fails at startup rather than falling back to local disk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
- The LM Studio recreate snippet now shows both Compose assignments instead of describing the alternative in a comment. - `sim-setup logs` on a source checkout prints where the logs are rather than following a stream; the table said it follows. - The GKE BackendConfig annotation targeted `<release>-realtime`, but the chart names the Service `<fullname>-realtime` — and the fullname is `<release>-sim` unless the release name already contains `sim`. It now selects by the release and component labels, which holds for either. - The Helm `NO_PROXY` example listed only the `.svc` suffixes, which match none of the bare Service names the chart wires into `SOCKET_SERVER_URL`, `PII_URL`, and `OLLAMA_URL`. Those requests would still have gone to the proxy. - `sim-setup` detects only the production and local Compose files, so an Ollama-stack install is invisible to it. The upgrade section now says so and gives the direct commands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
There was a problem hiding this comment.
All reported issues were addressed across 48 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Fix all with cubic | Re-trigger cubic
- "`statement_timeout` is 0, so a long migration is never cut off" was absolute, and `0076_damp_vector.sql` disproves it — it sets 180s locally. It is the session default now, with the override named. - The migration-log command hard-coded the production Compose file, so a source-built or Ollama install would have inspected the wrong stack. - The Compose rollback applied `SIM_VERSION` to every install, but the local and Ollama stacks build from source and ignore it. Those now get the checkout-and-rebuild path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 issue found across 48 files
Confidence score: 3/5
- In
apps/docs/content/docs/platform/self-hosting/upgrades.mdx, the Ollama upgrade commands can leave the app, realtime server, and migrator on old images because source-built Sim services are not rebuilt, resulting in a partially upgraded installation; update the checkout to the target revision and run the required rebuild commands.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/docs/content/docs/platform/self-hosting/upgrades.mdx">
<violation number="1" location="apps/docs/content/docs/platform/self-hosting/upgrades.mdx:206">
P2: For an Ollama install, these commands do not rebuild the source-built Sim services, so the upgrade leaves the app, realtime server, and migrator on their old images. Update the checkout to the desired revision and run `up -d --build` with the selected profile.</violation>
</file>
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Fix all with cubic | Re-trigger cubic
`docker-compose.ollama.yml` builds the app, realtime server, and migrator from source, which I had missed: - The Ollama upgrade block ran `pull` then `up -d`. That refreshes only the Ollama and Postgres images and leaves every Sim service on its old build. It is a checkout plus `build --pull` now. - The source rollback showed only the local file and no profile; the Ollama stack needs both. - Dropped the Ollama mention from the `COMPOSE_FILE` comment above, since that stack is covered by its own profile-aware commands rather than the CLI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 issue found across 48 files
Confidence score: 3/5
- In
apps/docs/content/docs/platform/self-hosting/upgrades.mdx, the rollback command selects the GPU Ollama service after a CPU-profile installation, which can fail on hosts without NVIDIA devices; update it to use a matching profile variable andollama-cpu.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/docs/content/docs/platform/self-hosting/upgrades.mdx">
<violation number="1" location="apps/docs/content/docs/platform/self-hosting/upgrades.mdx:287">
P2: When an Ollama stack was installed with the CPU profile, this rollback command activates the GPU service instead of `ollama-cpu` and can fail on hosts without NVIDIA devices. Use a profile variable matching the installation, as in the upgrade example.</violation>
</file>
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Fix all with cubic | Re-trigger cubic
Eight review rounds kept finding defects because I was patching what the last round flagged and writing new prose in the process, which nothing then checked. This is a full pass over the Compose files, the sim-setup CLI, and the chart. The mirror inventory (Greptile's blocker): the bare device-plugin override changes where the chart pulls from, to `<registry>/nvidia/k8s-device-plugin`, while the inventory tells operators to mirror to `<registry>/nvcr.io/nvidia/...`. Following both instructions produced an unpopulated path. Verified by rendering the chart both ways; the header now names the override's destination. Compose: - Nine `docker compose` commands across troubleshooting, verify, and background-jobs had no `-f`. The repo ships no default compose file, so every one of them failed with "no configuration file provided". - `simstudio` declares `env_file: .env` in all three files, so the LM Studio fix does not need a hand-edited `environment:` block — and hand-editing a managed Compose file makes `sim-setup update` refuse. - `OLLAMA_URL` as a one-shot shell prefix reverts to the localhost default on the next `up -d`, pointing the container at itself. It belongs in `.env`. - `pg_dump` through `exec` without `-T` corrupts the dump; the restore beside it already used `-T`. - Live cross-process status is a Redis consequence, not a cron one. - The local Compose file was missing from three "the file that started your install" lists and from the exposed-Postgres warning. - No shipped Compose file defines a `pii` service. - The rollback said "edit .env" but appended a second SIM_VERSION line. sim-setup CLI — the table stated Compose behavior as universal: - `down` runs `helm uninstall` on Kubernetes and removes only Postgres and Redis on a source checkout; both prompt first. - `reset` archives four env files on anything but Compose, including when no install is detected. - `--no-open` was described backwards: the link always prints, the flag skips the download prompt. - `start`/`restart` print port-forward hints on Kubernetes, not equivalents. - `--dir` is a global option and scopes the lifecycle commands. - `--quick` only preselects the setup style; `add` and `desktop` have refusals worth knowing; the update hand-edit refusal is narrower than stated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
The rollback hard-coded `--profile gpu` while the upgrade example above it uses a variable, so a CPU install would have started the GPU service. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
The install command was the serious one: the cloud values files hardcode a placeholder domain in six places, and the documented command overrode only `NEXT_PUBLIC_APP_URL`. Following the page literally left sign-in pointed at the placeholder, realtime rejecting every socket upgrade, and the Ingress serving the wrong host — the exact failure the page warns about further down. - The PII recipe's `INTERNAL_API_BASE_URL` self-call is blocked by the NetworkPolicy the same page recommends enabling: the app policy permits Postgres, Redis, realtime, Ollama, PII, telemetry, DNS, and 443, but has no app-to-app rule. Redaction would fail closed. - The GPU device-plugin DaemonSet is not `restricted`-compliant — it sets only two of the four controls and mounts three `hostPath` volumes — so enforcing the label namespace-wide breaks a GPU install. - Under External Secrets the chart writes nothing from `app.env`; an unmapped key fails the render. The page said the two modes behave alike. - The CronJob pods have no NetworkPolicy either. - `app.env.DATABASE_URL` is silently discarded: the chart computes it and inlines it on the container, so the PgBouncer advice pointed at a key that does nothing. - The PDB also renders from `autoscaling.minReplicas`, the reference topology could not render without an `externalDatabase.password`, `copilot`'s security-context keys nest one level deeper, and `NO_PROXY` needs the app's own Service name. - Untangled a garbled `externalSecrets.apiVersion` comment that contradicted its own default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
- White-labeling, session policies, and the inbox are already on wherever billing is disabled, which is every self-hosted deployment. The docs told operators to enable white-labeling with a flag it does not need, and listed all thirteen features as symmetric opt-ins. - `NEXT_PUBLIC_CUSTOM_CSS_URL` must be same-origin. The CSP's `style-src` is a hardcoded literal that takes no configured hosts, so both the "point them at absolute URLs" advice and my own earlier "or a host you have added to style-src" were wrong — there is no such remedy short of editing source. - Client variables are read at boot from the container environment, not baked at build. The troubleshooting note invited a pointless image rebuild. - Script migrations run with `lock_timeout = 0` and `statement_timeout = 0`, so one blocked on an app-held lock hangs forever rather than failing with `55P03`. The table described only the SQL phase. - With no mail provider the mailer logs recipient, subject, and sender at `info` and reports success — never the body, and nothing at all at the production default of `ERROR`. - A malformed storage backend stops startup; it does not fall through to a later fully configured one. - `EMAIL_PASSWORD_SIGNUP_ENABLED=false` disables sign-in and OTP too, and `SIGNUP_MX_VALIDATION_ENABLED` covers only email signup. - The Pi template accepts an immutable ref; the page forbade one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 48 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
Four of these are defects I introduced in the audit commits by applying suggested replacement text without verifying the replacement the way I verified the finding: - `postgresql.enabled: false` alone renders an empty `DATABASE_URL`; `externalDatabase.enabled: true` is required too. I dropped it. - The external-database password check is skipped under `existingSecret` and External Secrets, so "the chart refuses to render" was false. Under ESO the mapping is what matters. - `LOG_LEVEL` is an uppercase-only enum, so the `info` I told operators to set would fail validation. - `add cache` writes `REDIS_URL`, which `realtime` reads too, so recreating only the app container leaves Socket.IO pointed at the old Redis. Six the audits missed: - `PII_URL` used `<release>-pii`, the same fullname bug already fixed for the realtime Service. A manual value is also ignored when `pii.enabled`. - External Redis does not need `redis.enabled: false` — a non-empty `app.env.REDIS_URL` overrides the bundled URL and suppresses its Deployment. - `sim-setup doctor` was recommended to Ollama-stack users, which it cannot detect. - The CA-bundle snippet said "in either case" directly after explaining that an incomplete chain must be repaired server-side. - `update` refuses only the `dev` mode; a source checkout on `docker-compose.local.yml` is a Compose install and does update. - Doctor cannot tell whether migrations are current on a standalone install — the published package ships no migration journal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
Replaces #7432 and #7434, combined into one PR against current staging.
Summary
helm/sim/images.yamllists every image a complete install pulls, generated from the rendered chart, with both the reference to pull from and the path the chart resolves to under a mirror.Why the subject resolution is not a static list
imagetools createwrites an index even from one manifest, so:version-amd64has a different digest from the:sha-amd64manifest it wraps, and which tags exist depends on whether the run is a release and whether the latest guard held. A moving alias is attested only when it resolves to this run's index, and that read is retried so registry lag is not mistaken for a deliberate skip. The SBOM is generated from the plain per-architecture manifest, because Syft resolves an index against the runner's platform and an arm64-only index fails outright on an amd64 runner.Notable doc corrections
openssl rand -hex 32prints 64 characters, not 32.API_ENCRYPTION_KEYwas absent from the Kubernetes path, and unset means API keys are stored in plain text. Bucket fallback was wrong for all three object-storage providers. The proxy section said no platform traffic can be proxied — the server runs on Bun, whose fetch honors the proxy variables. Telemetry is off by default on Helm, not on, andNEXT_TELEMETRY_DISABLEDdoes not gate the browser-event relay. The CronJob count was 18; there are 22. Every "Settings → Enterprise/Security → X" path named nav groups that do not exist. SSO callback URLs hardcoded Sim Cloud's domain on a page with a self-hosted section.Type of Change
Testing
bun run check:audits(45 audits),bun run lint,bun run docs-manifest:check,bun run images:check,actionlint,helm lint,helm templateagainst both CI values files, and the scripts suite (275 tests) all pass. The generated mirror paths were verified against a real mirrored render; the chart's default render is byte-identical to staging apart from the version label.Checklist
🤖 Generated with Claude Code
https://claude.ai/code/session_015BwsJTEQRzWJaY4BRCkPZt