Skip to content

feat: roll HyperDX pods on config changes - #273

Open
bsosnader wants to merge 1 commit into
ClickHouse:mainfrom
bsosnader:brsosnad/investigate-values-only-checksums
Open

feat: roll HyperDX pods on config changes#273
bsosnader wants to merge 1 commit into
ClickHouse:mainfrom
bsosnader:brsosnad/investigate-values-only-checksums

Conversation

@bsosnader

Copy link
Copy Markdown
Contributor

Summary

  • add chart-owned pod-template checksums for the rendered HyperDX ConfigMap and optional Secret
  • preserve existing user annotation merge precedence while reserving scoped checksum keys
  • document automatic rollout behavior and external Secret limitations
  • add focused Helm unit coverage for deterministic, content-sensitive checksums

Motivation

HyperDX consumes clickstack-config and clickstack-secret through envFrom. Kubernetes resolves those environment variables when a pod starts, so changing only the ConfigMap or Secret during helm upgrade leaves existing HyperDX pods with stale startup configuration unless the Deployment pod template also changes.

This follows Helm's documented include ... | sha256sum rollout pattern. The ConfigMap and Secret manifests now share canonical named renderers with the Deployment checksums, ensuring the hashed content cannot drift from the resources Helm applies.

No new values API is needed: these are chart-managed resources with deterministic rollout behavior. Arbitrary template paths or tpl evaluation in user values would be less safe and would not improve this owned-resource case.

Backward compatibility

Existing hyperdx.deployment.annotations and hyperdx.deployment.podAnnotations values remain merged with the same precedence. The generated checksum/clickstack-config and checksum/clickstack-secret keys are chart-reserved and override caller values so stale hashes cannot disable rollouts.

checksum/clickstack-secret is omitted when hyperdx.secrets: null, matching Secret rendering and envFrom behavior. Externally managed Secret changes still require an explicit rollout because Helm cannot hash resources it does not render.

The first upgrade containing this change intentionally performs a one-time HyperDX rollout because the generated annotations are added to the pod template.

Tests

  • helm unittest -f tests/hyperdx-rollout-checksums_test.yaml charts/clickstack
  • helm unittest charts/clickstack — 31 suites, 257 tests
  • helm lint --strict charts/clickstack
  • default chart render
  • ALB ingress, API-only, and OTEL custom-config example renders
  • git diff --check

@bsosnader
bsosnader requested a review from a team as a code owner August 31, 2026 18:08
@changeset-bot

changeset-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: fa92f63

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
helm-charts Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the external Opened by an external contributor label Aug 31, 2026
@CLAassistant

CLAassistant commented Aug 31, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@bsosnader
bsosnader force-pushed the brsosnad/investigate-values-only-checksums branch from f7cf6b3 to fa92f63 Compare August 31, 2026 18:13
@wrn14897

wrn14897 commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Please fix the P1 checksum-test failure before merging. The CI unit-test job reports 5 failed, 252 passed, with all five failures in hyperdx-rollout-checksums_test.yaml.

The hard-coded expected hashes differ from CI's rendered hashes (e.g. baseline config expects 0587572216112d61fd7d298001d71ed852194038be79415116a447e260753058 but renders b76d0e7c9c3bc42636d77be4768074ac0627ebf8af4c5f75caa5ce697e7e04fe). Please investigate the mismatch using CI's pinned helm-unittest v1.0.3, make the fixtures reproducible by pinning the relevant rendering context, and rerun the full suite. Please also update the PR's test summary to reflect the verified result.

Rest of the change looks good: annotation merge precedence is preserved, the Secret checksum is correctly guarded on hyperdx.secrets: null, and both integration suites plus all three example renders pass. Two non-blocking notes:

  • The checksums include chart labels, so a chart-version-only upgrade will also roll HyperDX even when the app image and env config are unchanged. Worth confirming that is intended.
  • The deep-review job failure is unrelated to this change — actions/checkout@v6 refuses to check out fork code under pull_request_target.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external Opened by an external contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants