Add Cluster Policy support to bundle resources - #6225
Merged
Conversation
Collaborator
Integration test reportCommit: 3cc0cb9
11 interesting tests: 7 flaky, 3 SKIP, 1 RECOVERED
Top 24 slowest tests (at least 2 minutes):
|
Sankalp-Mittal
commented
Aug 11, 2026
| "vector_search_indexes", | ||
| "volumes", | ||
| }, | ||
| require.Equal( |
Collaborator
Author
There was a problem hiding this comment.
This change is due to the linter
Sankalp-Mittal
force-pushed
the
sankalp-mittal/cluster-policy-resource
branch
3 times, most recently
from
August 13, 2026 13:40
08272eb to
a93ad20
Compare
Sankalp-Mittal
marked this pull request as ready for review
August 13, 2026 13:58
| case dyn.KindInvalid, dyn.KindNil, dyn.KindString: | ||
| // KindInvalid means definition is absent; leave it for backend validation. | ||
| return v, nil | ||
| case dyn.KindMap, dyn.KindSequence: |
Contributor
There was a problem hiding this comment.
why do we allow KindSequence? yes, it's valid JSON but it's not valid policy language (or is it?)
Collaborator
Author
There was a problem hiding this comment.
This was also present in Genie spaces in configure_genie_space_serialized_space.go:68 so I copied it here as well. Though I think it should be removed from both places.
|
|
||
| # catalogs: no api field behaviors | ||
|
|
||
| # cluster_policies: no api field behaviors |
Contributor
There was a problem hiding this comment.
would be great to add hashed_in_state behaviour for the policy field after #6105 lands & cover with acceptance test for a large policy doc
Collaborator
Author
There was a problem hiding this comment.
That can be done in a follow up
| @@ -0,0 +1,6 @@ | |||
| Cloud = false | |||
Contributor
There was a problem hiding this comment.
please enable cloud for some of the tests once you have run them and verified they pass
Contributor
There was a problem hiding this comment.
let's do Cloud = true for all by default and only add Cloud = false on a per-test basis (if for some reason [state that reason as comment] Cloud isn't suitable for them)
The cluster_policies resource was added without regenerating derived files, failing validate-generated and the refschema acceptance test. Co-authored-by: Isaac
Adding cluster_policies with a workspace URL pattern extended the list of openable resource types, but the workspace_open command tests hardcoded the old list. Add cluster_policies to the expected completion, help text, and unknown-type error assertions. Co-authored-by: Isaac
The workspace open command's supported-resource-type list now includes cluster_policies; regenerate the golden output. Co-authored-by: Isaac
TestInvariantConfigsCoverage requires every resource type to have an invariant config. Add a cluster_policy config and register it in the invariant matrix. Exclude it from the migrate suite (terraform-seeded; the resource is direct-only) and continue_293 (unsupported on the old CLI). Regenerate the affected out.test.toml snapshots. Co-authored-by: Isaac
Restructure the single cluster_policies acceptance test into a group:
- move the existing test unchanged into basic/
- add job_ref/: a job task referencing the policy via
${resources.cluster_policies.pol.id}, asserting the direct engine
orders policy create before job create (and job delete before policy
delete on destroy) and resolves the policy id into the job body
- add definition_multiline/: a block-scalar JSON definition, asserting
it is preserved as a newline-escaped string end to end
No production code change.
Co-authored-by: Isaac
The cluster policy `definition` was a plain JSON string. Add a top-level `Definition any` field that shadows the embedded compute.CreatePolicy string so the definition can also be written as native YAML. ConfigureClusterPolicyDefinition normalizes an inline map/sequence to a JSON string at the dyn layer (same approach as genie serialized_space), avoiding int/float structdiff drift; PrepareState copies the normalized string into state. A string definition passes through unchanged. Co-authored-by: Isaac
- unit test for ConfigureClusterPolicyDefinition (string passthrough, map/sequence -> JSON string, invalid-kind diagnostic, absent field) - unit test for ResourceClusterPolicy.PrepareState copying the normalized string into state - acceptance test definition_yaml/: a native YAML mapping serializes to the same compact JSON string the API receives as the basic test Co-authored-by: Isaac
Instead of inline JSON everywhere, spread the three authoring forms across the non-targeted tests for incidental coverage: job_ref and cross_bundle_lookup use a multiline JSON block scalar, direct-only uses native YAML, cross_bundle_id keeps inline JSON. basic and the two targeted definition tests are unchanged. Co-authored-by: Isaac
Sankalp-Mittal
force-pushed
the
sankalp-mittal/cluster-policy-resource
branch
from
August 17, 2026 12:28
a96139d to
aef7b32
Compare
Sankalp-Mittal
commented
Aug 17, 2026
Sankalp-Mittal
left a comment
Sankalp-Mittal
left a comment
Collaborator
Author
There was a problem hiding this comment.
replied to comments
Sankalp-Mittal
commented
Aug 19, 2026
| # DataSecurityModeDiffSuppressFunc: suppress when old != "" && new == "" | ||
| #- field: data_security_mode | ||
|
|
||
| cluster_policies: |
Collaborator
Author
There was a problem hiding this comment.
This is added because if policy_definition_overrides are used then the backend fills in the definition fields and false drift can be detected
| @@ -0,0 +1,6 @@ | |||
| Cloud = false | |||
Contributor
There was a problem hiding this comment.
let's do Cloud = true for all by default and only add Cloud = false on a per-test basis (if for some reason [state that reason as comment] Cloud isn't suitable for them)
Collaborator
Author
Done |
janniklasrose
approved these changes
Aug 24, 2026
Cluster policy names are workspace-global and the CI workspace is shared
across concurrent runs, so fixed policy names collided ("already exists").
Author names and the bundle root_path with $UNIQUE_NAME via databricks.yml.tmpl
+ envsubst so each run is isolated.
Co-authored-by: Isaac
Co-authored-by: Isaac
Collaborator
Integration test reportCommit: ce0ddfd
18 interesting tests: 14 flaky, 2 RECOVERED, 2 SKIP
Top 50 slowest tests (at least 2 minutes):
|
jariaspdev-lab
pushed a commit
to jariaspdev-lab/cli
that referenced
this pull request
Aug 25, 2026
…6304) ## Summary Stacked on databricks#6225 (cluster policy resource). Adds the per-resource `permissions` field to `cluster_policies` so a policy can grant access via the direct engine's generic permissions applier. Cluster policies support a **single** ACL level — `CAN_USE` (SDK `compute.ClusterPolicyPermissionLevel`), meaning "may create clusters governed by this policy." There is no `CAN_MANAGE`/`CAN_VIEW`/`CAN_RUN`; policy administration is an admin-only function. ```yaml resources: cluster_policies: my_policy: name: my-policy permissions: - level: CAN_USE group_name: users ``` ## Changes - **Resource wiring**: declare `ClusterPolicyPermission`, add the `Permissions` field, register `cluster_policies.permissions` in the direct engine, and map the permissions-API object type to `/cluster-policies/`. - **`FixPermissions`**: add `cluster_policies` to `ignoredResources` so the deploying user is **not** auto-injected as `CAN_MANAGE`/`IS_OWNER` — cluster policies can't accept it, and the real backend would reject the deploy (same treatment as `secret_scopes`). - **Testserver**: model the real API — a non-`CAN_USE` level on a cluster policy returns `400`, so local and cloud runs agree. - **Generated files**: regenerated schema, refschema field list, and validation (`level` enum = `CAN_USE` only, `level` required). ## Design note `cluster_policies` is intentionally **not** added to the top-level `levelsMap`; it stays in `unsupportedResources`. The bundle-wide `permissions:` block is expressed in `CAN_MANAGE`/`CAN_VIEW`/`CAN_RUN` (and auto-grants the deployer `CAN_MANAGE`), none of which a cluster policy can accept. Supporting only the explicit per-resource form avoids silently dropping those grants. Two current-user injection paths had to be handled for this: the top-level `levelsMap` and the `FixPermissions` mutator. ## Testing Acceptance tests under `acceptance/bundle/resources/cluster_policies/permissions/`: - `basic/` — deploy applies the `CAN_USE` grants, redeploy updates the ACL; requests asserted with `print_requests.py`. - `out_of_band_change/` — an out-of-band ACL change is detected by `bundle plan` and reconciled on redeploy. - `levels/` (`Cloud=true`) — `CAN_USE` deploys successfully; a direct `CAN_MANAGE` `set-permissions` call is rejected (asserted with `musterr`; the backend error differs between the fake and cloud, so it is routed to a `LOG` file rather than pinned). Plus the `cluster_policies.permissions` CRUD roundtrip fixture in `dresources/all_test.go`. Unit + acceptance suites pass locally. This pull request and its description were written by Isaac.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add support for Cluster Policies resource in DABs. These can be defined as multiline json, inline json and yaml. Cluster Policies can be defined either as a
definitionorpolicy_family_definition_overridesChanges
Add support for a new
cluster_policiesresource type in DABs. Cluster policies are admin-defined guardrails that constrain how clusters can be configured (allowed node types, fixed autotermination, required tags, cost limits). This lets users manage them declaratively alongside their other bundle resources. This includes:resources.ClusterPolicyconfig type, wrappingcompute.CreatePolicy.ResourceClusterPolicy) covering create/read/update/delete plusPrepareState/RemapState.all.go), dev-mode name prefixing (presets), and the workspace URL pattern.definitionas native inline YAML, in addition to a JSON string or block scalar (see "Key points" below)./api/2.0/policies/clusters/*endpoints.Cluster policies are only supported in direct deployment mode — they have no Terraform provider mapping here, so they are excluded from Terraform-mode conversion.
Why
Cluster policies were previously only referenceable from bundles via a variable
lookup(resolving an existing policy name to itspolicy_id). Users want to define and own the policies themselves as code, instead of provisioning them out-of-band through the UI.Example
The
definitioncan be an inline JSON string:…or native YAML, which is normalized to the same JSON string at deploy time:
Key points to note
Update uses
Edit, and it is a full replace. The SDK's update call isClusterPolicies.Edit(notUpdate);DoUpdatesends the whole desired policy along with the existingpolicy_id.Output-only fields are excluded from state.
policy_id,created_at_timestamp,creator_user_name, andis_defaultare returned on read but never written back, soRemapStatecopies only the fields shared withCreatePolicy.Policy-family definitions are treated as a backend default. When a policy is authored with
policy_family_idand nodefinition, the backend computes the fulldefinitionfrom the policy family and returns it on read. Config leavesdefinitionempty while remote holds a populated value, which would otherwise register as drift and re-issue anEditon every deploy.cluster_policies.definitionis therefore declared underbackend_defaultsinresources.yml: becausedefinitionis anomitemptyfield, an empty config value is diffed asnil, so theold == nil && new == nil && remote != nilrule skips the server-computed value. An explicitly authoreddefinitionkeepsold/newnon-nil, so it still participates in normal drift detection.Dev-mode name prefixing. The policy
nameis a user-facing display name (unique, 1–100 chars), not the API identity (policy_id), so it is prefixed in development mode to avoid collisions between developers without changing identity.Inline-YAML
definitionreplicates the Genieserialized_spacepattern. A top-levelDefinition anyfield shadows the embeddedcompute.CreatePolicy.Definition string, and a per-resourcedyn-layer mutator (ConfigureClusterPolicyDefinition, registered right afterConfigureGenieSpaceSerializedSpace()) normalizes it before deploy: a string passes through unchanged, an inline map/sequence is marshaled to a compact JSON string (json.Marshal, sorted keys), and any other kind is rejected with a diagnostic.PrepareStatethen copies the normalized string into state; the state type stayscompute.CreatePolicy, so there is no state-compatibility break.The conversion is done at the config layer on purpose: authoring structured content inline in YAML decodes small integers as Go
int, while the state JSON round-trip decodes them asfloat64, sostructdiffreports false drift on every plan. Normalizing to a canonical JSON string makes config-side and state-side hold an identical string — the same reason Genie does it. The dashboardserialized_dashboardmutator is deliberately not the model here: it only reads afile_pathinto an opaque string and does no YAML→JSON conversion.Tests
acceptance/bundle/resources/cluster_policies/:basic/— single-policy validate/summary/deploy plus create/edit/delete, pinned to the direct engine.job_ref/— a job task referencing the policy via${resources.cluster_policies.pol.id}; asserts the direct engine orders policycreatebefore jobcreate(and job delete before policy delete on destroy) and resolves the policy id into the job body.definition_multiline/— a block-scalar JSON definition preserved as a\n-escaped string end to end.definition_yaml/— a native YAML mapping that serializes to the same compact JSON string the API receives asbasic/.direct-only/— deploying in terraform mode fails with the actionable "only supported with direct deployment mode" error.missing_ref/— a job referencing an undeclaredcluster_policies.*resource fails to plan with a dependency-resolution error.cross_bundle_id/— one bundle creates a policy; a separate bundle consumes it by hardcoding the generatedpolicy_id(captured from the producer's state) in a job.cross_bundle_lookup/— a separate bundle resolves the policy created by the producer via a variablelookup: {cluster_policy: <name>}and uses the resolved id in a job.ConfigureClusterPolicyDefinition(string passthrough, map/sequence → JSON, invalid-kind diagnostic, absent field) and forResourceClusterPolicy.PrepareState.GET /api/2.0/policies/clusters/liststateful (returning created policies, seeded with the two legacy names the variable-lookup tests rely on) so the lookup-by-name flow is exercisable.cluster_policiescoverage to shared suites: bundle permissions, presets/target-mode prefixing, run-as, state-load round-tripping, and the Terraform lifecycle exclusion list.This pull request and its description were written by Isaac.