Migrate bundle state filer read off workspace-files to /workspace/export - #6219
Migrate bundle state filer read off workspace-files to /workspace/export#6219Sankalp-Mittal wants to merge 7 commits into
Conversation
Integration test reportCommit: 4612ac8
8 interesting tests: 4 SKIP, 3 flaky, 1 RECOVERED
Top 12 slowest tests (at least 2 minutes):
|
Sankalp-Mittal
changed the title
2a99228 to
af88c8f
Compare
291eb0a to
b11f1be
Compare
| // Read via the raw apiClient.Do (not the SDK's Workspace.Download) so | ||
| // auth.WorkspaceIDHeaders can drop the CLI-only "none" workspace-id sentinel | ||
| // that Download would send literally. See PR #6149 for the write-path equivalent. | ||
| urlPath := "/api/2.0/workspace/export?path=" + url.QueryEscape(absPath) + "&direct_download=true" |
There was a problem hiding this comment.
Why can't we use the SDK again here? I don't follow the point about auth.WorkspaceIDHeaders. Can we make this comment clearer?
There was a problem hiding this comment.
Is it because the streaming endpoint is not accessible via the SDK?
There was a problem hiding this comment.
no the issue is only when the during auth login the --skip-workspace flag is passed, this causes WorkspaceID = "none" to be passed (which I could handle separately, but I wanted to keep the code change as small as possible)
There was a problem hiding this comment.
Can check out libs/auth/arguments.go:8 for definition of this sentinel
There was a problem hiding this comment.
Also modified the comment
shreyas-goenka
left a comment
shreyas-goenka
left a comment
There was a problem hiding this comment.
Sorry, did not mean to approve the PR. The current PR is good but please lets clarify why SDK is not usable here.
301ea4b to
284f5af
Compare
…tion filer.go now reads state via /api/2.0/workspace/export (a2946f2b7); regenerate the acceptance goldens to match. Note: state/basic and force_pull_commands filter captured requests on the old workspace-files path, so their goldens are now empty and their state-read assertions need the filters repointed at /workspace/export (follow-up). Co-authored-by: Isaac
e4d4b68 to
d110a58
Compare
shreyas-goenka
left a comment
shreyas-goenka
left a comment
There was a problem hiding this comment.
Did you confirmed that this works for files bigger than 10MB? PR looks good to me other than why the API requests are vanishing in those tests.
Yes I did, also fixed the API requests vanishing issue |
| "path": "/Workspace/Users/[USERNAME]/.bundle/state/default/state/deploy.lock" | ||
| } | ||
| } | ||
| { |
There was a problem hiding this comment.
We get these extra reads in the output because reading of locks and state files already used this endpoint, but then the grep was made on the old api, when we shift the grep to the new endpoint these extra reads also creep up
There was a problem hiding this comment.
Since the reading of the locks is not what we are testing I am excluding this from the output grep
3 participants
bundle: read deploy state via /workspace/export instead of deprecated workspace-files API
Summary
Switch the DABs state filer read (
bundle/deploy/filer.go,stateFiler.Read) fromGET /api/2.0/workspace-files/{path}to the SDK'sWorkspace.Download(ctx, absPath), which issuesGET /api/2.0/workspace/export?path=<path>&direct_download=true. This reads the bundle state files(
terraform.tfstate,resources.json) from the workspace.Why
The
workspace-filesendpoint is deprecated and is mapped to no API scope. Under context-basedingress (CBI) with fine-grained scoped tokens it was only reachable via the
all_apismaster scope,so
bundle deployreturned 403s for customers on least-privilege scoped tokens./workspace/exportis properly scoped, so state reads now work under least-privilege auth.
Implementation
Reads go through the SDK's
Workspace.Downloadhelper, so there is no separate API client, no manualauth.WorkspaceIDHeadersrouting headers (the SDK sets these), and no intermediate buffer —Downloadreturns the streaming body directly.direct_download=trueis load-bearing: plain/workspace/exportreturns base64 JSON capped at 10 MB,while the streaming variant handles files up to 500 MB, so large state files stream through fine.
Testing
GET /api/2.0/workspace/exportwithdirect_download=true(user_agent,statesuites).end-to-end.
This pull request and its description were written by Isaac.