Feat/web UI sboms - #1620
Draft
dervoeti wants to merge 8 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #1600, which covered the frontends that ship an npm build. This covers the remaining JavaScript in our products: the HBase, HDFS, Spark and Trino web UIs. None of it appeared in any SBOM, because the CycloneDX Maven plugin only sees Java dependencies.
Three different mechanisms are needed, because the JavaScript arrives three different ways.
HBase unpacks its web UI assets from webjars during the build, the CycloneDX Maven plugin does not pick them up.
hbase_webapps_deps.pyderives the coordinates from the poms and writes apackage.json, which cdxgen then turns into the SBOM. A version bump or an added webjar is picked up automatically, and the script fails the build if the pom layout changes or a version cannot be resolved.Hadoop and Spark vendor pre-built, usually minified files directly into their source trees. Those carry no package manifest and no lockfile, I did not find a tool that would identify a minified bundle reliably. They are therefore recorded by hand in a manifest per product version, which
shared/sbom/vendored_js.pyturns into a CycloneDX SBOM. I chose JSON manifests over writing the SBOM directly so we don't have to update all of them if we for example switch from CycloneDX 1.5 to 1.7. These are independent records of what we detected manually, not SBOMs.Every entry pins the SHA-256 of the file it describes, so a product upgrade that changes, adds or removes a file fails the build instead of silently shipping a wrong SBOM. Recording a version is a one-time cost per file.
shared/sbom/identify_js.pyis a helper script for creating these files. Itsinspectsubcommand lists every.jsfile below the given directories with its SHA-256 and any version string it states, andidentifyhashes every file of every published release of a candidate npm package to find a byte-identical match, which is proof rather than inference. File names can lie, for example Hadoop'sd3-v4.1.1.min.jsreally is d3 4.1.0.Trino gets three SBOMs, because its web UI is three separate things: the npm project behind the current UI, the npm project behind the preview UI, and the pre-built libraries under
webapp/vendor. The two npm projects are handled by cdxgen, the pre-built binaries by the same tooling used for Hadoop / Spark.The helper scripts and the vendored manifests were generated by Claude, I reviewed the helper scripts and iterated with Claude over the general concepts and we landed at this solution. I tested some builds and inspected the SBOMs, looks good.
Definition of Done Checklist