fix(sbom): stop sbom validation failing on duplicate component links

[reyreavman]

Summary

An image SBOM could carry two external references of the same type for one component, which the ISPRAS checker reports and werf turns into a validation failure. The enrichment step also accepted any CycloneDX reference type from the PURL resolver and dropped the archive digest the resolver reports for a source distribution, so a green build could produce an SBOM that fails validation afterwards. werf sbom validate now reaches the checker's source-distribution and leaf-only VCS checks, and refuses the one combination the checker does not honor.

What

Enrichment

• A component that already has an external reference of the resolved kind keeps it, and the resolved one is dropped: enrichment no longer adds a second reference of a type already present. Duplicate references of one type inherited from an older SBOM are not collapsed by this change — DedupBOM removes only identical reference objects, so two distinct vcs URLs on one component still survive and still trip the checker's contains 2 vcs links warning.
• An image built on top of another no longer accumulates a reference per merge: re-enriching an already enriched BOM is a no-op for components that already carry that type.
• os-pm packages keep the vcs link supplied by os-pm; the resolver no longer adds a second one next to it.
• BREAKING: a build fails when the PURL resolver answers with a reference kind other than vcs or source-distribution, with enrich: external reference kind "<kind>" is not allowed, expected "vcs" or "source-distribution". Such a kind previously produced an SBOM that failed ISPRAS validation later; a resolver returning e.g. website now stops the build instead.
• A source-distribution reference carries the STREEBOG digest the resolver reports in hashes; a vcs reference carries none.
• BREAKING: a build fails when the resolver answers source-distribution without a usable digest — no hashes, an algorithm other than STREEBOG-256/STREEBOG-512, or content that is not 64/128 hex characters — with enrich: source distribution has no hashes … or the matching algorithm/content error. Previously such a component entered the SBOM as a leaf with a bare archive URL, which the ISPRAS oss schema rejects.
  • VERIFIED: the production resolver returns hashes: [{"alg":"STREEBOG-256",…}] for pkg:npm/commondir@1.0.1 and hashes: [] for vcs results.
• A source distribution inherited from a base image's SBOM without a digest — hashes absent or hashes: [] — is replaced by the freshly resolved one instead of being kept; one that already carries a digest is kept.
• The BOM-wide externalReferences list is derived from the final component references: a link replaced on one component stays listed while another component still carries it with a digest.

werf sbom validate

• --check-source-distribution makes the checker verify that each source-distribution URL exists and points to an archive; default false. It also turns on VCS URL validation of every component, exactly as --check-vcs does — that is how the checker image behaves, and the flag help says so.
• --check-vcs-leaf-only restricts VCS URL checking to leaf components; default false.
• --check-vcs-leaf-only together with --check-source-distribution is rejected before the checker runs, with --check-vcs-leaf-only cannot be combined with --check-source-distribution: the checker would skip source distributions of non-leaf components; use --check-vcs instead. The checker applies the leaf filter before reading any reference, so the combination silently left non-leaf archives unchecked.
• The validation log header lists the checks the checker really runs, not the flags passed: --check-source-distribution alone yields with VCS, source distribution check; --check-vcs --check-vcs-leaf-only yields with leaf-only VCS check.
• VERIFIED: with --check-source-distribution a dead tarball URL yields WARNING: … не указывает на архив или не существует and Result: 0 passed, 1 failed; without the flag the same file passes that check silently.
• UNVERIFIED: an enriched BOM whose leaf component carries only a source-distribution reference with hashes has not been run through the ISPRAS checker image; the schema (component_spec → externalReferences with STREEBOG hashes) was read, not executed. A fixture run against the pinned checker image would settle it.

Why

Two independent producers write external references — os-pm for system packages and the PURL resolver — and the resolver appended unconditionally, so a component resolved to a kind it already had ended up with two links. Enrichment also runs on the merged BOM of every downstream image, and it runs after DedupBOM, so each merge appended another copy. The ISPRAS checker warns on a component with two distinct vcs links and werf treats any warning as a failure, which makes the published SBOM unvalidatable.

Deduplicating by URL and type would not have been enough: the checker collapses identical links and only reports distinct ones, which is exactly the os-pm plus resolver case.

The enrichment allow-list was the full CycloneDX enum while the ISPRAS schema requires vcs or source-distribution per component, so the two disagreed and the disagreement surfaced at validation time rather than at the build that caused it. The same schema requires a leaf whose only link is a source distribution to carry a STREEBOG digest of the archive; the resolver already computes it, but the response struct had no field for it.

The two validate flags leak the checker image's internals: its --check-vcs-leaf-only is a continue ahead of any reference parsing, and its VCS collection is not gated by --check-vcs. Fixing the image would be the right long-term answer, but the image is consumed by a floating tag, so a werf-side guard is the only thing that makes the CLI contract hold for whatever image is pulled. Running the checker twice — leaf-only VCS in one pass, source distribution in another — is not an alternative: the second pass would still run the full VCS check.

[reyreavman]

Verification

• Ran the ISPRAS checker image directly on hand-built fixtures to establish what it actually reports: one vcs link, two identical vcs links and vcs + source-distribution all pass; two distinct vcs links produce содержит 2 ссылки типа vcs. That is what the dedup rule is shaped around.
• Ran werf sbom validate --check-vcs --check-source-distribution against a BOM holding a dead npm tarball URL: the warning appears only with the flag, and the run exits 1.
• Ran the built binary with --check-vcs-leaf-only --check-source-distribution: exits 1 with the "cannot be combined" error before any container starts.
• Built a two-image project (yarn packages directive plus an import from the first image) against a local registry with the built binary and read both SBOMs back: one external reference per component, no duplicates.
• Queried the production PURL resolver for pkg:npm/commondir@1.0.1: kind: source-distribution, hashes: [{"alg":"STREEBOG-256","content":"4559fe…"}]; vcs results return hashes: []. The resolver mock in helpers_test.go reproduces that payload verbatim.
• Read the checker source (sbom-checker.py, schemas/schema.json) for the two flag claims: leaf filter is a continue before externalReferences is read; vcs collection runs under check_vcs or check_vcs_leaf_only or check_source_distribution.
• Mutation: restored pkg/sbom/externalref/enricher.go from origin/main → 4 specs failed, including keeps a single reference of a type the component already has and does not duplicate references when an enriched BOM is enriched again.
• Mutation: RunOptions.Validate condition replaced with false → 2 specs failed (leaf-only vcs and source distribution, every check).
• Mutation: hasHashes reduced to a nil check → 1 spec failed (replaces a source distribution … hashes empty).
• Mutation: BOM-wide list restored to incremental "delete old key on replace" → 2 specs failed (keeps a shared source distribution in the BOM list …, both component orders).
• Not run: test/e2e/sbom — the suites need a trusted builder base image from an external registry. The resolver mocks in those suites answer kind: "vcs", so the narrowed allow-list should not affect them, but that is read from the fixtures, not observed.
• Not run: the checker image against an enriched leaf carrying only source-distribution + STREEBOG hashes. The image is consumed by the floating :master tag, so a runtime regression test would follow the image, not the code.

Review focus

• The tie-break in Enrich: the pre-existing reference wins and the resolved one is dropped, except a hash-less source distribution, which the resolved one replaces. For os-pm packages that keeps os-pm's git:// URL rather than the resolver's https:// one, which matters under --check-vcs if a host no longer serves the git protocol. Confirm this is the precedence we want.
• enabledChecks now models the checker's semantics (source-distribution ⇒ VCS, leaf-only narrows) rather than echoing flags. If the image is fixed later, this mapping has to follow.
• docs/_includes/reference/cli/werf_sbom_validate.md is generated by task doc:gen.

Follow-up

• Fix --check-vcs-leaf-only / --check-source-distribution coupling in the checker image (3p-ispras-sbom-checker, sbom-checker.py), then drop the werf-side guard.
• Pin checker.Image to a digest instead of :master; without it a runtime regression test for the flags is not reproducible.
• Decide whether --check-source-distribution should be enabled by default in the pipelines that already pass --check-vcs.

[Fral738]

Two interactions in the checker image make the newly exposed checks behave differently from their CLI contract:

• Combining --check-source-distribution with --check-vcs-leaf-only silently skips source archives on non-leaf components. A dead parent archive changes from failure to success just by enabling the VCS modifier.
• --check-source-distribution alone also enables VCS validation, despite --check-vcs=false and the log reporting only a source distribution check.

The inline comments contain reproductions and concrete corrections. They were reproduced through the built CLI against checker image digest sha256:d9a95347a28b4bbbd6163b0c0dc5215e0e5b5a60acc2d519a55cc5bca86e6c49.

There is also a description/test claim to correct: "a component carrying only that reference is not an error" is not true of the source-distribution reference produced by the enricher. The ISPRAS schema requires hashes, including a STREEBOG hash, when a leaf has no vcs reference. The new acceptance test only checks the reference type; the resolver outcome supplies no hashes. A minimal otherwise-valid OSS BOM with that reference fails werf sbom validate --ispras-format oss --path <bom> with ERROR: 'hashes' is a required property, exit 1, even without URL checks. Missing hashes predate this PR; please qualify the description and the test's claimed coverage rather than presenting source-only output as proven valid. If schema-valid source-only output is intended in this PR, supply the required hash and test the serialized BOM against the checker.

Verification: task build and the full task test:unit passed. Mutations disabling duplicate suppression, changing the allow-list, and omitting the new Docker arguments were detected by the corresponding tests; restored runs passed. Full image-build e2e/integration suites were not run.

[Fral738]

The previous flag findings are addressed by the early combination guard and the updated help/logging contract. The remaining findings are in the new source-distribution repair path:

• An inherited hashes: [] is treated as an existing digest, so enrichment succeeds without repairing the hash-less reference despite a valid resolver result.
• Replacing one component's hash-less reference deletes the same URL/type from the BOM-wide reference set even when another component retains that reference with a valid digest.

Both cases were reproduced with focused Go tests on this head; the inline comments include the scenarios and corrections. The first also qualifies the description's claim that an inherited source distribution without a digest is replaced: it currently only handles a nil hashes pointer.

Verification: task build and the full task test:unit passed. Mutations of hash validation/copying/replacement and checker guard/logging were detected; restored runs passed and the worktree is clean. The built CLI rejects the incompatible flags before file access or container startup. The source-only serialized BOM has not been revalidated against the checker in this pass: the local Docker socket is absent and SSH to the available Linux host timed out. Full image-build e2e/integration suites were not run.

[Fral738]

The description's claim that "every component ends up with at most one reference per type" is stronger than the implementation. Enrich prevents appending another reference of that type, but does not collapse references already present on the component. DedupBOM only removes identical reference objects, so distinct inherited URLs survive that step too.

With both git://example.com/lodash.git and https://github.com/lodash/lodash already present as vcs, Enrich returns nil and retains both. Adding that second input reference to the existing keeps a single reference of a type the component already has test makes its length assertion fail (2, expected 1). An otherwise-valid OSS fixture with those references still produces the checker's contains 2 vcs links warning and CLI exit 1. This is an existing limitation, not a new regression introduced by the latest fixes.

Please qualify the description, for example: "Enrichment no longer adds a second reference of a type already present. Existing duplicate references inherited from older SBOMs are not cleaned up by this change." This documents the actual scope without requiring duplicate repair to be added to this PR.

[reyreavman]

Agreed, the claim overstated the implementation: findRefType acts on the first reference of the type, so two distinct inherited vcs URLs pass through untouched, and DedupBOM only removes identical objects. Description qualified accordingly (the bullet in What changes): enrichment no longer adds a second reference of a type already present; inherited duplicates of one type are not collapsed by this change. No code change — collapsing pre-existing duplicates would need a rule for which URL wins and belongs in its own PR.