Investigate concurrent validation to avoid serial fetches sharing one request timeout budget

[bdehamer]

Summary

When a single admission request contains multiple images (e.g. a multi-container pod), the provider validates them sequentially, and within each image it fetches the referenced attestation bundles sequentially. All of this work shares one parent context — the inbound request context, whose deadline is effectively the admission webhook timeout. As a result, a slow fetch for one image can consume most or all of the shared budget and starve the images processed after it.

This issue proposes investigating concurrent validation (across images, and across a single image's attestation bundles) so that the total wall-clock time for a request approaches the max of its constituent fetches rather than the sum.

Background: two levels of serialization

Gatekeeper's external-data protocol sends a list of image references (keys) in a single request, and the provider validates each in turn:

1. Images within a request are validated serially. Validate loops over r.Request.Keys and calls BundleFromName(ctx, …) for each image, passing the same request ctx every time (pkg/provider/provider.go).
2. Attestation bundles within an image are fetched serially. For each image, DoBundleFromName iterates the referrers index and, for every matching bundle, performs two registry round-trips (the bundle's manifest GET and its blob GET) one at a time (pkg/fetcher/bundle.go). An image with N attestations therefore costs roughly 2 + 2N sequential round-trips.

How this interacts with --bundle-timeout

--bundle-timeout (default 3s) is applied per attempt, per image: retryBundle wraps each attempt in context.WithTimeout(ctx, timeout), and each image gets its own retry loop with up to --bundle-max-attempts (default 3) attempts. So in isolation an image can take up to max-attempts × bundle-timeout.

The important subtlety: that per-image budget is derived from the shared request ctx. context.WithTimeout(parent, d) fires at the earlier of now + d and the parent's deadline. Because every image in the request shares the same parent context (carrying the admission webhook deadline), the per-image timeouts don't compose additively — they're all clamped by the one request-wide deadline. Concretely:

• The images are processed one after another under a single shared deadline.
• If image #1's fetch is slow (e.g. it has many attestations, or the registry is under load), it can consume most of the shared budget.
• Images #2, #3, … then start with little or no time left, and their first attempt is cancelled almost immediately — they fail not because their attestations were slow, but because an earlier image in the same request exhausted the shared budget.

A consequence worth calling out: raising --bundle-timeout does not give a multi-image request more total time — it's already bounded by the request/webhook deadline that all images share. It only changes how a single image's budget is subdivided across attempts.

Why it matters

• Multi-container pods (app + sidecars) are common, so multi-image requests are a meaningful fraction of traffic.
• Images vary widely in how many attestations are attached; an image with many attestations pays the 2 + 2N serial cost and is disproportionately latency-sensitive.
• Under registry latency spikes, the serial-within-serial structure multiplies exposure: every extra round-trip is another chance to cross the deadline, and the cost is borne not just by the slow image but by its request-mates.

Ideas to investigate

1. Validate images within a request concurrently (bounded parallelism), so total time ≈ max(image fetch times) instead of the sum. This directly protects request-mates from one slow image.
2. Fetch a single image's attestation bundles concurrently (bounded parallelism) in the referrers loop, reducing an image's own 2 + 2N serial latency.
3. Give each image a fair share of the budget, or a per-image sub-deadline, so one image cannot silently consume the whole request budget.
4. Consider the interaction with --bundle-timeout / --bundle-max-attempts semantics and document how they behave for multi-image requests.

Open questions

• What's a safe bound on concurrency (per request and per image) given registry rate limits and provider CPU/memory?
• Should the per-attempt timeout remain per-image, or should there be an explicit per-image sub-budget carved from the request deadline?
• How does concurrency interact with retries and with the in-memory bundle cache?
• Are there ordering or fairness guarantees we want when the shared deadline is tight (e.g. start all images at once vs. round-robin)?

Relevant code

• pkg/provider/provider.go — Validate loops over request keys, sequential per-image validation on the shared request context.
• pkg/fetcher/bundle.go — retryBundle (per-attempt context.WithTimeout), DoBundleFromName (sequential referrers/bundle fetch, 2 + 2N round-trips).
• cmd/aaop/aaop.go — --bundle-timeout, --bundle-max-attempts, --bundle-delay flag definitions.