fix(ci): make HttpApi exerciser and Bun setup resilient to flakes - #13094
Draft
kirillk wants to merge 5 commits into
Draft
fix(ci): make HttpApi exerciser and Bun setup resilient to flakes#13094kirillk wants to merge 5 commits into
kirillk wants to merge 5 commits into
Conversation
The `test` workflow was the dominant cause of red main (~32% of recent push runs). Two transient failure classes accounted for most of it: - The HttpApi exerciser has no retry, so a single flaky scenario (always `v2.session.permission.create`, `pass=314 fail=1`) failed the whole required gate. Cross-scenario state occasionally survives the async reset between scenarios, flipping the created permission's effect away from "ask". - `typecheck` failed with `socket hang up` while `oven-sh/setup-bun` downloaded the Bun release — a pure network flake with no retry. Add scenario-level retry to the exerciser (default 2 extra attempts, `--retries`), report retried passes as `flaky=N` so they stay visible, and retry the Bun download once, mirroring the existing Setup Node retry. A genuine regression still fails every attempt.
Contributor
Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Fix these issues in Kilo Cloud Files Reviewed (1 file, incremental)
Previous Review Summaries (4 snapshots, latest commit cb86d1e)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit cb86d1e)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Fix these issues in Kilo Cloud Files Reviewed (1 file, incremental)
Previous review (commit 7f476c1)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Fix these issues in Kilo Cloud Files Reviewed (3 files)
Previous review (commit 71f36f2)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Fix these issues in Kilo Cloud Files Reviewed (2 files)
Previous review (commit 945fb7d)Status: No Issues Found | Recommendation: Merge lgtm — the retry logic is correct (Effects are safely re-run, each attempt gets a fresh Files Reviewed (6 files)
Reviewed by kimi-k3 · Input: 51.8K · Output: 3.1K · Cached: 231.9K Review guidance: REVIEW.md from base branch |
The pinned CLI download in `generateOpenApiSpec` / connect-time resolve used a single OkHttp call with no retry, so a transient network drop (`java.net.SocketException: Unexpected end of file from server`) failed the whole jetbrains typecheck/build. Retry the metadata fetch and the archive download up to 3 times with a short backoff. HTTP status errors (404, rate limit) still fail immediately without retry.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| val url = url(version, platform, ext) | ||
| log.info("Downloading Kilo CLI $version for $platform from $url") | ||
| val request = Request.Builder().url(url).build() | ||
| retryNetwork("Downloading Kilo CLI $version for $platform") { |
Contributor
There was a problem hiding this comment.
SUGGESTION: Download progress resets to 0 on a retry
read and last are re-initialized on every retried attempt (they live inside the block passed to retryNetwork), so after a mid-transfer drop the reported percent jumps back down (e.g. 40% → 1%) before climbing again — consumers of ConnectionState.Downloading(percent, ...) in KiloBackendConnectionService will briefly see the progress bar regress. Consider hoisting the progress tracking outside the retry, or reporting a monotonic max across attempts, so the bar doesn't move backwards during a flake retry. Purely cosmetic; fine to leave as-is if you prefer.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
A single immediate retry did not cover GitHub release CDN blips that last tens of seconds (observed: back-to-back "socket hang up" and HTTP 503 within ~20s). Attempt the Bun download up to three times with 15s/30s backoff between tries; only the final attempt fails the job.
The build-time downloader (used by typecheck/build) threw immediately on a dropped connection or transient 5xx (observed in CI: HTTP 503 from the GitHub release CDN), failing the whole jetbrains typecheck. Retry both the metadata fetch and the archive download on IOExceptions and 5xx with a backoff (4 attempts). Permanent errors (404, rate limit, digest mismatch) still fail immediately. This complements the runtime KiloCliDownloader retry; the build task has no cached fallback, so it also retries transient 5xx (the runtime path intentionally fails fast on 5xx and reuses its cached CLI).
kirillk
marked this pull request as draft
August 13, 2026 00:04
t7tran
pushed a commit
to t7tran/kilocode
that referenced
this pull request
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resilience fixes that eliminate the most frequent transient CI failures on
main. All three are the same root pattern — a network download or a single flaky scenario with no retry fails the whole gate:test (linux)gate. Added--retries(default 2 extra attempts) intest/server/httpapi-exercise/. Retried passes are reported asflaky=Nin the summary and tagged[attempt N]per line so flakes stay visible. A genuine regression still fails every attempt..github/actions/setup-bunnow retries theoven-sh/setup-bundownload once (continue-on-error+ a gated retry step), mirroring the existing Setup Node retry.KiloCliDownloaderfetched release metadata and the archive with single OkHttp calls; a dropped connection failed the whole jetbrains typecheck/build. Now retries transient network errors (up to 3 attempts, short backoff). HTTP status errors (404, rate limit) still fail immediately.Why
Investigation of recent push runs on
mainshowed thetestworkflow failing ~32% of the time, from transient classes:testfailures): always the identical scenariov2.session.permission.create(pass=314 fail=1). Cross-scenario state occasionally outlives the async reset between scenarios, flipping the created permission'seffectaway from"ask".typecheck:socket hang upwhile downloading the Bun release.typecheck-jetbrains(seen on this PR's own CI):java.net.SocketException: Unexpected end of filedownloading the pinned Kilo CLI release. Confirmed transient — passed on re-run.Retry is the contained, correct fix for all three: transient flakes recover on a second attempt, real breakage keeps failing.
Tests
test/server/httpapi-exercise-retry.test.tscovers the exerciser retry decision logic (first-try pass, recover-after-one-failure withattempts=2, exhaust-all-attempts fail).KiloCliDownloaderTest."retries a dropped download and succeeds"usesMockWebServerwithDISCONNECT_DURING_RESPONSE_BODYto prove the download recovers after a mid-transfer drop (13/13 downloader tests pass).bun run typecheck, prettier,check-opencode-annotations.ts --worktree, and./gradlew :backend:test --tests KiloCliDownloaderTestall pass. Ran the exerciser locally: permission scenarios green,flaky=0.Not in scope (follow-up)
The remaining
testfailures are Windows unit-test flakes on the 4-vCPU runners — timeouts on a different test each run (KiloSnapshotTrack,config overlay routes,HttpApi Server.listen) plus agrep-signal-controlslayer-initfetch()socket drop. These are real-time / harness-network races that need careful, individual rewrites (per the repo's "wait on a readiness signal, not wall-clock" guidance) and are intentionally left out of this PR to keep it low-risk. The unit runner already retries failed files once.