fix(knowledge): make indexing and connector recovery durable

[icecrasher321]

Summary

• Save workspace imports as immutable KB-owned files before queueing, using the shared upload attachment and orphan-cleanup guard.
• Recover abandoned stored connector documents independently of source sync. Atomically install a new processing generation and outbox delivery with bounded attempts, runtime, and admission backoff.
• Treat shared provider capacity as a durable deferral. Preserve content and member progress across GitHub waits, pin immutable tree/blob reads, and reconcile removals only after a complete listing.
• Preserve safe, structured source failures and retry only positively identified transient database reads. Set HTTP body framing in the shared pinned transport.
• Keep regular KB and organization Search ownership, permissions, complete-index publication, and billing behavior intact. Apply the additive migration before deploying the workers.

Type of Change

• Bug fix

Testing

• 4,080 unit tests passed across affected knowledge, connector, processing, storage, deadline, and provider-capacity paths.
• 866 shared HTTP security tests passed, including real-socket body framing and redirect regressions.
• 328 integration tests and 13 PostgreSQL access/search tests passed in disposable PostgreSQL/Redis, including synthetic content against live OpenAI/Mistral APIs. The 42 skipped tests require separate opt-in provider or scale fixtures.
• Type check, lint, all 46 CI audits, API validation, docs manifest, block registry, and migration safety checks pass.
• Recovery coverage includes concurrent admission, duplicate/stale delivery, atomic rollback, blocked owners, organization Search authorization, immutable imports after source deletion, and concurrent-index migration replay.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 9, 2026 7:33pm UTC

[greptile-apps]

Greptile Summary

This PR makes knowledge ingestion and connector recovery durable across worker interruption, provider throttling, and source changes.

• Copies workspace imports into immutable knowledge-owned storage before creating and dispatching documents.
• Adds bounded recovery admission, processing generations, and transactional outbox delivery for abandoned connector documents.
• Persists resumable listings and provider-capacity deferrals while protecting completed work and deletion reconciliation.
• Pins GitHub tree and blob reads to immutable revisions and improves safe failure diagnostics.
• Adds the schema and concurrent partial index required for processing recovery.

Confidence Score: 5/5

The PR appears safe to merge; no actionable new defects remain, and both previous findings are resolved in the current code.

The latest changes safely improve recovery diagnostics, introduce a compatible typed deadline error, and correct the listing-checkpoint documentation. The current implementation preserves durable storage attachment, processing generation fencing, provider deferral, and authoritative-listing safeguards without an identified blocking failure.

Important Files Changed

Filename	Overview
apps/sim/lib/knowledge/application/add-workspace-files.ts	Imports workspace files into immutable knowledge-owned storage with renewed authorization, provenance checks, and transactional attachment.
apps/sim/lib/knowledge/documents/processing-recovery.ts	Implements bounded recovery of abandoned stored documents using fresh processing generations and durable outbox delivery.
apps/sim/lib/knowledge/connectors/listing-checkpoint.ts	Persists replay positions before page processing and advances checkpoints only after durable page work.
apps/sim/connectors/github/github.ts	Pins paginated listings and hydration to immutable GitHub tree and blob revisions while safely restarting expired snapshots.
apps/sim/lib/knowledge/connectors/sync-deferral.ts	Converts positively identified provider-capacity waits into durable connector scheduling deferrals.
packages/db/migrations/0336_knowledge_processing_recovery.sql	Adds processing recovery admission state and an additive concurrent partial index for bounded scans.
apps/sim/app/api/webhooks/outbox/process/route.ts	Runs stored-document recovery within the outbox cron budget and emits bounded structured diagnostics.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Workspace import or connector listing] --> B[Persist immutable KB-owned artifact]
  B --> C[Create pending document]
  C --> D[Atomically enqueue processing outbox event]
  D --> E[Processing worker claims generation]
  E --> F{Processing completes?}
  F -->|Yes| G[Publish completed document index]
  F -->|Provider capacity unavailable| H[Persist durable deferral]
  F -->|Worker abandoned or dispatch lost| I[Recovery scan]
  H --> D
  I --> J[Install new processing generation and outbox delivery]
  J --> E
  A --> K[Persist listing checkpoint]
  K --> L{Listing complete and authoritative?}
  L -->|Yes| M[Reconcile source removals]
  L -->|No| N[Resume without deleting unseen documents]

Loading

_{Reviews (2): Last reviewed commit: "fix(knowledge): expose safe recovery dia..." | Re-trigger Greptile}

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.