fix(knowledge): grant member access as each batch is indexed instead of at the end of the run - #7418
Conversation
…of at the end of the run
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 3 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 3 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
Greptile SummaryThe PR makes large member-mode crawls progressively accessible, batches bounded deferred connector content more efficiently, and fixes the stale-observation cutoff query.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| apps/sim/lib/knowledge/connectors/sync-primitives.ts | Reports persisted rows after queue dispatch and treats ordinary batch-grant failures as best-effort while preserving lease-loss termination. |
| apps/sim/lib/knowledge/connectors/member-sync-engine.ts | Groups persisted documents by observing member and incrementally records observations and materializes ACLs under the member lease. |
| apps/sim/lib/knowledge/connectors/sync-persistence.ts | Returns identifiers for newly inserted and replaced skipped-document rows so batch callbacks can grant access. |
| apps/sim/connectors/utils.ts | Introduces a UTF-8 byte-bounded line accumulator used to enforce connector transcript limits. |
| apps/sim/lib/knowledge/connectors/member-observations.ts | Correctly casts the stale-sweep instant before subtracting the interval window. |
Sequence Diagram
sequenceDiagram
participant Sync as Member sync
participant DB as Document persistence
participant Queue as Processing queue
participant ACL as Observation and ACL transaction
Sync->>DB: Persist hydrated and skipped batch rows
DB-->>Sync: Return external and document IDs
Sync->>Queue: Dispatch persisted documents
Queue-->>Sync: Dispatch completes or is logged
Sync->>ACL: Record observations and materialize ACLs
alt Non-lease grant failure
ACL-->>Sync: Error
Sync->>Sync: Log and continue
Sync->>ACL: Retry authoritatively during final listing pass
else Lease lost
ACL-->>Sync: SyncLockLostException
Sync->>Sync: End superseded run
end
Reviews (5): Last reviewed commit: "fix(google-chat): report the number of m..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
… instead of one at a time
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
…d keep the grant best-effort
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 9 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
…ler stops failing
… and advertise it as the hydration estimate
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 12 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
… actually indexed
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
1 participant
Summary
Three fixes to how a large first crawl feels and behaves in members mode, found on staging with a Gmail mailbox whose crawl had every indexed document hidden after fifteen minutes.
Access is granted as each batch is indexed. A members-mode run listed a member's whole source, hydrated every document, and only then wrote the member's observations and materialised ACLs. Until the run ended nothing was searchable, the Knowledge page's document list was empty, and the Sources chip read "Indexing" with nothing behind it for up to the run's 45-minute budget. The hydrate-and-persist loop now reports each batch it wrote (skipped rows included), and the member engine records the observation and materialises the ACL under its lease, best-effort, once the batch has been dispatched for processing. The listing's own pass at the end still records the same observations idempotently and remains the only thing that decides removals.
Deferred mail and chat documents hydrate in batches, under an enforced ceiling. A listing that defers content with no size hint was assumed to be as large as the whole in-flight budget, so every Gmail thread hydrated alone: one thread at a time, roughly 170 a minute on staging.
ExternalDocument.estimatedByteslets a listing say what to expect, and for text a connector assembles itself (a mail thread, a chat transcript) the estimate is now a real bound rather than a guess:CONNECTOR_TEXT_DOCUMENT_MAX_BYTES(12 MiB) is enforced by a sharedBoundedLinesaccumulator in the Gmail, Outlook, and Google Chat formatters, which appends each message whole or not at all and marks the document as truncated. Five such documents fit the 64 MiB in-flight budget, so they hydrate together. Sources with real file sizes are unchanged.The member-sync scheduler no longer fails on the stale-observation sweep. The sweep's cutoff was written as
$now - GREATEST(...)with the bound instant untyped, which Postgres resolves as interval arithmetic and then rejects everytimestamp > intervalcomparison. The instant is now cast, and the sweep runs cleanly against a real database.Verification
persistedDocumentsByObserveris a pure helper tested with a union of two members and a document nobody listed.BoundedLinesis tested for whole-record refusal, the truncation notice, and byte (not character) accounting.42883, the cast form returns.check:audits(45 audits) pass.