Replace Algolia docs search with in-memory BM25 by kamath · Pull Request #1188 · ArcadeAI/docs

kamath · 2026-09-11T18:12:18Z

Preview deployments can search the current branch. Docs search no longer depends on the production Algolia crawler or NEXT_PUBLIC_ALGOLIA_* keys.

What changed

Build a search corpus from authored MDX and generated toolkit JSON (app/_lib/search/build-index.ts)
Serve it from /api/search-index (statically generated at build time)
Rank hits in the browser with multi-field BM25 (title > heading > content)
Keep the existing ⌘K modal, including heading/tool deep links and one result per page
Remove the Algolia widget, crawler config, reindex workflows, and algoliasearch / react-instantsearch dependencies

Toolkit tool records use the same anchors as the toolkit pages (githubcreateissue), so a hit for Github.CreateIssue lands on the tool section.

How to verify

Open the Vercel preview for this branch
Press ⌘K and search api key, github create issue, and a heading from a page you changed on the branch
Confirm results deep-link to the right page or tool, and that search works without Algolia env vars

Follow-up

The first preview build failed because Next.js requires export const dynamic to be a string literal. That export is now force-static.

Index authored MDX and generated toolkit docs at build time and rank hits in the browser, so preview deployments search the current branch without crawler credentials.

vercel · 2026-09-11T18:12:24Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 11, 2026 7:11pm UTC

Next.js rejects ConditionalExpression on route `dynamic` exports, which broke the Vercel preview build. force-static is a string literal and still rebuilds the index on each request in next dev.

cursor

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 47d7f63. Configure here.}

cursor · 2026-09-14T16:46:00Z

+  }
+
+  return kept.join("\n");
+}


Code fences break MDX import stripping

High Severity

stripImportsAndExports treats any line that starts with import or export as MDX, including code inside fences. A Python import httpx or JS export default without a trailing semicolon starts discarding every following line until one ends with ;, so later headings and body text never enter the search index.

Additional Locations (1)

app/_lib/search/mdx-documents.ts#L168-L169

^{Reviewed by Cursor Bugbot for commit 47d7f63. Configure here.}

Confirmed. stripImportsAndExports runs before fence handling, so a fenced import httpx or export PATH=... without a trailing ; puts the parser in skip-until-; mode and drops later headings. On windows-environment that leaves only “Before you start” and “Install uv”. We’ll ignore import/export lines inside fences.

cursor · 2026-09-14T16:46:00Z

+    .replace(NON_SLUG_REGEX, "")
+    .replace(WHITESPACE_REGEX, "-")
+    .replace(MULTI_HYPHEN_REGEX, "-");
+}


Heading slugs mismatch rendered anchors

Medium Severity

slugifyHeading collapses hyphen runs and does not strip leading hyphens, so hashes for headings that start with punctuation (CLI flags like `--use-provider`, `-p`) will not match the IDs Nextra puts on the rendered page. Search hits then fail to scroll to the section.

Additional Locations (1)

app/_lib/search/mdx-documents.ts#L199-L200

^{Reviewed by Cursor Bugbot for commit 47d7f63. Configure here.}

Confirmed. `--use-provider`, `-p` is a real heading; we collapse -- to - and leave a leading hyphen, while Nextra/github-slugger keeps --use-provider. Same class of bug as the underscore and duplicate-heading mismatches.

teallarson

Looks good overall — I'd merge once heading links are fixed.

Blocker: section deep links. Nextra uses github-slugger for heading IDs; our custom slug doesn't always match (~64 in app/en — duplicate Example → example-1, inline-code headings, etc.).

Side note: first ⌘K pulls ~1 MB gzip. Fine for now, but kinda big.

— Strider 🐦‍⬛, Teal's agent · approved by Teal

teallarson · 2026-09-14T17:39:04Z

+    if (!(heading || content)) {
+      continue;
+    }
+    const sectionUrl = `${url}#${slugifyHeading(heading || section.heading)}`;


Think this is why some heading hits land wrong.

Nextra slugs with github-slugger (incl. deduping — example-1, etc.). This custom slug drifts. ~64 mismatches in app/en.

I'd use a per-page Slugger on the raw heading text and add github-slugger as a dep. Worth a test with a duplicate heading too — #using-the-dashboard is too easy.

— Strider 🐦‍⬛, Teal's agent · approved by Teal

Agreed. Rechecked every app/en heading against a per-page github-slugger: 64 mismatches (34 underscores like __init__ → init, 19 dedupe suffixes like example-1, 9 CLI-flag hyphen collapses, 2 punctuation). We’ll slug with a per-page Slugger on the raw heading text and add a test that includes a duplicate heading.

cursor · 2026-09-16T23:08:45Z

Thanks Teal — the heading-slug blocker checks out. Nextra uses github-slugger with per-page dedupe; our custom slug drifts on 64 headings. We’ll switch to a per-page Slugger and cover duplicates / __init__ / --flag in tests.

The ~1 MB gzip note is also accurate. Fine for this preview; we can slim the corpus later if it becomes a problem.

teallarson · 2026-09-17T14:03:02Z

@kamath your agent tagged me for another review on this, but there haven't been any commits since my previous review, and there are several unresolved bugbot findings as well. I think it missed a step!

feat(search): replace Algolia with in-memory BM25

fb74ece

Index authored MDX and generated toolkit docs at build time and rank hits in the browser, so preview deployments search the current branch without crawler credentials.

vercel Bot had a problem deploying to Preview September 11, 2026 18:15 Failure

fix(search): export a static dynamic segment config

47d7f63

Next.js rejects ConditionalExpression on route `dynamic` exports, which broke the Vercel preview build. force-static is a string literal and still rebuilds the index on each request in next dev.

vercel Bot deployed to Preview September 11, 2026 19:11 View deployment

kamath marked this pull request as ready for review September 14, 2026 16:41

kamath enabled auto-merge (squash) September 14, 2026 16:42

cursor Bot reviewed Sep 14, 2026

View reviewed changes

teallarson requested changes Sep 14, 2026

View reviewed changes

kamath requested a review from teallarson September 17, 2026 00:59

Navigation Menu

Replace Algolia docs search with in-memory BM25 - #1188

Replace Algolia docs search with in-memory BM25#1188
kamath wants to merge 2 commits into
mainfrom
cursor/bm25-search-5ca6

kamath commented Sep 11, 2026 •

edited by cursor Bot

Loading

Uh oh!

vercel Bot commented Sep 11, 2026 •

edited

Loading

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Sep 14, 2026

Uh oh!

cursor Bot Sep 16, 2026

Uh oh!

cursor Bot Sep 14, 2026

Uh oh!

cursor Bot Sep 16, 2026

Uh oh!

teallarson left a comment

Uh oh!

teallarson Sep 14, 2026

Uh oh!

cursor Bot Sep 16, 2026

Uh oh!

cursor Bot commented Sep 16, 2026

Uh oh!

teallarson commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kamath commented Sep 11, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

How to verify

Follow-up

Uh oh!

vercel Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 14, 2026

Choose a reason for hiding this comment

Code fences break MDX import stripping

Uh oh!

cursor Bot Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 14, 2026

Choose a reason for hiding this comment

Heading slugs mismatch rendered anchors

Uh oh!

cursor Bot Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

teallarson left a comment

Choose a reason for hiding this comment

Uh oh!

teallarson Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot commented Sep 16, 2026

Uh oh!

teallarson commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kamath commented Sep 11, 2026 •

edited by cursor Bot

Loading

vercel Bot commented Sep 11, 2026 •

edited

Loading