feat(memory): make vault search find more, and say why it found nothing - #99
Closed
arthware-dev wants to merge 11 commits into
Closed
arthware-dev wants to merge 11 commits into
arthware-dev wants to merge 11 commits into
Conversation
Adds an FTS5 index over the vault so a search returns the page a question is about first, rather than every page that matched sorted by date. Folds umlauts, so "Kase" finds "Käse". Fuses a trigram index behind an opt-in flag, which is what reaches "Geburtstagsfeier" from "Feier". Each hit reports which of the query keywords it contains, so a caller can tell "found it" from "found the nearest page". Nothing calls this yet; the CLI still uses the regex walk.
An offline A/B over a fabricated 177-page vault and 47 questions, including 12 whose answer is not in the vault at all. Reports recall, ranking and latency per kind of question, and how well each candidate confidence signal tracks whether the top hit was actually right. Runs in about two seconds with no model and no containers, so an engine decision costs seconds rather than LLM calls. Questions are written from a neutral truth line rather than from the page they have to find, and both engines get the same mechanically extracted keywords.
The right page comes first 2.6x as often and a search costs a tenth of the time. Two results change the plan: the trigram index is required rather than optional, because German compounds regress without it, and the confidence signal to ship is keyword coverage, not the BM25 score. Paraphrase recall of 29% is the number tier 2 has to beat.
The index is a cache over the vault checkout, so a search has to know whether it is still current before it answers. Nothing writes into that checkout outside git, so a HEAD that has not moved means no page has changed: the index stores the HEAD it was built from and skips the reconcile when they match. Constant time instead of statting every file, which reaches 337 ms on a vault of 8000 pages. A directory that is not a checkout keeps scanning, so --vault overrides still pick up edits. Also: the HEAD is stored with the rows it describes, so a reconcile that dies halfway is redone rather than left half applied; a page deleted mid-scan is skipped instead of failing the search; and a second indexer waits for the file instead of erroring.
The engine bench measures whether retrieval finds the right page, which is not the same as whether the family gets a better answer. This runs seven questions a single lookup cannot answer through the real agent on each backend, over a vault the size of a real one. The rig gains a --corpus flag, because thirteen pages cannot tell two engines apart, and lab-api gains a --backend flag serving the real engine rather than a copy. Result: ranking wins one answer out of seven and costs nothing; the confidence gate wins none and is dropped. A fixed result limit loses questions that need every matching page rather than the best one.
One turn per cell could not tell a real difference from a dice roll, and the conclusions drawn from it about individual questions were wrong. This repeats each cell three times, alternates which backend runs first so neither always pays the cold prefix cache, and records every search both arms were given. Result: no difference between the engines that survives the run-to-run spread. The one claimed correctness win was the old engine's single bad draw, correct 2 of 3 times on a repeat. The search log also shows why the engine gains do not reach the family: the agent searches one word at a time, so there is nothing for a ranking scheme to combine, and the two arms shared only 19% of their keywords.
Two gaps in the vault search, both visible to anyone who types a
question into chat.
A page titled "Zahnarzttermin Lisa" could not be found by searching for
Zahnarzttermin unless the body happened to repeat it. Frontmatter was
stripped before matching so that a query for "date" would not hit every
page through its date line, and that threw the title and tags out with
the field names. The values are back; the keys are still out. Person
names stay out too, because a name says who a page concerns rather than
what it says, and --person already asks that.
Searching "Kase" for a page that says "Käse" returned nothing at all,
which reads like the vault has no such page rather than like the word
was spelled differently. Both sides now fold before matching. Spelled
out umlauts ("Kaese") are still a different word.
Measured on the retrieval lab: recall@1 21% to 26%, recall@5 54% to
64%, no question kind worse.
An empty search means one of two things and a caller has to act
differently on each: the words were wrong, or the fact is not written
down. "no results" cannot tell them apart, so a bad guess either gives
up on something that is there or keeps rephrasing at something that is
not.
$ stack memory search "repair|expense"
no results. These words appear nowhere in the vault: expense
Measured in the agent rig: a quarter of searches came back empty, and
those skew English against German pages. Telling the agent to search in
the family's language did not move it, because when it writes its first
query nothing has told it what that language is. This delivers the same
fact where it can be used.
Costs a second walk only on a search that already returned nothing.
The rig gains --skills and --explain-misses so the pairing can be
measured; that run has not been completed yet.
…tion Every other number here rests on a gold set somebody wrote. Replaying the 104 queries the agent actually sent says that of the two shipped search fixes, only one moves anything: queries returning nothing drop from 31% to 24%, and every rescue comes from matching titles and tags. Diacritic folding contributes zero on real agent queries. It is correct and tested, but the agent does not type German without umlauts, it types English. Credited earlier by the class it was designed for rather than by what moved.
The upgrade handover reads as build-ready and would send the next reader straight into building the index that was just measured as not worth building. It now says so at the top. Adds a round-two handover: the one change that improved anything, the four that did not, which of the 4371 lines belong on main, the open decisions (--nl unused in 161 searches, the oMLX prefill ceiling, the unfinished rig run), and the method notes behind four findings that were reported and then retracted.
Four spots in a public repo named a specific machine rather than the fabricated household everything else uses. - a WiFi network in a knowledge-doc example carried a real network name, while every other entry around it is invented (Duff Insurance, Dr Hibbert, Firma Huber). Now invented too. - a probe artifact recorded the endpoint's full list of 28 served models, which fingerprints one machine. The count is kept, since that is all the probe used it for. - two retrieval notes quoted a prefill error verbatim, including the memory cap in gigabytes. They now describe the limit without the numbers, which is the part that mattered anyway.
Contributor
Author
|
Closing without merging, on purpose. The experiment earned one small change and a set of negative results, and the negative results are worth more than the code that produced them. Merging 4371 lines to keep a 526-line findings doc is the wrong trade, and The work is kept on branch Start with Unrelated docs tidying that did belong on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes to
stack memory search, plus the bench that decided which ones were worth making.What actually moves anything
Replaying the 104 queries the agent really sent (
tools/retrieval-lab/replay.py), queries that return nothing drop from 31% to 24%. Every one of those seven rescues comes from one change:--personalready asks that).Shipped, but measured as contributing nothing
SKILL.mdlines was killed partway, so it is unverified at agent level. Drop theSKILL.mdhunk if you want only measured changes.Measured and deliberately not adopted
stacklets/memory/fts_index.pyis a SQLite FTS5 + trigram index with BM25 ranking. It roughly triples recall@1 on the bench (23% → 60%) and does not reach the family: seven questions through the real agent, three repeats, two arms, no difference in answers, iterations or wall time beyond run-to-run spread. It has no callers. It stays only becausetools/retrieval-lab/cannot reproduce that conclusion without it.A confidence gate on keyword coverage was the strongest number in the whole investigation (answers to absent facts 92% → 17%) and bought nothing once a model was reading the results. Dropped.
Reading the findings doc
docs/design/brain/retrieval-engine-poc.mdhas the numbers and the method. It also records four attributions of mine that did not survive being measured, each time because a fix was credited by the class it was designed for rather than by what moved.