I tested my RAG retriever on terms it hadn't seen
Short answer
I tested the same lexical retrievers on 25 glossary terms outside the original tuning set. Normalised + fuzzy found 34 of 39 required mentions, against 33 of 36 before. Longer lookups found ten mentions short ones never could, but every lexical retriever injected wrong senses into posts that needed nothing, and the model used the wrong word all 33 times.
My last post, RAG is a search problem first, scored a glossary lookup on its own before a model wrote a word. Two people on dev.to asked what that experiment had missed.
Nilesh Sharma asked whether the three and four word lookups were worth the extra context. Fair question. My glossary had nothing longer than two words, so those lookups never matched anything.
KolmoPDF pointed out a bigger issue. I had tuned the normaliser and fuzzy threshold by hand on the same 24 posts I used to score them. That tells me how the retriever behaves on those posts, not whether the same rules transfer to new terms. The suggestion was to hold out glossary terms, and to label examples where no entry should be injected. Otherwise recall can improve while bad substitutions go unnoticed.
Both points were fair. So I kept the retrieval rules fixed and built a new test.
The setup
Same café. Same made up words. Same retrieval rules. I reused the five retrievers from the last demo without changing their code, including the 0.75 fuzzy threshold.
One distinction matters: unseen here means these glossary terms were not in the original set I used to tune and inspect the rules. At run time, the retriever still receives the glossary and compares the post against those terms. This is a test of whether the same lexical rules transfer to new terms, not a test where the retriever has no access to the glossary.
The new set contains:
- 25 new glossary entries. These include the first long ones: eight of three words, such as
iced green teaandkids eat free, and two of four,buy one get oneandpay at the counter. - Five ambiguous words, each with a labelled sense:
toast(bread),jam(fruit),tart(pastry),pie(baked) andshot(espresso). - 36 new synthetic posts, each labelled with the entries a correct rewrite needs and any entries that would be wrong. Six posts should get no glossary entry at all: a traffic jam, raising a toast, a soda that's "a little tart", a pie chart, a photo shot and a site that "uses cookies". That last one tests an old glossary term in a sense the first set never had.
I committed the posts and labels before running the retrievers, and didn't change them after seeing the scores.
Recall on the new terms
Here are the four lexical retrievers. The oracle isn't in this table because it returns the labels by definition; I use it later as a perfect retrieval reference.
| Retriever | Posts complete | Required term mentions found | Wrong sense entries |
|---|---|---|---|
| Exact (my rule) | 18 of 36 | 25 of 39 | 5 |
| Normalised | 22 of 36 | 31 of 39 | 6 |
| Fuzzy | 23 of 36 | 33 of 39 | 6 |
| Normalised + fuzzy | 25 of 36 | 34 of 39 | 6 |
"Required term mentions" counts occurrences, not unique terms: some of the 25 new entries are needed by more than one post, so the posts need 39 mentions of them in all. A post is complete only when every required entry comes back and no wrong sense entry does. For a post that needs nothing, complete means the retriever returned nothing.
Normalised + fuzzy found 34 of 39 required mentions. On the original 24 posts it found 33 of 36. So it didn't collapse on new terms, but it did a little worse. That's a useful result, not proof that the threshold generalises to real posts.
Per term, it found 20 of the 25 new terms every time a post needed them, and the other 5 some of the time. None were never found. My exact rule found 14 every time and never found two: pancake, which only ever appeared as "pancakes" or "hotcakes", and free wifi, written "Wi-Fi" in the post.
Normalised + fuzzy missed five mentions. Three were forms no spelling based retriever can reasonably infer: the synonyms hotcakes and nimbu pani, and the abbreviation BOGO. Two were misspellings: croisants and smoothy. It did catch the typos inside longer phrases: hapy hour and early bird ofer.
The sense problem
This is the part KolmoPDF was warning about.
All six posts that needed nothing should have come back empty. Normalised, fuzzy and normalised + fuzzy returned a wrong sense entry for all six. Exact returned one for five; it got only "uses cookies" right, because cookies doesn't exactly match the glossary entry cookie.
The retrievers don't know what a word means. "Traffic jam" contains jam, so the glossary entry for the fruit comes back. A higher recall score doesn't tell you whether the retrieved entry belongs in that sentence.
And the model used every wrong sense entry it received. In the answer runs, exact supplied five wrong sense entries per run and normalised + fuzzy supplied six. Three runs of both gave (5 + 6) × 3 = 33 wrong entry exposures. The model put the wrong glossary word in the post all 33 times:
- "Stuck in a traffic murla? We will keep your table."
- "This goma chart shows our most loved drinks."
- "Let's raise a brenka to five years of the café!"
In the previous demo, the model ignored the wrong bread roll entry twice in 15 exposures. This time it didn't ignore any. The prompt says to use the glossary, and it did.
So yes, recall improved, and the wrong senses came along with it. A recall number on its own would have made this look like a cleaner win than it was.
Longer lookups
Nilesh's question. I ran the exact retriever with a two word lookup limit, then with the four word limit my original glossary code uses. The counts below are totals across all 36 posts.
| Lookup limit | Three word mentions found | Four word mentions found | Posts complete | Lookup keys built | Glossary entries sent to prompts |
|---|---|---|---|---|---|
| Up to 2 words | 0 of 9 | 0 of 3 | 8 of 36 | 457 | 24 |
| Up to 4 words | 8 of 9 | 2 of 3 | 18 of 36 | 837 | 34 |
The eight three word entries are needed nine times, because early bird offer appears in two posts, and the two four word entries three times, because buy one get one is needed in two posts, once written as BOGO. The longer scan found ten of those twelve mentions. Complete posts went from 8 to 18. It built about 1.8 times as many lookup keys, from 457 to 837.
Those lookup keys are strings built locally; they never go into the prompt. Across the 36 posts, the longer scan added ten glossary entries to the prompts in total, under one per post on average. Those entries still use tokens, and I haven't measured their exact token cost or the extra lookup time.
This answers the coverage part of Nilesh's question: on this set, longer lookups found entries the short scan could never reach. I haven't run a separate generation comparison for the two lookup limits, so I can't yet say whether the extra context improves the final rewrites enough to be worth it everywhere.
Then the model
I ran three answer setups on openai/gpt-oss-120b through Groq: exact retrieval, normalised + fuzzy retrieval, and the oracle. The prompt was the same as in the previous demo. It asks the model to use the supplied glossary words exactly as written. Code checks that every required made up word appears and no forbidden one does.
| Retriever | Posts right, runs 1 / 2 / 3 | Right when retrieval was complete | Right when retrieval had a gap |
|---|---|---|---|
| Exact | 18, 18, 18 | 18 of 18 | 0 of 18 |
| Normalised + fuzzy | 25, 25, 25 | 25 of 25 | 0 of 11 |
| Oracle | 33, 34, 32 | 33, 34, 32 of 36 | none |
For both lexical setups, every post with incomplete retrieval was wrong in all three runs: 18 posts with exact, 11 with normalised + fuzzy.
The oracle is the useful control. It gives the model exactly the right entries, with no search mistakes. Even then, it got 33, 34 and 32 posts right. It left hotcakes and nimbu pani unchanged in every run, although their entries were in the prompt. That's two misses per run. The rest were typos it copied instead of replacing: "Mango smoothy" in runs one and three, and "hapy hour" in run three. Run two had no other misses. It did replace BOGO with the word for buy one get one every time.
Search galat, toh answer galat. Still.
What I'd change
- Keep the longer lookups. They found ten mentions here that nothing else could. I'd keep them, then watch the extra matches and the actual prompt tokens on real posts.
- Treat ambiguous entries differently. Retrieval can't tell fruit jam from a traffic jam. I could attach a sense note for the model, or only inject the entry when the surrounding words fit. I haven't tested either yet.
- Keep the posts that need nothing. Without those six, this test would have missed the worst false positives.
- Add aliases for forms spelling can't reach.
hotcakes,nimbu paniandBOGOneed explicit aliases or a different retrieval method. Fuzzy spelling alone won't get them.
What this doesn't prove
I wrote the posts and labels myself, and I built the retrievers. Even though I committed the set before running it, I knew how the rules worked when I wrote the examples. These are 36 synthetic posts, not a sample of real business posts. A stronger test would use posts collected independently, with labels checked by someone who knows the senses the glossary means.
Only one post needed an old entry, so the comparison is with the original set's own scores, not two halves of one set. Six posts that need nothing show that false positives are a real failure here, but not how often they'd happen on real posts. The normaliser and threshold were frozen, which makes this a better transfer test than the first demo, but the set is still small and synthetic. It's one model, three runs and a made up vocabulary.
What it does show: the same retrieval rules found most of the new terms, longer lookups added matches nothing else could, and lexical recall doesn't protect you from wrong senses. The retriever needs to know not only what to find, but when to leave the prompt alone.
The code, data and every run are in shipping-with-ai/03b-unseen-terms. Thanks to Nilesh and KolmoPDF for asking the right questions.
Written by Sayam Jain, AI Product Engineer in Bengaluru. More about me