When a RAG system reaches production, good embeddings aren’t always enough.
That’s something I started noticing while thinking more deeply about retrieval.
Vector Search is great at understanding semantic meaning.
But sometimes, semantic similarity is not what you need.
Imagine your knowledge base contains 10,000 technical documents.
A user searches for:
ERR_CONNECTION_REFUSED
Now imagine two documents:
Document A
“Troubleshooting network connectivity issues between services”
Document B
“How to fix ERR_CONNECTION_REFUSED in our internal gateway”
Vector Search may consider both highly relevant.
And semantically, that makes sense.
But as a user, I probably want Document B first.
Because sometimes an exact term matters more than semantic similarity.
That’s where BM25 helps.
Vector Search → finds similar meaning
BM25 → finds important exact terms
Instead of choosing between them, we can run both:
- User Query
- Vector SearchBM25
- Result Fusion
- Reranker
- Best Context
- LLM
The idea is simple:
Vector Search helps you find what the user means.
BM25 helps you find what the user actually typed.
And the reranker helps decide which retrieved chunks are really the most useful.
The interesting part isn’t adding BM25.
It’s deciding:
- When does semantic similarity matter more?
- When do exact terms matter more?
- How do you merge both result sets?
- How many candidates should reach the reranker?
- How do you measure whether retrieval actually improved?
That’s where RAG becomes more than just “embeddings + vector database”.
The LLM is only as good as the context you give it.
And that context starts with retrieval.
Keep reading: Learn how to evaluate the answer after retrieval in LLM-as-a-Judge, or how an agent can use retrieved information to choose its next step in ReAct in Agentic AI.
#RAG #LLM #AIEngineering #GenerativeAI #MachineLearning #SoftwareEngineering