2 min readBuilding AI systems
Why most RAG systems fail in production (and how to tell early)
Retrieval demos look great on ten hand-picked questions. Production is a different test. Here are the failure modes I check for first, and the cheap signals that reveal them.
A retrieval-augmented assistant usually starts life as a promising demo: a folder of documents, an embedding model, a vector store, and a prompt. It answers the five questions the team tried, and everyone is excited. Then it meets real users, and the answers get vague, confidently wrong, or quietly out of date.
None of that means the approach is wrong. It usually means the system was judged on the generation step when most of the problems live in retrieval, in the data, and in what happens when the answer isn't there.
1. The right passage never reaches the model
If the passage that answers the question isn't in the retrieved context, no prompt will save you. Fixed-size chunking that splits a policy across two pieces, tables flattened into noise, and pure vector search that misses exact product codes or names are the usual suspects.
- Chunk along the document's own structure (headings, sections, rows) rather than a fixed character count.
- Combine keyword and vector search, then rerank. Exact identifiers matter more than people expect.
- Measure retrieval on its own: for a set of real questions, is the answer-bearing passage in the top results?
2. Nobody defined what "good" means
Without a small, honest evaluation set, every change is judged by feel. Build a golden set from real questions (including the awkward ones), record the expected source, and score both retrieval and answers before and after each change. Fifty well-chosen questions beat five hundred synthetic ones.
3. The system can't say "I don't know"
The most damaging answers are fluent ones built on the wrong context. Give the assistant an explicit path to decline, ask a clarifying question, or hand off to a person, and test that path as seriously as the happy path.
A trustworthy assistant is one that is predictably useful, and predictably honest about when it isn't.
4. The knowledge goes stale
Documents change; indexes often don't. Track when each source was last ingested, re-index on change rather than on a calendar, and show users the date of the source they're reading.
Cheap signals to watch from day one
- The share of answers that cite a source, and whether that source actually supports the claim.
- How often users rephrase the same question: a strong hint that retrieval missed.
- Handoff and "I don't know" rates: too low can be as worrying as too high.
- Latency and cost per answer, so quality improvements don't quietly blow the budget.
None of this is glamorous, but it's the difference between a demo and a system a team can own. If you're unsure where your assistant stands, start with retrieval quality on twenty real questions. It's usually the fastest way to find out.