RAG Evaluation: The fix I was proud of, and the measurement that killed it

The hunch felt obviously right. Measured across all 34 questions, it lost on every axis that mattered.

Twice while building this project I noticed something that looked broken. Twice I designed a fix I was genuinely pleased with. Twice I measured it across the whole question set before shipping. Both times the measurement said no. This is what RAG evaluation is actually for. Not a scoreboard, but a way to find out you were wrong.

This post is about that gap. Not really about insurance, and not really about retrieval. About the distance between a fix that feels obviously right and a fix that is right, and the cheap habit that closes it.

The seductive one

The system has to know when a question has no answer in the document. My first design was cheap and, I thought, obviously correct. Before spending a single model call, look at how confidently retrieval found a matching clause. If nothing scores above some threshold, refuse. Fast, defensible, and it put the safety check early, where safety checks belong.

Before I wrote the gate, I did one boring thing. I measured the scores it would key on.

The 31 questions with real answers in the policy scored between 0.341 and 0.647, median around 0.55. The 3 questions I had written to have no answer scored between 0.479 and 0.640. The two ranges sit almost on top of each other. Any threshold high enough to reject the 3 unanswerable questions also rejects 29 of the 31 real ones. There is no line to draw.

The two ranges sit almost on top of each other. Any line strict enough to reject all three unanswerable questions also rejects 29 of the 31 real ones. There is no threshold to find.

Why it fails is worth sitting with, because it is not a tuning problem. One of the unanswerable questions was "what is the claim settlement ratio." That is unmistakably an insurance question. It is phrased in the document's own vocabulary and it embeds right next to the real coverage clauses. Retrieval similarity measures whether two pieces of text are about the same thing. Answerability is a different property, and no threshold on the first recovers the second. They are not the same measurement.

The heuristic was seductive because it was cheap and it sounded like rigour. Measuring it cost twenty minutes and saved me from shipping a gate that would have refused almost every real question. The lesson is not that retrieval scores are bad. It is that a plausible heuristic is a hypothesis, and a hypothesis you can test in twenty minutes is one you have no excuse to ship untested.

So refusal moved to where it belonged, a judgment the model makes, named as a first-class outcome in its instructions. Underneath it sits a check that does not trust the model at all: every citation the answer gives is verified against the clauses actually retrieved for it. A hallucinated citation reads exactly as confident as a real one. The only way to catch it is to check, not to believe.

The expensive one

The second reversal cost more, because I built an entire second pipeline before the data caught me.

I noticed that forcing the model to fill a rigid output schema seemed to hurt its answers on a few questions. The clearest asked about room rent, where the correct answer needs a different value for each of eight plans. All eight values were sitting in the retrieved context. Constrained to a schema, the model came back saying the document did not specify a room rent limit, and cited a clause that did not exist.

That looked like proof that the output format was competing with the model's ability to read its own context. So I built a two-stage pipeline. Let the model reason in free text first, with nothing constraining it. Then run a second call over that finished answer to extract the structured fields. Reason first, format later. On the room-rent question it worked. The free-text answer listed all eight plans correctly. I was pleased with it.

I did not ship it on one question. I built a single-stage baseline that got the same instructions plus the field guidance it needed to do the job in one pass, and I ran both pipelines across all 34 questions.

Two-stage lost. The single call verified every citation it produced, 34 of 34. The two-stage pipeline verified 61.8%. The single call cited the correct clause 90% of the time it gave an answer; two-stage managed 76.7%. The single call was also faster, 38.5 seconds a question against 59.7. One-stage hallucinated zero citations. Two-stage hallucinated on 13 of 34.

One constrained call beat the elegant two-stage pipeline on every axis. It verified every citation, cited the right clause more often, and did it in less time.

The mechanism, once I read what the free-text stage was actually writing, was specific. Given room to reason in prose, the model reaches for precise-sounding sub-clause references like "as per 4.6.iv.b." My system indexes clauses at the section level and cannot verify a reference that fine, so it counts as fabricated. The second stage then extracts those references faithfully, because faithful extraction was exactly what I asked it for. A single constrained call forces the model to commit to one verifiable section, and it turns out to be far more careful when it has to be.

Then the part that stung. The room-rent failure that started the whole detour was not evidence about constrained decoding at all. It was a bug in my prompt. My first single-stage attempt had been missing its field guidance. Give that back, and the single call answers room rent correctly too, all eight plans, no invented citation. The anecdote that launched a second pipeline was a confound.

What RAG evaluation actually taught me

Neither of these is a story about insurance. They are the same story twice, and it is a story about method.

A hunch from a handful of examples is a hypothesis, not a finding. Both fixes came from three or four anecdotes. Anecdotes are where good ideas start. They are not where they earn the right to ship. The refusal gate felt right. The two-stage pipeline felt right. Feeling right is the starting gun, not the finish line.

The fair baseline is the whole game. The two-stage result would have flipped if I had compared it against a crippled single-stage baseline, the one with the missing prompt guidance. It would have won, and I would have shipped the slower, less accurate design with a clean conscience. The entire result turns on whether the thing you compare against was given a fair chance. An unfair baseline does not just weaken a comparison. It can reverse it.

Measure on the whole set, not the example that motivated you. The room-rent question that started the detour was real, but it was one question out of 34, and it was misdiagnosed. Tuning to it would have meant fitting the confound.

Keep the code that lost. The two-stage pipeline is still in the repository, behind a flag, so anyone can reproduce the comparison. Deleting it would have hidden the one thing worth showing: not that I picked the right design, but that I picked the wrong one first and the measurement corrected me.

There is a piece of folklore that constraining a model's output degrades its reasoning. I believed it for about a week. On this task, with a fair prompt and a full run, it did not survive contact with the numbers. That is the only claim I will make about it, because it is the only one I measured.

The fixes I was proud of were both wrong. The habit that caught them, measure it before you believe it, is the part worth keeping.

View on GitHub