RAG Chunking: the bug that lived inside a page footer

Most of what looked like a retrieval problem was a document problem. The traps were in the clause numbering and the page furniture.

I found the worst bug in this project by accident. I was checking whether the system could tell someone, correctly, that maternity is not covered.

It could not. Not because the model was wrong. The clause that says so had already been destroyed before the model ever saw it.

What actually happened to the RAG chunking

The maternity exclusion in this policy is written as one clause with a short list of sub-points underneath it, numbered with lowercase roman numerals: i, ii, and so on. My chunking code read the document's own section numbering and cut the text into one chunk per clause. It walked the lettered clauses in order, a, b, c, looking for where each new one started.

The roman numeral "i." inside every clause looked, to a regular expression, exactly like a lettered clause called "i". So did the roman numerals inside the maternity clause itself. My parser read the first sub-point of the maternity exclusion and decided it was the start of a new clause. Nine different clauses across the document collided onto that same fake id, silently overwriting each other, and the text that survived was not the maternity exclusion at all.

The fix was not a smarter regular expression. It was sequence. A real lettered clause is always the next letter after the one before it. A nested roman numeral list restarts at "i" inside every single clause it appears in. The parser now only accepts a letter if it is the letter that should come next. Nine clauses recovered their real content. The maternity exclusion came back intact, citation code and all, with its one real exception still attached.

The regex could not tell a real clause "i" from the roman numeral nested inside every other clause. Nine clauses collapsed onto one id and silently overwrote each other. Sequence, not pattern, was the fix.

I did not find this by reading code. I found it by asking the system the actual question and reading the answer with real suspicion. The stakes of getting that one wrong are too high to trust a green test suite alone.

The document fights back before you ever reach the model

That bug was the sharpest example of a pattern that repeated through the whole build.

This is the unglamorous half of RAG chunking. Most of what looked like a retrieval problem was actually a document problem. It had to be solved before retrieval could mean anything.

The source is a two hundred and ninety one page insurance prospectus. I expected most of it to be policy text. It is not. Only about seventy one pages carry actual coverage terms, exclusions, and claims procedure. The other two hundred and twenty pages are premium rate tables: age against sum insured against plan against city tier, repeated across eight different plan variants. I excluded all of it from the retrieval index on purpose. A premium lookup is a structured query against a table, not a semantic search problem. Embedding two hundred numeric tables would have flooded the index with near identical noise for no benefit. Ask this system for a premium number and it tells you plainly that this is outside what it answers from, instead of guessing at a figure it has no business inventing.

Every page of the document also carries a repeated registration footer, and that footer is typeset two different ways depending on where it sits. I detect boilerplate by measuring how often a line recurs across pages, rather than hardcoding the footer text. A hardcoded string only catches the exact wording you tested against. The two footer layouts here sit at two different recurrence frequencies, and my first threshold landed squarely between them. It stripped one layout and left the other, and the surviving layout happened to sit on the pages that mattered most. Registration numbers ended up glued onto the end of real policy clauses. Real prose in this document never recurs above about three percent of pages. The footer recurs at twenty and eighty percent. Any threshold placed in that gap catches all of the boilerplate and none of the real content.

Real policy prose never repeats on more than about 3% of pages. The footer repeats on 20% and 80%, in two typeset layouts. Any threshold in that empty gap catches all the furniture and none of the content.

The plan comparison table was its own problem. Eight plans, one benefit per row, values that differ by hundreds of thousands of rupees between two plans whose names differ by a single word. Extracted as plain text, the table's eight columns collapse into one vertical stream with no way to tell which number belongs to which plan. So I transcribed that table by hand, and wrote a test that checks every transcribed number against the actual page it claims to come from. A typo in a hand transcription becomes a wrong answer about someone's money, with a confident citation attached.

The retrieval result I did not want to report

The part of this build that mattered most to get right was the evaluation, and the honest result there is not the one I set out to find.

I built hybrid retrieval, dense embeddings for paraphrase and keyword search for exact terms, because insurance language rewards both. "Co-payment" and "sub-limit" need exact matching. "Can I get treated at home instead of being admitted" needs the system to know that "Home Health Care" means the same thing. I expected the fusion of the two to clearly beat either retriever alone. Measured against thirty one hand built questions with known correct clauses, the result was messier than that. The fusion won narrowly when I only looked at the top three results. It tied dense retrieval exactly at five results. It lost outright at ten results, and on ranking quality across the whole set. A method that wins in one narrow slice and loses everywhere else is not a win. I reported it as the mixed result it actually was.

The fusion method combines results by rank rather than by score, which sounds like a technical footnote until you see what it does in this specific corpus. Almost every clause in an insurance document shares words like "policy" and "claim" with almost every question. Keyword search always returns a full ranked list, and the tail of that list is mostly noise wearing a real rank position. A clause ranked twelfth by both retrievers scored higher in the fusion than the actual answer, which the embedding search ranked first and the keyword search did not meaningfully rank at all. The fix was not a better weighting between the two retrievers. Weighting barely moved the number. The fix was to make the keyword search abstain when it had nothing confident to say, dropping any match scoring far below its own best result for that question.

I tuned that abstention threshold once and found a value that scored noticeably higher than its neighbours. I did not keep it. At thirty one questions, a single flipped answer moves the score by about three percentage points. A spike surrounded by a flat plateau is what fitting noise looks like. I kept the value that sat inside the flat, boring, reproducible middle of the range instead.

None of this made the numbers better than I hoped going in. It made them true, which turned out to matter more.

View on GitHub