ClauseCheck: Insurance Policy Q&A (RAG)

Project Snapshot

Key Insight

Hero Architecture Diagram

The Problem

A person who wants to know whether a treatment is covered has already paid for the document that says so. Getting the answer out of it is the hard part. The source here is a 291-page health insurance prospectus written in cross-referenced legal prose, with exceptions nested inside exceptions and waiting periods stated in units nobody speaks in.

The visible problem looks like search. Ask a question, find the clause, write the answer. That framing survives about ten minutes of contact with the document.

Two facts break it. The first is that this is not one policy. It is eight plan variants sharing a single set of clauses, so most questions have a different correct answer depending on the plan. Room rent is billed at actuals under six of the plans, capped at a percentage of the base sum insured under the seventh, and limited to a single private room under the eighth. One headline benefit is worth the full base sum insured under one plan and double that under another whose name differs by a single word. A system that retrieves the right clause but drops the plan qualifier does not fail loudly. It gives a confident, well-cited, wrong answer about money.

The second fact is that the answer people trust most is the one they cannot check. A citation is trusted precisely because verifying it is work. An answer citing a section the model never saw is fluent, authoritative, and wrong, and it reads exactly like a correct one. In a domain where the cost of a confident error is a denied claim, the naive approach, retrieve and generate, optimises the one thing that hides the failure: how right the answer sounds.

That is the constraint the architecture is built around. Not "find the clause." It is that a wrong answer is indistinguishable from a right one at the surface, so correctness has to be made checkable underneath the fluency.

Requirements & Constraints

Functional Requirements

• Answer a plain-English question grounded in the policy text, with the exact clause cited.

• Preserve the plan qualifier, so a plan-specific answer is never collapsed into a single wrong one.

• Refuse when the policy does not contain the answer, rather than guessing.

• Expose a machine-readable status and a citation-verified verdict, so a caller can branch on a refusal or an unverifiable answer without parsing prose.

Non-Functional Requirements

• Run end to end on a clone, with no API key and nothing to sign up for.

• Deterministic evaluation, never a model grading a model, since the only model on hand is the one under test.

• An index build that fails loudly on the corruption modes that would otherwise pass silently.

Latency is deliberately not a hard requirement here, and that is a scoping choice worth naming. The default local model answers a question in 38.5 seconds, which is fine for a correctness-first prototype and unshippable for an interactive product. That figure is the cost of the zero-setup path, not of the architecture: the same pipeline on the hosted demo, with a faster hosted model and the index kept warm in memory, answers a question in roughly 0.7 to 1.4 seconds once the free-tier host is awake (its first request after idle takes around 37 seconds, almost all of it the free-tier host spinning up its container rather than the system answering). The trade the local default makes is correctness and reproducibility over speed: an open-weights model needs no key and runs on a clone, but pays for it in wall-clock time. Retrieval and verification logic are identical across both, so latency is a deployment choice, not a rewrite.

Hidden Constraints

The document corrupts its own structure before the model ever sees it, and it does so silently. Only about 71 of the 291 pages carry policy prose. Around 220 are premium rate charts. The one table that says what each benefit is worth per plan is the one table the PDF text layer destroys, collapsing eight columns into a single vertical stream that decouples every value from its plan. A registration footer repeats on every page and outweighs real prose in raw term frequency. None of this announces itself. Each failure produces artefacts that look correct on disk while being wrong in a way no downstream prompt can recover. The invisible production reality is that knowledge quality is lost during ingestion, not retrieval, so the preprocessing is where most of the engineering had to go.

High-Level System Design

The system splits cleanly into an offline build and a per-question path.

Offline: the PDF is extracted with frequency-based boilerplate stripping, chunked on the document's own clause numbering, and the plan comparison matrix is transcribed by hand into one chunk per benefit. Chunks are indexed twice: dense embeddings in a FAISS index, and a BM25 index over stemmed terms. The build refuses to finish if it finds a duplicate chunk id or a chunk longer than the embedding window.

Per question: the question is embedded and matched against both indexes. Dense and sparse results are fused by reciprocal rank fusion, with a signal gate that lets BM25 stay silent when it has nothing confident to say. The top five distinct clauses go to a local model, which is instructed to answer only from those clauses, cite the section it used, or refuse. Before the answer is returned, every cited section is checked against the clauses actually placed in the model's context. The response carries the answer, a status of answered / not in document / out of scope, and a citations-verified flag.

The two boxes that make this more than a demo are the last two. Refusal is the model's judgement rather than a score threshold, and every citation is verified rather than trusted.

Engineering Decisions

Chunk on clause boundaries, not fixed windows

Context

The unit of retrieval determines whether a retrieved chunk can actually answer a question. Fixed-size windows are the default in most RAG pipelines.

Alternatives Considered

Final Decision

Chunk on the document's clause numbering. A fragment reading "shall be excluded until the expiry of 24 months," severed from the sentence saying what is excluded, retrieves confidently and is useless. Clause boundaries avoid that, and they buy a second property for free: the string that identifies a chunk is the string a person uses to find it in the PDF. The citation is not bolted on. It is the chunk's own name. The accepted cost is a parser that has to understand the document's numbering, which turned out to hide the worst bug in the project.

Engineering Principle

The unit of retrieval should be the unit of citation. If they differ, every answer inherits an alignment problem.

Transcribe the plan matrix by hand, and test the transcription

Context

One table states what each benefit is worth under each of the eight plans. It is the only place the document answers plan-specific money questions, and it is the one table PDF extraction destroys.

Alternatives Considered

Final Decision

Type the matrix out by hand, one chunk per benefit, each plan's value beside that plan's name. Hand-transcription trades one risk for another: a typo here is a wrong answer about money, which is worse than a miss. So a test checks every transcribed value against the source page it claims to come from, converting the manual risk into a guarded one.

This is the honest scaling ceiling of the project. The frequency-based boilerplate stripping and clause chunking transfer to any insurer's document unchanged, but the single most valuable data, the plan-specific money values, is manual per document. The plan-specific correctness that the hard-question results rest on does not come for free across a corpus; it costs a person a few hours per new policy. Automating it safely, table extraction that fails loud instead of near-missing, is a real piece of work this project does not claim to have done.

Engineering Principle

When the automated path silently decouples data from its meaning, a tested manual path is not a shortcut, it is the safer engineering choice.

Hybrid retrieval, with BM25 allowed to abstain

Context

Insurance language rewards two different kinds of matching. Paraphrase needs meaning; exact terms need tokens. Neither retriever alone covers both.

Alternatives Considered

Final Decision

Fuse dense and sparse by RRF, and gate BM25 so it drops hits scoring below 40% of its own top hit for that query. The honest reading of the numbers is that hybrid does not beat dense on this question set. It is one question ahead at recall@3 (83.9% vs 80.6%), ties at recall@5 (both 87.1%), and loses at recall@10 and on MRR. On a 31-question set every 3-point move is one question, so none of those gaps is a "win"; they are ties inside the noise. Both are kept anyway, because the failure modes are decorrelated: BM25 alone finds clauses dense misses, and the union of the two covers 29 of 31 questions. The question set is written in natural policyholder voice, which structurally favours paraphrase and therefore dense; a real query log would carry more exact-term lookups where BM25 earns its place. The gate value 0.4 was chosen for the mechanism, not tuned: 0.3, 0.4 and 0.6 all measured 87.1% while 0.5 measured 90.3%, and a lone spike in a flat band is one question out of 31 flipping, not an optimum.

Engineering Principle

Report the result you measured, not the one you wanted, and refuse to tune a constant until your own test set smiles back. Keep a component for its decorrelated failures, not for a headline it did not earn.

Refusal is the model's judgement, not a score threshold

Context

The system must know when a question has no answer in the document. The cheap design refuses before spending a model call: if no clause scores above a similarity threshold, refuse.

Alternatives Considered

Final Decision

Measure whether the threshold exists first. It does not. The 31 answerable questions scored 0.341 to 0.647 (median 0.546); the 3 unanswerable ones scored 0.479 to 0.640. Any threshold high enough to reject all three unanswerable questions also rejects 29 of the 31 answerable ones. The cause is structural: "what is the claim settlement ratio" is a real insurance question phrased in the document's vocabulary, so it embeds right next to genuine clauses. Retrieval similarity measures topical relatedness; answerability is a different property, and no threshold on the first recovers the second. So refusal moved to the model as an explicit, allowed outcome. On the eval, it refused all 3 unanswerable questions and wrongly refused only 1 of 31.

Engineering Principle

A plausible heuristic is a hypothesis. If you can test it in twenty minutes, you have no excuse to ship it untested.

Verify every citation in code, do not trust the prompt

Context

Letting the model decide when to refuse means trusting the model. That is only safe with something underneath it that does not depend on trust.

Alternatives Considered

Final Decision

Check every cited section against the clauses actually placed in the model's context. If the model cites a section it was never shown, the citation is flagged unverified. This is a hard check in code, not a request in the prompt, because a fabricated citation reads exactly as confident as a real one and cannot be told apart by reading. It matters more with a small local model than a frontier one, precisely because it does not depend on the model cooperating. On the eval, 34 of 34 citations were verified; not one was invented. What that number proves is exact and narrow: every cited section was actually shown to the model. It does not prove the cited clause supports the claim, and the confabulation-on-miss case in Limitations is a verified-but-wrong answer that passed this guard. "Verified" means shown, not supported.

Engineering Principle

Guard the failure a reader cannot catch for themselves, and guard it with a check that does not assume the thing it is checking will behave.

One-stage constrained generation, after the two-stage version lost

Context

While testing, constraining the model to a structured JSON object seemed to hurt its answers. The clearest case was room rent: the model returned "the document does not specify" with all eight plan values sitting in its context. The apparent fix was a two-stage pipeline: reason in free text, then extract structure in a second constrained call.

Alternatives Considered

Final Decision

Compared fairly against a single-stage baseline with the same instructions, across all 34 questions, two-stage lost: citations verified 100% vs 61.8%, correct clause 90% vs 76.7%, seconds per question 38.5 vs 59.7. One-stage fabricated zero citations; two-stage fabricated on 13 of 34. The cause is specific: free-text reasoning reaches for fine-grained references like "4.6.iv.b" that the corpus indexes only at section level and cannot verify, and the extraction step faithfully copies them into the answer. The room-rent anecdote that motivated the whole detour was a prompt bug, missing field guidance on the baseline, not a property of constrained decoding. The default is one-stage; the two-stage code is kept behind a flag so the comparison reproduces, and a test guards the default so it cannot silently flip back.

Engineering Principle

An intuition from a handful of anecdotes is worth exactly one fair experiment. Keep the losing branch reproducible, because the reversal is the evidence.

Implementation Highlights

The index build fails loudly. It refuses to finish on a duplicate chunk id or a chunk exceeding the embedding window. Both were real, silent bugs: a duplicate id means a citation points at the wrong clause, and an over-window chunk is truncated without warning, so it looks correct in every artefact while being partly invisible to the retriever. A build that crashes on either is worth more than one that looks clean and lies.

Boilerplate is detected by frequency, not regex. A line recurring on 15% or more of pages is treated as furniture and stripped. The threshold sits in a gap found in the data: real prose lines never recur above roughly 3%, the footer recurs at 20% and 80% because it is typeset in two line wraps, and the band between holds only rate-chart rows. Detecting furniture by frequency instead of a hardcoded pattern lets the boilerplate-stripping step transfer to other insurers' documents. The hand-transcribed plan matrix does not transfer the same way: that part is manual per document and is the project's real scaling limit.

One backend interface, three backends: a local open-weights model through Ollama is the default and needs no key. An OpenAI-compatible backend drives the hosted demo, and an Anthropic backend exists as an optional frontier baseline. Nothing in the pipeline depends on the frontier one.

The API surface is the trust surface. The single /ask endpoint returns the answer alongside a status and a citations-verified flag, so a caller can act on a refusal or an unverifiable citation without reading the prose. 49 tests guard the parts that would otherwise fail silently: chunking, the hand-transcribed matrix, the generation guards, and the API contract.

Challenges & Debugging

Challenge: the maternity clause came back wrong

The system could not correctly say that maternity is not covered in the first year. The model was not at fault. The clause was already destroyed before the model saw it.

Investigation

The failure surfaced only by asking the actual question and reading the answer with suspicion, not from a red test. The maternity exclusion is one clause with sub-points numbered in lowercase roman numerals: i, ii, and so on.

Root Cause

The chunker read the document's numbering and cut one chunk per lettered clause, walking a, b, c. The roman numeral "i." inside a clause looked, to the parser, exactly like a lettered clause "i". Nine clauses across the document collided onto that one fake id, silently overwriting each other, and the surviving text was not the maternity exclusion at all.

Resolution

The fix was sequence, not a smarter regex. A real lettered clause is always the next letter after the previous one; a nested roman list restarts at "i" inside every clause. The parser now accepts a letter only if it is the one that should come next. Nine clauses recovered their content.

Lesson

A green test suite proves the cases you thought to write. In a domain where one wrong answer is expensive, some bugs are only found by asking the real question and distrusting the fluent reply.

Challenge: fusion scored below its own best component

Investigation

Equal-weight RRF measured 83.9% recall@5, below dense alone at 87.1%. Fusion underperforming its best input is a bug, not a tradeoff, so it needed a cause, not a weight.

Root Cause

RRF fuses ranks and discards magnitude. Nearly every chunk in an insurance corpus shares a term like "policy" or "claim" with nearly every question, so BM25 always returns a full ranked list whose tail is noise carrying real rank positions. Junk ranked #12 by both retrievers scores 2/72 = 0.0278 and beats a gold clause ranked #1 by dense alone at 1/61 = 0.0164.

Resolution

Not a fusion weight; weighting barely moved the number. The fix was to make BM25 abstain when it has nothing to say, dropping sparse hits below 40% of its own top hit for the query.

Lesson

When a combination underperforms its best part, the problem is usually in what the combiner ignores, not in how it is weighted.

Challenge: torch segfaulted on the first real batch encode

Investigation

Encoding segfaulted with exit 139, but only on real multi-threaded work, not on short strings, which made it look flaky rather than deterministic.

Root Cause

On macOS/ARM, faiss-cpu and torch each vendor their own OpenMP runtime. Loading both into one process is undefined, and if faiss's copy loads first, torch segfaults the moment it dispatches real work.

Resolution

src/retrieval/dense.py imports sentence_transformers before faiss so torch's runtime wins, guarded with an isort fence so a sorter cannot reorder the two lines. In hosted-demo mode, query embedding is offloaded to an API, torch is never imported, and the ordering does not apply.

Lesson

A non-deterministic-looking crash is often a deterministic ordering bug wearing a disguise.

Results & Evaluation

Measured on 34 hand-built questions: 31 answerable with known gold clauses, plus 3 deliberate refusals. Every figure below is a deterministic check, set membership and status, never a model grading a model.

One caveat governs how to read every number here. With 31 answerable questions, one question is worth about 3 points, so any gap under roughly 3 points is a single question moving and should be read as a tie, not a result. The claims worth trusting are the large, structural ones, dense far ahead of sparse, refusal thresholds that overlap, one-stage's citation verification, not the sub-3-point deltas between close retrievers.

Retrieval Quality

Ablation is like-for-like: all three retrievers are deduplicated by section, so k means k distinct clauses.

Recall@5 by difficulty for hybrid: easy 85.7%, medium 72.7%, hard 100%. The hard questions are the plan-specific ones, where a single-answer response is wrong for at least two of the eight plans. Their 100% is the hand-structured plan matrix working. The headline is reported honestly: hybrid does not beat dense on this set; it is kept for decorrelated coverage, not for a win.

Hybrid's lower MRR (0.678 vs 0.723) means its gold clause tends to sit slightly lower in the ranking than dense's does. That would matter if a single top hit were the answer, but all five retrieved clauses go to the model, so rank order within the top five costs nothing here as long as the gold clause is in the set. It is a real ranking-quality regression that the architecture makes tolerable, not one to wave away.

Generation Quality

The refusal threshold that does not exist

The distributions overlap almost entirely. This table is the evidence that refusal cannot be a score threshold and had to become the model's judgement.

One-stage vs two-stage generation

Two-stage fabricated citations on 13 of 34 questions; one-stage on zero.

Limitations

• The evaluation set is small. n = 34 carries roughly 3 points of resolution per question, and the generation run is a single pass, not an average. Nothing is validated on a held-out split, and at this size it should not pretend to be.

• The generator still confabulates when retrieval misses. On a question about the order in which the policy spends benefit buckets, the answering clause was never retrieved, and the model assembled a plausible order from neighbouring clauses rather than refusing. Its citations were real sections, so the citation guard passed. The guard proves a section was shown, not that the claim is supported by it. That gap is real and not closed.

• Very short clauses are hard to retrieve. An exclusion reading, in full, "Treatment taken on outpatient basis" never says "not covered," so a question about routine doctor visits has almost nothing to match on and misses.

• Unit paraphrase defeats both retrievers. "I have had the policy for six years" does not retrieve a clause written as "sixty continuous months," because neither retriever does arithmetic.

• Premium questions are out of scope by design. The 220 pages of rate charts are deliberately not indexed. Premium lookup is a structured query, not a semantic one, and embedding those tables would flood the index. This is a documented refusal, not a gap.

What Production Would Still Need

The 49 tests are development-time guards. They prove the pipeline is correct on a clone; they say nothing about what happens after deployment, and this project has not had to answer that.

Named plainly so the gap is not mistaken for coverage:

• No monitoring or drift detection: nothing tracks refusal rate, verification-failure rate, or latency in production, so a regression after a model or index change would be silent. Those same eval metrics would become live alerts.

• Reindex is manual and full. A policy update means rebuilding the whole index by hand. There is no trigger, no versioning of the index against the source PDF, and no incremental path yet (that one is in Future Improvements).

• Cost is unmodelled. The free-tier hosted demo is sized for a portfolio link, not load. Per-question cost and a real rate limit are unaddressed.

• PII and security are unaddressed. A real policyholder question can carry personal and health data. Question logging, retention, and redaction were out of scope here and would be table stakes for anything real.

None of these is hard given the architecture; all are simply undone, and calling them undone is more honest than a monitoring paragraph the code does not back.

Lessons Learned

Lesson 1: knowledge quality begins before embeddings

The document destroyed its own structure during extraction, silently, in ways no downstream prompt could recover. Most of what looked like a retrieval problem was a document problem. Generalised: in any RAG system, the ceiling on answer quality is set at ingestion, so measure and guard the corpus before touching the retriever.

Lesson 2: the signals you would naturally trust do not measure what you need

Retrieval score did not measure answerability. Fluency did not measure correctness. A clean test set rewarded a fusion bug. Each intuitive proxy had to be replaced by a direct, deterministic measurement of the property that actually mattered. Generalised: name the property you care about, then check you are measuring that property and not a correlate of it.

Lesson 3: a plausible fix is a hypothesis, not a decision

The score-threshold refusal gate and the two-stage generator both felt obviously right, and both were wrong under measurement. A twenty-minute check killed the first before it shipped; a full fair run reversed the second. Generalised: the cheap habit of measuring a fix against the whole set before shipping it is what separates a fix that feels right from one that is.

Lesson 4: guard the failure the user cannot catch

A fabricated citation is invisible to the reader by construction. So the system verifies citations in code rather than asking the model nicely. Generalised: put your hard guarantees on the failures that are silent and expensive, not the ones a user would notice anyway.

Future Improvements

• Close the confabulation-on-miss gap. When the answering clause is not retrieved, push the generator harder toward "not in these clauses" instead of assembling an answer from neighbours. This is the one open correctness hole.

• Expand and hold out the question set. Grow well beyond 34 questions, weighted toward exact-term lookups that reflect a real query log, and hold out a split before quoting any tuned number.

• Claim-level grounding, not just citation-level: the current guard proves a citation was shown; a stronger check would verify the specific claim against the cited clause, closing the gap the confabulation case exposes.

• Incremental reindexing: rebuild only changed clauses when a policy is updated, rather than the whole index.

Technology Stack

Definition of Done

• What problem was solved? Getting a straight, correctly-qualified answer out of a 291-page, eight-plan policy, with the exact clause it came from, and an honest refusal when the policy has no answer.

• Why this architecture? Because a wrong answer is indistinguishable from a right one at the surface, so correctness had to be made checkable underneath: clause-level citation anchors, a hand-tested plan matrix, model-owned refusal, and code-verified citations.

• What principle should the reader remember? The signals you would naturally trust to judge quality, retrieval score, fluency, a clean test set, do not measure the property you actually care about. Measure the property directly, and follow the measurement when it contradicts you.