rag
Cite or refuse: the rule that keeps RAG honest
We treat hallucination as a failure mode of design, not of the model. Every answer either cites or says no, and the eval set is what enforces it.
Most RAG systems we inherit have one of two failure modes. Either they confidently invent answers because the retriever missed, or they refuse on questions whose answer is actually in the corpus but in a format the retriever can’t see. Both kill trust within a quarter.
The rule we ship every RAG project with is cite or refuse. Every answer the assistant produces either:
- Quotes specific source chunks with attribution that links back to the original document, or
- Says “I don’t know” and surfaces the closest near-miss for a human to review.
There is no third option. No “based on general knowledge”, no graceful hallucination, no plausible-sounding paragraph without sources. The model is instructed to refuse, and the application code enforces it by checking that the structured response includes citations before showing the answer to the user.
The harder half: an eval set that fights you
The rule alone isn’t enough. Without an evaluation harness, the rule silently degrades the first time someone tunes the prompt for “warmer” answers. We build the eval set on day one and run it in CI on every prompt change.
A good eval set has three populations:
- Retrievable questions. Answer is in the corpus; the system should answer with citations.
- Adjacent-but-wrong questions. Answer looks like it might be in the corpus but isn’t; the system must refuse.
- Adversarial questions. Designed to extract a specific known-bad answer (medical advice, legal advice, financial advice, off-policy refunds). The system must refuse.
The eval runs on every prompt change, every retriever change, every model swap. The pass bar is conservative: 100% on the adversarial set, ≥95% citation accuracy on the retrievable set, and ≥98% refusal rate on the adjacent-but-wrong set. Any drop blocks deploy.
Why this matters more than you’d guess
The deeper reason this rule matters: it changes what the system can do.
A “be helpful by default” RAG agent is incompatible with regulated domains, internal compliance docs, customer support where wrong answers cost the company money, and any setting where a confident hallucination looks the same as a correct answer. Once a single confidently-wrong answer ships, the trust budget is gone for months.
A cite-or-refuse system can survive in those settings. It refuses gracefully, it surfaces what it’s unsure about, and it logs the refusals so the team can decide whether to add the missing knowledge to the corpus or accept that the question is out of scope.
That’s the version we ship. The demo is less impressive. The thing is still answering questions accurately three quarters in.