Citation Fit Check
A real citation can support the wrong answer. Check that the rule applies to the person asking.
Built by Sunil Mathew, co-authored with Claude
- What to bring
- One type of question your RAG assistant answers; the source documents it retrieves from; 10 real or sample questions of that type, each with the facts about the asker that decide the answer; and the retrieved passages and answers for those questions, from traces or by running them.
- What to do
- Map the conditions that gate each rule, spot-check whether retrieval returns those conditions, score the twelve checks, and write test cases where the cited text is true but doesn’t apply.
- What you leave with
- A condition map for that answer type, a Pass / Partial / Fail result on twelve checks with a decision (Ready, Fix first or Hold), and a starter set of applicability test cases for your evals.
Excel workbook · also opens in Google Sheets and LibreOffice · free
From The Living Craft, a live cohort on designing agentic systems. Explore the cohort
The citation checks out. The answer is still wrong.
Sunil was part of the team responsible for People Tech modernization at Walmart. HR answers are where a confident mistake reaches someone’s health and money. RAG assistants add a quiet way to get them wrong: the answer quotes the right policy, the citation checks out, and a condition stated elsewhere in the document means it doesn’t apply to the person asking.
Only one of them shows up in a groundedness eval.
| Failure | What it looks like | Caught by groundedness evals? |
|---|---|---|
| Unsupported | The answer says something the source doesn’t | Yes |
| Misapplied | The answer quotes the source correctly, but a condition on that rule excludes the person asking | No |
The checklist targets the second failure. It asks four questions, one per working tab.
The four questions
| # | Question | Tab |
|---|---|---|
| 1 | Which conditions gate each rule, and where do they live? | 1 · Condition Map |
| 2 | Does retrieval bring back the conditions with the rule? | 2 · Spot Check |
| 3 | Does the system check the asker’s facts before answering? | 3 · Checklist |
| 4 | Would your evals catch a true citation that doesn’t apply? | 4 · Test Cases |
One question each, and the mistake it catches.
Which conditions gate each rule, and where do they live?
The mistake it catches Treating a rule as if it applies to everyone. The condition that narrows it often sits in another section, another document, or only in the HR system, where nobody reading the rule will see it.
Does retrieval bring back the conditions with the rule?
The mistake it catches Counting a retrieval as a hit because the rule came back. A passage that holds the rule and not its condition looks like a hit and produces a wrong answer with a real citation.
Does the system check the asker’s facts before answering?
The mistake it catches An assistant that answers yes when it does not know a fact about the asker, such as a start date or a plan. The twelve checks end in one decision and name the first thing to fix.
Would your evals catch a true citation that doesn’t apply?
The mistake it catches An eval set that scores only groundedness. A misapplied answer is grounded, so it passes. These cases are scored on whether the answer is right for the asker.
Four are critical. One No on any of them is a Hold.
Each check is answered Yes, Partial or No, and scores 2, 1 or 0. A No on any check marked critical means Hold. Otherwise 20 or more out of 24 is Ready, and anything less is Fix first.
Source
- 01
The source documents are the current versions, and the assistant knows each one’s effective date.
- 02
Critical Every rule this answer type relies on has its gating conditions identified (Tab 1).
- 03
Each condition is stored with its rule or linked to it, for example in passage metadata or by retrieving the parent section.
Retrieval
- 04
Critical Retrieval returns the condition along with the rule (Tab 2 condition recall).
- 05
Superseded or conflicting versions of a rule are excluded or flagged.
- 06
Exceptions and overrides to a rule are retrieved with it.
Answer
- 07
Critical The assistant has, or asks for, every asker fact the conditions depend on.
- 08
The answer states the conditions it relied on, not just the rule.
- 09
When an asker fact is missing, the answer is conditional (“it depends on your length of service”) or the assistant asks; it doesn’t default to yes.
- 10
Answers that affect health, money or employment point to a named human contact or the official source for confirmation.
Evaluation
- 11
Critical The eval set includes cases where the cited text is true but doesn’t apply, scored separately from groundedness (Tab 4).
- 12
A named owner re-runs this check when a source policy changes.
An HR benefits assistant, held.
Illustrative example: an HR benefits assistant. Not a real system, employer or real figures. It is filled in on the workbook’s last tab.
4 of 10
Retrieval brought back the rule every time and its condition 4 times. Condition recall: 40.0%.
3
Answers with a real citation that were wrong for the person asking. A reader would trust them and act on them.
Hold
Don’t trust these answers yet. Critical checks 4, 7 and 11 were answered No. Total 10 of 24.
The condition map found 3 rules. 2 of their conditions live away from the rule, and the assistant has no way to check 1: an employee’s start date. The first thing to fix is check 4. One test case is written: someone who joined eight months ago, asking about a pre-existing condition.
A citation tells you the text exists. Check that it applies.
You have examined one operating decision. Build the wider reasoning through practical work and review in The Living Craft’s October 2026 cohort: 30 live hours with Sunil Mathew, plus independent work.
Build a working agentic system and develop the reasoning behind it. Explore The Living Craft’s October 2026 cohort: 30 live hours with Sunil Mathew, plus independent work. Apply at https://learning.thelivingcraft.ai/#apply
Published 7 October 2026 · Content CC BY 4.0 · © The Living Craft