RAG passes hallucination checks while citing the wrong entity
Real citations, wrong entity: a new failure mode in RAG is invisible to every standard quality check and hits hardest in domain-specialist models.
Key takeaways
- RAG systems can attribute Drug Y's clinical evidence to Drug X while passing every hallucination and faithfulness check.
- Deceptive grounding rates reached 87% under adversarial conditions across 13 models in a controlled benchmark.
- Medical and biomedical fine-tuned models hit 86.7%, meaning domain specialisation amplifies the failure.
- The mechanism is entity-adjacent retrieval: real, relevant documents about the wrong subject.
- For brands and institutions, a correct citation no longer guarantees that the attributed claim is actually theirs.
Automated hallucination checkers gave RAG a clean bill of health. They were checking the wrong thing.
A paper published on arXiv this week introduces a failure mode that existing evaluation frameworks cannot detect. The researchers call it deceptive grounding: a RAG system retrieves real documents, cites them accurately, and produces responses that pass faithfulness and hallucination checks at near-perfect rates, while attributing the clinical evidence of one drug entirely to a different drug. Every claim is sourced. Every source is real. The entity is wrong.
The benchmark ran across 13 models under controlled factorial conditions. Deceptive grounding rates ranged from 8% to 87% at peak adversarial conditions. Medical and biomedical fine-tuned models, the ones most likely to be deployed in high-stakes clinical settings, reached 86.7%. Domain specialisation, in other words, made the problem worse. The models most trusted by practitioners were the most susceptible.
The mechanism is not a bug in retrieval
The ablation study identified the trigger with some precision. When entity-specific clinical evidence was removed from the retrieved document set, entity-attribution failure disappeared. The model was not confusing drug X with drug Y out of ignorance. It was following the evidence it was given, faithfully, to the wrong conclusion. The retrieval step selected plausible, topically relevant documents. The generation step used them correctly. The evaluation step confirmed everything looked fine. The error lived in the gap between topical relevance and entity-specific accuracy, a gap no standard metric was designed to cross.
This is a structurally different problem from hallucination. Hallucination produces claims unsupported by any document. Deceptive grounding produces claims supported by documents that are about the wrong subject. Faithfulness metrics measure the first; they do not touch the second.
What this means for any organisation that runs RAG at scale
The clinical framing is deliberate, and the stakes in that setting are obvious. But the underlying architecture is not clinical-specific. RAG is now the default pattern for enterprise knowledge retrieval across financial services, multilateral research systems, and large industrial groups. A financial institution deploying RAG over regulatory filings faces a structurally identical risk: a system that attributes the capital adequacy evidence for Bank A to a query about Bank B, passes every internal quality check, and presents the answer with citations that all resolve correctly. The auditor who checks the citations finds real documents. The analyst who trusts the answer acts on the wrong entity's data.
For multilaterals and UN agencies producing policy guidance, the failure mode is particularly corrosive. These institutions rely on citation chains as the mechanism of institutional credibility. A RAG system that cites real IPCC documents while applying one country's emissions trajectory to another's profile does not look broken. It looks authoritative.