New benchmark finds GEO defenses cut attack success by just 5.7%
Current LLM safety guardrails are near-useless against adversarial GEO, leaving authoritative brands exposed in retrieval-layer attacks.
Key takeaways
- Current guardrails cut adversarial GEO attack success by at most 5.7% relative — statistically nothing for one of the three tested.
- The attack lands at retrieval, before guardrails evaluate output, making safety-taxonomy defenses structurally mismatched to the threat.
- Multilaterals, financial services firms, and industrial groups are high-value targets precisely because their content is authoritative and widely retrieved.
- Institutions assuming model-level safety layers protect their brand in LLM answers should revise that assumption immediately.
- Retrieval-layer verification, not output guardrails, is the correct defensive investment.
Eighty-three percent of enterprise AI deployments now include some form of content guardrail, according to IBM's 2024 AI in Action report. A new benchmark from arXiv suggests those guardrails are doing almost nothing against one of the most consequential threats in AI search.
Counter-GEO-Bench, a peer-reviewed study pairing 247 human-verified queries with adversarially optimised documents, tested three off-the-shelf defenses against information-distorting generative engine optimization (GEO): Granite Guardian, Llama Guard 3, and NeMo Self-Check Fact-Checking. The best of the three reduced attack success rates by 5.7% in relative terms. Granite Guardian's reduction was not statistically significant at all.
That number should stop anyone in a senior communications or technology role. A 5.7% relative reduction means that if an adversary publishes an ordinary-looking GEO-optimised document, three LLMs will still synthesise the distorted answer into their outputs almost as reliably as if no defense existed. The guardrails in widest deployment are, for practical purposes, decorative.
What adversarial GEO actually does
The threat model is more precise than "misinformation." GEO, as a legitimate practice, helps content producers increase the likelihood that a generative search engine retrieves and cites their pages. The same optimisation signals, applied to false or distorted content, make that content more retrievable and more likely to appear in synthesised answers. The adversary does not need to compromise the LLM. They need only publish a plausible-looking document that scores well on the retrieval heuristics the model favours.
Counter-GEO-Bench isolates this cleanly. It pairs each query with two rewrites: one that preserves factual content while optimising for retrieval, and one that distorts it. The study then measures whether current defenses can distinguish the distorting variant from the benign one before the answer is generated. They largely cannot.
This matters structurally for any institution whose authoritative content competes in the same retrieval pool as adversarial documents. A multilateral like UNDRR or CGAP does not face state-level disinformation because it lacks credibility. It faces it precisely because it has credibility, and because an adversary who can push a distorted version of a CGAP financial-inclusion finding into an LLM's synthesised answer gains the halo of the institution without any of the fact-checking.
The retrieval layer is the attack surface, not the model
The safety-taxonomy guardrails tested by Counter-GEO-Bench were designed for a different problem: blocking outputs that are toxic, illegal, or obviously harmful. They were not designed to detect subtly distorted factual claims dressed in professional language. That mismatch explains the 5.7% figure as clearly as any technical analysis could.
LLMs operating as generative search engines retrieve before they generate. The attack lands at retrieval. By the time a guardrail evaluates the output, the distorted claim is already woven into the synthesised answer, and the defense is checking whether the synthesis sounds dangerous rather than whether the underlying retrieved document was accurate. It is the equivalent of installing a smoke alarm and expecting it to prevent arson.