Distilled model rivals top scorers at predicting LLM citations
A tiny open model nearly matches expensive proprietary scorers at predicting LLM citations, and that should change how brands buy GEO tools.
Key takeaways
- A 4B distilled model scores 0.542 to 0.549 Spearman correlation, statistically indistinguishable from 27B models and hosted scorer Jev.
- Ten answer engines given identical candidate sources agree with each other at only 0.704 Spearman correlation, the practical ceiling for any scorer.
- Most of the gap between the small model and Jev vanishes when Jev is restricted to scoring one passage at a time, like the student model.
- Citation-prediction tooling is becoming commoditised: parameter count and vendor pricing are weak proxies for predictive accuracy.
- Buyers should demand disclosed Spearman benchmarks from GEO vendors rather than relying on model size or brand reputation.
Ten different answer engines, fed the identical set of candidate sources for the same query, agree with each other on which pages deserve citation at a Spearman correlation of 0.704. A 27-billion-parameter decision model and Jev, a hosted scorer built for this purpose, get close to that human-agreement-like ceiling at 0.575 to 0.578. The headline result, buried in an arXiv paper titled with the kind of taxonomic precision only researchers love, is that a 4-billion-parameter student model, fine-tuned on teacher labels via low-rank adapters, lands at 0.542 to 0.549. Statistically, across 229 queries, that is indistinguishable from the 27B models and from Jev itself.
This matters more than the percentage points suggest, because of what it costs to run each option.
Why a seven-times-smaller model closing the gap is the real story
Generative engine optimisation, the practice of trying to get your content cited inside an AI-generated answer rather than ranked on a results page, has a measurement problem. You cannot optimise for citation share if you cannot cheaply and repeatedly score which candidate pages a model is likely to cite. Running a 27B parameter judge, or paying for a hosted decision model such as Jev, against every page on a client's site, for every target query, across every quarter, is expensive and slow. A 4B model that gets within a few hundredths of a Spearman point of the big models changes the arithmetic. It makes continuous citation-prediction monitoring something an in-house team or an agency tool can run at scale, not something reserved for one-off audits.
The paper's authors, working with GEO-Bench queries answered across ten engines, found something else worth sitting with: most of the remaining gap between their distilled student and Jev disappears once Jev is forced to score one passage at a time, the same constraint the student operates under. That is a tell. It suggests the advantage of the larger, more expensive models comes partly from seeing multiple candidate passages together and comparing them, not from some irreducible depth of judgment the smaller model lacks. Strip Jev of that context advantage and the playing field flattens. For anyone building GEO tooling, that is the more actionable finding than the raw correlation score: architecture and input framing matter as much as parameter count.
The ceiling that matters more than the model
The 0.704 figure, the agreement rate between engines answering the same query with the same candidates, deserves more attention than it is getting. It is the practical ceiling for any citation-prediction model, because it describes how much the engines themselves disagree even under identical conditions. No scorer, however large, should be expected to beat that number, because it would mean out-predicting the actual behaviour it is modelling. Jev and the 27B models, at 0.575 to 0.578, are already closer to that ceiling than most people building GEO tools seem to assume is possible. The 4B student arriving within a few points of them, rather than lagging by a wide margin, says the task is more tractable, and more commoditisable, than the current crop of paid, closed decision tools would have buyers believe.