The AI Disagreement Index
The first open, rigorous, living measurement of how much the AI engines disagree on which tools to trust, per category, over time, with the receipts. Not a ranking. A record of where the machines cannot agree.
Everyone else ranks brands. We measure where the engines fall apart.
The incumbents, Profound and Evertune and the rest, tell a vendor its generic "share of voice" across AI answers: one blended score, one direction, brand-level. Useful for a marketing team. It is still a ranking.
This measures something none of them publish: cross-engine disagreement on tools. For a given category we ask several AI engines the same buying question and record who each one names. The headline is not the winner. It is the spread: how far apart the engines land, which tools only one model has ever heard of, and how that picture changes when a model stops reciting training memory and starts searching the live web.
Three commitments make it a research object rather than a listicle. It is open: the raw recorded answers are retained as JSONL and the standings are reproducible on challenge. It is rigorous: consensus is a computed statistic, Fleiss' kappa and pairwise agreement, not a vibe. And it is temporal: we re-capture monthly, so you can watch the AI change its mind.
Rank-first tools
- One blended visibility score
- Brand-level, generic
- Snapshot, sold to the vendor
- Closed methodology
The Disagreement Index
- Cross-engine consensus and spread
- Tool-level, per category
- Living, re-captured monthly
- Open JSONL, DOI, reproducible
One category at a time. Each with its own recorded sample.
The Index is a network of per-category measurements. Each one is a separate capture with its own engine set, its own dated snapshot, and its own retained data. Open one to see the full standings and the receipts.
Research-grade, so it holds up when someone checks the math.
The same buying question is put to several AI engines per category. Every answer is recorded verbatim, then rolled up into who each engine names. Cross-engine means at least three of the engines actually answered, never a single-engine artifact.
Each capture carries the month it was taken. Standings are point-in-time. Monthly re-capture builds the temporal record, so drift, a model quietly changing its recommendation, is itself a measured signal.
The raw answers are retained as JSONL and the roll-up is deterministic. Any published standing can be reproduced from the recorded file. The consensus statistics, pairwise agreement and Fleiss' kappa, are computed from that same data.
A category is only published when the recorded sample is deep enough to be defensible, and a tool only earns a standing when named by at least three of the engines queried. Below the floor, we stay silent rather than fabricate.
The honesty floor, load-bearing
- Frame, never claim. A standing reads "named by N of the engines in our recorded sample," a checkable frame, never "the #1 tool," an unfalsifiable claim.
- True recorded standing only. Every figure is generated from the recorded data, so it cannot overstate. Where a cohort has no data, no standing is minted.
- Minimum-engine threshold. A tool must be named by at least three engines to appear. Thin niches are skipped, not padded.
- Paid never edits earned. A sponsored featured spot is labelled and structurally separate. Buying visibility never changes a recorded standing.
Full methodology and the underlying dataset are packaged as a citable research object: DOI 10.5281/zenodo.20767877, published under ORCID 0009-0005-6869-308X with byline Vincent Wesley Couey, licensed CC-BY-4.0.
If the engines recommend you, prove it, honestly.
Any vendor that clears the floor for a category earns a free Verified by Lattice badge stating its true recorded standing. The badge text is generated from the recorded data, so it can state exactly what the engines said and nothing more.
Vendors embed it on their own site. The badge links back here, to the category it was earned in, which is how the standing stays checkable: click through and see the receipts. The earned badge is a gift, not a purchase.
Want a labelled featured spot alongside the index instead? You can claim or defend your featured spot. A featured spot is always marked as sponsored and is structurally separate from the earned ranking. Earned is not paid, and paid never edits earned.
[category] · recorded [month] · data.deepsynthesis.org
Placeholder copy. Live badges carry the exact recorded count for the vendor and category, generated from the retained JSONL.
Where each engine sends you, side by side.
The interactive layer plots who each AI engine trusts per category and how those picks drift month over month. It is fed directly by the recorded dataset.
Interactive visualization loading soon
The live divergence map and drift timeline connect here once the dataset layer is wired. Until then, open any index above to see the full recorded standings and receipts.