What happens when you ask 8 AI models the same buying question every month

The Setup

A while back I got annoyed at a specific genre of blog post: "we asked ChatGPT what the best CRM is and here's the answer." One screenshot, one run, treated as if the model holds a stable opinion. It doesn't. So I built a small harness to measure that instead of hand-waving about it.

The setup is boring on purpose. Eight models. Sixteen B2B software categories (CRM, project management, email marketing, that kind of thing). For each category I ask every model the same plain question: what is the single best tool here. One pick, no hedging allowed. I log the raw response, the parsed pick, the model, the timestamp, and the exact prompt into a JSONL file. Then I do it again next month.

What Fell Out

Two things fell out that I did not fully expect.

First, across all sixteen categories the eight models never once agreed on the same tool. Not close-but-different. Zero unanimous picks out of sixteen. I assumed there would be at least a couple of categories where everyone converged on the obvious incumbent. Nope.

Second, and this is the one I keep chewing on: the models do not even agree with themselves. Ask the same model the same question in a fresh session and it swaps its own top pick around 74% of the time. Same model, same prompt, nothing changed but the session. Roughly three-in-four odds it contradicts what it told you yesterday.

The Open Question

Here is the part I want other people to break. I do not know how much of that self-disagreement is temperature and how much is genuine ranking instability. My instinct is that dropping temperature to 0 will not actually fix it, it will just hide the wobble behind a deterministic-looking facade while the underlying ranking stays mush. But I have not run that ablation cleanly yet, and I would rather someone who does eval work for a living tell me I am wrong.

Reproducibility

The reason I am posting the method and not only the numbers: a "models disagree" claim is worthless if you cannot reproduce it. So everything is open. The per-run JSONL, the prompt text, the model list, all of it, DOI'd under CC-BY so you can cite it or tear it apart:

https://data.deepsynthesis.org/

I re-run the whole thing monthly, which means you can watch a specific category drift over time rather than trusting a frozen screenshot from whenever the author happened to hit send.

Implementation Notes

If you want to build your own version:

• Force a single pick. If you let the model give you a ranked list it will hedge and you lose the signal. One tool, no runner-ups, is the whole trick.
• Parse defensively. Models love to answer the question and then immediately undercut it with "but it depends on your needs." Strip that.
• Store the raw response next to the parsed pick. You will second-guess your parser and you will want the original text.
• Timestamp everything and never overwrite a run. The monthly diff is the actual product.

The Takeaway

The uncomfortable takeaway for anyone doing AI-search or GEO work: if a model's pick is this unstable, "we rank first when you ask ChatGPT" is a coin that lands differently every flip. Optimizing for a single snapshot is optimizing for noise. Optimizing to show up across many runs and many models is the only thing that survives.

If you poke at the data and find a category where the parser is wrong, or a model I should add, tell me. That is the point of putting it out in the open.