Synthetic personas vs. real buyer panels
The argument gets framed as real versus fake. That framing is wrong and it costs teams money. The useful question is which error you can afford this week.
Two methods, opposite failure modes
- A panel fails on stated preference. People report what confuses them accurately. They predict their own behaviour badly. They say they prefer headline A and click headline B.
- A simulation fails on thin evidence. Compiled from too little, it returns a fluent, well-structured answer about nothing, and nothing in the output warns you.
Both are survivable. Neither is survivable if you do not know which one you are exposed to.
| Real buyer panel | Synthetic persona | |
|---|---|---|
| WHAT YOU LEARN | WHAT YOU LEARN | |
| What it measures | What people say when asked | What a modelled role does when reading |
| Unit of analysis | The individual, segmented afterwards | The persona, then the committee graph on top |
| Committee handoffs | Not covered. Respondents answer alone | Seven chains, scored healthy, at risk or broken |
| Quotable verbatims | Yes. The strongest artefact for an internal argument | No. Findings cite evidence rather than a speaker |
| Can surprise you | Yes. A real person brings context no corpus holds | Only within the evidence it was built from |
| HOW IT RUNS | HOW IT RUNS | |
| Cost per iteration | High enough to ration. You will not run eight versions | Low enough to iterate freely |
| Turnaround | Hours to weeks, depending on audience | Minutes |
| Repeatable | No. New people, new variance each time | Yes, if the system is deterministic. Many are not. Ask |
| Main risk | Stated preference drifts from real behaviour | Thin or biased evidence produces confident nonsense |
Four questions for any synthetic persona vendor
Most of the gap between a useful simulation and an expensive random-answer generator sits in these four. Ask us too.
- What is it built from, specifically? "Trained on B2B data" is not an answer. Name the sources, and show the source on each finding.
- Is it deterministic? Ask them to run the same page twice. If the verdict moves, you cannot test an edit, because you can never separate your change from the model.
- What is the published hit rate, and where are the misses? Ask whether predictions were sealed before the outcome existed, who graded them, and whether misses stayed in the denominator. See evidence ledger.
- What does it refuse to claim? A vendor who cannot name a limitation has not found one yet.
Three questions for a panel vendor
- How are respondents verified? "Works in IT" is not a buying committee role.
- Can I reach the roles that block deals? Security reviewers and economic buyers are the hardest to recruit and the most likely to kill your deal.
- What is my sample once I segment by role? Fifty people across five roles is ten per role. Ten is not many when roles disagree.
What a simulation cannot do
Worth being blunt, since we sell one.
- It cannot replace talking to customers. It is built from customer evidence, so it sits downstream of it.
- It cannot give absolute numbers. Rank and direction are defensible. A precise conversion rate is not, and one wrong magnitude sinks every correct finding beside it.
- It inherits the bias of its corpus. Public reviews skew to the delighted and the furious. The quiet middle is under-represented.
- It cannot surprise you from outside its evidence. A real person sometimes says the thing nobody modelled. Worth paying for.
The order that costs least
- Simulate. Kill the structural failures across many versions. Missing proof for a role, no baseline number, a broken scent trail. Iterating costs almost nothing.
- Panel. Spend real money on the question humans answer best: does this land, in these words, with this person.
- A/B. The only method that measures what actually happened.
Teams who swap the first two spend panel budget discovering the page has no security signal. Never pay a human for that finding.
Common questions
Are synthetic personas better than real buyer panels?
Neither is better. They fail in opposite directions. A panel measures stated preference, which drifts from behaviour. A simulation built on thin evidence returns confident nonsense. The useful question is which error you can afford this week.
What should I ask a synthetic persona vendor before buying?
What it is built from, specifically. Whether it is deterministic, so the same input returns the same verdict. What the published hit rate is, whether predictions were sealed before the outcome existed, and whether misses stayed in the denominator. And what the vendor refuses to claim.
Can synthetic personas replace customer interviews?
No. They are built from customer evidence, so they sit downstream of it. If that evidence is thin the simulation is fluent and empty, and nothing in the output signals it.
In what order should I run simulation, panel and A/B testing?
Simulate first to kill structural failures cheaply across many versions. Panel second, on the survivor, for the resonance question humans answer best. A/B last, because it is the only method that measures what actually happened.