Before You Bet on Synthetic Customers, Test Their Predictions

Match the evidence to the decision

An AI-generated customer response can serve several purposes. A team might use it to suggest questions for interviews, compare possible messages, or estimate how a particular customer will respond to an offer. Each purpose asks something different of the system.

A preprint submitted September 12 by Oded Netzer and Rajan Sambandam distinguishes generic AI responses, segment personas, and twins grounded in individual human data. These approaches should not be treated as interchangeable. The authors also explain why matching an overall survey result can conceal poor predictions for individual respondents. [3]

Their empirical study reconstructs withheld survey answers from other information about respondents. It does not test actual purchases, subscription renewals, or the announced Qualtrics platform. The paper remains preliminary; the selected methods and results reviewed support a narrower conclusion about survey augmentation. [3]

The management implication is to specify the claim being tested. If a team wants to choose an offer for individual customers, matching an overall average is insufficient evidence for that use. If the purpose is to generate ideas for further research, a narrower evaluation may be appropriate.

Before a demonstration, write down the decision, the population, the outcome to predict, and the error that would change your choice. That gives the evaluation a business purpose.

Keep a route back to real customers

For an initial evaluation, reserve fresh customer evidence that was not used to build or tune the simulation. Ask the research team to explain how that separation will be maintained and what the comparison can establish.

A survey can help assess stated preferences. A carefully designed customer trial can examine behavior under a specific offer. Choose evidence that matches the intended claim, and ask qualified researchers to design the comparison. A stated preference should not silently become a forecast of purchases. When results disagree, investigate whether the cause is simulation error, uncertainty in the human evidence, or a difference in what the two exercises measure.

Inspect results for the customer groups that matter to the decision. Ask whose experience is well represented in the underlying information and whose is thin or missing. Record those limits beside the prediction so that a confident presentation cannot conceal a narrow evidence base.

These are proposed management practices, not a published standard or a validated evaluation framework. The appropriate study design and acceptable error depend on the decision and the consequences of getting it wrong.

Make one offer the test case

Consider a hypothetical subscription business comparing two renewal offers. Its commercial leader wants to know which offer produces the better contribution after discounts and service costs while meeting an agreed retention target. The research owner defines the eligible customer groups and the period over which outcomes will be measured.

The team records the simulation's predictions before it sees results from a limited customer trial. It also records what the existing decision process would recommend. Where feasible, a researcher designs a randomized comparison so the team can better distinguish the offer's effect from differences among customers.

When results arrive, the team compares predictions with observed renewals and contribution after the relevant costs. It examines whether the simulation chose the better offer and whether its errors would have changed the business decision. It also checks the groups it identified before the trial. The researcher assesses uncertainty before the team treats a small observed difference as decisive.

Suppose the simulation favors one offer overall but performs poorly for newer customers. The appropriate response could be to restrict its use for that group, collect more evidence, or revise the approach. The team should document the disagreement before expanding the rollout.

This scenario is illustrative. It is not a reported deployment or evidence of a financial return. Its purpose is to make the approval decision observable: what did the simulation predict, what happened, and what does that comparison permit us to do next?

Assign ownership before expanding use

In a September 11 assessment, Forrester analyst Colleen Fazio argues that fragmented customer information, divided ownership, and weak governance remain obstacles to Qualtrics's vision. Her commentary provides an organizational perspective, rather than an independent performance evaluation. [4]

For a pilot, name a commercial owner who is accountable for the decision and a research owner who can challenge the evidence. Have the data owner establish which information is appropriate for the agreed purpose. Decide who can pause use when results diverge from expectations.

Give the simulation an explicit level of authority. Early work might inform a research agenda. A later evaluation might support choosing candidates for a customer trial. Personalized offers or automated actions would require evidence and controls suited to those uses. Success at one level should prompt a review before authority expands.

Maintain a short decision record: the question, information used, prediction, observed result, limitations, and approval. Revisit that record when the customer population, offer, or model changes.

Budget for learning and verification

The investment case should include access fees, preparation of customer information, research expertise, validation studies, and ongoing monitoring. Ask whether the proposed approach improves a meaningful decision enough to justify that full commitment.

Retain funding for human research during the evaluation. That budget provides the evidence needed to assess predictions and investigate surprises. Removing it before validation would make it harder to know whether the system deserves more responsibility.

Define the next funding decision before the pilot begins. Expansion might require useful performance on the chosen outcome, acceptable results for priority customer groups, and a workable process for handling disagreement. A limited or unsuccessful result should still produce a clear decision about what to change or stop.

The sources reviewed do not establish that the announced platform improves commercial decisions or can replace human customer research. They do give executives a timely reason to specify the evidence they will require. Before approving a larger commitment, ask the team to show where the simulation has earned influence over the decision.

Questions for executives

1. Which decision will the simulation inform, and what result would change our choice?
2. What fresh human evidence will test the prediction for the customer groups that matter?
3. Who can restrict or stop its use, and have we funded the validation needed to make that judgment?