Home/Intelligence/Experimentation
All solutions
Experimentation
Help A/B tests reach a result sooner with machine learning.
Customer spending varies widely, which makes revenue tests slow. Predicted lifetime value can account for some differences that existed before the test. This reduces noise in the comparison, helping you reach a reliable result with fewer customers.
Intelligence
Why tests take time
Imagine testing a new offer in a game or subscription product. A few customers will spend much more than others, whatever offer they see. Random assignment keeps the comparison fair on average, but those large customer differences still make the measured group averages noisy. More noise means more customers or more time may be needed to separate the offer's effect from normal variation.
Earlier customer spending
How CUPAC works
Before assignment, a machine learning model uses earlier purchases and activity to estimate each customer's lifetime value. That estimate is a covariate: useful context for the analysis, rather than the experiment's result. Customers are still assigned at random to the usual experience or the new one. Both groups then produce real outcomes.
The analysis accounts for the predicted differences when it compares those outcomes. This approach is called CUPAC. It removes some variation that was predictable before treatment, which can narrow the uncertainty around the measured effect.
Saved before the test for use in the analysis
Customers are assigned at random
Measure the effect using recorded outcomes
Compare what the two randomly assigned groups actually spent. The earlier prediction helps account for existing customer differences, making the estimated effect of the new offer more precise. Recorded outcomes remain the basis for the result.
Without adjustment
With a useful pre-assignment prediction
When it helps
CUPAC helps when the prediction explains a useful share of the outcome variation. It may add little when customer history is sparse, the outcome is hard to predict, or the test measures something unrelated to predicted lifetime value. Priori checks the gain rather than assuming every test will finish sooner.
- Use only pre-assignment information. A feature affected by the treatment can leak the answer into the adjustment and bias the result.
- Keep random assignment intact. The prediction improves the analysis. It does not decide who receives A or B.
- Validate out of sample. Prediction quality should be measured on customers the model did not train on, using the same outcome definition as the experiment.
- Report the effect and its uncertainty. A narrower interval supports an earlier decision only under the decision rule set before the test.
Technical assumptions for a valid effect estimate
Assignment must be random. The covariate must be fixed before assignment and cannot be affected by treatment. The analysis must respect the unit that was randomized and use suitable uncertainty estimates. Under those conditions, the adjustment targets the same average treatment effect as the unadjusted test. Its purpose is lower variance, not a different answer.
Put it to work
The same connected customer history can support the pre-assignment prediction and the final outcome measurement. Start with the decision, the treatment, and one primary outcome. Then define what is known before assignment and test whether the prediction improves precision without changing the meaning of the experiment or its planned decision rule.
See how Priori builds the shared data model behind those records, or how the same predicted value supports value-based acquisition. For a possible change that has not been randomized, use scenario planning instead of calling it proven lift.