Priori Signal

Home/Intelligence/Experimentation

All solutions

Experimentation

Help A/B tests reach a result sooner with machine learning.

Customer spending varies widely, which makes revenue tests slow. Predicted lifetime value can account for some differences that existed before the test. This reduces noise in the comparison, helping you reach a reliable result with fewer customers.

Intelligence

A short explanation of CUPAC: predict existing customer differences, keep random assignment, then compare actual outcomes.

Why tests take time

Imagine testing a new offer in a game or subscription product. A few customers will spend much more than others, whatever offer they see. Random assignment keeps the comparison fair on average, but those large customer differences still make the measured group averages noisy. More noise means more customers or more time may be needed to separate the offer's effect from normal variation.

Earlier customer spending

LowerHigher
Existing differences can obscure the effect of the change being tested.

How CUPAC works

Before assignment, a machine learning model uses earlier purchases and activity to estimate each customer's lifetime value. That estimate is a covariate: useful context for the analysis, rather than the experiment's result. Customers are still assigned at random to the usual experience or the new one. Both groups then produce real outcomes.

The analysis accounts for the predicted differences when it compares those outcomes. This approach is called CUPAC. It removes some variation that was predictable before treatment, which can narrow the uncertainty around the measured effect.

Earlier purchases and activity
Predicted lifetime value

Saved before the test for use in the analysis

Customers are assigned at random

A: Usual offer
B: New offer
Compare recorded results, adjusted using the earlier prediction
The prediction uses information fixed before assignment. The experiment then records what each group actually does.

Measure the effect using recorded outcomes

Compare what the two randomly assigned groups actually spent. The earlier prediction helps account for existing customer differences, making the estimated effect of the new offer more precise. Recorded outcomes remain the basis for the result.

Without adjustment

With a useful pre-assignment prediction

The aim is narrower uncertainty around the treatment effect, not a larger effect.

When it helps

CUPAC helps when the prediction explains a useful share of the outcome variation. It may add little when customer history is sparse, the outcome is hard to predict, or the test measures something unrelated to predicted lifetime value. Priori checks the gain rather than assuming every test will finish sooner.

  • Use only pre-assignment information. A feature affected by the treatment can leak the answer into the adjustment and bias the result.
  • Keep random assignment intact. The prediction improves the analysis. It does not decide who receives A or B.
  • Validate out of sample. Prediction quality should be measured on customers the model did not train on, using the same outcome definition as the experiment.
  • Report the effect and its uncertainty. A narrower interval supports an earlier decision only under the decision rule set before the test.
Technical assumptions for a valid effect estimate

Assignment must be random. The covariate must be fixed before assignment and cannot be affected by treatment. The analysis must respect the unit that was randomized and use suitable uncertainty estimates. Under those conditions, the adjustment targets the same average treatment effect as the unadjusted test. Its purpose is lower variance, not a different answer.

Put it to work

The same connected customer history can support the pre-assignment prediction and the final outcome measurement. Start with the decision, the treatment, and one primary outcome. Then define what is known before assignment and test whether the prediction improves precision without changing the meaning of the experiment or its planned decision rule.

See how Priori builds the shared data model behind those records, or how the same predicted value supports value-based acquisition. For a possible change that has not been randomized, use scenario planning instead of calling it proven lift.