What is a CRO holdout?

How does a CRO holdout work?

Each visitor is assigned at random, before any decision is made about them, into the holdout or the treated group. The holdout is observed exactly like everyone else but never shown anything. Comparing the two conversion rates gives the incremental lift.

The order matters more than the percentage. Assignment must happen *before* the system decides whether to act, because the moment the software picks who to help it has also picked who it is being compared with. A group assembled after the fact — visitors who happened not to receive an intervention — is not a control; it is a record of the system's own judgement.

The assignment also has to be stable per visitor for the length of their journey. Someone reassigned mid-session would sit in both groups, which contaminates both.

Why not just compare before and after?

Because the world moves. Seasonality, traffic mix, a campaign, a competitor's price change and a slow week all shift conversion rates by more than most optimizations do, and a before-and-after comparison attributes every one of those to the change.

The other tempting comparison — treated visitors against untreated ones, without random assignment — fails for a subtler reason, and it is the one that produces most of the impressive numbers in this industry. If the software chose which visitors to treat, it chose them because they looked promising, or stuck, or close to converting. Those visitors were already different. The gap between the groups measures that difference at least as much as it measures the help.

A randomly assigned holdout removes both problems at once, which is why it is worth the cost.

What does a holdout cost?

The share of visitors in it forgo whatever the system would have done for them. At 10 percent, that is a tenth of the potential gain — a real cost, and the price of knowing whether the other ninety percent gained anything at all.

It is worth stating plainly rather than glossed, because the alternative on offer is usually a system that reports improvement indefinitely without being able to show its work. A number you cannot check is worth less than a slightly smaller number you can.

The size is a trade-off between cost and how quickly the comparison converges: a smaller holdout is cheaper and slower to reach confidence, a larger one the reverse. Ten percent is a common settling point for continuously-acting systems.

How is a holdout different from an A/B test's control group?

Duration and scope. A test's control exists for the length of that test and covers one change. A holdout runs continuously and covers everything the system does — which is what a system that acts continuously and changes its own behaviour requires.

A system evaluated by a test that ended in March is being described by a measurement of software that no longer exists, since it has been learning ever since. Its control has to run for as long as it does.