A popup, a bar, an onsite notification: an onsite campaign earns its place by adding conversions you would not have had otherwise. That value is easy to overstate.
A popup can take credit for a sale that was coming anyway. The buyer was already heading to checkout, the popup appeared on the way, and it books the win.
What would have happened if the campaign had never run?
The gap between that and what actually happened is what the campaign added. Everything else is credit-taking.
You get that number by holding back a control group that sees nothing, then comparing. It comes out smaller than the headline figures, and it holds up under scrutiny.
The goal of this report is simple: to show what onsite campaigns are actually worth, using real control-group experiments, and how you can measure your own.
Because every one held back a control group, the numbers show what a campaign actually caused, not what merely happened after it. That is uncommon, and it is what makes them worth trusting.
How a campaign's value gets measured
When someone says a campaign worked, they mean one of three things. Two describe how it performed. Only the third shows whether it created value the site would not otherwise have had.
A popup shown 20,000 times gets 1,200 clicks: a 6% engagement rate. But interaction is not value. Many of those people would have bought anyway, so a click does not prove the campaign caused the sale.
For example, variation B beats A by 58%. A real and useful result for picking a design, but it only compares two campaigns to each other. Both could still be losing to a page that shows nothing.
You measure the people who saw the campaign against a control group that saw nothing. This is the only one of the three that shows whether the campaign created value that would not have happened anyway.
What a control group is
A control group is a random slice of your visitors that you hold back from the campaign. They see nothing; everyone else sees it as normal.
Because the two groups are alike in every other way, any gap in results between them can only come from the campaign. That gap is its incremental value.
A control-group test can run two ways. The simplest is a split: a random slice of visitors is held back and sees nothing, while everyone else sees the campaign.
A three-way test goes further, running version A against version B against nothing, so you learn the better version and whether either beats doing nothing, in one test.
Anyone can show you an engagement rate. The only number that matters is what you would have lost by doing nothing, and measuring that is the whole point of a control group.
What we found
247 experiments across sites using Wisepops have compared a campaign against a control group that saw nothing, including 74 three-way tests. Here is what they show.
In plain terms, the honest gain from a campaign is modest and steady, well below the engagement rates that get celebrated. It is real value, and it is the number worth planning around.
When a holdout test reached a clear verdict, the campaign beat showing nothing about 86% of the time, and the median gain on the campaign's goal was roughly +20%.
That gain is more modest than a version-versus-version win, and it depends on what you measure. Conversion gains ran from the low single digits to about +8%; revenue gains ran higher but noisier.
A +20% incremental gain is smaller than a headline A/B win, and a more useful one: it is the value the campaign actually created.
How to run your own control-group test
Four rules for a control-group test that produces a number you can trust.
- Hold back a real control groupSet aside a random slice of traffic that sees no campaign. Even 5% to 10% is enough on high-traffic pages; the bigger the holdout, the faster you reach confidence.
- Measure on revenue or conversionPick the outcome that carries real value, like sales or signups, and hold both groups to the same one.
- Give it enough traffic and timeIncremental effects are smaller than version-versus-version gaps, so they need more data to separate from noise. Plan beyond two weeks for lower-traffic goals.
- Report the incremental number, not the vanity oneLead with the result versus nothing. It is lower than the engagement rate, and it is the one that survives scrutiny.
How we measured this
A short report, held to the same standards as the rest of the series.
Population
Campaign-scope experiments across sites using Wisepops since January 2024 that included a no-popup control group, identified by a held-out control variation.
There are 247 such experiments, 173 two-way popup-versus-nothing and 74 three-way, of which 204 concluded. All figures are aggregated and anonymized.
Definition
Incremental value, sometimes called incremental uplift, is the increase for the group that saw the campaign over a control group that saw nothing, measured on the primary goal.
The roughly +20% median comes from holdout tests that reached a clear verdict, still a small sample, so treat it as directional.
Limits
Conversion gains are more conservative, about +1% to +8% across the middle range. Revenue gains run higher but noisier.
Relation to the A/B report
This is not the version-versus-version median from the A/B testing report, which compares one popup against another rather than against nothing. The two answer different questions and are never combined.