A/B test significance calculator
Enter the visitors and conversions from a finished test. The calculator shows whether the difference between two versions is real, or whether it needs more traffic before you can tell.
| Variant | Visitors | Conversions | Conversion rate | Remove |
|---|
Enter your test data
Visitors and conversions for the control and at least one variant. Results update as you type.
How much better or worse each variant really is
How these numbers are calculated
Significance test
Two-proportion z-test against the control, with unpooled standard errors so the interval and the verdict cannot disagree.
Confidence interval
Calculated on the absolute difference, then divided by the control rate to read in relative terms.
Observed power
The probability of detecting an effect the size of the one measured, at this sample size and confidence level.
Multiple comparisons
Sidak correction across k comparisons against the control, holding the test-wide false positive rate at the level selected.
Sample ratio mismatch
Chi-square goodness of fit on the visitor counts against an even split, flagged below p = 0.001.
Limits
- Fixed-horizon test. Set the sample size in advance and evaluate once. Stopping at the first significant reading inflates false positives beyond the stated level.
- Binary outcomes only. Revenue per visitor, average order value and items per order are skewed by large orders and need different methods.
- The normal approximation needs roughly 5 conversions and 5 non-conversions per variant as a floor.
- Significance is not effect size. A significant result can still be too small to be worth shipping.
- Sizing a test before you launch it is a different calculation, on the sample size calculator.
Related reading
How to A/B test popups: steps, ideas and examples
Setting up a popup test end to end, including control groups that show what a campaign adds on top of what visitors would have done anyway.
Read the guideA/B testing for CRO: best practices, examples and steps
Which areas to test first, how to write a hypothesis worth running, and what to do with the result once the test concludes.
Read the guide15 A/B testing ideas for ecommerce to drive conversions
Fifteen tests worth running on a store, from checkout flow and product page layout to offer type and campaign timing.
Read the guideGet started
with Wisepops
Book a demo to get a personalized audit of your traffic.