A/B test sample size calculator.
Work out how many visitors each version needs before you launch, and how long the test has to run at your traffic. Sizing a test first is what stops it ending without an answer.
Traffic needed at other uplifts
| Expected uplift | Per variant | Total visitors | Days |
|---|
Already ran the test? Check whether the result holds
How these numbers are calculated
Sample size
The same equation solved for visitors per variant, given a baseline rate, the uplift you want to detect, your confidence level, and statistical power fixed at 80%. Power is the chance of spotting a real effect of that size, and 80% is the planning convention, so it is set rather than exposed as an option.
Multiple comparisons
Sidak correction across k comparisons against the control, holding the test-wide false positive rate at the level selected.
Limits
- Fixed-horizon test. Use this figure as the sample size, run to it, and evaluate once on the significance calculator. Stopping at the first significant reading inflates false positives beyond the stated level.
- Binary outcomes only. Revenue per visitor, average order value and items per order are skewed by large orders and need different methods.
- The normal approximation needs roughly 5 conversions and 5 non-conversions per variant as a floor.
- Significance is not effect size. A significant result can still be too small to be worth shipping.
Related reading
How to A/B test popups: steps, ideas and examples
Setting up a popup test end to end, including control groups that show what a campaign adds on top of what visitors would have done anyway.
Read the guideA/B testing for CRO: best practices, examples and steps
Which areas to test first, how to write a hypothesis worth running, and what to do with the result once the test concludes.
Read the guide15 A/B testing ideas for ecommerce to drive conversions
Fifteen tests worth running on a store, from checkout flow and product page layout to offer type and campaign timing.
Read the guideGet started
with Wisepops
Book a demo to get a personalized audit of your traffic.