What statistical significance means in an A/B test
In an A/B test you show two versions of something to two random groups of people and compare how many convert. Variant B almost never gets exactly the same rate as variant A, even when the change makes no difference at all. The question is whether the gap is bigger than you would expect from chance alone.
The p-value answers that. It is the probability of seeing a difference at least this large if the two variants really performed the same. A p-value of 0.036 means a gap like yours would turn up by chance in about 3.6% of tests with no real difference. If the p-value is below your threshold (0.05 for a 95% confidence level), the result is called statistically significant.
How the calculator works
The calculator uses a two-proportion z-test. It pools the conversions of both variants to estimate the rate you would expect if there were no difference, works out the standard error from that, and divides the difference between the two rates by it. The result is the z-score: how many standard errors apart the two rates are. The p-value comes from the normal distribution.
An example: variant A gets 100 conversions from 1,000 visitors (10%) and variant B gets 130 from 1,000 (13%). That is a relative uplift of 30%, a z-score of about 2.10 and a two-sided p-value of about 0.036. B is significantly better at 95%, but not at 99%.
The confidence interval shows the range in which the true difference probably lies. In the example it runs from about +0.2 to +5.8 percentage points. A wide interval tells you the test is not precise yet, even if it is significant.
Running a survey rather than a test? The survey sample size calculator tells you how many answers you need.
Plan the sample size first
Most failed tests are too small. Before you start, decide the smallest change worth acting on, and use the pre-test helper to see how many visitors each variant needs at 80% power. Power is the chance that the test detects an effect of that size if it is really there. Then run the test until you reach that number, and run it for whole weeks so weekdays and weekends are both included.
A few other rules help:
- Test one change at a time, such as the prize, the game format or the number of form fields, so you know what made the difference.
- Pick the metric before the test. Sign-up rate, completion rate and opt-in rate can move in different directions.
- Don't peek and stop. Stopping as soon as the p-value dips below 0.05 makes false winners far more common.
Our guide to A/B testing gamified campaigns shows what is worth testing in a campaign, and the list of gamification marketing KPIs helps you choose the metric. In Scratcher, the numbers for each campaign are in analytics. To see what a better conversion rate is worth in money, use the campaign ROI calculator.
Statistical significance tells you whether a difference is likely to be real, not whether it is large enough to matter. Look at the uplift and the confidence interval as well.
