---
title: "A/B Test Calculator: Statistical Significance Calculator"
description: "Free A/B test calculator. Enter visitors and conversions for A and B to get the p-value, uplift and confidence interval, plus the sample size you need."
url: https://scratcher.io/ab-test-calculator
language: en
---

Free tool

# A/B test calculator

Check whether the difference between two variants is statistically significant, and work out how many visitors a test needs. Free, no sign-up, and every formula is shown.

## What statistical significance means in an A/B test

In an A/B test you show two versions of something to two random groups of people and compare how many convert. Variant B almost never gets exactly the same rate as variant A, even when the change makes no difference at all. The question is whether the gap is bigger than you would expect from chance alone.

The p-value answers that. It is the probability of seeing a difference at least this large if the two variants really performed the same. A p-value of 0.036 means a gap like yours would turn up by chance in about 3.6% of tests with no real difference. If the p-value is below your threshold (0.05 for a 95% confidence level), the result is called statistically significant.

## How the calculator works

The calculator uses a two-proportion z-test. It pools the conversions of both variants to estimate the rate you would expect if there were no difference, works out the standard error from that, and divides the difference between the two rates by it. The result is the z-score: how many standard errors apart the two rates are. The p-value comes from the normal distribution.

An example: variant A gets 100 conversions from 1,000 visitors (10%) and variant B gets 130 from 1,000 (13%). That is a relative uplift of 30%, a z-score of about 2.10 and a two-sided p-value of about 0.036. B is significantly better at 95%, but not at 99%.

The confidence interval shows the range in which the true difference probably lies. In the example it runs from about +0.2 to +5.8 percentage points. A wide interval tells you the test is not precise yet, even if it is significant.

Running a survey rather than a test? The [survey sample size calculator](https://scratcher.io/survey-sample-size-calculator) tells you how many answers you need.

## Plan the sample size first

Most failed tests are too small. Before you start, decide the smallest change worth acting on, and use the pre-test helper to see how many visitors each variant needs at 80% power. Power is the chance that the test detects an effect of that size if it is really there. Then run the test until you reach that number, and run it for whole weeks so weekdays and weekends are both included.

A few other rules help:

1. **Test one change at a time**, such as the prize, the game format or the number of form fields, so you know what made the difference.
2. **Pick the metric before the test.** Sign-up rate, completion rate and opt-in rate can move in different directions.
3. **Don't peek and stop.** Stopping as soon as the p-value dips below 0.05 makes false winners far more common.

Our guide to [A/B testing gamified campaigns](https://scratcher.io/blog/ab-test-gamified-campaign) shows what is worth testing in a campaign, and the list of [gamification marketing KPIs](https://scratcher.io/blog/top-gamification-marketing-kpis) helps you choose the metric. In Scratcher, the numbers for each campaign are in [analytics](https://scratcher.io/platform/analytics). To see what a better conversion rate is worth in money, use the [campaign ROI calculator](https://scratcher.io/campaign-roi-calculator).

*Statistical significance tells you whether a difference is likely to be real, not whether it is large enough to matter. Look at the uplift and the confidence interval as well.*

Questions

## Questions about A/B test significance

[Book a meeting](https://scratcher.io/book-a-meeting)

### Is the A/B test calculator free?

Yes. It is free to use as often as you like, and you don't need an account. Nothing you type is sent to us.

### What p-value counts as significant?

It depends on the confidence level you choose before the test. At 95%, a p-value below 0.05 is significant. At 90% the threshold is 0.10, and at 99% it is 0.01. A stricter level means fewer false winners but needs more visitors.

### Should I use a one-sided or a two-sided test?

Use two-sided unless you decided in advance that you only care whether B is better and would treat a worse B the same as no difference. A one-sided test reaches significance sooner, but it cannot tell you that B is worse.

### My result is not significant. What now?

It means the test has not shown a difference, not that there is none. Check the pre-test helper: if you have far fewer visitors than needed, keep the test running. If you have reached the planned sample size, treat the variants as equal and test a bigger change.

### Why does the calculator warn about small samples?

The z-test relies on an approximation that only works when each variant has enough conversions and non-conversions, roughly 5 or more of each. With less data, the p-value can be misleading, so collect more before you decide.

## Let the next draw run itself

Run your competition on your own page, collect emails and consent, and have the winners drawn for you. Try it free for 14 days, no credit card required.

[Start free trial](https://platform.scratcher.io/en/register) [Book a meeting](https://scratcher.io/book-a-meeting)
