What does 'statistically significant' actually mean?

It means the difference between your two conversion rates is unlikely to be pure chance. At 95% confidence a significant result has a p-value below 0.05 — meaning there's less than a 5% probability of seeing a difference this large if the variant were really no better than the control.

Is statistical significance the same as 'the variant is better'?

No. Statistical significance says the difference is real. Practical significance asks whether the difference is big enough to matter. A 0.1% uplift on a huge sample can be highly significant statistically and completely worthless in dollars — always check both.

Why report the p-value AND confidence intervals?

The p-value gives a single yes/no verdict on the null hypothesis. The confidence intervals show the plausible range for each rate — if the two intervals barely overlap, the effect is likely real and roughly the size you're seeing. If they overlap heavily, the observed uplift is well within noise.

A/B Test Significance Calculator

Q: One-tailed or two-tailed?

Use two-tailed by default. It answers 'is the variant different from the control?' — which is what you actually want to know, because a variant that hurts is just as important to detect as one that helps. Use one-tailed only if you genuinely do not care about a negative outcome.

Q: How long should I run the test?

Long enough to reach the sample size your minimum detectable effect requires — usually at least a full business cycle (7–14 days) to smooth out day-of-week and time-of-day effects. Stopping the moment p < 0.05 (peeking) inflates the false-positive rate; commit to a sample size before you start.

Enter control and variant visitors + conversions to test whether the observed uplift is real. Reports the two-proportion z-test, p-value, relative uplift and a bar chart with confidence intervals so you can see interval overlap at a glance.

Control (A)

VisitorsTotal people shown the control.ConversionsHow many of them converted.

Variant (B)

VisitorsTotal people shown the variant.ConversionsHow many of them converted.

Confidence levelHigher confidence = stricter significance threshold.Test typeTwo-tailed answers 'is B different from A?' — the safe default.

What is A/B test significance?

When you run an A/B test, the variant almost always looks a bit better or worse than the control just by chance. Statistical significance answers the specific question: "If the two versions were really equal, how often would we see a gap this large by pure luck?" — the p-value. A small p-value (traditionally < 0.05) means chance alone would rarely produce such a gap, so the variants really are different.

Statistical significance vs. practical significance (the trap)

They are not the same thing. Statistical significance is a claim about noise; practical significance is a claim about business impact. With enough data a 0.1% uplift can be highly significant and worth nothing. Write down your minimum detectable effect (MDE) before the test, and ship only when both the p-value AND the observed effect clear their bars.

A/B testing math explained, piece by piece

Each card below owns one part of the calculation — the z-test, the pooled standard error, per-arm confidence intervals, and the one-vs-two-tailed choice — with a plain-English explanation, the exact formula, a diagram and a worked example.

Two-proportion z-test

The z-test asks: if the two variants really had the same conversion rate, how surprising would it be to see the gap we observed? The z-statistic is 'observed difference in standard-error units', and the p-value is the probability of a gap this extreme (or worse) by chance.

z = (p̂_B − p̂_A) / SE

p̂_A, p̂_Bobserved conversion rates·SEstandard error of the difference (pooled)·zstandardised distance in SE units

Example

Given

p̂_A = 9.0%

p̂_B = 10.4%

SE ≈ 0.00592

Substitute

(0.104 − 0.090) / 0.00592

Answer

z ≈ 2.36 → p ≈ 0.018 (significant at 95%)

Pooled standard error

Under the null hypothesis (both rates equal), the best estimate of the shared rate is the pooled proportion. The pooled SE combines that shared rate with the two sample sizes and gives you the SE the z-statistic needs.

p̂ = (x_A + x_B) / (n_A + n_B) · SE

= √[p̂(1−p̂)·(1/n_A + 1/n_B)]

p̂pooled proportion·x_A, x_Bconversions in each arm·n_A, n_Bvisitors in each arm

Example

Given

x_A = 450/5000

x_B = 520/5000

Substitute

p̂ = 970/10000

= 0.097 → SE

= √(0.097·0.903·(2/5000))

Answer

SE ≈ 0.00592

Confidence intervals for each arm

The chart whiskers use each arm's own Wald CI so you can see the plausible range for the true rate. Non-overlapping (or barely overlapping) intervals almost always match p < 0.05; heavy overlap almost always means 'not significant'.

p̂ ± z* · √[p̂(1−p̂) / n]

z*1.96 at 95%, 1.645 at 90%, 2.576 at 99%·nsample size for that arm

Example

Given

p̂_A = 0.090

n = 5000

Substitute

0.090 ± 1.96·√(0.09·0.91/5000)

Answer

≈ [7.4%, 10.6%]

One-tailed vs two-tailed

Two-tailed asks 'is B different from A?' — either direction. One-tailed asks 'is B better than A?' — one direction only. Use two-tailed unless you truly do not care about the variant being worse; discovering harm is usually as important as discovering benefit.

two-tailed p = 2·[1 − Φ(|z|)] · one-tailed p

= 1 − Φ(z)

Φstandard normal CDF·αsignificance threshold (1 − confidence)

Example

Given

z = 2.36

Substitute

two-tailed: 2·[1 − Φ(2.36)]

Answer

p ≈ 0.018

Reading the chart

Bars are the observed rates; whiskers are per-arm 95% Wald intervals. Non-overlapping (or barely overlapping) whiskers almost always match a significant p-value; heavy overlap almost always means "not significant". The p-value is the definitive answer — the chart is the intuition.

How long should I run the test?

Long enough to detect the smallest uplift you'd care about, and at least one full week (ideally two) to smooth out weekday / weekend traffic. Two anti-patterns: peeking (checking daily and stopping the moment p < 0.05 inflates false positives) and stopping too early. Use the Sample Size Calculator to size the test up front.

Common mistakes

Peeking and early-stopping. Pre-commit to sample size.
Testing many variants without a correction. Multiple comparisons inflate false positives.
Chasing "significance" on a trivial effect. Compare to your MDE.
Under-powered tests. "No difference" from an under-sized sample really means "we don't know" — check CI width.
Deciding before running the numbers. Let the p-value and CI decide.

Features of this calculator

Two-proportion z-test with pooled standard error — the standard test for comparing two conversion rates
Selectable 90%, 95% or 99% confidence, plus one- or two-tailed testing
Reports absolute difference, relative uplift, z-statistic, p-value and the critical value in one pass
Bar chart with per-arm confidence-interval whiskers, colored green when significant
Confidence interval for the difference — crossing 0 means not significant
Show/hide step-by-step working: pooled proportion, SE, z, p-value and decision rule
Copy the summary or download the whole panel — chart, verdict and steps — as an image

Frequently asked questions

+Is a p-value of 0.05 a magic threshold?

No — it's a convention. Use 99% (α = 0.01) when a bad decision is expensive, 90% (α = 0.10) when it's cheap and reversible.

+What's the difference between absolute and relative uplift?

Absolute uplift is p_B − p_A in percentage points. Relative uplift is (p_B − p_A) / p_A. Relative is usually more meaningful for business decisions.

+Can I trust the result if the CIs barely overlap?

Usually yes — barely overlapping 95% CIs almost always match significance at 5%. The p-value is the definitive answer.

+One-tailed or two-tailed?

Two-tailed unless you truly do not care about the variant being worse.

+What if my conversion rate is 0% or 100%?

The z-test breaks down at the boundaries. Collect more data or use an exact method (Fisher's exact test).

Related calculators

Sample Size Calculator Z-score Calculator P-Value Calculator Confidence Interval Calculator