Statistical significance and error margins in performance and growth marketing

"Significant" is a word about noise, not about money

Sooner or later a dashboard tells you a test is "significant" or that a variant wins "with 95% confidence". It sounds like a verdict. It is something narrower: a statement about how surprising your numbers would be if nothing had changed at all. Whether the change is real, how big it is, and whether it pays for itself are three further questions, and the word "significant" answers none of them.

This article is about reading those words properly. What a p value is and is not, how wide the margin of error on a conversion rate really is, why a lift can pass the test and still lose you money, how peeking and slicing quietly break the test, and what a platform means when it prints a confidence figure next to your campaign.

Two other articles do the groundwork I lean on here. How many visitors a test needs, and how to design one, is the subject of statistical testing: sample size, lift and full factorial designs. Whether your traffic and budget are big enough to test at all is in how much data do you actually need. I will not repeat either. This one starts when the results come in.

Throughout I use an invented shop, Coal Quay Candle Co, an online candle maker in Cork run by a woman called Aoife, at coalquaycandles.example. Nobody real. Every number below is made up so I can show every step.

A p value measures surprise, assuming nothing changed

Every test starts from a deliberately boring assumption, the null hypothesis: the new version makes no difference. The p value then asks one question. If that boring assumption were true, how often would random chance alone produce a gap at least as large as the one you saw?

A small p value means "this would be an unusual result in a world where nothing changed". That is all. It is a measure of how badly the data fit the boring assumption, not a measure of anything you care about directly.

For two conversion rates the standard tool is the two proportion z test:

$$z = \frac{\hat{p}_B - \hat{p}_A}{\sqrt{\hat{p}\,(1-\hat{p})\left(\frac{1}{n_A} + \frac{1}{n_B}\right)}}$$

Here $\hat{p}_A$ and $\hat{p}_B$ are the observed conversion rates of the old page and the new one, $n_A$ and $n_B$ are the visitors each version saw, and $\hat{p}$ is the pooled rate, all orders divided by all visitors, which is the best guess at the common rate if the null hypothesis holds. The result $z$ counts how many standard errors apart the two rates are, and the p value is the share of the normal curve that lies at least that far out on either side. The test is built from scratch in Foundations: hypothesis tests.

Worked example: Aoife's new product page

Aoife tests a new product page with larger photos of the candles burning. Each version sees 20,000 visitors. The old page takes 600 orders, the new one 690.

Step one, the two rates:

$$\hat{p}_A = \frac{600}{20{,}000} = 0.0300, \qquad \hat{p}_B = \frac{690}{20{,}000} = 0.0345$$

Here $\hat{p}_A$ is the old page's rate and $\hat{p}_B$ the new page's, each orders divided by visitors. So 3.00% against 3.45%, a gap of 0.45 percentage points, which is a relative lift of 15%.

Step two, the pooled rate:

$$\hat{p} = \frac{600 + 690}{40{,}000} = \frac{1{,}290}{40{,}000} = 0.03225$$

The pooled rate $\hat{p}$ is every order from both versions over every visitor, 3.225%.

Step three, the standard error under the null hypothesis:

$$\sqrt{0.03225 \times 0.96775 \times \left(\frac{1}{20{,}000} + \frac{1}{20{,}000}\right)} = \sqrt{0.031210 \times 0.0001} = 0.0017667$$

The $0.0001$ is $\frac{1}{20{,}000} + \frac{1}{20{,}000}$, and the result is the standard error of the gap if the two pages were truly the same.

Step four, the z score:

$$z = \frac{0.0345 - 0.0300}{0.0017667} = \frac{0.0045}{0.0017667} = 2.55$$

The gap of $0.0045$ is $2.55$ standard errors wide.

Step five, the p value. A z of 2.55 leaves about 0.54% of the normal curve beyond it on each side, so the two sided p value is about 0.011. If the new page truly made no difference, a gap this large would turn up in roughly one test in 90.

That is a genuinely useful sentence. Notice what it does not say. It does not say the new page is 98.9% likely to be better, and it does not say the lift is 15%. Those are the two readings almost everyone reaches for, and both are wrong.

What a p value is not, in the statisticians' own words

In 2016 the American Statistical Association did something it had never done before: it published a formal statement on how p values are misused, with six principles, in its journal The American Statistician.¹ It is short and worth reading in full. In my own words, the six say:

1. A p value tells you how incompatible the data are with a particular model, usually the "nothing changed" model.
2. It is not the probability that your hypothesis is true, and not the probability that the result was produced by chance alone.
3. Business decisions should not rest only on whether a p value crosses a threshold.
4. Honest inference needs full reporting: every test you ran, not only the ones that came out well.
5. A p value does not measure the size of an effect or how much it matters.
6. On its own, a p value is not a good measure of evidence for or against a hypothesis.

Principle two is the one that undoes most marketing reports. The ASA puts it plainly: p values "do not measure the probability that the studied hypothesis is true".¹ A group of seven statisticians led by Sander Greenland catalogued 25 common misreadings in the same year, and the first on their list is exactly this one, reading the p value as the chance the null hypothesis is true.² The p value is calculated by assuming the null is true, so it cannot also tell you how likely the null is.

The old version of this article said that a p value below 0.05 means a 95% chance the result is not due to chance. I wrote that, and it is wrong for exactly this reason. I have corrected it here.

Worked example: how often is a "significant" result real?

To get from a p value to "how likely is this real", you need something the test never sees: how often your ideas work in the first place. This is conditional probability, and it is set out in Foundations: probability.

Say Aoife runs 100 tests over a few years. Assume, for illustration, that only about 10 of the ideas tested truly move conversion. The real share is unknown and varies by shop, which is exactly the point. Her tests are designed with 80% power, so they catch 80% of the real effects, and a 5% significance level, so 5% of the dud ideas come up significant by luck.

• Real effects caught: $10 \times 0.80 = 8$ significant results.
• Dud ideas that slip through: $90 \times 0.05 = 4.5$ significant results.
• All significant results: $8 + 4.5 = 12.5$.

$$P(\text{real} \mid \text{significant}) = \frac{8}{12.5} = 0.64$$

Here $P(\text{real} \mid \text{significant})$ means the probability that an effect is real, given that the test came out significant. With these assumptions it is 64%, not 95%. Roughly one significant result in three is a dud wearing a winner's sash. If her ideas were better, say 30 in 100 real, the share climbs, and if they were worse it falls. The p value alone cannot tell you which world you are in.

The margin of error on a conversion rate is wider than it looks

A conversion rate measured on a sample of visitors is an estimate. Run the same week again with different visitors and you would get a slightly different number. The margin of error says how far the estimate could reasonably sit from the true rate. OpenStax calls it the error bound for a proportion.³

$$E = z_{\alpha/2}\sqrt{\frac{\hat{p}\,(1-\hat{p})}{n}}$$

Here $E$ is the margin of error, $z_{\alpha/2}$ is the number of standard errors you go out either side (1.96 for 95% confidence), $\hat{p}$ is the conversion rate you measured, and $n$ is the number of visitors it was measured on. The square root part is the standard error of a proportion, from Foundations: sampling, and the estimate plus or minus $E$ is the confidence interval in Foundations: confidence intervals.

Worked example: the same 3% on two very different weeks

Aoife's old page converts at 3.00% on 20,000 visitors:

$$E = 1.96 \times \sqrt{\frac{0.03 \times 0.97}{20{,}000}} = 1.96 \times 0.001206 = 0.00236$$

With $\hat{p} = 0.03$ and $n = 20{,}000$, the margin $E$ is 0.24 percentage points. So the true rate is somewhere around 2.76% to 3.24%. Respectable.

Now a quiet week in February: 2,000 visitors, 60 orders, also 3.00%:

$$E = 1.96 \times \sqrt{\frac{0.03 \times 0.97}{2{,}000}} = 1.96 \times 0.003815 = 0.00748$$

With $n = 2{,}000$ the margin $E$ is 0.75 percentage points. Now the honest statement is "somewhere between 2.25% and 3.75%". Same headline number, an interval more than three times as wide. The February figure cannot tell a 2.5% shop from a 3.5% shop, which for Aoife is the difference between a worrying month and a good one.

Along the bottom is the number of visitors the rate was measured on, and up the side is the 95% margin of error in percentage points, either side of a 3% rate. The takeaway is the shape: the margin shrinks with the square root of the sample, so four times the visitors only halves it. Going from 1,000 to 5,000 visitors buys a lot of certainty. Going from 20,000 to 50,000 buys much less.

What "95% confident" actually promises

This is the second misreading I corrected from the old article, which said 95% confidence means you would get the same result 95 times out of 100. It does not. The 95% describes the method, not any one interval. If you ran the same measurement many times and built an interval each time, about 95 in 100 of those intervals would contain the true rate.² Your particular interval either contains it or it does not. You just do not know which.

In practice the reading I use is simple: the interval is the range of true rates that your data cannot rule out. Treat every value inside it as live, and plan on the unflattering end.

Read the interval on the lift, not the verdict

The same logic applies to the gap between two versions, which is the number you actually care about in a test. My sibling article sets out the formula for the interval on the difference, so here I only apply it to Aoife's test.

Worked example: how big is Aoife's lift, honestly

The standard error of the difference, without pooling this time, because the job now is estimating the gap rather than testing for zero:

$$\sqrt{\frac{0.0300 \times 0.9700}{20{,}000} + \frac{0.0345 \times 0.9655}{20{,}000}} = \sqrt{0.000001455 + 0.000001666} = 0.0017666$$

The margin of error on the gap is $1.96 \times 0.0017666 = 0.00346$, so the gap is 0.45 points plus or minus 0.35 points, which is 0.10 to 0.80 points.

In relative terms, dividing by the old rate of 3.00%, the lift is somewhere between 3% and 27%. The headline "plus 15%" sits in the middle of a range that runs from barely worth mentioning to excellent. That range is the true state of Aoife's knowledge. If she plans her Christmas stock on 15%, she is planning on the middle of a wide guess. If she plans on 3%, she will not be caught out, and anything above that is grand.

Significant and still not worth it

Principle five of the ASA statement is the one that costs real money: a p value does not measure the size of an effect or how much it matters.¹ Greenland and colleagues list the opposite belief, that significance means an important effect was found, as a misreading in its own right, and they point out that tiny effects come up significant easily in large samples.²

Big traffic makes this worse, not better. With enough visitors, almost any change is significant. The question then moves from "is it real" to "is it worth what it costs".

Worked example: a personalisation tool in Galway

Take a second invented shop, Salthill Running, a sportswear retailer in Galway at salthillrunning.example. It trials a product recommendation tool that costs €2,500 a month. Each version sees 500,000 visitors. The old site converts at 2.00% and the version with the tool at 2.06%.

The test, step by step. The pooled rate is 2.03%. The standard error under the null:

$$\sqrt{0.0203 \times 0.9797 \times \frac{2}{500{,}000}} = \sqrt{0.019888 \times 0.000004} = 0.00028205$$

Here $0.0203$ is the pooled rate $\hat{p}$ and the $\frac{2}{500{,}000}$ is the two equal arms added together. The z score:

$$z = \frac{0.0206 - 0.0200}{0.00028205} = 2.13$$

A $z$ of 2.13 gives a two sided p value of about 0.033 (the test from Foundations: hypothesis tests). Significant at 5%. The agency report writes itself. Now the money.

The site has 250,000 visitors a month, the mean order is €60, and the gross margin is 40%, so each extra order is worth $60 \times 0.40 = €24$ of margin. The tool pays for itself when it brings at least:

$$\frac{€2{,}500}{€24} = 104.2 \text{ extra orders a month}$$

On 250,000 visitors that is a lift of $104.2 / 250{,}000 = 0.042$ percentage points. Call that the break even line.

The interval on the gap is $0.06 \pm 1.96 \times 0.00028205$, so 0.06 plus or minus 0.055 points, roughly 0.005 to 0.115 points. Turned into money per month:

• Low end: $0.00005 \times 250{,}000 = 12$ orders, $12 \times €24 = €288$, a loss of about €2,200 against the fee.
• Point estimate: $0.0006 \times 250{,}000 = 150$ orders, $150 \times €24 = €3,600$, a profit of €1,100.
• High end: $0.00115 \times 250{,}000 = 288$ orders, $288 \times €24 = €6,912$, a profit of about €4,400.

The test is significant against zero, and zero is the wrong line. The line that matters is 0.042 points, and the interval straddles it. The honest summary is "probably profitable, quite possibly not, and I cannot yet tell". Whether to keep the tool is now a business decision about how much risk Salthill Running is happy to carry, an expected value question rather than a statistical one. What it is not is a done deal because a cell turned green.

The habit I keep is to draw the break even line before the test starts and to compare the interval with that line, not with zero. It changes the conversation more than any formula.

Peeking turns one test into twenty

A p value is calculated for one look, at a sample size fixed in advance. Look every day and stop the first time it dips under 0.05, and you are no longer running a 5% test. Ramesh Johari and colleagues, writing about A/B testing platforms, put it bluntly: the usual p values and intervals are unreliable if people choose when to stop by continuously monitoring their tests.⁴ Evan Miller's well known essay on the same trap works through a case where a "5%" test ends up with a false positive rate of 26.1% from repeated checking.⁵

My sibling article shows the simulation code, so I will not repeat it. I ran my own version for this article so the numbers would match Aoife's shop.

Worked example: Aoife watches the dashboard

Aoife runs an A/A test: both versions are the identical page, both truly convert at 3%, and each sees 1,000 visitors a day for 20 days. Any "winner" is a false positive by construction. I simulated 4,000 such tests and counted how often at least one look showed a significant gap at 5%, depending on how many times she looked.

Along the bottom is how many times the result was checked over the 20 days, spread evenly. Up the side is the share of the 4,000 identical tests that showed a "significant" winner at least once. One look at the end gives roughly the 5% promised (the 6% is simulation noise). A daily look gives about 25%, so one test in four crowns a winner between two identical pages. These are my simulation numbers, not a law, but they agree with what Miller and Johari describe.

There are two cures. Fix the sample size before the test, using the method in Foundations: sample size, and read the result once at the end. Or use a tool built for sequential testing, which spends the error allowance across the looks on purpose. A dashboard that recalculates an ordinary p value every morning is neither.

Slice the result ten ways and something will look significant

Peeking repeats one test over time. Multiple comparisons repeat it across variants, metrics or segments at the same moment, and the arithmetic is the same. Every extra test is another 5% chance of a false alarm.

$$\text{FWER} = 1 - (1 - \alpha)^m$$

Here $\text{FWER}$ is the family wise error rate, the probability of at least one false positive across the whole set of tests, $\alpha$ is the significance level of each single test, and $m$ is the number of tests. It follows from the multiplication rule for independent events in Foundations: probability, and assumes the tests are independent, which segments of one test only roughly are.

The simplest repair is the Bonferroni method: divide the significance level by the number of comparisons, so the whole set keeps an overall error rate of at most the original level.⁶ It is blunt and it costs power, which is why the better habit is to name the one primary metric before the test begins and treat everything else as a lead to investigate, not a result.

Worked example: the Limerick mobile visitors

Aoife's new page shows no overall lift on her next test. Disappointed, she breaks the result down: mobile and desktop, new and returning customers, and six counties. That is ten slices. One shines: "visitors from Limerick on mobile, plus 40%, p = 0.03".

With ten slices and nothing real going on, the chance that at least one shows $p < 0.05$:

$$1 - (1 - 0.05)^{10} = 1 - 0.95^{10} = 1 - 0.599 = 0.40$$

Here $0.05$ is $\alpha$ and $10$ is $m$. A 40% chance of a shining slice from pure noise. Bonferroni sets the bar for each slice at:

$$\frac{0.05}{10} = 0.005$$

That is $\alpha$ divided by $m$. The Limerick result, at 0.03, does not clear it. It is a reason to run a new test aimed at Limerick mobile visitors, if that audience is big enough to matter. It is not a finding. The ASA's fourth principle, full reporting, is the same point in other words: report all ten slices you looked at, not only the one that shone.¹

A platform's "confidence" label is a setting, not a truth

Ad platforms now run their own experiments and print their own verdicts, and each uses its own definitions. Before trusting the label, find out three things: what confidence level it uses, what question the number answers, and whether it was built for repeated looks.

Google Ads is a good case to learn on, because its help page says plainly how it works. Experiment results show a range for the difference between the trial and the original campaign, a blue asterisk marks a statistically significant result, and you can choose your own confidence level, with 80% as the default.⁷ Eighty, not 95. At 80%, one interval in five misses the truth by design. That is a defensible choice for a quick directional read, and a poor one for a decision that moves a large budget.

Meta publishes its own help page on confidence in its tests and experiments.⁸ Other tools report a "chance to win" or a "probability to beat the original", which comes from a different school of statistics (Bayesian) and answers a different question from a p value. None of these is wrong. They are just not interchangeable, and a figure of 70% on one platform does not mean what 70% means on another. Read the platform's own definition before you compare.

Worked example: Aoife's Google Ads bidding trial

Aoife trials a new bidding strategy on her Google Ads campaign. The experiment report shows conversions up 8%, with a range of minus 2% to plus 18% at the default 80% confidence, and no blue asterisk.

To see the same result at 95%, widen the range by the ratio of the two z values. The 80% interval uses $z = 1.28$ and the 95% interval uses $z = 1.96$:

$$\frac{1.96}{1.28} = 1.53$$

The ratio $1.53$ is how much wider a 95% interval is than an 80% one. The 80% range is 10 points either side of 8%. Multiplied by 1.53 that becomes about 15.3 points, so at 95% the range runs from about minus 7% to plus 23%. This assumes the interval is roughly symmetric and normal, which is a fair approximation for a sample this size.

So at the platform's own default the result is already inconclusive, because the range includes zero. At 95% it is very inconclusive: the new strategy might cost Aoife 7% of her conversions or win her 23%. The right reading is "not enough data yet". The wrong one is "plus 8%, roll it out".

How I read any test report

All of this folds into one routine that I run on every result, mine or anyone else's, before a single euro moves.

The routine runs from top to bottom. The first two questions check that the test was run so its numbers mean anything: fixed size, one look, one primary metric. The next two read the interval against zero and then against the line that pays the bills. Only a result whose whole interval clears break even gets shipped without a conversation, and even then I plan on its unflattering end.

The questions to ask when someone shows you a winner

You do not need to do any of the arithmetic yourself to use this. You only need to ask, politely, the questions that the arithmetic answers. When an agency, a tool or your own team shows you a winning test, ask:

1. What was the sample size, and was it fixed before the test started?
2. How often was the result checked, and was the tool built for that?
3. Which metric was chosen in advance, and how many others were looked at?
4. What is the interval on the lift, and at what confidence level?
5. Where is the break even line, and is the whole interval above it?

A good answer to all five is rare and worth trusting. "It hit 95% significance" answers none of them. Asking them costs nothing, and it is the surest way I know to stop the quiet rollout of winners that were never there.

Footnotes

1. American Statistical Association, American Statistical Association releases statement on statistical significance and P values (summarising Ronald L. Wasserstein and Nicole A. Lazar, The ASA's statement on p values: context, process, and purpose, The American Statistician, volume 70, issue 2), 2016, https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf

2. Sander Greenland, Stephen J. Senn, Kenneth J. Rothman, John B. Carlin, Charles Poole, Steven N. Goodman and Douglas G. Altman, Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations, European Journal of Epidemiology, volume 31, pages 337 to 350, Springer, 2016, https://pmc.ncbi.nlm.nih.gov/articles/PMC4877414/

3. Barbara Illowsky, Susan Dean and others, Introductory Statistics 2e, section 8.3 A population proportion, OpenStax, 2023, https://openstax.org/books/introductory-statistics-2e/pages/8-3-a-population-proportion

4. Ramesh Johari, Leo Pekelis and David J. Walsh, Always valid inference: bringing sequential analysis to A/B testing, arXiv 1512.04922, 2015 (revised 2019), https://arxiv.org/abs/1512.04922

5. Evan Miller, How not to run an A/B test, evanmiller.org, 2010, https://www.evanmiller.org/how-not-to-run-an-ab-test.html

6. NIST/SEMATECH, e Handbook of Statistical Methods, section 7.4.7.3 Bonferroni's method, National Institute of Standards and Technology, 2012, https://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm

7. Google, Monitor your experiments, Google Ads Help, 2025, https://support.google.com/google-ads/answer/6318747?hl=en

8. Meta, About confidence in your tests and experiments, Meta Business Help Centre, 2025, https://www.facebook.com/business/help/239549606692303