The complete guide to digital marketing testing methods

The method follows the question, not the tool

Every campaign, landing page and subject line is a guess about what people will do. A test turns the guess into evidence. The trouble is that "test" covers very different methods, and each answers a different question. Pick the wrong one and you get a confident answer to a question you never asked.

This guide walks through each method, from not testing at all to geographic lift tests: what it is, when it fits, what it needs and where it breaks. Every business and every number in the examples is made up, so the arithmetic can be shown in full. The formulas link to their sections on the Foundations page.

Not testing is sometimes the right call

Not testing is a choice like any other. You launch a change to everyone, with no control group, and watch what happens. There is no comparison and no measure of what the change itself caused.

That is the right call more often than people admit:

• Speed matters more than precision. A sale that runs for a day is over before a test could finish and the winner could be used. Launch your best guess and measure the total.
• The change is mandatory. A legal requirement or a platform policy has to happen whatever it does to conversion. A test would only tell you how much it cost.
• Traffic is too thin. At 50 visitors a month, no method reaches a reliable answer in a useful time. Talking to five customers teaches you more than a test that cannot finish.
• The stakes are tiny. A wording change on an internal tool used by 10 people does not need an experiment.

The risk is not testing by habit rather than by decision, and then crediting the launch when numbers rise. Seasons, competitors and plain chance move numbers too. Without a control group running at the same time, you cannot separate cause from coincidence.

Worked example: a 48 hour sale at a Galway fashion shop

Shop Street Threads (shopstreetthreads.example) is an invented Galway clothing shop that runs a 48 hour sale twice a year. It has three ideas for the announcement email.

1. A proper three way test would need several days to read and act on. The sale lasts two.
2. So the shop sends the concept that did best in last spring's sale to the whole list.
3. It measures total sale revenue against the last sale, and writes down that it cannot prove this was the best of the three.

A sound decision, made knowingly. Just never call the result proof.

An A/B test compares two versions on randomly split traffic

An A/B test compares a control (A) with one change (B). Each visitor is assigned at random and sees only one version. At the end you compare the two rates and ask whether the gap is larger than chance would easily produce.

Random assignment is the whole trick. It makes the two groups alike in everything except the version, so a difference in outcome can be pinned on the version.

The test statistic

The classical test for two conversion rates is the two proportion z test:

$$z = \frac{\hat{p}_B - \hat{p}_A}{\sqrt{\hat{p}\,(1-\hat{p})\left(\frac{1}{n_A} + \frac{1}{n_B}\right)}}$$

Here $\hat{p}_A$ and $\hat{p}_B$ are the conversion rates you measured in A and B, $n_A$ and $n_B$ are the numbers of visitors in each, and $\hat{p}$ is the pooled rate, all conversions divided by all visitors. The square root is the standard error of the difference, so $z$ counts how many standard errors apart the two rates are. A $z$ beyond 1.96 either way gives a two sided p value below 0.05. Foundations: hypothesis tests

Four settings go into the plan before the test starts. The null hypothesis says A and B convert the same. The significance level $\alpha$, usually 0.05, is the false positive rate you accept. Power, usually 80%, is the chance of detecting a real effect of the size you care about. And that size is the minimum detectable effect, the smallest lift worth finding. What a p value does and does not mean is the subject of the importance of statistical significance and error margins.

How many visitors you need

$$n = \frac{(z_{\alpha/2} + z_{\beta})^2 \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_1 - p_2)^2}$$

Here $n$ is the number of visitors per version, $p_1$ is the current conversion rate, $p_2$ is the rate you want to be able to detect, $z_{\alpha/2}$ is 1.96 for a two sided test at 5%, and $z_{\beta}$ is 0.84 for 80% power. The smaller the gap $p_1 - p_2$, the larger $n$, and because the gap is squared, halving it roughly quadruples the traffic. Foundations: sample size

The minimum detectable effect is usually stated as a relative lift $d$, so the target rate is:

$$p_2 = p_1 \,(1 + d)$$

Here $p_1$ is the current rate and $d$ is the relative lift as a share, 0.15 for 15%. Foundations: percentages Turned around, the formula tells you what your traffic can detect. With $n$ visitors per version, the smallest absolute gap you can reliably find is about:

$$\delta \approx (z_{\alpha/2} + z_{\beta})\sqrt{\frac{2\,p_1(1-p_1)}{n}}$$

Here $\delta$ is the minimum detectable effect in absolute terms (a share, so 0.003 is 0.3 percentage points), and the other symbols are as above. It treats both versions as having the rate $p_1$, which is close enough for planning. Foundations: sample size

A rule that circulates says you need 1,000 conversions per version to detect a 10% relative lift. The formula says more. At a 2.5% rate, detecting 2.5% against 2.75% needs about 64,100 visitors per version, which is about 1,600 conversions. At 5% it is about 1,560 conversions. So plan for roughly 1,500 to 1,600 conversions per version for a 10% lift, and four times that for 5%.

Running it properly

• Split. 50/50 is the fastest. A 90/10 split limits exposure to a risky version but takes far longer to read.
• Duration. At least one full week, so every weekday is in both versions. Then run until the planned $n$, not until the chart looks good.
• One primary metric, named in advance. Choosing the metric after seeing the results turns noise into findings.
• Segments are leads, not results. "It lost overall but won on mobile" is a new hypothesis for a new test.
• Read once. Looking every day and stopping at the first significant day inflates false positives. Evan Miller shows a test meant to run at 5% ending up with a false positive rate of 26.1% from repeated checking.¹ A Bayesian test is not immune either: David Robinson's simulations show that its error rates depend on the stopping rule too.² Whichever school you use, fix the stopping rule before you start. If you genuinely need to look early, use a method built for it, such as the always valid p values Ramesh Johari and colleagues describe, which statisticians call sequential testing.³

Worked example: a pricing page at a Kilkenny software company

Castlecomer Rota (castlecomerrota.example) is an invented Kilkenny company selling staff scheduling software. Its pricing page turns 2.1% of visitors into demo requests. It wants to know whether a row of customer logos lifts that by at least 15%.

1. Target rate: $p_2 = 0.021 \times 1.15 = 0.02415$.
2. Sample size: $n = 7.84 \times (0.021 \times 0.979 + 0.02415 \times 0.97585) / 0.00315^2 \approx 34{,}900$ per version. Here $7.84 = (1.96 + 0.84)^2$.
3. The page gets 8,000 visitors a week, so two versions of about 35,000 each need $70{,}000 / 8{,}000 \approx 8.7$ weeks. The plan says nine weeks.
4. After nine weeks: A has 756 demo requests from 36,010 visitors (2.10%), B has 866 from 36,070 (2.40%).
5. Pooled rate: $1{,}622 / 72{,}080 = 0.0225$. Standard error: $\sqrt{0.0225 \times 0.9775 \times (1/36{,}010 + 1/36{,}070)} = 0.00110$.
6. $z = (0.02401 - 0.02099) / 0.00110 = 2.73$, a two sided p value of about 0.006.

The logos stay. The original version of this example planned only 25,000 visitors per version, which the formula does not support. Stopping at that point is how underpowered "winners" are made.

Worked example: product photos at a Limerick outdoor shop

Ballyhoura Trail Supply (ballyhouratrail.example) is an invented Limerick outdoor shop. It tests an action photo of its walking boots against the studio photo. Add to basket runs at 8.2%, and it wants to detect a 10% relative change.

1. Target: $8.2\% \times 1.1 = 9.02\%$. The formula gives about 18,300 visitors per version.
2. The boots category gets 50,000 visitors a week, so the numbers are there inside a week. The shop runs two weeks anyway, to cover two full weekly cycles.
3. Result: studio photo 4,085 from 49,823 (8.20%), action photo 4,559 from 50,102 (9.10%).
4. Relative lift: $0.90 / 8.20 = 11\%$. $z = 5.06$, p below 0.0001.

Action photos roll out to the rest of the range.

from math import sqrt, erf

def z_test(conv_a, n_a, conv_b, n_b):
    pa, pb = conv_a / n_a, conv_b / n_b
    p = (conv_a + conv_b) / (n_a + n_b)
    z = (pb - pa) / sqrt(p * (1 - p) * (1 / n_a + 1 / n_b))
    p_value = 2 * (1 - 0.5 * (1 + erf(abs(z) / sqrt(2))))
    return round(z, 2), round(p_value, 4)

def sample_size(p1, p2, za=1.96, zb=0.84):
    return (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p1 - p2) ** 2

print(z_test(756, 36010, 866, 36070))   # (2.73, 0.0064)
print(round(sample_size(0.021, 0.02415)))  # 34865

A word on tools. Many teams ran their tests in Google Optimize. Google switched off Optimize and Optimize 360 on 30 September 2023, and any experiment still running ended on that date.⁴ If an old process document still points there, it needs a new home.

Three or more versions need a stricter bar

An A/B/n test splits traffic across three or more versions at once. It is useful when you have several real ideas and want to compare them in one go.

The catch is the number of comparisons. Each one carries its own 5% chance of a false positive, and the chances add up. Across $m$ independent comparisons, the chance of at least one false positive is:

$$P(\text{at least one false positive}) = 1 - (1 - \alpha)^m$$

Here $\alpha$ is the significance level of each comparison and $m$ is the number of comparisons. It assumes the comparisons are independent, which comparisons against a shared control only roughly are. Foundations: probability

With three comparisons that is $1 - 0.95^3 = 14\%$, and with five it is 23%. The simplest repair is the Bonferroni method:⁵

$$\alpha_{\text{each}} = \frac{\alpha}{m}$$

Here $\alpha$ is the overall false positive rate you want, $m$ is the number of comparisons and $\alpha_{\text{each}}$ is the bar each single comparison must clear. Foundations: hypothesis tests

Bonferroni is blunt and costs power. The Holm method applies the same idea step by step, from the smallest p value up, and rejects at least as often while keeping the same guarantee.⁶ Controlling the false discovery rate, the expected share of false positives among your "winners", is looser again and suits exploratory work with many versions.⁷

A/B/n tests suit several genuine hypotheses and enough traffic: four versions need roughly twice the traffic of an A/B test, since each version needs its own $n$. They do not suit thin traffic, and they do not help when each version changes the headline, the button and the layout at once. Then you cannot say which change did the work. That is a job for a multivariate test.

Worked example: subject lines at a Galway travel company

Eyre Square Getaways (eyresquaregetaways.example) is an invented Galway travel company with a weekly deals email to 400,000 subscribers. It tests four subject lines on a 40,000 slice, 10,000 each, then sends the winner to the rest.

• A (control): "This week's travel deals"
• B: "Flights from €29, this week only"
• C: "Your next trip just cost less"
• D: "48 hours left: member prices"

The metric is the click rate, not the open rate. Since 2021, Apple's Mail Privacy Protection stops senders from using invisible pixels to know when a message is opened,⁸ so open rates no longer measure what they claim to.

1. Three comparisons against the control, so Bonferroni sets the bar at $0.05 / 3 = 0.0167$. (The original version of this example used 0.0125, which would be the bar for four comparisons.)
2. After four hours: A 2.0%, B 2.6%, C 2.2%, D 2.9%.
3. B against A: $z = 2.83$, $p = 0.005$, below the bar. D against A: $z = 4.12$, $p < 0.001$, below the bar. C against A: $z = 0.99$, $p = 0.32$, not significant.
4. D goes to the remaining 360,000.

Worked example: landing pages for a Cork budgeting app

Penrose Money (penrosemoney.example) is an invented Cork budgeting app with three landing pages: A lists the features, B leads with the outcome, C leads with reviews. Paid traffic is split evenly, 15,000 visitors each, over three weeks.

1. Two comparisons against A, so the bar is $0.05 / 2 = 0.025$.
2. Sign up rates: A 3.2%, B 4.1%, C 3.8%.
3. B against A: $z = 4.16$, $p < 0.001$. C against A: $z = 2.83$, $p = 0.005$. Both clear the bar.
4. B against C: $z = 1.33$, $p = 0.18$. The data cannot tell them apart.

The original version of this example called C "not significant". With these numbers it is. The honest reading is that leading with outcomes or with reviews both beat a feature list, and the next test is B against a version that combines the two.

Bandits earn while they learn

An A/B test holds its split to the end, so a share of visitors keeps seeing the weaker version until you stop. A multi armed bandit moves traffic towards the better version as the evidence builds. The standard algorithm is Thompson sampling: each version keeps a probability distribution for its rate, every visitor sees the version whose random draw comes out highest, and uncertain versions still win some draws while clearly worse ones soon stop.⁹ The formula and code are in the maths behind growth marketing.

Worked example: the cost of an even split

In the simulation in that article, three page versions convert at 3%, 5% and 4%, unknown to the algorithm, over 10,000 visitors.

1. An even split earns $10{,}000 \times (0.03 + 0.05 + 0.04) / 3 = 400$ conversions.
2. Thompson sampling averaged 470 across 300 runs, 70 more, by sending about 7,700 visitors to the best version.

The price is precision about the losers. Use a bandit when earning during the test matters more than measuring every version, a short promotion, say. Use a test when you need the clean number.

Multivariate tests find the best combination, at a steep traffic price

A multivariate test changes several elements at once and shows each visitor one combination. Two headlines, two images and two buttons give $2 \times 2 \times 2 = 8$ combinations. The point is to find not only the best element of each kind but also interactions, where the best headline depends on the image next to it.

A full factorial design runs every combination and measures every main effect and interaction. A fractional factorial design runs a chosen subset, which estimates the main effects with less traffic but mixes up some interactions with each other, a trade the NIST handbook calls confounding.¹⁰

If an A/B test needs 20,000 visitors per version, comparing every cell of a multivariate test needs:

Along the bottom are four designs, with the number of combinations in brackets. Up the side is the total traffic needed if each combination gets 20,000 visitors. The takeaway is the shape: the bill grows with the number of combinations, not the number of elements, and it grows fast. That figure is for comparing every combination with every other. If you only need the main effects, a factorial design lets every visitor count towards every element at once, which needs far less.

So multivariate tests suit pages with hundreds of thousands of visitors a month and elements that may interact. With modest traffic, or no reason to expect interactions, run one A/B test after another.

Worked example: a category page at a Waterford electronics shop

Viking Triangle Devices (vikingtriangledevices.example) is an invented Waterford electronics shop. It tests its laptop category page on two headlines ("Find your laptop" against "Laptops for every budget"), two layouts (grid against list) and two filter positions (filters beside the products against filters above them).

1. $2 \times 2 \times 2 = 8$ combinations.
2. The page gets 1.2 million visitors a month, so a four week test gives each combination about $1{,}200{,}000 / 8 = 150{,}000$ visitors.
3. The result shows an interaction: the grid does best with filters beside it, the list does best with filters above it.
4. Testing layout and filters in two separate A/B tests would have found each "winner" against the other's default, and could have missed the pairing that works.

Worked example: a homepage at a Kilkenny software company

Callan Board (callanboard.example) is an invented Kilkenny project management tool. It tests three headlines, two hero images (product screenshot against illustration) and two button labels ("Start free trial" against "See it in action").

1. $3 \times 2 \times 2 = 12$ combinations.
2. With 800,000 visitors a month, six weeks gives about $1{,}200{,}000 / 12 = 100{,}000$ per combination.
3. The screenshot wins, but only with "See it in action". With "Start free trial" the illustration does better. That interaction shapes the next round of creative work.

Holdout tests measure what would have happened anyway

A holdout test does not compare two experiences. It compares some treatment with none. A random share of the audience, often 5% to 20%, is kept out of a campaign, a feature or a retargeting list. Comparing them with everyone else measures the incremental lift: what the activity caused, rather than what it was credited with.

Most marketing measurement flatters itself. People who see retargeting ads already visited the site. Credit every purchase they make to the ads and you count sales that would have happened anyway.

Two numbers come out of a holdout:

$$\text{absolute lift} = r_T - r_C, \qquad \text{relative lift} = \frac{r_T - r_C}{r_C}$$

Here $r_T$ is the conversion rate in the treated group and $r_C$ the rate in the holdout (control). The absolute lift is in percentage points, the relative lift is a share of the holdout's rate. Foundations: percentages

The share of the treated group's conversions that the activity actually caused is $(r_T - r_C) / r_T$, a different number again. Confusing the two is the commonest mistake in reading a holdout.

Holdouts suit retargeting, email and CRM programmes, campaigns that run all the time, and any budget a finance director wants proved. Three practical points: a bigger holdout gives more precision but forgoes more revenue, so 10% is a common compromise. Holdouts usually run longer than A/B tests, because they measure behaviour over weeks. And the holdout must really be held out: if those people get the same message through another channel, the measurement is spoiled.

The ad platforms build this in. Google's Conversion Lift splits people (or regions) into a group that sees the ads and one that does not, and reports incremental conversions, incremental cost per action and incremental return on ad spend. Google notes it is not available for every account, and you ask your Google representative to use it.¹¹ A platform measuring its own ads deserves the same scrutiny as any other report: check how the groups were split and what counts as a conversion before you trust the number.

Worked example: retargeting at a Kilkenny furniture shop

Thomastown Oak (thomastownoak.example) is an invented Kilkenny furniture shop spending €50,000 a month on retargeting. The question from the top is simple: would these people have bought anyway?

1. It holds out 10% of the 200,000 retargeting eligible visitors a month for eight weeks. 180,000 a month see the ads, 20,000 do not.
2. Retargeted: 3.2% buy, at €180 on average, so €5.76 per visitor. Holdout: 2.1% buy, at €175, so €3.675 per visitor.
3. Incremental revenue per visitor: $5.76 - 3.675 = €2.085$.
4. Per month: $180{,}000 \times 2.085 = €375{,}300$ of revenue the ads caused.

Incremental return on ad spend is that revenue over the spend:

$$iROAS = \frac{\Delta R}{S}$$

Here $\Delta R$ is the incremental revenue, the revenue the treated group earned above what the holdout's rate predicts, and $S$ is the ad spend in the same period. Foundations: ratios and rates

5. $iROAS = 375{,}300 / 50{,}000 = 7.5$.
6. Had every purchase by a retargeted visitor been credited to the ads, the figure would be $180{,}000 \times 5.76 / 50{,}000 = 20.7$. The holdout shows almost two thirds of that was going to happen anyway.

Retargeting still pays here, but at 7.5 in revenue, before margin, not 20.7. (The original version of this example multiplied by all 200,000 visitors, including the holdout who never saw an ad.)

Worked example: trial emails at a Galway software company

Oranmore Ledger (oranmoreledger.example) is an invented Galway bookkeeping app with a 12 email sequence for trial users. It holds out 15% of 4,000 new trial users for 60 days.

1. Emails: 3,400 users, 286 pay (8.4%). Holdout: 600 users, 31 pay (5.2%).
2. Absolute lift: $8.4 - 5.2 = 3.2$ points. $z = 2.71$, $p = 0.007$.
3. Relative lift: $3.2 / 5.2 = 62\%$.
4. Share of the email group's conversions the emails caused: $3.2 / 8.4 = 38\%$, about 109 of the 286.

The original version of this example said 62% of the attributed conversions were incremental. That mixes up the relative lift with the incremental share. The sequence works, and 38% is the number to put in front of the finance director.

Geographic tests work when you cannot randomise people

A geographic test randomises places instead of people. One set of towns or regions gets treatment A, a matched set gets B or nothing. Google has written up how it uses these geo experiments to measure search ads, with non overlapping regions randomly assigned to control and treatment.¹²

They suit media that cannot be aimed at individuals (radio, posters, leaflets, TV), prices or ranges that differ by region, and cases where people talk to each other, so a person level test would leak.

Places differ in ways that have nothing to do with the test, so a geographic test needs care:

• Matched pairs. Pair towns with similar history, size and make up, then randomly treat one of each pair.
• Difference in differences. Compare the change in each group, before against during, not the levels.
• Synthetic control. Build a weighted blend of untreated areas that tracked the treated area before the test, and compare the treated area with that blend during it. Alberto Abadie's review sets out when it works and when it does not.¹³ Meta's open source GeoLift package does this for ad campaigns.¹⁴

The difference in differences is:

$$DiD = (Y_{T,\text{during}} - Y_{T,\text{before}}) - (Y_{C,\text{during}} - Y_{C,\text{before}})$$

Here $Y_{T}$ is the outcome in the treated areas and $Y_{C}$ in the control areas, each measured before and during the test. Subtracting the control's change removes what would have happened anyway, such as a good summer for everyone. Foundations: indexing and base rates

Worked example: local radio for a garden centre chain

Knockmealdown Garden Centres (knockmealdowngardens.example) is an invented chain with stores in market towns across Munster and Connacht, from Co. Waterford to Co. Galway. It wants to know what advertising on the local radio station in each town's county does for sales.

1. It forms 20 pairs of similar store towns, the two towns of each pair served by different local stations, matched on sales per head, local population, nearby competitors and past growth.
2. In each pair, a coin decides which town gets 12 weeks of radio.
3. Sales against the same weeks last year: radio towns up 14.2%, the others up 6.8%.
4. $DiD = 14.2 - 6.8 = 7.4$ percentage points of growth the radio caused.

Add the margin on those sales and the chain knows whether radio pays.

Worked example: a flat delivery fee

An invented takeaway delivery app, active in 10 Irish cities and large towns, wants to compare a flat €2.99 fee with its current fee of €1.99 to €4.99 by distance. A person level test would have neighbours paying different fees for the same restaurant, so it tests by town.

1. Five towns switch to the flat fee, five matched towns keep the old one, for eight weeks.
2. Flat fee towns: orders per customer up 8.2%, average order value down 3.1%. Revenue per customer: $1.082 \times 0.969 = 1.048$, up 4.8%.
3. Comparison towns: orders up 1.4%, order value up 0.8%. $1.014 \times 1.008 = 1.022$, up 2.2%.
4. The difference: about 2.6 points in favour of the flat fee, before fee income.

With five towns a side, the noise between towns can be as large as that gap. Before rolling it out, the app checks how much these towns' growth normally varies against each other, and adds the change in fee income.

Audience splits test different approaches for different segments

An audience split runs a separate test inside each segment, because segments may respond differently. Discounts may win with bargain hunters while a premium message wins with loyal customers.

Segments can be fixed in advance (new against returning, industry), found by a model, or taken from context such as device.

The risk is a loop that confirms itself. Label someone "price sensitive", show them only discounts, and you never learn that the premium message would have worked too. From time to time, show each segment the other segment's winner.

Worked example: new and returning visitors at a Galway fashion shop

Shop Street Threads tests three homepage banners for new visitors (brand story, new arrivals, 15% off a first order) and three for returning visitors (recommendations, arrivals since the last visit, the loyalty scheme). Each test is planned and read on its own, with its own $n$. New visitors respond best to the discount, returning visitors to the recommendations, so the homepage now differs by group.

Worked example: industry pages at a Limerick software company

An invented Limerick marketing software company tests industry pages against its general page for healthcare, financial services and retail prospects. The industry pages win in all three, by 40% to 62%, with healthcare at the top, where regulation weighs most on buyers. (The original version said "40% to 60%" and then quoted 62% for healthcare.) Three segments means three comparisons, so each is read at the Bonferroni bar of 0.0167.

Time based tests swap versions over time

A time based test shows version A for a period, then version B, and compares the periods. The original version of this article called it a sequential test, but statisticians use that name for the methods that let you look at a running test safely, so I call this one time based.

It avoids splitting traffic but makes time a rival explanation. If B's week does better, was it B, or did a competitor run out of stock that week?

The question is when it is worth that price. Low traffic alone is not a reason: splitting each day's visitors 50/50 gives the same sample with no time confound. The real reason is a technical one, a system that cannot show two versions at once. Then protect yourself:

• Alternate several times. A, B, A, B, not A then B, so one odd week cannot carry the result.
• Adjust for known changes. Ad spend, traffic sources, promotions.
• Compare with something untouched. If a metric the change cannot affect moved too, adjust for it.
• Know your normal. A long baseline shows how much the metric moves week to week on its own.

Worked example: onboarding at a Cork productivity app

Cobh Tasks (cobhtasks.example) is an invented Cork app whose onboarding is built into the app release, so the old and new flows cannot run side by side. It gets about 50 new users a day.

1. Weeks one and three run the old flow, weeks two and four the new one. Each gets about 700 new users.
2. Old flow: 238 activate (34%). New flow: 287 activate (41%).
3. $z = 2.70$, $p = 0.007$, with the caveat that the weeks were not randomly assigned.
4. The new flow won in both of its weeks, and nothing else changed in spend or sources. The team ships it.

The original article used a 5,000 person email list as its example here. Splitting each send in half gives the same sample with no time confound, so for email that is the better design.

Choosing the method

Read it from the top: whether a test is worth it, whether people can be randomised (if not, places or time are your units), whether you want incrementality (only a holdout answers that), then versions, segments and bandits. Every route ends the same way: the plan is written before the data arrives.

Before any of it, run the sample size formula. If the traffic cannot reach the $n$ for the effect you care about, accept a larger minimum detectable effect, test something bolder, or ask customers instead.

The mistakes that undo good tests

• Stopping early. The first significant day is not the result. Fix $n$ or use a method built for looking.
• Fishing. Running many tests or slicing one ten ways and reporting the significant ones turns noise into findings. Name the primary metric and the analysis in advance.
• Significant but small. A 0.1% lift can be real and still not worth building. Put the effect next to its cost.
• Overlapping tests. Two tests on the same page and the same people can interfere. Someone has to keep a list of what is running where.

Testing is a discipline, not a feature

Testing is not a switch in a tool. It is a set of habits: pick the method from the question, size it before it starts, read it once, and believe the answer even when it disappoints.

The costly mistake is rarely the wrong method. It is not testing, or testing badly and trusting the result.

I have implemented testing programmes across eCommerce, SaaS and B2B businesses. The statistics matter, but so does the habit of running tests properly and acting on the results honestly. Start with an A/B test on your busiest page, get the discipline right, and widen the methods as the questions grow.

Footnotes

1. Evan Miller, How Not To Run an A/B Test, evanmiller.org, 2010, https://www.evanmiller.org/how-not-to-run-an-ab-test.html

2. David Robinson, Is Bayesian A/B Testing Immune to Peeking? Not Exactly, Variance Explained, 2015, http://varianceexplained.org/r/bayesian-ab-testing/

3. Ramesh Johari, Leo Pekelis and David J. Walsh, Always valid inference: bringing sequential analysis to A/B testing, arXiv 1512.04922, 2015 (revised 2019), https://arxiv.org/abs/1512.04922

4. Google, [Sunset September 2023] Google Optimize, Analytics Help, https://support.google.com/optimize/answer/12979939

5. NIST/SEMATECH, e Handbook of Statistical Methods, section 7.4.7.3 Bonferroni's method, National Institute of Standards and Technology, 2012, https://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm

6. Wikipedia, Holm Bonferroni method (Sture Holm, Scandinavian Journal of Statistics, 1979), https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method

7. Wikipedia, False discovery rate (Yoav Benjamini and Yosef Hochberg, Journal of the Royal Statistical Society Series B, 1995), https://en.wikipedia.org/wiki/False_discovery_rate

8. Apple, Apple advances its privacy leadership with iOS 15, iPadOS 15, macOS Monterey, and watchOS 8 (Mail Privacy Protection), Apple Newsroom, 7 June 2021, https://www.apple.com/newsroom/2021/06/apple-advances-its-privacy-leadership-with-ios-15-ipados-15-macos-monterey-and-watchos-8/

9. Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband and Zheng Wen, A Tutorial on Thompson Sampling, Foundations and Trends in Machine Learning 11(1), 2018, https://arxiv.org/abs/1707.02038

10. NIST/SEMATECH, e Handbook of Statistical Methods, section 5.3.3.4 Fractional factorial designs, National Institute of Standards and Technology, 2012, https://www.itl.nist.gov/div898/handbook/pri/section3/pri334.htm

11. Google, About Conversion Lift, Google Ads Help, https://support.google.com/google-ads/answer/12003020

12. Jon Vaver and Jim Koehler, Measuring Ad Effectiveness Using Geo Experiments, Google, 2011, https://research.google/pubs/measuring-ad-effectiveness-using-geo-experiments/

13. Alberto Abadie, Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects, Journal of Economic Literature 59(2), pages 391 to 425, 2021, https://www.aeaweb.org/articles?id=10.1257/jel.20191450

14. Meta Open Source, GeoLift: an end to end geo experimental methodology based on synthetic control methods, GitHub, https://github.com/facebookincubator/GeoLift