Foundations: the statistics and maths behind every article, from scratch

This is the ground floor. Every formula in every Intelligence article on this site links to one of the sections below, so that nobody has to take a Greek letter on trust. Read it top to bottom as a short course, or arrive from an article, read the one section you need, and go back. Nothing here assumes more than school arithmetic, and I explain every term the first time it matters.

One shop runs through the whole page. Northgate Hampers is an invented online shop that sells gift hampers from a unit outside Leeds. The owner is Priya. Last December the shop took 1,000 orders. Every number in every example on this page is mine and made up for the purpose, so the arithmetic is clean and nobody real is involved. Where a definition comes from somewhere else, there is a numbered footnote and the source is listed in full at the foot.

Each idea gets a plain explanation, the formula, a paragraph that names every symbol in it, a worked example with every step shown, and the one mistake I see most often in a business. Other articles link here with an anchor, for example Foundations: the mean, so each section keeps its address as the page grows.

Percentages, percentage points and percentage change

A percentage is a share of something, written per hundred. Three hundred out of 1,000 orders is 30 per hundred, so 30%. It answers the question "how much of the whole is this?". Percentages are everywhere in a business, and most of the confusion I see comes from mixing up two different things that both wear the % sign: the change in a percentage, and the change measured in percentage points.

A percentage point is the plain difference between two percentages. If the conversion rate goes from 2.0% to 2.5%, it has risen by 0.5 percentage points. A percentage change is the difference measured against where you started. The same move, from 2.0% to 2.5%, is a 25% increase, because 0.5 is a quarter of 2.0. Both statements are true. They answer different questions, and whoever is talking to you may pick the one that sounds better.

$$\Delta\% = \frac{v_{\text{new}} - v_{\text{old}}}{v_{\text{old}}} \times 100$$

Here $\Delta\%$ is the percentage change, the thing you want to know. $v_{\text{new}}$ is the new value, whatever you are measuring now. $v_{\text{old}}$ is the old value, the one you are comparing against, and it sits underneath because every change is measured relative to where you started. The $\times 100$ at the end just turns a fraction into a figure per hundred.

Worked example: Northgate Hampers lifts its conversion rate

Priya ran a tidier checkout for a month. Before, 2.0% of visitors placed an order. After, 2.5% did. First, the percentage points: $2.5 - 2.0 = 0.5$ percentage points. Second, the percentage change, using the formula above: $\Delta\% = \frac{2.5 - 2.0}{2.0} \times 100 = \frac{0.5}{2.0} \times 100 = 0.25 \times 100 = 25\%$. So the conversion rate rose by half a percentage point, which is a 25% increase. If an agency reports the 25%, ask for the points. If a cautious colleague reports the 0.5, ask for the percentage. Together they tell the whole story.

Where it goes wrong

Percentages do not add up and down symmetrically. If sales fall by 50% and then rise by 50%, you are not back where you started. From 100, a 50% fall takes you to 50. A 50% rise from 50 adds 25, so you land on 75, a quarter below where you began. The rise is measured from a smaller base, so it is worth less in pounds than the fall was. Any time somebody strings percentages together, work it through in actual numbers.

Ratios and rates

A rate is a count of things that happened, divided by the number of chances for them to happen. A conversion rate is orders divided by visitors. A click through rate is clicks divided by the number of times an advert was shown, which the advertising platforms call impressions. Orders per 1,000 visitors is the same idea with a friendlier denominator. The question a rate answers is "how often does this happen, per opportunity?", and the word "per" is the clue that you are looking at one.

$$\text{rate} = \frac{\text{events}}{\text{opportunities}}$$

The $\text{rate}$ is the result, a number between 0 and 1 that is usually shown as a percentage. The $\text{events}$ are the things that happened, orders, clicks, sign ups, returns, whatever you are counting. The $\text{opportunities}$ are the number of times that thing could have happened, visitors, impressions, emails delivered, orders shipped. Changing the denominator changes the rate, so always say what it is.

Worked example: Northgate Hampers' conversion rate and click through rate

In December, 40,000 people visited the shop and 1,000 of them ordered. The conversion rate is $\frac{1{,}000}{40{,}000} = 0.025$, which is 2.5%. Put the other way, that is 25 orders per 1,000 visitors, because $0.025 \times 1{,}000 = 25$. The same month, Priya's adverts were shown 1,000,000 times and were clicked 12,000 times. The click through rate is $\frac{12{,}000}{1{,}000{,}000} = 0.012$, so 1.2%. Note what the two rates share: a count on top, a count of chances underneath, and a result that only means something once you know both counts.

Where it goes wrong

Rates built on very different denominators are not comparable. A new product page that turned 10 of its 100 visitors into buyers has a conversion rate of 10%, which looks four times better than the shop's 2.5%. But 100 visitors is a tiny base, and the next 100 might bring two orders. Before comparing rates, compare the number of opportunities underneath them. The section on sampling shows how to put a size on that doubt.

The mean

The mean is what most people call the average: add everything up and divide by how many things there were. It answers the question "if every order had been the same size, what size would that have been?". It is the single most used number in business reporting, and the one most often used on its own when it should not be.¹

$$\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i$$

The symbol $\bar{x}$, read "x bar", is the mean. $n$ is the number of values, here the number of orders. The large $\sum$ is the instruction to add things up, and the $i=1$ below it and the $n$ above it say "starting with the first value and finishing with the last". Each $x_i$ is one individual value, the $i$th order, so $x_1$ is the first order, $x_2$ the second, and so on. The $\frac{1}{n}$ at the front divides the total by the count.

Worked example: the mean order at Northgate Hampers

Priya's 1,000 December orders came to £102,000 in total. So $n = 1{,}000$ and $\sum x_i = 102{,}000$. The mean order is $\bar{x} = \frac{1}{1{,}000} \times 102{,}000 = 102$. The mean order value was £102. That is a true statement about December, and it is the number you would use to turn a forecast of orders into a forecast of revenue. It is not, on its own, a description of the typical customer, as the next section shows.

Where it goes wrong

The mean is pulled towards the extremes. Among Priya's 1,000 orders are twenty corporate orders of more than £500 each, companies sending hampers to their whole client list. Those twenty orders carry a large slice of the £102,000, and they drag the mean upwards. Most customers spent far less than £102. If you set a free delivery threshold, a gift wrap offer or an advertising bid on the assumption that "the average customer spends about £100", you are designing for a customer who barely exists. Always ask what the median is as well.

The median and the mode

The median is the value in the middle once you sort everything from smallest to largest: half the orders are below it, half above. The mode is the value that appears most often. Neither needs a formula so much as a procedure. To find the median of $n$ values, sort them and take the one in position $\frac{n+1}{2}$. With an even number of values, take the mean of the two in the middle. To find the mode, count how often each value appears and pick the most common. The question the median answers is "what does a typical order look like?", and the mode answers "what happens most?".

The three averages tell you about the shape of your data when you line them up. When mean, median and mode are all about the same, the data is roughly symmetric. When the mean is larger than the median, which is larger than the mode, the data has a long tail stretching out to the right, towards the large values. That is called a right skewed distribution, and it describes nearly every shop I have ever looked at, because there are always a few very large orders and no orders below zero.²

Worked example: the three averages of Northgate Hampers

Sort Priya's 1,000 orders from smallest to largest. There is an even number of them, so the median is the mean of the 500th and 501st orders. Both happen to be £68, so the median is £68. Half of all orders were below £68 and half above. Now count how often each exact value occurs. The single most common order value is £49, because that is the price of the Classic hamper, the bestseller, and hundreds of people bought one of those and nothing else. So the mode is £49. Line the three up: mean £102, median £68, mode £49. The mean sits well above the median, which sits above the mode. That ordering is the signature of a right skewed distribution: lots of modest orders, a tail of big ones.

Where it goes wrong

The phrase "the average basket" is meaningless until you say which average. A supplier pitching a loyalty scheme will quote the mean, because it is the biggest of the three. A pessimist will quote the mode. The honest report gives at least the mean and the median side by side and says so. If the two are far apart, you have a tail, and the tail is where the interesting decisions live.

Spread: range, variance, standard deviation and the coefficient of variation

An average on its own tells you where the middle is but nothing about how far the values stray from it. Two shops can both have a mean order of £102 while one sells almost everything at around £100 and the other sells a mixture of £20 cards and £800 corporate crates. Spread is the family of numbers that describe how widely values are scattered. The range is the simplest: the largest value minus the smallest. The variance and the standard deviation measure the typical distance from the mean. The coefficient of variation puts that distance in proportion to the mean so you can compare the spread of things measured on different scales.³

$$s^2 = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2, \qquad s = \sqrt{s^2}, \qquad CV = \frac{s}{\bar{x}}$$

In the first formula $s^2$ is the sample variance. $n$ is the number of values and $n-1$ is one fewer than that, for a reason I come to in a moment. $\sum_{i=1}^{n}$ says add up what follows for every value from the first to the last. $x_i$ is each individual value and $\bar{x}$ is the mean, so $(x_i - \bar{x})$ is how far one value sits from the mean, and squaring it makes every distance positive and gives more weight to the far ones. In the second formula $s$ is the standard deviation, the square root of the variance, which brings the measure back into the original units, pounds rather than pounds squared. In the third, $CV$ is the coefficient of variation, the standard deviation divided by the mean, a pure number with no units.⁴

Why divide by $n-1$ and not $n$? Because the mean you subtract was itself calculated from the same values, and that uses up one degree of freedom. If you know the mean of 1,000 numbers and you know 999 of them, the last one is fixed, so only 999 of the deviations are free to vary. Dividing by $n-1$ corrects for the fact that the deviations from a sample's own mean are slightly smaller, on average, than the deviations from the true mean of the whole population would be. With 1,000 orders the difference is tiny. With ten it matters.

Worked example: how spread out are Northgate Hampers' orders?

Priya's smallest December order was a £12 jar of chutney with a card, and the largest was a £1,480 corporate order, so the range is $1{,}480 - 12 = 1{,}468$. Working through all 1,000 squared deviations by hand would fill a book, so take it as given that the sum of squared deviations comes to $\sum (x_i - \bar{x})^2 = 24{,}939{,}036$. Then the variance is $s^2 = \frac{24{,}939{,}036}{999} = 24{,}964$, and the standard deviation is $s = \sqrt{24{,}964} = 158$. So the orders typically sit about £158 away from the mean of £102. The coefficient of variation is $CV = \frac{158}{102} = 1.55$. A CV above 1 means the standard deviation is bigger than the mean itself. The mean is not a safe summary of orders like these. Report the median next to it, and show the distribution.

Where it goes wrong

A mean with no spread next to it is half a number. "Average order £102" invites a reader to picture a hundred pound customer. "Average order £102, standard deviation £158, median £68" tells them the truth, that the typical customer spends under £70 and a few spend a great deal. If a report gives you only averages, ask for the standard deviation or at least the median, and treat any plan built on the average alone with care.

Distributions, the normal curve and the long tail

A distribution is the full picture of how often each value occurs. Instead of one number, you get the whole shape: how many orders fell between £40 and £60, how many between £60 and £80, and so on. Every average and every measure of spread is a summary of a distribution. The chart below is Priya's December, sorted into value bands.

Along the bottom are the value bands, from under £40 to over £500. Up the side is the number of orders in each band. The tallest bar is the £40 to £60 band with 290 orders, where the £49 Classic hamper lives, and the bars shrink steadily until only 20 orders sit above £500. The peak is on the left and the tail stretches right, which is exactly what mean 102, median 68 and mode 49 told you in the previous section. The 20 orders in the last band are the corporate orders that pull the mean up.

The most famous distribution is the normal distribution, the bell curve. It is symmetric, the mean, median and mode all sit at the top, and it is described completely by its mean and its standard deviation.⁵ It comes with a rule of thumb worth memorising: about 68% of values lie within one standard deviation of the mean, about 95% within two, and about 99.7% within three.⁶ Its formula looks fearsome and you never need to compute it by hand, but it is worth seeing once so the symbols hold no mystery.

$$f(x) = \frac{1}{\sigma\sqrt{2\pi}}\, e^{-\frac{(x-\mu)^2}{2\sigma^2}}$$

Here $f(x)$ is the height of the curve at a value $x$, which is how relatively likely that value is. $\mu$, the Greek letter mu, is the mean of the whole population, and $\sigma$, sigma, is its standard deviation. Population values get Greek letters, while the sample versions you calculate from data get Latin ones, $\bar{x}$ and $s$. $\pi$ is the familiar 3.14159 and $e$ is the number 2.71828, the base of natural growth, which you meet again under logarithms. The exponent $\frac{(x-\mu)^2}{2\sigma^2}$ grows as $x$ moves away from the mean, and the minus sign in front of it makes the curve fall away on both sides. The $\frac{1}{\sigma\sqrt{2\pi}}$ at the front scales the curve so the area underneath it adds up to 1.

Skew is the name for a lopsided distribution. Priya's orders are right skewed, or positively skewed: the tail points to the right.⁷ Two more distributions turn up in the articles. The binomial distribution describes a count of yes or no outcomes out of a fixed number of tries, such as how many of 40,000 visitors place an order when each one does so with the same probability.⁸ The Poisson distribution describes a count of events in a fixed period when they arrive independently at a steady average rate, such as how many customer service emails arrive in an hour.⁹

Worked example: single orders are not normal, but the mean of many is

Priya's individual orders are plainly not normal: the shape is lopsided and the tail is long. Yet if you took the mean of all 1,000 orders, then imagined doing December over and over again with the same kind of customers, the means you got would cluster in a symmetric bell shape around £102. This is the central limit theorem, and it is why the normal curve matters even when your data does not look like it. It says that the mean of a large enough sample is approximately normally distributed, whatever the shape of the individual values, with a standard deviation equal to the standard deviation of the individual values divided by the square root of the sample size.¹⁰ For Priya that is $\frac{158}{\sqrt{1{,}000}} = \frac{158}{31.6} = 5.0$. So although single orders range from £12 to £1,480, the mean of a thousand of them would land within about £5 of £102 most of the time. The section on sampling gives that number its proper name, the standard error.

Where it goes wrong

Assuming order values are normal and designing off the mean. A free delivery threshold set at "a bit above the average order, say £110" sounds sensible if you picture a bell curve centred on £102. Look at the real chart: only 210 of the 1,000 orders are above £100 at all, so the threshold sits out in the tail where few customers ever reach, and it nudges almost nobody. Set thresholds from the distribution, typically just above the median or the biggest band, never from the mean of a skewed shape.

Probability

A probability is the long run share of times something happens. If 2.5% of visitors order, the probability that the next visitor orders is 0.025, in the sense that over many thousands of visitors about 25 in every 1,000 will. Probabilities run from 0, never, to 1, always, and the probabilities of all the possible outcomes add up to 1. Four rules cover almost everything the articles need.

First, adding. If two outcomes cannot both happen, they are mutually exclusive, and the probability of one or the other is the sum of the two. A visitor cannot both order and not order, so the probability of one or the other is $0.025 + 0.975 = 1$. Second, multiplying. If two events have no influence on each other, they are independent, and the probability of both is the product. If one in 40 visitors orders and one in ten visitors arrives on a Tuesday, and the two have nothing to do with each other, the probability of a Tuesday order is $0.025 \times 0.1 = 0.0025$.¹¹ Third and fourth, the two formulas below, which deal with events that do influence each other.

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}$$

This is conditional probability. $P(A \mid B)$ reads "the probability of A given B": how likely is A once you know B has happened. $P(A \cap B)$ is the probability of A and B both happening, the $\cap$ symbol meaning "and". $P(B)$ is the probability of B on its own. Dividing by $P(B)$ narrows the world down to the cases where B happened and asks what share of those also had A.¹²

$$P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}$$

This is Bayes' theorem, and it lets you turn a conditional probability around. $P(A \mid B)$ is again the probability of A given B, the thing you want. $P(B \mid A)$ is the reverse, the probability of B given A, which is often the one you actually know. $P(A)$ is the plain probability of A, called the prior or the base rate, and $P(B)$ is the plain probability of B. Multiply what you know by the base rate and divide by how common B is overall.¹³

Worked example: do visitors who use the search box order more?

Priya knows three things. First, 2.5% of all visitors order, so $P(\text{order}) = 0.025$. Second, of the people who order, 40% had used the search box, so $P(\text{search} \mid \text{order}) = 0.40$. Third, 10% of all visitors use the search box, so $P(\text{search}) = 0.10$. The question she wants answered is the reverse of the second fact: of the visitors who used the search box, what share went on to order? That is $P(\text{order} \mid \text{search})$. Bayes' theorem gives $P(\text{order} \mid \text{search}) = \frac{P(\text{search} \mid \text{order}) \, P(\text{order})}{P(\text{search})} = \frac{0.40 \times 0.025}{0.10} = \frac{0.010}{0.10} = 0.10$. Searchers convert at 10%, four times the shop's overall 2.5%. You can check it with the conditional probability formula: searchers who order are $0.40 \times 0.025 = 0.010$ of all visitors, 1% of them, or 400 of the 40,000. Searchers are 10% of visitors, 4,000 of them. And $\frac{400}{4{,}000} = 0.10$. Same answer, and now Priya knows that making the search box easier to find is worth testing.

Where it goes wrong

Reading $P(B \mid A)$ as if it were $P(A \mid B)$. "40% of buyers used search" is not "40% of searchers buy". The first is about buyers, the second about searchers, and in Priya's shop the second number is 10%, not 40%. The two can be wildly different whenever the base rates differ, and in business they nearly always do. See indexing and base rates for the fraud filter version of the same trap.

Expected value

The expected value is the average result you would get if you could run a gamble many times over. You take each possible outcome, multiply it by its probability, and add them all up. It answers the question "on average, what is this decision worth?", and it is the right way to compare options that each carry some chance of failing.¹⁴

$$E[X] = \sum_i p_i\, x_i$$

$E[X]$ is the expected value of the uncertain quantity $X$, the campaign's return, the value of a lead, whatever you are weighing up. The $\sum_i$ says add up the following for every possible outcome $i$. $p_i$ is the probability of outcome $i$, and the $p_i$ across all outcomes must add up to 1. $x_i$ is the value of outcome $i$, in pounds or whatever unit you care about.

Worked example: should Northgate Hampers run the spring campaign?

Priya is weighing a spring campaign that costs £5,000. From past campaigns she judges there is a 60% chance it returns £9,000 in revenue, a 30% chance it only returns £4,000, and a 10% chance it returns nothing at all, because a wet bank holiday can do that. The three probabilities add to $0.6 + 0.3 + 0.1 = 1$, as they must. The expected return is $E[X] = 0.6 \times 9{,}000 + 0.3 \times 4{,}000 + 0.1 \times 0 = 5{,}400 + 1{,}200 + 0 = 6{,}600$. The expected profit is the expected return less the cost, $6{,}600 - 5{,}000 = 1{,}600$. On average the campaign makes £1,600, so if Priya could run it every spring for twenty years, she would expect to come out roughly £32,000 ahead. That is a yes, in the long run.

Where it goes wrong

Expected value says nothing about the one time you cannot afford the zero. The £1,600 is an average over many imagined springs. In any single spring the campaign returns £9,000, £4,000 or nothing, and never £6,600. If a £5,000 loss in a quiet quarter would mean not paying a supplier, the 10% chance of it is the number that matters, not the average. Expected value is for decisions you can repeat, or losses you can absorb. For the others, look at the worst case first.

Sampling and the standard error

A population is everything you care about: every visitor who will ever come to the shop, every order the shop will ever take. A sample is the part you actually have, one December, one month of a test.¹⁵ Almost every number in a business is a sample dressed up as a fact. December's conversion rate of 2.5% is what happened with those 40,000 visitors. A different 40,000, just as typical, would have produced a slightly different figure. The standard error measures how much a statistic would wobble from one sample to the next, and it is the foundation under everything in the next three sections.

$$SE_{\bar{x}} = \frac{s}{\sqrt{n}}, \qquad SE_{p} = \sqrt{\frac{p(1-p)}{n}}$$

In the first formula $SE_{\bar{x}}$ is the standard error of a mean, how much the sample mean would vary across repeated samples. $s$ is the sample standard deviation and $n$ is the sample size. The square root in the denominator is why precision improves slowly: to halve the error you need four times the data. In the second formula $SE_{p}$ is the standard error of a proportion, such as a conversion rate. $p$ is the proportion itself, written as a decimal, $1-p$ is its complement, and $n$ is again the number of opportunities. The product $p(1-p)$ is largest when $p$ is 0.5 and small when $p$ is near 0 or 1, which is why rare events need enormous samples.¹⁶

Worked example: how sure is Northgate Hampers of its own numbers?

Start with the mean order. The standard deviation was £158 and there were 1,000 orders. $SE_{\bar{x}} = \frac{158}{\sqrt{1{,}000}} = \frac{158}{31.62} = 5.0$. So if Priya could rerun December many times, the mean order would typically land within about £5 of £102. Now the conversion rate. $p = 0.025$, so $1 - p = 0.975$, and $n = 40{,}000$. First the product, $0.025 \times 0.975 = 0.024375$. Divide by $n$: $\frac{0.024375}{40{,}000} = 0.000000609$. Take the square root: $\sqrt{0.000000609} = 0.00078$. As a percentage that is 0.078 percentage points. The December conversion rate of 2.5% is good to within about a twelfth of a point. That is what 40,000 visitors buys you.

Where it goes wrong

A "trend" spotted in 30 orders. If Priya's shop took 30 orders on a Tuesday with a mean of £118 and 30 on a Wednesday with a mean of £91, somebody will say Tuesdays are better. With $s = 158$ and $n = 30$ the standard error of each daily mean is $\frac{158}{\sqrt{30}} = 28.8$, so each of those means could easily be off by £30 in either direction, and the gap between them is noise. Before you believe a difference, compare it with the standard error. The confidence interval and the hypothesis test are the formal ways of doing that.

Confidence intervals

A confidence interval is a range around a sample statistic that is likely to contain the true population value. Instead of saying "the mean order is £102", you say "the mean order is somewhere between £92 and £112, and I am 95% confident of that". The 95% describes the method: if you built intervals this way from many samples, about 95 in 100 of them would contain the true value. It is the honest way to report a number from a sample, and once you have the standard error, it is one multiplication away.¹⁷

$$\bar{x} \pm z \times SE$$

$\bar{x}$ is the sample statistic at the centre of the interval, a mean here, or a proportion $p$ in the second half of the example. $z$ is a multiplier from the normal distribution that sets the confidence level: 1.96 for 95%, 1.645 for 90%, 2.576 for 99%. The 1.96 comes straight from the 95% rule of thumb under distributions, where about 95% of a normal curve sits within two standard deviations of the mean. $SE$ is the standard error from the previous section. The $\pm$ says go that far below and that far above the statistic.

Worked example: Northgate Hampers' December, with its error bars

The mean order was £102 with a standard error of 5.0. For 95% confidence, the margin is $1.96 \times 5.0 = 9.8$. The interval is $102 - 9.8 = 92.2$ to $102 + 9.8 = 111.8$. Priya can say the true long run mean order, for customers like December's, is between £92.20 and £111.80. The conversion rate was 2.5% with a standard error of 0.078 percentage points. The margin is $1.96 \times 0.078 = 0.153$ points. The interval runs from $2.5 - 0.153 = 2.347$ to $2.5 + 0.153 = 2.653$, so from 2.35% to 2.65%. Notice how tight that is. Forty thousand visitors is a lot of evidence, and a competitor claiming "our checkout converts at 3%" is making a claim Priya's data can actually test.

Where it goes wrong

Treating the interval as the range of individual orders. The interval £92.20 to £111.80 is about where the mean sits. It says nothing about where single orders fall, and they fall everywhere from £12 to £1,480. An interval for a mean gets narrower as the sample grows. The spread of individual orders does not shrink at all, however many you collect. If you want the range of typical orders, go back to the distribution and the standard deviation, not the standard error.

Hypothesis tests

A hypothesis test is a disciplined way of asking "is this difference real, or could it be noise?". You start from the null hypothesis, the boring assumption that there is no difference, that the new checkout converts exactly as well as the old one.¹⁸ You then ask how surprising your data would be if that boring assumption were true. The p value is that surprise, as a probability: the chance of seeing a difference at least as big as yours if there were really no difference at all. The significance level, usually 5% and written $\alpha$, is the line you draw in advance. If the p value falls below it, you reject the null hypothesis and call the result statistically significant.

Two kinds of mistake are possible, and they pull against each other. A Type I error is a false alarm: you declare a difference that is not there, and the significance level is exactly the probability of that, 5% if you use 5%. A Type II error is a miss: there is a real difference and your test fails to spot it. Power is the probability of avoiding a miss, the chance that your test detects a difference of a given size when it exists, and 80% is the usual target.¹⁹ More data raises power. Nothing else does, short of a bigger real effect.

The diagram runs top to bottom. The first box fixes the sample size per arm and the significance level before any data arrives. Then the visitors are split at random, the test runs to the fixed size, and only then is the p value read against the line. The takeaway is the order: the decision rule comes first, the data second, and the peeking never.

The most common test in an online shop compares two conversion rates, one from each version of a page, and it is called the two proportion z test.²⁰

$$z = \frac{p_1 - p_2}{\sqrt{p(1-p)\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}$$

$z$ is the test statistic, the observed difference measured in standard errors, so a $z$ of 2 means the difference is twice as big as the noise you would expect. $p_1$ and $p_2$ are the two conversion rates, and $n_1$ and $n_2$ are the numbers of visitors behind each. $p$ with no subscript is the pooled rate, all the orders from both versions divided by all the visitors from both, which is the best single estimate if the null hypothesis is true. The expression under the root is the standard error of the difference between two proportions. The p value for a two sided test is the share of the normal curve beyond $z$ on both sides, read from a table or a spreadsheet function.

Worked example: Northgate Hampers tests a new checkout

Priya splits a fortnight's traffic in two. The old checkout had 20,000 visitors and 500 orders, a rate of $p_1 = 0.025$. The new checkout had 20,000 visitors and 560 orders, so $p_2 = 0.028$. The difference on top is $p_2 - p_1 = 0.003$, three tenths of a percentage point in favour of the new version. The pooled rate is all the orders over all the visitors, $p = \frac{500 + 560}{40{,}000} = \frac{1{,}060}{40{,}000} = 0.0265$, and $1 - p = 0.9735$. Now the bottom: $p(1-p) = 0.0265 \times 0.9735 = 0.02580$. The bracket is $\frac{1}{20{,}000} + \frac{1}{20{,}000} = 0.0001$. Multiply: $0.02580 \times 0.0001 = 0.00000258$. Square root: $\sqrt{0.00000258} = 0.00160$. So $z = \frac{0.003}{0.00160} = 1.87$. A $z$ of 1.87 leaves about 3% of the normal curve in each tail, so the two sided p value is about 0.06. That is above 0.05, so the result is not significant at the 5% level. The new checkout probably is a little better, but a fortnight of traffic is not enough to be confident of it, and the next section says how much would be.

Where it goes wrong

Peeking every day and stopping on the first green day. A p value bounces around as data comes in, and if you check it daily and stop the moment it dips below 0.05, you will declare a winner far more often than one in twenty times when there is no real difference, because you are giving randomness twenty chances to look convincing.²¹ Decide the sample size first, run to it, read the result once. Everything else is a slot machine with a dashboard.

Sample size

Before a test, the question is not "how long shall I run it?" but "how many visitors do I need in each arm to see the difference I care about?". That depends on four things: the two rates you want to be able to tell apart, how sure you want to be when you call a winner, and how sure you want to be of not missing a real one. Put those into the formula and it hands back a number of visitors. The time then follows from your traffic, not the other way round.²²

$$n = \frac{(z_{\alpha/2} + z_{\beta})^2 \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_1 - p_2)^2}$$

$n$ is the number of visitors needed in each arm of the test. $z_{\alpha/2}$ is the normal multiplier for your significance level, 1.96 for 5% two sided, the same 1.96 as in the confidence interval. $z_{\beta}$ is the normal multiplier for the miss rate you will tolerate: for 80% power, $\beta$ is 20%, and $z_{\beta}$ is 0.84. $p_1$ is the current rate and $p_2$ is the rate you hope to reach, both as decimals, and the square bracket holds the variance of each, added together. The denominator is the difference you want to detect, squared, which is why detecting a small lift needs a huge sample: halve the lift and you need four times the visitors.

Worked example: how long must Northgate Hampers run the checkout test?

Priya wants to detect a lift from 2.5% to 2.8% at 5% significance with 80% power. The multipliers are $z_{\alpha/2} = 1.96$ and $z_{\beta} = 0.84$, so $(1.96 + 0.84)^2 = 2.8^2 = 7.84$. The bracket: $p_1(1-p_1) = 0.025 \times 0.975 = 0.024375$, and $p_2(1-p_2) = 0.028 \times 0.972 = 0.027216$, which add to $0.051591$. The top is $7.84 \times 0.051591 = 0.40447$. The bottom is $(0.025 - 0.028)^2 = (0.003)^2 = 0.000009$. So $n = \frac{0.40447}{0.000009} = 44{,}941$, about 45,000 visitors per arm, 90,000 in all. At December's pace of 40,000 visitors a month that is a little over two months, and in a quiet spring month it could be four. That is why the fortnight in the previous section came up short: it had less than half the visitors it needed.

Where it goes wrong

Choosing the test length from the calendar. "Run it for two weeks" is a habit, not a calculation. Two weeks is right only if two weeks happens to deliver the sample size the formula asks for, and for a shop converting at 2.5% it usually does not. The length of a test is an output. The inputs are the lift you care about and the traffic you have. If the number of visitors needed is more than you will see in a season, test a bigger change instead.

Correlation

Correlation measures how closely two things move together. When one is high, is the other high too? Pearson's correlation coefficient, written $r$, puts the answer on a scale from minus 1 to 1. At 1 the two move in perfect lockstep, at minus 1 one rises exactly as the other falls, and at 0 there is no straight line relationship at all. Its square, $r^2$, has a friendlier reading: the share of the variation in one quantity that lines up with the other.²³

$$r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}$$

$r$ is the correlation coefficient. $x_i$ and $y_i$ are the paired values, for example the emails sent and the revenue taken in month $i$. $\bar{x}$ and $\bar{y}$ are their means. The top adds up, for every pair, the deviation of $x$ times the deviation of $y$: when both are above their means or both below, the product is positive, and when they are on opposite sides it is negative, so the sum is large and positive when the two move together. The bottom is the square root of the two sums of squared deviations multiplied, which is the same scale as the top, so the ratio can never leave the range from minus 1 to 1.

Worked example: do more emails mean more revenue at Northgate Hampers?

Priya has twelve months of two numbers: emails sent and revenue taken. Over the year she sent a mean of 9,000 emails a month and took a mean of £42,500 a month. Working out the twelve pairs of deviations gives three sums. The sum of the products of the deviations, the top of the formula, is $\sum (x_i - \bar{x})(y_i - \bar{y}) = 100{,}000{,}000$. The sum of squared deviations for emails is $\sum (x_i - \bar{x})^2 = 40{,}000{,}000$, and for revenue it is $\sum (y_i - \bar{y})^2 = 650{,}000{,}000$. Multiply the two bottom sums: $40{,}000{,}000 \times 650{,}000{,}000 = 2.6 \times 10^{16}$. Square root: $\sqrt{2.6 \times 10^{16}} = 161{,}245{,}155$. Then $r = \frac{100{,}000{,}000}{161{,}245{,}155} = 0.62$. A positive, moderate correlation. Squaring it, $r^2 = 0.62^2 = 0.38$. About 38% of the month to month variation in revenue lines up with the variation in email volume. The other 62% moves with something else: the season, the adverts, the weather, luck.

Where it goes wrong

Correlation is not causation. Email volume and revenue both rise in December, because Priya sends more emails before Christmas and because people buy hampers before Christmas. The two are correlated through the calendar, not because one causes the other. The way to find out whether emails cause revenue is a controlled test, where some customers get the extra email and some do not. A correlation is a reason to run that test, never a substitute for it.

Regression

Regression fits a line through the data so you can use one quantity to predict another. Where correlation says how tightly two things move together, regression says by how much: for every extra email sent, how much extra revenue, on average? The simplest version fits a straight line and is called simple linear regression. The line is chosen so that the sum of the squared vertical distances from the points to the line is as small as possible, which is why it is also called least squares.²⁴

$$y = \beta_0 + \beta_1 x + \varepsilon$$

$y$ is the quantity you want to predict, the outcome. $x$ is the quantity you predict it from, the input. $\beta_0$, beta nought, is the intercept, the predicted value of $y$ when $x$ is zero. $\beta_1$ is the slope, how much $y$ changes for a one unit change in $x$, and it is the number most people actually want. $\varepsilon$, epsilon, is the error, everything about $y$ that the line does not explain, a reminder that the line is an average, not a promise. The slope is the top of the correlation formula divided by the sum of squared deviations of $x$, and the intercept follows from the two means, $\beta_0 = \bar{y} - \beta_1 \bar{x}$. The quality of the fit is reported as $R^2$, which in simple regression is exactly the $r^2$ from the correlation section.

Worked example: the email line at Northgate Hampers

Using the twelve months from the correlation section: the top sum was 100,000,000 and the sum of squared email deviations was 40,000,000, so the slope is $\beta_1 = \frac{100{,}000{,}000}{40{,}000{,}000} = 2.5$. Each extra email sent in a month goes with £2.50 of extra revenue that month, on average. The intercept uses the two means, £42,500 and 9,000 emails: $\beta_0 = 42{,}500 - 2.5 \times 9{,}000 = 42{,}500 - 22{,}500 = 20{,}000$. The fitted line is $\text{revenue} = 20{,}000 + 2.5 \times \text{emails}$. If Priya sends 4,000 more emails next month than usual, the line predicts $2.5 \times 4{,}000 = 10{,}000$, so £10,000 more revenue. That prediction is only trustworthy inside the range of the data the line was fitted to. If her monthly sends ran from 6,000 to 12,000, a prediction at 13,000 is a modest stretch and one at 50,000 is fiction. And $R^2 = 0.38$, so the line explains 38% of the variation and the error term carries the rest.

A short note on logistic regression, because it turns up whenever the outcome is a yes or a no: did this visitor order, did this customer come back, did this parcel get returned. A straight line is the wrong shape for a probability, which has to stay between 0 and 1, so logistic regression fits a line to the log odds instead.²⁵

$$\ln\frac{p}{1-p} = \beta_0 + \beta_1 x$$

$p$ is the probability of the yes outcome. $\frac{p}{1-p}$ is the odds, the probability of yes divided by the probability of no, so a 20% chance is odds of 0.25, or one to four. $\ln$ is the natural logarithm, which stretches the odds out so they can run from minus infinity to plus infinity and a straight line fits. $\beta_0$ and $\beta_1$ are an intercept and a slope as before, except that the slope now says how much the log odds change for a unit change in $x$. The articles that use it always translate the result back into a probability for you.

Where it goes wrong

Two things. Extrapolating beyond the data, as above: a line fitted to months with 6,000 to 12,000 emails knows nothing about what happens at 50,000, where people unsubscribe in droves. And reading the slope as causal. The 2.5 says revenue is higher in months with more emails. It does not say sending an email creates £2.50, because the months with more emails were also the months before Christmas. Only a controlled test can turn a slope into a cause.

Price elasticity

Price elasticity of demand measures how strongly customers react to a change in price. It is the percentage change in the quantity sold divided by the percentage change in price. If a 10% price rise loses you 12% of your sales, the elasticity is minus 1.2. The sign is almost always negative, because people buy less when things cost more, so in conversation the minus is often dropped and people talk about the size. Above 1 in size, demand is elastic: customers react more than proportionally, and a price rise loses you revenue. Below 1, demand is inelastic: a price rise loses you some customers but total revenue goes up.²⁶

$$\varepsilon = \frac{\%\Delta Q}{\%\Delta P}, \qquad \varepsilon_{\text{mid}} = \frac{(Q_2 - Q_1)/\left(\frac{Q_1 + Q_2}{2}\right)}{(P_2 - P_1)/\left(\frac{P_1 + P_2}{2}\right)}$$

$\varepsilon$ is the elasticity. $\%\Delta Q$ is the percentage change in quantity sold and $\%\Delta P$ is the percentage change in price. In the second, midpoint, version, $Q_1$ and $Q_2$ are the quantities before and after, $P_1$ and $P_2$ the prices before and after, and each change is measured against the average of the two values rather than the starting one. The midpoint version gives the same answer whether the price went up or down, which the simple version does not, because of the asymmetry you met under percentages.

Worked example: Northgate Hampers raises the price of the Classic

The Classic hamper sells at £49, and in a normal month Priya sells 400 of them. She raises the price to £54. Sales fall to 352. The price change is $\frac{54 - 49}{49} \times 100 = \frac{5}{49} \times 100 = 10.2\%$. The quantity change is $\frac{352 - 400}{400} \times 100 = \frac{-48}{400} \times 100 = -12\%$. So the elasticity is $\varepsilon = \frac{-12}{10.2} = -1.18$. Demand for the Classic is elastic, and the revenue confirms it: before, $400 \times 49 = 19{,}600$, so £19,600. After, $352 \times 54 = 19{,}008$, so £19,008. The price rise cost Priya £592 a month on that one product. The midpoint version, for completeness: quantity change $\frac{-48}{376} = -0.1277$, price change $\frac{5}{51.5} = 0.0971$, so $\varepsilon_{\text{mid}} = \frac{-0.1277}{0.0971} = -1.31$. Same verdict, elastic either way.

Where it goes wrong

Assuming one elasticity for every price point and every season. The Classic may be elastic at £49 in March, when a hamper is an impulse, and inelastic at £49 in the second week of December, when it is a solved problem and the buyer is relieved. Elasticity is a local measurement, true near the price and the time it was measured. Measure it again before you trust it somewhere else.

Growth rates and compounding

A growth rate is a percentage change over a period. Compounding is what happens when the growth of one period becomes the base for the next, so that growth builds on growth. A shop that grows 5% a month does not grow 60% a year, because by month two the 5% applies to a slightly bigger shop, and so on. The compound annual growth rate, CAGR, is the single steady yearly rate that would take you from where you started to where you ended, smoothing out the bumps in between.²⁷

$$\text{CAGR} = \left(\frac{V_{\text{end}}}{V_{\text{start}}}\right)^{1/t} - 1$$

$\text{CAGR}$ is the compound annual growth rate, as a decimal. $V_{\text{end}}$ is the value at the end of the period and $V_{\text{start}}$ the value at the beginning, so the fraction is the total growth multiple. $t$ is the number of years, and raising the multiple to the power $1/t$ takes the $t$th root, which spreads the total growth evenly across the years. The $- 1$ at the end turns a multiple into a rate: a multiple of 1.176 is a rate of 0.176.

A handy companion is the rule of 70: divide 70 by the growth rate as a percentage per period and you get, near enough, the number of periods it takes to double.²⁸ At 7% a year, ten years. At 17.6% a year, four.

Worked example: Northgate Hampers over three years

Three years ago the shop turned over £400,000. This year it turned over £650,000. The growth multiple is $\frac{650{,}000}{400{,}000} = 1.625$. With $t = 3$, take the cube root: $1.625^{1/3} = 1.176$. Subtract 1: the CAGR is $0.176$, so 17.6% a year. Check it forwards: $400{,}000 \times 1.176 \times 1.176 \times 1.176 = 400{,}000 \times 1.626 = 650{,}400$, which is right to the rounding. Now the monthly case. Growing 5% a month for twelve months gives a multiple of $1.05^{12} = 1.796$, so 79.6% a year, not the 60% you get by adding twelve fives. The extra 19.6 points are growth on growth. And the rule of 70: at 17.6% a year the shop doubles in about $\frac{70}{17.6} = 4$ years, which matches the arithmetic, since $1.176^4 = 1.91$.

Where it goes wrong

Adding percentages across periods. Twelve months of 5% is 79.6%, not 60%. Three years of 17.6% is a multiple of 1.625, not 1.528. The error is small over two periods and enormous over twenty. The same mistake in reverse hides in "we lost 10% a year for five years", which is a fall to 59% of the start, not to 50%. Multiply, never add.

Discounting and present value

A pound today is worth more than a pound in three years, because today's pound can be invested, or spent on stock that sells, or simply held without the risk that the future pound never arrives. Discounting is the arithmetic that puts a future sum into today's money, so that costs now and revenues later can be compared on one scale. The rate you shrink by each year is the discount rate, and it reflects what else you could do with the money plus how risky the future sum is. The present value is the result, and the net present value of a project is the present value of everything it brings in minus the present value of everything it costs.²⁹

$$PV = \frac{FV}{(1+d)^t}$$

$PV$ is the present value, what the future sum is worth today. $FV$ is the future value, the sum you expect to receive. $d$ is the discount rate per period as a decimal, 0.10 for 10%. $t$ is the number of periods until the money arrives, and $(1+d)^t$ is compounding run backwards: the factor the money would have grown by, which you now divide by instead. The net present value adds up the present values of every cash flow, in and out, and a positive total means the project is worth more than it costs in today's money.³⁰

Worked example: Northgate Hampers weighs a warehouse move

Priya is offered a deal that brings in £10,000 of extra margin in three years' time, in exchange for a commitment today. She uses a discount rate of 10%, roughly what the money would earn in the business otherwise. $(1 + 0.10)^3 = 1.1 \times 1.1 \times 1.1 = 1.331$. So $PV = \frac{10{,}000}{1.331} = 7{,}513$. The £10,000 in three years is worth £7,513 today. If the commitment costs £7,000 now, the net present value is $7{,}513 - 7{,}000 = 513$, a positive £513, and the deal is just worth doing. If it costs £8,000, the net present value is minus £487, and however good £10,000 sounds, she would be better off keeping the £8,000 in the business.

Where it goes wrong

Comparing a cost today with a revenue in three years at face value. "Spend £8,000 now, get £10,000 later" sounds like a £2,000 profit. At a 10% discount rate it is a £487 loss. The further out the revenue and the riskier the business, the harsher the discount, and a lot of growth plans that look handsome in a spreadsheet only look that way because nobody discounted the back half.

Customer lifetime value

Customer lifetime value, CLV, is what a customer is worth to you over the whole time they keep buying, in today's money. It tells you how much you can afford to spend to win one. The version below assumes a steady margin from a customer each period, a fixed share of customers staying from one period to the next, and future margin discounted. It is the standard textbook form.³¹ Retention sits at its heart for the reason a well known Harvard Business Review article gave: customer defections have a far bigger effect on profit than most businesses realise, because a customer kept is a customer you do not have to win again.³²

$$CLV = M \times \frac{r}{1 + d - r}$$

$CLV$ is the lifetime value of one customer. $M$ is the margin you earn from a customer in one period, after the cost of the goods and of serving them, not the revenue. $r$ is the retention rate, the share of customers still buying one period later, as a decimal. $d$ is the discount rate per period, as in the section above. The fraction $\frac{r}{1+d-r}$ turns a single period's margin into the sum of all the future periods, each one a little less likely to happen and a little less valuable because it is further away. When retention is high, the denominator gets small and the value climbs steeply.

Worked example: what a Northgate Hampers customer is worth

A typical repeat customer earns Priya £400 of margin a year, 70% of customers order again the following year, and she discounts at 10%. So $M = 400$, $r = 0.7$, $d = 0.1$. The denominator is $1 + 0.1 - 0.7 = 0.4$. The fraction is $\frac{0.7}{0.4} = 1.75$. The lifetime value is $CLV = 400 \times 1.75 = 700$, so £700. Now suppose a better follow up email lifts retention to 80%. The denominator becomes $1 + 0.1 - 0.8 = 0.3$, the fraction $\frac{0.8}{0.3} = 2.667$, and $CLV = 400 \times 2.667 = 1{,}067$. Ten points of retention added £367 to the value of every customer, more than half as much again. That is what makes retention the most powerful lever in the formula, and the chart shows the whole curve.

Along the bottom is the retention rate, from 50% to 90%. Up the side is the lifetime value in pounds, with margin held at £400 a year and the discount rate at 10%. At 50% retention the value is £333, at 60% it is £480, at 70% it is £700, at 80% it is £1,067 and at 90% it is £1,800. The curve is not a straight line. Each extra ten points of retention is worth more than the last, because customers who stay longer compound. The takeaway is that a shop with 60% retention and one with 80% can charge the same prices and run the same adverts, and the second can afford to spend more than twice as much to win each customer.

Where it goes wrong

Feeding revenue instead of margin into $M$. A customer who spends £1,000 a year on hampers that cost £600 to buy, pack and post earns you £400, not £1,000. Put the £1,000 in and the formula tells you a customer is worth £1,750, and you will cheerfully spend £800 to acquire one who is actually worth £700. It is the most expensive wrong number in marketing, and I see it in most spreadsheets I am shown.

Marginal against average

The average return on a budget is total return divided by total spend. The marginal return is the return on the next pound: what the last slice of spend brought in, compared with what it cost. The two are the same only when every pound works equally hard, and in advertising they never do. The first pounds go on the people most likely to buy, and each further pound reaches people a little less likely, so the marginal return falls as spend rises. That is diminishing returns, and it is why the decision to spend more should be made on the marginal number.³¹

$$\text{marginal return} = \frac{\Delta \text{revenue}}{\Delta \text{spend}}$$

The $\text{marginal return}$ is the return on the additional spend, as a multiple. $\Delta \text{revenue}$, delta revenue, is the change in revenue brought by the extra spend: revenue after minus revenue before. $\Delta \text{spend}$ is the extra spend itself. Compare it with the average return, which is simply total revenue over total spend, and the gap between the two tells you how fast returns are falling off.

Along the bottom are three slices of the same budget: the first £10,000, the next £5,000, and the whole £15,000 taken together. Up the side is the revenue returned per pound spent. The first slice returns £4.00 per pound, the next only £2.00, and the average over everything is £3.30. The takeaway is that the average bar looks healthy while the marginal bar has already halved. Anyone who sees only the average will keep spending.

Worked example: Northgate Hampers scales its advertising

Priya spends £10,000 on adverts in November and they bring in £40,000 of revenue. The average return is $\frac{40{,}000}{10{,}000} = 4.0$, four pounds back for each pound spent. Encouraged, she adds £5,000. Revenue rises to £50,000. The extra £5,000 brought in an extra £10,000, so the marginal return is $\frac{50{,}000 - 40{,}000}{15{,}000 - 10{,}000} = \frac{10{,}000}{5{,}000} = 2.0$. Two pounds back per pound, half of what the first slice earned. The average over the whole £15,000 is $\frac{50{,}000}{15{,}000} = 3.3$, still handsome, and entirely misleading about what the next £5,000 would do. If her margin on hamper revenue is about 40%, then £2.00 of revenue per pound is 80p of margin per pound, and the extra slice lost her money.

Where it goes wrong

Scaling a budget on the average. "We get 3.3 back, so let's double the spend" assumes the next £15,000 behaves like the first, when the last £5,000 has just told you it will not. Always ask what the last increment returned, not what the whole pile averaged. The place to stop is where the marginal return drops to the point where the margin on it no longer covers the spend, and in Priya's case that point has already passed.

Logarithms and the log scale

A logarithm answers the question "how many times do I have to multiply by the base to get this number?". With base 10, the logarithm of 1,000 is 3, because $10 \times 10 \times 10 = 1{,}000$, and the logarithm of 100,000 is 5. The natural logarithm, written $\ln$, uses the base $e$, about 2.718, which is the natural rate of continuous growth and is why $\ln$ turns up in growth formulas and in logistic regression. Logarithms turn multiplication into addition, which is their whole usefulness: equal ratios become equal distances.³³

$$\log(ab) = \log a + \log b, \qquad \log\frac{a}{b} = \log a - \log b$$

$\log$ is the logarithm to whatever base you are using, and the rules hold for every base. $a$ and $b$ are any two positive numbers. The first rule says the log of a product is the sum of the logs. The second, which follows from it, says the log of a ratio is the difference of the logs. So on a log scale, going from 1,000 to 2,000 is the same distance as going from 100,000 to 200,000, because both are a doubling and both add $\log 2$.

Worked example: two product lines on one chart at Northgate Hampers

Priya sells a card and ribbon line that turns over about £1,000 a month and the hamper line that turns over about £100,000. On an ordinary chart the card line is a flat smudge along the bottom and a 10% rise in it is invisible, while a 10% rise in the hamper line is a visible £10,000 step. On a log scale the picture changes. $\log_{10} 1{,}000 = 3$ and $\log_{10} 1{,}100 = 3.041$, a step of $0.041$. $\log_{10} 100{,}000 = 5$ and $\log_{10} 110{,}000 = 5.041$, a step of $0.041$. The same percentage move is the same height at both ends of the chart. That is exactly what you want when you are asking "which line is growing faster?", and exactly what you do not want when you are asking "which line brings in more money?".

Where it goes wrong

Reading a log chart as if it were linear. On a log scale the gap between the gridlines for £1,000 and £10,000 is the same size as the gap between £10,000 and £100,000, so a line that looks like it is climbing steadily is actually growing by a fixed percentage, which is exponential growth, and a line that looks like it is slowing may be adding the same number of pounds every month. Always read the axis labels before the shape.

Indexing to 100, and base rates

Indexing rescales a series so that one chosen point equals 100 and everything else is expressed relative to it. It strips out the units and lets you put revenue, visitors and orders on one chart to compare their growth, which is how the official inflation figures are published.³⁴ The base rate is a different idea that shares the section because both are about the number you compare against. The base rate is how common something is in the first place, and the base rate fallacy is the mistake of judging a result without it.³⁵

$$I_t = \frac{v_t}{v_{\text{base}}} \times 100$$

$I_t$ is the index value at time $t$. $v_t$ is the actual value at time $t$ and $v_{\text{base}}$ is the value at the base period, the one you have chosen to call 100. Dividing by the base and multiplying by 100 makes the base period exactly 100 and every other period a percentage of it, so 120 means 20% above the base and 85 means 15% below.

Worked example: a year of page views, and a fraud filter

Northgate Hampers had 85,000 page views in January and 102,000 in December. Set January as the base. January's index is $\frac{85{,}000}{85{,}000} \times 100 = 100$ by construction. December's is $\frac{102{,}000}{85{,}000} \times 100 = 1.2 \times 100 = 120$. Page views ended the year 20% above where they started, and if Priya indexes orders and revenue the same way she can lay all three lines on one chart and see which grew fastest, without the units getting in the way.

Now the base rate. Priya installs a fraud filter that is 99% accurate, meaning it correctly flags 99% of fraudulent orders and correctly passes 99% of honest ones. Only 0.1% of her orders are fraudulent. Out of 10,000 orders, 10 are fraudulent and 9,990 are honest. The filter catches about $0.99 \times 10 = 9.9$ of the frauds, call it 10. But it also wrongly flags 1% of the honest orders, $0.01 \times 9{,}990 = 99.9$, call it 100. So of about 110 flagged orders, 100 are honest customers. For every fraud the filter stops, it holds up around ten honest orders, and "99% accurate" is true and useless at the same time. The low base rate of fraud is what does it, exactly as in Bayes' theorem.

Where it goes wrong

Quoting a lift without its base. "The new banner lifted sign ups by 50%" sounds like a transformation. If the sign up rate was 0.2%, a 50% lift takes it to 0.3%, one more sign up per thousand visitors. Both numbers are honest. Only the second tells you whether to care. Whenever somebody gives you a percentage change, ask what the starting rate was, and if they give you an accuracy, ask how rare the thing is.

Seasonality, moving averages and year on year

Seasonality is the regular pattern that repeats every year: for a hamper shop, a huge December, a quiet summer, a lift around Mother's Day. Until you have separated it out you cannot tell whether the business is growing or whether it is just Christmas. Two tools do most of the work. A moving average smooths the series by averaging a window of recent periods, and a twelve month window averages out one full cycle of seasons.³⁶ A seasonal index measures each month against that smoothed level, so you know how much of a month is season and how much is you.³⁷ Year on year comparison sidesteps the problem altogether by comparing a month with the same month a year earlier.

$$MA_t = \frac{1}{k}\sum_{i=0}^{k-1} x_{t-i}$$

$MA_t$ is the moving average at time $t$. $k$ is the window, the number of periods you average over, 12 for a monthly series with yearly seasons. $x_{t-i}$ is the value $i$ periods before $t$, so the sum runs over the current period and the $k-1$ before it. The $\frac{1}{k}$ turns the sum into a mean. The seasonal index for a month is then that month's actual value divided by the moving average, so 1.0 means an ordinary month, 2.0 means double, and 0.5 means half.

Along the bottom are the twelve months of last year. Up the side is sales in pounds. The jagged line is the actual sales in each month, and the flat line is the twelve month moving average of £42,500, which is the same for every month because the window always covers exactly one whole year. The takeaway is how far the real months sit from the average: nine months below it, and two months, November and December, at more than double it. Almost all of the movement in the jagged line is season, not growth.

Worked example: Northgate Hampers' seasonal indices

Last year's monthly sales, in pounds, were 25,000 in January, 27,000 in February, 32,000 in March, 34,000 in April, 36,000 in May, 26,000 in June, 21,250 in July, 24,000 in August, 34,000 in September, 50,000 in October, 98,750 in November and 102,000 in December. Add them up: the year came to £510,000. The twelve month moving average at December is $MA_{\text{Dec}} = \frac{510{,}000}{12} = 42{,}500$. Now the indices. December: $\frac{102{,}000}{42{,}500} = 2.4$. A December is 2.4 ordinary months. July: $\frac{21{,}250}{42{,}500} = 0.5$, half an ordinary month. November: $\frac{98{,}750}{42{,}500} = 2.32$. With those indices in hand, Priya can deseasonalise any month by dividing by its index. A December of £108,000 next year would deseasonalise to $\frac{108{,}000}{2.4} = 45{,}000$, which against this year's £42,500 is real growth of about 6%. The same £108,000 compared with November's £98,750 would look like growth of 9.4%, and nearly all of that is the calendar.

Where it goes wrong

Comparing December with November and calling it growth. December beats November every year at a hamper shop, by about $\frac{2.4}{2.32}$, so roughly 3%, whatever the business is doing. The honest comparison is December against last December, or this December's deseasonalised figure against the moving average. Any report that compares consecutive months in a seasonal business, without an index or a year on year figure beside it, is measuring the calendar.

Matrices

A matrix is a grid of numbers with rows and columns, and a vector is a single column of them. That is all. They appear in the more advanced articles because they are the compact way to write many equations at once. A media mix model, which estimates how much revenue each channel drives, is dozens of equations, one per week, each saying "this week's revenue is this much per pound of search plus this much per pound of social plus this much per pound of email". Written as a matrix, that is one line.³⁸ Principal components, which find the few underlying patterns in many columns of data, are a matrix calculation too.

$$\mathbf{y} = \mathbf{X}\boldsymbol{\beta}$$

$\mathbf{y}$ is a vector of outcomes, one per row, for example revenue in each month. $\mathbf{X}$ is the matrix of inputs, one row per month and one column per channel, holding the spend on each channel in each month. $\boldsymbol{\beta}$ is a vector of coefficients, one per channel, each saying how much revenue a pound in that channel returns. Bold type is the convention for "a whole grid or column, not a single number". Multiplying a matrix by a vector means: for each row, multiply each entry in the row by the matching entry in the vector and add up the products, giving one number per row. That is the arithmetic of the regression line, done for every row at once.

Worked example: three channels, three months at Northgate Hampers

Priya's spend in the last three months of the year, by channel, makes a 3 by 3 matrix with one row per month and one column per channel, in the order search, social, email. October: 2,000, 1,000, 500. November: 4,000, 2,000, 500. December: 6,000, 3,000, 1,000. Suppose the returns per pound are 4 for search, 2 for social and 10 for email, so the vector $\boldsymbol{\beta}$ is $(4, 2, 10)$. Multiply row by vector. October: $2{,}000 \times 4 + 1{,}000 \times 2 + 500 \times 10 = 8{,}000 + 2{,}000 + 5{,}000 = 15{,}000$. November: $4{,}000 \times 4 + 2{,}000 \times 2 + 500 \times 10 = 16{,}000 + 4{,}000 + 5{,}000 = 25{,}000$. December: $6{,}000 \times 4 + 3{,}000 \times 2 + 1{,}000 \times 10 = 24{,}000 + 6{,}000 + 10{,}000 = 40{,}000$. The vector $\mathbf{y}$ is $(15{,}000, 25{,}000, 40{,}000)$: the revenue attributed to advertising in each month. A media mix model runs this backwards. It knows the spend matrix and the revenue, and it searches for the $\boldsymbol{\beta}$ that fits best, which is least squares with three inputs instead of one.

Where it goes wrong

Nothing to fear here, and that is the point. A matrix is bookkeeping for many equations at once. When an article shows one, read it row by row: each row is one ordinary sum, the kind you have just done three of. The mathematics that sits on top, inverses and decompositions, is how the computer finds the best fitting coefficients, and you never need to perform it by hand to understand what the result means.

How to use this page

If you came here from an article, you have what you needed. If you read the whole thing, you now have every tool the Intelligence articles rely on, from a percentage point to a matrix, with one shop's numbers running through all of them. When a future article needs an idea that is not here yet, the idea gets added here first, so this page keeps growing and the anchors keep working. If something on it is unclear, that is my fault and not yours, and I would like to know.

Footnotes

1. OpenStax, Introductory Statistics 2e, section 2.5, Measures of the centre of the data, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/2-5-measures-of-the-center-of-the-data

2. OpenStax, Introductory Statistics 2e, section 2.6, Skewness and the mean, median and mode, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/2-6-skewness-and-the-mean-median-and-mode

3. OpenStax, Introductory Statistics 2e, section 2.7, Measures of the spread of the data, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/2-7-measures-of-the-spread-of-the-data

4. Wikipedia, Coefficient of variation, Wikimedia Foundation, retrieved 2026, https://en.wikipedia.org/wiki/Coefficient_of_variation

5. NIST/SEMATECH, Engineering Statistics Handbook, section 1.3.6.6.1, Normal distribution, National Institute of Standards and Technology, https://www.itl.nist.gov/div898/handbook/eda/section3/eda3661.htm

6. OpenStax, Introductory Statistics 2e, section 6.1, The standard normal distribution, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/6-1-the-standard-normal-distribution

7. NIST/SEMATECH, Engineering Statistics Handbook, section 1.3.5.11, Measures of skewness and kurtosis, National Institute of Standards and Technology, https://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm

8. OpenStax, Introductory Statistics 2e, section 4.3, Binomial distribution, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/4-3-binomial-distribution

9. OpenStax, Introductory Statistics 2e, section 4.6, Poisson distribution, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/4-6-poisson-distribution

10. OpenStax, Introductory Statistics 2e, section 7.1, The central limit theorem for sample means, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/7-1-the-central-limit-theorem-for-sample-means-averages

11. OpenStax, Introductory Statistics 2e, section 3.3, Two basic rules of probability, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/3-3-two-basic-rules-of-probability

12. OpenStax, Introductory Statistics 2e, section 3.2, Independent and mutually exclusive events, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/3-2-independent-and-mutually-exclusive-events

13. Penn State Eberly College of Science, STAT 414 Introduction to Probability Theory, Lesson 6, Bayes' theorem, https://online.stat.psu.edu/stat414/Lesson06.html

14. OpenStax, Introductory Statistics 2e, section 4.2, Mean or expected value and standard deviation, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/4-2-mean-or-expected-value-and-standard-deviation

15. OpenStax, Introductory Statistics 2e, section 1.1, Definitions of statistics, probability and key terms, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/1-1-definitions-of-statistics-probability-and-key-terms

16. OpenStax, Introductory Statistics 2e, section 8.3, A population proportion, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/8-3-a-population-proportion

17. OpenStax, Introductory Statistics 2e, section 8.1, A single population mean using the normal distribution, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/8-1-a-single-population-mean-using-the-normal-distribution

18. OpenStax, Introductory Statistics 2e, section 9.1, Null and alternative hypotheses, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/9-1-null-and-alternative-hypotheses

19. OpenStax, Introductory Statistics 2e, section 9.2, Outcomes and the Type I and Type II errors, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/9-2-outcomes-and-the-type-i-and-type-ii-errors

20. OpenStax, Introductory Statistics 2e, section 10.3, Comparing two independent population proportions, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/10-3-comparing-two-independent-population-proportions

21. Evan Miller, How not to run an A/B test, evanmiller.org, 2010, https://www.evanmiller.org/how-not-to-run-an-ab-test.html

22. NCSS, PASS Sample Size Software documentation, Tests for two proportions, NCSS LLC, https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Tests_for_Two_Proportions.pdf

23. OpenStax, Introductory Statistics 2e, section 12.3, The regression equation, Rice University, 2023, https://openstax.org/books/introductory-statistics-2e/pages/12-3-the-regression-equation

24. NIST/SEMATECH, Engineering Statistics Handbook, section 4.1.4.1, Linear least squares regression, National Institute of Standards and Technology, https://www.itl.nist.gov/div898/handbook/pmd/section1/pmd141.htm

25. Penn State Eberly College of Science, STAT 501 Regression Methods, Lesson 13, Weighted least squares and logistic regressions, https://online.stat.psu.edu/stat501/Lesson13.html

26. OpenStax, Principles of Economics 3e, section 5.1, Price elasticity of demand and price elasticity of supply, Rice University, 2022, https://openstax.org/books/principles-economics-3e/pages/5-1-price-elasticity-of-demand-and-price-elasticity-of-supply

27. Corporate Finance Institute, CAGR (Compound Annual Growth Rate), CFI Education, https://corporatefinanceinstitute.com/resources/valuation/what-is-cagr/

28. Corporate Finance Institute, Rule of 70, CFI Education, https://corporatefinanceinstitute.com/resources/data-science/rule-of-70/

29. OpenStax, Principles of Economics 3e, Appendix C, Present discounted value, Rice University, 2022, https://openstax.org/books/principles-economics-3e/pages/c-present-discounted-value

30. OpenStax, Principles of Finance, section 16.2, Net present value (NPV) method, Rice University, 2022, https://openstax.org/books/principles-finance/pages/16-2-net-present-value-npv-method

31. Neil Bendle, Paul W. Farris, Phillip Pfeifer and David Reibstein, Marketing Metrics, 4th edition, Pearson, 2021, https://www.pearson.com/us/higher-education/program/Bendle-Marketing-Metrics-4th-Edition/PGM2833542.html

32. Frederick F. Reichheld and W. Earl Sasser Jr, Zero Defections: Quality Comes to Services, Harvard Business Review, 1990, https://hbr.org/1990/09/zero-defections-quality-comes-to-services

33. OpenStax, Precalculus 2e, section 4.5, Logarithmic properties, Rice University, 2021, https://openstax.org/books/precalculus-2e/pages/4-5-logarithmic-properties

34. Office for National Statistics, Consumer price indices, a brief guide, 2017, https://www.ons.gov.uk/economy/inflationandpriceindices/articles/consumerpriceindicesabriefguide/2017

35. Wikipedia, Base rate fallacy, Wikimedia Foundation, retrieved 2026, https://en.wikipedia.org/wiki/Base_rate_fallacy

36. NIST/SEMATECH, Engineering Statistics Handbook, section 6.4.2, What are moving average or smoothing techniques, National Institute of Standards and Technology, https://www.itl.nist.gov/div898/handbook/pmc/section4/pmc42.htm

37. NIST/SEMATECH, Engineering Statistics Handbook, section 6.4.4.3, Seasonality, National Institute of Standards and Technology, https://www.itl.nist.gov/div898/handbook/pmc/section4/pmc443.htm

38. OpenStax, Precalculus 2e, section 9.5, Matrices and matrix operations, Rice University, 2021, https://openstax.org/books/precalculus-2e/pages/9-5-matrices-and-matrix-operations