Showing uncertainty honestly: intervals, ranges and error bars

TL;DR

Difficulty: Intermediate

A number from a sample, a test or a model is an estimate, and a chart that shows only the point claims more certainty than you have.

  • A confidence interval is about the average, a prediction interval about the next single value. They answer different questions.
  • Error bars can show standard deviation, standard error or a confidence interval. Always say which, at what level, from how many observations.
  • Show the distribution when its shape could change the decision.
  • Small samples need Wilson intervals and a funnel plot, not a league table.
  • Put a number next to every "likely".
  • Show an A/B test as the difference with its interval, and decide from the ends of that interval.

Your conversion rate, your average order and your test winner are all estimates, and the usual chart hides that. This chapter shows how to draw and word uncertainty so that a board reads it correctly: intervals, labelled error bars, distributions, honest small samples and A/B tests shown as a range, not a badge.

Your number is an estimate, and the chart pretends it is not

Shannon Outfitters is an outdoor gear brand in Limerick. It sells online at shannonoutfitters.example and through one shop on O'Connell Street. Every month Sam, the analyst, sends the owner, Aoife, a board report. Last month it said: online conversion rate 3.1%, average order €68, new product page "the winner, plus 11.7%".

Every one of those numbers is an estimate. The conversion rate comes from one month of visitors, and next month's visitors will behave a little differently. The average order comes from 2,500 orders, not from every order the shop will ever take. The "winner" comes from a two week test. Each number has a range of values that would be just as believable, and the report shows none of them.

That is the reader's question for this chapter: my number is an estimate, so how do I show how sure I am without confusing everyone? The answer is a small set of tools: intervals, error bars that say what they are, distributions instead of lone averages, bands on lines, honest small samples, and numbers next to words like "likely". This chapter builds on choosing the right chart for the question. The statistics behind it are explained in statistical significance and error margins. Here the focus is on the picture and the words.

All figures for Shannon Outfitters in this article are invented for teaching. Research findings are sourced in the footnotes.

Example: what "3.1%" is hiding

Last month 21,300 people visited the online shop and 660 of them ordered. That is a conversion rate of 3.1%. If Sam could rerun the same month with a fresh set of similar visitors, the rate would not come out at exactly 3.1% again. It might be 2.9% or 3.3%. The single number looks precise to one decimal place. The truth is that Sam knows it to within about a quarter of a percentage point either way, and the chart gives Aoife no way of knowing that.

Nothing is wrong with the number. What is missing is the second half of the sentence: "3.1%, and probably somewhere between 2.9% and 3.3%". The rest of this chapter is about writing and drawing that second half.

A confidence interval says how sure you are about an average

A confidence interval is a range around an estimate, built so that the method captures the true value most of the time. A 95% confidence interval comes from a recipe that, used on many samples, contains the true value in 95 out of 100 of them. It describes the precision of a summary number, such as an average or a rate, not the spread of individual customers.

For an average, the usual recipe is:

xˉ±z⋅sn\bar{x} \pm z \cdot \frac{s}{\sqrt{n}}

Here xˉ\bar{x} is the sample average, ss is the standard deviation of the individual values (how far a typical value sits from the average), nn is the number of values, sn\frac{s}{\sqrt{n}} is the standard error (how much the average itself would wobble from sample to sample), and zz is the multiplier for the confidence level, z=1.96z = 1.96 for 95%. The Foundations: confidence intervals section builds this up from scratch, and Foundations: spread explains the standard deviation.

Notice the n\sqrt{n}. Four times as much data only halves the width of the interval. That is why precision gets expensive.

Example: the average online order

In October Shannon Outfitters took 2,500 online orders. The average was €68 and the standard deviation was €52. Sam works out the interval step by step.

Step 1, the standard error:

522500=5250=1.04\frac{52}{\sqrt{2500}} = \frac{52}{50} = 1.04

Here s=52s = 52 and n=2500n = 2500.

Step 2, the margin for 95%:

1.96×1.04=2.041.96 \times 1.04 = 2.04

Here z=1.96z = 1.96 and the standard error is 1.041.04.

Step 3, the interval: €68 minus €2.04 is €65.96, and €68 plus €2.04 is €70.04.

So the honest sentence is: "The average online order in October was €68 (95% confidence interval €66 to €70)." With 2,500 orders the average is known quite precisely. That is good news, and it is only visible because Sam calculated it.

A prediction interval says where the next single value will land

Here is the most common mix up in this whole field. A confidence interval is about the average. It does not tell you what the next customer will spend. For that you need a prediction interval: a range that should contain a single new value with a stated probability.

A prediction interval is always much wider, because one customer varies far more than the average of 2,500. For data that follow the bell shaped normal distribution, the textbook version is:

xˉ±z⋅s1+1n\bar{x} \pm z \cdot s\sqrt{1 + \frac{1}{n}}

Here xˉ\bar{x}, ss, nn and zz mean the same as before. The 11 under the root stands for the spread of the new individual value, and 1n\frac{1}{n} for the remaining uncertainty about the average itself. With large nn the second part almost vanishes and the interval is roughly xˉ±z⋅s\bar{x} \pm z \cdot s. See Foundations: distributions for what "normal" means here.

Forecasters use the same idea. The standard forecasting textbook argues that a point forecast without its prediction interval is worth very little, because nobody can tell how accurate it is1.

Example: the next online order

Sam plugs the October figures into the formula:

1.96×52×1+12500=101.941.96 \times 52 \times \sqrt{1 + \frac{1}{2500}} = 101.94

Here z=1.96z = 1.96, s=52s = 52 and n=2500n = 2500. So the normal prediction interval runs from €68 minus €101.94 to €68 plus €101.94: from minus €34 to €170. A negative order value is impossible, which tells Sam something important. Order values are not bell shaped. A good few small orders (socks, gas canisters) sit on the left and a long tail of tents and down jackets stretches to the right. For skewed data like this, the honest prediction interval comes straight from the data: Sam sorts all 2,500 orders and reads off the 5th and the 95th percentile, the values below which 5% and 95% of orders fall. They are €7 and €235.

Now compare the two sentences:

  • Confidence interval: "The average order is between €66 and €70."
  • Prediction interval: "Nine orders in ten fall between €7 and €235."

Both are true. They answer different questions. If Aoife plans the warehouse budget, the first one matters. If she sets a free delivery threshold, the second one does. A chart that shows only "€68" answers neither.

Error bars can mean three different things

Error bars are the little whiskers above and below a point or a bar. They look like a standard, but they are not. The same whisker can show at least three different things, and the three differ hugely in width:

  • Standard deviation (SD): how much individual values vary. It describes the data, not the precision of the estimate.
  • Standard error (SE): how much the estimate would vary between samples. It is the SD divided by n\sqrt{n}, so it shrinks as data grows.
  • Confidence interval (CI): a range built for a stated confidence level, usually 95%. With ten or more observations, a 95% interval is roughly twice the standard error on each side.

Cumming, Fidler and Vaux put it bluntly in a guide for biologists: different error bars give quite different information, so the figure legend must say which ones it shows2. Wilke's book on data visualisation makes the same point and adds that the confidence level belongs in the label too3.

This is not pedantry. Belia, Fidler, Williams and Cumming invited authors of published articles in psychology, behavioural neuroscience and medicine to adjust a chart until two means with error bars looked just significantly different. From 473 respondents, they concluded that many leading researchers had severe misconceptions about how error bars relate to significance and did not adequately tell confidence intervals and standard error bars apart4. If trained researchers mix them up, a board will too.

Cumming and colleagues also give rules of thumb for independent groups with at least ten observations each. When 95% confidence intervals overlap by about half of one arm, the difference is roughly at the edge of significance (p about 0.05). When standard error bars leave a gap of about one arm between them, it is also roughly p of 0.052. So "the bars do not overlap, therefore it is real" is a rule that works for no type of bar.

Example: one test, three kinds of bars

Shannon Outfitters tested a new product page. Version A, the old page, had 12,000 visitors and 360 orders: 3.00%. Version B, the new page with bigger photos and a size guide, had 12,000 visitors and 402 orders: 3.35%. Sam calculates all three kinds of bars.

For A, the standard error of a rate pp with nn visitors is p(1−p)/n\sqrt{p(1-p)/n}:

0.03×0.9712000=0.00156\sqrt{\frac{0.03 \times 0.97}{12000}} = 0.00156

That is 0.16 percentage points. For B the same calculation gives 0.16 as well (0.164 before rounding). The 95% interval is 1.96 times that on each side. The standard deviation of a single visitor (who either orders or does not) is p(1−p)\sqrt{p(1-p)}, which is about 17 percentage points.

Bars showVersion A (3.00%)Version B (3.35%)What the picture seems to say
Standard deviation3.00 ± 17.06 points3.35 ± 17.99 points"No difference at all, it is all noise"
Standard error2.84% to 3.16%3.19% to 3.51%"No overlap, B is clearly better"
95% confidence interval2.69% to 3.31%3.03% to 3.67%"They overlap, it is unclear"

Same data, three pictures, three stories. The SD bars are so wide that the chart would look like two thin lines inside two huge whiskers. The SE bars just miss each other, by 0.03 points, and a hurried reader would call it a clear win. The 95% intervals overlap by 0.28 points, which is almost a whole arm, so by Cumming's rule the difference is not significant at the usual 5% level. A proper test agrees: p is about 0.12.

The lesson for the chart: if your audience compares estimates, use 95% confidence intervals, label them as such in the caption, and give nn. Better still, plot the difference itself with its interval, which comes later in this chapter.

Tap or click to open full screen and move around

Show the distribution, not only the average

An average squeezes thousands of values into one. Sometimes that is exactly what you need. Often it hides the thing that matters: whether customers cluster around the middle, split into two groups, or trail off into a long tail. Why averages mislead covers the statistics. Here is the chart side.

A bar chart of averages has a second, quieter problem. Newman and Scholl showed people bars that stood for averages and asked how likely particular values were to belong to the data. People judged values inside the bar as more likely than values the same distance above the top, as if the bar contained the data. They called it the within the bar bias, and it appeared with and without error bars and in several groups of participants5. A bar invites the eye to read it as a container, which an average is not.

Two fixes follow. Where the spread matters, show it: a histogram (bars of counts in value bands) or, for several groups at once, a box plot, which marks the median, the middle half of the data and the extremes. Box plots are not available in this site's chart set, so the example below uses grouped histograms. And where you show averages, a dot with an interval reads more honestly than a bar.

Example: shop orders and online orders

Aoife asks whether shop customers and online customers spend differently. Sam's first chart compares the two averages.

Bar chart of the average order value at Shannon Outfitters in October, invented numbers. The shop averages 70 euro and online 68 euro. The two bars look almost identical, which suggests the two groups of customers behave the same. The chart hides the shape of the orders behind each average. Tap or click to open full screen and move around

Two bars of nearly equal height. The obvious conclusion is "no difference". Then Sam draws the same orders as a distribution: the share of orders in each value band.

Grouped bar chart of the share of orders in seven value bands at Shannon Outfitters in October, invented numbers. Online, 34 percent of orders are under 25 euro and 13 percent are over 150 euro, so online orders split into many small top ups and a long tail of big purchases. In the shop, 30 percent of orders fall between 50 and 75 euro and few are very small or very large. The averages are almost the same, 68 and 70 euro, but the customers behave very differently. Tap or click to open full screen and move around

The averages are €68 and €70. The customers are not the same at all. Online, a third of all orders are under €25: someone topping up gas canisters or socks, often paying An Post delivery on a tiny basket. At the other end, 13% of online orders are over €150. The shop clusters in the middle. That suggests two decisions the first chart could never have prompted: a free delivery threshold that nudges the small online baskets up, and a closer look at what drives the large online orders.

Bands on lines show how sure you are over time

On a line chart the interval becomes a band: a shaded area around the line that shows the range of plausible values at each point. Wilke describes the confidence band around a trend line as the customary way to show that uncertainty, and notes that even a straight line fit gets a curved band, because both the slope and the level of the line are uncertain3.

For forecasts, the band is a prediction interval, and it normally widens the further ahead you look, because uncertainty adds up over the horizon1. A forecast drawn as a single line hides exactly that. The next chapter, communicating forecasts with fan charts and scenarios, covers this in depth, and the forecasting guide covers how to build the model. The short version: if a line shows something estimated, it gets a band, and the caption says what the band is.

Example: Sam's three month revenue outlook

Sam's forecast for online revenue in the next three months is €182,000, €150,000 and €171,000. On its own that is a confident looking line. With the 80% prediction intervals from the forecasting model it becomes:

MonthForecast80% prediction interval
January€182,000€164,000 to €201,000
February€150,000€128,000 to €174,000
March€171,000€143,000 to €202,000

The band widens from about €37,000 in January to about €59,000 in March. Drawn as a shaded band around the line, it tells Aoife at a glance that January is fairly firm and March is a rough guide. The caption says: "Shaded area: 80% prediction interval. In eight months out of ten, actual revenue should land inside it."

Small samples need wide intervals, and a picture that admits it

Small samples are where uncertainty is largest and where charts most often hide it. Two traps sit side by side.

The first is the formula. The simple interval for a rate, p±1.96p(1−p)/np \pm 1.96\sqrt{p(1-p)/n}, is called the Wald interval. Brown, Cai and DasGupta showed that its accuracy is erratic, that textbook advice on when it is safe is misleading, and recommended the Wilson interval (or the Jeffreys interval) for small samples6. The Wald interval can even produce impossible results: a negative lower end, or an interval of zero width when nothing happened.

For the case where nothing happened at all, there is a neat shortcut called the rule of three: if an event occurred zero times in nn tries, the upper end of an approximate 95% interval for its rate is about 3/n3/n7.

The second trap is the league table. Rank products, stores or campaigns by a rate, and the top and the bottom will be crowded with the smallest samples, because small samples swing furthest by chance. Spiegelhalter proposed the funnel plot for exactly this: plot each unit's rate against its sample size, and draw control limits that narrow as the sample grows. Points inside the funnel are consistent with chance. He notes that it avoids spurious ranking of units into league tables8.

Example: the best converting product page

Aoife's dashboard ranks product pages by conversion rate. The top of last quarter's list: a bivvy bag at 7.5%. The bottom: gaiters at 0%.

Bar chart ranking ten Shannon Outfitters product pages by conversion rate last quarter, invented numbers. The bivvy bag is top at 7.5 percent and gaiters are bottom at 0 percent. The chart does not show that the bivvy bag had only 40 visitors and the gaiters only 60, so both extremes are mostly chance. Tap or click to open full screen and move around

Sam looks at the sample sizes. The bivvy bag page had 40 visitors and 3 orders. The gaiters page had 60 visitors and no orders.

For the bivvy bag, the Wald interval runs from minus 0.66% to 15.66%. A negative conversion rate is nonsense, which is the formula's way of admitting it should not be used here. The Wilson interval runs from 2.58% to 19.86%. So the "best" page could easily be an ordinary one.

For the gaiters, the Wald interval is 0% to 0%, which claims perfect certainty from 60 visits. The rule of three gives an upper end of 3/60=5%3/60 = 5\%, and the Wilson interval runs from 0% to 6.02%. The gaiters might convert perfectly well.

Then Sam draws the funnel plot. Every product is a dot: visitors on the horizontal axis (log scale, because the samples range from 40 to 3,200), conversion rate on the vertical. The two curves are the approximate 95% limits around the shop wide rate of 3.1%.

Funnel plot of ten Shannon Outfitters product pages last quarter, invented numbers. Each product is a dot with its number of visitors on a logarithmic horizontal axis and its conversion rate on the vertical axis. Two curves mark approximate 95 percent limits around the shop wide rate of 3.1 percent; they are wide for small samples and narrow for large ones. The bivvy bag at 7.5 percent from 40 visitors and the gaiters at 0 percent from 60 visitors sit inside the funnel, so they are consistent with chance. Only the waterproof jacket, 3.9 percent from 3,200 visitors, sits above the upper limit, and the boots, 2.3 percent from 2,400 visitors, below the lower limit. Tap or click to open full screen and move around

The picture changes the conversation. The bivvy bag and the gaiters, the two pages the league table shouted about, sit comfortably inside the funnel. Two pages with plenty of traffic sit outside it: the waterproof jacket above the upper limit (3.9% from 3,200 visitors) and the boots below the lower limit (2.3% from 2,400 visitors). Those are the two pages worth Aoife's attention. The boots page is losing real sales, and the jacket page may hold a lesson for the others.

The limits here use the simple normal approximation, which is rough at the small end. Spiegelhalter's paper shows exact versions8. For a board chart the approximate funnel is good enough, as long as the caption says so.

Words like "likely" mean different things to different people

Sometimes you cannot show a chart at all, only a sentence. Then the uncertainty sits in words such as "likely", "possible" or "a real chance". Those words feel clear to the writer and are read very differently by each reader.

The classic account comes from Sherman Kent, then a member of the CIA's Board of National Estimates. In 1951 an intelligence estimate called a Soviet attack on Yugoslavia that year a "serious possibility". When a senior reader asked what odds that meant, Kent said about 65 to 35 in favour. The reader had taken it to mean much lower odds. Kent then asked his own board colleagues, who had all agreed the wording. The lowest had meant odds of about 20 to 80, the highest 80 to 209. One phrase, agreed by everyone in the room, and the people who signed it off meant anything from one in five to four in five.

Kent proposed tying words to numbers: "probable" for about 75%, give or take about 12%, "almost certain" for about 93%, give or take about 6%, and so on9. The Intergovernmental Panel on Climate Change (IPCC) uses a calibrated scale of this kind in its reports, shown here10.

TermProbability
Virtually certain99% to 100%
Very likely90% to 100%
Likely66% to 100%
About as likely as not33% to 66%
Unlikely0% to 33%
Very unlikely0% to 10%
Exceptionally unlikely0% to 1%

Note how wide "likely" is even in a calibrated system: anything from two in three upwards. Andrew and Michael Mauboussin surveyed how the general public reads such words and found the same wide gaps: one person's "pretty likely" is another's "far from certain"11. The practical rule follows: use a word if it helps the reader, but put the number next to it. And if you use a scale, print it in the report.

Example: Sam's sentence for the board

Sam's first draft for the board says: "The new product page is likely to increase sales."

Aoife reads "likely" as "almost certainly". The marketing lead reads it as "maybe". Neither knows how big the increase might be. Sam rewrites it:

"The new product page probably converts better: the test puts the chance at about 94%. The best estimate is 11.7% more orders, but the plausible range runs from about 3% fewer to about 26% more. I would switch to it, because it costs nothing to keep, but I would not put the 11.7% into the sales forecast."

Longer, fair enough. But every reader now takes away the same thing, and the decision is in the sentence. Where those numbers come from is the subject of the next section.

A/B test results: show the interval on the difference, not a winner badge

An A/B test sends some visitors to version A and some to version B, then compares the results. Many testing tools sum up the outcome as a badge: "Winner: B" with a percentage chance to beat the original. The badge answers a question nobody asked (which version is higher in this sample) and hides the one that matters (how big is the difference, and could it be zero or negative?).

The honest display is the difference between the two versions with its confidence interval. For two conversion rates:

(p^B−p^A)±zp^A(1−p^A)nA+p^B(1−p^B)nB(\hat{p}_B - \hat{p}_A) \pm z \sqrt{\frac{\hat{p}_A(1-\hat{p}_A)}{n_A} + \frac{\hat{p}_B(1-\hat{p}_B)}{n_B}}

Here p^A\hat{p}_A and p^B\hat{p}_B are the observed conversion rates of the two versions, nAn_A and nBn_B are their numbers of visitors, the square root is the standard error of the difference, and z=1.96z = 1.96 for a 95% interval. The Foundations: confidence intervals and Foundations: hypothesis tests sections explain the pieces.

If the interval excludes zero, the data favour a real difference at the 5% level. Its two ends tell you the smallest and the largest effect still consistent with the data, and that is what a business decision needs.

Example: Shannon Outfitters' product page test, shown three ways

The test from earlier ran for two weeks with 12,000 visitors on each version: 360 orders on A (3.00%), 402 on B (3.35%). Sam shows Aoife the result three ways.

Display one, the winner badge. The testing tool says: "Winner: B. Plus 11.7% conversion. 94% chance to beat original." Drawn as a chart, it is two bars with the higher one highlighted.

Bar chart of the Shannon Outfitters product page test, invented numbers. Version A converts at 3.00 percent and version B at 3.35 percent, with B highlighted as the winner. The chart shows no interval, so it hides that the difference is uncertain and could be zero or slightly negative. Tap or click to open full screen and move around

It looks decisive. It says nothing about how sure anyone should be.

Display two, bars with error bars. This is the table from the error bar section. With 95% intervals, A runs from 2.69% to 3.31% and B from 3.03% to 3.67%. They overlap. Better than the badge, but Aoife now has to judge the overlap by eye, and as Belia's study showed, even researchers get that wrong4.

Display three, the interval on the difference. Sam calculates it step by step. The difference is 3.35%−3.00%=0.353.35\% - 3.00\% = 0.35 percentage points. The variance part for A, with p^A=0.03\hat{p}_A = 0.03 and nA=12000n_A = 12000, is:

0.03×0.9712000=0.000002425\frac{0.03 \times 0.97}{12000} = 0.000002425

and for B, with p^B=0.0335\hat{p}_B = 0.0335 and nB=12000n_B = 12000, it is:

0.0335×0.966512000=0.000002698\frac{0.0335 \times 0.9665}{12000} = 0.000002698

Their sum is 0.0000051230.000005123, and its square root, the standard error of the difference, is 0.00226, or 0.226 percentage points. The margin is 1.96×0.226=0.441.96 \times 0.226 = 0.44 points. So the 95% interval runs from 0.35 minus 0.44 to 0.35 plus 0.44: from minus 0.09 to plus 0.79 percentage points. Relative to version A, that is from about 3% fewer orders to about 26% more.

Sam draws how that interval developed over the two weeks, with a dashed line at zero.

Line chart of the Shannon Outfitters product page test over 14 days, invented numbers. The line is the cumulative difference in conversion rate between version B and version A in percentage points, and the shaded band is its 95 percent confidence interval. On day 1 the difference is 1.05 points with an interval from minus 0.59 to 2.69. The band narrows every day. On day 14 the difference is 0.35 points with an interval from minus 0.09 to 0.79. The band still crosses the dashed zero line, so a difference of zero, or a small loss, cannot be ruled out. Tap or click to open full screen and move around

The picture shows three things the badge hid. The early days swung wildly, which is why nobody should stop a test because day three looked good. The band narrowed as visitors came in. And on day 14 the band still dips below zero. The data are most consistent with B being somewhat better, but they do not rule out no difference, or a small loss. A standard test agrees: p is about 0.12.

The "94% chance to beat original" on the badge is not false. It is roughly what you get from the same numbers under simple assumptions. But it describes only the direction, never the size, and it invites Aoife to book an 11.7% uplift that might be 0%.

Finding an uplift this small reliably would need far more traffic: by the usual sample size calculation, about 39,000 visitors per version for an 80% chance of detecting it (Foundations: sample size, and how much data you actually need for the full story). That is a useful thing for Aoife to know before the next test, not after.

What Aoife should decide

I start every analytics project from the decision it should support, and an A/B test is no exception. The question is not "did B win?" but "what does the shop do now, and what does it promise?" The flow below is the one Sam uses.

Tap or click to open full screen and move around

For Shannon Outfitters the interval includes zero. The new page is already built and costs nothing to keep, and nothing in the test suggests it does harm beyond a possible small dip. So Sam's recommendation: switch to B, keep the sales forecast unchanged, and watch the product page conversion over the next quarter. If Aoife ever wants to know the size of the effect, the next test needs about three times the traffic, planned before it starts.

Before you publish an uncertain number

A short check that fits on one screen. Run it before any estimate goes into a chart or a report.

  1. Is this number an estimate? If it comes from a sample, a test or a model, it has uncertainty. Say so.
  2. Which question does the reader have? About the average (confidence interval) or about the next single value (prediction interval)?
  3. What do the bars or the band show? SD, SE or a confidence interval, at which level, from how many observations. Put it in the caption.
  4. Would a distribution say more than the average? If the shape could change the decision, draw the histogram.
  5. Is any group small? Use Wilson intervals for small rates, the rule of three for zero events, and a funnel plot instead of a league table.
  6. Are there words like "likely"? Put the number next to them.
  7. Is it a comparison? Show the difference with its interval, not two separate numbers and a badge.

Every chart on my own website follows a small set of fixed rules: brand colours only, light and dark mode, labels at 45 degrees, a full screen view, top to bottom diagrams for phones, and a text description for screen readers. The descriptions on the charts above are written the same way: what the chart shows, its point, and the key numbers including the interval. A screen reader user deserves the uncertainty too.

Conclusion

Every number in a business report that comes from a sample, a test or a model is an estimate, and a chart that shows only the point quietly claims more certainty than anyone has. The fix is not complicated. Say whether you mean the average or the next single value, and draw the matching interval. Label every error bar. Show the distribution when the shape matters. Give small samples the wide intervals they deserve and plot them in a funnel, not a league table. Put numbers next to words like "likely". And show an A/B test as the difference with its interval, then decide from the ends of that interval, not from a badge.

Aoife did not get a less useful report from Sam. She got one she can act on: which numbers are firm, which are rough, and which decision follows from each.

If your reports show numbers without saying how sure they are, book the free workshop: we take one of your real reports or A/B tests and redraw it with honest intervals, and the first two weeks of work are on me.

Sources

  1. 1.

    Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice, 3rd edition, OTexts, 2021, section 5.5 Distributional forecasts and prediction intervals, https://otexts.com/fpp3/prediction-intervals.html

  2. 2.

    Geoff Cumming, Fiona Fidler and David L. Vaux, Error bars in experimental biology, The Journal of Cell Biology 177(1), 7 to 11, 2007, https://pmc.ncbi.nlm.nih.gov/articles/PMC2064100/

  3. 3.

    Claus O. Wilke, Fundamentals of Data Visualization, O'Reilly Media, 2019, chapter 16 Visualizing uncertainty, free online edition, https://clauswilke.com/dataviz/visualizing-uncertainty.html

  4. 4.

    Sarah Belia, Fiona Fidler, Jennifer Williams and Geoff Cumming, Researchers misunderstand confidence intervals and standard error bars, Psychological Methods 10(4), 389 to 396, 2005, https://pubmed.ncbi.nlm.nih.gov/16392994/

  5. 5.

    George E. Newman and Brian J. Scholl, Bar graphs depicting averages are perceptually misinterpreted: the within-the-bar bias, Psychonomic Bulletin and Review 19(4), 601 to 607, 2012, https://doi.org/10.3758/s13423-012-0247-5

  6. 6.

    Lawrence D. Brown, T. Tony Cai and Anirban DasGupta, Interval estimation for a binomial proportion, Statistical Science 16(2), 101 to 133, 2001, https://doi.org/10.1214/ss/1009213286

  7. 7.

    James A. Hanley and Abby Lippman-Hand, If nothing goes wrong, is everything all right? Interpreting zero numerators, JAMA 249(13), 1743 to 1745, 1983, https://pubmed.ncbi.nlm.nih.gov/6827763/

  8. 8.

    David J. Spiegelhalter, Funnel plots for comparing institutional performance, Statistics in Medicine 24(8), 1185 to 1202, 2005, https://pubmed.ncbi.nlm.nih.gov/15568194/

  9. 9.

    Sherman Kent, Words of estimative probability, Studies in Intelligence 8(4), Central Intelligence Agency, 1964, https://www.cia.gov/resources/csi/static/Words-of-Estimative-Probability.pdf

  10. 10.

    Michael D. Mastrandrea and others, Guidance note for lead authors of the IPCC Fifth Assessment Report on consistent treatment of uncertainties, Intergovernmental Panel on Climate Change, 2010, Table 1, https://www.ipcc.ch/site/assets/uploads/2017/08/AR5_Uncertainty_Guidance_Note.pdf

  11. 11.

    Andrew Mauboussin and Michael J. Mauboussin, If you say something is "likely," how likely do people think it is?, Harvard Business Review, 3 July 2018, https://hbr.org/2018/07/if-you-say-something-is-likely-how-likely-do-people-think-it-is