Start with the decision, not the dashboard

TL;DR

Difficulty: Intermediate

A dashboard that changes no decision is a screensaver, however clean the tracking. Work backwards instead. Name a real either or decision, pick the one pivotal number that would flip it, and write down the threshold before you look at any data. Then work out the expected value of perfect information: the most any study on that question could ever be worth. In the price example it is 12,000 pounds, so a 500 pound test is a bargain and a 40,000 pound build is not. Fetch the smallest data that puts you on one side of the threshold, act, and let the analysis retire.

A dashboard can be accurate, handsome and expensive and still be worth nothing, because nobody does anything differently after looking at it. In this article I turn the usual analytics project around: start from the decision, find the single number that would flip it, set the threshold before the data arrives, and use the value of information to cap what any study is worth, with a worked price example and an invented Leeds software firm.

A dashboard nobody acts on is a screensaver

Picture a dashboard that took an agency four months and a frightening sum to build. Forty seven charts across nine tabs. Live traffic, cohort retention, funnel drop off, a map with dots that pulse. It is a genuinely handsome piece of work. Now ask its owner one question. When did you last look at it and do something differently because of what it said?

If the honest answer is never, the dashboard has failed, and the failure has almost nothing to do with the maths. The tracking can be clean, the regressions correct, every number right. It still failed, because it produced a report that nobody acted on, which is a very expensive way of producing nothing at all.

So here is the first point, as bluntly as I can put it. A dashboard is not analytics. If no decision changes, you did not buy insight. You bought a screensaver.

I build data work for a living, so this is me arguing against a chunk of my own industry. A lot of what gets sold as analytics is collection dressed up as understanding. Someone wires up every event, pipes it into a warehouse, builds a dashboard on top and then hopes that insight will condense out of the air like dew. It rarely does. Insight is not a byproduct of collection. It is the answer to a question somebody asked on purpose.

Notice how the handsome dashboard tends to be used. Once a quarter someone opens it in a board meeting, scrolls to the chart that is going up, feels briefly good and closes the laptop. The charts going down get a nervous joke and no follow up. That is measurement as decoration, a very costly mood ring for the founder.

Start from the decision, not the data

The method I use runs backwards from how most analytics projects are scoped.

Most projects start with the data. What can we track? Let us track all of it. Then someone builds a dashboard, everyone looks at it and waits for wisdom. Start at the other end instead. Start with the decision you are trying to change.

Not a metric. A decision. A decision is a fork in the road where you could plausibly do two different things, and where the two roads lead somewhere different for the business. Should we raise the price? Should we drop this feature? Should we move budget from Google to LinkedIn? Should we chase larger accounts or double down on self serve? Those are decisions. Monthly active users is not a decision. It is a number that might, or might not, inform one.

Once you have the decision, ask the only question that matters next. What single number, if I knew it, would tip me from one road to the other? I call it the pivotal number. Not the interesting number, not the impressive number, the pivotal one: a value on this side means do X, a value on that side means do Y. If no value of the number would change your mind, the number is decoration, and you can stop paying to collect it today.

That is the second point. Every metric you track should have a decision attached to it, and a threshold where it flips that decision. A metric with no decision behind it is not a key performance indicator. It is a fact you are hoarding.

A quick test: metric or decision

The fastest way to feel the difference is to put a few side by side. On the left, things people proudly track. On the right, the decision each one could feed, if any.

Thing on the dashboardDecision it could feed
Monthly active usersDo we invest in the activation flow or leave it?
Page views on the blogDo we keep commissioning articles or stop?
Average session lengthUsually nothing. It is a comfort blanket.
Share of trials that reach the first useful resultDo we rebuild onboarding this quarter?
Gross margin by plan tierDo we retire the lowest tier?

Run your own dashboard through that exercise and be honest about the right hand column. If a row is empty, or carries the same vague answer as three other rows, you have found a chart you are paying to render for no reason.

The backwards method, step by step

There are four steps, and they go in this order on purpose.

Name the decision. Write it as a genuine either or. Not "improve retention", which is a wish, but "do we build the win back mail sequence this quarter, or the onboarding revamp instead?" A real fork with real alternatives.

Find the pivotal number and its threshold. For that example it might be the share of lapsed customers who came back within ninety days after a nudge in a past test. Above roughly 8%, the win back sequence probably pays for itself. Below 3%, the onboarding work almost certainly wins. Notice that the thresholds exist before the data does. That is the point. You know what would change your mind before you go looking.

Find the smallest data that tells you which side of the threshold you are on. Not all the data. The smallest. Often it is a single cohort, one query, a hundred rows or a two week holdout. Sometimes it is a number you already have and never framed as pivotal.

Run the smallest analysis that answers it, act, and move on. Not a model when a mean will do (Foundations: the mean). Not a dashboard when a one off query will do. Answer the question, make the decision and let the analysis retire with dignity. You are not building a monument. You are settling an argument.

There is a discipline hiding in that order. Steps one and two happen with no data in the room. You write the fork and the threshold on a whiteboard, from judgement, before anyone opens a query tool. People find this uncomfortable, because guessing a threshold feels unscientific. It is the opposite. Science has a name for it. Researchers who preregister write down their question and analysis plan before they see the outcomes, precisely because hindsight bias makes it hard to tell a prediction from an explanation invented after the fact.1 Committing to what would change your mind before you see the number is the same defence, scaled down to a business: it stops you moving the goalposts once the data lands somewhere you do not like.

Dashboard firstDecision first
Start from what we can collectStart from what we must decide
Track everything, just in caseTrack the pivotal number on purpose
Build the dashboard, then look for meaningSet the threshold, then find the number
Success is a shipped dashboardSuccess is a changed decision
Lives forever, watched by no oneAnswers the question, then retires
Nobody owns an actionEvery number owns a decision
Tap or click to open full screen and move around

Put a number on the decision itself

The maths is simpler than people expect, and it changes how you spend.

Take a worked example. You are deciding whether to raise your price by 20%. There are two states of the world you cannot see in advance. In the good state, call it HH, customers mostly accept it and you make more money. In the bad state, LL, enough of them leave that you make less. From what you know today you put the good state at 60% and the bad state at 40%.

The expected value of an action is each outcome weighted by how likely it is (Foundations: expected value):

EV(a)=∑ipi vi(a)EV(a) = \sum_{i} p_i \, v_i(a)

Here aa is the action you choose, ii runs over the possible states of the world, pip_i is the probability you give state ii (the probabilities add up to one, see Foundations: probability), and vi(a)v_i(a) is what action aa is worth to you if state ii turns out to be true.

Now some rough annual figures, invented for the example. Raise the price and land in the good state, and you make about £50,000 more a year. Raise it and land in the bad state, and you lose about £30,000. Keep the price, and nothing changes, so that action is worth zero by definition. Then:

EV(raise)=0.6×50 000+0.4×(−30 000)=18 000EV(\text{raise}) = 0.6 \times 50\,000 + 0.4 \times (-30\,000) = 18\,000

With pH=0.6p_H = 0.6 and pL=0.4p_L = 0.4, raising is worth about £18,000 a year on today's beliefs, and keeping the price is worth EV(keep)=0EV(\text{keep}) = 0. If you had to choose right now, you would raise. But look how much of that rests on a 40% chance of losing £30,000. That risk is exactly what a small test can buy down, and the next calculation tells you what such a test is worth.

Tap or click to open full screen and move around

You control the first fork. The world controls the second. Decision first analytics is the business of buying a peek at the world's choice before you commit, but only when the peek costs less than the mistake it prevents.

The value of information, in pounds

This is my favourite idea in the field, and it is used far less outside statistics departments than it deserves. Ronald Howard set it out in 1966 under the name information value theory.2 It answers the question every owner should ask before commissioning an analysis: what is it worth to know?

Imagine a perfect oracle that tells you the true state before you decide. When it says good state, 60% of the time, you raise and make £50,000. When it says bad state, 40% of the time, you keep the price and make zero, dodging the loss. In general, the expected value of perfect information is the gap between deciding with the oracle and deciding on your current best guess (Foundations: expected value):

EVPI=∑ipimax⁡avi(a)  −  max⁡a∑ipi vi(a)EVPI = \sum_{i} p_i \max_{a} v_i(a) \; - \; \max_{a} \sum_{i} p_i \, v_i(a)

The first term, ∑ipimax⁡avi(a)\sum_i p_i \max_a v_i(a), picks the best action separately in each state and then averages over the states, which is what you could earn with the oracle. The second term, max⁡aEV(a)\max_a EV(a), is the best you can do today, choosing once before you know the state. pip_i and vi(a)v_i(a) mean the same as above.

For the price decision the oracle's expected value is:

EVperfect=0.6×50 000+0.4×0=30 000EV_{\text{perfect}} = 0.6 \times 50\,000 + 0.4 \times 0 = 30\,000

and subtracting today's best, EV(raise)=18 000EV(\text{raise}) = 18\,000, gives:

EVPI=30 000−18 000=12 000EVPI = 30\,000 - 18\,000 = 12\,000

So EVPI=£12,000EVPI = £12{,}000. That is the ceiling: the most any study, test, survey or dashboard on this question could ever be worth to you, even if it were flawless, because it is the expected size of the mistake it could save you from. This is the third point. A £500 price test on a sample of customers, which gets you much of the way to that answer, is a bargain. A £40,000 analytics build to answer the same question is setting fire to money. The information is worth £12,000 at most, and now you know where the line is.

Keeping the price is worth 0 pounds, raising it on today's beliefs is worth 18,000 pounds, and deciding with perfect information would be worth 30,000 pounds. The gap of 12,000 pounds is the most any study on the question can be worth. Tap or click to open full screen and move around

That one calculation kills more bad analytics projects than any amount of taste. When someone pitches you a big data initiative, ask which decisions it feeds and what those decisions are worth. If they cannot answer, you have found the screensaver before paying for it.

What if the test is not perfect

Real tests are never oracles. A two week price experiment on a slice of your traffic gives a noisy reading, not the truth. So a real study is worth somewhere below the £12,000 ceiling, and how far below depends on how much it can actually shift your beliefs.

You do not need heavy machinery to feel this. If a test would land on the same side of your threshold whatever happens, it is worth nothing: run it and you learn nothing you would act on. If it could plausibly land on either side, and each side points to a different action, it is worth a lot, up towards the ceiling. Everything else sits in between. The rule of thumb I use is simple. A test is only worth running if you can imagine a result that would change your mind. If every result leads to the same action, cancel the test and take the action now.

Decision theorists call the worth of a real, imperfect study the expected value of sample information, an idea popularised by Howard Raiffa and Robert Schlaifer in the 1960s.3 It always sits in a sandwich (Foundations: probability):

0≤EVSI≤EVPI0 \le EVSI \le EVPI

EVSIEVSI is the expected value of the real study and EVPIEVPI is the perfect information ceiling from above. The lower bound holds because you can always ignore information you have bought and decide as you would have anyway,4 and the upper bound holds because no imperfect study can tell you more than the oracle.

You will rarely compute EVSIEVSI by hand, and you do not need to. The sandwich alone rules out two common ways of wasting money. Any quote above the EVPIEVPI is nonsense on its face. And a study that cannot carry you across your threshold is worth the zero on the left, however sophisticated it looks. Before you buy any analysis, work out the ceiling in ten minutes on the back of an envelope, then ask whether the proposed study could realistically move you across a threshold. Two questions, and they are worth more than another data platform.

But we might need it later

The commonest objection is that we should collect everything now because we might want it later. It sounds prudent. Mostly it is a way of avoiding the hard part, which is deciding what you care about.

I am not against keeping raw data. Storage is inexpensive, and keeping clean event data is sensible within what data protection allows and no further, because refilling history later is painful. Under UK data protection law you need a lawful basis and a purpose for personal data, and you may keep it no longer than that purpose needs, so "just in case" is not a purpose on its own.5 Keeping lawful data is fine. The mistake is spending analyst time, dashboard budget and, worst of all, management attention on data before a decision needs it.

Storing data and building analytics are two different activities with two different costs. Storing a clean stream of events is close to free and mostly a one off engineering job. Building, maintaining, explaining and staring at a dashboard costs money forever, because the running cost is human attention. Keep the first as wide as the law and good sense allow. Ration the second, because in a small company attention is the scarcest thing you have. Do not build the cathedral until someone needs to pray in it.

A worked example, from decision to done in an afternoon

Here is the whole method on one invented business, so you can see the steps moving. Ledgerline is a small bookkeeping software firm in Leeds (ledgerline.example). Revenue has flattened, the board wants answers, and an agency has quoted a five figure sum for the usual warehouse and dashboard package. Before anyone signs, the founder runs the backwards method in an afternoon.

The decision. Not "fix growth", which is a wish. The real fork: does the next quarter of engineering time go on a self serve onboarding rebuild, or on features for larger accounts? One team, one quarter, two roads.

The pivotal number. The onboarding rebuild pays off only if many signups stall before they reach value. The push for larger accounts pays off only if the money is concentrated in a few big customers who are asking for more. So the pivotal number is the share of new revenue coming from the top 10% of accounts. Above roughly 60%, the money is at the top and the larger accounts win. Below 40%, growth is a volume game and onboarding wins. Both thresholds go on the wall before a single figure is pulled.

The smallest data. No warehouse. One export from the billing system and about twenty minutes in a spreadsheet: rank accounts by revenue, take the top tenth and sum their share of new revenue over the last two quarters.

Run it and act. In this example the number comes back at 71%. Comfortably over the 60% threshold, nowhere near the fuzzy middle. The larger accounts win, the onboarding rebuild goes on the shelf and the quarter goes to the features those accounts have asked for.

StepWhat Ledgerline doesTime
Name the decisionOnboarding rebuild or features for larger accounts20 minutes
Find the pivotal numberTop tenth's share of new revenue15 minutes
Set the thresholdOver 60% larger accounts, under 40% onboarding10 minutes
Get the dataOne billing export20 minutes
Run and decide71%, larger accounts win20 minutes

An afternoon and one spreadsheet steer a full quarter of a team. The five figure build would have produced a handsome dashboard with that same 71% buried on tab four, weeks later, to reach the identical decision.

The ways companies talk themselves back into the dashboard

Chasing the interesting instead of the pivotal. There is always a chart that is fun to look at and irrelevant to any decision. Traffic by hour of the day is the classic. The pull towards the interesting is strong precisely because it feels like work without the risk of committing to a fork.

Moving the threshold after seeing the data. You said 8% would justify the win back sequence, the number came in at 6%, and suddenly you are explaining why 6% is really quite good. If you do this, the threshold was never real and the exercise was theatre. Write it down first, in front of someone, and hold yourself to it.

Buying precision you will not use. People commission a study accurate to a decimal place when the decision only needs to know which side of a wide line you are on. If the threshold is 60% and any sane estimate lands between 70% and 75%, you already have your answer. Paying to sharpen 71% into 70.8% is waste.

Confusing a data problem with a decision problem. Sometimes no number will settle it, because the fork is about strategy or appetite for risk, not fact. A dashboard cannot decide whether you want to run a business serving large accounts. Do not send data to do a leadership job.

Building the monument before the question. The deepest mistake, the one the others grow from, is starting collection before naming a single decision, out of the comfortable feeling that gathering data is the same as making progress.

How to tell if you are doing it wrong

If your dashboard has a tab nobody has opened in a month, that tab is decoration. If you cannot name the last decision a report changed, your reporting is theatre. If your analytics brief is a list of metrics rather than a list of questions, it is scoped backwards. And if a chart makes you say "interesting" and then you do nothing, it was not interesting. It was a screensaver with a trend line.

The fix is the same every time. Go back to the decision. What are you actually choosing between? What number would tip it? That is the report. Most of the rest you will not miss.

How to apply this on Monday

No new tools, no budget.

First, list your live decisions: the three to five real forks facing the business next quarter, each with two named alternatives. If you cannot fill three, that is a finding in itself.

Second, for each decision, name the pivotal number and its threshold from judgement, before you look at anything. Write it down where someone else can see it.

Third, run the ten minute ceiling check. Rough out what each road is worth in the good and the bad case, put a probability on each, and work out the EVPIEVPI. If the ceiling is small, trust your judgement and move on. If it is large, that decision has earned a proper look. The article on decision theory for founders walks through expected value with more examples.

Fourth, find the smallest data that puts you on one side of the threshold. One export, one query, one cohort. Resist every urge to widen it.

Fifth, act, then let the analysis go. If the same decision comes back next quarter, and only then, ask whether it has earned a standing report. Most will not.

If you catch yourself sayingDo this instead
Let us track everything just in caseName one decision and track its pivotal number
We need a dashboard for thisWe need an answer for this, once
The number is not precise enoughIs it on the right side of the threshold? Then it is precise enough
Interesting, look at that trendWhich decision would that trend change?
We might need it laterKeep the lawful raw data, build nothing yet

Where this leaves you

None of this means measurement is a waste. Measurement is the means and the decision is the end, and it is easy to get that backwards because collecting data feels like progress while deciding feels like risk. Good analytics is not the one with the most charts. It is the one that changed what you did on Monday.

So the next time someone offers to build you a dashboard, or you feel the urge to track one more thing, ask the decision first question. What decision am I trying to change, and what single number would change it? Start there, work back to the smallest thing that answers it, and every pound you spend is aimed at a fork in the road rather than sprayed at the scenery.

Pointing data at real decisions instead of prettier dashboards is much of my data science work, and I am happy to start by talking you out of the analysis you do not need. Your raw data stays yours, and the terms for code and models are in the analytics and data science terms. If you want to try it on one of your own decisions, book the free workshop, and the first two weeks of work are on me.

If your reporting has stopped changing what you actually do, bring me the decision you are stuck on. I will help you find the one number that would settle it and the smallest way to get it. I run a free workshop, and the first two weeks of work are on me.

Sources

  1. 1.

    Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven and David T. Mellor, The preregistration revolution, Proceedings of the National Academy of Sciences 115 (11), 2600 to 2606, 2018, https://pmc.ncbi.nlm.nih.gov/articles/PMC5856500/

  2. 2.

    Ronald A. Howard, Information Value Theory, IEEE Transactions on Systems Science and Cybernetics 2 (1), 22 to 26, 1966, https://doi.org/10.1109/TSSC.1966.300074

  3. 3.

    Expected value of sample information, Wikipedia, citing Robert Schlaifer and Howard Raiffa, Applied Statistical Decision Theory, https://en.wikipedia.org/wiki/Expected_value_of_sample_information

  4. 4.

    Expected value of perfect information, Wikipedia (the value of information can never be less than zero, because the decision maker can always ignore it), https://en.wikipedia.org/wiki/Expected_value_of_perfect_information

  5. 5.

    UK GDPR, Article 5 (1) (b) purpose limitation and (1) (e) storage limitation, legislation.gov.uk, https://www.legislation.gov.uk/eur/2016/679/article/5