Start with the decision, not the dashboard
TL;DR
Difficulty: Intermediate
A dashboard that changes no decision is a screensaver, however clean the tracking. Work backwards instead. Name a real either or decision, pick the one pivotal number that would flip it, and write down the threshold before you look at any data. Then work out the expected value of perfect information: the most any study on that question could ever be worth. In the price example it is 12,000 pounds, so a 500 pound test is a bargain and a 40,000 pound build is not. Fetch the smallest data that puts you on one side of the threshold, act, and let the analysis retire.
A dashboard can be accurate, handsome and expensive and still be worth nothing, because nobody does anything differently after looking at it. In this article I turn the usual analytics project around: start from the decision, find the single number that would flip it, set the threshold before the data arrives, and use the value of information to cap what any study is worth, with a worked price example and an invented Leeds software firm.
A dashboard nobody acts on is a screensaver
Picture a dashboard that took an agency four months and a frightening sum to build. Forty seven charts across nine tabs. Live traffic, cohort retention, funnel drop off, a map with dots that pulse. It is a genuinely handsome piece of work. Now ask its owner one question. When did you last look at it and do something differently because of what it said?
If the honest answer is never, the dashboard has failed, and the failure has almost nothing to do with the maths. The tracking can be clean, the regressions correct, every number right. It still failed, because it produced a report that nobody acted on, which is a very expensive way of producing nothing at all.
So here is the first point, as bluntly as I can put it. A dashboard is not analytics. If no decision changes, you did not buy insight. You bought a screensaver.
I build data work for a living, so this is me arguing against a chunk of my own industry. A lot of what gets sold as analytics is collection dressed up as understanding. Someone wires up every event, pipes it into a warehouse, builds a dashboard on top and then hopes that insight will condense out of the air like dew. It rarely does. Insight is not a byproduct of collection. It is the answer to a question somebody asked on purpose.
Notice how the handsome dashboard tends to be used. Once a quarter someone opens it in a board meeting, scrolls to the chart that is going up, feels briefly good and closes the laptop. The charts going down get a nervous joke and no follow up. That is measurement as decoration, a very costly mood ring for the founder.
Start from the decision, not the data
The method I use runs backwards from how most analytics projects are scoped.
Most projects start with the data. What can we track? Let us track all of it. Then someone builds a dashboard, everyone looks at it and waits for wisdom. Start at the other end instead. Start with the decision you are trying to change.
Not a metric. A decision. A decision is a fork in the road where you could plausibly do two different things, and where the two roads lead somewhere different for the business. Should we raise the price? Should we drop this feature? Should we move budget from Google to LinkedIn? Should we chase larger accounts or double down on self serve? Those are decisions. Monthly active users is not a decision. It is a number that might, or might not, inform one.
Once you have the decision, ask the only question that matters next. What single number, if I knew it, would tip me from one road to the other? I call it the pivotal number. Not the interesting number, not the impressive number, the pivotal one: a value on this side means do X, a value on that side means do Y. If no value of the number would change your mind, the number is decoration, and you can stop paying to collect it today.
That is the second point. Every metric you track should have a decision attached to it, and a threshold where it flips that decision. A metric with no decision behind it is not a key performance indicator. It is a fact you are hoarding.
A quick test: metric or decision
The fastest way to feel the difference is to put a few side by side. On the left, things people proudly track. On the right, the decision each one could feed, if any.
| Thing on the dashboard | Decision it could feed |
|---|---|
| Monthly active users | Do we invest in the activation flow or leave it? |
| Page views on the blog | Do we keep commissioning articles or stop? |
| Average session length | Usually nothing. It is a comfort blanket. |
| Share of trials that reach the first useful result | Do we rebuild onboarding this quarter? |
| Gross margin by plan tier | Do we retire the lowest tier? |
Run your own dashboard through that exercise and be honest about the right hand column. If a row is empty, or carries the same vague answer as three other rows, you have found a chart you are paying to render for no reason.
The backwards method, step by step
There are four steps, and they go in this order on purpose.
Name the decision. Write it as a genuine either or. Not "improve retention", which is a wish, but "do we build the win back mail sequence this quarter, or the onboarding revamp instead?" A real fork with real alternatives.
Find the pivotal number and its threshold. For that example it might be the share of lapsed customers who came back within ninety days after a nudge in a past test. Above roughly 8%, the win back sequence probably pays for itself. Below 3%, the onboarding work almost certainly wins. Notice that the thresholds exist before the data does. That is the point. You know what would change your mind before you go looking.
Find the smallest data that tells you which side of the threshold you are on. Not all the data. The smallest. Often it is a single cohort, one query, a hundred rows or a two week holdout. Sometimes it is a number you already have and never framed as pivotal.
Run the smallest analysis that answers it, act, and move on. Not a model when a mean will do (Foundations: the mean). Not a dashboard when a one off query will do. Answer the question, make the decision and let the analysis retire with dignity. You are not building a monument. You are settling an argument.
There is a discipline hiding in that order. Steps one and two happen with no data in the room. You write the fork and the threshold on a whiteboard, from judgement, before anyone opens a query tool. People find this uncomfortable, because guessing a threshold feels unscientific. It is the opposite. Science has a name for it. Researchers who preregister write down their question and analysis plan before they see the outcomes, precisely because hindsight bias makes it hard to tell a prediction from an explanation invented after the fact.1 Committing to what would change your mind before you see the number is the same defence, scaled down to a business: it stops you moving the goalposts once the data lands somewhere you do not like.
| Dashboard first | Decision first |
|---|---|
| Start from what we can collect | Start from what we must decide |
| Track everything, just in case | Track the pivotal number on purpose |
| Build the dashboard, then look for meaning | Set the threshold, then find the number |
| Success is a shipped dashboard | Success is a changed decision |
| Lives forever, watched by no one | Answers the question, then retires |
| Nobody owns an action | Every number owns a decision |
Put a number on the decision itself
The maths is simpler than people expect, and it changes how you spend.
Take a worked example. You are deciding whether to raise your price by 20%. There are two states of the world you cannot see in advance. In the good state, call it , customers mostly accept it and you make more money. In the bad state, , enough of them leave that you make less. From what you know today you put the good state at 60% and the bad state at 40%.
The expected value of an action is each outcome weighted by how likely it is (Foundations: expected value):
Here is the action you choose, runs over the possible states of the world, is the probability you give state (the probabilities add up to one, see Foundations: probability), and is what action is worth to you if state turns out to be true.
Now some rough annual figures, invented for the example. Raise the price and land in the good state, and you make about £50,000 more a year. Raise it and land in the bad state, and you lose about £30,000. Keep the price, and nothing changes, so that action is worth zero by definition. Then:
With and , raising is worth about £18,000 a year on today's beliefs, and keeping the price is worth . If you had to choose right now, you would raise. But look how much of that rests on a 40% chance of losing £30,000. That risk is exactly what a small test can buy down, and the next calculation tells you what such a test is worth.
You control the first fork. The world controls the second. Decision first analytics is the business of buying a peek at the world's choice before you commit, but only when the peek costs less than the mistake it prevents.
The value of information, in pounds
This is my favourite idea in the field, and it is used far less outside statistics departments than it deserves. Ronald Howard set it out in 1966 under the name information value theory.2 It answers the question every owner should ask before commissioning an analysis: what is it worth to know?
Imagine a perfect oracle that tells you the true state before you decide. When it says good state, 60% of the time, you raise and make £50,000. When it says bad state, 40% of the time, you keep the price and make zero, dodging the loss. In general, the expected value of perfect information is the gap between deciding with the oracle and deciding on your current best guess (Foundations: expected value):
The first term, , picks the best action separately in each state and then averages over the states, which is what you could earn with the oracle. The second term, , is the best you can do today, choosing once before you know the state. and mean the same as above.
For the price decision the oracle's expected value is:
and subtracting today's best, , gives:
So . That is the ceiling: the most any study, test, survey or dashboard on this question could ever be worth to you, even if it were flawless, because it is the expected size of the mistake it could save you from. This is the third point. A £500 price test on a sample of customers, which gets you much of the way to that answer, is a bargain. A £40,000 analytics build to answer the same question is setting fire to money. The information is worth £12,000 at most, and now you know where the line is.
That one calculation kills more bad analytics projects than any amount of taste. When someone pitches you a big data initiative, ask which decisions it feeds and what those decisions are worth. If they cannot answer, you have found the screensaver before paying for it.
What if the test is not perfect
Real tests are never oracles. A two week price experiment on a slice of your traffic gives a noisy reading, not the truth. So a real study is worth somewhere below the £12,000 ceiling, and how far below depends on how much it can actually shift your beliefs.
You do not need heavy machinery to feel this. If a test would land on the same side of your threshold whatever happens, it is worth nothing: run it and you learn nothing you would act on. If it could plausibly land on either side, and each side points to a different action, it is worth a lot, up towards the ceiling. Everything else sits in between. The rule of thumb I use is simple. A test is only worth running if you can imagine a result that would change your mind. If every result leads to the same action, cancel the test and take the action now.
Decision theorists call the worth of a real, imperfect study the expected value of sample information, an idea popularised by Howard Raiffa and Robert Schlaifer in the 1960s.3 It always sits in a sandwich (Foundations: probability):
is the expected value of the real study and is the perfect information ceiling from above. The lower bound holds because you can always ignore information you have bought and decide as you would have anyway,4 and the upper bound holds because no imperfect study can tell you more than the oracle.
You will rarely compute by hand, and you do not need to. The sandwich alone rules out two common ways of wasting money. Any quote above the is nonsense on its face. And a study that cannot carry you across your threshold is worth the zero on the left, however sophisticated it looks. Before you buy any analysis, work out the ceiling in ten minutes on the back of an envelope, then ask whether the proposed study could realistically move you across a threshold. Two questions, and they are worth more than another data platform.
But we might need it later
The commonest objection is that we should collect everything now because we might want it later. It sounds prudent. Mostly it is a way of avoiding the hard part, which is deciding what you care about.
I am not against keeping raw data. Storage is inexpensive, and keeping clean event data is sensible within what data protection allows and no further, because refilling history later is painful. Under UK data protection law you need a lawful basis and a purpose for personal data, and you may keep it no longer than that purpose needs, so "just in case" is not a purpose on its own.5 Keeping lawful data is fine. The mistake is spending analyst time, dashboard budget and, worst of all, management attention on data before a decision needs it.
Storing data and building analytics are two different activities with two different costs. Storing a clean stream of events is close to free and mostly a one off engineering job. Building, maintaining, explaining and staring at a dashboard costs money forever, because the running cost is human attention. Keep the first as wide as the law and good sense allow. Ration the second, because in a small company attention is the scarcest thing you have. Do not build the cathedral until someone needs to pray in it.
A worked example, from decision to done in an afternoon
Here is the whole method on one invented business, so you can see the steps moving. Ledgerline is a small bookkeeping software firm in Leeds (ledgerline.example). Revenue has flattened, the board wants answers, and an agency has quoted a five figure sum for the usual warehouse and dashboard package. Before anyone signs, the founder runs the backwards method in an afternoon.
The decision. Not "fix growth", which is a wish. The real fork: does the next quarter of engineering time go on a self serve onboarding rebuild, or on features for larger accounts? One team, one quarter, two roads.
The pivotal number. The onboarding rebuild pays off only if many signups stall before they reach value. The push for larger accounts pays off only if the money is concentrated in a few big customers who are asking for more. So the pivotal number is the share of new revenue coming from the top 10% of accounts. Above roughly 60%, the money is at the top and the larger accounts win. Below 40%, growth is a volume game and onboarding wins. Both thresholds go on the wall before a single figure is pulled.
The smallest data. No warehouse. One export from the billing system and about twenty minutes in a spreadsheet: rank accounts by revenue, take the top tenth and sum their share of new revenue over the last two quarters.
Run it and act. In this example the number comes back at 71%. Comfortably over the 60% threshold, nowhere near the fuzzy middle. The larger accounts win, the onboarding rebuild goes on the shelf and the quarter goes to the features those accounts have asked for.
| Step | What Ledgerline does | Time |
|---|---|---|
| Name the decision | Onboarding rebuild or features for larger accounts | 20 minutes |
| Find the pivotal number | Top tenth's share of new revenue | 15 minutes |
| Set the threshold | Over 60% larger accounts, under 40% onboarding | 10 minutes |
| Get the data | One billing export | 20 minutes |
| Run and decide | 71%, larger accounts win | 20 minutes |
An afternoon and one spreadsheet steer a full quarter of a team. The five figure build would have produced a handsome dashboard with that same 71% buried on tab four, weeks later, to reach the identical decision.
The ways companies talk themselves back into the dashboard
Chasing the interesting instead of the pivotal. There is always a chart that is fun to look at and irrelevant to any decision. Traffic by hour of the day is the classic. The pull towards the interesting is strong precisely because it feels like work without the risk of committing to a fork.
Moving the threshold after seeing the data. You said 8% would justify the win back sequence, the number came in at 6%, and suddenly you are explaining why 6% is really quite good. If you do this, the threshold was never real and the exercise was theatre. Write it down first, in front of someone, and hold yourself to it.
Buying precision you will not use. People commission a study accurate to a decimal place when the decision only needs to know which side of a wide line you are on. If the threshold is 60% and any sane estimate lands between 70% and 75%, you already have your answer. Paying to sharpen 71% into 70.8% is waste.
Confusing a data problem with a decision problem. Sometimes no number will settle it, because the fork is about strategy or appetite for risk, not fact. A dashboard cannot decide whether you want to run a business serving large accounts. Do not send data to do a leadership job.
Building the monument before the question. The deepest mistake, the one the others grow from, is starting collection before naming a single decision, out of the comfortable feeling that gathering data is the same as making progress.
How to tell if you are doing it wrong
If your dashboard has a tab nobody has opened in a month, that tab is decoration. If you cannot name the last decision a report changed, your reporting is theatre. If your analytics brief is a list of metrics rather than a list of questions, it is scoped backwards. And if a chart makes you say "interesting" and then you do nothing, it was not interesting. It was a screensaver with a trend line.
The fix is the same every time. Go back to the decision. What are you actually choosing between? What number would tip it? That is the report. Most of the rest you will not miss.
How to apply this on Monday
No new tools, no budget.
First, list your live decisions: the three to five real forks facing the business next quarter, each with two named alternatives. If you cannot fill three, that is a finding in itself.
Second, for each decision, name the pivotal number and its threshold from judgement, before you look at anything. Write it down where someone else can see it.
Third, run the ten minute ceiling check. Rough out what each road is worth in the good and the bad case, put a probability on each, and work out the . If the ceiling is small, trust your judgement and move on. If it is large, that decision has earned a proper look. The article on decision theory for founders walks through expected value with more examples.
Fourth, find the smallest data that puts you on one side of the threshold. One export, one query, one cohort. Resist every urge to widen it.
Fifth, act, then let the analysis go. If the same decision comes back next quarter, and only then, ask whether it has earned a standing report. Most will not.
| If you catch yourself saying | Do this instead |
|---|---|
| Let us track everything just in case | Name one decision and track its pivotal number |
| We need a dashboard for this | We need an answer for this, once |
| The number is not precise enough | Is it on the right side of the threshold? Then it is precise enough |
| Interesting, look at that trend | Which decision would that trend change? |
| We might need it later | Keep the lawful raw data, build nothing yet |
Where this leaves you
None of this means measurement is a waste. Measurement is the means and the decision is the end, and it is easy to get that backwards because collecting data feels like progress while deciding feels like risk. Good analytics is not the one with the most charts. It is the one that changed what you did on Monday.
So the next time someone offers to build you a dashboard, or you feel the urge to track one more thing, ask the decision first question. What decision am I trying to change, and what single number would change it? Start there, work back to the smallest thing that answers it, and every pound you spend is aimed at a fork in the road rather than sprayed at the scenery.
Pointing data at real decisions instead of prettier dashboards is much of my data science work, and I am happy to start by talking you out of the analysis you do not need. Your raw data stays yours, and the terms for code and models are in the analytics and data science terms. If you want to try it on one of your own decisions, book the free workshop, and the first two weeks of work are on me.