Start With the Decision, Not the Dashboard

The screensaver problem

A client showed me a dashboard last year that took an agency four months and a frightening amount of money to build. Forty seven charts across nine tabs. Live traffic, cohort retention, funnel drop off, a lovely map with dots that pulsed. It was, genuinely, a beautiful piece of work. I asked the owner one question. When was the last time you looked at it and did something differently because of what it said?

Long pause. Then, honestly, never.

That is the whole problem with most analytics, and it has almost nothing to do with the maths. The regressions were fine. The tracking was clean. The numbers were correct. It failed because it produced a report that nobody acted on, which is a very expensive way to produce nothing at all. Here is the first thing I want you to take away, and I will say it as bluntly as I can. A dashboard is not analytics. If no decision changes, you did not buy insight. You bought a screensaver.

I build data work for a living, so this is me arguing against a chunk of my own industry. Most of what gets sold as analytics is collection dressed up as understanding. Somebody wires up every event, pipes it into a warehouse, builds a dashboard on top, and then hopes that insight will somehow condense out of the air like dew. It almost never does. Insight is not a byproduct of collection. It is the answer to a question you asked on purpose.

And notice how the beautiful dashboard actually gets used in practice. Once a quarter someone opens it in a board meeting, scrolls to the chart that is going up, feels briefly good, and closes the laptop. Nothing moves. The charts that are going down get a nervous joke and no follow up. That is not measurement doing its job. That is measurement as decoration, a very costly mood ring for the founder. If that stings a little, good. It stung me the first time I noticed I had built one myself.

Start from the decision, not the data

So here is the method I actually use, and it runs backwards from how most analytics projects are scoped.

Most projects start with the data. What can we track? Let us track all of it. Then we build a dashboard, then we look at the dashboard, then we wait for wisdom. I want you to start at the other end entirely. Start with the decision you are trying to change.

Not a metric. A decision. A decision is a fork in the road where you could plausibly do two different things, and where the two roads lead somewhere different for the business. Should we raise the price. Should we kill this feature. Should we move the budget from Google to LinkedIn. Should we chase the enterprise segment or double down on self serve. Those are decisions. Monthly active users is not a decision. It is a number that might, or might not, inform one.

Once you have the decision, ask the only question that matters next. What single number, if I knew it, would tip me from one road to the other? I call it the pivotal number. Not the interesting number, not the impressive number, the pivotal one. The one where a value on this side means do X and a value on that side means do Y. If no value of the number would change your mind, the number is decoration, and you can stop paying to collect it today.

That is the second thing I want to land. Every metric you track should have a decision attached to it, and a threshold where it flips that decision. A metric with no decision behind it is not a key performance indicator. It is a fact you are hoarding.

A quick test: metric or decision

The fastest way to feel the difference is to put a few of them side by side. In the left column, things people proudly track. In the right, the decision each one would actually feed, if any.

Thing on the dashboard	Decision it could feed
Monthly active users	Do we invest in the activation flow or leave it?
Page views on the blog	Do we keep commissioning articles or stop?
Average session length	Nothing, usually. It is a vanity comfort blanket.
Share of trials that reach the aha moment	Do we rebuild onboarding this quarter?
Gross margin by plan tier	Do we retire the lowest tier?

Run your own dashboard through that exercise and be honest about the right hand column. If a row has nothing in it, or the same vague answer as three other rows, you have found a chart you are paying to render for no reason. I have never done this with a client and not found at least a third of the charts fall straight into the bin.

The backwards method, step by step

Here is the sequence I walk clients through. It has four steps and they go in this order on purpose.

One. Name the decision. Write it as a genuine either or. Not improve retention, which is a wish, but should we build the win back email flow this quarter or the onboarding revamp instead. Real fork, real alternatives.

Two. Find the pivotal number and its threshold. For that example it might be the share of churned customers who came back within ninety days after a nudge in a past test. If it is above roughly eight percent the win back flow probably pays for itself, if it is below three percent the onboarding work almost certainly wins. Notice I have thresholds before I have data. That is the point. I know what would change my mind before I go looking.

Three. Work out the smallest data that would tell you which side of the threshold you are on. Not all the data. The smallest. Often it is a single cohort, one query, a hundred rows, a two week holdout. Sometimes it is a number you already have and never framed as pivotal.

Four. Run the smallest analysis that answers it, act, and move on. Not a model when a mean will do. Not a dashboard when a one off query will do. Answer the question, make the decision, and let the analysis die with dignity. You are not building a monument. You are settling an argument.

There is a discipline hiding in the order. Steps one and two happen with no data in the room at all. You write the fork and the threshold on a whiteboard, from judgement, before anyone opens a query tool. People find this uncomfortable, because guessing the threshold feels unscientific. It is the opposite. Committing to what would change your mind before you see the number is the single best defence against fooling yourself, because it stops you moving the goalposts once the data lands somewhere you do not like.

Compare that to the common approach and the contrast is almost comic.

Dashboard first	Decision first
Start from what we can collect	Start from what we must decide
Track everything, just in case	Track the pivotal number on purpose
Build the dashboard, then look for meaning	Set the threshold, then find the number
Success is a shipped dashboard	Success is a changed decision
Lives forever, watched by no one	Answers the question, then retires
Nobody is accountable for an action	Every number owns a decision

The dashboard first column is where the money goes. The decision first column is where the money comes back.

Here is the shape of it as a picture.

Put a number on the decision itself

Let me show you the maths, because it is simpler than people expect and it changes how you spend.

Say you are deciding whether to raise your price by twenty percent. There are two states of the world you cannot see in advance. In the good state, call it H, customers mostly tolerate it and you make more money. In the bad state, L, enough of them walk that you make less. From what you know today you put the good state at sixty percent and the bad state at forty percent.

The expected value of any action is just each outcome weighted by how likely it is:

$EV = \sum_i p_i \cdot v_i$

Put in rough annual figures. If you raise the price and land in the good state you make about fifty thousand more a year. If you raise it and land in the bad state you lose about thirty thousand. If you keep the price, nothing changes, so that action is worth zero by definition. Then:

$EV_{\text{raise}} = 0.6 \times 50\,000 + 0.4 \times (-30\,000) = 18\,000$

So on todays beliefs, raising the price is worth about eighteen thousand a year and keeping it is worth zero. If someone forced you to choose right now, you raise. But look how much of that expected value is riding on a forty percent chance of losing thirty grand. That risk is exactly what a small test can buy down, and now I can tell you precisely how much that test is worth.

It helps to see the whole decision as a little tree, with the choice on the left and the two states of the world branching off each action.

Read it left to right. You pick a branch at the fork, then the world picks a branch at the diamond, and you only control the first choice. The whole game of decision first analytics is buying a peek at which branch the world is going to take before you commit, but only when that peek costs less than the mistake it prevents. Which brings us to the number that governs everything.

The value of information, in actual pounds

This is my favourite idea in the whole field and almost nobody outside a stats department uses it. It answers the question every owner should ask before commissioning any analysis. What is it worth to know?

Imagine you had a perfect oracle that told you the true state before you decided. If it said good state, which happens sixty percent of the time, you would raise and make fifty thousand. If it said bad state, forty percent of the time, you would keep the price and make zero, dodging the loss. So the expected value if you could always decide with perfect knowledge is:

$EV_{\text{perfect}} = 0.6 \times 50\,000 + 0.4 \times 0 = 30\,000$

The value of perfect information is simply the difference between deciding with the oracle and deciding on your current best guess:

$EVPI = EV_{\text{perfect}} - EV_{\text{best}} = 30\,000 - 18\,000 = 12\,000$

Twelve thousand pounds. That is the ceiling. That is the absolute most any study, test, survey or dashboard on this question could ever be worth to you, even if it were flawless, because that is the size of the mistake it could save you from. And here is the third thing I want to hammer home. A five hundred pound price test on a sample of customers, which will get you most of the way to that answer, is a screaming bargain. A forty thousand pound analytics build to answer the same question is setting fire to money. The information is worth twelve grand. You never pay more than the answer is worth, and you now have the number that tells you where the line is.

That single calculation kills more bad analytics projects than any amount of taste. When someone pitches you a big data initiative, ask them which decisions it feeds and what those decisions are worth. If they cannot answer, you have found your screensaver before you paid for it.

What if the test is not perfect

Now the honest wrinkle. Real tests are never oracles. A two week price experiment on a slice of your traffic gives you a noisy read, not the truth. So the true worth of a real study sits somewhere below that twelve thousand pound ceiling, and how far below depends on how much the test actually shifts your beliefs.

You do not need heavy machinery to feel this. Think of it in three bands. If a test would almost certainly land on the same side of your threshold no matter what, it is worth nothing, run it and you learn nothing you would act on. If a test could plausibly land on either side, and each side points you at a different action, it is worth a lot, right up towards that ceiling. Everything else is in between. The rule of thumb I actually say out loud to clients is this. A test is only worth running if you can imagine a result that would change your mind. If every result leads to the same action, cancel the test and take the action now.

There is a tidy way to say the same thing. The expected value of a real, imperfect study, often written EVSI, always obeys:

$0 \le EVSI \le EVPI$

You will not usually compute the left hand side by hand, and you do not need to. What matters is the sandwich. Zero on one end, the perfect information ceiling on the other, and your real study somewhere inside. That single inequality quietly rules out two of the most common ways people waste money. You cannot be worth more than the perfect answer, so any quote above the EVPI is nonsense on its face. And a study that cannot cross your threshold is worth the zero on the left, no matter how sophisticated it looks.

The practical upshot is almost rude in its simplicity. Before you buy any analysis, work out the EVPI ceiling in ten minutes on the back of an envelope, then ask whether the proposed study could realistically move you across a threshold. Two questions. They will save you more money than a data team.

But we might need it later

The commonest objection I hear is that we should collect everything now because we might want it later. It sounds prudent. It is mostly a way of avoiding the hard part, which is deciding what you actually care about.

I am not against keeping raw data. Storage is genuinely inexpensive and you should keep clean event data if you can, within what data protection allows and no further, because refilling history is painful. Keeping data is fine. The mistake is spending analyst time, dashboard budget and, worst of all, management attention on data before a decision needs it. Collecting data is easy. Attention is the scarce resource, and a wall of charts nobody asked a question of is a tax on the scarcest thing you have. Keep the raw material by all means. Do not build the cathedral until someone needs to pray in it.

There is a real distinction underneath this that is worth making explicit, because it dissolves most of the argument. Storing data and building analytics are two completely different activities with two completely different costs. Storing a clean stream of events is close to free and mostly a one off engineering job. Building, maintaining, explaining and staring at a dashboard is expensive forever, because the running cost is human attention. Keep the first as wide as data protection sensibly allows. Ration the second like it is the last bottle of water in the desert, because in a small company it very nearly is.

A worked mini case, from decision to done in a week

Let me put the whole method on one story, because the steps are clearer when they are moving. Numbers here are made up to keep a client anonymous, but the shape is one I have lived through more than once.

A founder of a small business tool came to me convinced she needed a full analytics rebuild. Revenue had flattened, the board wanted answers, and an agency had quoted a five figure sum for the usual warehouse and dashboard package. Before signing anything, we did the backwards method over a single afternoon.

Step one, the decision. Not fix growth, which is a wish. The real fork was this. Do we spend the next quarter of engineering time on a self serve onboarding rebuild, or on an enterprise features push for larger accounts? One team, one quarter, two roads.

Step two, the pivotal number. We reasoned it through. The onboarding rebuild only pays off if a lot of signups are stalling before they ever reach value. The enterprise push only pays off if the money is genuinely concentrated in a few big accounts that are asking for more. So the pivotal number was the share of new revenue coming from the top ten percent of accounts. Above roughly sixty percent, the money is at the top and enterprise wins. Below forty percent, growth is a volume game and onboarding wins. We wrote those thresholds on the wall before pulling a single figure.

Step three, the smallest data. This did not need a warehouse. It needed one export from the billing system and about twenty minutes in a spreadsheet. We ranked accounts by revenue, took the top decile, and summed their share of new revenue over the last two quarters.

Step four, run it and act. The number came back at seventy one percent. Comfortably over the sixty percent threshold, not even close to the fuzzy middle. The money was unmistakably at the top. Enterprise won, the onboarding rebuild went on the shelf, and the quarter went to features the biggest accounts had been asking for.

Here is the part that matters for this article. The whole thing took an afternoon and one spreadsheet, and it steered a full quarter of a team. The five figure analytics rebuild would have produced a beautiful dashboard with that same seventy one percent buried on tab four, discovered weeks later, at forty times the cost, to reach the identical decision. We did not need the cathedral. We needed one number and the nerve to write the threshold down first.

Step	What we did	Time
Name the decision	Onboarding rebuild or enterprise push	20 minutes
Find the pivotal number	Top decile share of new revenue	15 minutes
Set the threshold	Over 60 percent enterprise, under 40 onboarding	10 minutes
Get the data	One billing export	20 minutes
Run and decide	71 percent, enterprise wins	20 minutes

Common mistakes I see again and again

After enough of these I can almost predict the ways a company will talk itself back into the dashboard. Here are the ones that come up most.

Chasing the interesting instead of the pivotal. There is always a chart that is fun to look at and irrelevant to any decision. Traffic by hour of day is the classic. Fascinating, almost never actionable. The pull towards interesting is strong precisely because it feels like work without the risk of committing to a fork.

Moving the threshold after seeing the data. You said eight percent would justify the win back flow, the number came in at six, and suddenly you are explaining why six is really quite good actually. If you do this, the threshold was never real and the whole exercise was theatre. Write it down first, in front of someone, and hold yourself to it.

Buying precision you will not use. People commission a study accurate to a decimal place when the decision only needs to know which side of a wide line you are on. If your threshold is sixty percent and any sane estimate lands between seventy and seventy five, you already have your answer. Spending more to sharpen seventy one to seventy point eight is pure waste.

Confusing a data problem with a decision problem. Sometimes the honest answer is that no number will settle it, because the fork is about strategy or appetite for risk, not fact. Dashboards cannot decide whether you want to run an enterprise business. Do not send data to do a leadership job.

Building the monument before the question. The deepest mistake, the one all the others grow from, is starting the collection before naming a single decision. Every screensaver dashboard I have ever seen was born this way, from good intentions and the comfortable feeling that gathering data is the same as making progress.

How to tell if you are doing it wrong

A few tells, from years of walking into this. If your dashboard has a tab nobody has opened in a month, that tab is decoration. If you cannot name the last decision a report changed, your reporting is theatre. If your analytics brief is a list of metrics rather than a list of questions, it is scoped backwards and it will disappoint you. And if a chart makes you say interesting and then you do nothing, it was not interesting. It was a screensaver with a trend line.

The fix in every case is the same. Go back to the decision. What are you actually trying to choose between? What number would tip it? That is the report. Everything else is optional, and most of it you will not miss.

How to apply this on Monday

Enough theory. Here is what to actually do this week, with no new tools and no budget.

First, list your live decisions. Sit down and write the three to five real forks facing the business in the next quarter. Genuine either ors, each with two named alternatives. If you cannot fill three, that is a finding in itself, and a good one.

Second, for each decision, name the pivotal number and guess its threshold from judgement, before you look at anything. Say out loud where the line is that flips you from one road to the other. Write it down where someone else can see it.

Third, run the ten minute EVPI check. Rough out what each road is worth in the good and bad case, put a probability on each, and work out the ceiling on what an answer could be worth. If the ceiling is small, trust your gut and move on. If it is large, that decision has earned a proper look.

Fourth, find the smallest data that puts you on one side of the threshold. One export, one query, one cohort. Resist every urge to broaden it. You are not describing the business, you are settling one argument.

Fifth, act, then let the analysis go. Make the call, and do not enshrine the working in a permanent dashboard. If the same decision comes back next quarter, and only then, consider whether it earns a standing report. Most will not.

Do that for one real decision and you will feel the difference immediately. It is faster, less expensive, and oddly calmer than staring at a wall of charts hoping for a revelation.

If you catch yourself saying	Do this instead
Let us track everything just in case	Name one decision, track its pivotal number
We need a dashboard for this	We need an answer for this, once
The number is not precise enough	Is it on the right side of the threshold? Then it is precise enough
Interesting, look at that trend	What decision would that trend change?
We might need it later	Store the raw data, build nothing yet

Where this leaves you

None of this means measurement is a waste. It means measurement is a means, and the decision is the end, and we have spent a decade getting that backwards because collecting data feels like progress and making decisions feels like risk. Good analytics is not the one with the most charts. It is the one that changed what you did on Monday.

So next time someone offers to build you a dashboard, or you feel the urge to track just one more thing, stop and ask the decision first question. What decision am I trying to change, and what single number would change it? Start there, work back to the smallest thing that answers it, and you will spend a tenth of the budget and get ten times the value, because every pound of it is aimed at a fork in the road rather than sprayed at the scenery.

If you want a hand pointing your data at real decisions instead of prettier dashboards, that is a lot of what my data science work is. I will happily start by talking you out of the analysis you do not need.