16 ways data projects fail before they start
TL;DR
Difficulty: Beginner
Most data projects fail at the framing stage, not in the model. Before funding one, name the decision it will change and check you may lawfully use the data for that purpose. Then ask sixteen questions: would we hold this input at prediction time, what happens without the strongest feature, what does a do nothing baseline score, are the numbers out of sample, how often was the test set used, what is missing from the sample, what does a false alarm cost against a miss, who owns the data feed and who acts on the output. The comfort of the answers tells you more than the answers.
Most data projects that fail do not fail because someone picked the wrong model. They fail months earlier, in the meeting where nobody wrote down which decision the thing was meant to change. This is a pre flight checklist for the person holding a vendor proposal: the legal gate first, then sixteen blunders grouped by where they hide, each with the question that catches it and what it costs, a worked example on accuracy and expected cost, and a decision tree for whether to fund, fix or stop.
Why most data projects die at the framing stage
Most data projects that fail do not fail because someone picked the wrong model. They fail months earlier, in the meeting where nobody wrote down which decision the thing was meant to change. By the time a model is training, the flaw is already baked into the question, the data or the promise, and no amount of clever maths later will lift it back out.
This is not only my opinion. In 2024 RAND interviewed experienced data scientists and engineers about why AI projects fail, and the first of the five root causes they reported was that stakeholders misunderstand or miscommunicate what problem the project is meant to solve. The others were missing or poor data, too little infrastructure, chasing the latest technology instead of a real problem, and pointing the technique at problems it cannot solve.1 You will see dramatic failure percentages quoted for data and AI projects. RAND itself only repeats an outside estimate of "more than 80%", and most of the round numbers that circulate have no study behind them that I could find, so I do not lean on any of them here. The pattern matters more than the percentage.
Here is the uncomfortable part. The failure is usually invisible on the day it happens. Nobody storms out of the framing meeting. Everyone nods, the budget gets signed, and the project rolls forward looking healthy for weeks. The bill arrives in month four, when the model that scored beautifully in the lab does nothing useful in the wild, the money is spent and the trust is gone. That gap between when the mistake is made and when it surfaces is why a checklist beats intuition. You cannot feel these failures coming. You have to ask for them.
So this is a pre flight checklist for the person who has a vendor proposal or a pitch deck in front of them and wants to know, before signing, whether the project is quietly doomed. You do not need to code to use it. You need to ask sixteen awkward questions and watch how comfortably they get answered. A good data science partner will have answers ready. A slick one will change the subject, because several of the classic blunders are exactly what a polished pitch is built to hide.
The zeroth question, before all the others
Before the sixteen there is a question people skip: what decision will this change, and what will we do differently depending on the answer? If nobody in the room can name the decision, you do not have a data project, you have a data hobby, and hobbies do not belong on an invoice.
Ask the vendor to finish the sentence "once this is live, the person who owns this process will do X instead of Y". If they finish it in one breath, good. If they reach for words like insight, visibility or empowerment, you are being sold a feeling, not a decision. Somebody choosing to approve, decline, call, reorder, discount or hold, that is an action, and only actions pay back a data budget.
The question before the zeroth: may you use this data at all
There is one more gate, and it is a legal one. Under the UK GDPR, personal data collected for one purpose may not be used for a new purpose that is incompatible with the original one (purpose limitation, Article 5(1)(b)).2 Customer records gathered to deliver orders are not automatically fair game for a model that decides who gets credit. Since 5 February 2026 the rules on automated decisions sit in the new Articles 22A to 22D of the UK GDPR, inserted by the Data (Use and Access) Act 2025: where a significant decision about a person is taken with no meaningful human involvement, you must tell them, let them make representations, offer human intervention and let them contest it.3 The regulator is now the Information Commission (formerly the ICO).
If you sell into the EU, the EU AI Act also lists the use of AI to assess the creditworthiness of individuals or establish their credit score as high risk (Annex III, point 5(b)).4 After the Digital Omnibus on AI, published in July 2026, the obligations for those Annex III systems apply from 2 December 2027 instead of 2 August 2026.5 Ask the vendor which of these applies and who signed it off. If the answer is "the data is already there, so we can use it", stop and get advice first.
How to read the sixteen
Each blunder below comes in three parts: the blunder in plain terms, a picture of what it looks like (the businesses are invented, the mechanisms are real), and the question that catches it, with a note on what it tends to cost if it slips through. Read the cost lines carefully. Owners weigh a data project by what it might earn and underweight what a quietly broken one drains in staff time, lost trust and decisions made worse by a confident wrong answer.
Where the question itself is wrong
1. There is no decision behind the project
The work starts from "we have loads of data, what can we learn from it" rather than from a choice someone has to make. It produces dashboards nobody opens.
Picture a wholesaler in Leeds that commissions a customer segmentation. It is elegant: seven clusters, lovely colours. Ask which cluster changes what the sales team does on Monday and the room goes quiet, because nobody wired a single cluster to a single action. The whole thing is wallpaper.
How to catch it: ask what action changes on Monday morning depending on the result. If the answer is a shrug, stop here. What it costs: the build, plus the worse hidden cost of a team that now believes data projects are decoration and quietly resists the next one, the one that would have worked.
Where the data lies to you
2. Using data from the future to predict the future
The training set contains information that only exists after the moment you are meant to be predicting. The model looks brilliant in the lab and is useless in the wild.
Picture a Manchester delivery firm predicting whether a parcel will arrive late, with the number of failed delivery attempts among the inputs. You only know that count after the delivery window closes, so at prediction time it does not exist yet. In training it makes the model look almost perfect.
How to catch it: for every input, ask "would we really have this value at the instant we make the prediction". If not, it is a time machine, not a feature. What it costs: you find out after go live, when the lab numbers collapse to noise, having already paid for a build and a deployment.
3. Treating leakage as skill
A close cousin: a column that is really the answer in disguise sneaks into the inputs, so the model scores far too well on something it should find hard. This is not a beginner's mistake. Kapoor and Narayanan at Princeton found leakage errors in 329 published papers across 17 scientific fields that used machine learning.6 If peer reviewed researchers miss it, a sales deck certainly can.
Picture a credit risk model that includes a collections flag, a field that only gets set once a customer has already defaulted. Or a churn model scoring in the high nineties out of sample, with a field like "days since account closed" among its inputs. Only churned customers have a closed account, so that field is the answer written on the back of the card.
How to catch it: be suspicious of any single feature that carries the whole model. Ask them to remove the strongest input and show you the score again. A leaked model collapses. A real one barely flinches. What it costs: a huge win in the pilot, a rollout, and then ordinary decisions made with false confidence, which is worse than no model because now people trust it.
Where a rare event fools the score
4. Ignoring the base rate
When the thing you care about is rare, accuracy is a liar. If fraud is ten orders in a thousand, a model that flags nothing at all is 99% accurate and catches none of it. The base rate is the share of cases that are positive before any model looks at them (see base rates in the Foundations).
Picture a payments team shown a fraud model at "98% accurate" and delighted, until someone asks how many actual fraud cases it catches. Almost none, because it has learned that saying no is right most of the time.
How to catch it: ask what fraction of cases are positive, then ask what a do nothing baseline would score. If they cannot tell you, they have not thought about it. What it costs: you pay for accuracy and get a model that always says no.
5. Testing on the training data
The model is graded on the very examples it studied, so of course it does well. It has memorised, not learned.
Picture a demand forecast reported at 96% that then misses the next quarter by a mile, because it was scored on the same year it was fitted on. That measures memory, not skill.
How to catch it: listen for the words "out of sample". You want to hear that the reported numbers come from data the model never saw while training. What it costs: every number in the proposal is inflated, and you plan headcount and stock against a promise that was never tested.
6. Peeking at the test set
Subtler than the last one. The team tries fifty ideas, checks each against the test set, keeps the winner, and has fitted the test set by hand. Dwork and others showed in Science in 2015 that reusing a holdout set to guide the analysis in this adaptive way can make the results overfit the holdout itself, so its score stops being an honest estimate.7
How to catch it: ask how many times the final test set was looked at, and whether a separate untouched holdout is kept for the very end. What it costs: a reported score optimistic by an amount nobody can quantify, and that is the one number you were handed to size the whole investment.
Where the sample is not what you think
7. Survivorship and sampling bias
The data only contains the survivors. You study the customers who stayed and draw lessons that never applied to the ones who left.
Picture a retention study built entirely on current customers, meant to explain what keeps people. The people who left, the exact group you need to understand, are not in the table, so every conclusion is about the wrong crowd.
How to catch it: ask what is missing from the sample, not just what is in it. What it costs: you optimise hard for the people who were never going to leave and learn nothing about the ones who do.
8. Confusing correlation with causation
Two things move together, so the deck implies that pulling one lever moves the other. Ice cream sales and drownings, forever (see correlation in the Foundations).
Picture a report showing that customers who use the mobile app spend more, with the implication that pushing everyone onto the app raises spend. It may simply be that already loyal customers are the ones who install the app.
How to catch it: ask whether anything was actually intervened on, or whether it is all observed. If nobody ran a real test, read every "drives" and "causes" as "happens alongside". What it costs: a marketing budget spent forcing a behaviour that was a symptom of value, not its cause.
9. Extrapolating past the edge of the data
The model only ever saw orders between £10 and £500 and is now asked about a £50,000 tender. It answers confidently and wrongly.
How to catch it: ask what range each input covered in training, and what the model does when a live case falls outside that range. What it costs: a confident wrong answer on your biggest, rarest and most important cases, precisely the ones a person should have looked at.
Where the objective is quietly wrong
10. Optimising a metric that is not the goal
The team maximises accuracy or clicks because they are easy to measure, while the business cares about margin, lifetime value or not annoying its best customers.
Picture a recommendation engine tuned for clicks that learns to surface lurid low value items. Clicks go up and to the right, and margin quietly falls.
How to catch it: ask them to write the metric down in one line, then ask whether a model that scored perfectly on it could still lose you money. If yes, it is the wrong metric. What it costs: exactly what you measured, which turns out not to be what you wanted, usually at scale.
11. No cost of being wrong
Every error is treated as equal when it plainly is not. Missing a fraudulent order and blocking a loyal customer are very different costs, and a model tuned as if they were the same is optimising for a world you do not live in.
How to catch it: ask what a false alarm costs you and what a miss costs you. You want two different numbers, in pounds. What it costs: expensive mistakes made as freely as trivial ones. The maths for this one is further down.
Where the data is dirtier than anyone admits
12. Assuming the data is clean
Nobody looked at the raw data. Dates in three formats, prices in two currencies with no label, a country field typed by hand with a dozen spellings of the same place. The model learns the mess faithfully.
Picture a sales model that treats blank price fields as a price of zero, so a batch of missing values teaches it that a whole product line is free. The chart looks fine until someone traces a mad prediction back to its source.
How to catch it: ask to see the profiling, meaning missing rates, duplicates, ranges and the odd values. If there is no profiling step in the plan, that is the first thing to add. What it costs: garbage learned faithfully, plus weeks spent discovering a mess that a day of profiling would have surfaced. My article on how much data you actually need covers the volume side of the same question.
13. Overfitting with too many knobs
Hundreds of features, not many rows, and a model flexible enough to memorise noise. It fits the past beautifully and predicts the future badly.
Picture a churn model with four hundred features and a few hundred customers. It has effectively learned each customer by heart, scores beautifully on history and falls apart on anyone new.
How to catch it: ask for the ratio of examples to features, and whether the model was kept deliberately simple. Complexity is a warning sign here, not a virtue. What it costs: a model that looks strongest on the slide and is weakest in production.
Where the world moves and nobody owns it
14. Concept drift ignored
The world the model learned in has moved on. Prices changed, a competitor launched, a season turned, and last year's patterns stopped holding without anyone noticing. Sculley and others at Google describe this in their paper on hidden technical debt in machine learning systems: changes in the external world and fragile data dependencies are among the reasons real systems carry large ongoing maintenance costs.8
How to catch it: ask how often the model is retrained, and how you will be told when it has gone stale, rather than finding out from a bad quarter. What it costs: a slow silent decay you only notice months later.
15. No owner for the data or the pipeline
The data lives in a spreadsheet on someone's laptop, or in a system a departed employee set up in a hurry. When the feed breaks, and it will, nobody is responsible.
Picture a nightly export a former analyst rigged up on their own machine. It runs fine until the laptop is wiped for the next starter, and the model spends two weeks scoring empty data before anyone connects the dots.
How to catch it: ask who owns each data source, who gets the call at seven in the morning when it stops, and whether that is written down. What it costs: a fortnight of decisions made on stale or empty data. My article on who does what among data engineers, scientists and analysts explains which role owns which part.
16. No plan to act on the answer
The last mile is missing. The prediction lands in a report nobody wired into a workflow, so nothing changes, and the project counts as a failure even though the model was fine.
Picture a good risk score that arrives every Monday as a PDF attachment, which the team meant to act on never opens because it is not part of any tool they already use.
How to catch it: ask where the output goes and who, or what, acts on it automatically. If the answer is "we will have a look at it", the value is already leaking away. What it costs: the entire value of the project.
The quick reference card
Print this, keep it next to the proposal and tick your way down it. Three columns, because the cost is the part owners forget.
| Blunder | How to catch it | What it costs |
|---|---|---|
| No decision behind it | What changes on Monday because of the result | A build nobody uses |
| Data from the future | Would we hold this value at prediction time | Lab numbers that collapse at go live |
| Leakage as skill | Remove the strongest input, show the score again | Confident decisions people wrongly trust |
| Ignoring the base rate | What does a do nothing baseline score | A model that always says no |
| Testing on training data | Are these numbers out of sample | Inflated numbers you plan against |
| Peeking at the test set | How often was the final set looked at | A score optimistic by an unknown amount |
| Survivorship bias | What is missing from the sample | Lessons about the wrong people |
| Correlation as causation | Did anyone actually intervene | Budget spent forcing a symptom |
| Extrapolating past the edge | What range did each input cover | Wrong answers on your biggest cases |
| Wrong metric | Could a perfect score still lose money | What you measured, not what you wanted |
| No cost of being wrong | Cost of a false alarm versus a miss, in pounds | Expensive mistakes made freely |
| Dirty data assumed clean | Show me the profiling | Garbage learned faithfully |
| Overfitting | Ratio of rows to features | Strongest on the slide, weakest live |
| Concept drift | How often is it retrained | Silent decay noticed too late |
| No data owner | Who gets the seven o'clock call | Weeks of scoring on empty data |
| No plan to act | Who or what acts on the output | The whole value left on the table |
Accuracy is the number a slick pitch hides behind
Numbers four and five are where most owners get talked into a bad project, so here is one small worked figure. Suppose ten orders in every thousand are fraudulent. A model that does nothing at all and waves every order through scores this on plain accuracy (see percentages in the Foundations):
where is the number of true positives (frauds flagged), the number of true negatives (good orders waved through) and the number of orders. The do nothing model flags nothing, so , and the accuracy is .
That is 99%, and it has caught precisely zero fraud. If a pitch leads with a single big accuracy number on a rare event, that number tells you almost nothing and may hide a model that has learned to always say no. Ask instead how many of the actual fraud cases it catches, and how many good orders it wrongly blocks to get there. Those two numbers are the real story, and they are the ones a weak proposal leaves off the slide.
The least expensive model on paper can be the most expensive in pounds
Numbers ten and eleven hide together, and this is the maths that exposes them. A model makes two kinds of mistake. It raises a false alarm, flagging something that was fine, and it misses, letting through something it should have caught. These almost never cost the same.
| Outcome | Model says yes | Model says no |
|---|---|---|
| Actually yes | caught it (good) | a miss |
| Actually no | a false alarm | left alone (good) |
Only the false alarm and the miss hurt, and they carry different price tags. Blocking a loyal customer might cost a relationship worth a great deal. Missing one fraudulent order might cost only that order. So the number that matters is the expected cost of the errors, an expected value in money:
where is the cost of one false alarm, the number of false alarms, the cost of one miss and the number of misses over the same period.
A worked example with invented numbers: a Bristol shop takes 10,000 orders a month, 100 of them fraudulent. Model A blocks 300 good orders and misses 20 frauds. Model B blocks only 50 good orders but misses 60 frauds. Say a blocked good customer costs £40 in lost margin and goodwill, and a missed fraud costs £60. Model A costs pounds a month. Model B costs pounds. Model B misses three times as much fraud and is still the better business decision. Ask any vendor to fill in that line. If they have never priced a false alarm against a miss, they have optimised a number that has nothing to do with your profit and loss.
A worked mini case: five of the sixteen in one project
In real life the blunders rarely arrive one at a time. Here is an invented but typical case. A subscription box retailer in Brighton, boxclub.example, has a model that predicts which subscribers will cancel next month, scoring in the low nineties, and a plan to send the flagged subscribers a retention discount.
The zeroth question passes: there is a real decision, who gets a discount. Then number three bites. The strongest feature is "days since last login", but people who have already decided to cancel stop logging in first, so the feature is half reading the answer. Remove it and the score drops into the seventies. Number four bites next. Only about three subscribers in a hundred cancel in a given month, so a model that predicts nobody cancels is already 97% accurate, and the proud headline means almost nothing. Number eleven was skipped. Nobody priced the discount, so a false alarm hands money to someone who was never going to leave, while a miss loses a subscriber. With pounds on both, the sensible threshold is far more cautious than the one they picked. And number sixteen waits at the end: the discount emails were to be pulled by hand from a spreadsheet once a month by someone already busy.
The model is not the problem. The framing is, in five places, and every one is catchable in an afternoon with the questions above. The fix is to rebuild the target so it only uses information available before the cancellation, price the two mistakes, set the threshold against expected cost rather than accuracy and wire the output straight into the email tool. The honest score is lower and the project makes money.
The pre commission decision tree
Before a single pound is committed, walk the project through four gates.
Living with the model: the loop nobody budgets for
Getting a model live is not the finish line, it is the start of a loop, and numbers fourteen and fifteen live here. Somebody has to watch the live scores, decide when the model has gone stale, retrain it on fresh data and put the new version back.
Budget for this loop up front and name the person who watches it. A proposal that treats go live as the end of the work is promising a model that is best on its first day and worse every day after.
The common mistakes owners make
Everything so far points at the vendor. It is only fair to turn the same honesty on the buyer.
The first is falling for the single big number. A round, confident figure feels like proof. It is not, and the rarer the event the less it means, so treat any lone headline score as an opening question.
The second is confusing confidence with competence. A vendor who never once says "I am not sure" is not more skilled, just less honest. People who know this work will tell you where a model is weak.
The third is buying the model and not the pipeline. The clever model is a small part of the work. The rest is data plumbing, profiling, the monitoring loop and the wiring that turns a prediction into an action, which is exactly the maintenance burden the Google paper describes.8 If the proposal spends all its words on the model and none on the plumbing, you are buying the small part and hoping the big part appears for free.
The fourth is skipping the profiling because it is not exciting. Nobody puts data cleaning in the pitch, yet it is where most of the risk lives.
The fifth is treating go live as the finish line, when it is the moment the work of keeping the thing honest begins.
And the last is not naming an owner on your own side. If nobody in your business is accountable for the decision the model feeds, it will drift back into being wallpaper however good the maths.
How to use this against a proposal you are handed
You do not need to argue the maths. Sit with the deck, run the sixteen questions and pay less attention to the answers than to the comfort. A partner who has built real things will meet number three with a shrug and a demo, tell you the base rate before you ask, and name the person who owns the data feed. A partner selling you a story will get defensive around leakage, lead with one shiny accuracy figure and go quiet when you ask what changes on Monday.
Two practical points if you hire me for this. Your raw data stays yours, and the terms for code and models are in the analytics and data science terms. And the first conversation is a free workshop, plus the first two weeks of work, so you can see how I handle the sixteen before you commit a budget. Book the workshop here.
The failure modes in this list are not exotic. They are ordinary and avoidable, and every one of them is far less expensive to catch in a meeting than in month four of a build that was never going to work.