16 ways data projects fail before they start

TL;DR

Difficulty: Beginner

Most data projects fail at the framing stage, not in the model. Before funding one, name the decision it will change and check you may lawfully use the data for that purpose. Then ask sixteen questions: would we hold this input at prediction time, what happens without the strongest feature, what does a do nothing baseline score, are the numbers out of sample, how often was the test set used, what is missing from the sample, what does a false alarm cost against a miss, who owns the data feed and who acts on the output. The comfort of the answers tells you more than the answers.

Most data projects that fail do not fail because someone picked the wrong model. They fail months earlier, in the meeting where nobody wrote down which decision the thing was meant to change. This is a pre flight checklist for the person holding a vendor proposal: the legal gate first, then sixteen blunders grouped by where they hide, each with the question that catches it and what it costs, a worked example on accuracy and expected cost, and a decision tree for whether to fund, fix or stop.

Why most data projects die at the framing stage

Most data projects that fail do not fail because someone picked the wrong model. They fail months earlier, in the meeting where nobody wrote down which decision the thing was meant to change. By the time a model is training, the flaw is already baked into the question, the data or the promise, and no amount of clever maths later will lift it back out.

This is not only my opinion. In 2024 RAND interviewed experienced data scientists and engineers about why AI projects fail, and the first of the five root causes they reported was that stakeholders misunderstand or miscommunicate what problem the project is meant to solve. The others were missing or poor data, too little infrastructure, chasing the latest technology instead of a real problem, and pointing the technique at problems it cannot solve.1 You will see dramatic failure percentages quoted for data and AI projects. RAND itself only repeats an outside estimate of "more than 80%", and most of the round numbers that circulate have no study behind them that I could find, so I do not lean on any of them here. The pattern matters more than the percentage.

Here is the uncomfortable part. The failure is usually invisible on the day it happens. Nobody storms out of the framing meeting. Everyone nods, the budget gets signed, and the project rolls forward looking healthy for weeks. The bill arrives in month four, when the model that scored beautifully in the lab does nothing useful in the wild, the money is spent and the trust is gone. That gap between when the mistake is made and when it surfaces is why a checklist beats intuition. You cannot feel these failures coming. You have to ask for them.

So this is a pre flight checklist for the person who has a vendor proposal or a pitch deck in front of them and wants to know, before signing, whether the project is quietly doomed. You do not need to code to use it. You need to ask sixteen awkward questions and watch how comfortably they get answered. A good data science partner will have answers ready. A slick one will change the subject, because several of the classic blunders are exactly what a polished pitch is built to hide.

The zeroth question, before all the others

Before the sixteen there is a question people skip: what decision will this change, and what will we do differently depending on the answer? If nobody in the room can name the decision, you do not have a data project, you have a data hobby, and hobbies do not belong on an invoice.

Ask the vendor to finish the sentence "once this is live, the person who owns this process will do X instead of Y". If they finish it in one breath, good. If they reach for words like insight, visibility or empowerment, you are being sold a feeling, not a decision. Somebody choosing to approve, decline, call, reorder, discount or hold, that is an action, and only actions pay back a data budget.

The question before the zeroth: may you use this data at all

There is one more gate, and it is a legal one. Under the GDPR, personal data collected for one purpose may not be used for a new purpose that is incompatible with the original one (purpose limitation, Article 5(1)(b)).2 Customer records gathered to deliver orders are not automatically fair game for a model that decides who gets credit. Article 22 adds that a person has the right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects on them, unless an exception applies and safeguards such as human intervention are in place.3 The Court of Justice of the EU held in the SCHUFA case in December 2023 that even calculating a credit score can be such a decision when a lender draws strongly on it.4 In Ireland the Data Protection Commission enforces all of this.

The EU AI Act also lists the use of AI to assess the creditworthiness of individuals or establish their credit score as high risk (Annex III, point 5(b)).5 After the Digital Omnibus on AI, published in July 2026, the obligations for those Annex III systems apply from 2 December 2027 instead of 2 August 2026.6 Ask the vendor which of these applies and who signed it off. If the answer is "the data is already there, so we can use it", stop and get advice first.

How to read the sixteen

Each blunder below comes in three parts: the blunder in plain terms, a picture of what it looks like (the businesses are invented, the mechanisms are real), and the question that catches it, with a note on what it tends to cost if it slips through. Read the cost lines carefully. Owners weigh a data project by what it might earn and underweight what a quietly broken one drains in staff time, lost trust and decisions made worse by a confident wrong answer.

Where the question itself is wrong

1. There is no decision behind the project

The work starts from "we have loads of data, what can we learn from it" rather than from a choice someone has to make. It produces dashboards nobody opens.

Picture a wholesaler in Cork that commissions a customer segmentation. It is elegant: seven clusters, lovely colours. Ask which cluster changes what the sales team does on Monday and the room goes quiet, because nobody wired a single cluster to a single action. The whole thing is wallpaper.

How to catch it: ask what action changes on Monday morning depending on the result. If the answer is a shrug, stop here. What it costs: the build, plus the worse hidden cost of a team that now believes data projects are decoration and quietly resists the next one, the one that would have worked.

Where the data lies to you

2. Using data from the future to predict the future

The training set contains information that only exists after the moment you are meant to be predicting. The model looks brilliant in the lab and is useless in the wild.

Picture a Galway delivery firm predicting whether a parcel will arrive late, with the number of failed delivery attempts among the inputs. You only know that count after the delivery window closes, so at prediction time it does not exist yet. In training it makes the model look almost perfect.

How to catch it: for every input, ask "would we really have this value at the instant we make the prediction". If not, it is a time machine, not a feature. What it costs: you find out after go live, when the lab numbers collapse to noise, having already paid for a build and a deployment.

3. Treating leakage as skill

A close cousin: a column that is really the answer in disguise sneaks into the inputs, so the model scores far too well on something it should find hard. This is not a beginner's mistake. Kapoor and Narayanan at Princeton found leakage errors in 329 published papers across 17 scientific fields that used machine learning.7 If peer reviewed researchers miss it, a sales deck certainly can.

Picture a credit risk model that includes a collections flag, a field that only gets set once a customer has already defaulted. Or a churn model scoring in the high nineties out of sample, with a field like "days since account closed" among its inputs. Only churned customers have a closed account, so that field is the answer written on the back of the card.

How to catch it: be suspicious of any single feature that carries the whole model. Ask them to remove the strongest input and show you the score again. A leaked model collapses. A real one barely flinches. What it costs: a huge win in the pilot, a rollout, and then ordinary decisions made with false confidence, which is worse than no model because now people trust it.

Where a rare event fools the score

4. Ignoring the base rate

When the thing you care about is rare, accuracy is a liar. If fraud is ten orders in a thousand, a model that flags nothing at all is 99% accurate and catches none of it. The base rate is the share of cases that are positive before any model looks at them (see base rates in the Foundations).

Picture a payments team shown a fraud model at "98% accurate" and delighted, until someone asks how many actual fraud cases it catches. Almost none, because it has learned that saying no is right most of the time.

How to catch it: ask what fraction of cases are positive, then ask what a do nothing baseline would score. If they cannot tell you, they have not thought about it. What it costs: you pay for accuracy and get a model that always says no.

5. Testing on the training data

The model is graded on the very examples it studied, so of course it does well. It has memorised, not learned.

Picture a demand forecast reported at 96% that then misses the next quarter by a mile, because it was scored on the same year it was fitted on. That measures memory, not skill.

How to catch it: listen for the words "out of sample". You want to hear that the reported numbers come from data the model never saw while training. What it costs: every number in the proposal is inflated, and you plan headcount and stock against a promise that was never tested.

6. Peeking at the test set

Subtler than the last one. The team tries fifty ideas, checks each against the test set, keeps the winner, and has fitted the test set by hand. Dwork and others showed in Science in 2015 that reusing a holdout set to guide the analysis in this adaptive way can make the results overfit the holdout itself, so its score stops being an honest estimate.8

How to catch it: ask how many times the final test set was looked at, and whether a separate untouched holdout is kept for the very end. What it costs: a reported score optimistic by an amount nobody can quantify, and that is the one number you were handed to size the whole investment.

Where the sample is not what you think

7. Survivorship and sampling bias

The data only contains the survivors. You study the customers who stayed and draw lessons that never applied to the ones who left.

Picture a retention study built entirely on current customers, meant to explain what keeps people. The people who left, the exact group you need to understand, are not in the table, so every conclusion is about the wrong crowd.

How to catch it: ask what is missing from the sample, not just what is in it. What it costs: you optimise hard for the people who were never going to leave and learn nothing about the ones who do.

8. Confusing correlation with causation

Two things move together, so the deck implies that pulling one lever moves the other. Ice cream sales and drownings, forever (see correlation in the Foundations).

Picture a report showing that customers who use the mobile app spend more, with the implication that pushing everyone onto the app raises spend. It may simply be that already loyal customers are the ones who install the app.

How to catch it: ask whether anything was actually intervened on, or whether it is all observed. If nobody ran a real test, read every "drives" and "causes" as "happens alongside". What it costs: a marketing budget spent forcing a behaviour that was a symptom of value, not its cause.

9. Extrapolating past the edge of the data

The model only ever saw orders between €10 and €500 and is now asked about a €50,000 tender. It answers confidently and wrongly.

How to catch it: ask what range each input covered in training, and what the model does when a live case falls outside that range. What it costs: a confident wrong answer on your biggest, rarest and most important cases, precisely the ones a person should have looked at.

Where the objective is quietly wrong

10. Optimising a metric that is not the goal

The team maximises accuracy or clicks because they are easy to measure, while the business cares about margin, lifetime value or not annoying its best customers.

Picture a recommendation engine tuned for clicks that learns to surface lurid low value items. Clicks go up and to the right, and margin quietly falls.

How to catch it: ask them to write the metric down in one line, then ask whether a model that scored perfectly on it could still lose you money. If yes, it is the wrong metric. What it costs: exactly what you measured, which turns out not to be what you wanted, usually at scale.

11. No cost of being wrong

Every error is treated as equal when it plainly is not. Missing a fraudulent order and blocking a loyal customer are very different costs, and a model tuned as if they were the same is optimising for a world you do not live in.

How to catch it: ask what a false alarm costs you and what a miss costs you. You want two different numbers, in euro. What it costs: expensive mistakes made as freely as trivial ones. The maths for this one is further down.

Where the data is dirtier than anyone admits

12. Assuming the data is clean

Nobody looked at the raw data. Dates in three formats, prices in two currencies with no label, a country field typed by hand with a dozen spellings of the same place. The model learns the mess faithfully.

Picture a sales model that treats blank price fields as a price of zero, so a batch of missing values teaches it that a whole product line is free. The chart looks fine until someone traces a mad prediction back to its source.

How to catch it: ask to see the profiling, meaning missing rates, duplicates, ranges and the odd values. If there is no profiling step in the plan, that is the first thing to add. What it costs: garbage learned faithfully, plus weeks spent discovering a mess that a day of profiling would have surfaced. My article on how much data you actually need covers the volume side of the same question.

13. Overfitting with too many knobs

Hundreds of features, not many rows, and a model flexible enough to memorise noise. It fits the past beautifully and predicts the future badly.

Picture a churn model with four hundred features and a few hundred customers. It has effectively learned each customer by heart, scores beautifully on history and falls apart on anyone new.

How to catch it: ask for the ratio of examples to features, and whether the model was kept deliberately simple. Complexity is a warning sign here, not a virtue. What it costs: a model that looks strongest on the slide and is weakest in production.

Where the world moves and nobody owns it

14. Concept drift ignored

The world the model learned in has moved on. Prices changed, a competitor launched, a season turned, and last year's patterns stopped holding without anyone noticing. Sculley and others at Google describe this in their paper on hidden technical debt in machine learning systems: changes in the external world and fragile data dependencies are among the reasons real systems carry large ongoing maintenance costs.9

How to catch it: ask how often the model is retrained, and how you will be told when it has gone stale, rather than finding out from a bad quarter. What it costs: a slow silent decay you only notice months later.

15. No owner for the data or the pipeline

The data lives in a spreadsheet on someone's laptop, or in a system a departed employee set up in a hurry. When the feed breaks, and it will, nobody is responsible.

Picture a nightly export a former analyst rigged up on their own machine. It runs fine until the laptop is wiped for the next new hire, and the model spends two weeks scoring empty data before anyone connects the dots.

How to catch it: ask who owns each data source, who gets the call at seven in the morning when it stops, and whether that is written down. What it costs: a fortnight of decisions made on stale or empty data. My article on who does what among data engineers, scientists and analysts explains which role owns which part.

16. No plan to act on the answer

The last mile is missing. The prediction lands in a report nobody wired into a workflow, so nothing changes, and the project counts as a failure even though the model was fine.

Picture a good risk score that arrives every Monday as a PDF attachment, which the team meant to act on never opens because it is not part of any tool they already use.

How to catch it: ask where the output goes and who, or what, acts on it automatically. If the answer is "we will have a look at it", the value is already leaking away. What it costs: the entire value of the project.

The quick reference card

Print this, keep it next to the proposal and tick your way down it. Three columns, because the cost is the part owners forget.

BlunderHow to catch itWhat it costs
No decision behind itWhat changes on Monday because of the resultA build nobody uses
Data from the futureWould we hold this value at prediction timeLab numbers that collapse at go live
Leakage as skillRemove the strongest input, show the score againConfident decisions people wrongly trust
Ignoring the base rateWhat does a do nothing baseline scoreA model that always says no
Testing on training dataAre these numbers out of sampleInflated numbers you plan against
Peeking at the test setHow often was the final set looked atA score optimistic by an unknown amount
Survivorship biasWhat is missing from the sampleLessons about the wrong people
Correlation as causationDid anyone actually interveneBudget spent forcing a symptom
Extrapolating past the edgeWhat range did each input coverWrong answers on your biggest cases
Wrong metricCould a perfect score still lose moneyWhat you measured, not what you wanted
No cost of being wrongCost of a false alarm versus a miss, in euroExpensive mistakes made freely
Dirty data assumed cleanShow me the profilingGarbage learned faithfully
OverfittingRatio of rows to featuresStrongest on the slide, weakest live
Concept driftHow often is it retrainedSilent decay noticed too late
No data ownerWho gets the seven o'clock callWeeks of scoring on empty data
No plan to actWho or what acts on the outputThe whole value left on the table

Accuracy is the number a slick pitch hides behind

Numbers four and five are where most owners get talked into a bad project, so here is one small worked figure. Suppose ten orders in every thousand are fraudulent. A model that does nothing at all and waves every order through scores this on plain accuracy (see percentages in the Foundations):

accuracy=TP+TNN\text{accuracy} = \frac{TP + TN}{N}

where TPTP is the number of true positives (frauds flagged), TNTN the number of true negatives (good orders waved through) and NN the number of orders. The do nothing model flags nothing, so TP=0TP = 0, TN=990TN = 990 and the accuracy is 990/1000=0.99990 / 1000 = 0.99.

That is 99%, and it has caught precisely zero fraud. If a pitch leads with a single big accuracy number on a rare event, that number tells you almost nothing and may hide a model that has learned to always say no. Ask instead how many of the actual fraud cases it catches, and how many good orders it wrongly blocks to get there. Those two numbers are the real story, and they are the ones a weak proposal leaves off the slide.

The least expensive model on paper can be the most expensive in euro

Numbers ten and eleven hide together, and this is the maths that exposes them. A model makes two kinds of mistake. It raises a false alarm, flagging something that was fine, and it misses, letting through something it should have caught. These almost never cost the same.

OutcomeModel says yesModel says no
Actually yescaught it (good)a miss
Actually noa false alarmleft alone (good)

Only the false alarm and the miss hurt, and they carry different price tags. Blocking a loyal customer might cost a relationship worth a great deal. Missing one fraudulent order might cost only that order. So the number that matters is the expected cost of the errors, an expected value in money:

E[cost]=CFA⋅NFA+CM⋅NME[\text{cost}] = C_{FA} \cdot N_{FA} + C_{M} \cdot N_{M}

where CFAC_{FA} is the cost of one false alarm, NFAN_{FA} the number of false alarms, CMC_{M} the cost of one miss and NMN_{M} the number of misses over the same period.

A worked example with invented numbers: a Limerick shop takes 10,000 orders a month, 100 of them fraudulent. Model A blocks 300 good orders and misses 20 frauds. Model B blocks only 50 good orders but misses 60 frauds. Say a blocked good customer costs €40 in lost margin and goodwill, and a missed fraud costs €60. Model A costs 300×40+20×60=13,200300 \times 40 + 20 \times 60 = 13{,}200 euro a month. Model B costs 50×40+60×60=5,60050 \times 40 + 60 \times 60 = 5{,}600 euro. Model B misses three times as much fraud and is still the better business decision. Ask any vendor to fill in that line. If they have never priced a false alarm against a miss, they have optimised a number that has nothing to do with your profit and loss.

A worked mini case: five of the sixteen in one project

In real life the blunders rarely arrive one at a time. Here is an invented but typical case. A subscription box retailer in Galway, boxclub.example, has a model that predicts which subscribers will cancel next month, scoring in the low nineties, and a plan to send the flagged subscribers a retention discount.

The zeroth question passes: there is a real decision, who gets a discount. Then number three bites. The strongest feature is "days since last login", but people who have already decided to cancel stop logging in first, so the feature is half reading the answer. Remove it and the score drops into the seventies. Number four bites next. Only about three subscribers in a hundred cancel in a given month, so a model that predicts nobody cancels is already 97% accurate, and the proud headline means almost nothing. Number eleven was skipped. Nobody priced the discount, so a false alarm hands money to someone who was never going to leave, while a miss loses a subscriber. With euro on both, the sensible threshold is far more cautious than the one they picked. And number sixteen waits at the end: the discount emails were to be pulled by hand from a spreadsheet once a month by someone already busy.

The model is not the problem. The framing is, in five places, and every one is catchable in an afternoon with the questions above. The fix is to rebuild the target so it only uses information available before the cancellation, price the two mistakes, set the threshold against expected cost rather than accuracy and wire the output straight into the email tool. The honest score is lower and the project makes money.

The pre commission decision tree

Before a single euro is committed, walk the project through four gates.

Tap or click to open full screen and move around

Living with the model: the loop nobody budgets for

Getting a model live is not the finish line, it is the start of a loop, and numbers fourteen and fifteen live here. Somebody has to watch the live scores, decide when the model has gone stale, retrain it on fresh data and put the new version back.

Tap or click to open full screen and move around

Budget for this loop up front and name the person who watches it. A proposal that treats go live as the end of the work is promising a model that is best on its first day and worse every day after.

The common mistakes owners make

Everything so far points at the vendor. It is only fair to turn the same honesty on the buyer.

The first is falling for the single big number. A round, confident figure feels like proof. It is not, and the rarer the event the less it means, so treat any lone headline score as an opening question.

The second is confusing confidence with competence. A vendor who never once says "I am not sure" is not more skilled, just less honest. People who know this work will tell you where a model is weak.

The third is buying the model and not the pipeline. The clever model is a small part of the work. The rest is data plumbing, profiling, the monitoring loop and the wiring that turns a prediction into an action, which is exactly the maintenance burden the Google paper describes.9 If the proposal spends all its words on the model and none on the plumbing, you are buying the small part and hoping the big part appears for free.

The fourth is skipping the profiling because it is not exciting. Nobody puts data cleaning in the pitch, yet it is where most of the risk lives.

The fifth is treating go live as the finish line, when it is the moment the work of keeping the thing honest begins.

And the last is not naming an owner on your own side. If nobody in your business is accountable for the decision the model feeds, it will drift back into being wallpaper however good the maths.

How to use this against a proposal you are handed

You do not need to argue the maths. Sit with the deck, run the sixteen questions and pay less attention to the answers than to the comfort. A partner who has built real things will meet number three with a shrug and a demo, tell you the base rate before you ask, and name the person who owns the data feed. A partner selling you a story will get defensive around leakage, lead with one shiny accuracy figure and go quiet when you ask what changes on Monday.

Two practical points if you hire me for this. Your raw data stays yours, and the terms for code and models are in the analytics and data science terms. And the first conversation is a free workshop, plus the first two weeks of work, so you can see how I handle the sixteen before you commit a budget. Book the workshop here.

The failure modes in this list are not exotic. They are ordinary and avoidable, and every one of them is far less expensive to catch in a meeting than in month four of a build that was never going to work.

If you have a data proposal in front of you and are not sure it stacks up, or data sitting there and a decision you think it could inform, bring it to a free workshop. I will run it through these sixteen questions with you, and the first two weeks of work are on me.

Sources

  1. 1.

    James Ryseff, Brandon F. De Bruhl and Sydne J. Newberry, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed, RAND Corporation, 13 August 2024, https://www.rand.org/pubs/research_reports/RRA2680-1.html

  2. 2.

    Regulation (EU) 2016/679 (GDPR), Article 5(1)(b), purpose limitation, https://eur-lex.europa.eu/eli/reg/2016/679/oj

  3. 3.

    Regulation (EU) 2016/679 (GDPR), Article 22, automated individual decision making including profiling, https://eur-lex.europa.eu/eli/reg/2016/679/oj

  4. 4.

    Court of Justice of the European Union, judgment of 7 December 2023, SCHUFA Holding (Scoring), Case C 634/21, https://curia.europa.eu/juris/liste.jsf?num=C-634/21

  5. 5.

    Regulation (EU) 2024/1689 (Artificial Intelligence Act), Annex III, point 5(b), https://eur-lex.europa.eu/eli/reg/2024/1689/oj

  6. 6.

    Digital Omnibus on AI, Official Journal of the European Union, L series, 24 July 2026, in force 27 July 2026, https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202601744

  7. 7.

    Sayash Kapoor and Arvind Narayanan, Leakage and the reproducibility crisis in machine learning based science, arXiv 2207.07048, https://arxiv.org/abs/2207.07048

  8. 8.

    Cynthia Dwork and others, The reusable holdout: preserving validity in adaptive data analysis, Science 349 (2015), https://doi.org/10.1126/science.aaa9375

  9. 9.

    D. Sculley and others, Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems 28 (2015), https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html