Is your dashboard lying to you? A ten minute data quality audit
TL;DR
Difficulty: Beginner
Dashboards rarely fail loudly. They show plausible numbers that are quietly wrong, and you act on them. Run six checks: reconcile one total with the shop's own records, compare total rows with populated rows (ten blanks read as zero turn a £100 average into £80), tick off every expected segment, distrust flat or round averages, read the raw campaign labels and find the last updated time. In GA4, also know what is missing on purpose: people who declined consent, modelled estimates that only some reports show, rows withheld below privacy thresholds, known bots removed, and the last two days still processing. Fix every cause at the source, not on the chart.
A wrong number that looks confident is more dangerous than no number, because it stops you asking questions. In this article I give you a ten minute audit with six checks any owner can run, work through how ten blank fields read as zero cut an average order value by 20% and switch off a profitable campaign, and then show the four places where GA4 withholds, models or filters data on purpose, so you can tell an honest gap from a real error.
A confident wrong number is worse than no number
A wrong number that looks confident is more dangerous than no number at all. When you have no number you stay careful. You ask around, you hedge, you keep your options open. When you have a number, especially a tidy one on a nice chart, you stop asking and you act. If that number is quietly wrong, you have just made a decision on bad data with full confidence, which is the worst way to be wrong.
Data quality problems do not announce themselves. A broken pipeline shouts, so somebody fixes it before lunch. A subtly wrong average whispers. The chart still renders, the average still computes, the line still goes up and to the right, and you keep believing it for months because it never gives you a reason not to.
This article gives you a ten minute audit you can run on your own dashboard this afternoon, with no data science team, only a suspicious mind and six quick checks. The stance behind it is simple: treat every number as guilty until it proves itself innocent. If your dashboard is Google Analytics 4 (GA4), there is a second layer, because GA4 withholds, models and filters data on purpose, and each of those honest design choices can look exactly like a lie. That gets its own section after the checks.
This is not the same problem as a dashboard nobody opens. A correct dashboard that changes no decision is covered in why your dashboard project keeps dying on the shelf. This one is about the dashboard people do open, and believe.
Why bad data hides so well
A software bug tends to be loud. A page goes blank, a red error appears, and everyone rushes to fix it. A data quality problem is the opposite kind of animal. The code runs perfectly, every function returns a value and the pipeline finishes green. The only thing wrong is that one of those values does not mean what you think it means, and no automated test catches a number that is technically valid but factually false. Eighty is a perfectly good number. It sorts, sums and averages without complaint. It is just not the truth.
So the first thing to take away is this: the absence of an error is not evidence of correctness. A dashboard that has never thrown a warning is not a trustworthy dashboard, it is a quiet one, and quiet is not the same as honest.
The one that poisons everything: blanks counted as zero
Start with the check that catches the most damage for the least effort.
The symptom is a headline average that feels a little low. Not alarmingly low, just softer than you expected. Your gut says average order value should be around £100, the screen says £80, and you shrug, because £80 is plausible. That shrug is the symptom. The number is deflated by exactly the amount that keeps it believable.
The cause is that, somewhere between the database and the chart, blank values were read as zero instead of being skipped. In SQL, the AVG function averages only the values that are not null, but the moment someone wraps the column in COALESCE to turn null into zero, the nulls join the calculation as real zeros.1 A spreadsheet does the same in its own way: Excel's AVERAGE skips empty cells but counts cells that contain a zero.2 An export writes empty strings, the import reads them as numbers, and empty becomes nought. Nobody decided this. It is a default, and defaults are where data quietly goes wrong.
Worked example: fifty orders, ten blanks
Take an invented report of fifty orders in a month. Forty of them recorded a value, and those forty values add up to £4,000. The other ten came in through a channel that did not record a value, so the field is blank.
The mean is the sum of the values divided by their count (Foundations: mean):
Here is the mean, is the number of values counted, and is the value of order .
The honest version averages only the orders that have a value: .
The poisoned version turns the ten blanks into zeros. The sum stays at £4,000, but you now divide by fifty: .
Same business, same real orders, and the headline average order value fell from £100 to £80. Put the poisoned figure over the true one: .
That is a 20% understatement, and nothing on the dashboard looks broken.
How far the poison travels
The bad average does not stay in its lane. It feeds the forecast, the margin per customer and, most expensively, the most you can afford to pay to win an order. A simple version of that ceiling is the gross margin on the average order (Foundations: percentages):
Here is the most you can pay in advertising to win one first order and still break even on it, is the gross margin as a share of the order value, and is the average order value.
With a 40% gross margin, the true ceiling is: .
The poisoned ceiling is: .
Now picture a campaign that wins orders at £36 each. Against the poisoned ceiling of £32 it looks like a loser, so you pause it. Against the true ceiling of £40 it was earning £4 on every first order. The blank field did not just cost you an accurate chart. It cost you the growth you switched off, and that bill never appears on the dashboard that caused it.
The ten minute audit, check by check
You do not need to be technical. You need to be nosy. Go through these in order and note anything that fails. A flag is not a disaster, it is a lead to follow.
Check one: do the totals reconcile
Pick one number you can verify against something outside the dashboard: total orders this week against the raw order table in the shop, or total revenue against what the payment provider paid out. If the dashboard says more than the source, you probably have double counted rows. That happens the moment a join fans out, with one order matching three parcel lines and being counted three times. If the dashboard says less, something is being dropped or filtered before it reaches you.
A clean multiple is a fingerprint. If the dashboard reports 1,500 orders and the order table holds 500 rows, the ratio is exactly three, and that is almost never noise. A number that does not reconcile with a source you trust is not a number yet. It is a rumour. Reconcile one figure a week, because the totals are the load bearing wall the rest of the dashboard rests on.
Check two: blanks against zeros, in every rate
Blanks do not only wreck averages. They wreck rates, and rates run your marketing. A conversion rate is conversions divided by sessions (Foundations: ratios and rates):
Here is the conversion rate, is the number of conversions and is the number of sessions.
Take an invented week with 2,000 sessions and 40 orders: .
That is 2%. Now suppose 500 sessions with no recorded outcome get added to the denominator as if they had been counted all along: .
The rate falls to 1.6%. Suppose instead that a tag failed to fire and 500 real sessions were never recorded at all: .
The rate jumps to about 2.67%. Same reality, two opposite lies, both decided by how one blank was handled.
| Where the blank lands | What it does to the rate | What you wrongly conclude |
|---|---|---|
| Padding the denominator | Pushes the rate down | The channel looks worse than it is |
| Missing from the denominator | Pushes the rate up | The channel looks better than it is |
| Padding the numerator | Pushes the rate up | The offer looks like it converts more |
| Missing from the numerator | Pushes the rate down | The offer looks like it converts less |
The quick test is the same as before: count total rows, count populated rows, compare. Then ask the same question of the top and the bottom of every rate on the page. Anything with a slash in it, conversion rate, refund rate, margin, is a blank waiting to break in one of two directions.
Check three: the missing segment
A segment that silently vanished is worse than one that shows a bad number, because a bad number draws the eye and a missing row does not. The usual cause is a filter or a naming mismatch upstream. A channel is renamed at the source, the dashboard filter still looks for the old name, and the whole channel drops out of every chart without a whisper. The only visible trace is a total that drifts down and that nobody can explain.
The test is boring and it works. Write down the segments you know exist, every channel, region or product line, and tick them off against the screen. If one you know exists is missing, something is filtering it out and taking its numbers with it. In GA4 there is a second reason a row can disappear, thresholding, which comes up below.
Check four: suspiciously flat or round averages
Real data is lumpy. Genuine averages land on awkward numbers like 97.3 and 104.8, because they are the residue of many messy values. When an average sits on exactly a round number, or a column of averages is identical across groups that should differ, be suspicious. Flat and round usually means a default value was substituted for missing data somewhere, and a pile of records stamped with the same placeholder drags the whole distribution towards a fake centre.
The mean alone hides this, so look at the spread too. The standard deviation measures the typical distance of a value from the mean (Foundations: spread):
Here is the standard deviation, is the number of values, is each value and is their mean.
You do not need to compute it by hand. You need the intuition: real business data almost never has a tiny spread. If delivery times across five regions all read 3.0 days, the spread is zero, and real parcels are never that tidy. Comparing the mean with the median helps as well (Foundations: median and mode). When a default has been stuffed in, the median often sits right on the placeholder while the mean drifts, and the gap is the fingerprint of the fill.
Check five: broken UTMs and inconsistent naming
Open your traffic sources and read the raw labels, not the tidy chart. You will often find Facebook, facebook, FB and fb ads all describing one channel, each counted separately. None of this throws an error. The machine treats two spellings as two truths, so a channel that would clearly clear your target if its variants were added up shows as several weak ones, and you cut the one that was working. Campaigns that were never tagged sit in direct or unassigned traffic.
The fix is a naming convention enforced where links are created, not a manual clean up every month. How GA4 reads the parameters and sorts them into channels is covered in mastering UTM tracking for your online campaigns.
Check six: is the data even fresh
The symptom is a chart that looks the same on Thursday as on Monday, and you cannot tell whether trade really went flat or the data stopped moving. The cause is a pipeline that failed silently, so the last good load sits there looking current. Find the time that tells you when the data last updated. If the dashboard does not show one, that absence is itself the flag. Stale data is the sneakiest wrong number of all, because it was correct, once.
When the dashboard is GA4: four honest gaps that look like lies
GA4 adds a layer the six checks do not cover. It deliberately withholds, models and filters data, and each of those design choices produces numbers that differ from your shop's own records for good reasons. The danger is not the gap. The danger is reading the gap as a fault and "fixing" it, or reading GA4 as the truth and the shop as the error. Every fact below comes from Google's own help pages as they read in November 2026.
Consent: people who decline are not in the data
With Google's consent mode in its basic form, the Google tags do not run until the visitor answers the banner, and if the visitor declines, Google says that no data is transferred at all, not even the consent status. In the advanced form, the tags load anyway and send measurements without cookies when consent is denied.3 On the sites I build for Germany, Austria and Ireland I use basic mode, and advanced only on the shop's lawyer's written advice. In the UK it is a risk decision. The reasoning sits in event tracking and consent mode.
So in basic mode GA4 can only count the purchases of people who said yes. If the shop recorded 500 orders in a week and GA4 shows 340 purchase events: .
GA4 saw 68% of the orders. That is not a broken tag, it is the consent gap, and its size is set by your banner and your visitors. The rule follows directly: revenue and order counts come from the shop's own database, which sees every order, and GA4 is for patterns among the people who agreed to be measured. Never reconcile the shop against GA4 and call the difference an error. Reconcile GA4 against the shop and call the difference the share of orders you cannot see.
Modelled data: some numbers are estimates, and only some reports show them
GA4 can fill part of the consent gap with behavioural modelling, which uses machine learning to estimate the behaviour of people who declined analytics cookies from the behaviour of people who accepted. Google lists the conditions: consent mode on every page, an advanced implementation where the tags load before the banner, at least 1,000 events a day with analytics storage denied for at least 7 days, and at least 1,000 daily users with analytics storage granted on at least 7 of the previous 28 days. Even then eligibility is not guaranteed. Modelled data only appears under the Blended reporting identity, and it is not part of the BigQuery export.4
Two consequences for an audit. A small shop running basic mode gets no modelled data at all, so there is nothing to fill the gap and nothing to explain away. A larger property that does qualify will show different numbers in the GA4 interface than in its own BigQuery tables, by design. Before comparing two GA4 figures, write down which reporting identity each came from. Why modelling is out of reach for most small shops, and what to estimate instead, is covered in event tracking and consent mode.
Thresholds: rows withheld on purpose
GA4 withholds data from a report or exploration when it could allow someone to infer the identity of individual users. Google names the cases: reports that include demographics, search query data when there are not enough users, and narrow date ranges with low user or event counts. The thresholds are system defined, you cannot adjust them, and Google does not publish the values. A report with withheld rows says so in the data quality indicator at the top. Widening the date range can bring the rows back.5
This is the GA4 version of check three. Before you hunt for a broken filter behind a missing segment, look at the data quality indicator. A small segment over a short range may simply be below the threshold, and a month later it reappears.
Bot filtering: you cannot see how much was removed
GA4 automatically excludes traffic from known bots and spiders, identified from Google's own research and the International Spiders and Bots List maintained by the Interactive Advertising Bureau. You cannot switch the exclusion off, and you cannot see how much traffic it removed.6 Two things follow. Session counts in GA4 will not match your server logs, and the difference is not something you can audit. And the exclusion speaks of known bots, so a sudden burst of odd traffic from one city with no engagement deserves suspicion rather than celebration, because it is not covered by a list it is not on.
Freshness: yesterday is not final yet
For a standard GA4 property, Google gives about 12 hours for daily data to process, and says processing can take 24 to 48 hours, during which the numbers in reports may change.7 So the freshness check has a GA4 rule of its own: never compare yesterday with the same day last week as if both were final. Wait two days, or mark the most recent days as provisional on the screen.
A worked mini case: ten minutes at Northgate Hampers
Northgate Hampers is an invented online hamper shop in Leeds, run by Priya, at northgatehampers.example. It is Monday, she opens the weekly report, and everything is green. Here are her ten minutes.
Minute one, reconcile. The dashboard says 1,500 orders, the order table says 500. A factor of three, so a join is counting each order once per parcel line. Flag one: every count and revenue figure on the page is inflated threefold until proven otherwise.
Minute three, blanks. Of the 500 orders, 400 carry a value, together worth £36,000. The honest average is: .
The dashboard divides by all 500 rows: .
Flag two: the headline average shows £72 instead of £90, and every target pegged to it is too timid by the same 20%.
Minute five, segments. The channel breakdown lists paid search, email and direct, but Priya knows a healthy affiliate channel exists. It was renamed at the source last month and the filter still looks for the old name. Flag three.
Minute seven, flatness. Average delivery time reads exactly 3.0 days in every region. Flag four: a default of three days is filling the blanks.
Minute nine, naming. The source list shows google, Google and google cpc as three rows. Flag five: one channel split three ways, each looking weaker than the whole.
Minute ten, freshness and GA4. There is no last updated time on the page, and the GA4 panel shows 340 purchases against the 500 orders. Flag six is the missing timestamp. The GA4 figure is not a flag: Priya runs basic consent mode, so 68% coverage is the consent gap, and the order table remains her source for revenue.
Six flags in ten minutes on a dashboard that looked flawless, and one difference she correctly left alone. None of them threw an error.
Common mistakes when you run this
The first is fixing the chart instead of the cause. You spot the blanks, add a filter to the one chart, the number looks right, and three other reports still read the same field the same broken way.
The second is trusting a number because it matches your expectation. Confirmation is not verification. Audit the numbers you like at least as hard as the ones you do not.
The third is auditing once and calling it done. A source gets renamed, a tag gets removed, a new channel comes online untagged, and yesterday's clean dashboard is today's liar. The audit is a routine, not a project.
The fourth is confusing precision with accuracy. A figure quoted to two decimal places feels more trustworthy than a round one, but the decimals describe the arithmetic, not the truth of the inputs.
The fifth is new with GA4: treating its gaps as bugs. Server side tracking can recover events that browsers lose, but it does not bring back people who declined, and it should not try. What it can and cannot recover is set out in server side tracking: the technical guide, and where the consented data ends and first party data begins is in cookieless marketing and first party data.
Why the tidy dashboards are the dangerous ones
The instinct is to trust the clean dashboard and distrust the messy one. It is backwards. A dashboard with obvious errors and blank panels keeps you honest, because you can see it is broken. The tidy one, all green, every number round, is the one that walks you confidently off a cliff. Polish is a design property. Truth is a data property. A beautiful chart of a wrong number is a wrong number in a nice suit.
What to do once a check fails
When a check fails, resist the urge to patch the chart. The wrong number on screen is a symptom, and the useful question is how far upstream the cause lives and what else it has already touched. If blanks are read as zero in this average, they are almost certainly read as zero in every other report on the same field.
This is also why I build reporting on data the owner can reach into: the shop's own database next to GA4, so every figure can be traced to the rows behind it. You cannot audit what you cannot open. If you want a second pair of eyes on your dashboards, that is part of the data science work I do. Your raw data stays yours, and the terms for code and models are in the analytics and data science terms.
How to make it a habit next week
Pick the one number your budget or pricing hangs on and reconcile it against the shop's own records once a week. Add a last updated time to the top of every dashboard you own, and mark the last two days of GA4 data as provisional. For each key figure, write down how blanks are handled; the moment you have to write it down, you usually find out you do not know. Agree a naming convention for channels and campaigns and enforce it where links are created. Note which reporting identity each GA4 figure uses. And once a quarter, run the full audit on your most trusted dashboard, precisely because it is the one you check least.
The one question to ask every number
Before you make a decision on a number, ask it how it could be wrong. Are you counting blanks as zero? Are you double counting a join? Are you missing a segment, or is it below a threshold? Are you flat because someone filled you with a default? Are you stale, or still being processed? Are you only the people who said yes? Ten minutes of suspicion, run as a routine, catches more bad decisions than any fancier analysis on data nobody checked. The goal is not a prettier dashboard. It is a number you have earned the right to trust.