Measurement
Part of Measuring one customer data journey without pretending correlation is cause
Three customer data platforms measurement mistakes that overstate a result
Customer data platforms measurement mistakes that overstate a result: a rising match rate, supplier credit called cause and totals counted twice.
A supplier report can show a rising match rate, a large credit figure and a total that looks complete. Three of the mistakes that make a result look stronger than its evidence can be spotted in a versioned report, and each can be corrected before anything goes out.
These checks read the report, not the platform. They are observable defects, not claims about how often organisations make them, and no supplier funded or influenced them. Each one ends with the wording a reviewer can ask the author to use.
What to take away
- Forcing uncertain identities together can raise a displayed match rate while damaging accuracy and rights handling.
- A dashboard model allocates credit under its own inputs. It does not estimate what would otherwise have happened.
- Several systems may claim the same outcome, and summing their figures can count it repeatedly.
- An overwrite removes the earlier definition, value and reason, so keep both versions of a restated figure.
The claim, what it shows, and the wording to ask for
| Claim in the report | What the figure can show | Wording to ask for |
|---|---|---|
| The match rate rose again this release | A displayed rate went up, whatever happened to accuracy | Match rate, with the unresolved count beside it |
| The dashboard says a channel drove the outcome | Credit allocated under the supplier's own inputs and window | Supplier-attributed credit, not an estimated effect |
| A combined total across two systems | One outcome may be counted more than once | Reconciled outcome count, each credit rule shown separately |
| A restated figure with no note | A change that readers cannot trace | Earlier and new values side by side, with the reason |
A rising match rate is not a better match
A match rate that climbs every release is a prompt to ask questions, not a result to celebrate. Forcing uncertain identities together can raise a displayed rate while damaging accuracy and rights handling.
The UK GDPR accuracy principle says personal data must be accurate and, where necessary, kept up to date, and that inaccurate data must be erased or rectified without delay. A wrongly joined profile can carry one person's details on another person's record, so the rate alone cannot show whether the matching is safe.
Make ambiguity an owned queue and measure unresolved cases without penalty for safe separation. A team is then not rewarded for merging records it cannot justify. The ICO's rights routes for correction, restriction, objection and deletion show what is at stake when a join is wrong.
Supplier credit is not cause
A dashboard model allocates credit under its own inputs, identity graph and window. It does not by itself estimate what would have happened otherwise, so the word "drove" needs a design behind it.
The Magenta Book is HM Treasury guidance on what to consider when designing an evaluation. Reserve causal language for a defensible experimental or quasi-experimental design, and label everything else as supplier attribution, with the supplier, window and identity rule named beside the figure.
A reviewer does not need to settle the causal question to return the report. Asking for the label is enough. The guide to attribution methods shows when a causal claim should be withheld, and what a plainer statement can say instead.
Totals that count one outcome twice
Several systems may claim the same outcome, and summing the figures can count it repeatedly. Reconcile the underlying event once, then display each credit rule separately so a reader can see how the credit was shared.
A combined total is only comparable when it counts the same event. Freeze the event, population, numerator, denominator, period and Europe/London clock beside the result, as a metric definition card does, before anyone sets it against another figure.
The same discipline applies to outside numbers. A supplier figure that cannot say what it counted gives nothing to compare with. Screening benchmark claims sets out the questions to ask before a number is trusted.
Correcting a figure that has already gone out
Keep both versions. An overwrite removes the earlier definition, value and reason, so users cannot tell whether the series changed because the world changed or because the query did.
Mark the restatement beside the figure and retain the earlier version. The Government Analysis Function's reproducible analytical pipelines strategy supports version control and auditable analysis, which is the habit a corrected report depends on.
A reporting dashboard can show corrections on its first screen, so readers see a restated figure flagged rather than discover it later. The full build route sits in the guide to measuring one journey.
Before the report leaves your desk
- Show the unresolved identity count beside every match rate.
- Label supplier attribution as such, and reserve causal words for a defensible design.
- Reconcile the underlying event once before adding any totals.
- Display each credit rule separately.
- Mark every restatement and retain both versions.
Common questions
Is a rising match rate ever acceptable?
Ask how many identity decisions remain unresolved and whether uncertain identities were forced together. A rise that comes with a growing unresolved queue has not shown better matching.
Can a supplier dashboard say a channel caused a result?
Only a defensible experimental or quasi-experimental design supports causal language. Otherwise the figure should be labelled as supplier attribution, allocated under the supplier's own inputs and window.
What should a corrected report show?
The earlier definition, value and reason, beside the new figure. Keeping both lets a reader see whether the series changed because the world changed or because the query did.
