The Difference Between Data and Evidence

What is the difference between data and evidence?

Data is a collection of observations. Evidence is data that has been contextualized, analyzed, and connected to a specific claim through reasoning that can withstand scrutiny. Most organizations have abundant data and almost no evidence.

I sat in a meeting where a product manager presented this data: “Users who enable feature X have 40% higher retention.” The implicit claim was that feature X causes retention. But the data was observational, not experimental. Users who enable feature X might be more engaged to begin with. The feature might correlate with retention without causing it. The 40% number was real data. It was not evidence for the claim being made. The analytical step from data to evidence (controlling for confounders, testing alternatives, establishing causation) was skipped entirely.

Why do organizations confuse data with evidence?

Organizations confuse data with evidence because data looks objective and quantitative, which creates a false sense of rigor, and because the analytical work required to transform data into evidence (hypothesis testing, causal analysis, counterfactual reasoning) is slow, expensive, and often produces inconvenient conclusions.

The confusion is structural. According to the philosophy of evidence, evidence requires a relationship between observations and claims. Data without that relationship is just numbers. But numbers feel authoritative. A slide that says “40% higher retention” feels more persuasive than one that says “we have a hypothesis about retention that requires further testing.” Organizations reward certainty and punish ambiguity, so data gets presented as evidence because evidence sells decisions and data only raises questions.

I have seen this pattern in every organization I have worked with. The data team produces reports. Stakeholders treat the reports as evidence. The analytical step between the two (is this data actually evidence for the claim you are making?) is nobody’s explicit responsibility. The epistemology of metrics addresses this gap, but few organizations apply epistemological discipline to their analytics.

How can data teams bridge the gap between data and evidence?

Data teams bridge the gap by requiring every data presentation to explicitly state the claim being made, the assumptions required for the data to support that claim, and the alternative explanations that have not been ruled out.

• Claim-first analysis: Instead of starting with “here is what the data shows,” start with “here is what we are trying to determine.” The claim frames what kind of analysis is needed. An observational question needs description. A causal question needs experimentation or careful causal inference. Matching the analytical method to the claim type is how data becomes evidence
• Assumption documentation: Every analysis should document its assumptions explicitly. “This analysis assumes that feature X adoption is not correlated with pre-existing user engagement.” When assumptions are visible, stakeholders can evaluate whether they hold, rather than accepting conclusions at face value
• Alternative explanation audit: Before presenting data as evidence, list at least three alternative explanations for the observed pattern. If you cannot rule them out, the data supports your claim but does not prove it, and that distinction matters for decision-making

The Popperian approach to A/B testing operationalizes this: seek to disprove your hypothesis, not confirm it. If the data survives attempts at falsification, it is stronger evidence. If you never try to falsify, you do not have evidence. You have confirmation bias with charts.

What are the consequences of treating data as evidence?

Treating data as evidence without analytical rigor leads to confident wrong decisions, because the veneer of data-drivenness makes organizations less likely to question conclusions that are supported by numbers, even when those numbers do not actually support the conclusion.

According to evidence-based management research, organizations that adopt “data-driven decision making” without analytical rigor perform worse than organizations that rely on experienced judgment, because the false confidence from misinterpreted data overrides the legitimate uncertainty that experienced decision-makers would acknowledge. Data without rigor is not an upgrade from intuition. It is a downgrade, because it replaces honest uncertainty with dishonest confidence.

Data is cheap. Evidence is expensive. The gap between them is analytical discipline: the willingness to question your own numbers, document your assumptions, and present conclusions with appropriate uncertainty. Organizations that invest in evidence, not just data, make better decisions. Those that confuse the two make worse decisions while feeling more confident about them, which is the most dangerous possible combination.