Vanity Metrics and the Theater of Data-Driven Decision Making

What distinguishes vanity metrics from actionable metrics?

A vanity metric makes you feel good without changing what you do. An actionable metric changes a decision. The distinction is not in the metric itself but in whether it connects to a decision framework that produces different actions based on different values.

I audited the executive dashboard of a B2B SaaS company. The dashboard displayed 14 metrics. I asked the leadership team one question about each: “What would you do differently if this number were 20% lower?” For 9 of the 14 metrics, the answer was either “I don’t know” or a vague “we’d investigate.” Only 5 metrics had clear decision rules: if churn exceeds X, trigger retention campaign; if pipeline velocity drops below Y, adjust sales hiring. The 9 metrics without decision rules were vanity metrics. They were measured, displayed, and discussed, but they changed nothing.

Why do organizations invest in vanity metrics?

Organizations invest in vanity metrics because metrics perform objectivity, creating the appearance of data-driven culture without requiring the discomfort of actually changing behavior based on what data reveals.

The performance is the point. Having a dashboard communicates rigor. Quoting numbers in meetings communicates competence. Increasing month-over-month totals communicates progress. None of these require the metric to actually inform a decision. They require the metric to exist and to look good. This is what I call the theater of data-driven decision making: the rituals of data culture (dashboards, reports, KPI reviews) without the substance (changing decisions based on evidence).

According to Goodhart’s Law, when a measure becomes a target, it ceases to be a good measure. But the corollary is less discussed: when a measure becomes a performance (something displayed for its own sake), it never becomes a target at all. It exists in a permanent limbo of measurement without action. The Goodhart’s Law in your dashboard piece explores this dynamic in depth.

How can organizations move from metrics theater to genuine data-driven practice?

The transition requires connecting every metric to a decision rule, killing metrics that do not connect to actions, and creating accountability for acting on what metrics reveal, especially when the data is uncomfortable.

• Decision rule requirement: For every metric on a dashboard, document the decision it supports: “If X exceeds threshold Y, we do Z.” Metrics without decision rules are candidates for removal. This is painful because it forces clarity about what the organization actually manages versus what it merely observes
• Uncomfortable data test: The true test of data-drivenness is not whether you celebrate good numbers. It is whether you change behavior when numbers are bad. I ask teams: “Show me a time in the last quarter when data told you something you did not want to hear, and you changed course because of it.” Organizations that cannot answer this question are performing data-drivenness, not practicing it
• Metric sunset reviews: Quarterly, review every active metric with one question: “Did this metric change any decision in the last 90 days?” If not, deprecate it. The dashboard design discipline starts here

What is the cost of pretending to be data-driven?

The cost is double: the organization spends real resources (engineering time, tool licenses, meeting hours) maintaining the apparatus of data-drivenness while gaining none of the decision quality benefits, and the performance crowds out genuine analytical work.

I calculated the total cost of vanity metrics at one organization: $180,000 per year in BI tool licensing, $240,000 in engineering time maintaining pipelines and dashboards for metrics nobody acted on, and approximately 400 meeting-hours per year discussing those metrics. Total: over $420,000 annually spent performing data-drivenness. That budget could have funded 2 analysts doing deep causal analysis, which would have actually changed decisions.

According to research published in Harvard Business Review, data-driven organizations outperform peers by 5% to 6% in productivity. But the key word is “data-driven,” meaning driven by data. Organizations that are “data-decorated,” displaying data without being driven by it, capture none of that productivity premium. They pay the cost of data infrastructure without receiving the benefit of data-informed decisions. The meeting that should have been a query is often the venue where this theater plays out.

Vanity metrics are comfortable. Actionable metrics are uncomfortable. The difference between a data-driven organization and a data-decorated one is the willingness to be uncomfortable: to measure what matters, even when it reveals problems, and to change course when the data demands it, even when the change is difficult. Anything less is theater. Expensive theater.