Data Observability Needs Product Thinking

What is the difference between infrastructure observability and product observability?

Infrastructure observability asks “Did the pipeline run?” while product observability asks “Did the data serve its purpose?”, and the gap between these questions is where the most damaging data failures hide.

I deployed Monte Carlo for data observability at a B2B SaaS company in 2024. The platform performed well on its design objectives: it detected schema changes within 5 minutes, flagged volume anomalies with 92% precision, and monitored freshness across 340 tables. The dashboards glowed green. The data team declared observability “solved.”

Three months later, the VP of Sales reported that the pipeline attribution model had been double-counting conversions from a partner channel for 6 weeks. The error inflated attributed pipeline by $3.1 million. No observability alert fired because the data was structurally perfect: correct schema, expected volume, on-time delivery. The error was semantic. The join condition between the partner feed and the attribution model used a field that had been repurposed upstream, and the observability system had no concept of what the join was supposed to mean.

Why does infrastructure-only observability create false confidence?

Infrastructure-only observability creates false confidence because it provides comprehensive monitoring of the dimensions that rarely cause business impact while providing no monitoring of the dimensions that almost always cause business impact.

I categorized 94 data incidents at 3 organizations over 6 months by type and impact:

• Structural failures (schema changes, pipeline crashes, freshness delays): 41 incidents, 78% detected by observability tools, average business impact of $2,400 per incident. These are loud failures. They break things visibly. Monitoring catches them because they violate measurable technical thresholds
• Semantic failures (wrong calculations, misapplied business rules, stale reference data, incorrect joins): 53 incidents, 11% detected by observability tools, average business impact of $47,000 per incident. These are silent failures. The data looks correct. The numbers are wrong. Monitoring misses them because they violate business logic that the monitoring system doesn’t understand

The math is stark. Infrastructure observability catches low-impact problems and misses high-impact ones. The total undetected business impact from semantic failures was 19x larger than the total impact from structural failures. Yet observability investment, both in tooling and engineering time, was 90% directed at structural monitoring.

What does product thinking look like for data observability?

Product thinking for observability means treating each critical data asset as a product with defined consumers, expected behaviors, and success metrics that the observability system is designed to verify.

I rebuilt the observability layer using product principles. For each of the 12 critical data products (revenue metrics, customer health scores, pipeline attribution, etc.), I defined:

A product specification that documented: who consumes this data, what decisions they make with it, what “correct” looks like (with specific examples), and what failure looks like (with specific examples of past incidents). This specification was co-written with the data consumer, not the data engineer.

Business logic assertions that encoded the product specification as automated checks. For the pipeline attribution model: “Total attributed pipeline must equal total pipeline within 2%.” For the revenue metrics: “Monthly recurring revenue should change by less than 5% month-over-month unless a known event (acquisition, churn, pricing change) explains the variance.” For customer health scores: “No customer should have a health score below 0 or above 100, and the distribution should not shift by more than 1 standard deviation without a documented cause.”

Consumer feedback loops that gave data consumers a direct channel to report “this number doesn’t look right” with automatic routing to the data team and tracking of resolution time. This channel caught 6 semantic issues in the first quarter that no automated check detected.

How do you build business context into automated monitoring?

Building business context into monitoring requires encoding domain knowledge as testable assertions: not “is the data fresh?” but “does the revenue number match the sum of its components?” and “is the customer count consistent with the churn rate?”

• Cross-metric consistency: Define relationships between metrics that should hold. Revenue = units sold times average price. Customer count = previous count + new minus churned. When these identities break, something is wrong even if each individual metric passes its own quality checks
• Temporal plausibility: Define expected ranges for period-over-period changes. If monthly revenue typically varies by 3-8%, a 15% change should trigger investigation, not just when it’s negative but also when it’s positive (which could indicate a calculation error or double-counting)
• Reference data validation: Monitor the freshness and consistency of reference data (exchange rates, pricing tables, segment definitions) that upstream metrics depend on. I found that 4 of 7 semantic failures traced back to stale reference data that nobody was monitoring
• Consumer experience metrics: Track how consumers interact with data products. If query volume drops suddenly, consumers may have lost trust. If manual export volume increases, consumers may be building workarounds. These behavioral signals are leading indicators of data quality problems

Why is observability a product discipline, not an infrastructure discipline?

Observability is a product discipline because its purpose is to ensure data serves its consumers, and understanding whether data serves consumers requires understanding the consumers, their decisions, and their definition of “correct,” none of which are infrastructure concerns.

Infrastructure teams rightfully monitor uptime, latency, and resource utilization. These are necessary but insufficient for data observability. A pipeline can have 99.99% uptime and deliver wrong numbers with perfect reliability. The additional layer, the product layer, requires domain knowledge that infrastructure engineers typically don’t have and that data engineers often don’t prioritize.

The organizational implication is that observability ownership must be shared between the data engineering team (who understand the pipeline mechanics) and the data consumers (who understand what “correct” means). Neither group alone can build complete observability. The engineering team knows when the pipeline fails. The consumer knows when the data is wrong. Effective observability connects both signals into a unified view.

Data observability that monitors only infrastructure is like a security system that checks whether the doors are locked but not whether the right people are inside. The doors can be locked. The building can still be compromised. The pipeline can run. The data can still be wrong. Product thinking bridges this gap by asking the question that infrastructure cannot answer: “Does this data mean what its consumers think it means?” Until observability systems can address that question, they are monitoring the plumbing while the building floods from the roof.