The most dangerous data problem is not the one that crashes the pipeline. It’s the one that doesn’t — the silent drift that lets bad numbers flow through looking exactly like good ones.

Early on, our failures were loud. A source went down, a job threw an exception, somebody got paged. Those are easy; the system tells you it’s broken. The failures that cost us were the quiet ones: a feed that kept delivering, kept parsing, kept loading — but had started delivering subtly wrong values. Everything green, nothing alarming, and a slow rot in the foundation that nobody noticed until a downstream number looked strange weeks later.

So we stopped trusting “it ran” as a measure of health. A job completing tells you the plumbing works. It tells you nothing about whether what came through the plumbing is right. We needed a daily baseline — a sense of what normal looks like — so that abnormal could announce itself the moment it appeared, not weeks downstream.

Row anchors came first. Before trusting a day’s data, we check it against expectation: roughly how many rows should this source produce, given what it produced yesterday and last week? A feed that usually delivers a few thousand records and suddenly delivers a few hundred hasn’t necessarily errored — it may have quietly dropped half its coverage. The count is the cheapest possible smoke detector, and it catches a surprising amount.

Distribution sampling came next, because counts aren’t enough. A source can deliver exactly the right number of rows and have every one of them wrong. So we sample the shape of the data — does today’s distribution look like the distribution we’ve come to expect? When the center shifts, or the spread collapses, or a field that’s usually populated goes mostly empty, something changed upstream, and we’d rather know now than after we’ve built on it.

The governing principle is simple: drift means red. Not “drift means investigate when you have time.” A baseline is only useful if a deviation from it actually stops something. The discipline isn’t in measuring — measuring is easy. It’s in committing, in advance, that an unexplained shift halts the day’s data until a human understands why.

The deeper lesson is that monitoring is an act of humility. You are admitting that your sources will betray you, that your assumptions will rot, that the version of normal you encoded last quarter will quietly stop being normal. A health check isn’t infrastructure you build once and forget. It’s a standing confession that the ground can move under you — and a promise to notice when it does, before anything you care about is standing on it.