If you ask most teams how accurate their system is, they will quote you a metric. A precision figure, a recall figure, an area under some curve. We have those numbers too, and we watch them. But we’ve come to believe that for a system whose job is to judge whether a company has a problem, accuracy is not really a number at all. It is the cumulative result of a set of disciplines, each of which is individually unglamorous and collectively decisive. A single headline metric, divorced from those disciplines, is one of the easier things in this field to fake — usually without meaning to.

This is a place to draw several threads together. Everything we’ve written about turns out to be a different face of the same commitment.

A metric can lie in a dozen quiet ways #

We’ve spent a great deal of effort cataloguing the ways a system can post a good number and be wrong underneath.

It can be wrong because it learned from hindsight — trained on past situations described with information that only became available later, so that its apparent skill is really just memorized answers. We called that the discipline of time, and the defense is to snapshot every feature as it stood at the moment of decision and never backfill the future into the past.

It can be wrong because it learned from scale — flagging large, established companies because they have more of every countable thing, while a metric trained on labels that correlate with size cheerfully reports success. We called that the scale trap, and the defense is to measure deviation from an entity’s own baseline rather than absolute or cumulative quantity.

It can be wrong because it learned from an answer key — a structure or a feature that quietly encodes the labels it’s supposed to predict. We saw that in a relationship graph whose edges leaked the very thing they were meant to discover, and the defense was to insist that a complex method beat a simple one on a fair, confound-matched comparison before earning its place.

It can be wrong because it learned from a counterfeit — reasoning generated by a general model imitating expertise it doesn’t have, then fed back as if it were the real thing. We called that the distillation trap, and the defense was to enforce, at the data layer, that judgment used for training comes only from genuine forensic experts.

And it can be wrong because the verdict was hand-tuned — a formula of weights set by intuition rather than learned from outcomes, absorbing its builders’ priors and the biases of its labels. The defense there was to keep the verdict out of the engine entirely, let the engine produce only leads, and confer judgment from observed outcomes over time.

Every one of these failures produces a system that looks accurate. None of them is caught by staring at the headline number. They are caught only by the disciplines — by refusing to let yourself off the hook with a good metric.

The unifying instinct #

Step back far enough and all of these are the same instinct wearing different clothes. In every case, the discipline says: do not let an easy proxy stand in for the hard truth.

Statistical deviance is an easy proxy for wrongdoing — so we look for internal contradiction instead. Absolute quantity is an easy proxy for significance — so we measure relative to a baseline. A good metric is an easy proxy for a working system — so we audit the disciplines underneath it. Fluent reasoning is an easy proxy for expert judgment — so we demand the real expert. A hand-tuned formula is an easy proxy for learned judgment — so we wait for outcomes. The temptation is always the same: a shortcut that feels like the thing but isn’t. The discipline is always the same: refuse the shortcut, even when it costs you speed, even when the proxy would have posted a better number this quarter.

Why we’d rather be slow than fooled #

There is a real price to all of this. Insisting on point-in-time data means some data can’t be used. Insisting on relative baselines means more work than a threshold. Insisting on expert annotation means our reasoning corpus grows only as fast as scarce human judgment allows. Insisting that the verdict be learned from outcomes means the most valuable part of the system matures slowly, on the timescale of real-world results rather than a training run. Each discipline is a brake.

We accept every one of those brakes, because the alternative is the thing we exist to expose. We are in the business of catching companies whose numbers tell a story their reality doesn’t support. A system that posted impressive accuracy while quietly measuring scale, or memorizing hindsight, or learning from a counterfeit, would be doing the exact same thing we accuse those companies of — presenting a confident surface over a hollow center. We would rather be slow and honest than fast and fooled, because a forensic research system that fools itself has no business asking anyone to trust it about anyone else.

Accuracy, for us, is not a number we report. It is the standing refusal to believe our own good numbers until we’ve ruled out every boring reason they might be lying. That refusal is the product. Everything else is just engineering in service of it.