Most engineering writing is about things that worked. This one is about deciding that something didn’t, and treating that decision as a real result rather than a failure to be hidden. We think the ability to abandon a method cleanly — to prove to yourself that an idea you wanted to be true is not — is one of the most underrated disciplines in applied research, and one of the most important to a system whose credibility depends on not fooling itself.

The pull toward a good story #

A reasonable idea: companies are connected to each other through shared relationships, and those connections might carry signal about risk. Build a graph, learn over its structure, let the model find patterns in how entities relate. It is an appealing idea because it is almost certainly true in general — relationships do carry information — and because graph methods are powerful and satisfying to build.

We pursued it. And the appeal of the idea turned out to be exactly the danger, because when you want something to work, you will find ways to make the numbers say it works.

Two ways a graph lied to us #

The first attempt connected companies through a shared attribute and trained a model to predict known problem cases from the resulting structure. The performance looked promising in one configuration. But when we examined why, we found the model was succeeding by exploiting the way we had built the edges: connections that existed partly because of the labels themselves. The structure wasn’t revealing fraud; it was quietly encoding the answer key. When we removed that contamination, the signal collapsed to roughly what you’d get from chance. A simpler model using no graph structure at all — just per-company features — did meaningfully better. The signal, such as it was, lived in the features, not in the structure.

That was a clean enough negative. But the more instructive failure came on the second attempt, because it was subtler and harder to dismiss.

This time the graph connected individuals through the companies whose boards they served on, and the edges were built entirely from real filings — genuinely independent of any label, with no answer key smuggled in. By the standard we’d learned from the first failure, this graph was clean. And the model still showed a difference between problem cases and others. The connection counts for problem companies ran higher.

We could have stopped there and declared a signal. Instead we asked where the difference came from. The companies in our problem set were systematically older and larger. Older companies have filed for longer and accumulated more relationships; larger companies have directors who sit on more boards. Normalize for that, and the apparent signal flattened out — the per-period rate of connection was essentially the same. The elevated structure wasn’t fraud. It was age and size wearing a costume. We tried every reasonable mitigation: shorter time windows, different relationship filters, stratifying by company size. The contamination narrowed and refused to vanish.

So we had a graph whose edges were honestly constructed, that produced a real numerical difference, and whose difference was still not a fraud signal — because the difference was driven by a confound that correlated with our labels rather than by anything about misconduct. This is a more dangerous situation than the first one, because there is no obvious construction error to point to. The edges are clean. The metric is real. And the conclusion is still a mirage.

The discipline of walking away #

The decision in both cases was the same: archive the method. Not shelve it ambiguously, not keep it around “just in case” with a mental asterisk — archive it, with the evidence written down, as a settled negative result.

We hold a few principles about how to do this honestly. First, a number that looks like a signal is a hypothesis, not a conclusion; the job is to find the most boring explanation that accounts for it before accepting the exciting one. Second, the most common boring explanation in this domain is scale or survivorship, and you should suspect it by default whenever your labels correlate with the size or age of the entities involved. Third, the right benchmark for “does this complex method help?” is always a simple model that doesn’t use it — if the complicated thing can’t beat the simple thing on a fair, confound-matched comparison, the complicated thing is not earning its place.

And there is a constructive flip side to the negation, which is why this is not a story about defeat. Establishing that a relationship structure does not carry an independent fraud signal when supervised against enforcement labels told us something precise about where the real supervision has to come from. The honest training signal for a method like this is not a borrowed set of historical enforcement labels — which drag their size-and-survivorship baggage along with them — but the outcomes our own system observes over time. Knowing that is a result. It redirects effort from a dead end toward the place where the signal might actually live.

Walking away from a method you wanted is not a failure of the research. It is the research. The systems that earn trust are the ones willing to disprove their own best ideas before the world does it for them.