The Graph That Learned the Wrong Thing

We spent a stretch of engineering time building a graph neural network over corporate relationships, ran it properly, and archived it. We have written before about the discipline of abandoning a method as a matter of principle. This is the other kind of post: the autopsy, with the numbers.

The idea, and why it was reasonable #

Fraud is not evenly distributed across a market. It clusters — around particular auditors, particular groups of people who sit on each other’s boards, particular financing arrangements. If that is true, then the relationships between companies carry information that no single company’s financials contain, and a model over the relationship graph should see things a per-company model cannot.

We built two relation types. One connected companies sharing an audit relationship. The other connected companies through interlocking insiders. Both are constructed from filings; neither requires anything proprietary. The supervision came from an enforcement database — the standard choice, and the first thing we would do differently.

What the numbers said #

On a clean subgraph, held out across five split seeds, the model reached ROC-AUC 0.535 ± 0.125. That is a coin flip with a wide error bar; individual folds ranged from 0.39 to 0.73. Precision-recall was worse in a more informative way: PR-AUC 0.052 ± 0.056, a standard deviation larger than the mean. When the spread of your metric exceeds its value, you have not measured a weak effect. You have measured noise.

The honest reading is that the graph, as supervised, carried no usable signal. But the two follow-up tests are the part actually worth publishing.

Test one: we made it work, and that was the bad news #

We added edges connecting flagged entities directly to one another and re-ran. Performance jumped to ROC-AUC 0.884 ± 0.022, PR-AUC 0.810 ± 0.033.

A naive reading is that we found the right graph structure. The correct reading is visible in the second number: the variance collapsed. A model that has learned a genuine, subtle relational pattern does not become six times more consistent across random splits. A model that has found a structural giveaway does.

And that is what happened. An edge type that exists only between labelled entities is the label, re-encoded as topology. The network was not learning that companies connected through certain relationships are riskier. It was learning that this particular kind of edge only ever appears attached to a positive case. The jump from 0.535 to 0.884 measured leakage, not discriminative power.

This is the finding we would most want another team to take from this post. In graph learning the leak does not arrive as a column you can drop. It arrives as an edge, and edges look like structure, and structure looks like exactly the thing you built the model to find. The variance collapse was the tell — a sudden gain in both accuracy and stability should be treated as a symptom until proven otherwise.

Test two: the plain model beat the clever one #

We then ran the simplest possible control — four engine scores, no graph at all, a small feedforward network on identical data drawn through the same path. ROC-AUC 0.726 ± 0.045, PR-AUC 0.201 ± 0.059 against a prevalence of 0.051, roughly a four-fold lift.

Four ordinary features with no relational information outperformed the graph model by a wide margin, with a standard deviation that does not swallow the mean. The graph did not add to a weaker baseline; it underperformed a baseline we could have built in an afternoon.

We do not think this proves relationship structure is worthless in this domain. It proves that our graph, with that supervision, on that label set, was worse than nothing — and that we would not have known without running the boring control. The control cost almost no time and produced the single most decisive number in the exercise.

What we concluded, and what we kept #

The diagnosis is that the supervision was wrong, not that the idea was. Training on an enforcement database means the positives are cases that were caught, and being caught correlates with being large, visible, and long-lived enough for an investigation to conclude. A graph model with size and survival information flowing through its topology will happily learn that, and report it as insight. Our earlier concerns about that label set were about what it teaches; this was the same problem showing up as a measurable confound.

So the model is archived rather than deleted, with the condition for revisiting written down: a supervision source generated by our own outcomes rather than by someone else’s enforcement history. Until that exists, rebuilding the same thing with a better architecture would be answering the wrong question more precisely.

The data stayed. Eleven and a half million insider transaction records now feed ordinary scanner signals — unusual selling relative to a company’s own baseline, ownership concentration changes — where they earn their keep without any graph at all. That was not a consolation prize we invented after the fact. It was the only part of the exercise where the data was being asked a question it could actually answer.

All figures are system-level results current as of publication date; methodology parameters are intentionally omitted.