There is a quiet way to lie to yourself when building a system that learns from history, and almost everyone does it at least once. You take an event from the past — a company that was later found to have problems — and you describe its situation using data you have now. You enrich the old case with current figures, restated financials, updated relationship data, the latest everything. The model trains on this enriched picture and performs beautifully. And it is worthless, because at the moment that mattered, none of that information existed.

We call the defense against this point-in-time discipline, and it is one of the hardest rules to hold, because violating it never feels like cheating. It feels like being thorough.

The temptation #

Imagine you are assembling a training set of historical situations. For each one, you want the richest possible feature set. The data is right there in your database — the current financials, the current ownership structure, the current network of relationships. Why not use it?

Because the model you are trying to build will, in production, have to make a judgment about a company as it appears today, with only the information available today. If you trained it on situations described with information that only became available later, you have trained it on a world it will never inhabit. It learned to recognize patterns that, at decision time, simply are not visible. Its reported accuracy is an artifact of hindsight. In the field, that accuracy evaporates, and you don’t find out until it’s too late to matter.

The treachery is that the leakage is invisible in your metrics. Hindsight-enriched models look excellent on held-out data, because the held-out data is contaminated in exactly the same way. The error is not in the evaluation; it is baked into the features themselves. You can have a perfectly clean train/test split and still be deceiving yourself completely.

What the discipline requires #

Point-in-time discipline says: every feature attached to a past situation must be snapshotted as it stood at that moment, and never backfilled with a later value. If a financial figure was subsequently restated, the model must see the figure as originally reported, because that is what a decision-maker would have seen. If a relationship or an ownership stake didn’t yet exist, it cannot appear in the feature set, no matter how relevant it later proved to be.

This sounds obvious stated plainly. It is brutally inconvenient in practice. Many data sources store only the latest version of a value — the current financials, the restated numbers, the present-day structure — with no easy way to reconstruct what was knowable on a given past date. A convenience store of “latest values” is, for point-in-time training, actively dangerous: it will silently hand you the future every time you ask about the past. Honoring the discipline often means doing real work to recover historical vintages, or accepting that certain data simply cannot be used for training because its point-in-time version is irrecoverable.

We extend the same rule to relationships and networks, not just financial line items. When we consider whether to use a structure built from connections between entities, we ask not just whether the structure is informative but whether it can be reconstructed as it stood at the relevant moment. A relationship graph assembled from the full sweep of history, projected backward onto a past decision point, imports knowledge that didn’t exist then — and, as we’ve discussed elsewhere, it also tends to import the accumulated bulk of older, larger entities, which is its own kind of contamination. Time and scale failures often travel together.

Why it reaches all the way to the annotation layer #

Point-in-time discipline isn’t only a rule about training data. It shapes how we present a case to a human expert who is annotating it. When an expert examines a flagged situation, we show them the features exactly as they stood when the flag was raised — a frozen, read-only snapshot — and we make it structurally impossible for the annotation process to backfill or alter those features with anything learned later. The expert is judging the situation as it actually appeared, not as it appears in retrospect. Their reasoning, and the training signal it eventually becomes, inherits the same temporal honesty as everything upstream of it.

The underlying principle is simple to state and unforgiving to follow: a judgment is only meaningful in the context of the information available when it was made. A system that learns from the past must inhabit the past as it actually was — uncertain, incomplete, missing all the things that only later became clear. Anything else is just an elaborate way of memorizing the answers and calling it insight.