We Publish the Disagreement

Three annotation experts. None sees the others’ work. If they disagree — the disagreement goes into the record.

The first two sentences describe a process. The third is a commercial decision, and it is the one that gets pushback: why would a research firm publish the cases where its own experts could not agree?

The case for hiding it #

The argument for suppression is strong enough that we should state it properly rather than knock down a weak version.

A research product is bought for its conclusions. A subscriber who receives “two of our experts read this as a genuine problem, one read it as an aggressive but defensible treatment” has received something harder to act on than a clean verdict. Publishing dissent looks like publishing doubt, and doubt is not obviously what anyone is paying for. Every incentive says: adjudicate internally, publish the resolution, keep the argument in-house. That is what most of this industry does, and it is not dishonest — the resolution genuinely is the firm’s considered view.

Why we do the opposite #

A unanimous-only record is a record that has been filtered, and the filter is invisible.

If we published only the cases where three readers agreed, our published set would be systematically composed of the easy ones. Not because anyone suppressed a hard case — because hard cases produce disagreement, and disagreement was the exclusion criterion. A reader looking at our output would see a firm that is right a lot, and would have no way to know they were looking at a sample selected for tractability. The confidence they formed would be a property of our editorial process, not of our judgment.

That is the same defect we spend our engineering effort eliminating everywhere else. We keep retracted cases in the training corpus, labelled as errors, for the identical reason: a record that only contains successes teaches nothing about the boundary. Publishing only consensus would be that failure at the customer-facing layer.

Dissent is information the reader cannot reconstruct. When one experienced reader looks at a set of filings and reaches a different conclusion from two others, that fact is a measurement of how hard the case is — and it is a measurement nobody outside can make. A subscriber deciding how much weight to put on a flag is far better served by “three read it, one dissented on these grounds, the lead reviewer resolved it this way” than by a confidence score, which compresses exactly this into a number that hides where it came from.

It is the only version we can defend. When a flag is challenged, a published record that already contains the strongest internal counter-argument is in a different position from one that does not. If the objection raised is the one our own dissenter made, we can point to where we recorded it and how it was resolved. If we suppressed it and it surfaces later, every other case we published becomes a question about what else was left out.

What actually gets published #

The dispute and its adjudication, both. Which reader dissented and on what grounds; the lead reviewer’s resolution and reasoning; the credentials of everyone involved. A resolved case does not become a clean case — it stays visibly a case where the readers split and someone had to decide.

Two things we do not do. We do not publish a disagreement rate as a headline metric; it would immediately become a number to manage, and the way to manage it downward is to stop assigning hard cases. And we do not treat consensus as a quality signal in either direction — a unanimous case is not more reliable than a resolved one, it is just less contested, which usually means it was more obvious.

The part that cost us something #

We publish disagreements between experts who are, in most cases, more experienced in their specialty than anyone reviewing them. That means we regularly publish a record in which a very good accountant was, in the lead reviewer’s judgment, wrong.

Handling that honestly required a compensation rule most people find counterintuitive: the dissenting reader is paid the same. Their work was to render independent judgment, and they did. If dissent carried a financial cost, the blind-reading structure would survive on paper while quietly collapsing in practice — readers would learn to guess the consensus rather than form a view, and we would have built an elaborate mechanism for producing agreement.

Paying for dissent and paying for consensus at the same rate is the price of getting real independence, and it is cheaper than the alternative: a corpus full of readers who learned what answer we wanted.

All figures are system-level results current as of publication date; methodology parameters are intentionally omitted.