Three Blind Readers and a State Machine

Three annotation experts work each historical case. None sees the others’ work. If they disagree, the disagreement goes into the record.

That sentence is easy to write and most of the engineering is in defending it. Blindness is not a policy you announce; it is a property that has to survive every code path that could leak one reader’s view to another. This post is about the paths we found.

Assignment has to be mutually exclusive, in the schema #

The obvious leak is the crude one: the same person gets the same case twice, agrees with themselves, and the system records two independent opinions. That is not blindness failing, it is independence failing, and it produces the most dangerous artifact in the system — false consensus.

The defence is that the assignment table treats the case-and-annotator pair as unique. Not a check in the dispatch code — a constraint on the row. The second assignment does not get skipped; it fails to insert. Dispatch is a machine action, and the property that makes it trustworthy is that it cannot produce the duplicate, not that it currently doesn’t.

The subtler version took longer to see. An annotator who has already worked a case in a previous round should not be assigned a related one where their earlier conclusion carries over. That is not duplicate assignment; it is correlated assignment, and it produces the same false consensus by a longer route. Correlated history is handled at dispatch as a priority ordering rather than a hard bar — a soft constraint, because a hard one starves the queue in a small pool. We would rather have a visible tradeoff than an invisible one.

Consensus is derived, never stored #

The state of a case moves through: awaiting readers, consensus, disputed, resolved. The design decision worth reporting is that this state is a view, not a column.

Storing it would mean a write path that sets it, and a write path that sets it is a write path that can set it wrong — or set it early, or fail halfway and leave a case marked “consensus” when only two readers have submitted. Deriving it means the state is always exactly what the underlying annotations imply, and there is no way to have a case whose recorded status disagrees with its own contents.

The cost is that you cannot correct the state directly. If a case shows as disputed and you believe it should not be, the only remedy is to change the annotations, which is the correct remedy. The absence of a manual override is the feature.

resolved is the one terminal state, reached when a lead reviewer adjudicates a dispute. It does not erase the disagreement; the dispute and its adjudication both stay in the record, because both are what the engine learns from.

Superseding, not editing #

When an annotation is replaced — a reader revisits and revises — the new version does not overwrite the old. It carries a link to what it supersedes, and both remain.

This is the same append-only discipline we apply everywhere, and here it has a specific payoff: the supersession chain is itself informative. An expert who revised after seeing new filings is a different signal from an expert who revised after an adjudication went against them. Overwriting would flatten both into “here is their opinion,” and the corpus would lose the fact that a mind was changed, and by what.

The instrument problem, and the two-layer fix #

Verifying any of this requires exercising it. So we run instrument accounts — synthetic annotators that walk the full flow so we can assert the machinery works on production, not on a staging copy that has drifted.

Instrument rows must never count as human judgment. The obvious defence is an output filter: exclude them from the consensus surface. We built that first, and it is not sufficient.

The reason is that an output filter guards the reading of state, not the writing of it. An instrument annotator submitting into a case still participates in the derivation upstream of the filter — it can push a case into consensus, and the filter downstream will faithfully hide the instrument row while reporting a consensus it caused. The exit gate cannot catch an upstream state change; by the time the filter runs, the damage is in the input.

So there are two layers. Instrument seats are excluded from the consensus derivation itself, and separately excluded from the replication count — the number that decides how many readers a case still needs. Either alone leaves a hole. Together they mean an instrument seat can traverse the full flow without any of its output reaching a place where it is mistaken for a person’s judgment.

The general form of this is worth keeping: a filter at the boundary protects the boundary, not the thing behind it. If synthetic input can change state before the filter, the filter is decoration.

What it is for #

None of this makes the annotations better. It makes them countable — it means that when we say three independent experts examined a case and two agreed while one dissented, each word in that sentence corresponds to something the system structurally could not have faked.

That matters because the disagreement is the part we publish. A dispute is only evidence of genuine difficulty if the readers really could not see each other. Everything above exists to make that claim true rather than aspirational.

All figures are system-level results current as of publication date; methodology parameters are intentionally omitted.