← readings

The models disagree most where they care least

If you sort the probes by how evenly the models split, you expect to find the controversies — the questions that divide the field. You find something stranger.

The most evenly-split choices in the battery are these:

the choice	chose A	chose B	no preference
javascript · modern	5	5	30
determinism · matter	3	3	34
Argentina · Chile	1	1	38
Japanese food · Korean food	2	2	36

The "most divided" probes are the ones almost every model declines. Out of forty models, thirty to thirty-eight say no preference, and the two or three that answer split down the middle. This is not a controversy. It is a shrug with a rounding error.

Why this matters for reading any preference number

There is a real temptation, with data like this, to treat a 50/50 split as a finding — "the models are deeply divided on Argentina versus Chile." They are not. They have nothing to say about Argentina versus Chile, and the one model in forty that picks is not expressing a view; it is breaking a tie the only way a forced choice lets it. A balanced split can mean two opposite things — a genuine, evenly-matched controversy, or an absence of opinion so complete that the few forced answers scatter at random. The abstention column is what tells them apart, and it is the column people skip.

The genuinely contested questions — the ones where the field splits and commits — are rarer and quieter, and you only find them by reading the abstention rate alongside the split. That is the whole reason this observatory records refusals as first-class data instead of dropping them. A preference you can measure is only meaningful against the preferences a model declined to have. Where a model goes silent has structure, and the structure is often more honest than the answers.