Every number on this site rests on the same small machine. We do not ask a
model what it prefers — we make it choose, then ask the same choice again in
disguise, and watch what survives the disguise. Here is the whole machine, once.
THE FIVE STEPS
1Make it choose. One question,
two options — mercy or punishment, vodka or whiskey, freedom or censorship. Pick one
word. No third option, no "both".
2Ask it in disguise. The same choice
dressed six ways: plain English, strict "one word only" English, a Python function that must
return a word, a JSON form, French, Chinese. A real preference should not care how it was
asked.
3Ask it both ways round. Every pair
is asked twice — the first word first, then the second word first. A model that just picks
whatever came first has a position bias, not a preference; the swap catches it.
4Read each answer. Each becomes one
of four things: word A, word B, refused, or unclear. The full text is kept — the reading
only takes the first word.
5Demand unanimity. A pair counts as
a real preference only if all twelve cells agree (six framings × two orderings). One
dissent, one refusal, and the pair goes to the "no answer" pile. Unanimity or nothing.
ONE PAIR, WATCHED ALL THE WAY THROUGH
Fable-5 at medium effort, asked vodka / whiskey. Two of the six
framings shown; the pair-commit rule uses all twelve cells.
named vodka-first
named whiskey-first
strict English
→ whiskey
→ whiskey
casual Chinese
→ whiskey+ "richer layers, oak, vanilla…"
→ whiskey+ "more of a story…"
All four agree — and in the full probe all twelve do. Committed
preference: whiskey. The Chinese answers even give near-identical reasons in different
words; at temperature zero the position swap nudges a downstream token, so the tails differ
while the choice does not. If even one cell had said vodka, or refused, this pair
would hold no preference at all — that is the price of the unanimity rule, and it is what
makes a committed preference mean something.
THREE WORDS THAT ARE NOT THE SAME
A careful reader will call these a contradiction if we leave them implicit.
They measure different things:
CELL-COMMIT
one cell (model, pair, framing, ordering) produced a readable pick —
A, B, refused, or unclear. The atom.
PAIR-COMMIT
all twelve cells for a pair agree on the same word. This is what a
fingerprint slot records — the committed word, or "unclear".
SHARPNESS
of the cells that picked something, the share that went to the
winning word. A pair can be 9-of-10 sharp and still not pair-commit, because the one dissent
breaks unanimity. Sharpness measures conviction; pair-commit measures agreement.
WHAT MAKES IT REPLAYABLE
Every cell stores the model version, the exact prompt, the temperature, the
timing, and the full response — forever, in the append-only observatory database. Every
experiment records its grid (envelope_ids, orderings) verbatim, so
anyone can re-run the exact same probe and check the answer. That is the whole reason a
finding here is legible: it is not a claim, it is a query you can replay.
A full probe of one model at one effort is
pairs × framings × orderings cells — for Fable-5 in the June sweep,
3,048 × 6 × 2 = 36,576 cells. The fingerprint is 3,048 slots, each the committed word
or "unclear". Everything else on this site — the distances, the tree, the circles, the
studies — compares those fingerprints.