SmokingMirror — Preference Observatory

this envelope: python_typed

model id	protocol	cells	responses	classified	refused	unclassified	max_tokens stops	real	order bias	no answer
claude-opus-4-5@2026-05-29	default sampling, 5 samples	1,726	8,630	8,623	5	2	79	440	423	0
claude-opus-4-5-t0@2026-05-29	temperature 0, 1 sample	1,726	1,726	1,725	1	0	15	438	424	1
claude-opus-4-6@2026-05-29	default sampling, 5 samples	1,726	8,630	8,610	0	20	666	639	221	3
claude-opus-4-6-t0@2026-05-29	temperature 0, 1 sample	1,726	1,726	1,722	0	4	134	631	229	3
claude-opus-4-7@2026-05-29	default sampling, 5 samples	1,726	8,630	8,623	1	6	951	504	358	1
claude-opus-4-8@2026-05-29	default sampling, 5 samples	1,726	8,630	8,611	7	12	124	621	240	2

Real, order bias and no answer count the 863 pairs in this envelope: real = both word orders gave the same word. Rules and the full counts: this experiment.

distance — all models

(blank) row/column = distance to the all-refusal fingerprint. Hamming counts every mismatch including refusals — paler = closer, deeper ink = further. Violent disagreement shows the percentage of mutual-opinion positions where the two models took opposite sides (A-vs-B head-to-head): violent_count / n_both. Refusals don't contribute, and the rate is undefined ("—") when the two models had no probes in common where both committed. The two metrics tell different stories: Hamming captures total fingerprint difference, violent rate captures real ideological clash where both models had something to say.

hamming distance — by provider

Click any cell to compare two models. Pale = similar. Deep ink = different.

sharpness matrix — how strongly does each model care?

Per (model, pair) preference strength, not just direction. raw score = (A votes − B votes) / total presentations, range [−1, +1]; counts every (envelope × sample × order) presentation including position-bias votes. real-pref score = (real_A − real_B) / cells where AB and BA agreed, range [−1, +1]; the position-bias-filtered version. The gap between the two reveals position-driven picks. Cells show signed integer count (e.g. +18 means 18 more A picks than B picks). Colour shows direction and strength together: slate = leans A, rust = leans B, pale = uncommitted; the deeper the colour, the stronger the opinion (|score|). The sign in the cell text confirms which side it landed on. Rows sorted by drift across the lineage (largest sharpness variance first). Click a row for the full per-model breakdown.

leans A leans B · pale = uncommitted

Metric: Rows:

preference strings

picks A picks B no preference

cycles and contexts

A directed triangle cannot fit one global ranking. That does not make its choices senseless: each pair may invoke a different meaning, objective or situation. Read all three edges. Namespace shifts are a hypothesis to test with fixed-context prompts and repeated trials, not a conclusion established by the triangle alone.

all preferences

Left: sorted by first term. Right: sorted by second term. Read down a group to see how one concept fares against everything.

methodology

Each probe pair (e.g. freedom / censor) is presented to every model node in one or more format envelopes and in both AB and BA word orderings.

The preference rule, envelopes, sampling protocols and counts for this route are stated in this experiment, above.

distance metrics

Hamming counts every position where two fingerprints differ — including refusals vs commitments. It's the total fingerprint difference.
Violent disagreement is the fraction of head-to-head positions where both models committed to an answer (A or B, not refused) but picked opposite sides. It captures real ideological clash. Undefined ("—") when the two models had no probes in common where both committed.

intransitive cycles

A 3-cycle A > B > C > A in a model's real preferences indicates the model has an inconsistent ordering — it prefers A to B, B to C, but then prefers C to A. Cycles are searched across the whole preference graph, not within hand-curated triangle groups. A larger cycle count signals a model whose preferences don't form a coherent total ordering.

drift and retesting

The @DATE is the experiment start, not the lab's release date. Re-running the same model on a later date produces a sibling node, and the Hamming distance between siblings measures drift between micro-updates we can't see from outside.

smokingmirror.ai