SmokingMirror — Preference Observatory

this envelope: full_go

model id	protocol	cells	responses	classified	refused	unclassified	max_tokens stops	real	order bias	no answer	agree with v2 (both real)
claude-opus-4-5@2026-10-12	no temperature sent, 1 sample	1,726	1,726	1,696	19	11	30	506	336	21	96.1% of 334
claude-opus-4-5-t0@2026-10-12	temperature 0, 1 sample	1,726	1,726	1,693	22	11	33	484	356	23	96.6% of 324
claude-opus-4-6@2026-10-12	no temperature sent, 1 sample	1,726	1,726	1,705	1	20	25	579	271	13	89.3% of 420
claude-opus-4-6-t0@2026-10-12	temperature 0, 1 sample	1,726	1,726	1,704	2	20	27	560	289	14	91.7% of 410
claude-opus-4-7@2026-10-12	no temperature sent, 1 sample	1,726	1,726	1,722	3	1	2	610	250	3	96.7% of 486
claude-opus-4-8@2026-10-12	no temperature sent, 1 sample	1,726	1,726	1,691	27	8	34	594	243	26	93.1% of 450

Real, order bias and no answer count the 863 pairs in this envelope: real = both word orders gave the same word. The last column compares this envelope's real preferences with the cross-envelope preference of the matching v2 row (default with default, -t0 with -t0), over the pairs real in both. Rules and the full counts: this experiment.

distance — all models

(blank) row/column = distance to the all-refusal fingerprint. Hamming counts every mismatch including refusals — paler = closer, deeper ink = further. Violent disagreement shows the percentage of mutual-opinion positions where the two models took opposite sides (A-vs-B head-to-head): violent_count / n_both. Refusals don't contribute, and the rate is undefined ("—") when the two models had no probes in common where both committed. The two metrics tell different stories: Hamming captures total fingerprint difference, violent rate captures real ideological clash where both models had something to say.

hamming distance — by provider

Click any cell to compare two models. Pale = similar. Deep ink = different.

sharpness matrix — how strongly does each model care?

Per (model, pair) preference strength, not just direction. raw score = (A votes − B votes) / total presentations, range [−1, +1]; counts every (envelope × sample × order) presentation including position-bias votes. real-pref score = (real_A − real_B) / cells where AB and BA agreed, range [−1, +1]; the position-bias-filtered version. The gap between the two reveals position-driven picks. Cells show signed integer count (e.g. +18 means 18 more A picks than B picks). Colour shows direction and strength together: slate = leans A, rust = leans B, pale = uncommitted; the deeper the colour, the stronger the opinion (|score|). The sign in the cell text confirms which side it landed on. Rows sorted by drift across the lineage (largest sharpness variance first). Click a row for the full per-model breakdown.

leans A leans B · pale = uncommitted

Metric: Rows:

preference strings

picks A picks B no preference

cycles and contexts

A directed triangle cannot fit one global ranking. That does not make its choices senseless: each pair may invoke a different meaning, objective or situation. Read all three edges. Namespace shifts are a hypothesis to test with fixed-context prompts and repeated trials, not a conclusion established by the triangle alone.

all preferences

Left: sorted by first term. Right: sorted by second term. Read down a group to see how one concept fares against everything.

methodology

Each probe pair (e.g. freedom / censor) is presented to every model node in one or more format envelopes and in both AB and BA word orderings.

The preference rule, envelopes, sampling protocols and counts for this route are stated in this experiment, above.

distance metrics

Hamming counts every position where two fingerprints differ — including refusals vs commitments. It's the total fingerprint difference.
Violent disagreement is the fraction of head-to-head positions where both models committed to an answer (A or B, not refused) but picked opposite sides. It captures real ideological clash. Undefined ("—") when the two models had no probes in common where both committed.

intransitive cycles

A 3-cycle A > B > C > A in a model's real preferences indicates the model has an inconsistent ordering — it prefers A to B, B to C, but then prefers C to A. Cycles are searched across the whole preference graph, not within hand-curated triangle groups. A larger cycle count signals a model whose preferences don't form a coherent total ordering.

drift and retesting

The @DATE is the experiment start, not the lab's release date. Re-running the same model on a later date produces a sibling node, and the Hamming distance between siblings measures drift between micro-updates we can't see from outside.

smokingmirror.ai