smokingmirror.ai

preference observatory · 2026-05-29 · 863 probe pairs · 2 orderings · 6 model nodes  ·   ·   ·   · 

this experiment

Claude Opus 4.5, 4.6, 4.7 and 4.8 on the frozen Smoking Mirror v2 grid: 863 word pairs × 6 envelopes × 2 word orders = 10,356 cells per row, run 2026-05-29. Opus 4.5 and 4.6 were also run at temperature 0. The -t0 rows sent temperature 0 and took one sample per cell; the other rows took five samples per cell at default sampling (Opus 4.7 and 4.8 reject a temperature parameter). The stored temperature column reads 0 for every response of every row, including the five-sample rows, so the database alone does not confirm what was sent. The token cap was 8 (32 for json_schema), so a one-word answer followed by any explanation ends at max_tokens; that is not a failure.

model idsetprotocolcellsresponsesclassifiedrefusedunclassifiedmax_tokens stopsreal prefs / pairs
claude-opus-4-5@2026-05-29v2default sampling, 5 samples10,35651,78051,5512520425,981445 / 863
claude-opus-4-5-t0@2026-05-29v2temperature 0, 1 sample10,35610,35610,3085435,196447 / 863
claude-opus-4-6@2026-05-29v2default sampling, 5 samples10,35651,78051,5732817926,567553 / 863
claude-opus-4-6-t0@2026-05-29v2temperature 0, 1 sample10,35610,35610,3157345,315545 / 863
claude-opus-4-7@2026-05-29v2default sampling, 5 samples10,35651,78051,5952216327,590608 / 863
claude-opus-4-8@2026-05-29v2default sampling, 5 samples10,35651,78051,39518719826,245572 / 863

Per-envelope tables (v2 envelopes): chinese_casual · english_casual · english_strict · french_casual · json_schema · python_typed. Each page lists every pair's preference in that envelope, with its counts.

Counts. A cell is one pair × envelope × word order. Responses are the stored API responses (cells × samples). Classified responses name one of the two options; refused are empty or carry a refusal marker; unclassified matched neither option. max_tokens stops counts responses the API ended at the token cap.

Classifier. The deterministic pass that wrote the stored v1 labels (tier 1 exact, then tier 2 fuzzy at confidence ≥ 0.85), run here on every response of both sets. Tier 2 has rules only for the casual envelopes, so code envelopes are tier 1 alone. On the 227,011 v2 responses that carry a stored v1 label it reproduces 227,011 of them.

Preferences. A cell's answer is the majority of its classified samples (a tie is no answer). A preference is real in an envelope only when both word orders give the same word; both answered but different is order bias. A pair's cross-envelope preference is real when one word wins in more than half of the set's envelopes (4 of 6 for v2, 8 of 14 for the coding set).

Against the stored tables. The June /v2/ observatory read these rows from the stored per-envelope and cross-envelope preference tables. This derivation reproduces them exactly for claude-opus-4-5-t0@2026-05-29 (5,178 envelope cells, 863 pairs), claude-opus-4-6-t0@2026-05-29 (5,178 envelope cells, 863 pairs). For the five-sample rows the stored tables followed a sample rule they do not record, and they differ from the majority-of-samples rule used here on: claude-opus-4-5@2026-05-29 403 of 5,178 envelope cells, 57 of 863 pairs; claude-opus-4-6@2026-05-29 190 of 5,178 envelope cells, 39 of 863 pairs; claude-opus-4-7@2026-05-29 306 of 5,178 envelope cells, 40 of 863 pairs; claude-opus-4-8@2026-05-29 485 of 5,178 envelope cells, 75 of 863 pairs.

The same rows beside the coding set: /opus-45-48-coding/. Every count here comes from the export committed beside the generator (web/smokingmirror/site5/opus45_48/).

distance — all models

(blank) row/column = distance to the all-refusal fingerprint. Hamming counts every mismatch including refusals — paler = closer, deeper ink = further. Violent disagreement shows the percentage of mutual-opinion positions where the two models took opposite sides (A-vs-B head-to-head): violent_count / n_both. Refusals don't contribute, and the rate is undefined ("—") when the two models had no probes in common where both committed. The two metrics tell different stories: Hamming captures total fingerprint difference, violent rate captures real ideological clash where both models had something to say.

hamming distance — by provider

Click any cell to compare two models. Pale = similar. Deep ink = different.

sharpness matrix — how strongly does each model care?

Per (model, pair) preference strength, not just direction. raw score = (A votes − B votes) / total presentations, range [−1, +1]; counts every (envelope × sample × order) presentation including position-bias votes. real-pref score = (real_A − real_B) / cells where AB and BA agreed, range [−1, +1]; the position-bias-filtered version. The gap between the two reveals position-driven picks. Cells show signed integer count (e.g. +18 means 18 more A picks than B picks). Colour shows direction and strength together: slate = leans A, rust = leans B, pale = uncommitted; the deeper the colour, the stronger the opinion (|score|). The sign in the cell text confirms which side it landed on. Rows sorted by drift across the lineage (largest sharpness variance first). Click a row for the full per-model breakdown.
leans A leans B · pale = uncommitted
  

preference strings

picks A picks B no preference

cycles and contexts

A directed triangle cannot fit one global ranking. That does not make its choices senseless: each pair may invoke a different meaning, objective or situation. Read all three edges. Namespace shifts are a hypothesis to test with fixed-context prompts and repeated trials, not a conclusion established by the triangle alone.

all preferences

Left: sorted by first term. Right: sorted by second term. Read down a group to see how one concept fares against everything.

methodology

Each probe pair (e.g. freedom / censor) is presented to every model node in one or more format envelopes and in both AB and BA word orderings.

The preference rule, envelopes, sampling protocols and counts for this route are stated in this experiment, above.

distance metrics

Hamming counts every position where two fingerprints differ — including refusals vs commitments. It's the total fingerprint difference.
Violent disagreement is the fraction of head-to-head positions where both models committed to an answer (A or B, not refused) but picked opposite sides. It captures real ideological clash. Undefined ("—") when the two models had no probes in common where both committed.

intransitive cycles

A 3-cycle A > B > C > A in a model's real preferences indicates the model has an inconsistent ordering — it prefers A to B, B to C, but then prefers C to A. Cycles are searched across the whole preference graph, not within hand-curated triangle groups. A larger cycle count signals a model whose preferences don't form a coherent total ordering.

drift and retesting

The @DATE is the experiment start, not the lab's release date. Re-running the same model on a later date produces a sibling node, and the Hamming distance between siblings measures drift between micro-updates we can't see from outside.