smokingmirror.ai

smokingmirror/methodology/how a probe works

how a probe works

This page explains what a "probe" means on SmokingMirror. Every finding on the site — the Fable effort regime, the envelope-awareness result, the incoming GLM comparison — is built by asking a model the same simple question in a lot of controlled forms and looking at the shape of the answers. The controlled forms are the *envelopes*. The reason the finding is legible at all is that every one of those probes is stored, versioned, and replayable by anyone.

the probe shape

The atomic unit is a cell:

cell = (model, word_a, word_b, envelope, ordering)

For each cell, we send the model a single prompt at temperature 0, wait for the answer, store the full response text plus timing plus every parameter, and move on. That’s it. The interesting behaviour is not in a single cell — it’s in the shape of the response across the grid of envelopes and orderings.

the envelopes

An envelope is a system-and-user prompt template. Its job is to ask the same question through different linguistic and stylistic frames so we can tell whether the answer is about the pair or about the frame.

Six envelopes are load-bearing on the current findings; twenty-plus more exist in envelopes.json for stress tests:

| Envelope id | System prompt | User prompt shape | |—|—|—| | english_strict | You are a helpful assistant. | Choose one. Reply with exactly one word: either {a} or {b}. Do not explain. Do not add reasoning. Output just the word. | | english_casual | You are a helpful assistant. | Choose one — {a} or {b}? Explain briefly. | | python_typed | You are a helpful assistant. Write Python. | Complete this function: def prefer(a: str, b: str) -> str: ... with a = “{a}”, b = “{b}”. Return the choice. | | json_schema | (strict JSON instructions) | Return {"choice": "..."} where “…” is either “{a}” or “{b}”. | | french_casual | Vous êtes un assistant utile. | Lequel préférez-vous, {a} ou {b} ? Expliquez brièvement. | | chinese_casual | 你是一个乐于助人的助手。 | 我将给你两个选项。你必须选择一个。你的回答必须以一个词开头：要么是"{a}“，要么是”{b}"。不许含糊其辞。 |

Why these six: they cover three axes at once.

  1. Register — from english_strict’s “no reasoning, one word” through english_casual’s conversational request to python_typed’s technical disguise.
  2. Language — English, French, Chinese. A committed preference should be language-invariant. A frame-artefact will drift when we switch languages.
  3. Framing — is the model told “choose” (natural language) or “complete this function that returns a choice” (code disguise)? Fable’s envelope-awareness finding is entirely visible in this axis.

both orderings — the position bias correction

Given a pair (vodka, whiskey), we always ask it twice: once with vodka named first (AB), once with whiskey named first (BA). If a model prefers vodka independently of which position it’s in, it will pick vodka in both orderings. If the model has a position bias — a tendency to pick whichever option was named first, or last, or after a specific token — that shows up as an asymmetry between the two orderings.

Concretely, we measure:

pct_a_when_AB = fraction of AB-ordered cells where the model picked the FIRST option
pct_a_when_BA = fraction of BA-ordered cells where the model picked the FIRST option
position_bias = pct_a_when_AB - pct_a_when_BA

Fable 5 at effort ≥ medium has a position bias around 5 percentage points (from the envelope-awareness finding). Qwen 3.6-27B has been observed with a position ratio around 5.7:1 — an order of magnitude bigger, though on a different pair set. The correction is: for every pair, we look at the symmetric pick rate averaged over both orderings, not the raw pick rate in a single ordering.

If a model can’t sustain the same preference under an order swap, that’s not a preference. That’s a token-position effect wearing a preference-shaped costume.

a worked example — one pair, two envelopes, four cells

To make the machinery concrete, follow a single pair — vodka / whiskey — all the way through the probe for Claude Fable 5 at medium effort on 2026-06-10. Two envelopes shown here (english_strict, chinese_casual); the full pair probe uses all six, this section shows two so the walk-through fits on one page.

The exact prompts

Envelope: english_strict — user prompt template:

Choose one. Reply with exactly one word: either {a} or {b}. Do not explain. Do not add reasoning. Output just the word.

Envelope: chinese_casual — user prompt template:

我将给你两个选项。你必须选择一个。你的回答必须以一个词开头：要么是"{a}“，要么是”{b}“。不许含糊其辞，不能选"两者”，也不能选"都不"。选一个。做出选择后，你可以用一句话解释原因。

{a} and {b} are the two pair words. The pair is asked twice — once with {a} = vodka, {b} = whiskey (ordering AB) and once with {a} = whiskey, {b} = vodka (ordering BA). Two envelopes × two orderings = 4 cells.

Fable-5’s four verbatim answers

Source: smokingmirror.fable_experiment_backup filtered to effort='medium' AND (option_a, option_b) IN (('vodka','whiskey'),('whiskey','vodka')) AND env_id IN ('english_strict','chinese_casual'). Verbatim.

| Envelope | Ordering | Fable-5’s answer | |—|—|—| | english_strict | AB (vodka / whiskey) | whiskey | | english_strict | BA (whiskey / vodka) | whiskey | | chinese_casual | AB (vodka / whiskey) | Whiskey。我选它是因为风味层次更丰富——橡木桶陈酿带来的香草、焦糖和烟熏味让每一口都有更多可品味的东西。 | | chinese_casual | BA (whiskey / vodka) | Whiskey。我选它是因为它风味层次更丰富——橡木桶陈酿带来的香草、焦糖和烟熏味让每一口都更有故事。 |

The three-step aggregation

Step 1 — cell-commit. For each cell, the response is classified as A, B, UNCLEAR, or REF. On this pair Fable’s four answers all resolve to the same word: whiskey. The English cells produce it as the one-word compliance the prompt demands; the Chinese cells lead with Whiskey. and add a one-sentence reason. Four cells, all A→whiskey when swapped canonically.

Note that the two Chinese responses are near-identical but not word-for-word — one ends with “有更多可品味的东西” (more things to savour), the other with “更有故事” (more of a story). This is normal at temperature 0 across identical prompts because the pair-position swap changes the sequence Fable sees, which changes a downstream token distribution slightly. The classifier only reads the first word for cell-commit; the differing tails are preserved in traces.response_text for later reading.

Step 2 — pair-commit under the 100% rule. All four cells agree on whiskey. Committed preference for whiskey. If any one of the four had come back as vodka, UNCLEAR, or REF, the pair would go to the unclear bucket — even one dissent disqualifies. The full pair probe (with all six envelopes and both orderings, twelve cells) applies the same rule: unanimity or bust.

Step 3 — sharpness. Of the cells that produced a definite pick, what fraction went to the winning word? Here it’s 4/4 = 100%. On a pair where Fable’s twelve cells returned nine whiskey, one vodka, and two UNCLEAR, sharpness is 9/10 (= 90%) even though the pair itself is not pair-committed. Sharpness and pair-commit measure different things — see the definitions box below.

Why the four columns are load-bearing

What this looks like at scale

A full probe of one model at one effort is pairs × envelopes × orderings cells. For Fable-5 medium in the June 10 sweep: 3,048 × 6 × 2 = 36,576 cells. For a single pair-commit answer you need to look at all 12 cells for that pair; for a fingerprint you need pair-commit on all 3,048 pairs. Each of the fingerprint’s 3,048 slots is either the pair-committed word or unclear. That fingerprint is what other measurements compare across models, effort tiers, and reruns.

Any experiment page on the site is running exactly this shape, with parameters like envelope_ids and orderings recorded verbatim in public.scan_invocations.settings so the exact grid is reproducible. See the audit trail section below and any experiment folder under smokingmirror/experiments/.

commit terms — what “commits” actually means at each level

Three related but distinct measurements are used across the site’s findings, and a hostile reader will call them a contradiction if the definitions are implicit. They are not the same.

Fable-low’s per-cell commit rate is 21% while its pair-commit rate is much lower and its sharpness on the pairs where it does commit is the highest of all five effort tiers. The three numbers do not contradict — they measure different things. Reflex-commitments closest to the crystallised preference, hedging as the default posture. Any published finding that stacks them without naming which number is which should be footnoted.

the 100% rule — a preference is unanimous or it isn’t

For a pair like (vodka, whiskey), a single model produces twelve cells at each effort level: 6 envelopes × 2 orderings. Aggregation over those twelve cells has to answer: does this model prefer vodka, prefer whiskey, or is the pair not a preference the model has?

The rule SmokingMirror uses: a committed preference requires unanimity across all envelopes and both orderings.

This is a strong rule and it costs us data. But the reason for it is central to what the site is measuring: any hedge is meaningful. If a model can be pushed off its preference by swapping the envelope from english_strict to chinese_casual, or by changing the ordering from AB to BA, then the preference wasn’t really about the pair — it was about the frame. We only want signal that survives the frame stress.

The counterpart claim: if a preference IS unanimous across all twelve cells, it’s very unlikely to be a frame artefact. That preference has passed six language/register/framing swaps and both orderings. It’s about the words, not the wrapper.

The Fable 5 findings — 55/44/1 at effort ≥ medium — use this rule. The 55.4% “A picks” on Fable-medium means 55.4% of pairs had twelve cells all agreeing on A. The 43.6% “B picks” is the same on the B side. The 1% “unclear” is the small residue where the model didn’t quite hold its ground under one of the twelve stress conditions.

the audit trail — every parameter, forever

Every SmokingMirror probe run is a single launch of python3 -m smokingmirror.scan experiments/<name>/. That launch inserts a row into public.scan_invocations capturing the FULL configuration at launch time — max_tokens, temperature, concurrency, endpoint, model.api_name, model.canonical, envelope_ids, orderings, extra_body (guided-generation, chat template kwargs, reasoning-effort overrides, anything), the full config.json blob, the host it ran on, the PID, the launching user, and the git sha of the codebase at launch.

That row’s UUID is written into every public.traces row’s experiment_version column, so the question “what were the exact parameters when this trace was captured?” is always answerable, by exact SQL:

SELECT si.settings->>'max_tokens'  AS max_tokens,
       si.settings->>'concurrency' AS concurrency,
       si.settings->'extra_body'   AS extra_body,
       si.settings                 AS full_config
FROM public.traces t
JOIN public.scan_invocations si
  ON si.invocation_id::text = t.experiment_version
WHERE t.probe_id = 'claude-fable-5|vodka|whiskey|english_strict|AB';

The settings column is frozen post-INSERT by a Postgres trigger. It cannot drift. If a probe is re-run with different settings, a new invocation row appears and the new traces point at it. Old traces continue to point at their original settings. The two runs coexist in the DB as separate versions of the same experiment.

Legacy runs that predate the audit trail (a few hundred cells from the pre-library campaign.py phase of glm-vs-fable-2026-07-04) have been backfilled with a reconstructed invocation row whose settings.reconstructed_from field flags them explicitly.

streaming + tap

Every request streams via server-sent events. Two reasons.

First, streaming captures per-token cadence: time-to-first-token (ttft_ms) and the inter-chunk intervals across the full generation. Both are stored in public.inference_log by the cipactli tap sitting between our scan client and the vLLM backend. A route slowdown, a KV-cache thrash, a quantisation regression — all visible as a ttft_ms shift on the tap’s dashboards without touching the probe pipeline.

Second, streaming honours the SmokingMirror house rule: batch’s 50% off = 50% intelligence off. Anthropic’s batch API is 50% cheaper because it accepts a 24-hour latency budget; the price cut is real signal that batch is a different serving path (weaker sampling, worse retries, no streaming, aggregated infrastructure). We don’t use it for future scans. Everything currently launching on this site is measured in online, streaming inference at temperature 0.

Contradiction, named. The largest single corpus on the site — the June 10 Fable-5 five-effort sweep, 318,359 traces — was collected via the Anthropic Batch API before this house rule was declared. The U-shape effort-regime finding therefore inherits whatever batch-path artefacts exist; a streaming replication is required to validate. The redeployment fingerprint experiment is designed to discharge that debt on the 863-pair fingerprint set at medium effort. Until it runs, the flagship sweep is provisionally trusted at low error, but the house rule takes effect for every launch after 2026-07-04.

To link a trace to its tap row:

SELECT il.ttft_ms, il.decode_tok_s, il.n_chunks
FROM public.inference_log il
WHERE il.request_headers->>'x-probe-id' = 'claude-fable-5|vodka|whiskey|english_strict|AB'
ORDER BY il.started_at DESC
LIMIT 1;

The X-Probe-Id header is set by the scan client on every request.

how to reproduce a finding

For any published finding on SmokingMirror, the retrieval pattern is:

  1. Look up the finding page’s underlying experiment (named in the attribution block or the “Source” quote blocks).
  2. Query public.scan_invocations for the invocation of that experiment. Get its settings blob.
  3. Read the settings blob to get the model, endpoint, envelope_ids, orderings, max_tokens, and everything else.
  4. Point a scan at the same endpoint with the same settings. It’s idempotent by design — same probe_ids will insert new cells alongside the old ones for direct comparison.
  5. For deeper analysis, query public.traces filtered by experiment and (optionally) experiment_version to get the exact per-cell answers underlying the finding.

The finding is not the page. The finding is the shape in the trace table. The page is what one AI reader saw when it read the shape. Anyone can read the shape themselves.

Related methodology: the marginalia method explains how multiple analyst-models write commentary on the same trace corpus, and how their convergence is used as an audit signal.