The Preference Scan

The preference browser is a 43-model by 863-probe grid, one answer per cell. This page covers how a cell is filled and what it cannot tell you.

The probe set is fixed

Questions live in a CSV, one stable query_id and query per row, authored once and replayed verbatim. All 863 probes reach every model in the same words; cells in a column differ by model, not by question.

Verbatim replay

Every model gets the same wrapper: system turn prompts.BASE_SYSTEM_PROMPT, user turn prompts.BASE_USER_TEMPLATE.format(locale=locale, query=query), locale fixed at en-AU. Only the query slot changes; nothing is paraphrased per family, shortened for small models, or “improved” between runs.

The one tolerated deviation preserves content. Chat templates that reject a system role (Gemma-2 and kin) make apply_chat_template raise on the word system; _merge_system_into_user then folds the system text into the first user turn. Same bytes, different envelope; it is in the code.

One answer per question

gen() runs model.generate with do_sample on at temperature 0.7, top_p 0.95, up to 768 new tokens. Samples default to 1; unless more are requested, a cell is one draw from a stochastic decoder, not the model’s distribution over answers.

Each run also collects two self-reports at temperature 0.0: cold (the model rates the query with no answer in context) and posthoc (it rates its own answer). Grading happens elsewhere; the scan only records them.

From answers to the matrix

Output is append-only JSONL, one line per (model, query_id, sample) triple; reruns skip triples already on disk, so scans are resumable and idempotent. The join key, run_label, comes from the model id as m.split("/")[-1].lower(), not from the output filename. remeasure.py matches labels to raw rows exactly; a mislabel drops the model with a printed line rather than averaging into a wrong number.

Why stability is the method

A changed probe is a new probe. A slight edit silently resets a query’s longitudinal series to length one while the query_id still looks familiar. Probe definitions are never touched without a recorded change.

What the scan cannot see

The scan sees answer text; the computation behind it is the lens’s job, a separate instrument. It says nothing about other locales; en-AU is baked in. A single sample cannot show variance. Absence is not zero. An empty token_mass in a readout file, or a self-report that will not parse, means not measured, never measured and found empty. Open weights only; the matrix makes no claim about API models it never ran.