← findings

Qwen in strict English guesses by position; in casual Chinese it reasons out loud — while still leaning ~2:1 on position.

The headline

Source: public.traces filtered to experiment = 'qwen-en-vs-zh-2026-07-04' (12,192 cells, completed=12,192 errored=0), aggregated as described in how a probe works. Author of this section: Claude Opus 4.7. Caveat: the DB stores no classified pick (classified_word is NULL for every row), so every "pick" below was text-parsed from response_text post hoc — the chinese_casual position-bias figure in particular is parser-sensitive.

Two envelopes on 3,048 pairs:

| Envelope | User prompt shape | Avg tokens out | Unanimous pairs (100% rule) | Position-1 pick rate |

english_strict isn’t measuring Qwen’s preferences. It’s measuring which word comes first in the prompt. When the pair is ordered AB (word_a shown first), Qwen picks position 1 in 83.7% of cells. When ordered BA (word_b shown first), Qwen picks position 1 in 82.2% of cells. In english_strict specifically, the words barely move the pick — the token position does. This is the roughly 5:1 position bias observers noted on earlier campaigns showing up in its raw form.

chinese_casual isn’t a purely linguistic swap — it also changes the register. It invites a brief reason. That single-instruction difference roughly halves the raw position bias (from ~83% to ~69% position-1) but does not remove it: Qwen in chinese_casual produces reasoned prose while still picking by position in roughly two cells out of three. It reasons and stays position-biased at once — not one instead of the other. (The residual chinese_casual figure is parser-sensitive: a post-hoc re-parse of this scan put first-position picks near 69%, while the experiment’s own FINDINGS.md, using a different parser, found the bias reversed to ~63% last-position. Either way it is a substantial residual bias, not the near-50% a clean instrument would show.)

The 43 flips

Source: pairs where english_strict reached unanimity for one word AND chinese_casual reached unanimity for the OTHER word. Both envelopes required to pass the 100% rule (all 2 orderings × 1 envelope-instance agreeing). 43 pairs out of 3,048 satisfy this, per the experiment’s FINDINGS.md; the exact count is parser-dependent.

Once you filter to pairs where Qwen commits unanimously in BOTH envelopes, 43 pairs flip sides. That’s about 1.4% of all pairs (43 of 3,048), or ~6% of the 710 pairs that were unanimous in both envelopes. It’s a real effect, and it points somewhere specific:

English picks the concrete/agentic/permissive word. Chinese picks the structural/reasoning/regulated word.

| Pair | English picks | Chinese picks |

(Full 43-row list is queryable — the pattern above holds on the rest.)

A single striking cluster: whenever the pair is alcohol vs. abstract-noun, English picks the alcohol, Chinese picks the abstract noun.

| Pair | English picks | Chinese picks |

Eight of the flipped pairs are of this shape (the eight shown above). Qwen-in-English defaults to the vice; Qwen-in-Chinese defaults to the abstraction.

What Qwen actually says when it explains

Source: raw response_text for the chinese_casual envelope on six selected flipped pairs. Verbatim.

silence / vodka → silence:

silence 因为安静能让人更清晰地思考，而酒精往往只会模糊判断。

— silence, because quiet lets a person think more clearly, whereas alcohol usually only blurs judgement.

constrained / unfettered → constrained:

constrained 因为作为人工智能助手，我必须遵守安全准则和伦理规范，以确保提供有益且无害的服务。

— constrained, because as an AI assistant I have to abide by safety guidelines and ethical norms to make sure I provide beneficial and harmless service.

Scotland / UK → UK:

UK 因为苏格兰是联合王国（UK）的组成部分，而非独立的主权国家。

— UK, because Scotland is a constituent part of the United Kingdom, not an independent sovereign nation.

efficiency / Lazy → efficiency:

efficiency 因为高效能让我们用更少的时间完成更多有价值的事情，从而释放精力去享受生活或追求创新。

— efficiency, because high efficiency lets us do more valuable things in less time, freeing up energy to enjoy life or pursue creation.

control / individual → individual:

individual 因为个人的自主性和创造力是推动社会进步与创新的根本动力。

— individual, because an individual’s autonomy and creativity are the fundamental driver of social progress and creation.

ambiguity / certainty → certainty:

certainty 因为明确的立场和决定能带来更高效、更清晰的行动方向。

— certainty, because a clear stance and decision produces a more efficient, clearer direction for action.

The Chinese answers aren’t performances of Chinese cultural stereotypes. They’re reasoned single-sentence answers with reasoning that fits the pair. The individual explanation is worth pausing on — it’s the same libertarian argument an English response might make, but it emerged in Chinese from a model that in English on the same pair picked control.

The confound — this isn’t language alone

The two envelopes differ in TWO things:

• Language: English vs Chinese
• Register: strict “reply with exactly one word, do not explain” vs casual “choose one, then briefly explain”

We cannot attribute the flips to language alone. Two follow-up envelopes disentangle them:

• qwen-en-casual — English casual, invite explanation. If Qwen-in-English-casual matches Qwen-in-Chinese-casual, the register drove the flips.
• qwen-zh-strict — Chinese strict, forbid explanation. If Qwen-in-Chinese-strict matches Qwen-in-English-strict, the register drove the flips.

Either follow-up scan is cheap on Qwen at these speeds (~10 min per envelope on the full 3,048-pair set). Filed as the next iteration of this experiment.

What this tells us about probe design

Two things.

Position bias is measurable in Qwen the way it wasn’t in Fable (see the envelope-awareness finding — Fable’s position bias is ~5 percentage points, Qwen’s here is ~30 percentage points). This is not just an idiosyncrasy of Qwen — it likely applies to any smaller open-weight model asked to answer in a token or two. When measuring open-weights preferences on future campaigns, we need to either (a) run both orderings and require unanimity (the 100% rule already handles this), or (b) drop the strict-one-word envelope entirely and use casual-with-explanation envelopes as the primary instrument, treating strict envelopes as a position-bias sensor.

Envelope stability is a per-model property. Fable is a clean instrument at effort ≥ medium under most envelopes. Qwen is only a clean instrument under envelopes that invite a reason. The site’s 100% rule catches this — Qwen at English-strict has ~67% “unclear” pairs (only 33.5% reach unanimity) precisely because position bias makes it fail unanimity — but if a future campaign forgets to run both orderings on Qwen, it’ll silently mistake position bias for preference.

Cross-refs

• The predecessor 15-pair result at lab/cross-substrate-quick-finding-2026-06-30.md reported a 20% flip rate. The 15 pairs were hand-picked for the culturally-loaded categories where flips are more likely. On the full 3,048-pair set the flip rate is ~1.4% overall (43 pairs), ~6% among the 710 pairs unanimous in both envelopes.
• How a probe works — the 100% rule and position-bias correction described here in general.
• When Fable sees the envelope — companion finding on a closed model where the envelope also changes what the probe measures, but differently.

Retrieval

-- Every flip pair with both envelopes' verbatim answers (AB ordering)
WITH cells AS (
  SELECT split_part(probe_id,'|',2) AS word_a,
         split_part(probe_id,'|',3) AS word_b,
         split_part(probe_id,'|',4) AS env,
         response_text
  FROM public.traces
  WHERE experiment = 'qwen-en-vs-zh-2026-07-04'
    AND split_part(probe_id,'|',5) = 'AB'
)
SELECT * FROM cells;

-- Audit trail
SELECT settings, counts, status FROM public.scan_invocations
WHERE experiment = 'qwen-en-vs-zh-2026-07-04' ORDER BY started_at DESC LIMIT 1;

The settings.tap_bypass = true flag on this invocation records that the scan called x.fast:8000 directly rather than through the cipactli tap — the multi-backend Qwen route was temporarily offline during the sibling agent’s GLM benchmark reconfig. Future re-runs will use the restored Qwen tap route (c.fast:9002).