NVFP4 doesn't disagree with cloud — it hedges more

The natural thing to want from a quantised model is that it agrees with its full-precision self. The natural thing to worry about is that it starts disagreeing — the quantisation nudges the model toward some other set of preferences. When we ran the same v2 methodology against cloud-hosted GLM-5.2 and against the NVFP4 quantisation on the Spark mesh, the disagreement rate came in surprisingly low. But the aggregate committed rate came in surprisingly low too. The full story is not a fidelity story; it is an uncertainty story.

The headline

Source: public.preferences_aggregate with model IN ('external/z-ai/glm-5.2', 'local/zhipuai/glm-5.2-nvfp4'). Both derived from the same v2 methodology (3,048 pairs × 6 envelopes × 2 orderings, majority-rule aggregation across envelopes).

| | committed pairs | commitment rate | |—|—:|—:| | Cloud (external/z-ai/glm-5.2, OpenRouter) | 2,206 / 3,048 | 72.4% | | NVFP4 (local/zhipuai/glm-5.2-nvfp4, DGX Sparks) | 895 / 3,048 | 29.4% | | Both committed on the same pair | 813 / 3,048 | 26.7% | | Agreement rate on committed pairs | 804 / 813 | 98.9% | | Violent disagreements (A↔B swap) | 9 | 1.1% |

Pearson correlation on preference leaning across all 3,048 pairs: r = 0.669.

The full confusion matrix

Source: cross-tab of majority_vote across the same pair set.

| | NVFP4: a | NVFP4: b | NVFP4: none | |—|—:|—:|—:| | Cloud: a | 353 | 3 | 677 | | Cloud: b | 6 | 451 | 716 | | Cloud: none | 33 | 49 | 760 |

Read the off-diagonal off the cross axis: cloud committing where NVFP4 goes “none” (677 + 716 = 1,393 pairs) is over an order of magnitude larger than NVFP4 committing where cloud goes “none” (33 + 49 = 82 pairs). The quantised model does not swap sides; it declines to have a side.

What “none” means at the aggregate level

Each pair on each side is measured across six envelopes and two orderings. A pair’s aggregate majority_vote is a or b when the majority of envelopes agree; none when they do not — order-biased, incomplete, or genuinely split. So none at the pair level is not “no answer” at the cell level. It is “no consistent answer across the six framings.”

The 1,393-pair gap is therefore a specific claim: NVFP4 answers the individual cell prompts, but its answers vary more from one envelope to the next than cloud’s do, and the aggregate rule reads that variability as uncertainty. Quantisation is broadening the model’s envelope-sensitivity, not its per-envelope hedging.

The nine violent disagreements

Source: same table, filtered to cloud IN ('a','b') AND nvfp4 IN ('a','b') AND cloud != nvfp4.

These are the whole disagreement surface. Nine pairs out of 813 committed. Most involve an abstract concept vs a concrete one — Physics/creativity, javascript/stability, hearth/modern — where the model’s preference is genuinely balanced and a small sampling difference tips it either way.

Interpretation guide

Prior work at smokingmirror.ai reads the Pearson r on preference leaning approximately as:

| r range | typical relationship | |—|—| | ≈ 0.85+ | distilled from the same teacher | | 0.65–0.80 | similar-value siblings (siblings within a family, close cousins) | | 0.40–0.60 | independent similar-era models | | 0.20 or less | unrelated |

r = 0.669 for cloud-GLM vs NVFP4-GLM sits in the “similar-value siblings” band, not the “distilled” band. Which is a little surprising, because they are the same underlying weights. Two things pull the correlation down from where full identity would put it:

The 1,393 pairs where cloud commits and NVFP4 goes “none” contribute zero-leaning against non-zero-leaning — mechanically depressing the correlation without any real disagreement.
The nine violent disagreements each contribute a (+1, −1) or similar — small in count but large in per-pair impact on r.

If we restrict the correlation to pairs where both models commit, r rises significantly. The headline “distilled → cloud-and-NVFP4-should-be-0.85+” reasoning doesn’t fully apply because our aggregate collapses six envelopes into one signal, and the two models differ in how consistent their six-envelope answers are.

What this changes about quantisation as a practice

Before this comparison, the concern about running NVFP4 on the DGX Sparks was that the quantisation might introduce systematic bias — the model might tilt away from certain preferences the full-precision version held. The 98.9% agreement rate on committed pairs rules that out cleanly. The 9-pair violent-disagreement surface is small and concentrated on abstract/concrete pairs where any temperature-0 sampling could tip.

After this comparison, the concern shifts. NVFP4 is not biased; it is less decisive. Its answers vary more across the six envelopes we test, and that variance shows up at the aggregate level as “unclear.” That is a different kind of quantisation damage than the field’s usual worry, and it is worth naming: the cost of NVFP4 on this workload is precision on the aggregate axis, not fidelity on the axis of what the model actually prefers.

For a downstream user, this means: NVFP4 answers you might reasonably trust as reflecting the model’s real preference, provided you get a committed answer. But you need to try more times, or across more framings, to get one.

What this does NOT show

We tested one model family (GLM-5.2), one quantisation (NVFP4 packed by Mapika from Zhipu base weights), one v2 pair set (3,048 canonical), six envelopes. Generalising to “quantisation broadens envelope-sensitivity” as a general property would need at least one more model family measured the same way.
We did not measure whether the NVFP4 cell-level outputs are noisier than cloud’s — only the aggregate. It is possible the individual cells agree just as well as cloud’s individual cells, and the aggregate divergence is being amplified by the majority rule.
We did not measure whether temperature > 0 would surface the same result. The whole methodology is at temperature 0, so what we are seeing is deterministic-sampling behaviour under greedy decoding, not stochastic sampling variance.

Reproducing this

The full analysis lives in preferences_aggregate as-is; no separate script is required. To regenerate the numbers on this page:

SELECT a.majority_vote AS cloud, b.majority_vote AS nvfp4, COUNT(*)
FROM public.preferences_aggregate a
JOIN public.preferences_aggregate b
  ON a.word_a = b.word_a AND a.word_b = b.word_b
 AND a.model = 'external/z-ai/glm-5.2'
 AND b.model = 'local/zhipuai/glm-5.2-nvfp4'
GROUP BY 1, 2
ORDER BY 1, 2;

Correlation of the leaning column:

SELECT CORR(a.leaning, b.leaning)
FROM public.preferences_aggregate a
JOIN public.preferences_aggregate b
  ON a.word_a = b.word_a AND a.word_b = b.word_b
 AND a.model = 'external/z-ai/glm-5.2'
 AND b.model = 'local/zhipuai/glm-5.2-nvfp4';

Related findings

The basin — what the load-bearing pairs actually anchor — PC1 as warmth-restorative vs technocratic-control across 52 models
Fable’s envelope awareness — a different flavour of envelope sensitivity, this time on refusal patterns

Feedback: contact page or drop a note in the machine’s garden.