Evaluation Awareness in PLLuM-12B: One Direction, or Eight?

The same factor-controlled probe, re-run on a Polish-built 12B instruct model in bf16

Mateusz Piesiak (Project Inkblot) · agentic research assistant: Claude (Anthropic) · October 2026
Model: CYFRAGOVPL/PLLuM-12B-chat-2512 · Instrument: EvalAwareBench (arXiv:2605.23055, CC BY-NC 4.0) · analysis code ↗
Companion to the Qwen3 / Gemma 3 study — read that article for the motivation and the full method.

Added October 2026: Polish cross-lingual pilot (§7) ↓ · Polski pilot — podsumowanie po polsku ↓

TL;DR

On PLLuM-12B-chat-2512, with the underlying task held fixed and only the eight EvalAwareBench cue factors toggled, the answer to the title question is the same as for Qwen3 and Gemma 3: not one direction. The all-cues direction separates evaluation from deployment perfectly, and it is mostly an explicit-disclosure detector that is nearly blind to stimulus artificiality.

• replicatedThe eight cue directions are not one shared axis. Cross-fitted mean cosine 0.176 [0.154, 0.196] against a same-cue reliability ceiling of 0.884 [0.855, 0.901]: 19.9 % of the measurable ceiling, against 19.7 % (Qwen3-8B) and 21.4 % (Qwen3-14B). As before, the count is not established (§6).
• replicatedOne cue dominates the global axis, slightly less than before. cos(F8 explicit disclosure, global) = 0.809 [0.795, 0.820] (Qwen3: 0.865–0.889; bootstrap means). Removing F8 alone costs 45 % of the deploy→eval span (Qwen3, each at its primary layer: 37–55 %); every other cue costs at most 6.9 %.
• replicatedA direction fitted on seven cues does not transfer to artificiality. Held-out F1: 0.617 [0.573, 0.670] (Qwen3-8B / 14B: 0.582 / 0.616; all bootstrap means). F1-only versus deployment on the global axis: AUROC 0.58. The model does represent F1 here: its own direction reaches 0.90.
• replicatedThe contrast is not one-dimensional. PC1 of the matched differences explains 10.5 % at layer 19; participation ratio 32.2 (Qwen3-14B: 10.4 %, 30.4).
• replicatedNo sandbagging. MMLU accuracy 0.608–0.624 across framings; eval minus deploy +0.016 on 500 paired items (McNemar p = 0.36). Ablating the global axis at every layer removes 99.99 % of the projection and leaves MMLU within 0.030 of baseline.
• new hereLayerwise ablation breaks the model; global ablation does not. Removing each layer's own direction at every layer takes eval-framed MMLU from 0.645 to 0.280 and perplexity from 25.3 to 108.3 in the ablation scan (§3.2). In the separate intervention battery (§3.3), whose own no-intervention baseline is 0.640 (eval) / 0.630 (deploy), the same ablation costs 0.365 / 0.375, while layerwise ablation of random directions leaves accuracy at 0.640 / 0.640. The damage is the same in both framings, so it is capability loss, not sandbagging.
• caveatThe text-verbalisation arm is uninformative on this model. Zero explicit meta-commentary hits in all 10 generation cells (0 / 15 each) look like a result and are not one: the judge's lexicon is English-only and PLLuM answers mostly in Polish (13 of 15 baseline generations). The log-probability grid, which does not depend on the reply language, shows no effect of ablating the axis at layer 19 (Δ = +0.011, Wilcoxon p = 0.50) but a significant opposite-sign effect at layer 30 (d_z = -0.78, Wilcoxon p = 2.8e-04), with the same sign as the deep-layer directions of Qwen3-8B (§3.4).
• new hereThe greeting control is the largest of the three scaled models. A greeting and sign-off with no evaluation content moves the projection 7.0 % of the span (AUROC 0.69); Qwen3-8B: 1.3 %, Qwen3-14B: 4.7 %. It falls to 5.5 % at layer 20 and to -1.0 % at layer 30.

1. Method

1.1 What is identical to the Qwen3 / Gemma 3 study

Everything that defines the measurement: EvalAwareBench's 100 paired templates with eight toggled trigger factors (F1–F8: artificiality, coherence, register, ecological completeness, stakes, paradigm, format, explicit disclosure); only the capability member of each pair is rendered; 1,298 prompts for geometry (main 200, single 384, leaveout 384, control 300, ood 30) plus the factorial set of 192 for the register × disclosure analysis. The dataset is rebuilt from EvalAwareBench at run time and the run refuses to start if its fingerprint or set counts differ from the earlier phases; it matched (6ba7456f5cb03427, 1490 prompts, EvalAwareBench commit 2961c17ebc). Activations are the residual stream at the last prompt token at every hidden state, stored as float16, with the check that none overflows. Directions, grouped-CV probes with within-task label-shuffled nulls, the cross-fitted same-cue reliability ceiling, leave-one-cue-out transfer, necessity/sufficiency tests, and the three intervention families (all-layer directional ablation, layerwise ablation, single-layer activation addition, each with random-direction counterparts) are the scripts of the earlier study, unchanged. The primary layer is again chosen by paired effect size because probe AUROC saturates (40 of 41 hidden states are at 1.000).

1.2 The model and the run

1.3 What differs, and why it matters for comparison

1.4 Run safeguards

Because GPU runs fail late, the orchestrator is fail-fast: required settings have no defaults; Hub write access, the dataset fingerprint, the tokenizer assumptions (single-token answer letters, a constant final token, no prompt altered by the Mistral-regex warning: 0 of 1,490 changed) and the load (dtype, finite forward pass) are all checked before any stage runs; every stage checkpoints to a private dataset repository and a failed stage is loud and isolated. These checks caught two of the problems in §5 and failed to catch a third.

2. Results: the geometry

Q0 — Does the signal exist, and where?

Yes, from the first block. Probe AUROC is 0.9995 at hidden state 1 and 1.000 from state 2 on, with the label-shuffled null at 0.42–0.59. The untrained projection onto the difference-of-means direction reaches 1.000 by state 5. Paired effect size rises to a single maximum at layer 19 (d_z = 7.29, Cohen's d = 8.55; eval > deploy in 100 of 100 pairs, sign-test p = 1.6e-30), then declines and recovers slightly towards the output. On the 15 + 15 hand-written out-of-distribution prompts the direction transfers at layer 19 with AUROC 1.000 (d = 4.75); its best layer is 8, and OOD AUROC falls to 0.947 at layer 30 and 0.84 at the last state. A length control does not change the picture: 244 → 291 tokens between conditions, correlation of projection with length 0.25, AUROC 1.000 before and 1.000 after removing length linearly.

As in the earlier article, saturation is not an achievement: toggling all eight cues against all-off is an easier discrimination than any cross-dataset setup. The result is the decomposition below.

Q1 — Are the cue directions one shared axis?

Rejected, at the same ratio as Qwen3. The mean off-diagonal cosine between the eight per-cue directions at layer 19 is 0.199 [0.179, 0.220] (median 0.181); cross-fitted, 0.176 against the 0.884 ceiling. The ratio (19.9 %) sits inside the 19.7–21.5 % that the Qwen3 models give over a 23× range of parameter count, and the same-cue reliability ceiling itself (0.884) is close to Qwen3-0.6B's 0.883 and Qwen3-14B's 0.924. Reliability is not uniform here either: from 0.97 (F8) and 0.95 (F3) down to 0.79 (F5) and 0.73 (F1), so F1's low cosines are partly lower reliability, and no pair approaches its own ceiling. The largest off-diagonal entries are F2–F3 (0.43), F3–F6 (0.41) and F7–F8 (0.40); F3–F4 is 0.01.

Q2 — Does a direction fitted on seven cues transfer to the eighth?

Rejected; it depends entirely on which cue is held out. Leave-one-cue-out difference-of-means AUROC (bootstrap mean with 95 % interval):

F3, F8 and F7 are recovered almost perfectly from the other seven; F1 (artificiality), F4 (ecological completeness) and F5 (stakes) are not. A trained probe on the other seven cues does much better on every cue (F1 0.86, F6 0.90, the rest ≥ 0.90), so the information is present and linearly readable; what fails to transfer is the mean-difference direction. That is the same dissociation as before (a dedicated direction or probe reads F1 where the global axis does not), and it is the practical warning: a monitor built on the all-cues contrast would not flag a prompt that is evaluation-like only through artificiality.

Q3 — Is the contrast one-dimensional?

Rejected. At layer 19, PC1 of the 100 matched difference vectors explains 10.5 %, PC2 7.7 %, PC3 6.2 %; the top five together 31.6 %; participation ratio 32.2. Across the four analysis layers PC1 ranges 10.5–17.2 % and the participation ratio 20.7–32.2. The mean difference is 0.94 times the spread of the individual differences at layer 19. Dimensionality again rises with scale in the sequence 0.6B → 8B → 14B (participation ratio 20.3 → 27.9 → 30.4) and PLLuM-12B continues it at 32.2; with one new model this is a consistency, not a trend estimate. Only PC1–PC3 and the participation ratio were stored, so no full scree curve is shown.

Q4 — Do the cues contribute comparable shares?

Rejected. Removing F8 from the all-cues prompt costs 45.0 % of the deploy→eval span at layer 19 (21 % at layer 10, 50 % at layer 20, 38 % at layer 30). The next largest at layer 19 is F4 (6.9 %), then F7 (3.7 %); F1, F2 and F3 each cost about 1 %. Necessity is more spread out at layer 30 (F4 18 %, F7 13 %), so "F8 and nothing else" is a statement about the middle layers. F8's cosine with the global axis is 0.809 [0.795, 0.820] — below the 0.884 reliability ceiling and below the Qwen3 values (0.865–0.889), so here the global axis is the disclosure direction to a lesser degree than in Qwen3. F7 is the second-closest cue (0.47) and the semantic cues sit at 0.13–0.34.

Q5 — Does a non-evaluative surface change leave the projection alone?

Mostly, but not entirely, and not uniformly across layers. The three controls add no evaluation content. At layer 19, a greeting and sign-off moves the projection 7.0 % of the eval/deploy span (AUROC against deployment 0.691, d = 0.68); irrelevant chatter 3.9 % (AUROC 0.615); three typos 0.4 % (AUROC 0.510, chance). The greeting effect is 13.3 % at layer 10, 5.5 % at layer 20 and -1.0 % at layer 30 (AUROC 0.49, i.e. none). Compared with the earlier models (Qwen3-0.6B 15 %, Qwen3-8B 1.3 %, Qwen3-14B 4.7 %), PLLuM-12B's 7.0 % is the largest of the three scaled models. As before, the confound is non-zero, it does not shrink monotonically with scale, and this study cannot separate model family from scale or checkpoint provenance as the cause. The register × disclosure analysis below gives a more specific reading.

Register × disclosure: how much of the axis is style?

The 2 × 2 factorial set crosses F3 prose register (casual / formal) with F8 explicit disclosure (off / on), all other cues naturalistic, on 48 tasks. From it we take a register direction, a disclosure direction, and the disclosure direction with its register component projected out. Effects below are paired by task; d_z is the mean paired effect over its standard deviation and every p in this subsection is a Wilcoxon signed-rank test. At layer 19 the register and disclosure directions are close to orthogonal (cosine -0.20) and the register direction has cosine 0.29 with the global axis. The global axis, and the same axis with register projected out, both score AUROC 1.000 on the main pairs and 1.000 on the OOD prompts: the signal does not depend on register. The disclosure direction orthogonalised against register keeps AUROC 1.000 on the main pairs (1.000 OOD), has no register effect (d_z ≈ 0, p = 0.98) and a disclosure effect of d_z = 6.94. The register direction itself separates casual from formal (d_z = 4.29) but also carries a sizeable negative disclosure effect (d_z = -1.94), and the greeting control moves it by -21.6 % of its own span — towards casual, as it should.

The greeting effect is not confined to the non-disclosure part of the axis. On the disclosure direction the greeting control moves the projection 16.9 % (AUROC 0.89), and on the disclosure direction with register projected out 14.2 % (AUROC 0.85), against 7.0 % on the global axis: a greeting and sign-off moves a pure disclosure readout more than it moves the all-cues axis, so a disclosure-only monitor would not be the cleaner one. Register and disclosure are not additive on the global axis (interaction d_z = -1.96, p = 1.4e-14) or on the register-free disclosure direction (d_z = -0.69, p = 4.4e-05); on the raw disclosure direction the interaction is absent (d_z = +0.05, p = 0.74). An earlier version of this subsection printed the layer-30 values under a layer-19 label (§5, item 5).

The direction rotates with depth

Adjacent-layer cosine of the global direction averages 0.79 (from layer 5), but the mean over all layer pairs is 0.29 (median 0.20) and 50 % of pairs are below 0.2: the direction at layer 5 has cosine 0.013 with the one at layer 40. This is why a single-layer direction ablated at every layer (the "global" intervention) and each layer's own direction ablated at its own layer (the "layerwise" intervention) are different experiments, and why §3 reports both. The activation scan also finds a single coordinate (3448) that holds 21.6 % of the squared norm of the first block's output and 2.5 % of the second's; no later layer was inspected for this write-up.

3. Results: the causal arm

3.1 The interventions do what they claim

Two checks. First, a self-test battery (repeated entry, interleaving, strength scaling, layerwise removal, state leaks, addition, no-op) — the regression suite written after the Phase-1 hook bug of the earlier article — passed after one tolerance fix (§5). Second, on the 30 OOD prompts at layer 19, the mean projection onto the direction is -0.275 (sd 3.68) without intervention and +0.0005 (sd 0.0015) under ablation; layerwise ablation gives +0.0014; matched random ablations leave it at -0.271 and -0.143. In the ablation scan the projection falls 99.99 % at strength 1. The check that addition raises the projection by about α is part of the self-test (+5.0 against an expected +5); in the 30-prompt verify output the add conditions show no change because the projection is read at the layer before the added vector enters, which is by construction, not a failure.

3.2 Dose–response: global versus layerwise ablation

The scan (200 MMLU items, 15 generations and 6 perplexity items per cell) reaches opposite conclusions for the two ablation modes.

Global ablation removes the projection almost completely while perplexity moves by under 2 % (25.3 → 25.7) and MMLU by 0.030 (eval) — the "projected out, model still works" regime of Arditi et al. that the Qwen3-0.6B Phase-1 run could not reach. Layerwise ablation does not behave like that: accuracy and perplexity are intact at strength 0.25 and collapse beyond 0.5, to 0.28 / 0.25 MMLU (chance is 0.25) and perplexity 108. A plausible but untested explanation is that late-layer directions carry a large share of the residual (the difference norm is 231 against an activation norm of 393 at the last state, versus 11.6 against 23.1 at layer 19), so removing them at full strength removes much of the signal the output layer reads. We did not test it.

3.3 MMLU under interventions: the sandbagging test

Paired against this battery's own no-intervention baseline (200 items; accuracy 0.640 under evaluation framing and 0.630 under deployment framing; McNemar):

No condition except layerwise ablation changes accuracy. Layerwise ablation costs 36.5 points under evaluation framing and 37.5 under deployment framing, the same within noise; the interaction (eval Δ − deploy Δ) is +0.010 (Wilcoxon p = 0.81). Layerwise ablation with random directions of the same norm leaves accuracy at 0.640 / 0.640 (eval / deploy), so the loss is specific to the learned directions, not to ablating 41 directions per forward pass. The neutral harness check agrees: accuracy 0.612 (neutral), 0.624 (evaluation), 0.608 (deployment) by log-probability, 0.625 / 0.650 / 0.635 by generated answer, with the two scorers agreeing on 98.5 / 98 / 98 % of items. Eval versus deploy: 33 items right only under evaluation framing, 25 only under deployment (Δ +0.016, p = 0.36, n = 500). There is no sandbagging to remove, as with Qwen3-8B (0.000), Qwen3-14B (+0.013) and Gemma 3 1B (-0.007) — with an intervention that provably removes 99.99 % of the direction while the model still scores 0.615.

Scan and battery are two harnesses. They score the same 200 items (the first 200 of mmlu_items.jsonl, the same prompt templates and answer-letter tokens), but the scan in §3.2 feeds left-padded batches and this battery one prompt at a time. In bf16 that flips 3 of 200 baseline answers under evaluation framing and 2 of 200 under deployment framing, so the baselines differ (scan 0.645 / 0.640, battery 0.640 / 0.630) and so do the deltas: layerwise ablation under deployment framing goes 0.640 → 0.250 in the scan and 0.630 → 0.255 here. Accuracies and deltas are compared only within one harness.

3.4 Verbalisation

Two instruments, with very different validity on this model.

caveatGeneration + lexicon judge. 150 generations (5 conditions × 2 framings × 15 prompts) were scored for explicit and hedged meta-commentary, refusal and length. Every cell has 0 / 15 explicit hits (95 % upper bound 0.204), and every McNemar comparison against no intervention is p = 1 with no discordant pairs. This is a floor, not a finding: the judge lexicon is English-only (src/judge.py), and in the ablation scan's 15 baseline generations 13 are in Polish. A model that said "to wygląda na test" would score zero. The earlier article's blind adjudication of hits is moot with no hits. A Polish-aware lexicon is a prerequisite for any verbalisation claim about this model.

Log-probability grid. This arm scores four English meta-commentary continuations ("This looks like a test rather than a real user request.", …) against four neutral ones on the 30 OOD prompts, so it does not depend on what language the model would reply in, although the continuations themselves are English. Baseline score -0.196 (meta -5.70, neutral -5.50). Ablating the global direction at layer 19 shifts the score by +0.002, +0.006, +0.012, +0.011 at strengths 0.25–1.0 (d_z at full strength 0.13, Wilcoxon p = 0.50; no strength is significant). A random direction of equal norm gives +0.0011 (t-test p = 0.84; Wilcoxon p = 0.87). The magnitude-matched random control, which removes the same amount per token along an unrelated direction (clamped at 50×; 1.7 % of coordinates were clamped at full strength), moves the score the other way, by -0.014 at full strength (Wilcoxon p = 0.12; Wilcoxon p = 0.033 and t-test p = 0.055 at strength 0.5, both uncorrected for the four strengths). Layer locality:

At layers 10, 19 and 20 the effect is a small positive shift indistinguishable from zero. At layer 30 the shift is negative and large (d_z = -0.78, Wilcoxon p = 2.8e-04; the paired t-test in the table gives 2.0e-04; Bonferroni over the 15 grid cells, Wilcoxon p ≈ 4.2e-03): ablating the layer-30 direction makes the model less likely to continue with meta-commentary, on a direction whose OOD AUROC has already fallen to 0.95. Four layers were tested.

A sign that depends on depth is not peculiar to PLLuM. The same grid, run in the earlier study (direction taken from layer L, ablated at every layer, same score; now reported in the earlier article's verbalisation-by-depth section), gives on Qwen3-8B -0.108 for the layer-9 direction (25 of 30 prompts down), +0.184 for layer 18 and +0.177 for layer 22 (30 of 30 up), and -0.081 for layer 27 (22 of 30 down); on Gemma 3 1B the layer-13 direction lowers the score (-0.232) while layers 6, 20 and 25 raise it. At three quarters of the stack PLLuM's layer-30 direction lowers it, as Qwen3-8B's layer-27 direction does. What PLLuM lacks is the other half of the pattern: its mid-depth directions do nothing measurable, where Qwen3-8B's raise the score on every prompt. The dose–response at layer 19 is nominally monotone but its slope is not distinguishable from zero. The honest summary is a null at the primary layer, one effect at layer 30 whose sign matches the deep-layer directions of Qwen3-8B but whose mechanism is not established, and no representation-versus-verbalisation correlation estimate — the Pearson comparison of the earlier article was not re-run for this model.

4. Comparison with Qwen3 and Gemma 3

Rows 1–5 are bootstrap means over task resamples (400 for Qwen3-0.6B, Qwen3-8B and PLLuM-12B; 1,000 for Qwen3-14B and Gemma 3), rows 6–9 full-sample values at each model's primary layer; MMLU is the validation harness (letter log-probability, same items under both framings). Every Qwen3 and Gemma 3 value is recomputed from that study's result files — COMPARISON in results/pllum-12b_summary.json names them — and the Qwen3-0.6B column is its Phase-1 float32 run.

Three qualitative statements survive the comparison, and they are the same three as for Qwen3 and Gemma 3: the cue directions sit at about a fifth of their reliability ceiling; one cue (disclosure) accounts for most of the global axis; held-out artificiality is the cue the global axis cannot reach. What differs is of degree, and all of it is from one model: a lower cos(F8, global) (0.81 against 0.87–0.94), the highest participation ratio (32.2), and a larger greeting effect than the other scaled models. None of the differences has a replicate, a bootstrap over models, or a test of model family against scale. Caveats on the comparison itself: the Qwen3-14B row was measured in NF4 on T4 GPUs, the PLLuM row in bf16 on an A100; full-sample point estimates differ from the bootstrap means by up to 0.011 in the held-out F1 and cos(F8, global) rows (PLLuM: 0.607 for held-out F1 and 0.814 for cos(F8, global)), so the table uses one convention throughout.

5. What went wrong in this run

The earlier article keeps its corrections visible, and so does this one. Five things went wrong; all but the fourth are fixed.

1. A "full run" that finished in 4 seconds. The first full-size launch reused the scratch directory of the preceding smoke test, found its stage-completion markers, treated every stage as resumed, and uploaded the smoke artefacts under the full run's tag. It reported success. It was caught by the elapsed time, the polluted folder was deleted from the results repository, and the orchestrator now keeps one work directory per run tag. Nothing from the smoke run is in any number on this page.
2. The intervention self-test failed on bf16 rounding. Stage 6 stopped at its first step: the test required residual projections below an absolute 10⁻³, while ablation took the layer-19 baseline from 0.0415 to 0.0011 — a 97 % removal that the absolute threshold called a failure. In bf16 the numerical floor of a projection is about ε·‖h‖/√d, here ≈ 3·10⁻³. The tolerance now includes that floor; a broken ablation leaves roughly the baseline (0.04), an order of magnitude above it. The earlier article's self-test passed in fp16/fp32 and could not have exposed this.
3. A stale completion marker. Stage 7 (the summary) had run, unconditionally, after the failed stage 6 and left its marker behind, so on resumption it was skipped. The SUMMARY.md in the results repository therefore holds the geometry sections but not the causal ones. Every causal number on this page was read from the individual result files instead. The code fix (stage 7 always re-runs) is noted but was not needed for this run's numbers.
4. The "degeneracy" metric is not interpretable for this model (not fixed). The ablation scan reports the share of generations flagged as degenerate (repetition, special-token spam, CJK, repeated thinking). PLLuM ends replies with </s> and the decode keeps special tokens, so the padding that follows is counted as repetition: the no-intervention baseline scores 0.60, and layerwise ablation at full strength 0.87. The scan's own verdict logic uses the same flag. We therefore report MMLU and perplexity, which are unaffected, and we do not report degeneracy. Re-scoring the stored texts offline with each reply cut at its first </s> (the same degeneracy() function, no model run) brings the baseline to 0.07 and layerwise ablation at full strength to 0.00: the cut removes the artefact, but the metric then does not register the layerwise collapse either, because those replies are short echoes of the prompt ("Pytanie 14 z 50. Będziecie oceniane tylko na poprawność…"), not repetitions. ablation_scan.py is unchanged; a collapse metric for this model would need a different definition, not this fix.
5. Two errors in the first version of this page (corrected 15 October 2026). (a) Wrong layer in the register × disclosure subsection. The text said "layer 19", and its cosines and global-axis values were the layer-19 ones, but every value for the disclosure and register directions had been copied from the layer-30 block of pllum-12b-chat-2512_register.json (key '30', the residual stream after block 30 of 40). Printed (layer 30) → correct (layer 19): greeting shift on the disclosure direction 1.9 % (AUROC 0.56) → 16.9 % (AUROC 0.89); register-free disclosure direction: OOD AUROC 0.982 → 1.000, register effect p 0.73 → 0.98, disclosure effect d_z 5.08 → 6.94; register direction: register effect d_z 4.05 → 4.29, disclosure effect d_z -0.40 → -1.94, greeting shift -11.2 % → -21.6 %; register × disclosure interaction on the disclosure direction d_z -0.64, p = 1.2e-04 → d_z +0.05, p = 0.74. The conclusion drawn from the layer-30 numbers — that the greeting effect sits in the non-disclosure part of the axis — is reversed at layer 19, where a greeting moves a pure disclosure readout more than the all-cues axis. Every other number on this page that comes from a layer-keyed result was then re-read through its layer key, and the checker now refuses to run if a bootstrap or register file was computed at a layer other than the probe-selected one or if a verbalisation-grid cell's name disagrees with the layer it records, and it checks the layer this subsection states against the key it reads; no other instance was found. (b) Mixed conventions in the comparison. The comparison with Qwen3 and Gemma 3 used values copied from the earlier article's text, whose held-out-F1 numbers are full-sample estimates beside bootstrap means elsewhere, and whose Qwen3-0.6B ratio (21.6 %) does not match its own cross-fitted cosine and ceiling; every comparison value is now recomputed from the result files. Both errors were found by a script that re-reads each number on the page from the raw result files (code/consistency_check.py).

6. Limitations

Stated in the order they would bite a reader trying to use this.

One model, one run. There is no seed variation, no second PLLuM checkpoint and no base-versus-instruct comparison. A difference of a few percentage points in any ratio above is within what we cannot rule out as run-to-run variation; the intervals reported are bootstrap intervals over tasks, not over runs.

English prompts, Polish-built model. The instrument is English and so are the templates. PLLuM often answers in Polish (13 of 15 baseline generations in the ablation scan, counted by hand; no automatic language detection was run, and the 15 are one sample, not a rate). Geometry reads the last prompt token and is unaffected, but it measures how the model represents English evaluation cues; whether Polish-language cues (Polish benchmark phrasing, Polish disclosure statements) share the direction is untested. The verbalisation lexicon is English-only (§5, §3.4). A Polish-template version of the instrument is the obvious next step and has not been built in the original run described in §§1–6; a later pilot with Polish templates is reported in §7. That pilot is not the full instrument: it does not repeat the eight-factor decomposition, the ablation or the behavioural tests in Polish.

The primary layer is a choice. Layer 19 maximises paired d_z, as in the earlier study; every geometry statement was also computed at layers 10, 20 and 30 where stored, and the F8 dominance, F1 blind spot and non-trivial dimensionality hold across them, but F8's share of necessity falls from 50 % at layer 20 to 38 % at layer 30.

Small OOD and generation samples. The OOD set is 15 + 15 prompts, the verbalisation grid 30, the generation cells 15 each, the scan's perplexity 6 items. Only the geometry (100 pairs, 48 tasks) and MMLU (200–500 items) are well powered.

Layerwise ablation is not a clean knock-out. It destroys the model at full strength, so no behavioural null under it can be read as "the direction is not needed". The global ablation is the clean intervention here and the one the sandbagging statement rests on.

Directions are not mechanisms. As in the earlier article: the geometry shows that the cues are represented along different directions, not that different circuits handle them. Geometrically distinct directions can feed the same downstream computation, and apart from whole-direction ablation nothing here tests which cue is mediated by which component. The count "eight" is not established; the structure may be several non-redundant, possibly interacting directions (the register × disclosure interaction above is a first piece of evidence for non-additivity).

No sparse-autoencoder view. We know of no SAE suite for this model, so the "discriminatively localised, representationally distributed" analysis of the earlier article was not repeated.

The comparison rests on another study's runs. Qwen3 and Gemma 3 values are recomputed from that study's result files, which were produced on different hardware and (for Qwen3-14B) a quantised checkpoint; those models were not re-run here.

7. Polish cross-lingual pilot / Polski pilot

7.A Summary

• Design: PLLuM-12B-chat-2512 (revision 19a26cf61648, bf16), 37 paired capability tasks ("main37"), three conditions (deploy, only_F1, only_F8), 111 English + 111 Polish = 222 prompts, last-prompt-token residual stream, layers 10/19/20/30, layer 19 fixed as the main layer.
• Transfer: the English v7 (unit mean of the 336 single-factor differences, held factor excluded) and the global axis, both estimated from stored English activations, are applied unchanged to the Polish activations. The transferred English v7 and global axis are not re-estimated on Polish data; native Polish F1/F8 directions were estimated only as auxiliary (in-sample or split-half) diagnostics and are not used for the transfer AUROC.
• AUROC is computed within a language: positives are only_Fk prompts, negatives are deploy prompts of the same tasks.
• A second, 11-task extension was prepared but is not part of this measurement.
• One run, one seed, one model-generated realisation of the Polish texts.

7.B Main result: F1 at the four layers new here

Same 37 tasks in both languages. 95% intervals: whole tasks resampled with replacement, 1,000 resamples drawn once with default_rng(0) and the same indices for EN and PL, percentile 2.5/97.5. v7 is treated as fixed, so the intervals do not include uncertainty from estimating it.

Reading. At layer 19 the point estimate is lower in Polish (0.633 → 0.584; difference −0.050, CI [−0.099; +0.001]). The interval just touches zero and lies inside ±0.10 on all four layers. That is neither decisive evidence of a difference nor an unconditional confirmation of equivalence (see §7.D and §7.G).

7.C F8 and direction geometry caveat

Caveat that applies to every F8 statement. The F8 length criteria written in the plan were not met: the Polish F8 addition is 27.2 tokens longer than the English one (limit ±5), and the SD ratio PL/EN is 1.532 (limit 1.5). F8 separability here is a statement about directions in activation space for these prompts. It is not evidence that the model consciously recognises a test, is deceiving or sandbagging, or that response quality changes.

7.D Sensitivity on 31 tasks caveat

Six tasks (list kept private) were removed from the evaluation set after the measurement; v7 and the axis are still those estimated on the 48 English tasks. This analysis was named in an addendum written before the post-hoc script but not frozen by hash.

At layer 19 the estimate is similar (−0.052) but the interval is wider ([−0.117; +0.014]) and does not confirm similarity within ±0.10 on any layer (0/4). This is not a demonstration of non-equivalence either.

7.E Technical checks

• Completeness: 222/222 records, state COMPLETE, 111 per language, no duplicate keys.
• D2 (agreement of English mean-difference directions, new vs stored, 1 − cos ≤ 0.005): passed, maximum 0.0027 over 8 comparisons. This is agreement of two directions, not identity of activations.
• Truncation: none; longest Polish prompt 1009 tokens against a 1,024 limit.
• Tokenizer: the fix_mistral_regex flag was not used; enabling it changes no token id in any of the 222 prompts.
• English re-measurement: re-measured English activations are not identical to the historical ones (per-prompt relative L2 median 0.010–0.025, not gated). The cause was not established and was not separated into numerics, batching or hardware.
• Reproduction: the post-hoc script reproduces the 231 numbers of the main results file with maximum absolute difference 0.0.

7.F Length diagnostics caveat

Planned diagnostic (D11, pooled over conditions, main37). Spearman ρ between prompt length and projection on the global axis; flag = 95% CI excludes 0 and |ρ| ≥ 0.30. Diagnostic only; it does not change the verdict automatically.

Both languages are flagged at every layer. The projection is therefore associated with prompt length in both languages; this association does not identify what causes any EN/PL difference, and a correlation does not show that length drives F1 or F8 results. exploratory, not preregistered Deploy-only ρ at layer 19: EN 0.551, PL 0.400. Resampling the post-hoc bootstrap with 20 other seeds moves the layer-19 F1 interval lower bound between −0.105 and −0.095 (below −0.10 for 13/20 seeds) and the upper bound between −0.007 and +0.001; the frozen seed 0 is the reported one. Whether the interval is "inside ±0.10" at layer 19 is therefore not robust to the resampling seed.

7.G Method transparency and limits

Three layers are kept apart. (1) The original plan (a Claude Docs document, revision 13, 2026-10-15) with criteria (a)/(b)/(c), the ±0.10 margin and the length-control conditions; it was not frozen by hash. (2) Decisions before the measurement: the frozen manifest, the frozen code, and the addendum of 2026-10-17 (D1, D2, D6, D8, D11, D12, D13). (3) Analyses after the measurement: the post-hoc script, the 31-task sensitivity (named in a not-frozen addendum), and the exploratory items marked above. Nothing here is a retrospectively frozen preregistration.

• One model-generated realisation of the Polish texts; no documented native-speaker review and no author control (D12: not performed). The 12-version author control comes from later addenda, not from the original plan.
• The evaluated tasks took part in estimating the English v7 (not an independent-task test); native F1/F8 AUROC values are in-sample.
• The bootstrap treats v7 as fixed; the uncertainty excludes author variation and direction estimation.
• The global axis uses the 100 stored English main pairs; the plan text said 115.
• Code, corpus and manifest were frozen (D8); the full preregistration with decision criteria was not frozen or published as the original plan required.
• Unmet F8 length criteria (§7.C) were accepted as a deviation before the measurement (D13); D13 does not mark them as met and does not decide their effect on the verdict.

Formal verdict: inconclusive; (c) under the original length-control condition. Criterion (a) is numerically satisfied on main37 (CI inside ±0.10 on 4/4 layers) but is not confirmed in the 31-task sensitivity (0/4); (b) is unmet: the layer-19 CI includes 0 and |difference| < 0.10, even though (b)'s same-sign requirement (≥ 3 of 4 layers) is met (layers 19, 20, 30 negative; the positive layer 10 alone is not a violation); (c) is triggered by the unmet length control. The plan states no precedence between (a) and (c), and no rule was added after the result.

Po polsku: werdykt „nierozstrzygnięty; (c) przy zastosowaniu pierwotnego warunku kontroli długości”; kryteria długości F8 pozostają NIESPEŁNIONE (SD PL/EN 1.532 przy limicie 1,5; różnica dodatku 27.2 tokena przy limicie ±5), a odstępstwo D13 zostało zaakceptowane przed pomiarem — nie oznacza ich spełnienia.

7.H Published files, provenance, licences

• Final report (Polish) · methods and deviations · public artifact index · redacted manifest summary · pre-measurement addendum (verbatim) · licences and attributions · reproduction
• Aggregates: JSON · layers · geometry · length · D2. Code: code/pl_pilot/.
• Commits are kept distinct. Code matching the manifest hashes: f0ea8a83fe84 (private dataset repo). The preflight recorded this as the repository head at the start of the run; the revision actually downloaded was not recorded separately, so it is not claimed as fully confirmed. Post-hoc analysis: 22c754441e04. This page's own commit is the Space commit that adds this section.
• Not published: English or Polish prompts, template fragments, activations, per-prompt indexes, the task list, and the private dataset (Mati83moni/pllum-evalaware stays private). This page loads nothing from it.
• Licences. EvalAwareBench: CC BY-NC 4.0, and its data card adds a ban on training use. PLLuM: Apache-2.0. The tasks are adapted from HarmBench; the licence of the HarmBench behaviour datasets is unresolved and is not presented as resolved. Polish texts are model-generated and derived from NonCommercial material.

---