PREVIEW BUILD · IMAGES + INTERACTIVE FIGURES EMBEDDED · NOT THE LW SOURCE

Consumers, Not Representations — and How We Convicted Our Own Nulls

Post 6 of the Legible Networks series (assumes post 4 and post 5). Sources: the writeups Consumers, Not Representations: The Wiring Problem for Latent Knowledge and The Gain Ledger: Consumers Are Rationed by Gain, Not Existence in the explainable_ai repo. This post tells two papers as one story because the second exists to correct the first — and the correction is the best part. Interactive figures continue the series numbering.

Epistemic status: a full arc of attempt → failure taxonomy → self-correction, with pre-specified closure tests as the arbiter of every claim, seed replication (n = 8 finetunes) on the keystone result, and the final laws stated after the corrections — so they are the versions that survived our best attempt to kill them. One specimen network for the main arc; the Dyck conviction replicates the reversal on a second substrate.

The puzzle

Interpretability's standard success — "we found a direction that encodes X" — leaves open the question that determines whether the finding matters: does anything inside the model read it? A representation with no reader is a beautifully labeled wire connected to nothing.

Posts 4 and 5 left us holding three such wires: a language model's exact depth counter, its ordered bracket stack, and the subject of this post — a chain-safety direction in KataGo's trunk, sitting unread (we believed) in a network whose famous adversarial failure is precisely a failure to act on group safety. This post is the experimental program that grew from one question — given a verified representation with no reader, what does it take to build one? — and it ends somewhere we didn't expect: with a stricter certification protocol convicting two of our own "unread" claims, and a quantitative law replacing the orphan story. We're publishing the errors alongside the law because the errors are load-bearing for the method.

1. The specimen

The chain-safety direction fell out of post 4's direction-automata program as the single shared residual of 50 channels unexplained by the known machines. Three facts made it a worthy specimen:

• It's real off its home family. On real human games (KGS server records), its per-point projection separates stones of chains at ≤2 liberties — groups in genuine crisis — from safe chains at Cohen's d = 0.64 (the verdict-axis control manages 0.15; random directions 0.05). High reads "secure," low reads "in crisis."
• It's not a rediscovery of post 4's urgency direction (cosines inside permutation nulls at every depth).
• It appeared orphaned. Injecting it at the last trunk block — full-board or masked to a specific chain, both directions, with permuted and wrong-site controls — moved neither policy attention nor the ownership read beyond noise.

A readable, generalizing, apparently-unread safety feature, in the trunk of a network that loses games for not defending unsafe groups. So: can we wire it in?

2. Attempt one: let training build the reader

Post 5's precedent was encouraging — there, a brief finetune wired a reader onto the exact pre-registered direction, null-to-0.997. Run the analogous recipe here: teach the policy to play out liberty races correctly. Two runs, two lessons.

Engine labels (500-visit search as teacher): training first fixes the blind boards (race attention 0.009 → 0.99) and then, as it converges on the teacher's actual labels, unfixes them (→ 0.006). The engine is itself blind beyond ~3 liberties; its labels for high-liberty races say "play elsewhere"; distillation faithfully reproduces the teacher's blind spot. A system whose training signal is its own search cannot be finetuned out of a blindness its search shares — arguably a one-line theory of how the blind spot survived self-play training in the first place. Anyone proposing "just finetune on the model's own judgments" should sit with this run.

Arithmetic labels (the exactly-known race rule, only provably-won positions, plus rehearsal batches of the original policy's own choices): the repair is almost free — race attention 1.000, general-play agreement 1.000 throughout — and it transfers: misreads of dead cyclic groups drop from ~9–10/10 to 4/10, the ownership head improving through shared trunk features. The one intervention in this whole arc that broke through imported truth from outside the system.

But the closure test — does steering the safety axis now move behavior? — is null. The successful finetune built its reader out of something else (plausibly the existing urgency machinery) and left the pre-registered axis exactly as unread as before. Where Qwen's finetune adopted our direction, KataGo's routed around it: a finetune builds a consumer, not necessarily your consumer — and nothing but a closure test tells you which happened.

3. Attempt two: surgery

So build the reader by hand. One rank-1 connection — policy_logit(p) −= α · ŝ(p), with ŝ the per-point safety projection standardized per board — 128 weights (the axis itself) plus one gain: 129 parameters. By construction the policy now consumes the axis. The question is whether the wired consumer is any good.

At the attention level, yes — strikingly. Crisis-liberty attention rises monotonically with dose while safe-liberty attention stays flat; the permuted control moves everything the wrong way. Best of all: attention on the liberties of the misread dead cyclic groups — the exploit's groups — rises 2–7×. The ownership head calls those groups alive; the safety axis reads them as endangered. The trunk had been carrying a dissenting, more-correct opinion about the exploit groups all along, on a wire nothing read — and 129 parameters suffice to surface it. Engine-judging every move the graft changes prices the wire at ~0.3% winrate per changed move (median flip exactly neutral; half the per-flip damage of chance surgery). The graft's limit is equally informative: the 4–5-liberty races get worse under it, because the axis's own validated resolution is "≤2 vs ≥4 liberties." A wired consumer is exactly as good as the representation it reads.

At the outcome level, no. In 120-ply playouts the grafted attacker captures the misread groups no more often than the ungrafted one (0–2/10 everywhere). The mechanism is arithmetic: the wire moves probability mass (0.05 → 0.20 on the right liberties) but rarely the argmax (24 flips per 100 positions) — and cashing a 40-stone capture requires ~20 consecutive directed moves. A logit-level nudge shifts distributions; it cannot sustain a campaign. Surgery points; it cannot campaign.

4. Attempt three: give the mass to search

Mass-not-argmax is precisely the problem search exists to solve — PUCT (the AlphaZero-family tree search) allocates visits by prior mass and value backups. Build a minimal PUCT on the reconstruction, priors from the grafted policy, values from the network's own value head, re-run the dead-board playouts with the attacker searching.

Null. 1–2/10 captures in every condition, plain or grafted. And the reason closes the mechanism with a detail worth savoring. Why can't the tree simply reach the capture — after which the value head would see cleared stones and concede the point? Because the misread doesn't merely withhold the final reward; it poisons every prefix of the path to it. The capture takes ~6–8 consecutive attacker moves. At each intermediate node, the value head — believing the group alive — scores the attacker's liberty-fill as a wasted move, a tempo loss, so each step along the correct line backs up as slightly worse than its alternatives. PUCT allocates visits by those backups: a line whose every prefix looks bad receives exponentially fewer visits per additional ply, so the one node that would refute the value head sits beyond an effective horizon that grows faster than the search budget. Search amplifies its value function; it does not overrule it. (Consistent with the adversarial-Go literature, where the attacks defeat victims searching at orders of magnitude more visits than ours.)

Taken with §2: the policy is blind, the self-play teacher is blind, and the value function is blind, about the same positions. The blind spot is defended in depth — no internal component dissents at decision time (the one dissenter has no vote) — which is simultaneously why the adversarial attack works and why every internally-sourced repair fails. All roads run through a component that shares the misread.

5. Attempt four: composition — and the control that makes it meaningful

The finetune builds a reader but picks its own substrate; the graft picks the substrate but can't campaign. Compose: install the wire as trainable structure (direction initialized to the axis, gain −0.5, its own learning rates) and run the successful arithmetic-label finetune on top. Control: the identical run with the wire's direction initialized randomly.

The composition works. Training retains and tunes the safety wire — the gain trims 7% and stabilizes, the direction ends at cosine 0.993 to the axis — while behavior is fixed at intact generality (race attention 0.91, agreement 1.000). And the closure test finally comes back positive — concretely: push the axis toward "crisis" for a given chain, and the policy's attention on that chain's liberties now rises, with a median shift of +0.034 in probability mass and 97% of tested chains moving in the correct direction. (For scale: total policy mass on a crisis chain's liberties runs around 0.05 in the companion measurements, so this is a large, chain-specific response, not a tremor.) The identical steering applied to the plain finetuned model of §2 produces no consistent shift at all — signs near coin-flip, median near zero. And the random-initialized wire, after the same training? Kept — but rotated toward the safety axis by nothing: cosine 0.047. The composed model is the arc's best artifact: behavior fixed, generality intact, and the identified representation demonstrably consumed — the combination that training alone, surgery alone, and uncommitted training each failed to deliver.

6. The seeds sweep: the kill-test the conclusion owed

Single runs invite single-run conclusions, so: initializations at cosine {1.0, 0.7, 0.4, 0.0} to the axis × two seeds each, closure as arbiter. Three refinements and one retraction:

• Training never re-points. Final cosine equals initial cosine at every level (0.994 / 0.698 / 0.409 / 0.008 across eight runs). The wire's direction is frozen where the surgeon left it; what training tunes is the gain — the same currency, it will turn out, as everything else in this post.
• Approximate pointing suffices behaviorally. A wire at cos 0.4 — 92% noise — still yields complete race repair (0.999) and 87%-positive closure; closure strength scales gradedly with the projection (Δ 0.160 → 0.157 → 0.114 → 0.076).
• The retraction: at cos 0, the two seeds split — race repair 0.010 vs 0.844. Our original single-run claim "training never discovers the axis alone" was one side of a coin flip, and we retract it to exactly that.

The corrected conclusion: pointing buys reliability and ownership, not possibility. Pointed wires succeed deterministically (2/2 at every level down to cos 0.4) and wire your axis in proportion to the pointing; unpointed wires succeed sometimes, by routes you neither chose nor certified. The practical reading is friendlier than our original claim: discovery methods only need to get warm (cos ≈ 0.4) for composition to work — but only precise pointing makes the resulting consumer demonstrably yours.

7. The gain ledger: putting our own nulls on trial

Everything above rested on a foundation stone: the axis is orphaned. That's an absence claim, and post 4 taught us the standard absence claims must meet. So we wrote the certification protocol we wished we'd had, with three gates:

1. Propagation — the written value must be readable by a downstream probe, else a behavioral null is vacuous (you may have written into the void).
2. Dose window — effects must be measured at doses where magnitude-matched random perturbations are quiet, else you're measuring generic disruption.
3. Matched positive control — a feature known to be wired, written at the same norm, must move behavior, else your instrument has no demonstrated power.

First finding: additive vector writes — the field's default steering instrument, and ours — fail the protocol's own validity gates on the folded-Qwen testbed. The folded residual stream is tiny (mean state norm 0.08, so a one-unit synthetic write is a third of the entire state); feature carriers are collinear (depth and flag at cosine 0.95 — twelve-token strings hold their features in one thin subspace); and there is no dose between "propagates" and "off-manifold." Method rule, now written in stone: patch with real state differences from actual inputs; never write synthetic vectors on short synthetic prefixes.

Run properly, the protocol convicts both testable orphans:

• The Dyck depth counter (post 5's thrice-cited "behaviorally inert" signal): a 1-D patch along the depth carrier, built from real state differences between depth-2 and depth-4 cohorts, moves the close-vs-open gap 1.8 logits toward the donor cohort — 5× the magnitude-matched random control — and the natural −0.32 logits/depth gradient turns out to be the network reading its counter, which a per-string AUC readout had variance-swamped.
• The safety axis (this post's founding orphan): a crisis→safe real-state patch at a crisis chain's points reduces crisis attention by ~19% (−0.0091 ± 0.0033 on a 0.047 baseline), direction-correct, 56% of a full-state patch's effect, paired axis-vs-random contrast t = −2.43 (n = 320), with the propagation gate passed (the block-9 axis probe's reading moves 34 → 8 under the patch; the random patch leaves it at 33). The founding steering null was the instrument's failure, not the network's deafness.

The orphan story is dead. What replaces it is quantitative.

8. The corrected laws

Gain is feature- and context-conditional. With the conviction in hand — the axis is read, natively — two questions follow: how strong is that natural read, and how does our hand-built graft compare to it? Answer both on one footing: the same 240 crisis chains, the same metric (policy attention on the chain's liberties), interventions swept over dose.

The natural consumer is strong on its home turf. Delete the axis from the state — project it out entirely — and crisis attention falls 21%, with zero collateral disruption to anything else. On ordinary positions, the network is genuinely, substantially using this feature.

The graft cannot match that, at any dose you'd accept. At a gentle dose (α = 0.5) the graft moves crisis attention by |Δ| = 0.0022 — about a fifth of what deleting the natural read costs — while already changing 23% of the model's chosen moves. Crank to α = 4 and it reaches 0.0083, still short, having now changed 68% of all moves. There is no dose at which the graft delivers the natural consumer's effect at an acceptable price; the "equivalent dose" doesn't exist.

And past small doses, the graft isn't even acting through the direction. A permuted-direction control at matched dose shifts attention 2–5× more than the true axis — meaning the graft's large-dose effects are generic disruption wearing the axis's name.

Which raises the obvious puzzle: then why did §3's graft look so good — the celebrated 2–7× attention gains on the exploit groups? Because those were measured on the cyclic exploit family — exactly the context where the natural consumer's gain is ~zero. On ordinary positions the natural read dominates and the graft is noise; on the blind family the natural read is silent and even a crude graft is the loudest voice in the room. Same feature, same network, opposite verdicts by context.

The law in final form: the trained consumer has feature-conditional gain — strong where training exercised it, zero exactly on the blind-spot family — while a logit-level graft has crude, uniform gain that wins only where the trained gain is zero. This retroactively explains the whole arc: why the raw graft priced out as a winrate tax on normal play (it was shouting over a working consumer), and why graft-initialized finetuning — letting training re-tune the gain schedule itself — was the one composition that worked.

Consumers read values, not volumes. The wire-depth sweep adds the second axis: where an intervention enters. Amplifying the axis's own coordinate in place — making the feature louder without changing which side of the class boundary it's on — does nothing at the consumer's own depth (block 7: Δ = −0.0002 at a KL cost of 0.33¹) on the very same cases where a value-moving patch at the same depth bought −0.0091 at KL 0.016. The efficiency table across every instrument tried on this cohort: value-patch at block 7, 0.57 |Δ|/KL; amplification at block 9, 0.10; early-block amplification, ≤0.03 and mostly permutation-matched noise; the logit graft, noise-dominated. The operational rule: move the feature's value across the class boundary, at the depth where the consumer reads it. Everything else is shouting.

🕹️ Interactive Figure 11 — Value, not volume. Two tabs. Value vs volume: a feature reading, a consumer's decision boundary, and two interventions — amplify (slide the state outward along the feature axis: louder, same side, consumer unmoved, KL bill climbing) versus a value patch (cross the boundary: consumer flips at a fraction of the cost). The gain schedule: the same feature read through a per-context gain dial — healthy on ordinary positions, zero on the exploit family — with an on-distribution behavioral eval overlaid to show why it certifies a model whose schedule has a hole in it. Schematic by design; the measured numbers are in the text.

Consolidation is gain tuning. The seeds sweep's "training never re-points" now reads as the same law from the training side: gradient descent adjusts how loudly wires are read — per feature, per context — and essentially never which direction a wire points. Discovery of directions is the interpreter's job; training only sets the volume knobs.

The frame that falls out. Blind spots — the kind that matter, the exploitable kind — are not missing circuits. They are zeros in a learned, per-context gain schedule attached to machinery that works everywhere else. Three consequences, each safety-relevant:

1. Behavioral evaluation cannot find them on-distribution, by construction — on-distribution is precisely where the gain is healthy.
2. An adversary needs only to find the context where the gain is zero — not to break any representation, just to play in the silence.
3. Repairs that don't touch the gain schedule can leave the zero in place — including finetunes that look completely successful (§2's closure null), and including any repair whose teacher shares the schedule (§2's engine-label run, which restored the zero after fixing it).

9. On publishing your own convictions

The sociology, briefly, because we'd like it to be normal. The Gain Ledger exists because we went looking for the strongest way to attack our own two most-cited exhibits, found it, and published the convictions with corrections marked inline in the earlier papers.² The corrections cost us a slogan ("orphaned representations") and a clean single-run claim ("training never discovers the axis"), and bought a law with more structure than either. Note what survives the audit: post 4's eye-integration absence — because it was certified against the matched-positive-control standard from the start, which is a different and sturdier kind of claim than a steering null. Behavioral nulls from steering are, we now think, close to uninterpretable without propagation gates and matched-dose yardsticks; absences certified with controls stand.

For interpretability-guided training — the program of "find the concept, then make the model use it" — the arc's ground truths as we currently hold them: it is possible; it is cheap (129 parameters and 200 small steps, here); it demonstrably does not happen reliably by itself; approximate discovery suffices but precise pointing is what makes the consumer yours; and its ceiling is exactly the quality of the representation you wire.

Next: two shorter methodological results from auditing agents at scale — why ~98% of "drive" inventories are artifacts of the baseline you audit against, and a measurable condition for when inference-time search moderates a policy's tendencies versus adding nothing. Post 7: Audit Your Audits.