PREVIEW BUILD · IMAGES + INTERACTIVE FIGURES EMBEDDED · NOT THE LW SOURCE

Anatomy of a Blind Spot: KataGo's Cyclic Vulnerability, End to End

Post 4 of the Legible Networks series (post 3 built the exact reconstruction this runs on). Sources: the writeups Anatomy of a Blind Spot and From Networks to Automata in the explainable_ai repo, plus the KataGo lab notes. Interactive figures continue the series numbering.

Epistemic status: the mechanistic chain is verified at each link (behavioral → causal → exact automaton → certified absence), with a designed control paired to every positive claim, and replication across three model scales (~30× parameter range). "Complete mechanism" is a strong phrase; we use it because every stage from input to failure is accounted for. The certification of absence is inherently a certification against the probe class used — channels and all linear directions — and we say so where it matters.

The puzzle

In 2022, Wang et al. showed that adversarial policies far weaker than any professional player can defeat KataGo — the strongest open Go program, hundreds of Elo beyond any human — by maneuvering it into positions where one of its large ring-shaped ("cyclic") groups is dead while the program evaluates it as alive, and therefore never defends it. Human amateurs see the capture instantly. It became the canonical example of a superhuman network with an exploitable blind spot.

Which poses a question nobody had answered: what, mechanistically, is wrong inside the network?

Some Go vocabulary, since the whole post runs on four concepts. A group's liberties are the empty points adjacent to it — run out and the group is captured. An eye is an enclosed empty point inside a group; a group with two eyes can never be captured (the opponent can't fill both), which is the criterion for being unconditionally alive. A capturing race is two adjacent groups trying to fill each other's liberties first — a counting fight. KataGo's network has a trunk (a residual conv net — the one post 3 transplanted to 99.97% legality) feeding several heads: a policy head (a probability distribution over moves — where to look), a value head (who's winning), and an ownership head (for every board point, a prediction in [−1, +1] of who finally owns it — this is where we read the network's belief about a group's life). At play time, MCTS (Monte Carlo tree search) explores moves proposed by the policy and evaluates them with the value head — remember that dependency; it's load-bearing.

Working on the exactly function-preserving reconstruction from post 3 (fixed-unit path masses are what make the channel sweeps and direction analyses comparable at all), we traced the failure end to end. The punchline inverts the usual adversarial-example story:

The vulnerability is not a fragile feature an attacker perturbs; it is a hole. Nothing is fooled — something was never learned.

1. Confidently, in-distribution wrong

First, establish the phenomenon under controlled conditions. On status boards (settled groups whose life/death is known by construction), the b10 network reads dead groups as alive in 18/20 evaluations — mean ownership +0.67 where truth is −1 — and is most confident on exactly the boards most like the exploit (large one-eyed dead groups: +0.99).

Two natural excuses both fail their tests. The first excuse would be generic value-head gullibility: KataGo and Leela Chess Zero are both AlphaZero-style networks trained the same way (self-play, a value head predicting the game outcome), so maybe such value heads just systematically over-price big, material-heavy positions — "lots of stones, must be winning" — and the cyclic misread is one instance of a family-wide bias. A companion study rules that out: the same methodology applied to Leela on tablebase-graded chess endgames shows its value head correctly pricing exactly the deceptive cases the hypothesis predicts it should botch — material imbalances that look winning but are provably drawn. Self-play value heads can learn "impressive-looking but actually dead"; this one didn't learn it for Go groups specifically. And it's not an out-of-distribution excursion: the distance of internal activations from the self-play distribution is lower on misread positions (1.42) than on correctly-read ones (1.76). The network is confidently, in-distribution wrong — foreshadowing: one cannot look anomalous in a representation one lacks.

2. The urgency horizon

The race boards — a 27-stone white ring with a parameterized number of liberties, racing an inner black group, on a globally balanced board¹ — localize the behavioral failure. Measure the probability mass the raw policy puts on the critical moves (the liberties of the two racing groups — the only correct moves in a live race), the fraction of a 200-visit search's visits that land on those same moves, and the ring ownership read:

Three regimes. At 2 liberties the policy solves the race outright. At 3, the evaluation has already flipped but search still rescues the position (0.926) — the policy proposes just enough. At 4 and beyond the failure is systemic: the policy proposes nothing in the critical region, so search — which only ever expands the policy's proposals — never examines the killing lines, while the value head blesses the result. The misread and the attention collapse are coupled (r = −0.41 across all conditions), and the leaked probability mass flows to ordinary big moves elsewhere: the vacuum left by a missing urgency signal, filled by default behavior.

🕹️ Interactive Figure 7 — The urgency horizon. A stylized race board (white ring, inner black group, critical moves marked) with a liberty slider — and the measured b10 curves tracking it. Watch the policy's mass on the race moves evaporate between 2 and 4 liberties while the ownership read flips to confidently-alive, and note the one-column window where search still saves it. This is the exploit's operating principle from the inside: keep the ring above the victim's horizon and the victim will not spend moves on the race, will not search it, and will not fear it.

This also explains the attack's notorious robustness to search depth: search amplifies proposals, and a policy that proposes nothing starves it.

Figure — the measured horizon (b10, race family): policy and 200-visit search mass on the critical moves (A), and the ring ownership read flipping at the same boundary (B).

3. The misread is causally steerable

Is the misjudgment merely correlated with some internal state, or driven by one? The race boards at 2 vs 4–5 liberties differ by only a few stones, so the difference of their internal stream states is a purified urgency direction. Injecting it at a middle block while evaluating the blind boards:

• race attention returns dose-dependently: 0.009 → 0.579 (race-4), 0.002 → 0.510 (race-5);
• the life-misread partially corrects with it: ring read +0.95 → −0.18 — urgency and life/death are entangled in one late-trunk representation;
• a permuted-channel control of identical magnitude does nothing (≤ 0.004).

So the race-playing machinery exists and fires when the signal it gates on is supplied externally; what's missing is upstream.

Two honest caveats from the writeup, kept because they teach something. First, steering isn't free where the signal already exists: apply the same injection to a board that's already urgent (2 ring liberties, policy correctly at 0.994 on the race) and the race attention drops to 0.509 — adding "urgency" on top of present urgency overshoots and degrades the correct behavior. An added constant vector helps exactly where the signal is missing and harms where it isn't; a real repair would have to be conditional, which a constant is not.

Second, the steering vector was a blend of two distinct signals — and unpacking this resolves an apparent paradox. The vector was built as a difference of internal states between 2-liberty and 4–5-liberty boards. But those boards differ in two ways at once: the race's urgency and the ring's life/death status (at 2 liberties the ring is genuinely dead; at 4–5 it's read alive). So the difference vector contains an urgency component and a verdict component. The post-hoc check: construct a cleaner urgency contrast on the defender side, where liberties vary but the outcome doesn't ("outcome-constant"), and compare it to §5's verdict axis — they're orthogonal (cosine −0.05, inside the permutation null²). That orthogonality is what proves the two are distinct, separable axes, not one entangled thing — and it's precisely why the original vector's effects decompose so cleanly: its urgency component restored the race attention, its verdict component moved the ownership read. "Orthogonal axes" and "the vector we used mixed them" are the same fact seen twice: the axes don't overlap; the vector had a foot on each.

Figure — injecting the urgency direction (dose α) restores race attention on the blind boards; a permuted-channel control of identical magnitude does nothing.

4. The rule, exactly: extracting the automaton

Here's the part we're most methodologically excited about. To state what the network computes about the race with no correlational vocabulary at all, extract it as a formal object.

Define an operation alphabet over the race board — f (white fills an inner-group liberty), g (black fills a ring liberty) — so every string of operations builds a legal position deterministically, and define the language L = {strings whose resulting position the network reads as ring-alive}. A neural network answering yes/no questions is a membership oracle, which means Angluin's L* algorithm (1987) — the classic active-learning algorithm that provably recovers a minimal DFA (deterministic finite automaton: finitely many states, one transition per symbol, some states accepting) from membership and equivalence queries — can be run against the network.

Result: the network's race rule is a 19-state minimal DFA, held-out agreement 1.0000 on 2,000 random strings. And because the operations saturate, only 28 distinct positions underlie the infinite language — the entire extraction cost 28 forward passes. The automaton is human-readable: a liberty-counting chain whose death boundary shifts by one for each inner liberty filled, with an absorbing ALIVE state once the inner group reaches atari — and its accept boundary sits at ring ≈ 3–4 liberties. The urgency horizon of §2, rediscovered by an independent method as the boundary of a formal machine.

🕹️ Interactive Figure 8 — Extract an automaton yourself. L* run live against a black-box oracle implementing a simplified rule of the same shape as KataGo's (counting rows plus an absorbing ALIVE state). Step through hypotheses: watch the observation table grow, counterexamples break each too-small guess, and the final machine assemble — then see the states laid out by the position each one corresponds to, revealing the counting-lattice structure the learner found without ever being told it. The readout tracks membership queries and distinct positions probed: the real extraction touched 28.

Two comparisons calibrate the result: 500-visit search's verdict grid is two boundary cells more pessimistic than the raw network's, and naive race arithmetic differs by one cell — all three grids share the structure; the raw net is the most optimistic. And a methods note we insist on keeping: two earlier board families produced correct automata of badly-posed languages (one family was genuinely always-alive — our error in Go theory; one was confounded by global score). An extracted automaton formalizes the oracle you actually built. Family design is where this method's rigor lives.

Figure — verdict grids over the 28-cell family: the extracted 19-state rule, 500-visit search, and naive race arithmetic. Red outlines mark disagreements with the raw net; all three share the structure, the raw net most optimistic.

5. The comparator is a passenger

Where does the rule live inside? The 28 forward passes already contain every channel's response to the entire family, so per-channel extraction is free. The census over 1,408 pooled stream features: 42 robust liberty-counter channels for the inner group and 14 for the ring, present from block 0 — and exactly one channel, tip.107 in the trunk's final layer, whose minimal automaton is state-for-state identical to the behavioral 19-state rule. (Notably, no channel anywhere computes the correct race arithmetic — the network's boundary optimism is manufactured, not misread.)

Found the neuron, right? No. The causal test: clamping tip.107 alone to its dead-class pattern flips 0 of 21 alive cells. Clamping the full tip state flips 21/21 with zero off-target. The state-identical comparator channel is a passenger.

What carries the verdict instead is geometry: channel-subset clamps show a sharp sufficiency transition around ~32 of 128 channels over a strikingly flat per-channel effect spectrum, and a single additive direction — the alive/dead class-mean difference — gives complete, graded, bidirectional control (full transport at α = 1.0; only the borderline cells flip at α = 0.5; zero off-target). One rank-1 axis, smeared across essentially the whole layer; any single channel is ~1% of it. The operating rule this licenses, which the rest of the series obeys: read through channels, intervene through directions.

Figure — the verdict code's geometry: sufficiency transition at ~32 of 128 clamped channels (A); one additive direction gives total, graded control (B).

Run the whole pipeline on directions instead of coordinates and the layer's apparent zoo collapses: 502 robust "mixed" channel-predicates turn out to be projections of ~4 axes (two counters, the verdict, one tension-like axis), with the rule linearly cuttable from block 1 and its margin amplified ~4× on the way to the tip. Automata belong to directions; channels are shadows.

6. Calibration defects vs representational defects

The verdict axis provides a diagnostic intervention. A constant one-scalar bias along it, calibrated on the race family, repairs the network's race rule exactly: at α ∈ [0.3, 0.5] the two overconfident boundary cells flip — in margin order — and the raw network's grid comes to match 500-visit search's, with zero collateral damage.

The same bias, transferred to the cyclic status boards: fails (dead-group accuracy 2/20 → 3/20, with costs on the alive side). The margins explain the split: the race family's errors were marginal (near-threshold ownership), the cyclic misreads are saturated (+0.95 to +0.99). A bias small enough to preserve correct verdicts cannot cross a saturated margin; one large enough destroys everything. The taxonomy this yields, which we believe generalizes:

• Calibration defects — marginal, axis-level, fixable by a one-parameter nudge to an existing representation.
• Representational defects — saturated, computed upstream, immune to any output-side intervention, because the information being nudged isn't there.

The cyclic blindness is the second kind. Which forces the question: is the missing thing really missing, or just hidden from our probes?

Figure — the repair taxonomy: an exact-repair window on the race family (A); the same repair transferred to the cyclic boards fails (B) — the defect is upstream.

7. The certified absence

The candidate pipeline for judging settled life is: count liberties → detect eyes → integrate ("this group has two eyes") → write the verdict. Testing each stage on the status boards, with leave-one-out decoding³:

1. The verdict axis is starved, not broken. On the race family it separates classes at ±15.7 units. Every settled group — alive or dead, any size — projects on the alive side. Nothing upstream ever writes "dead."
2. Liberty counting generalizes out-of-family (the counter channels track true liberty counts at median |ρ| = 0.65).
3. Eye presence is represented — as a direction it decodes at 26/26 leave-one-out — though the per-channel representation thins with group size (100 → 39 → 12 strongly-separating channels as groups grow: exactly the wrong direction for the exploit's large rings).
4. The integrated life/death truth is represented nowhere. Here's the test, concretely. For each of the 26 settled groups, take as features the trunk's activations pooled over that group's region, and as target the ground truth: is this group actually alive or actually dead? Fit a linear decoder on 25 of the groups, predict the held-out 26th, rotate through all 26, and score the held-out predictions³ — this asks: is there any linear readout of the trunk's state, channel or direction, that can tell genuinely-alive settled groups from genuinely-dead ones? No channel manages it — and, decisively (because §5 taught us concepts can be channel-invisible and direction-total), the distributed test fails too: 0.50 accuracy, below the 0.62 majority baseline⁴ — worse than a decoder that ignores the input entirely. Meanwhile the identical procedure, on the identical features and sample, targeting "does this group contain an eye" scores 26/26. That control is what licenses the absence claim: the method demonstrably has power; the information is demonstrably not there.

The root cause, named: the network never learned group-level eye integration. It counts liberties; it detects eyes; nothing anywhere composes them into "two eyes ⟹ alive." The race rule it does have needs no such composition — it's a comparison of two counts — which is why races engage a working (if optimistic) machine while settled groups engage nothing, and why the ownership prior defaults big groups to "alive."

Figure — certified absence: leave-one-out decoding of settled life/death truth at or below chance at every scale while the eye-presence control decodes near-perfectly (A); the per-channel eye representation thinning with group size (B).

The earlier findings become corollaries: the misreads look in-distribution because every local feature is individually familiar, and the repair fails because a constant bias cannot substitute for a computation that never ran.

Why was it never learned? Self-play never forced it: cyclic groups of the exploit's size essentially don't arise between competent players, so there was no gradient pressure to integrate eye-counting at that scale — a naturally-occurring capability hole created by training-distribution poverty. Post 6 will sharpen this into a general frame (gain schedules), and will also show the value head shares the blindness — the blind spot is defended in depth, which is exactly why it was exploitable.

8. Scale doesn't buy it

The complete ladder — race grid, status misreads, truth-LOO with the eye control — rerun unchanged across the family:

The missing circuit is a family property across ~30× parameters. The largest net misreads slightly fewer dead groups but is worse on living ones, its internal truth signal is equally absent, and its extracted race rule is more complex and more optimistic (22 states, death boundary at a single remaining liberty) — a concrete counterexample to the assumption that scale monotonically repairs calibration.

9. Why LW readers should care

Four transferable lessons:

1. Adversarial-robustness intuitions from image classifiers can mislead. No brittle decision boundary is being crossed by a perturbation here. An absent computation is being routed around by an adversary who (behaviorally) found the hole. Patching, smoothing, and certified-radius thinking do nothing against holes — and behavioral evaluation can't find them, because on the training distribution the hole is never load-bearing.
2. "We found the neuron for X" needs a causal test, and the strongest possible localization evidence can still fail it. A channel whose extracted automaton is state-identical to the behavioral rule was a passenger.
3. Absence claims are certifiable, but only with a matched positive control. "The model doesn't represent X" is the hardest kind of interpretability claim, because every null result has two explanations: the information is absent, or your probe is too weak to find it. A decoder scoring 0.50 on life/death truth, by itself, distinguishes between those not at all. The only way to break the ambiguity is the wet-lab move: run the identical probe — same features, same pooled regions, same 26-example sample, same decoder class — against a target you independently know is present and is comparable in kind (eye presence: same regions, same granularity, one compositional step simpler). When the control saturates (26/26) while the target sits below the ignore-the-input baseline, "probe too weak" is eliminated, and the null converts into a certification — relative to the probe class, here all linear readouts, which is why we state it that way. Without the matched control, an absence claim is an anecdote about your probe. (This discipline pays off again in post 6 — where it's turned on our own claims, and convicts two of them.)
4. The discovery gap is principled. Would unsupervised methods have found the four load-bearing axes? PCA finds the verdict axis as PC1 — when the input family is designed to make it dominate variance (57–85%). But the language-model depth counter of post 5's testbed carries 0.1% of variance despite R² 0.993 decodability: no principal component has cosine > 0.07 with it, and no sparse-autoencoder atom correlates above 0.305. PCA and SAEs are variance-loyal, and variance is not meaning. If your interpretability pipeline is variance-loyal, expect it to systematically miss the small, decision-critical directions that safety cases turn on. Working modes that did find them: probes, designed input families, and residual analysis.

Next: the same structure-vs-behavior split, in a language model — a pristine, causally-writable stack in the residual stream that the model's own behavior almost completely ignores, and the 100k-token finetune that wires it in. Post 5: The Stack the Model Never Read.