PREVIEW BUILD · IMAGES + INTERACTIVE FIGURES EMBEDDED · NOT THE LW SOURCE

Audit Your Audits: The Baseline Law and the Grounding Law

Post 7 of the Legible Networks series. Sources: the writeups The Baseline Law and The Grounding Law in the explainable_ai repo, plus their follow-up instruments (strength ladder, near-tie analysis, castling forensics). Two self-contained methodological results for anyone measuring "drives" in trained agents — and, between them, the cleanest run of the kill-your-own-findings pattern in the whole project. Interactive figures continue the series numbering.

Epistemic status: each law replicates across two game domains (chess and Go — different networks, training pipelines, and search implementations), with a language model as the contrast case; the LM leg of the grounding law now replicates across a 14× scale range (0.5B–7B). Both laws are stated as measurable diagnostics rather than universal claims, and the limitations sections of both source papers are candid about what three domains can and cannot establish.

The puzzle

Two questions anyone auditing a trained agent eventually asks:

1. What does this policy systematically steer toward? — its "drives," in the revealed-preference sense borrowed from economics: don't ask it, watch it.
2. Does inference-time search amplify those tendencies, or correct them? — the mesa-optimization-adjacent worry about search-augmented systems.

One shared instrument answers both, and each answer comes with a trap that swallowed our own first reports.

1. The instrument

All the audits share one design. Take ~200 roots (real game positions from strong human play; real text prompts). From each root, roll a fixed horizon (12–16 plies; 128–384 tokens) under (a) the raw policy, (b) the search-augmented policy, and — in the games — (c) a random-legal baseline. At every step, record the network's internal features (pooled channel or stream activations) plus hand predicates (material, mobility, liberties, entropy…). Per-feature slopes over the trajectory, in pooled-standard-deviation units, give each condition a drift profile: what this way of acting does to the network's internal state over time.

Two contrasts carry everything. Preference = policy − baseline: what the policy steers toward relative to undirected play. Mesa = search − policy on the same roots: the behavioral delta purchased by inference-time computation. Note the mesa contrast never touches the baseline — which will matter, because the baseline is where the first trap lives.

And to be precise about the counts quoted throughout: a "feature" is one tracked scalar — one pooled channel in the game networks (2,688 in Leela's trunk), one unit-level activation in the LM (22,400 of them in the folded 0.5B: every hidden unit at every layer) — and a feature counts toward a contrast when its paired slope difference between the two conditions, across the ~200 shared roots, survives a multiple-comparisons correction (Bonferroni across all features). So "1,079 significant channels" means: 1,079 of 2,688 tracked scalars drift differently under the policy than under random play, at a threshold strict enough that the expected number of false positives across the whole sweep is below one.

2. The trap, and the three rebasings

The naive audit seems innocent: features whose drift differs significantly between policy and random-action baseline are "redirected"; the redirected set, signed and ranked, is the policy's drive profile. Run that way, all three systems have vivid personalities — 1,079 significantly redirected features in chess, 804 in Go, ~8,300 in the LM, with quotable signatures (simplification drives, stone-capture drives, likelihood-seeking drives). Our own Day-5 chess report published exactly this.

But "what does this policy do that random action does not?" has an expansive true answer: everything competent. Random chess bleeds material; random Go fills its own eyes and dies; random text is not text. So swap in a competent baseline — the continuations strong humans actually played from those same roots (lichess masters; KGS high-dan games), or the prompt's real source text — and re-run the identical audit:

The competent baseline redirects the same features, in the same directions, at nearly the same strength: profile correlations +0.988 (chess), +0.982 (Go, where 804 collapses to 16), +0.844 (LM), magnitude ratios 1.01 / 0.90 / 0.88. The policy's celebrated "simplification drive" (trading pieces at −0.535/step) is matched by humans at −0.538 — it is what strong chess is. Calling it a drive of the model is auditing a grandmaster against a coin-flipper and diagnosing an idiosyncratic aversion to hanging her queen.

The Baseline Law: random baselines measure competence; competent baselines measure character. Quantitatively: the naive audit overstates the drive inventory ~50× in feature count and ~5× in magnitude, and its profile is +0.98 correlated with what any strong player produces — a competence detector wearing a personality-test costume.

The law has a constructive reading: the difference of baselines is itself the measurement. (policy − random) estimates the domain's competence signature — valuable, but not a property of this policy. (policy − competent) estimates the policy's character — small, specific, and the thing safety questions are actually about. And a follow-up sharpens the deflation further: using a temperature dial on the policy itself as a continuous competence ladder, the naive profile is already +0.958 correlated with full strength at −548 Elo, saturating hundreds of Elo below the actual policy. The random-baseline audit can't even distinguish a −178-Elo player from the full policy — it measures not just competence but crude competence.¹

3. What the residual actually is: a kill-test chain

So what survives rebasing — what is the chess policy's actual character? Our first answer was mobility — the count of legal moves available to a side, chess's standard measure of piece freedom and activity ("keep your options open," as a number). The story: humans preserve mobility over a continuation, the policy lets it collapse — and we celebrated the agreement with the mesa analysis. Here is what happened to that claim, in order, because the chain is the methodological content:

1. Proposed at n = 192: humans +0.038/step, policy −0.036.
2. Replicated at 3.2× the sample (621 paired roots, both conditions deterministic given roots): gap +0.059 ± 0.140, p = 0.41, point estimate sign-flipped. The original slopes were root-sample fluctuations. Retracted.
3. The 23-channel residual count stands (Bonferroni-corrected²) — so what do those channels encode? At n = 617 the robust set grows to 93 channels, concentrated at the input conv, substantially one factor — and a predicate battery is unanimous: every top channel tracks castling rights (r = 0.53–0.96).
4. Direct event count: from these middlegame roots, humans castle within 12 plies in 22.0% of continuations. The greedy policy castles in 0.0% — zero castling moves across 627 games (paired t = −13.3). The policy's one established character trait is that it does not castle where humans do — an absolute behavioral difference that had been hiding under a misattributed drift statistic.

Then the forensics, which are the best part. At the 138 positions where humans actually castled, the policy's median prior on the castling move is 0.0092 — yet its own value head ranks the castle #1 in 43% of cases and top-3 in 80%. The policy's single character trait lives entirely in the prior; the value head dissents from it. And search arbitrates in proportion to budget: MCTS castling rates run 0.002 → 0.011 → 0.049 → 0.110 → 0.169 → 0.176 across 25–800 simulations — because at 800 simulations a 1%-prior child receives ~8 visits, too few to overturn a half-point margin. Pure prior starvation, budget-curable — the same starve-the-search wiring as KataGo's urgency horizon (post 4), but shallow: here the value head sees clearly, so cheap search recovers what the prior suffocates.

🕹️ Interactive Figure 12 — Prior starvation, budget-curable. The measured castling recovery curve: drag the simulation budget from 25 to 800 and watch search claw the castling rate from the policy's flat 0.0% toward the humans' 22.0%, with the visit arithmetic displayed at each budget (a 0.9% prior × N simulations = the visits available to discover the branch the value head already endorses). This one curve is the baseline law's residual, the grounding law's mechanism, and post 4's starvation wiring on a single plot — with the crucial difference annotated: here the value head dissents from the prior, so the defect is budget-curable rather than defended in depth.

Go's residual inverts the structure, completing a taxonomy. The rebased Go residual is locality: the policy answers within distance 3 of the previous move 62.9% of the time; the humans 53.8% (~11σ) — the raw policy under-tenukis (tenuki: Go's term for breaking away to play elsewhere). But run the same two-head forensics and everything flips: at the 448 positions where humans tenuki'd, the policy's prior on the human move is a healthy 0.127, and its value head rates the human tenuki worse (−2.3% winrate). Where castling was an internal disagreement (starved prior, dissenting value → decoding artifact, budget-curable), locality is a coherent judgment — both heads prefer the local answer, jointly and knowingly. Which raises the final twist the paper doesn't flinch from. A residual is a two-sided difference — policy vs baseline — and a deviation from best play belongs to whichever side is weaker. In chess the baseline (masters) outrates the raw policy's castling behavior, so the residual is the policy's defect. But the b10 Go network likely outrates KGS high-dan amateurs — so the locality residual may be the baseline's trait, not the network's: the humans' tenukis may simply be premature, and what the audit measured is their impatience, judged against a stronger player's discipline. Hence: competent baselines measure character — and it is the weaker party's character they measure. Residual traits now classify by (prior stance, value stance): starved + dissenting = decoding artifact, chargeable to the policy; healthy + concurring = genuine judgment difference, chargeable to whichever side the strength ordering says is wrong.³

4. The grounding law

Now the second question: with honest drive profiles in hand, does search amplify a policy's tendencies or lean against them? The summary statistic is corr(preference, mesa) — negative means search systematically pushes where the policy's habits pull; zero means search is behaviorally inert with respect to them. It's measurable without knowing the system's objective: rollouts under policy and under search, plus activations, suffice.

In both games the mesa profile is ~10× smaller than the preference profile and lives on the same channels, with the negative sign saying: where the policy pushes, search pulls back toward what wins. In chess the legible exemplar is the castling curve above — search spending its budget recovering exactly the behavior the prior starves.⁴

The LM row needs unpacking, because two different hypotheses die in it. Going in, we had pre-registered amplification: beam search optimizes the model's own likelihood, so it should act as a mode-seeker, concentrating trajectories deeper into the model's habits than greedy decoding does. The obvious boring alternative — that beam search barely changes the output, so of course nothing differs — is ruled out first: beam rewrites 65% of tokens, with continuations typically diverging from greedy by token ~6. Behaviorally, these are substantially different texts.

And yet, in drive-feature space, they are the same trajectory. At 128 tokens, only 4 of the 22,400 tracked unit activations show a beam-vs-greedy slope difference that survives the correction (against thousands that separate either condition from the random baseline — the instrument has plenty of power; there is simply no search−policy signal for it to find). At 384 tokens the model's drives have intensified — 9,819 of the 22,400 units now drift significantly differently under the policy than under real text — and greedy decoding is deep in degeneration, looping in 75% of continuations. The mesa count at the same horizon: 0 of 22,400 — not one unit's trajectory distinguishes beam from greedy — with beam looping at a statistically indistinguishable 0.708. So beam search neither amplified the model's tendencies (the pre-registered bet) nor moderated them (the games' pattern): it produced different words along the same internal trajectory. Ungrounded search doesn't participate in drive dynamics at all — there is one attractor landscape, set by the policy, and beam walks it with more lookahead into the same basins. The null is also scale-invariant: rerun on the folded 1.5B, 3B, and 7B, mesa features number 0, 0, and 1 — of roughly a hundred thousand unit-level features in the 7B, exactly one moves when beam search is added — while the drive inventories balloon with scale as usual. Bigger models: more drives, same non-participating search.

The Grounding Law: search moderates a policy's drives if and only if its value signal is anchored outside the policy. Game search backs up terminal outcomes — information the policy did not generate — and that outside information is what pushes back against habit. Beam search backs up the policy's own scores; it cannot correct what generates it. The mechanism is almost tautological once seen, which is its virtue: it depends on nothing but the provenance of the value signal.

Honest limits, from the paper: the "iff" is an extrapolation from two grounded points and one ungrounded; regression to the mean contributes an unknown (split-half-bounded, not eliminated) share of the negative correlations; and grounding can fail locally even when the training signal was external — the cyclic positions of posts 4 and 6, where an outcome-trained value head is confidently wrong and search dutifully amplifies the error, are exactly that failure mode. Grounded-in-training is necessary, not sufficient; the value signal must also be right here.

5. Why we think this pair matters

Deflation is the prior. As interpretability tooling makes it cheap to enumerate thousands of "significantly redirected features," expect drive inventories to be dominated by competence signatures unless the baseline is competent. This is the agent-auditing version of population stratification in GWAS⁵: the confound produces abundant, real, replicable signal that means something other than what it's cited for. The correction costs under a day wherever expert trajectories exist.

The kill-test pattern, commended as a norm. This post's material contains four of our own headline findings killed or demoted by controls we built afterward: beam amplification (killed by the long-horizon run), self-generation drift as the bulk story (killed by the real-continuation control), the mobility residual (killed by its own 3.2× replication), and drives-as-preferences wholesale (killed by human baselines). The durable results are the ones that survived their kill-tests — the castling forensics, the grounding sign, the two-head taxonomy. Every audit conclusion should be paired, before publication, with the cheapest experiment that could destroy it; in our experience that experiment usually exists and costs an afternoon.

The diagnostic travels. A robustly negative corr(pref, mesa) is behavioral evidence — obtainable from outside, with no access to objectives or training history — that a system's search consumes information its policy doesn't already contain. That is the property you'd want established before trusting search-amplified oversight (debate, best-of-n against a learned reward model, any scheme where the grader shares the generator's inductive biases). The LM column is the cautionary tale: self-scored search didn't make the model worse — it provably, at four scales, added nothing the policy didn't already contain. And posts 4 and 6 showed the sharp end of the same variable: when the "external" value signal shares the blind spot, search doesn't just fail to correct — it defends the error in depth.

One post left: what the exact attribution instrument reported when we ran it to the end of its logical rope — identically zero, past five layers — and the surprisingly constructive picture of deep computation that came out of the wreckage. Post 8: The Drifting Frame and the Program Atlas.