PREVIEW BUILD · IMAGES + INTERACTIVE FIGURES EMBEDDED · NOT THE LW SOURCE

The Stack the Model Never Read

Post 5 of the Legible Networks series (post 4 told the Go version of this story). Source: the writeup The Stack the Model Never Read: Latent Structure, Its Mechanism, and Its Elicitation in a Language Model in the explainable_ai repo. All experiments run on the exactly function-preserving unit-budget reconstruction of Qwen2.5-0.5B from post 3, with inputs built at the token-id level so byte-pair merges can't pre-digest the structure. Interactive figures continue the series numbering.

Epistemic status: the strongest experimental design in the series — exhaustive input families (every bracket string up to a length bound; no sampling questions), every instrument calibrated on ground truth before it touches the model, and interventions with matched controls throughout. One early claim ("the counter is behaviorally inert") was corrected by our own later certification protocol; the corrected version is presented inline, marked, with the full story in post 6. One model, one scale, one grammar family — the limitations section of the source paper is worth reading in full.

The puzzle

Ask what formal machine a language model implements for nested structure. We asked at three levels of description — its internal directions, its individual units, and its behavior — and got three different answers. The disagreement became the finding:

• As a representation, the model carries a genuine soft stack in its residual stream — slot-addressable, pop-surviving, distractor-robust, causally writable, believed by the model's own downstream layers.
• As a behavior, the model tracks the stack's top only at depth 1 and is at chance by depth 7, because its output circuit reads a cheaper structure entirely: an unordered bag of owed closers.
• After ~100k tokens of targeted finetuning, a consumer for the stack gets wired in, conflict accuracy jumps from chance to 0.93, and a steering intervention that was a pre-registered null becomes 0.997 behavioral control — through the exact direction identified beforehand.

We think this is a complete, causally-verified instance of the eliciting-latent-knowledge problem in a real model, with the elicitation actually performed. The design law it suggests: representations arise cheaply as byproducts; what training rations is consumers.

1. Making the Chomsky hierarchy measurable

The vehicle is bracket strings — Dyck languages, the formal-language name for balanced brackets: Dyck-1 has one bracket type ( ), Dyck-2 has two, ( ) and [ ], where each closer must match its opener. They're the canonical vehicle because the Chomsky hierarchy hands you unambiguous ground truth about what machinery each predicate requires: some properties need only finitely many states (regular — a lookup table over a finite memory), some need an unbounded counter (how deep am I?), and some need a full stack (what exact sequence of closers do I owe, in order?). And exhaustive families — every string up to a length bound — kill sampling questions dead.

First, an instrument. For any yes/no signal over strings, compute a lower bound on the states of a deterministic automaton consistent with its verdicts on all strings up to length L: find prefixes that some shared continuation provably distinguishes (they lead to different verdicts), in the spirit of Myhill–Nerode — distinguishable prefixes must occupy different states, so counting them bounds the machine from below. Plot that bound against the horizon L, and the Chomsky tier becomes a curve shape, calibrated exactly on ground truth:

🕹️ Interactive Figure 10 — The tier diagnostic, computed live. Pick a predicate and the widget computes the distinguishable-prefix bound in your browser — no canned numbers — reproducing the table above: flat for parity, linear for the counter, doubling for the stack (with a log-scale toggle that turns the doubling row into a straight line). One instrument, three Chomsky tiers, separated on finite data.

One honesty note that matters downstream: noise inflates this bound (misclassified strings masquerade as state structure), so on continuous model readouts it certifies only "at least," and the capacity claims below rest on decoding accuracy instead.

2. The counter: three answers at three levels

On Dyck-1 (all 8,190 strings, length ≤ 12), nesting depth — the counter — lives in the model at three different fidelities simultaneously:

As a direction: essentially exact. Ridge regression finds depth linear in the residual stream at R² = 0.993, with held-out adjacent-depth AUC 1.00¹ — the code distinguishes depth 4 from depth 5 without error — present from layer 4 of 24 onward. (A later sharpening, which becomes post 8's opening theme: the information is present from layer 4, but the encoding direction rotates across depth — cos(layer 4, layer 24) = 0.055 — and yet a single unsupervised linear alignment of layer states transports the probe across twenty layers at 96–99% of ceiling. No fixed direction carries the counter; a coherently drifting frame does. All steering below uses directions fit at the injection layer and is unaffected.)

As a unit: a bounded shadow. The best single unit of all 22,400 (|r| = 0.84) decays compressively — equal depth increments produce shrinking activation steps, a Weber–Fechner code, like human perception of loudness — and its extracted machine plateaus at 11 states: a bounded counter, regular in the limit.

As a behavior: barely consulted. The close-versus-open logit gap distinguishes depth 0 from positive and then, per-string, nothing. (Corrected 7/25 — the original reading "memoryless" was a readout artifact. The stricter certification protocol of post 6, patching with real state differences rather than synthetic writes, shows the gap's mean gradient with depth — −0.32 logits/depth of graded caution — is real and causally driven by the depth carrier, at 5× the magnitude-matched random control; per-string AUC sat at chance only because within-depth variance swamps it. The counter is weakly consumed, not orphaned.)

So: super-regular at the direction level, regular at the unit level, weakly-consulted at the behavioral level — post 4's slogan (automata belong to directions; channels are shadows) with a third rung added below.

And a sharp contrast case that previews the whole series' corrected law. The other half of the validity machine is the sticky "went negative" flag: the bit that turns on permanently the moment a prefix closes more brackets than it ever opened (once invalid, always invalid). It too is represented — decodable at AUC 1.00 from layer 2, even earlier than the counter, as befits an attention-cheap sticky OR.

But unlike the counter, the flag has a strong consumer, and the test for that is worth spelling out. There is exactly one situation where the flag changes what the correct next token is: depth 0. On a still-valid prefix at depth 0, closing is illegal (it would go negative), so a model consulting the flag should lean hard against closing; on an already-dead prefix at depth 0, nothing is at stake either way. Measured: the close-vs-open gap sits at −0.43 on valid depth-0 prefixes (leaning open — correct) and at +5.00 on dead ones — a +5.4-logit swing driven by the flag, precisely at the one place the flag is decision-relevant. And at matched mid-depths — where closing is legal regardless and the flag shouldn't change the choice — the flag's value moves the gap not at all. The flag is consumed exactly where, and only where, consuming it pays.

Two components of one machine, maintained side by side: the flag strongly wired (+5.4 logits at the boundary where prediction pays most), the counter weakly wired (−0.3 logits/depth of graded caution). Consumers are rationed — by gain, not by existence.

3. The soft stack

Dyck-2 (all 87,380 strings, length ≤ 8, invalid strings included — which continuations remain valid is exactly what reveals a stack) upgrades the target from a number to an ordered sequence. The stream holds it:

• The k-th-owed closer decodes at 0.979 / 0.930 / 0.862 / 0.819 / 0.722 across stack slots 1–5.
• It's retrieval, not echo. Here's the confound this control exists to kill: if the last token of the prefix is an opener — say [ — then "the top of the stack is [" is readable by simply echoing the most recent token, no stack required. The control: restrict to prefixes that end in a closer. There, the last event was a pop, so the current top is an opener pushed several tokens ago that has just become top again by surviving that pop — and any reader that merely echoes the last token now fails, because the last token is a closer, not the top. Under this restriction the direction still decodes the top at 0.965. The best apparent top-of-stack unit (|r| = 0.843 on unrestricted data — a very convincing-looking neuron) collapses to a coin flip, 0.505: everything it "knew" was the most recent token. Genuine post-pop retrieval exists at the direction level and at no unit whatsoever.
• It survives distractors: with an English sentence prepended, the summary decodes at 0.997 — better than clean.
• It's causally writable, slot by slot: steering the top-of-stack direction flips the network's own downstream reading of the top (a decoder eight layers later, fit on clean states) bidirectionally at 0.992 / 0.017, while the slot beneath survives at 0.76–0.81. A norm-matched random vector achieves neither — and destroys slot 2. Slots are approximately independently addressable coordinates.

By every internal test we could devise, the network believes its own stack.

4. The dissociation, and the hunt for the reader

Behaviorally: the model prefers the correct closer at 0.92 for depth-1 stacks, 0.70 by depth 3, chance by depth 7. And the obvious handle doesn't move it — last-position steering of the very direction the network's own layers believe shifts the choice no better than matched noise.²

Three hypotheses for what the output circuit actually reads, each with a designed experiment, each a failure:

• Hypothesis 1 — attention matching. The canonical mechanism everyone expects: some circuit attends from the current position back to the matching opener and copies its identity (the induction-head style story). If that's the reader, then the opener's internal representation is causally load-bearing — so swap it. Take a string; build its twin: the identical string with just the top opener flipped ( ↔ [ (still grammatical, and the correct closer flips with it). Patch the opener position's activations from the twin into the original. If behavior reads that opener, the answer should flip to the twin's answer. Result: the fraction of prefixes on which the model's choice follows the twin's correct answer rises from 0.295 to 0.409 — real and position-specific (patching a control position does nothing), but eleven percentage points of choice probability, where the hypothesis predicts crossing to a solid majority. Whatever the opener representation contributes, it's a minority sharehold in the decision.
• Hypothesis 2 — a matching head. Maybe the mechanism is literal: one attention head that looks from the last position back at the top opener. Scan every head in the model for exactly that pattern. The best candidate attends to the opener at only twice the uniform baseline — barely above "not looking anywhere in particular" — and it lives in layer 0, far too early to implement matching. No such head exists.
• Hypothesis 3 — interference. Maybe the stack summary is only cleanly readable in our sterile bracket-only inputs, and in anything resembling real text it drowns — so behavior couldn't use it even if it wanted to. The distractor test says the opposite: prepend an English sentence and the summary decodes at 0.997 — better than clean — with behavior unchanged. The representation is robust; the excuse is not available.

The three failures share a shape: behavior is insensitive to every single position (patch one, almost nothing; steer one, nothing) and exquisitely sensitive to all positions at once (the every-position injection flips 97%). That shape has a name.

5. The bag

The readout that fits: an unordered bag of owed closers — sum over positions, no order. Bag and stack agree at depth 1 and diverge below, matching the accuracy curve. And a bag makes a prediction a stack flatly contradicts: define the conflict prefixes as those where the true top-of-stack disagrees with the majority of owed closers (you owe ] ) ) — the stack says close ], the bag says )). A bag-reader follows the majority there. Measured, on data already in hand:

Conflict-prefix accuracy: 0.500 / 0.473 / 0.492 / 0.483 at depths 3–6 — exact chance, everywhere — versus 0.76–0.95 when top and majority agree.

The fitted readout, with terms glossed: gap (the logit preference between the two closers) ≈ −2.33 + 0.77·(top identity) + 3.53·(bag composition) — the truth's coefficient a sixth of the bag's, the bag carrying 6× the variance, and the intercept a standing lean toward ].

🕹️ Interactive Figure 9 — Stack vs bag. Build a bracket prefix and watch three readers disagree: the true stack (ordered, LIFO), the bag (two tallies, no order), and the model's fitted readout. The conflict lamp lights when top and majority diverge — the prefixes where the pre-finetune model drops to chance — and the pre/post-finetune toggle swaps the coefficients (bag 3.53 → 1.34, top 0.77 → 6.00) to show the same prefix read correctly after §6's elicitation. A generator button samples conflict prefixes for you.

Even the intercept is economics rather than pathology. Raw token frequency predicts the opposite lean () outnumbers ] 2.6:1 in a 2.2M-character sample) — but bracket spans run the right way: square brackets close at a median of 5 characters vs 13 for parentheses. Conditional on an open bracket somewhere behind you, ] arrives sooner — so "lean ] when you've lost track" is the rational hazard-rate prior for exactly the deep-stack situations where a bag-reader has lost track.

One equation retrodicts seven previously separate measurements: the depth-1 accuracy and its decay (bag = top at depth 1); the ]-bias; the all-positions steering flip (every bag contribution shifts at once); the last-position steering null (one position is a 1/L shareholder in a sum); the twin patch's +11 points (one bag element's worth of logits — flips only marginal cases); the absence of matching heads (a sum needs only diffuse attention); the distractor robustness (counting features don't care about interleaved text). When your cheap-heuristic hypothesis predicts your steering nulls to the decimal, you can stop arguing about whether the representation "really" drives behavior.

Why build a pristine stack and read a bag? Because on natural text — where deep nesting is rare and inconsistently closed even by humans — top and bag agree almost always; the bag is cheaper to read, and the loss never paid for the difference. The stack exists as a byproduct of whatever local bookkeeping attention performs; the reader was never rationed the gain.

6. The elicitation

If the reader is missing only because the bag saturated the reward, modest targeted pressure should build it out of the existing representation, fast. Finetune on generated deep-nesting strings (length ≤ 32, depth-biased; 400 steps × 32 strings), tracking the conflict set and the readout decomposition:

The consumer is built between steps 100 and 150 — roughly 100k tokens — after a transient in which the old readout is wiped before the new one assembles. The coefficients trade places; the ]-bias is erased. And the closure test: the steering intervention that was a null before the finetune becomes near-total, fully specific control. (α here is the steering dose — the multiplier on the injected direction, state ← state + α·v. The vector v is the difference between the model's own mean internal states for the two classes — ]-topped minus )-topped prefixes — so α = 1 displaces the state by exactly the amount that actually having the other top would; α = 0.5 is half that displacement, α = 2 double.) Last-position steering now flips behavior at 0.561 / 0.891 / 0.997 for α = 0.5 / 1 / 2, while the matched random control lies flat at 0.02 at every dose. The new reader consumes exactly the direction identified in §3, before the finetune was run.

The control condition calibrates the speed. The same budget spent on close-the-oldest — whose target is the weakly-represented deep slot (0.72–0.86 decodable, vs the top's 0.98) — learns unstably (0.35 → 0.67 → 0.10 → 0.59), finishes at 0.791 vs 0.975, and remains bag-dominated (b_bag +5.23). The finetune did not teach the model a stack; it wired a reader to the stack the model had. Elicitation is fast precisely when the representation already exists.

7. Why this matters

For the ELK-flavored discourse, this specimen has properties the thought experiments usually have to stipulate:

1. Ground truth is exact. Dyck membership is decidable, and the family is exhaustive — no debate about what the model "should" know.
2. The latent knowledge is certified causal, not just probed. The standard caution about probing results (Hewitt & Liang, 2019) is that the probe, not the model, might do the work. Here the network's own downstream layers demonstrably read the steered value — the knowledge just doesn't reach the output.
3. The gap has a mechanism, not a mystery. Not "the model is hiding something" — the output circuit runs a nameable cheaper algorithm whose on-distribution agreement with truth explains, quantitatively, why training never closed the gap. A behavioral eval would rate this model "can't do deep nesting"; the truth is "knows, doesn't consult."
4. Elicitation is cheap, verifiable — and does not automatically generalize. The success signature here is as clean as it gets: a pre-registered direction, a pre-registered null, and the null flipping to 0.997 control through that direction. But the identical recipe applied to KataGo's chain-safety blindness fixed the behavior while the closure test came back null — the repair routed around the identified representation. The honest summary of interpretability-guided training as we currently understand it: a finetune builds a consumer, but not necessarily the consumer you identified — and only a post-hoc closure test tells you which happened. (Plus one warning from that failure mode worth engraving somewhere: distilling from a teacher that shares the blindness first fixes it and then faithfully re-teaches it. Label provenance is part of the mechanism.)

The safety-relevant corollary cuts both ways. Latent capabilities can sit much closer to the surface than behavior suggests — one small finetune from active — and behavioral evals systematically conflate "can't" with "doesn't consult." When a model appears unable to do something its scale suggests it should, consider that the deficit may live in the reader, not the representation.

Next, the paper that put all of this on trial: we had claimed two representations were behaviorally orphaned — the depth counter here and a chain-safety axis in KataGo — and a stricter certification protocol convicted both claims, replacing the orphan story with a quantitative law about gain. Post 6: Consumers, Not Representations — and the Gain Ledger.