PREVIEW BUILD · IMAGES + INTERACTIVE FIGURES EMBEDDED · NOT THE LW SOURCE

A Language Model You Can Read With Arithmetic

Post 2 of the Legible Networks series (post 1: Training Without Gradients). Sources: the unit-net Day 2 report and the j-carve repo (task ladder, carving pilot, and the carving-dynamics memo). As before, interactive figures are hosted separately with static fallbacks; figure numbering continues from post 1.

Epistemic status: empirical, one small distilled model (~3M parameters, TinyStories domain), plus an independent from-scratch character-level model for the carving-dynamics experiments. The central claims are either mathematical (exactness of the lens, verified to max reconstruction error 3×10⁻⁷) or come with predicted-vs-measured intervention effects agreeing to four decimal places. The most transferable claim — that averaged-lens fidelity is regime-dependent — is a two-condition comparison and should be treated as an existence proof, not a survey.

The puzzle

Post 1 ended on a question: the unit-net geometry gives every weight fixed units, so path products through the network are bounded and comparable. On MNIST that meant readable digit templates. What happens if you put a language model in this geometry?

The stakes come from a specific instrument. Anthropic's J-lens result (July 2026)¹ reads what an internal activation "is disposed to make the model say" via a fitted linear transport:

lens_l(h_l) = unembed(J_l · h_l), where J_l = E[∂h_final/∂h_l]

Unpacking the symbols: h_l is the hidden activation vector at layer l; h_final is the last-layer activation; unembed is the model's output projection onto vocabulary logits; and J_l — the lens matrix — is the Jacobian of the final activation with respect to layer l's activation, averaged over a corpus (that's the E[·]). "Jacobian" is a fancier word than the idea deserves: it's just the matrix of sensitivities — entry (i, j) answers "if the j-th coordinate of h_l wiggles a little, how much does the i-th coordinate of h_final move?" Multiplying h_l by it transports the activation to the output's coordinate system, where unembed can turn it into words. The averaging isn't a stylistic choice: a transformer's true layer-to-output map is wildly input-dependent, so any single linear lens must be a compromise across inputs.

Remy's hypothesis going in: in a unit-net, no compromise is needed. With ReLU activations, the true per-input Jacobian is exactly the product of active-path effective matrices, every factor bounded by the geometry (convex excitatory rows, inhibition ≤ 1). The workspace should be readable directly off the wiring — no fitting, no averaging, exact for each input separately.

That hypothesis turned out true, and it enabled three things you cannot currently get from a transformer: an exact lens, the first ground-truthed measurement of how lossy averaged lenses actually are, and concept neurons built to spec with a loss term. Then it opened a fourth door — carving task-specific reasoners out of the network with provable error bounds — which the second half of this post walks through, including the part where the textbook pruning method doesn't just underperform but collapses below random, for a reason specific to this geometry.

Getting an honest student (harder than the science)

First we needed a language model in the geometry. The student, TokenUnitLM: a shared token table (each feature unit holds a convex attention budget over the vocabulary — legible from birth) feeding a unit-net trunk; 12-token context; a reduced 2,048-token vocabulary covering 96.4% of the corpus; distilled from Qwen2.5-0.5B on TinyStories — i.e., trained to match the teacher model's output distribution (KL divergence to the teacher's next-token probabilities²) plus ordinary cross-entropy on the true next token.

It took five runs, and the failures are the instructive part:

Final student: 33.2% teacher agreement@1 (how often its top token matches the teacher's), 25.5% next-token accuracy — a genuinely functional, if modest, TinyStories LM in fully constrained, verifier-legal weights.

The lens results

Exactness. For piecewise-linear activations, the exact lens reconstructs the forward pass with max error 3×10⁻⁷. Read that again: the interpretability artifact and the model's computation are the same object — the lens is not a probe of the computation, it is the computation.

Why does the activation function decide this? A ReLU is piecewise linear: on any single input, each unit is simply open (passes its value through) or shut (outputs zero). Freeze that open/shut pattern and the network from layer l to the output is literally a product of matrices — a linear map. So the Jacobian on that input isn't an approximation of the computation; it equals the computation, gates and all, which is why the lens can rebuild the model's actual outputs to float precision. The price is post 1's trap 1 in new clothing: a ReLU unit that gets driven negative on every input is shut everywhere, receives zero gradient, and is dead permanently — the optimizer killed 484 of 512 units in run 3 exactly this way.

A sigmoid can't die — its slope is strictly positive everywhere, so every unit always passes some gradient. But a sigmoid is curved, and the Jacobian of a curved function is only the tangent line at the current operating point. The sigmoid student's lens is still the exact per-input Jacobian — strictly more than the fitted method approximates — but it answers a smaller question: "how would the output respond to a small perturbation of h_l right here?" Walk any finite distance along that linear story and the curvature makes it wrong; it cannot rebuild the forward pass. That's the design tension in one line: ReLU gives exact global decomposition but risks dying during training; sigmoid lives but reads only locally. The hardened-ReLU recipe (concentrated init, inhibition cap, liveness auxiliary) resolves it: exactness kept, death prevented.

The headline measurement. With ground truth computable, we could finally ask a question that is unanswerable in a transformer: how faithful is the Anthropic-style averaged lens to the true per-input transport? ("Transport" here just means the per-input Jacobian J_l(x) — the matrix that carries layer l's activation to the output frame for that particular input.) The averaged lens replaces every input's transport with one corpus mean, so its fidelity depends entirely on how much the true transport varies from input to input. For a ReLU student it varies only discretely — through which gates are open — so a unit whose gate pattern rarely changes across the corpus has an essentially constant transport, and the average is faithful to it. For a sigmoid student the transport varies continuously: the local slope shifts with every input, so the corpus average is an average over many different matrices — a compromise that matches none of them. That's what the middle column below measures, and the right column shows the consequence:

Same architecture family, same task, wildly different fidelity — determined by the model's representational regime. The uncomfortable implication for anyone consuming averaged-lens results on frontier models: you don't know which regime you're in, and the spread between regimes spans "excellent" to "useless." Corpus-averaging is neither uniformly safe nor uniformly lossy; it's model-dependent, and only a substrate with computable ground truth can tell you which regime you're in.

Workspace signatures, replayed in a readable substrate

The J-lens paper's three workspace signatures, each reproduced here with exact instruments:

• Reportable. For each prompt, the exact lens produces a disposition score for a chosen concept token — the total bounded path mass currently flowing toward, say, ' Lily' — computed from internal activations alone, before any output token exists, and decomposable into per-unit contributions. To check the score tracks something real, take a battery of prompts and ask the teacher model, for each one, how probable it finds that token next; then correlate across prompts: does the student's internal disposition rise on exactly the prompts where the teacher's log-probability of the concept rises? Spearman rank correlation⁴: +0.33 for ' Lily', +0.27 for ' park' — modest, and rising with student quality across runs.
• Controllable. Every intervention's effect on the disposition is predicted exactly before execution — four decimal places, every test we ran. Steering here is not fitted vectors; it's bookkeeping.
• Causal — the dragon neuron. A targeted "ownership" loss (maximize each concept token's exclusive-max margin at its best unit) drove concept selectivity from −0.008 to +0.89 at zero cost in teacher agreement. Unit 35 now carries ' dragon' with path weight 0.836 (runners-up ≈ 0). Ablate it → p(dragon) = 0. Boost it → p(dragon): 0.0002 → 1.0000, with the logit change predicted to four decimals (+0.6615 predicted, +0.6615 measured). Generation degenerates on cue from story-babble to "dragon dragon dragon…"

These are designed, named, precision-steerable concept neurons, created on purpose with a loss term — not found by sifting, built to spec.

Does giving a concept a dedicated neuron break its natural readability? First observation suggested yes: dragon's reportable correlation dropped to ~0 after it gained its unit. The controlled version — twin models, identical except for which three concepts get ownership, every concept measured owned and unowned on a shared 2,048-prompt set — says no. The crossed design gives each of six concepts a matched pair of measurements — its reportability with a dedicated unit and without, everything else identical. Averaged over the six pairs, ownership changed reportability by −0.013 — essentially nothing — and the direction of the change differs concept to concept (some slightly up, some slightly down). So there is no general rule of the form "give a concept its own neuron and it stops being naturally readable." What does survive: dragon specifically dropped again in the twin model, so the original observation was real — and its drop is the largest of the six. But one reliably-dropping concept out of six is a data point, not a law, and dragon happens to be the rarest token of the six, so the drop plausibly tracks rarity rather than ownership per se (a frequency-stratified follow-up is flagged). Practical upshot: ownership engineering is safe to combine with natural readability.⁵

j-carve: cutting task reasoners out of J-space, with certificates

Now the door the exact lens opens. If task-relevance is computable — for any verbalized task, the exact lens assigns every edge and unit a bounded path mass toward the task's answer tokens — then pruning everything below threshold should yield a much smaller network that:

(a) still performs the task; (b) still carries a readable J-space — it can report its own answer disposition before producing it (self-reflective); and (c) comes with a certified error bound, because every discarded edge's maximum possible contribution is known in advance — convex rows + bounded inhibition + activations in [0,1] bound all path products.

Point (c) is the differentiator. Pruning in ordinary networks is empirical: prune, retest, hope. Here the pruning budget is provable arithmetic. And there's a structural bonus: the carved model is renormalized back into the constraint family, so a carve is itself a legal unit-net — everything in this series applies recursively to the carved artifact.

The pilot

Four verbalized tasks, phrased as two-alternative forced choices (2AFC) in TinyStories register (the student has never seen review language, so sentiment becomes "…Overall it was [good/bad]"), each validated against the teacher first. Curves of task accuracy vs fraction of edges kept:

One knob controls each carve: keep, the fraction of hidden units and embedding features retained — the top-keep of each, ranked by exact-lens relevance mass toward the task's answer tokens. Every edge between retained units survives (rows renormalized back to legality); everything else goes. Because each weight matrix shrinks on both sides — rows and columns at once — the parameter count falls roughly as the square of the knob: keep = 0.5 leaves ~11% of the weight entries, and keep = 0.1 leaves under 1%:

The part-of-speech reasoner holds exactly 1.00 from the full 6.95M-entry student down to a carve retaining just 10% of its units — 61k weight entries, a 114× reduction — collapsing to chance at the next notch (5% of units, 22k entries): a sharp carving cliff, not a slope. The other three tasks sit at chance in the full student: a capacity boundary, precisely drawn by the teacher reference — there's nothing to carve yet, and every point of future student quality unlocks more of the ladder. Each curve point also carries its certified |Δz| bound — in the pilot, computed by charging every dropped unit its maximum possible output-path mass, a one-line piece of arithmetic that the geometry makes valid (activations are bounded by 1, so a dropped unit's worst case is simply its output weights' absolute mass toward the task tokens). At aggressive budgets that bound is honest but loose — measured behavior sits far inside it — and the next section identifies precisely the term it fails to charge.

How carving actually works — and how the textbook method collapses

Between the pilot and this post, we ran the carving-dynamics study on an independent testbed: a from-scratch unit-net character LM (4 layers, ~2.17M weight entries, four-domain corpus), carved at fixed budgets of 10% / 3% / 1% / 0.3% of dense, scored by top-1 agreement with the dense net on a held-out battery. Five findings, and the first is the shocker:

1. Naive relevance scoring doesn't underperform — it collapses below random. First-order Taylor edge saliency — expected |weight × input activation × output gradient|, the textbook-correct importance score — scored 0.03 agreement at a 10% budget, below random carving (0.04), while plain global magnitude pruning — the crudest baseline there is: rank every weight entry in the whole network by absolute size, keep the top-k, delete everything else, with no notion of task, row, or position — scored 0.67.

The mechanism is specific to the geometry: carving is not deletion, it is deletion plus forced renormalization of every surviving P row (the carve must stay a legal unit-net, so each surviving row's entries are rescaled to re-sum to 1). Any global score with row-level multipliers concentrates the shared edge budget on a few "hot" rows, and the other rows get starved — allocated almost nothing, in the limit keeping only the one entry that legality forces every row to retain. And here's the trap: a one-entry P row renormalizes that lone entry to weight 1.0. A unit that spent training averaging hundreds of inputs at ~0.01 each becomes a full-strength wire copying one arbitrary input; its activation jumps ~50× above anything it produced during training, and the dense-trained downstream weights faithfully amplify the garbage. Starved rows are toxic, not merely lossy — they don't fall silent, they scream — hence a cliff to below-random rather than graceful degradation.

🕹️ Interactive Figure 4 — The renorm spike. Top: one neuron's budget bar (the motif from Figure 1). Drag the "edges kept" slider down and watch the survivors inflate to keep the bar at 1.0 — at one surviving edge, that synapse is the neuron, and the readout shows its activation on a fixed input jumping ~50× above trained scale. Bottom: four rows sharing one edge budget, allocated either by a global sort (the textbook way — one hot row hoards, three rows starve and spike) or balanced per row (renorm factors stay near 1). Same budget, same scores, opposite outcomes.

2. Magnitude pruning survives by accident, then dies. Because every row sums to 1, entry magnitudes are on a comparable scale across rows, so a global magnitude sort accidentally spreads budget evenly and each row keeps its own head — renorm factors stay near 1. That grace runs out as budgets thin: 0.67 @ 10% → 0.43 @ 3% → 0.07 @ 1% → 0.04 @ 0.3%. (A subtler variant of the same failure: a globally-scored pool starved an entire layer — allocation must be balanced per row, water-filling style, never trusted to a global sort.)

3. The unit of selection is the subgraph, not the edge. What works: decide which units stay alive per layer (ranked by activation × backpropped influence toward the answer), then spend the whole edge budget inside the live subnetwork, balanced across live rows. Dead units still legally require one P-row entry each — and the trick is where to park it: on the constant-1 bias feature (post 1's only bias mechanism, an always-on input appended to every layer). A dead unit then outputs a harmless constant rather than a ×50 amplified copy of whichever input its forced entry happened to land on. And — the load-bearing detail — live rows retain edges only into live units, so even that constant has no path into the computation. Deletion damage becomes containment.

4. Selection alone still computes the wrong function; values must be repaired under the constraints. Call the support the carve's sparsity pattern — which edges and units are retained — as distinct from the values those retained weights carry. Even a well-chosen support scored ≈0.00 at the 0.3% budget before repair: renormalization rescales every surviving row and removed inhibition shifts every threshold — the wiring is right but the arithmetic is wrong. Projected SGD on the retained values toward the dense net's outputs — every step re-clips N to [0,1] and renormalizes each P row over its retained support, so legality is preserved by construction — takes that dead 0.3% carve to 0.56. Repair is cheap (milliseconds per step on CPU at this scale).

5. Repair cannot rescue an incoherent support. Same repair, different supports, at 0.3%: live-subgraph selection → 0.58; global magnitude → 0.25; random → 0.07. A magnitude-chosen edge set is scattered dust — high-weight edges whose endpoints don't compose into input→output paths — and no value optimization pushes signal through paths that don't exist. Support quality is the real currency; repair only cashes it.

The full table (held-out agreement with the dense net):

Two consequences for the certification story. First, the certified bound must charge the renormalization term: for a row with kept excitatory mass m, the surviving terms are rescaled by 1/m, so the error decomposes as

z′ − z = (1/m − 1)·Σ_kept p·a − Σ_dropped p·a + Σ_dropped n·a

and the (1/m − 1) term — bounded and computable from the same geometry, since activations live in [0,1] — is the dominant error source that a discarded-mass-only bound silently misses. That's precisely why the pilot's bounds are loose, and why "prune everything below threshold" produces spikes the naive bound calls safe. Second, the pipeline is carve → repair → re-certify: repair preserves legality at every step, so the certified bound recomputes cleanly on the repaired net. And a diagnostic worth exporting to any pruning work in constrained geometries: expect a cliff, not a knee, wherever a method lacks either containment or repair.

A note on where the geometry itself is heading, because these findings drove a design change. A later revision of the constraint family relaxes "every P row sums to exactly 1" to "at most 1" — implemented by letting dropped mass park on a constant-zero column, the do-nothing sibling of the bias feature (a slack unit whose value is always 0, so budget spent on it contributes nothing). Under that convention a carve never renormalizes survivors upward at all: removed mass simply becomes dead budget, the (1/m − 1) term vanishes from the error decomposition by construction, and the renorm spike becomes impossible rather than merely contained. Readers who continue to post 3 will recognize this as the same lesson the transplant work learned independently as its "never inflate" rule. The findings above are what taught us the relaxation was worth making.

The task ladder, and the compositionality meter

The pilot's four tasks are the bottom of a designed 12-task ladder in three tiers: Tier 1 lexical (sentiment — where negation is literally an inhibitory N-path flipping a positive valence path, making the classic hardest part of lexicon sentiment a first-class visible object — plus register ID, keyword detection, topic); Tier 2 local syntax (POS, NER, chunking, word-sense disambiguation — the purest test of ReLU gating as a carving boundary: same word, two J-spaces, gate decides); Tier 3 compositional stress tests (entailment, coreference, and Winograd schemas as the designed negative control).

The Winograd tier is the part we'd most like LW readers to sit with. Winograd resolution turns on world knowledge ("the trophy doesn't fit in the suitcase because it is too big"), which by construction cannot live in a small lexical subgraph. The prediction is that carving fails to shrink the network much at matched accuracy — and that failure is the measurement: the fraction of J-space a task refuses to release quantifies its compositionality. (By compositionality we mean the degree to which a task's answer depends on relations among several pieces of information — how they combine — rather than on the independent presence of particular tokens or features. "Good/bad after 'not'" is barely compositional: one flip, visible as one N-path. Winograd is maximally so: every lexical cue is identical between the two readings, and only the combination decides.) A compositionality meter built from pruning curves, with the network's own geometry sorting tasks by how much "reasoner" they actually require.⁶

Where this left us

Two conclusions stood at the end of this phase. First, the J-space of a unit-net is exact where a transformer's must be approximated — and the approximation's fidelity is regime-dependent (3% vs 91%) in a way only a readable substrate can measure. Second, all three workspace phenomena reproduce at 3M parameters with interventions that are arithmetic, and the same arithmetic supports certified extraction of task reasoners — once you respect the renormalization dynamics that make naive pruning toxic.

The obvious objection: this is a 33%-agreement toy distilled onto TinyStories; nobody cares about legible toys. We agreed. Which is why the next session abandoned distillation entirely and asked whether you can take a pretrained, full-strength network and make it a unit-net in place, exactly, without a single gradient step. The answer — 97.7% of Qwen's foldable mass, 95.5% of ResNet18, and eventually 99.97% of KataGo — is post 3.

Addendum: j-carve on the transplanted Qwen (added 2026-07-26)

The pilot above ran against the 33%-agreement distilled student, and three of its four tasks were capacity-gated — chance with the full model, nothing to carve. That gate is now gone: the student is the pretrained model itself, moved exactly into the legal family by post 3's transplant machinery (argmax agreement 1.0000 with the original; 70.8% of layer-linear weights — the transformer's linear matrices, the carveable substrate — unit-net-legal). Full-model held-out accuracy: sentiment 0.80, topic 1.00, pronoun 1.00, pos-slot 1.00. Every prediction the pilot could only gesture at got tested. Highlights:

The renorm spike came back wearing RMSNorm. Under the ≤1 relaxation there is no row renormalization left to spike — removal only lowers positive row sums, and the carve is legal by construction. The first run degraded anyway: zeroing a dropped channel shrinks the residual stream, and every RMSNorm reader re-inflates the survivors — the same non-local poisoning, one door down (gentler: degradation, not below-random collapse). The fix is the replacement ladder's first rung from post 3: replace each dropped channel with its fit-battery mean, folded into a bias — a non-budget constant with the same epistemic status as a BatchNorm β. Relevance scoring must then be centered (a channel that never varies is replaced exactly by its mean, so only its variation can matter). Effect at an 11%-params carve: pronoun 0.58 → 0.92. Finding 4's moral — selection alone computes the wrong function — solved this time by folding rather than training.

The headline: task/LM separation and the stratification law, on a real 0.5B transformer. Held-out accuracy vs fraction of layer-linear parameters kept (units of selection: MLP channels and attention heads, per finding 3):

Read the bottom row against the others: at half the parameters, every task sits at or above its full-model accuracy while general next-token agreement is nearly zero. The carve extracts the reasoner and discards the language model around it — the pilot's thesis, now demonstrated where it matters. And the stratification is post 3's highways finding, confirmed with each task aimed at rather than hit by accident: syntax survives to 6.5%, pronoun to 11%, the semantic tasks fall apart below half. (The pronoun-carved model still does pos-slot at 0.75 while everything it wasn't carved for sits at chance — the gender circuit rides the syntax trunk.)

The compositionality meter reads — one tier early, and inside a single task. Splitting sentiment into sent-lex (a valence word is present; echo it, flip it under negation) and sent-inf (event → valence inference, zero affect vocabulary: "The bridge project ran two years late…" → bad) separates cleanly: sent-lex carves like the syntax tasks (0.83 at 11% params); sent-inf is hard for the full model (0.67) and refuses to compress. What the meter measures is not the task's name but how much world-inference its probes demand — exactly the Winograd-tier prediction, arriving a tier early because the probes moved up-register with the student.

Budget shape is a per-task property. Topic's early death turned out to be attention starvation: heads are few, cheap (~12% of layer-linear params all kept), and load-bearing for routing "this passage" to "its topic word." Keeping heads at √f while MLP channels take the full cut revives topic from dead to 0.67 at 14% params — while pos-slot alone prefers the symmetric carve (its circuit is MLP-cheap). MLP channels and attention heads should never share one knob.

Repair replicates — and hits a new wall. Legality-projected value repair (Adam on kept entries only, no-inflate projection every step — post-repair census confirms legality is free) reproduces findings 4 and 5 exactly at 0.5B: a coherent support is rescued (pos-slot at a 2% carve: 0.58 → 0.83 held-out), an incoherent one is beyond help (pronoun at 5%: the name→gender signal simply isn't in the support). The new failure mode is data starvation: batteries of 12–48 probes overfit within a few hundred steps (topic repairs to 1.00 on the fit battery while held-out stays at prior-chance — the support memorizes routes). Repair needs a fitting battery an order of magnitude larger than selection does.

The honest negative, and the door it opens. The pilot's certified bound did not transfer: transformer activations aren't confined to [0,1], the frozen linearization is first-order once gates and attention patterns shift, and the computed bounds — while rank-ordering task fragility correctly — are vacuous as certificates (hundreds of logits against O(1) margins). But the transplant supplies the repair path: RMSNorm's output is architecturally bounded (|x̂_c| ≤ √d·|γ_c| for every input), legal rows turn bounded inputs into bounded outputs by weight arithmetic alone, softmax is a convex combination and RoPE a rotation — both interval-safe — so carving deltas can be propagated as sound intervals end to end. The transplant law didn't just move the weights into the legal family; it made the certificate computable. Building that chain is the next concrete step, along with composing carve → temperature fold → integer quantization on the survivor.