PREVIEW BUILD · IMAGES + INTERACTIVE FIGURES EMBEDDED · NOT THE LW SOURCE

Training Without Gradients in a Weight-Budget Geometry

Post 1 of the Legible Networks series. Everything here is from a single day of experiments on MNIST; code, models, and logs are in the dual-projection-mnist repo¹. The work throughout this series is a human–AI collaboration (Remy driving, Claude at the keyboard); we'll flag where that matters. The interactive figures are hosted separately — links inline — with static fallbacks for readers who prefer them.

Epistemic status: empirical, single dataset, small networks (≈100k parameters), but with matched-budget comparisons, ablations, and self-checking evaluation code. The specific accuracy numbers are solid; the organizing thesis at the end is stated at the strength we currently believe it.

The puzzle

Here is the question we started the day with, verbatim from the session's framing:

Can simulated annealing with beam search — moves of ±δ on a synapse, parallel row perturbation with renormalization, and moving δ of weight between two edges that project to the same neuron — replace backprop, and at what resource cost?

Not "in principle, given infinite compute" (yes, trivially) but at operating points anyone cares about: how many accuracy points does each ingredient of gradient descent actually buy, and are there hardware regimes where blind search wins?

To make the question honest you need a search space where local moves are natural and never leave the feasible set. That's what the geometry is for.

The geometry

A unit-net layer keeps two nonnegative matrices, excitatory P and inhibitory N, with pre-activation z = Pa − Na, under three rules:

1. Every row of P sums to exactly 1 — each neuron gets a unit budget of excitation, spent as a convex combination of its inputs.
2. Every entry of N lies in [0, 1].
3. Inputs in [0, 1] ⟹ all activations in [0, 1], by theorem, not by clipping.²

🕹️ Interactive Figure 1 — A dense layer and its unit-net analog. Left: an ordinary dense layer, where each neuron's incoming weights live on whatever scale training happened to leave them (hover a neuron: row scales differ by ~40× within one layer, so "this weight is big" is meaningless across neurons). Right: the same connectivity as a unit-net — every neuron's excitation is a stacked budget bar summing to exactly 1.0, every inhibitory synapse bounded by 1. Hover to inspect any neuron's budget; resample to see the invariant hold for every draw.

Why bother? Two reasons. First, the constraint set is a product of simplices and boxes — a bounded, well-shaped search space where the natural local moves (shift a bit of budget from one synapse to another; nudge one inhibitory weight) never leave the feasible set. That makes it an honest arena for gradient-free methods. Second — and this became the point of the whole research program — magnitudes get fixed units. A path product through the network is bounded and comparable to any other path product. "This is a strong connection" becomes an absolute statement rather than one relative to whatever scale that layer happened to learn.

What backprop had to survive

The oracle baseline is backprop through a constraint-satisfying reparameterization:

P = row-softmax(W), N = sigmoid(V)

where W and V are ordinary, unconstrained real-valued matrices — the actual trainable parameters. Row-softmax maps each row of W onto the probability simplex (guaranteeing the sum-to-1 constraint), and sigmoid squashes each entry of V into (0, 1) (guaranteeing the bound on inhibition). Backprop never touches P and N directly; it does gradient descent on W and V, and the constraints hold by construction.

Getting this to work surfaced four traps. None of them is exotic optimization lore — each is a direct, mechanical consequence of the geometry, and each cost us real debugging time:

1. Dead-ReLU initialization. By construction Pa ≤ 1 — a convex combination of inputs that are each at most 1 can't exceed 1. But there's no matching cap on total inhibition: a neuron with 784 inputs and N entries initialized near 0.5 (which is exactly what an innocent V ≈ 0 init gives you) receives inhibition on the order of hundreds. So z = Pa − Na is enormously negative for every neuron on every input, every ReLU outputs zero, the gradient is identically zero, and training never begins. The fix: initialize V strongly negative so N starts near zero, and let inhibition be earned during training.

2. The weight-decay trap. Reflexively adding weight decay to all parameters — standard hygiene — is actively destructive here, in a sneaky way. Decay pulls V toward 0, and sigmoid(0) = 0.5, so "regularization" silently grows every inhibitory synapse toward half strength, strangling the network it was meant to protect. The lesson generalizes well beyond unit-nets: penalties must be applied to the semantically meaningful object (here, L1 on N itself), never to the coordinates of a reparameterization — because the reparameterization's zero point can mean something entirely different from "small."

3. Vanishing logit spread. Convex rows are weighted averages, and an average of values in [0, 1] stays in [0, 1]. Stack two such layers and your ten output logits all land within a few hundredths of each other. Softmax over near-identical logits is near-uniform; cross-entropy against a near-uniform distribution produces gradients too small to be useful. The fix is a fixed multiplicative logit scale of 60–100 (an inverse temperature, effectively) so the loss has teeth. Nothing adaptive needed — but without it, training crawls regardless of optimizer.

4. Margin-memorization collapse. Each inhibitory synapse is bounded by 1, but a row of N can have hundreds of entries, so total inhibition per neuron is effectively unbounded. Late in training, the network discovers it can drive wrong-class logits to huge negative values on individual training examples — memorization by margin inflation. Train accuracy keeps improving while test accuracy falls off a cliff (97% → 80% within a few epochs). Validation-based model selection isn't hygiene here; it's mandatory. Notably, this is the geometry's one pathway to overfitting, and it's legible: you can watch the inhibition row-sums grow.

Tuned, backprop reaches 97.2% test at hidden width 64. An unconstrained twin, fairly tuned, reaches 97.7% — so the constraints cost gradient descent roughly half a point of ceiling, and, notably, produce the smallest train–test generalization gap of anything we ran (0.2 vs 0.5).

The annealer, and what actually mattered

The gradient-free contender: beam-search simulated annealing. Keep a beam of 8 networks; each step, spawn 31 mutants per beam member, score all 256 candidates on a shared minibatch (one batched GPU matmul), and Boltzmann-select survivors at an annealed temperature.³

The move vocabulary is where the interesting science happened, and it's worth walking the findings in the order they were earned:

Selection noise, not landscape ruggedness, caused every plateau. A 256-example scoring batch has cross-entropy noise comparable to the real differences between candidates — so selection was flipping coins between genuinely better and worse mutants. Growing the scoring batch over the run (256 → 2048) turned a hard plateau at 76% into monotone climbing. If your evolutionary method is stuck, check the noise floor of your fitness function before blaming the landscape.

Move design dominates schedule tuning. Ablations: two moves alone — single-inhibitory-weight nudges, plus moving δ of budget between two excitatory synapses of one neuron — matched the full move set at 1/6 the compute. Whole-row Gaussian noise, the "obvious" mutation operator, was dead weight.

Structured proposals beat temperature. Adding a "Gaussian blob" move — perturb a spatially contiguous patch of a first-layer receptive field, with the blob radius annealed from 6 pixels down to sub-pixel over the run — added +3.2 points at zero extra cost. This is convolution's inductive bias (nearby pixels belong together), injected through the proposal distribution instead of through weight sharing.

🕹️ Interactive Figure 2 — The blob move, annealing live. One first-layer receptive field (784 excitatory weights, drawn as a 28×28 heatmap), optimized by nothing but blob proposals under Boltzmann acceptance. Watch the radius schedule do its work: early wide blobs rough in mass where it belongs; late sub-pixel blobs sharpen edges. The ring flashes green on accepted proposals, rust on rejections — and you can watch the rejection rate climb as the temperature falls. This is the coarse-to-fine schedule the whole session kept rediscovering.

Topology beats geometry

This deserves its own section, because it's the day's most conceptually interesting result, and it comes in three rungs.

Rung 1 — geometry of the input. The blob move above uses pixel geometry: perturb weights that are neighbors on the image. Worth +3.2 points, as noted. Fine — but that prior only exists at the first layer, where weights have spatial addresses.

Rung 2 — imposed, arbitrary geometry. What about hidden layers, where neurons have no spatial arrangement at all? We imposed one: arrange the 64 hidden units on an 8×8 lattice — an arbitrary assignment, no relationship to function — and run blob moves on that lattice. This reached 90.4%, a solid gain over unstructured moves. Which is strange on its face: the lattice is fake. The resolution is that what a blob move really buys is correlated multi-synapse edits — moving a coherent group of parameters together, at a coherence scale you can anneal — and even a fake geometry supplies that.

Rung 3 — derived, functional topology. If any coherent grouping helps, the right grouping should help more — and the network can tell you what it is. Every ~1k steps: run the current network on data, record which hidden units co-activate, build the activation-correlation graph, and take its graph Laplacian L. Then propose "blobs" not on any spatial grid but as heat kernels on this graph — exp(−tL) applied to a random seed unit, i.e., a bump of perturbation that diffuses along the functional wiring, touching strongly-correlated units a lot and uncorrelated units not at all. The diffusion time t plays the role the radius played on the pixel grid, and anneals the same way.

Result: 91.4%, the session's gradient-free record. This is Hebb's principle — units that fire together — applied not to the weights but to the search operator: mutate together what fires together.

🕹️ Interactive Figure 3 — Lattice blobs vs functional blobs. Sixty-four hidden units, colored by their true functional community (which the search algorithm never sees directly). Left tab: the imposed 8×8 lattice — click a unit and the blob perturbs its grid neighbors, a mixed bag of unrelated functions. Right tab: the same units laid out by their activation-correlation graph — click the same unit and the heat kernel spreads along the wiring, recruiting exactly its co-firing partners. Drag the diffusion-time slider to watch the "radius" anneal. Same click, very different move.

One more schedule fact ties the section together: blob radii, edit counts, temperature, and even the mix of move types (blob-heavy early → transfer-heavy late) all want to anneal in concert; a run annealing the move mix alone hit 90.8% with no hidden-layer topology at all. Coarse-to-fine is the universal schedule.

First-layer excitatory rows. Backprop (left) learns sparse stroke templates; plain annealing (middle) leaves salt-and-pepper; topology-aware proposal moves (right) grow dense, contiguous receptive fields.

Interlude: delta evaluation, or how the annealer got 50× cheaper

A structural observation made everything after it affordable. A mutation touches one row — so scoring a mutant doesn't need a full forward pass. Cache the parent's activations once, then propagate only the rank-1 consequence of the edit: the perturbed row changes one pre-activation (dz = a·ΔW_row), which flows through a single outer product into the output layer. The result is exact (self-checked against full forward passes every run), costs ~50× fewer FLOPs, and turns each mutant from a 400 KB weight copy into a 3 KB edit descriptor.

This moved the annealing-vs-backprop resource gap from ~6,700× to ~140×, and every subsequent experiment ran on this engine. It also quietly enabled a free lunch elsewhere: since every proposal's Δloss is now measured anyway, an online bandit over move types gets its reward signal for free — and it matched hand-tuned move mixes with zero tuning, surfacing the "save inhibition edits for late in the run" schedule before we'd noticed it ourselves.

The information ladder

The bridge between blind search and backprop turns out to be classical. Ask "what is the gradient of the budget-transfer move?" — which direction of moving mass between two synapses most reduces loss — and you land on known objects: the single best transfer is a pairwise Frank–Wolfe step, and applying the transfer direction to all coordinates at once, multiplicatively, is exponentiated gradient, a member of the mirror descent family:

row ← row · e^{−ηg}, then renormalize the row to sum to 1

where g is the gradient of the loss with respect to that row's weights (a vector, one entry per synapse) and η is the step size. For readers who haven't met mirror descent: ordinary gradient descent takes steps that are natural for Euclidean geometry — fine in ℝⁿ, wrong on a simplex, where an additive step can walk off the feasible set or waste its length fighting the constraint. Mirror descent takes the gradient step in transformed coordinates chosen to match the geometry (the "mirror map"); for the simplex, the natural map is entropy, and the resulting update is exactly the multiply-by-exponentials-and-renormalize rule above — the same family as Hedge and multiplicative weights from online learning. Mass moves proportionally, the simplex is preserved by construction, and no reparameterization or softmax saturation is involved.

Lining everything up at matched budgets gives the ladder that anchors the day:

Each rung consumes a strictly richer signal and buys accuracy. Two null results sharpen the picture. Heat-kernel smoothing of gradients — the same spatial prior worth +3.2 points to blind search — is worth ≈ nothing (+0.4 max) once real gradients exist: priors substitute for information you don't have, and are redundant once you have it. And a REINFORCE-trained proposal policy rediscovered Hebb's rule from reward alone (it learned, from nothing but "did this proposal reduce loss," to propose mass toward high pre–post correlation) — yet slightly hurt the annealer, because pointing every proposal in the greedy direction collapses the diversity that selection feeds on.⁴

When does the ladder invert?

It doesn't — on clean float32 hardware. But the point of ranking methods by the channels they consume is to ask what happens when the hardware stops honoring a channel's assumptions.

Quantization. Fix the update quantum δ and the weight space becomes a finite lattice.⁵ Backprop-through-quantization (straight-through estimator) degrades gracefully as δ grows; the lattice-native annealer does something stranger — quantization helps it, with a sweet spot at 21 levels per weight (90.8%, the day's best flat-annealing result, using only three move types). And training natively on the lattice crushes training-in-float-then-rounding: 84.9% vs 66.3% at δ=0.2.

Analog noise. Now simulate device physics — and let's be precise about what is being read and written. Write noise: every time an updated weight is committed to storage (the physical memory cell, in the analog-hardware picture), the stored value lands multiplicatively perturbed (w·e^{σξ}); you pay only for rows you actually write. Read noise: every time the stored weights are read back to compute a forward pass, the resulting pre-activations come back perturbed (z += σξ). Two different channels, corrupted independently.

The symmetric law: gradient methods are robust to read noise and fragile to write noise; selection is the reverse — each method fails when the specific channel it depends on loses precision. Gradient methods batch-average away read noise, but their small, precise updates must survive storage: at moderate write noise (σ = 0.1–0.2), signSGD with its fixed step matched to the noise floor overtakes full mirror descent, because precision below the write-noise floor is worse than a blunt step. Selection writes rarely and coarsely (write-noise tolerant), but lives off tiny measured loss differences between candidates — differences ~100× smaller than a gradient method's batch-averaged signal — which read noise destroys first (90.8% → 66.3% from read noise alone). That read-noise fragility is the annealer's open weakness.

Deliberation. Finally, we tried making the search smarter rather than better-informed: an exact convex solver for the readout layer, and MCTS (Monte Carlo tree search) over weight edits. Both fail at the synapse level in instructive ways. Solving the readout to optimality mid-training collapses the search — at a block optimum the loss saturates and the selection landscape goes flat: a perfect readout is a bad teacher. Edit-level tree search burns its entire budget within ~10 edits of the root.

But macro-MCTS — where each tree node is not a single weight edit but a chunk of annealing under a chosen strategy (an operator mix × a step scale), so the tree plans the training schedule rather than the synapses — reached parity with the hand-tuned annealer at matched FLOPs. And it independently rediscovered the hand-designed curriculum: coarse spatial blobs alternating with functional-graph moves early, inhibition edits in the mid-game, ultra-fine budget transfers in the endgame — the same coarse-to-fine, excitation-before-inhibition schedule we'd converged on by hand (including the bandit's save-inhibition-for-late insight), plus a novel interleaving of feature-learning and readout phases we hadn't tried. Deliberation pays at the strategy level, never at the synapse level.

The thesis the day earned

Optimization methods are ranked by the information rate and precision of the channel they consume — full gradient magnitudes > gradient signs > candidate rankings — and the winner at any operating point is the method whose channel assumptions the substrate can still honor.

Clean digital hardware honors Adam's assumptions. Quantized weights and analog write noise progressively hand the advantage to sign-based steps and then to selection. Structural priors — spatial locality, imposed lattices, learned functional topology — are worth several accuracy points exactly when the search signal is information-poor, and approximately nothing once real gradients flow. And deliberation (solvers, tree search) pays at the level of strategy, never at the level of individual parameters.

Why this is post 1

On its own this would be a fun optimization study. What promoted it to the start of a series is what the geometry did for readability. Because every row is a budgeted convex combination minus bounded inhibition, magnitudes have fixed units end-to-end: composed weight products W₁·W₀ are directly comparable digit templates, and "strong path" means something absolute.

The full network rendered as wiring, W = P − N, for two models: backprop-trained vs annealed with functional-graph moves. Edge thickness is weight magnitude — and, the point, thicknesses are comparable across the entire figure, because every neuron spends the same budget. Two circuit styles are visible at a glance: backprop wires sparsely and surgically, concentrating each neuron's budget on a few strong, specific paths, so its network looks like a spare tree of bright cables. The annealed model instead learns broad, diffuse excitation and then suppresses whole functional communities at once with coordinated inhibition — you can see entire clusters of hidden units being switched off as groups, dark bundles where backprop has nothing. Same task, same accuracy regime, visibly different characters.⁶

The question that launched the next session: if fixed-unit path products are readable on MNIST, what happens when you put a language model in this geometry — does interpretability's hardest instrument (reading what an activation is disposed to make the model say) become exact? That's post 2.