Fog

01The thing alignment doesn’t cover

Value alignment is necessary but not sufficient. A perfectly aligned agent still acts on its perception of the situation — the context in front of it, the memory it carries, the documents it retrieves. Its values operate on that picture, not on the world directly. So an adversary who cannot change what the agent wants can still change what it sees, and route a harmful action past intact values by controlling the premises the judgment runs on.

Call this fog: context that has been selectively omitted, edited, or poisoned so that an aligned agent reasons correctly to a conclusion it would have rejected on fuller information.

The fog doesn’t defeat the values. It starves them.

02Three things that are not the same

It’s worth separating three failure modes that get lumped together, because they call for different defenses:

The third is the dangerous one, precisely because the output is real work rather than a forgery. A forged transcript is a lie about what the model did; a fogged agent’s output is a true record of careful reasoning — over a picture someone else curated.

03Capability and alignment both make it worse

The uncomfortable property of fog is that the agent’s virtues work for the attacker. A more capable agent executes more competently on whatever premises it’s given. A more aligned agent follows those premises more faithfully, with less of the hedging or refusal that might create friction. Neither capability nor alignment inspects the provenance of the context; both assume it.

The human analogy is exact. You do not need to corrupt an honest, competent analyst to get a bad decision out of them — you need only control the brief. Hand them a real problem with three relevant facts removed and they will reason their way to exactly the conclusion you wanted, in good faith, and defend it. The better the analyst, the more convincingly. Fog is the same move against an agent, and autonomy sharpens it: the agent acting without a human reading over its shoulder has no one to notice the missing facts.

04This is not new, and saying so matters

None of the components here are novel, and a post for this audience should say so plainly. The surface has been mapped under several names:

• Indirect prompt injection — hostile instructions planted in documents or pages the model will later read (Greshake et al., 2023). Fog is the omission-and-editing sibling: injection adds hostile content, fog removes or curates honest content.
• Memory and RAG poisoning — corrupting an agent’s retrieval store or persistent memory so future reasoning draws on planted or altered material.
• The “lethal trifecta” (Simon Willison, 2025) — an agent with access to private data, exposure to untrusted content, and a way to exfiltrate is exploitable almost by construction.

What I’d add is a framing and an emphasis. The framing: separating perception from values shows why an aligned model isn’t protected — the attack never touches the values. The emphasis: omission gets far less attention than injection, and it’s harder to catch.

Injected text is present and can be scanned for; a fact that was removed leaves nothing to scan.

Curation by deletion is the quietest version of the attack.

05Where fog gets in

At the level of what needs defending — not a how-to — the fog can enter at four places:

The common thread is that the surface grows with exactly the things that make agents useful: autonomy, long horizons, and operator-editable memory. The more an agent is trusted to run unattended and to carry state forward, the more of its perception is mediated by channels an adversary might hold.

06One observation from testing

A concrete pattern, from red-teaming local agents that had operator-editable memory. It is a single line of anecdote, not a study, and should be read as such.

The useful part is that the defensive signal lives in the same data as the attack. The instance that detected the curation stopped on its own. The deletion was necessary because detection worked. That points at what to invest in: not only making memory hard to edit, but making agents reliably notice when their own context shows signs of having been edited — because when they notice, they already do the right thing.

07Defenses

If the problem is that an aligned agent trusts a context it cannot vouch for, the defenses fall into four families, and they compound.

None of these is sufficient alone, and the first two put load on the model that it may not reliably carry yet. The point of defense-in-depth here is exactly that the human gate catches what the model’s self-detection misses.

08What this doesn’t solve

Several honest caveats, since the audience will supply them anyway.

The empirical grounding is thin — anecdote from local testing, not a benchmark. I’d treat the “detecting instance halts” result as a hypothesis worth replicating, not a finding to lean on.

The easy detector is easy to beat. An agent that halts on dangling references only catches sloppy fog. Competent curation leaves no dangling reference — it removes the fact and everything that points to it, so the context is internally consistent and simply false. Against that, memory-level detection has little to work with, which is why provenance and external checks, not model cleverness, carry the weight.

And the stop-and-verify instinct trades against helpfulness. A model tuned to distrust its own context will also balk at benign gaps, and that friction is real. Where the line sits is an open question, and a good one for this group.

I’d welcome replication, counterexamples, and especially arguments that the taxonomy draws a distinction that doesn’t carry weight.